A behavior recognition method, device, equipment and storage medium

Abstract

The embodiments of the present application provide a behavior recognition method, apparatus, device and storage medium, wherein the method comprises: obtaining a video frame sequence including an object to be recognized; determining, in the video frame sequence, a first image area set where the object to be recognized is located; based on the behavior category of the object to be recognized in the first image area set, adjusting the feature distance between different first image areas to obtain a second image area set; based on the second image area set, recognizing the behavior of the object to be recognized to obtain a recognition result.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer vision, and are related to but not limited to a behavior recognition method, apparatus, device and storage medium. Background Art

[0002] For human-centered video behavior recognition, the input video sequence is subjected to full-image data enhancement and then fed into the classification model for prediction. Since the video captured by the camera often contains more information and covers a larger field of view, the location of the target event of the pedestrian and the scale of the human body are also random, which affects the accuracy of behavior recognition. Summary of the invention

[0003] The embodiment of the present application provides a technical solution for updating a panoramic image.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides a behavior recognition method, the method comprising:

[0006] Acquire a video frame sequence including an object to be identified;

[0007] In the video frame sequence, determining a first image region set where the object to be identified is located;

[0008] Based on the behavior category of the object to be identified in the first image region set, adjusting the feature distances between different first image regions to obtain a second image region set;

[0009] Based on the second image area set, the behavior of the object to be identified is identified to obtain an identification result.

[0010] In some embodiments, determining the first image area set where the to-be-recognized object is located in the video frame sequence includes: detecting the to-be-recognized object in each video frame to obtain multiple detection frames of the to-be-recognized object; adjusting the areas of the multiple detection frames in each video frame to obtain multiple adjusted areas; and determining the first image area set in the multiple adjusted areas of each video frame. In this way, by adjusting the areas of the multiple detection frames in each video frame, a part of the multiple adjusted areas is selected as the first image area of ​​the video frame, which can reduce repeated recognition.

[0011] In some embodiments, determining the first image area set in the multiple adjusted areas of each video frame includes: determining a target adjusted area in the multiple adjusted areas of each video frame whose overlap with the reference adjusted area is greater than a preset overlap threshold and whose area is less than a preset area threshold; removing the target adjusted area from the multiple adjusted areas of each video frame to obtain the first image area set of each video frame; wherein the reference adjusted area is any adjusted area in the multiple adjusted areas whose first confidence is greater than a first confidence threshold. In this way, the amount of calculation for behavior recognition in the adjusted area can be reduced, and the accuracy of recognition can be improved by using the first image area with higher quality for behavior recognition.

[0012] In some embodiments, before the feature distances between different first image regions are adjusted based on the behavior categories of the objects to be identified in the first image region set to obtain the second image region set, the method further includes: selecting video frames less than a preset number of frames from the video frame sequence as target video frames; and classifying the behavior of the objects to be identified based on the first image region in each target video frame to obtain the behavior category of the objects to be identified. In this way, by selecting a small number of target video frames in the video frame sequence to classify the behavior of the object, the amount of calculation for behavior classification can be reduced.

[0013] In some embodiments, the feature distances between different first image regions are adjusted based on the behavior category of the object to be identified in the first image region set to obtain the second image region set, including: reducing the feature distances between the first image regions with the same behavior category and / or increasing the feature distances between the first image regions with different behavior categories to obtain the second image region set. In this way, the differences between the first image regions of different categories can be made greater, which facilitates the discrimination of the behavior category of the object to be identified.

[0014] In some embodiments, when the video frame in the video sequence includes at least one second image area, the behavior of the object to be identified is identified based on the set of the second image areas to obtain the identification result, including: determining that the behavior category of each second image area in the video frame is a second confidence level of a preset category; in at least one second image area of ​​the video frame, determining candidate image areas whose second confidence level is greater than or equal to a second confidence level threshold to obtain a set of candidate image areas; in the set of candidate image areas, eliminating candidate image areas whose spatial distance from the set of candidate image areas meets a preset condition to obtain a set of eliminated candidate image areas; based on the set of eliminated candidate image areas in the video frame, the behavior of the object to be identified is identified to obtain the identification result. In this way, the performance of subsequent behavior identification based on the set of eliminated candidate image areas is improved.

[0015] In some embodiments, the candidate image regions whose spatial distances from the candidate image region set meet preset conditions are eliminated from the candidate image region set to obtain the eliminated candidate image region set, including: determining a subset of candidate image regions whose second confidence is equal to a second confidence threshold in the candidate image region set; determining the spatial distance between any candidate image region and other candidate image regions in the candidate image region subset; wherein the other candidate image regions are candidate image regions in the candidate image region subset other than the any candidate image region; determining candidate image regions to be eliminated whose spatial distances are greater than a preset spatial distance threshold; and eliminating the candidate image regions to be eliminated from the candidate image region set to obtain the eliminated candidate image region set. In this way, by eliminating outlier candidate image regions in the candidate image region set, the consistency of the center points of the candidate image regions is improved, and multiple eliminated candidate image regions are made close to each other in space.

[0016] In some embodiments, the identifying the behavior of the object to be identified based on the set of candidate image regions after elimination in the video frame to obtain the identification result includes: determining at least one target region sequence in the video frame sequence based on the set of candidate image regions after elimination in at least one video frame; and identifying the behavior of the object to be identified in the at least one target region sequence to obtain the identification result. In this way, the behavior recognition network can be more focused on identifying the behavior of the object to be identified, and pay more attention to how to distinguish different movement details of the object to be identified, rather than the difference between the object to be identified and irrelevant objects.

[0017] In some embodiments, the determining at least one target region sequence in the video frame sequence based on the eliminated candidate image region set in at least one video frame includes:

[0018] Based on the set of eliminated candidate image regions, any eliminated candidate image region is selected from each video frame of the at least one video frame; the selected eliminated image regions are merged to obtain at least one merged region; in the video frame sequence, a target region sequence matching each merged region is determined to obtain the at least one target region sequence. In this way, region extraction is performed in the video frame sequence according to the merged region to obtain a target region sequence matching each merged region, which can make the picture content in the target region sequence more focused on the object to be identified itself, thereby improving the effective receptive field of practice.

[0019] In some embodiments, the identifying the behavior of the object to be identified in the at least one target area sequence to obtain the identification result includes: adjusting the side length of the target area in each target area sequence to a preset side length to obtain an adjusted target area sequence; identifying the behavior of the object to be identified in each adjusted target area sequence to obtain the identification result. In this way, by adjusting the side length of the target area in the target area sequence to a uniform length, subsequent behavior identification is facilitated, and the efficiency of behavior identification can be improved.

[0020] The present application provides a behavior recognition device, the device comprising:

[0021] A first acquisition module, used to acquire a video frame sequence including an object to be identified;

[0022] A first determination module, used to determine, in the video frame sequence, a first image region set where the object to be identified is located;

[0023] A first adjustment module, configured to adjust the feature distances between different first image regions based on the behavior categories of the objects to be identified in the first image region set, to obtain a second image region set;

[0024] The first recognition module is used to recognize the behavior of the object to be recognized based on the second image area set to obtain a recognition result.

[0025] Correspondingly, an embodiment of the present application provides a computer storage medium, on which computer executable instructions are stored. After the computer executable instructions are executed, the above-mentioned behavior recognition method can be implemented.

[0026] An embodiment of the present application provides an electronic device, which includes a memory and a processor, wherein the memory stores computer executable instructions, and the processor can implement the above-mentioned behavior recognition method when running the computer executable instructions on the memory.

[0027] The embodiments of the present application provide a behavior recognition method, apparatus, device and storage medium. In a video frame sequence of a video stream, a first image area set of an object to be recognized in each video frame is first determined; then, according to the behavior category of the object to be recognized in the first image area set, the feature distances between different first image areas are adjusted so that the obtained second image area set is more conducive to subsequent behavior recognition; finally, the behavior of the object to be recognized is recognized through the second image area set in the video frame to obtain a recognition result; in this way, the behavior of the object to be recognized is recognized based on the second image area after the feature distance is adjusted, which can improve the effective receptive field of the recognition network, thereby improving the accuracy of behavior recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of the implementation flow of the behavior recognition method provided in the embodiment of the present application;

[0029] Figure 2 A schematic diagram of another implementation flow of the behavior recognition method provided in an embodiment of the present application;

[0030] Figure 3 A schematic diagram of another implementation flow of the behavior recognition method provided in an embodiment of the present application;

[0031] Figure 4 A schematic diagram of another implementation flow of the behavior recognition method provided in the embodiment of the present application;

[0032] Figure 5 A schematic diagram of an application scenario of the behavior recognition method provided in an embodiment of the present application;

[0033] Figure 6 A schematic diagram of another application scenario of the behavior recognition method provided in an embodiment of the present application;

[0034] Figure 7 This is a schematic diagram of the structure of the behavior recognition device according to an embodiment of the present application;

[0035] Figure 8 It is a schematic diagram of the composition structure of the electronic device according to the embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the invention will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0037] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0038] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0040] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0041] 1) Computer vision refers to the use of cameras and computers to replace the human eye to identify, track and measure targets, and further perform graphic processing so that the computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection.

[0042] 2) Non-Maximum Suppression (NMS) searches for local maximum values ​​and suppresses the maximum value. Take target detection as an example. During target detection, a large number of candidate boxes will be generated at the same target location. These candidate boxes may overlap with each other. In this case, non-maximum suppression is needed to find the best target bounding box and eliminate redundant bounding boxes.

[0043] 3) Outliers, also known as escape values, refer to one or more values ​​in the data that are significantly different from other values. If the probability that a value deviates from the observed mean is less than or equal to 1 / (2n), the data should be discarded (where n is the number of observations and the probability can be estimated based on the distribution of the data). In the embodiment of the present application, an outlier can represent an image area that is spatially far away from other image areas.

[0044] The following describes an exemplary application of the behavior recognition device provided in the embodiment of the present application. The device provided in the embodiment of the present application can be implemented as various types of user terminals such as a laptop computer with data processing functions, a tablet computer, a desktop computer, a mobile device (for example, a personal digital assistant, a dedicated messaging device, a portable gaming device), and can also be implemented as a server. The following describes an exemplary application when the device is implemented as a terminal or a server.

[0045] The method can be applied to electronic devices, and the functions implemented by the method can be implemented by calling program codes by a processor in the electronic device. Of course, the program codes can be stored in a computer storage medium. It can be seen that the electronic device at least includes a processor and a storage medium.

[0046] The present application embodiment provides a behavior recognition method, such as Figure 1 As shown, combined with Figure 1 The following steps are explained:

[0047] Step S101: acquiring a video frame sequence including an object to be identified.

[0048] In some embodiments, in a video stream that includes an object to be identified, a sequence of video frames is determined. The video stream may be video data collected in any scene, for example, video data collected by a camera in the scene in any scene, or video data received from other devices. The object to be identified may be a movable object in the scene where the video stream is located; the object to be identified may be one or more objects, for example, if the video stream is collected for pedestrians, then the object to be identified is a pedestrian. If the video stream is collected in a traffic scene, then the object to be identified may be a vehicle in the traffic scene; if the video stream is an image collected on a grassland, then the object to be identified may be animals such as cattle and sheep on the grassland.

[0049] In some possible implementations, the video stream may be video data collected by a camera in a scene within a short period of time, such as a 3-second video stream collected for pedestrians. The video frame sequence of the video stream is a plurality of video frames obtained by sampling the video stream; for example, by sampling a 3-second video stream at equal intervals, 8 video frames, i.e., a video frame sequence, may also be obtained by randomly sampling a 3-second video stream.

[0050] Step S102: determining, in the video frame sequence, a first set of image regions where the object to be identified is located.

[0051] In each video frame of the video frame sequence, a first image region where the object to be identified is located in each video frame is determined. The first image region is obtained by preprocessing a detection frame of the object to be identified in the video frame, and the preprocessing process includes expanding the size of the detection frame and screening the expanded detection frame.

[0052] In some possible implementations, first, in each video frame of the video frame sequence, the object to be identified is detected to obtain a detection frame of each object to be identified in the video frame; if the video frame includes multiple objects to be identified, then the multiple objects to be identified are detected in the video frame to obtain a detection frame of each object to be identified, that is, multiple detection frames. Then, the length and width of each detection frame are adaptively expanded by a certain proportion. Finally, in each video frame, the multiple expanded detection frames are spatially non-maximum suppressed to eliminate redundant expanded detection frames. In this way, by expanding the detection frame of the object to be identified in the video frame, the first image area with higher picture quality is screened out, and redundant detection frames can be eliminated, thereby reducing the amount of calculation.

[0053] Step S103: adjusting the characteristic distances between different first image regions based on the behavior categories of the objects to be identified in the first image region set to obtain a second image region set.

[0054] In some embodiments, by adopting a binary classification model, the behavior category of the object to be identified in each first image area is identified to determine whether the behavior of the object to be identified is abnormal; in this way, the behavior category of the object to be identified includes abnormal and non-abnormal. The first image area set is divided into multiple subsets according to the behavior category of the object to be identified; for example, first image areas with the same behavior category are placed in one subset, and first image areas with different behavior categories are placed in another subset. The feature distance between different first image areas can be the feature distance between first image areas with the same behavior category, or it can be the feature distance between first image areas with different behavior categories.

[0055] In some possible implementations, the feature distances between first image regions of the same behavior category in the first image region set are shortened to implement regularization constraints on first image regions of the same behavior category; at the same time, the feature distances between first image regions of different behavior categories are extended to make first image regions of different behavior categories easier to distinguish. In this way, by shortening the feature distances between first image regions within a class in the first image region set and extending the feature distances between first image regions between classes, a second image region set with adjusted feature distances is obtained, thereby improving the discrimination ability of the binary classification model, making the obtained second image region set easier to identify the behavior of the object to be identified.

[0056] Step S104: Based on the second image area set, the behavior of the object to be identified is identified to obtain an identification result.

[0057] In some embodiments, the behavior of the object to be identified may be identified through all the second image areas in the second image area set to obtain a recognition result. In some possible implementations, the multiple second image areas obtained are merged according to different video frames, so that the behavior of the object to be identified can be identified based on the merged area to obtain the recognition result. The recognition result includes the category to which the behavior of the object to be identified belongs and the specific behavior content of the object to be identified (for example, the category is abnormal behavior, and the specific behavior content is a gathering of multiple people). For video frames including second image areas, select any second image area in each video frame, and spatially merge the second image areas selected from multiple video frames to obtain a merged area. Based on the merged area, the behavior of the object to be identified can be identified to obtain an accurate recognition result.

[0058] In some embodiments, the behavior of the object to be identified is also identified through some second image areas in the second image area set to obtain an identification result. In some possible implementations, some second image areas with a high confidence that the behavior category belongs to the preset behavior category are screened out from the second image area set. That is, the confidence that the behavior of the object to be identified belongs to the preset behavior is high. For the second image area of ​​each video frame, a second image area whose confidence in the category corresponding to the preset behavior in the classification result is greater than a certain threshold is selected. Based on the selected second image area set with a high confidence, the behavior category of the object to be identified is identified to obtain the identification result.

[0059] In an embodiment of the present application, in a video frame sequence of a video stream, firstly, a set of first image regions of an object to be identified in each video frame is determined; then, according to the behavior category of the object to be identified in the first image region set, the feature distances between different first image regions are adjusted to implement regularization constraints on the first image region set, so that the obtained second image region set is more conducive to subsequent behavior recognition; finally, the behavior of the object to be identified is identified through the second image region set in the video frame to obtain a recognition result; in this way, by identifying the behavior of the object to be identified based on the second image region after adjusting the feature distance, the effective receptive field of the recognition network can be improved, thereby improving the accuracy of behavior recognition.

[0060] In some embodiments, the detection frame of the object to be identified in the video frame is preprocessed to obtain the first image area in each video frame, that is, the above step S102 can be implemented by the following steps S121 to S123 (not shown):

[0061] Step S121 , detecting the object to be identified in each video frame to obtain multiple detection frames of the object to be identified.

[0062] In some embodiments, the objects to be identified are at least two objects. In each video frame of the video frame sequence, the objects to be identified are detected to obtain a detection frame of each object to be identified, that is, multiple detection frames.

[0063] In some possible implementations, a video frame sequence is input into a detection network to detect the object to be identified, and a detection result marked with a detection frame is output. For example, if the object to be identified is a plurality of pedestrians, then by detecting the plurality of pedestrians in each video frame, a detection frame of each pedestrian in the video frame is obtained, so that each video frame includes at least one detection frame.

[0064] Step S122, adjusting the areas of the multiple detection frames in each video frame to obtain multiple adjusted areas.

[0065] In some embodiments, the length and width of each detection frame are adaptively expanded according to the preset ratio to increase the coverage area of ​​the detection frame. For example, the length and width of each detection frame are adaptively expanded by 1.5 times to obtain the adjusted area corresponding to the detection frame. In the video frame where the detection frame is located, the side length of the detection frame is expanded by the preset ratio to obtain the adjusted area.

[0066] Step S123: determining the at least one first image region among the multiple adjusted regions in each video frame.

[0067] In some embodiments, part of the adjusted areas in the multiple adjusted areas of each video frame is selected as the at least one first image area. For example, one adjusted area is selected from the multiple adjusted areas in each video frame as the first image area in the video frame. Alternatively, multiple adjusted areas are screened in each video frame to obtain at least one first image area in the video frame. In this way, after adjusting the areas of multiple detection frames in each video frame, a part of the multiple adjusted areas is selected as the first image area of ​​the video frame, which can reduce repeated recognition.

[0068] In some embodiments, spatial non-maximum suppression is performed on multiple adjusted regions of each video frame to remove redundant regions in at least one adjusted region, thereby obtaining a first image region with higher image quality. That is, the above step S123 can be implemented by the following process:

[0069] In the first step, a target adjusted region is determined among the multiple adjusted regions of each video frame, wherein the overlap with the reference adjusted region is greater than a preset overlap threshold and the area is smaller than a preset area threshold.

[0070] In some embodiments, the reference adjusted region is any adjusted region of the multiple adjusted regions whose first confidence is greater than a first confidence threshold. The reference adjusted region may be an adjusted region with the highest confidence among the multiple adjusted regions. Among the multiple adjusted regions of the video frame, the adjusted region with the highest confidence of the detection frame is determined, i.e., the reference adjusted region; the reference adjusted region thus obtained is the region with the highest confidence in detecting the object to be identified, indicating that the clarity and completeness of the object to be identified detected in the reference adjusted region are the best.

[0071] In the multiple adjusted areas of each video frame, the overlap between each adjusted area and the reference adjusted area is determined, so as to determine the adjusted areas with an overlap greater than a preset overlap threshold, and further determine the adjusted areas with an area less than the preset area threshold among the adjusted areas with larger overlap, i.e., the target adjusted areas. The target adjusted areas may be one or more. In a video frame, if there are 5 adjusted areas, by determining the overlap between four of the adjusted areas and the reference adjusted area, if the overlap between two of the adjusted areas and the reference adjusted area is greater than the preset overlap threshold, then further determine the target adjusted area with an area less than the preset area threshold among the two adjusted areas.

[0072] In the second step, the target adjusted area is eliminated from the multiple adjusted areas of each video frame to obtain the first image area set of each video frame.

[0073] In some embodiments, target adjusted regions with high overlap and small area are deleted from the multiple adjusted regions of the video frame, and the remaining adjusted regions are used as a first image region set with higher quality.

[0074] In an embodiment of the present application, multiple detection frames of the object to be identified in each video frame are enlarged and then the adjusted area with higher quality is selected as the first image area; thereby, the amount of calculation for behavior recognition in the adjusted area can be reduced, and the behavior recognition can be performed using the first image area with higher quality, which can improve the accuracy of recognition.

[0075] In some embodiments, by selecting a small number of target video frames in the video frame sequence to classify the behavior of the object to be identified, the computational overhead can be reduced. That is, before the above step S103, the following steps may be included:

[0076] In the first step, video frames with a number less than a preset number of frames are selected from the video frame sequence as target video frames.

[0077] In some embodiments, the preset number of frames is less than the total number of frames in the video frame sequence, for example, the preset number of frames is set to be much smaller than the total number of frames. In the video frame sequence, a small number of video frames are selected as target video frames. In some possible implementations, the video frames arranged at preset positions in the video frame sequence are determined as target video frames, so that the same number of target video frames are determined at several preset positions. For example, the preset positions are the first position, the middle position, and the last position, then the target video frames include the first frame, the middle frame, and the last frame.

[0078] In the second step, based on the first image area in each target video frame, the behavior of the object to be identified is classified to obtain the behavior category of the object to be identified.

[0079] In some embodiments, the first image area in each target video frame is input into a behavior classification network, the behavior of the object to be identified is classified, and classification results of multiple first image areas in the target video frame are obtained. Taking the target video frame including the first frame, the middle frame and the last frame as an example, multiple first image areas in the first frame are input into the behavior classification network to obtain the classification result corresponding to each first image area in the first frame; at the same time, multiple first image areas in the middle frame and the last frame are respectively input into the behavior classification network to obtain the classification result corresponding to each first image area in the middle frame and the classification result corresponding to each first image area in the last frame.

[0080] In some possible implementations, whether the behavior of the object to be identified is abnormal behavior can be classified based on the first image area in each target video frame, and the classification result is the confidence that the behavior of the object to be identified in the first image area is abnormal behavior, and the confidence that the behavior of the object to be identified in the first image area is non-abnormal behavior. In this way, by selecting a small number of target video frames in the video frame sequence to classify the object behavior, the amount of calculation for behavior classification can be reduced.

[0081] In some embodiments, regularization constraints are imposed on the feature distances between different first image regions according to the behavior categories of the objects to be identified, so as to improve the discrimination ability of the binary classification model. Figure 2 The steps shown are explained below:

[0082] The method comprises the following steps S201:

[0083] Step S201 : reducing the feature distance between the first image regions with the same behavior category and / or increasing the feature distance between the first image regions with different behavior categories, to obtain the second image region set.

[0084] In some embodiments, in a video frame sequence, first, a rough prediction is made for the behavior category of the object to be identified in each first image area, and then, the feature distance of the first image areas of the same behavior category is reduced, or, the feature distance of the first image areas of different behavior categories is increased, or, on the basis of reducing the feature distance of the first image areas of the same behavior category, the feature distance of the first image areas of different behavior categories is further increased, or, on the basis of increasing the feature distance of the first image areas of different behavior categories, the feature distance of the first image areas of the same behavior category is further reduced; in this way, in the obtained second image area, features of the same category are brought closer, and features of different categories are pushed away, so that on the basis of the second image area, the behavior category of the object to be identified is identified, which can make the identification result more accurate.

[0085] In some possible implementations, the above step S201 may be implemented in the following two ways:

[0086] Method 1: First, the first image region is zoomed in within the class, and then zoomed out between the classes to obtain the second image region set, that is, the above step S201, which can be implemented by the following steps S211 and S212 (not shown in the figure):

[0087] Step S211 : reducing the feature distance between the first image regions with the same behavior category to obtain a first constrained region set.

[0088] In some embodiments, the first image region set is divided according to the behavior category. For the first image regions with the same behavior category, the Euclidean distance between the feature vectors of these first image regions or the mean square distance between the feature vectors is determined; the feature distance between the first image regions with the same behavior category is reduced, for example, the feature distance between the first image regions with the same behavior category is shortened, which can be arbitrarily reducing the Euclidean distance or the mean square distance between the feature vectors of the first image regions, or reducing the Euclidean distance or the mean square distance to a certain value.

[0089] In some possible implementations, if the behavior categories include abnormal and non-abnormal categories, then the feature distance between the first image regions with the same behavior category belonging to the abnormal category is shortened to obtain multiple first constrained regions with the behavior category belonging to the abnormal category; at the same time, the feature distance between the first image regions with the behavior category belonging to the non-abnormal category is also shortened to obtain multiple first constrained regions with the behavior category belonging to the non-abnormal category; thereby obtaining a first constrained region set.

[0090] Step S212: increasing the characteristic distance between the first constrained regions with different behavior categories to obtain the second image region set.

[0091] In some embodiments, for first constrained regions of different behavior categories, the Euclidean distance between the feature vectors of these first constrained regions is determined; the feature distance between the first constrained regions of different behavior categories is increased, for example, the feature distance between the first constrained regions of different behavior categories is pushed further away, and the Euclidean distance between the first constrained regions of different behavior categories can be arbitrarily increased; for example, the Euclidean distance is increased to a certain value.

[0092] In some possible implementations, based on the first constrained region set, if the behavior categories include abnormal and non-abnormal, the feature distance between the first constrained region with abnormal behavior category and the first constrained region with non-abnormal behavior category is extended to obtain the first constrained region obtained after adjusting the feature distance between the two first constrained regions, thereby obtaining the second image region set. In this way, by shortening the feature distance between first image regions of the same category and extending the feature distance between first image regions of different categories, the difference between first image regions of different categories can be made greater, which is convenient for the binary classification model to distinguish.

[0093] Method 2: Alternatively, the second image region set may be obtained by first performing inter-class extension and then intra-class extension on the first image region, including the following steps S213 and S214 (not shown):

[0094] Step S213: increasing the characteristic distance between the first image regions with different behavior categories to obtain a second constrained region set.

[0095] Here, the implementation process of step S213 is similar to step S212, that is, the Euclidean distance between the first image regions with different behavior categories is appropriately increased to make the difference between the first image regions with different categories larger.

[0096] Step S214: reducing the feature distance between the second constrained regions with the same behavior category to obtain the second image region set.

[0097] Here, the implementation process of step S214 is similar to step S211, that is, the Euclidean distance between the first image regions with the same behavior category is appropriately reduced so as to make the difference between the first image regions with the same category smaller.

[0098] In some embodiments, when a video frame in a video sequence includes at least one second image region, by selecting a candidate image region with a larger confidence from the second image regions of multiple video frames and removing outlier candidate image regions, the candidate image regions for identifying the behavior of the object to be identified are made more effective, that is, the above step S104 can be performed by Figure 3 The steps shown achieve:

[0099] Step S301 : determining a second confidence level that the behavior category of each second image region in the video frame is a preset category.

[0100] In some embodiments, for each video frame in a video frame sequence (the video frame may be a target video frame in the video frame sequence, or may be any video frame in the video frame sequence), a second confidence level that the behavior category of each second image area in the video frame is a preset category is determined. For example, if there are three second image areas in a video frame, a second confidence level that the behavior category of the third second image area is a preset category is determined. The preset category is determined based on the categories included in the classification process. For example, if the classified categories include abnormal behavior and non-abnormal behavior, then the preset category may be abnormal behavior. In a specific example, if the object to be identified is a vehicle, a traffic accident is set as abnormal behavior, then the confidence level that a traffic accident occurs in each second image area is determined. If the object to be identified is a pedestrian, a gathering of pedestrians is set as abnormal behavior, then the confidence level that a gathering of pedestrians occurs in each second image area is determined.

[0101] Step S302: determining, in at least one second image region of the video frame, candidate image regions whose second confidence level is greater than or equal to a second confidence level threshold, to obtain a set of candidate image regions.

[0102] In some embodiments, the second confidence threshold is greater than or equal to the minimum second confidence corresponding to the video frame, or the second confidence threshold is customized to a larger value, for example, the second confidence threshold is set to 0.8. In at least one second image region of the video frame, a second image region whose behavior category is a preset category and whose confidence is greater than a preset confidence threshold is used as a candidate image region; thereby, at least one candidate image region with a higher confidence can be selected in each video frame.

[0103] In some possible implementations, taking the video frame as the target video frame as an example, the user customizes the second confidence threshold; or, different confidence thresholds can be set for different target video frames, and the second confidence threshold can be set based on the minimum second confidence corresponding to at least one second image region of the target video frame, for example, the confidence threshold is set to be greater than or equal to the minimum second confidence; thus, by analyzing the second confidence, the second image region with a higher second confidence in the target video frame can be screened out from at least one second image region of each target video frame, and the second image region with a higher second confidence is used as a candidate image region; thus, a set of candidate image regions is obtained. In this way, by selecting the second image region with a higher second confidence in the target video frame as the candidate image region, the performance of subsequent behavior recognition based on the candidate image region can be improved.

[0104] Step S303 , in the candidate image region set, the candidate image regions whose spatial distances from the candidate image region set meet a preset condition are eliminated to obtain a eliminated candidate image region set.

[0105] In some embodiments, the preset condition is to select candidate image regions as outliers of the candidate image region set, for example, any candidate image region from the candidate image region set is determined to be an outlier if the spatial distance between the spatial position and the spatial position of other candidate image regions is greater than a certain threshold (for example, the threshold is set to be greater than or equal to the average distance between multiple candidate image regions). In the candidate image region set, the outlier candidate image regions are eliminated, that is, the eliminated candidate image region set is obtained.

[0106] In some possible implementations, candidate image regions with a large spatial distance from other candidate image regions are eliminated from the candidate image region set, so that the center points of the image regions in the eliminated candidate image region set are more consistent, that is, the above step S303 can be implemented by the following steps S331 to S334 (not shown):

[0107] Step S331: Determine, in the candidate image region set, a subset of candidate image regions whose second confidence is equal to a second confidence threshold.

[0108] In some embodiments, the second confidence of the candidate image region is equal to the second confidence threshold, indicating that the possibility that the behavior category of the candidate image region is a preset category is small, or that the behavior category of the candidate image region is likely to be the preset category in the end.

[0109] In some possible implementations, the candidate image regions whose second confidence is equal to the second confidence threshold are screened out from the candidate image region set, that is, a candidate image region subset is obtained; it can also be that in the target video frames of the video frame sequence, each target video frame predicts the second image region that may eventually be the candidate image region; in this way, the candidate image region subset can be obtained based on multiple target video frames. For example, there are 3 target video frames, and a second image region that may eventually be the candidate image region is predicted in each target video frame, so that a candidate image region subset including three candidate image regions can be obtained.

[0110] Step S332: determining the spatial distance between any candidate image region and other candidate image regions in the candidate image region subset.

[0111] In some embodiments, the other candidate image regions are candidate image regions other than any candidate image region in the candidate image region subset. In the candidate image region subset, for each candidate image region, the spatial distance between the center point of the candidate image region and a plurality of other candidate image regions in spatial position is determined; or, for each candidate image region, the average value of the spatial distance between each candidate image region and each other candidate image region is determined, and the average value is used as the spatial distance between any candidate image region and the other candidate image regions. The spatial distance can be used to determine whether the candidate image region is far away from the remaining other candidate image regions.

[0112] Step S333: determining the candidate image regions to be eliminated whose spatial distance is greater than a preset spatial distance threshold.

[0113] In some embodiments, the preset spatial distance threshold may be set based on the average distance of the spatial distances between multiple candidate image regions in the candidate image region subset, for example, the preset spatial distance threshold is set to be greater than or equal to the average distance. If the spatial distance between the center of a candidate image region and other candidate image regions is greater than the preset spatial distance threshold, it means that the candidate image region is an outlier in the candidate image region subset. In a specific example, if the candidate image region subset includes 6 candidate image regions, and the spatial distance between one candidate image region and the other candidate image regions is greater than the preset spatial distance threshold, it means that the candidate image region is not within a reasonable range and has a relatively large deviation.

[0114] Step S334: Eliminate the candidate image regions to be eliminated from the candidate image region set to obtain the eliminated candidate image region set.

[0115] In some embodiments, in the candidate image area set, the candidate image areas to be eliminated with large deviations are eliminated, so that the center points of the candidate image area set after elimination are consistent. In a specific example, if the candidate image area subset includes 6 candidate image areas, and these 6 candidate image areas are scattered in different spatial positions, then the minimum bounding box that can surround the 6 candidate image areas will be very large, and the purpose of locating the local area cannot be achieved. Therefore, the candidate image area subset is clustered, and the candidate image areas that deviate far are discarded; for example, more than half of the candidate image areas in the 6 candidate image area subsets are within a reasonable spatial range, and only a few have more deviations due to inaccurate predictions. In the end, these outlier candidate image areas are discarded when taking the minimum bounding box to improve the accuracy of the candidate image area set after elimination. In this way, by eliminating the outlier candidate image areas in the candidate image area set, the consistency of the center points of the candidate image areas is improved, and multiple eliminated candidate image areas are close to each other in space.

[0116] Step S304: Based on the eliminated candidate image region set in the video frame, the behavior of the object to be identified is identified to obtain the identification result.

[0117] In some embodiments, by selecting one of the eliminated candidate image regions from different video frames for merging, the behavior of the object to be identified is identified through the merged region to obtain the identification result of the object to be identified. In this way, by selecting the second image region with a higher second confidence level in the video frame as the candidate image region, and eliminating the outlier candidate image region in the candidate image region set, the performance of subsequent behavior recognition based on the eliminated candidate image region set is improved.

[0118] In some embodiments, when the video frame includes at least one frame, by merging any second image regions in different video frames, behavior recognition is performed in the original video frame sequence based on the merged image region to improve the perception of the effective region during the recognition process, that is, the above step S304 can be implemented by the following steps S341 and S342 (not shown in the figure):

[0119] Step S341 : determining at least one target region sequence in the video frame sequence based on the eliminated candidate image region set in at least one video frame.

[0120] In some embodiments, each video frame includes at least one candidate image region after elimination. Any candidate image region after elimination of different video frames is spatially merged, and regions are extracted from the original video frame sequence according to the merged regions to obtain a target region sequence. In this way, multiple target region sequences can be extracted from the video frame sequence based on multiple merged regions.

[0121] In some possible implementations, taking the at least one video frame as at least one target video frame as an example, a target region sequence is obtained by merging a candidate image region selected after elimination in each target video frame, and performing region extraction in the original video frame sequence according to the merged region; that is, the above step S341 can be implemented by the following steps:

[0122] In the first step, based on the set of eliminated candidate image regions, any eliminated candidate image region is selected from each video frame of the at least one video frame.

[0123] In some embodiments, a candidate image region after elimination is randomly selected in each video frame, so that there are several video frames, that is, several candidate image regions after elimination are obtained. Taking the target video frame as an example, for example, the target video frame includes the first frame, the middle frame and the last frame, then a candidate image region after elimination is selected in the first frame, the middle frame and the last frame respectively, and a sequence including three candidate image regions after elimination is obtained; if the first frame, the middle frame and the last frame each include two candidate image regions after elimination, one candidate image region after elimination is selected in the first frame, the middle frame and the last frame respectively, then four candidate image region sequences after elimination can be obtained, and each candidate image region sequence after elimination includes three candidate image regions after elimination.

[0124] The second step is to merge the selected removed image regions to obtain at least one merged region.

[0125] In some embodiments, by selecting a removed image region in each video frame, the selected removed image regions are merged, so that multiple merged regions can be obtained based on the second image regions in multiple video frames. Multiple removed image regions in each removed image region sequence are spatially merged to obtain a merged region corresponding to the removed image region sequence, so that the number of merged regions is the same as the number of sets.

[0126] In some possible implementations, multiple eliminated image regions in a eliminated image region sequence may be enclosed in a frame, and the region covered by the frame is the merged region.

[0127] In the third step, in the video frame sequence, a target region sequence matching each merged region is determined to obtain the at least one target region sequence.

[0128] In some embodiments, for any merged region, a cutout is performed in the video frame sequence according to the merged region to extract an image region corresponding to the merged region, thereby obtaining a target region sequence; thus, several merged regions can determine the same number of target region sequences. The target region sequence can characterize the behavior trajectory of the object to be identified.

[0129] In an embodiment of the present application, by selecting a removed image area in each video frame, merging the selected removed image areas, and performing area extraction in the video frame sequence according to the merged areas, a target area sequence matching each merged area is obtained, which can make the picture content in the target area sequence more focused on the object to be identified itself, thereby improving the effective receptive field in practice.

[0130] Step S342: identifying the behavior of the object to be identified in the at least one target area sequence to obtain the identification result.

[0131] In some embodiments, since the image of the target area in the target area sequence is focused on the object to be identified, the interference of most irrelevant information in the image is reduced. Therefore, by inputting multiple target area sequences into the behavior recognition network, the behavior recognition network can be more focused on identifying the behavior of the object to be identified, and pay more attention to how to distinguish the different motion details of the object to be identified, rather than the difference between the object to be identified and irrelevant objects.

[0132] In some possible implementations, the side lengths of the target areas are adjusted so that the side lengths of the target areas in the target area sequence are uniform, which facilitates behavior recognition of the objects to be recognized in the target area sequence. That is, the above step S342 can be implemented by the following steps:

[0133] In the first step, the side length of the target area in each target area sequence is adjusted to a preset side length to obtain an adjusted target area sequence.

[0134] In some embodiments, the side length of each target area is adjusted using a preset side length to obtain an adjusted target area. For a target area in any target area sequence, the length and width of the target area are adjusted to the preset side length, for example, the length and width of the target area are adjusted to 224, so that the size of the adjusted target area in the adjusted target area sequence is 224×224.

[0135] The second step is to identify the behavior of the object to be identified in each of the adjusted target area sequences to obtain the identification result.

[0136] In some embodiments, by inputting each adjusted target area sequence into a behavior recognition network for behavior recognition, the confidence level of whether the behavior of the object to be recognized is a preset behavior is obtained; in this way, by adjusting the side lengths of the target areas in the target area sequence to a uniform length, subsequent behavior recognition is facilitated, thereby improving the efficiency of behavior recognition.

[0137] Below, an exemplary application of an embodiment of the present application in an actual application scenario will be described, with the example of identifying preset behaviors of pedestrians in a video captured in a complex scene.

[0138] Anomaly detection in videos is an important problem in the field of computer vision and has a wide range of applications in the field of video management, such as detecting traffic accidents and some unusual events. Thousands of video acquisition cameras are deployed around the world. However, most cameras only record the dynamics of each moment and do not have the ability to automatically manage (often requiring manual inspection by personnel). Due to the huge number of videos, it is obviously unrealistic to filter the content in the video by manpower alone. Therefore, it is necessary to use computer vision and deep learning technology to automatically detect abnormal events occurring in the video.

[0139] In the related art, it is extremely difficult to identify preset behaviors in videos, for example, due to the scarcity of labeled data caused by low-probability events, large inter-class / intra-class variance, differences in the subjective definition of abnormal events, low resolution of managed videos, etc.

[0140] For the detection of preset behaviors in video scenes, how to accurately locate the area where the preset behaviors occur in the entire picture of the video frame sequence (under different viewing angles), so as to replace the entire picture with the local area and input it into the recognition network for behavior classification, will help improve the machine's effective perception range for the target event, reduce the interference of most irrelevant information in the picture, and make the model more focused on how to distinguish the different movement details of the subject, rather than the differences between the target group and irrelevant passers-by. At the same time, it supports common indoor and outdoor scenes such as urban streets and rail transit, so that the automatic analysis of preset behaviors in video content provides convenient services for users.

[0141] In the related art, in the scenario of behavior recognition, the input video sequence is subjected to data enhancement or other preprocessing of the entire image and then sent to the classification model for prediction. However, this method is only applicable to human-centered video behavior recognition. For videos shot by cameras, they often contain more information and cover a larger field of view. At the same time, the location where the preset behavior occurs and the scale of the human body are also random. Therefore, it is obviously unreasonable to simply use the entire image as the model input. In this way, by introducing category-related prior information to extract the area, the results of each frame are unstable, which can easily lead to an excessively large extraction range.

[0142] Based on this, the present application embodiment provides a behavior recognition method, such as Figure 4 As shown, Figure 4 Another implementation flow diagram of the behavior recognition method provided in the embodiment of the present application, combined with Figure 4 The steps shown are explained below:

[0143] Step S401: extracting a pedestrian detection frame in each video frame in a video frame sequence.

[0144] In some embodiments, video data captured by a camera is obtained, and a full-image video frame sequence is obtained by sampling the video data; a detection frame of a pedestrian in the video frame sequence is extracted by calling an upstream structured detection model, such as Figure 5 As shown in the pedestrian detection 501 and 502 in FIG.

[0145] Step S402: enlarge each detection frame by m times to obtain an adjusted area corresponding to each detection frame.

[0146] In some embodiments, the length and width of each detection frame are expanded m times to expand the area of ​​the detection frame. For example, m can be set to 1.5. In this way, the image area corresponding to each detection frame is the area obtained by expanding the length and width of the detection frame m times. Figure 5 As shown, the pedestrian detection frame 501 is enlarged m times to obtain the adjusted area 511, and the pedestrian detection frame 502 is enlarged m times to obtain the adjusted area 521.

[0147] Step S403: sort each adjusted region based on the area of ​​the adjusted region to obtain a sorting result.

[0148] In some embodiments, the sorting result is obtained by arranging the enlarged detection frames in descending order according to the areas of the adjusted regions.

[0149] Step S404: Based on the sorting result, spatial non-maximum suppression is performed on multiple adjusted regions in the same frame to obtain a first image region in the same frame.

[0150] In some embodiments, since the collected video is collected for pedestrians, there may be people close to each other in the same frame, and the corresponding adjusted areas may overlap in space. In order to reduce repeated recognition, non-maximum suppression of the adjusted areas is performed, and multiple adjusted areas are sorted in descending order according to their areas. In the same frame, the adjusted areas with high overlap and small area are discarded to obtain the first image area in the frame.

[0151] Step S405, performing binary classification on the multiple first image regions corresponding to the first frame, the middle frame and the last frame of the video frame sequence, and based on the classification results, narrowing the intra-class feature distances of the multiple first image regions and expanding the inter-class feature distances.

[0152] In some embodiments, the intra-class feature distances of multiple first image regions corresponding to the first frame, the middle frame, and the last frame in the video frame sequence are shortened, and the inter-class feature distances are extended. The multiple first image regions corresponding to the processed first frame, the middle frame, and the last frame are input into the binary classification model for binary classification to determine whether the picture content in the first image region is an abnormal class or a non-abnormal class. The classification result includes the confidence that each first image region belongs to the abnormal class. In this way, the regularization constraint of extending the feature distance of the image region between classes and shortening the feature distance of the image region within a class is introduced during the training process, which can improve the discrimination ability of the binary classification model.

[0153] Step S406: Based on the classification result, second image regions whose confidence of abnormal class is greater than a preset confidence threshold are determined from the plurality of first image regions after feature distance adjustment to obtain a set of second image regions.

[0154] In some embodiments, after the classification network has identified multiple first image regions, the score of each first image region belonging to the abnormal class is output, and the first image region whose score ranks in the top half of each frame in the first frame, the middle frame, and the last frame after the feature distance adjustment is used as the second image region. Figure 6 As shown, the second image region combination includes second image regions 61 to 68; wherein the second image region 64 is an image region where detection fails, that is, no valid pedestrian is detected in the image region.

[0155] Step S407, removing the second image regions corresponding to the outliers in the second image region set to obtain a third image region set, taking one third image region each from the first frame, the middle frame and the last frame to merge, to obtain multiple merged regions.

[0156] In some embodiments, in the second image region set, outlier second image regions are eliminated, and second image regions that are close to each other in space are retained, that is, a third image region set is obtained. In the third image region set, a third image region is taken from each of the first frame, the middle frame, and the last frame, and the three taken third image regions are taken as a union to obtain a merged region; and so on, multiple merged regions are obtained. Figure 6 As shown, a merged region 601 , a merged region 602 , and a merged region 603 are obtained by merging a third image region in different frames among the second image regions 61 to 68 .

[0157] Step S408 , extracting a corresponding image region sequence from the video frame sequence based on the merged region to obtain a plurality of target region sequences.

[0158] In some embodiments, corresponding regions of the merged regions are cut out on the original video frame sequence to obtain multiple target region sequences.

[0159] In some possible implementations, taking the determination of the Kth target area as an example, the long side of the Kth target area is scaled to 224, the short side of the Kth target area is scaled proportionally, and the area less than 224 in the target area is padded with black borders at the top and bottom, and the final size of the target area is 224×224.

[0160] Step S409: identifying the behavior of pedestrians in the picture based on the multiple target area sequences.

[0161] In some embodiments, multiple target region sequences are input into a video classification network to identify whether the behavior of pedestrians in the multiple target regions is abnormal. Figure 5 As shown, first, after the adjusted area 511 and the adjusted area 521 are subjected to spatial non-maximum suppression, the obtained first image area is input into the classification network model to obtain the score of abnormal behavior of pedestrians in each first image area; then, the first image area with a score greater than a preset threshold in the classification result is taken as the second image area to obtain a second image area set; the second image areas in the same frame are merged, and the image area is extracted in the original frame based on the merged area; finally, the target area sequence is input into the network model 503 to identify whether the area includes abnormal behavior of pedestrians, and the score 512 of abnormal behavior of pedestrians in each target area sequence is obtained, and the target area with a score greater than the preset threshold is output in the set 504; the target area with a score less than the preset threshold is output in the set 505. In this way, the perception ability of the network model to the target area in the video is effectively improved, the search scope and the amount of calculation are greatly reduced, and the preprocessing estimation results of the network model for different abnormal behavior labels are more stable, the effective receptive field of the event is improved, and the recall rate of the preprocessing is improved.

[0162] In the embodiment of the present application, the target area is determined based on the pedestrian detection frame and the image area that can be learned and preprocessed, which increases the effective perception area of ​​the model and reduces the search range of irrelevant backgrounds; and the image feature distances of the first image area after non-maximum suppression in the first, last, and middle frames are respectively shortened within the class and pushed away between classes in a learnable manner, and the processed image area is input into the binary classification model for abnormal behavior recognition, which can improve the model's ability to distinguish and discriminate in complex and dense scenes, thereby improving recognition accuracy; the abnormal second image area with a higher score predicted by each frame is used, and the outlier image area in the second image area set is eliminated to improve the consistency and robustness of the center point. The third image area in the third image area set is merged into the corresponding target area, and the behavior of the target area is recognized separately, which can improve the recall rate of the second image area and the effective perception area of ​​the network model for the event.

[0163] The present application embodiment provides a behavior recognition device, Figure 7 Schematic diagram of the structure of the behavior recognition device in the embodiment of the present application. Figure 7 As shown, the behavior recognition device 700 includes:

[0164] A first acquisition module 701 is used to acquire a video frame sequence including an object to be identified;

[0165] A first determination module 702 is used to determine, in the video frame sequence, a first image region set where the object to be identified is located;

[0166] A first adjustment module 703, configured to adjust the characteristic distances between different first image regions based on the behavior category of the object to be identified in the first image region set, to obtain a second image region set;

[0167] The first recognition module 704 is used to recognize the behavior of the object to be recognized based on the second image area set to obtain a recognition result.

[0168] In some embodiments, the first determining module 702 includes:

[0169] A first detection submodule, configured to detect the object to be identified in each video frame to obtain multiple detection frames of the object to be identified;

[0170] A first adjustment submodule, configured to adjust the areas of the plurality of detection frames in each video frame to obtain a plurality of adjusted areas;

[0171] The first determining submodule is used to determine the first image area set among the multiple adjusted areas of each video frame.

[0172] In some embodiments, the first determining submodule includes:

[0173] A first determining unit is used to determine, among the multiple adjusted regions of each video frame, a target adjusted region whose overlap with the reference adjusted region is greater than a preset overlap threshold and whose area is smaller than a preset area threshold;

[0174] A first elimination unit, configured to eliminate the target adjusted area from the multiple adjusted areas of each video frame to obtain the first image area set of each video frame;

[0175] The reference adjusted region is any adjusted region of the multiple adjusted regions whose first confidence is greater than a first confidence threshold.

[0176] In some embodiments, the apparatus further comprises:

[0177] A first selection module, configured to select video frames less than a preset number of frames from the video frame sequence as target video frames;

[0178] The first classification module is used to classify the behavior of the object to be identified based on the first image area in each target video frame to obtain the behavior category of the object to be identified.

[0179] In some embodiments, the first adjustment module 703 is further configured to:

[0180] The feature distance between the first image regions with the same behavior category is reduced and / or the feature distance between the first image regions with different behavior categories is increased, so as to obtain the second image region set.

[0181] In some embodiments, when a video frame in the video sequence includes at least one second image region, the first identification module 704 includes:

[0182] A second determination submodule, configured to determine a second confidence level that the behavior category of each second image area in the video frame is a preset category;

[0183] A second determination submodule is used to determine, in at least one second image region of the video frame, a candidate image region whose second confidence is greater than or equal to a second confidence threshold, to obtain a set of candidate image regions;

[0184] A first elimination submodule is used to eliminate, from the candidate image region set, candidate image regions whose spatial distances from the candidate image region set meet a preset condition, to obtain a set of eliminated candidate image regions;

[0185] The first recognition submodule is used to recognize the behavior of the object to be recognized based on the eliminated candidate image area set in the video frame to obtain the recognition result.

[0186] In some embodiments, the first elimination submodule includes:

[0187] A second determining unit, configured to determine, from the candidate image region set, a subset of candidate image regions whose second confidence is equal to a second confidence threshold;

[0188] a third determining unit, configured to determine a spatial distance between any candidate image region and other candidate image regions in the candidate image region subset; wherein the other candidate image regions are candidate image regions in the candidate image region subset other than the any candidate image region;

[0189] A fourth determining unit, configured to determine a candidate image region to be eliminated whose spatial distance is greater than a preset spatial distance threshold;

[0190] The second elimination unit is used to eliminate the candidate image regions to be eliminated from the candidate image region set to obtain the eliminated candidate image region set.

[0191] In some embodiments, the first identification submodule includes:

[0192] a fifth determining unit, configured to determine at least one target region sequence in the video frame sequence based on the eliminated candidate image region set in at least one video frame;

[0193] The first recognition unit is used to recognize the behavior of the object to be recognized in the at least one target area sequence to obtain the recognition result.

[0194] In some embodiments, the fifth determining unit includes:

[0195] A first selection subunit, configured to select any eliminated candidate image region from each video frame of the at least one video frame based on the eliminated candidate image region set;

[0196] A first merging subunit, configured to merge the selected removed image regions to obtain at least one merged region;

[0197] The first determining subunit is used to determine, in the video frame sequence, a target region sequence matching each merged region to obtain the at least one target region sequence.

[0198] In some embodiments, the first identification unit includes:

[0199] A first adjustment subunit is used to adjust the side length of each target area in the target area sequence to a preset side length to obtain an adjusted target area sequence;

[0200] The first recognition subunit is used to recognize the behavior of the object to be recognized in each of the adjusted target area sequences to obtain the recognition result.

[0201] It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0202] It should be noted that in the embodiment of the present application, if the above-mentioned behavior recognition method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable an electronic device (which can be a terminal, server, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a sports hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0203] Correspondingly, an embodiment of the present application further provides a computer program product, which includes computer executable instructions. After the computer executable instructions are executed, the steps in the behavior recognition method provided in the embodiment of the present application can be implemented.

[0204] Accordingly, an embodiment of the present application further provides a computer storage medium, on which computer executable instructions are stored. When the computer executable instructions are executed by a processor, the steps of the behavior recognition method provided in the above embodiment are implemented.

[0205] Accordingly, an embodiment of the present application provides an electronic device, Figure 8 Schematic diagram of the structure of the electronic device according to the embodiment of the present application. Figure 8As shown, the electronic device 800 includes: a processor 801, at least one communication bus, a communication interface 802, at least one external communication interface and a memory 803. The communication interface 802 is configured to realize the connection and communication between these components. The communication interface 802 may include a display screen, and the external communication interface may include a standard wired interface and a wireless interface. The processor 801 is configured to execute the image processing program in the memory to implement the steps of the behavior recognition method provided in the above embodiment.

[0206] The description of the above behavior recognition device, electronic device, and storage medium embodiments is similar to the description of the above method embodiments, and has similar technical descriptions and beneficial effects as the corresponding method embodiments. Due to space limitations, the above method embodiments can be used as examples, so they will not be repeated here. For technical details not disclosed in the behavior recognition device, electronic device, and storage medium embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0207] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0208] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0209] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0210] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0211] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional units. A person of ordinary skill in the art can understand that all or part of the steps of the above method embodiments can be completed by hardware related to program instructions, and the above program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above method embodiments are executed; and the above storage medium includes: mobile storage devices, read-only memory (ROM), disks or optical disks, etc. Various media that can store program codes.

[0212] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be embodied in the form of a software product in essence or in other words, the part that contributes to the prior art can be stored in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks. The above is only a specific implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the present application, and they should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A behavior recognition method, characterized in that: The method comprises: Acquire a video frame sequence including an object to be identified; In the video frame sequence, determining a first image region set where the object to be identified is located; Based on the behavior category of the object to be identified in the first image region set, adjusting the feature distances between different first image regions to obtain a second image region set; Based on the second image area set, identifying the behavior of the object to be identified, and obtaining an identification result; The step of adjusting the feature distances between different first image regions based on the behavior category of the object to be identified in the first image region set to obtain the second image region set includes: The feature distance between the first image regions with the same behavior category is reduced and / or the feature distance between the first image regions with different behavior categories is increased, so as to obtain the second image region set.

2. The method according to claim 1, characterized in that Determining, in the video frame sequence, a first image region set where the object to be identified is located includes: Detecting the object to be identified in each video frame to obtain multiple detection frames of the object to be identified; Adjusting the areas of the multiple detection frames in each video frame to obtain multiple adjusted areas; The first image region set is determined among the multiple adjusted regions of each video frame.

3. The method according to claim 2, characterized in that Determining the first image area set from the multiple adjusted areas of each video frame includes: Determine, among the multiple adjusted regions of each video frame, a target adjusted region whose overlap with the reference adjusted region is greater than a preset overlap threshold and whose area is smaller than a preset area threshold; Eliminating the target adjusted area from the multiple adjusted areas of each video frame to obtain the first image area set of each video frame; The reference adjusted region is any adjusted region of the multiple adjusted regions whose first confidence is greater than a first confidence threshold.

4. The method according to claim 1, characterized in that Before adjusting the characteristic distances between different first image regions based on the behavior categories of the objects to be identified in the first image region set to obtain the second image region set, the method further includes: Selecting, from the video frame sequence, video frames with a number less than a preset number of frames as target video frames; Based on the first image region in each target video frame, the behavior of the object to be identified is classified to obtain the behavior category of the object to be identified.

5. The method according to any one of claims 1 to 4, characterized in that: In a case where a video frame in the video frame sequence includes at least one second image region, identifying the behavior of the object to be identified based on the set of second image regions to obtain an identification result includes: Determining a second confidence level that the behavior category of each second image region in the video frame is a preset category; In at least one second image region of the video frame, determining a candidate image region whose second confidence is greater than or equal to a second confidence threshold, and obtaining a set of candidate image regions; Eliminate, from the candidate image region set, candidate image regions whose spatial distances from the candidate image region set meet a preset condition, to obtain a set of eliminated candidate image regions; Based on the eliminated candidate image area set in the video frame, the behavior of the object to be identified is identified to obtain the identification result.

6. The method according to claim 5, characterized in that The step of eliminating candidate image regions whose spatial distances from the candidate image region set meet a preset condition from the candidate image region set to obtain the eliminated candidate image region set includes: Determine, from the candidate image region set, a subset of candidate image regions whose second confidence is equal to a second confidence threshold; In the candidate image region subset, determining the spatial distance between any candidate image region and other candidate image regions; wherein the other candidate image regions are candidate image regions in the candidate image region subset except the any candidate image region; Determine a candidate image region to be eliminated whose spatial distance is greater than a preset spatial distance threshold; In the candidate image region set, the to-be-eliminated candidate image region is eliminated to obtain the eliminated candidate image region set.

7. The method according to claim 5, characterized in that The identifying the behavior of the object to be identified based on the eliminated candidate image area set in the video frame to obtain the identification result includes: Determine at least one target region sequence in the video frame sequence based on the eliminated candidate image region set in at least one video frame; The behavior of the object to be identified is identified in the at least one target area sequence to obtain the identification result.

8. The method according to claim 7, characterized in that The step of determining at least one target region sequence in the video frame sequence based on the eliminated candidate image region set in at least one video frame comprises: Based on the set of eliminated candidate image regions, selecting any eliminated candidate image region from each video frame of the at least one video frame; Merging the selected removed image regions to obtain at least one merged region; In the video frame sequence, a target region sequence matching each merged region is determined to obtain the at least one target region sequence.

9. The method according to claim 7 or 8, characterized in that: The step of identifying the behavior of the object to be identified in the at least one target area sequence to obtain the identification result includes: Adjusting the side length of each target area in the target area sequence to a preset side length to obtain an adjusted target area sequence; The behavior of the object to be identified is identified in each of the adjusted target area sequences to obtain the identification result.

10. A behavior recognition device, characterized in that: The device comprises: A first acquisition module, used to acquire a video frame sequence including an object to be identified; A first determination module, used to determine, in the video frame sequence, a first image region set where the object to be identified is located; A first adjustment module, configured to adjust the feature distances between different first image regions based on the behavior categories of the objects to be identified in the first image region set, to obtain a second image region set; A first recognition module, configured to recognize the behavior of the object to be recognized based on the second image region set to obtain a recognition result; The first adjustment module is further used to reduce the feature distance between the first image regions with the same behavior category and / or increase the feature distance between the first image regions with different behavior categories to obtain the second image region set.

11. A computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, which, after being executed, can implement the method steps described in any one of claims 1 to 9.

12. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory stores computer executable instructions, and the processor can implement the method steps described in any one of claims 1 to 9 when executing the computer executable instructions on the memory.

Images