Behavior recognition method, device, and storage medium

Abstract

The application provides a behavior recognition method and device and a storage medium, which comprise obtaining continuous first interval frame images in a current scene; based on determined optical flow regions, the first interval frame images are screenshot to obtain first screenshot images corresponding to the first interval frame images; optical flow calculation is performed on the first screenshot images of the front and rear frames to obtain first optical flow maps corresponding to the optical flow regions between the front and rear frames; based on the first optical flow maps, first comprehensive optical flow losses corresponding to the optical flow regions are obtained; and based on all the first comprehensive optical flow losses, first behavior information of a second monitoring target in the first interval frame images relative to a first monitoring target in the first interval frame images is determined. In this way, the motion direction and displacement amplitude of each pixel in the optical flow regions can be accurately determined by using the first comprehensive optical flow losses of the optical flow regions obtained under any field of view, so that the first behavior information of the second monitoring target relative to the first monitoring target can be accurately determined.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly provides a behavior recognition method, device and storage medium. BACKGROUND

[0002] At present, the node records in various operation processes such as airport apron flight operation processes are mostly completed manually. For example, during the period from the start of any flight to the departure of the flight, the airport staff determines and records the events occurring through on-site observation or monitoring cameras, such as recording the entry of the aircraft into the position when the aircraft enters the port and stops, recording the arrival of the refueling vehicle when the refueling vehicle arrives to prepare for refueling, and recording the docking event of the conveyor belt vehicle when the conveyor belt vehicle starts to approach the aircraft to prepare for conveying luggage. There are dozens to hundreds of such event records. Manual input is large, and errors are prone to occur due to personnel fatigue. Therefore, an operation process node automatic recognition system based on visual AI emerges as the times require. Some existing node recognition algorithms can support automatic recognition of the behaviors of various monitoring targets in the operation process, such as the behaviors of various monitoring targets in dozens of node events such as aircraft entry and exit, refueling vehicle entry and exit, luggage conveyor belt vehicle docking and withdrawing from the aircraft, terminal head docking and withdrawing from the aircraft, terminal head exit and reset, and the like.

[0003] In actual airport operation scenarios, a left, middle and right monitoring camera is usually arranged to monitor the operation scene in all directions, and various operation node events are automatically recognized based on visual means, which is in line with the demand trend of intelligent upgrading of airport equipment. However, due to the differences in various operation scenarios and camera position angles, the generalization of AI automatic analysis of the nodes is poor, and the effect is good in some scenarios and cannot be used in some scenarios.

[0004] Taking the terminal head as an example, the related events include four node events of terminal head docking with the aircraft, terminal head withdrawing from the aircraft, terminal head exit (i.e. leaving the berth in the non-working state), and terminal head reset (i.e. returning to and stabilizing in the berth in the non-working state). A conventional terminal head behavior recognition algorithm trains target detection by directly labeling whether the terminal head is docked with the aircraft passenger cabin door, thereby obtaining the terminal head docking with the aircraft behavior or withdrawing from the aircraft behavior. However, the left or right monitoring camera has a view overlapping with the movement direction of the terminal head, and it is difficult to determine whether it is in the docking state or the withdrawing state, which often leads to misjudgment. Some terminal head behavior recognition algorithms also use terminal head detection and tracking to determine the movement state of the terminal head, but they also cannot solve the problem of the left and right side angles of the field of view, and the main reason is that the pixel displacement changes very little when the terminal head moves, which leads to misjudgment of the algorithm.

[0005] Therefore, how to accurately recognize the behavior information of the monitoring target in the operation process is a technical problem to be solved by those skilled in the art. SUMMARY

[0006] In order to overcome the above-mentioned defects, the present application is proposed to provide a behavior recognition method, device and storage medium for solving or at least partially solving the technical problem of low accuracy of behavior information of a monitoring target in a recognition operation process.

[0007] In a first aspect, the present application provides a behavior recognition method, comprising:

[0008] obtaining a continuous first interval frame image under a current scene; wherein the first interval frame image comprises a first monitoring target and a second monitoring target;

[0009] based on the determined optical flow region, taking a screenshot of the first interval frame image to obtain a first screenshot image corresponding to the first interval frame image;

[0010] performing optical flow calculation on the first screenshot images of the front and rear frames to obtain a first optical flow map corresponding to the optical flow region between the front and rear frames;

[0011] based on the first optical flow map, obtaining a first comprehensive optical flow loss corresponding to the optical flow region;

[0012] determining first behavior information of the second monitoring target relative to the first monitoring target according to all the first comprehensive optical flow losses.

[0013] Further, in the above-mentioned behavior recognition method, based on the first optical flow map, the first comprehensive optical flow loss corresponding to the optical flow region is obtained, comprising:

[0014] extracting a first preset number of first motion vectors from the first optical flow map;

[0015] selecting a first motion vector with a vector length greater than a first preset length as a first target motion vector;

[0016] if the ratio of the total number of first target motion vectors to the first preset number is greater than a first preset proportion, averaging all the first target motion vectors to obtain a first average vector;

[0017] if the vector length of the first average vector is greater than a second preset length, the first average vector is taken as the first comprehensive optical flow loss corresponding to the optical flow region between the front and rear frames.

[0018] Further, in the above-mentioned behavior recognition method, according to all the first comprehensive optical flow losses, the first behavior information of the second monitoring target relative to the first monitoring target is determined, comprising:

[0019] If the ratio of the number of first integrated light loss quantities to the total number of image frames is greater than the second preset ratio, determine the first angle between each first integrated light loss quantity and the orthogonal direction of the first monitoring target; wherein, the orthogonal direction of the first monitoring target is the direction perpendicular to the orientation of the first monitoring target;

[0020] If all the first included angles are smaller than the first preset included angle, the first behavioral information of the second monitored target relative to the first monitored target is determined to be a close proximity state.

[0021] If all the first included angles are greater than the second preset included angle, the first behavior information of the second monitored target relative to the first monitored target is determined to be a withdrawal state; wherein, the second preset included angle is greater than the first preset angle;

[0022] If a portion of the first included angle is greater than or equal to the first preset included angle, and / or a portion of the first included angle is less than or equal to the second preset included angle, the first behavioral information of the second monitored target relative to the first monitored target is determined to be in a stationary state.

[0023] Furthermore, in the above-described behavior recognition method, if the first behavior information is an approaching state or a withdrawing state, the method further includes:

[0024] Acquire consecutive second-interval frame images in the current scene; wherein, the second-interval frame images include a first monitoring target and a second monitoring target; the first interval time corresponding to the first interval frame image is less than the interval time corresponding to the second interval frame image;

[0025] Based on the determined optical flow region, a screenshot is taken of the second interval frame image to obtain the second screenshot image corresponding to the second interval frame image;

[0026] Optical flow calculation is performed on the second screenshot images of the preceding and following frames to obtain the second optical flow map corresponding to the optical flow region between the preceding and following frames;

[0027] Based on the second optical flow map between consecutive frames, the second comprehensive optical loss corresponding to the optical flow region between consecutive frames is obtained;

[0028] Based on all the second comprehensive light loss, determine the second behavioral information of the second monitored target relative to the first monitored target;

[0029] Based on the first behavioral information and the second behavioral information, the final behavioral information of the second monitoring target relative to the first monitoring target is determined.

[0030] Furthermore, in the above-described behavior recognition method, the second comprehensive optical loss corresponding to the optical flow region between consecutive frames is obtained based on the second optical flow map between consecutive frames, including:

[0031] Extract a second preset number of second motion vectors from the second optical flow map;

[0032] A second motion vector with a length greater than a third preset length is selected as the second target motion vector;

[0033] If the ratio of the total number of second target motion vectors to the second preset number is greater than the third preset ratio, the second average vector is obtained by averaging all the second target motion vectors.

[0034] If the length of the second average vector is greater than the fourth preset length, the second average vector is used as the second comprehensive optical loss amount corresponding to the optical flow region between the previous and next frames.

[0035] Furthermore, in the behavior recognition method described above, determining the second behavior information of the second monitored target relative to the first monitored target based on all the second comprehensive light loss quantities includes:

[0036] Determine the second angle between each second integrated optical loss and the orthogonal direction of the first monitoring target; wherein, the orthogonal direction of the first monitoring target is the direction perpendicular to the orientation of the first monitoring target;

[0037] If all the second included angles are less than the third preset included angle, the second behavioral information of the second monitored target relative to the first monitored target is determined to be a close state.

[0038] If all second included angles are greater than the fourth preset included angle, the second behavior information of the second monitored target relative to the first monitored target is determined to be a withdrawal state; wherein, the fourth preset included angle is greater than the third preset included angle;

[0039] If a portion of the second included angle is greater than or equal to the third preset included angle, and / or a portion of the second included angle is less than or equal to the fourth preset included angle, the second behavioral information of the second monitored target relative to the first monitored target is determined to be in a stationary state.

[0040] Furthermore, in the behavior recognition method described above, determining the final behavior information of the second monitoring target relative to the first monitoring target based on the first behavior information and the second behavior information includes:

[0041] If both the first behavioral information and the second behavioral information are in a close proximity state, then the final behavioral information of the second monitoring target relative to the first monitoring target is determined to be in a close proximity state.

[0042] If both the first behavioral information and the second behavioral information are in a evacuation state, then the final behavioral information of the second monitoring target relative to the first monitoring target is determined to be in a evacuation state.

[0043] If the first behavioral information and the second behavioral information are inconsistent, it is determined that the final behavioral information of the second monitoring target relative to the first monitoring target is a static state.

[0044] Furthermore, in the behavior recognition method described above, before acquiring the first consecutive interval frame images in the current scene, the method further includes:

[0045] Obtain the third consecutive frame image in the current scene;

[0046] If the first monitoring target and the second monitoring target are detected in the third interval frame image, a detection box for the first monitoring target and a detection box for the second monitoring target are generated;

[0047] Based on the detection frame of the first monitoring target, the key points of the first monitoring target and the foreground area of ​​the preset part in the first monitoring target are determined;

[0048] Based on the key points of the first monitored target, determine the orientation of the first monitored target;

[0049] Determine the overlap between the detection frame of the second monitored target and the foreground area of ​​the preset location;

[0050] If the orientation of the first monitored target is within the first field of view and the overlap is greater than the first preset overlap, or if the orientation of the first monitored target is within the second field of view and the overlap is greater than the second preset overlap, the area corresponding to the detection frame of the second monitored target shall be taken as the optical flow area.

[0051] Wherein, the first preset overlap degree is less than the second preset overlap degree.

[0052] In a second aspect, the present invention provides a behavior recognition device, the behavior recognition device including a processor and a storage device, the storage device being adapted to store a plurality of program codes, the program codes being adapted to be loaded and run by the processor to perform any of the behavior recognition methods described above.

[0053] In a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a plurality of program codes adapted to be loaded and executed by a processor to perform the behavior recognition method described in any of the preceding claims.

[0054] The above-described technical solutions of the present invention have at least one or more of the following beneficial effects:

[0055] In implementing the technical solution of this invention, after acquiring consecutive first-interval frame images in the current scene, a screenshot of the first-interval frame image can be taken based on the determined optical flow region to obtain a first screenshot image corresponding to the first-interval frame image; optical flow calculation is performed on the first screenshot images of the preceding and following frames to obtain a first optical flow map corresponding to the optical flow region between the preceding and following frames; based on the first optical flow map, a first comprehensive optical loss corresponding to the optical flow region is obtained; and based on all the first comprehensive optical loss values, the first behavioral information of the second monitoring target relative to the first monitoring target is determined. In this way, under any field of view, the movement direction and displacement amplitude of each pixel within the optical flow region can be accurately determined using the obtained first comprehensive optical loss value of the optical flow region, thereby accurately determining the first behavioral information of the second monitoring target relative to the first monitoring target. Attached Figure Description

[0056] The disclosure of this invention will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. Furthermore, similar numbers in the drawings are used to denote similar components, wherein:

[0057] Figure 1 This is a schematic diagram of the main steps of a behavior recognition method according to an embodiment of the present invention;

[0058] Figure 2 This is a flowchart illustrating the process of determining the optical flow region;

[0059] Figure 3 This is a schematic diagram of the result of object detection in an image;

[0060] Figure 4 It is a schematic diagram of the key points for determining the first monitoring target and the foreground area of ​​the preset part in the first monitoring target;

[0061] Figure 5 This is a main structural block diagram of a behavior recognition device according to an embodiment of the present invention. Detailed Implementation

[0062] Some embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0063] In the description of this invention, "module" and "processor" can include hardware, software, or a combination of both. A module can include hardware circuitry, various suitable sensors, communication ports, memory, and may also include software components, such as program code, or a combination of software and hardware. A processor can be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and / or signal processing capabilities. The processor can be implemented in software, in hardware, or a combination of both. Non-transitory computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The terms "at least one A or B" or "at least one of A and B" have a similar meaning to "A and / or B" and can include only A, only B, or A and B. The singular terms "a" or "this" can also include plural forms.

[0064] In various operational processes, such as flight operations on the airport tarmac, front-view cameras can effectively observe the behavioral information of monitored targets, such as the docking and disengagement of jet bridges and aircraft, and the repositioning and departure of jet bridge heads. However, left- or right-view cameras often show situations where the movement direction of the monitored target is in the same or opposite direction to the camera's line of sight, or the pixel displacement change during jet bridge head movement is very small, resulting in lower accuracy in recognizing the behavioral information of the monitored target.

[0065] Therefore, in order to solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0066] See appendix Figure 1 , Figure 1 This is a schematic flowchart illustrating the main steps of a behavior recognition method according to an embodiment of the present invention. Figure 1 As shown, the behavior recognition method in this embodiment of the invention mainly includes the following steps 101-105.

[0067] Step 101: Obtain the first consecutive interval frame images in the current scene;

[0068] In a specific implementation, a surveillance camera positioned in the current scene can be used to acquire consecutive first-interval frame images of the current scene. The first-interval frame images include a first monitored target and a second monitored target. The first monitored target can be a stationary target, such as an aircraft parked on an airport tarmac, and the second monitored target can be a target that needs to be manipulated, such as a jet bridge. The time interval corresponding to the first-interval frame images can be 1 second.

[0069] Step 102: Based on the determined optical flow region, take a screenshot of the first interval frame image to obtain the first screenshot image corresponding to the first interval frame image;

[0070] In one specific implementation, an optical flow region can be predetermined, and a screenshot of the first interval frame image can be taken using this optical flow region to obtain a first screenshot image corresponding to the first interval frame image. For example, the optical flow region can be a region enclosed by a rectangular frame, so the obtained first screenshot image will be a rectangular image.

[0071] In a specific implementation process, the optical flow region can be a pre-defined region, or it can be determined in the following way. Once the optical flow region is determined in the following way, the optical flow region can be fixed and used directly in the future without having to determine the optical flow region again. Figure 2 This is a flowchart illustrating the process of determining the optical flow region. For example... Figure 2 As shown, determining the optical flow region can be achieved through the following steps 201-206:

[0072] Step 201: Obtain the third consecutive interval frame image in the current scene;

[0073] In a specific implementation, a surveillance camera positioned in the current scene can be used to acquire a third consecutive interval frame image of the current scene. The time interval corresponding to the third interval image can be the same as or different from the time interval corresponding to the first interval image.

[0074] It should be noted that the reason for acquiring the third interval frame image is to use these third interval frame images to determine the optical flow region. Subsequent optical flow analysis cannot be performed before the optical flow region is determined. Therefore, after determining the optical flow region, only subsequent images need to be analyzed, without needing to analyze the previous images again. In other words, the first interval frame image is the image located after the third interval frame image.

[0075] Step 202: If the first monitoring target and the second monitoring target are detected in the third interval frame image, generate the detection box of the first monitoring target and the detection box of the second monitoring target;

[0076] In a specific implementation, a pre-trained YOLOv5 detection model can be used to detect the third interval frame image. If the first monitoring target and the second monitoring target are detected in the third interval frame image, the detection boxes of the first monitoring target and the second monitoring target can be generated. Figure 3 This is a schematic diagram illustrating the results of object detection in an image. For example... Figure 3As shown, taking the aircraft as the first monitored target and the jet bridge as the second monitored target as an example, the third interval frame image is input into the YOLOv5 detection model. If an aircraft is detected, it indicates that the aircraft is in a stopped state. At this time, the aircraft is outlined with the first detection box a, and the jet bridge is outlined with the second detection box b.

[0077] It should be noted that if the first monitoring target is not detected in the third interval frame image, then subsequent detection box annotations are not required.

[0078] Step 203: Based on the detection frame of the first monitoring target, determine the key points of the first monitoring target and the foreground area of ​​the preset part in the first monitoring target;

[0079] In a specific implementation, a sub-image of the first monitored target can be cropped using the detection bounding box of the first monitored target. The pixels of the cropped sub-image are adjusted to a preset pixel width and preset pixel height, and the pixel values ​​are normalized. Then, the sub-image is input into a keypoint detection model to infer the keypoints of the aircraft and the foreground region of a preset part in the first monitored target. The preset pixel width and preset pixel height can be 256. Figure 4 This is a schematic diagram illustrating the key points of the first monitoring target and the foreground area of ​​a preset part within the first monitoring target. For example... Figure 4 As shown, taking an airplane as an example, the key points of an airplane can include the nose point ( Figure 4 The middle part is indicated by the nose, and the left end of the main wing ( Figure 4 shown as the left end of the main wing), the right end of the main wing ( Figure 4 (represented by the right end of the main wing), midpoint of the fuselage, and front point of the tail ( Figure 4 (represented by the tail section), tail tip ( Figure 4 The tail fin tip is indicated in the middle, and the right end of the tail fin is indicated in the middle. Figure 4 (represented by the right tail fin) and the left end of the tail fin ( Figure 4 (The left side of the tail is indicated by the left side of the main wing, the right side of the main wing, the right side of the tail, and the left side of the tail are defined relative to the direction the aircraft is facing. The foreground area d of the fuselage can be obtained using masking technology.)

[0080] Step 204: Determine the orientation of the first monitored target based on its key points;

[0081] In a specific implementation process, after determining the key points of the first monitoring target, the orientation of the first monitoring target can be determined using these key points. For example... Figure 4 As shown, the aircraft's orientation can be determined by comparing the nose point with the center point of the cabin or the tail point. Figure 4 The Vp direction. It should be noted that other key points can also be used to determine the aircraft's orientation.

[0082] Step 205: Determine the overlap between the detection frame of the second monitoring target and the foreground area of ​​the preset location;

[0083] In a specific implementation process, the overlap between the detection box of the second monitored target and the foreground region of the preset part can be obtained according to the calculation formula (1):

[0084] IOU ab =n p / (w b *h b (1)

[0085] Among them, IOU ab For overlap, w b h is the pixel width of the detection bounding box of the second monitored target. b n is the pixel height of the detection bounding box of the second monitored target. p The total number of pixels in the foreground area of ​​the preset part that are within the detection frame of the second monitored target.

[0086] Step 206: If the orientation of the first monitored target is within the first field of view and the overlap is greater than the first preset overlap, or if the orientation of the first monitored target is within the second field of view and the overlap is greater than the second preset overlap, the area corresponding to the detection frame of the second monitored target shall be taken as the optical flow area.

[0087] In a specific implementation, for a scenario with three surveillance cameras, the field of view of the front surveillance camera can be set as the first field of view. Specifically, it can be the field of view of the first monitored target's orientation Vp. Figure 4The angle between the center line c and the second monitoring target is ±35°. The field of view of the monitoring cameras on the left and right sides can be set as the second field of view, which is the field of view outside the first field of view. When the orientation of the first monitoring target is within the first field of view, the front camera can easily identify the second monitoring target. Therefore, a small overlap can be set as the first preset overlap. Then, the obtained overlap is compared with the first preset overlap. When the obtained overlap is greater than the first preset overlap (0.15), the area corresponding to the detection frame of the second monitoring target can be used as the optical flow area. Otherwise, the obtained overlap is compared with the first preset overlap. When the orientation of the first monitoring target is within the second field of view, the monitoring cameras on the left and right sides cannot easily identify the second monitoring target. Therefore, a large overlap can be set as the second preset overlap (0.3). Then, the obtained overlap is compared with the second preset overlap. When the obtained overlap is greater than the second preset overlap, the area corresponding to the detection frame of the second monitoring target can be used as the optical flow area. Otherwise, the obtained overlap is compared with the second preset overlap.

[0088] Step 103: Perform optical flow calculation on the first screenshot images of the preceding and following frames to obtain the first optical flow map corresponding to the optical flow region between the preceding and following frames;

[0089] In a specific implementation, for sparse optical flow, algorithms such as the Lucas-Kanade algorithm and the pyramid Lucas-Kanade algorithm can be used to calculate the optical flow of the first screenshots of the preceding and following frames, obtaining the first optical flow map corresponding to the optical flow region between the preceding and following frames. For dense optical flow, algorithms such as the Farneback algorithm and the RLOF algorithm can be used to calculate the optical flow of the first screenshots of the preceding and following frames, obtaining the first optical flow map corresponding to the optical flow region between the preceding and following frames. Further examples will not be provided here.

[0090] It should be noted that, in order to reduce the amount of computation and improve computational efficiency, the first screenshot images of the preceding and following frames can be scaled up first. The first screenshot image is scaled up to an image with a pixel height of 160 and a pixel width of 160, and then converted to a grayscale image before optical flow calculation is performed.

[0091] Step 104: Based on the first optical flow map, obtain the first comprehensive optical loss corresponding to the optical flow region;

[0092] In a specific implementation, the first overall optical loss corresponding to the optical flow region can be calculated according to the following steps:

[0093] (1) Extract a first preset number of first motion vectors from the first optical flow map;

[0094] In a specific implementation, a first preset number can be set based on the pixel width and pixel height of the optical flow map, and the first preset number of first motion vectors can be extracted from the first optical flow map. For example, for an optical flow map with a pixel height of 160 and a pixel width of 160, one motion vector can be extracted every 4 pixels, and then the motion vectors at the edges can be removed, resulting in 39*39=1521 first motion vectors.

[0095] (2) Select a first motion vector whose vector length is greater than a first preset length as the first target motion vector;

[0096] In a specific implementation, when obtaining the first optical flow map, the vector length of each first motion vector can be obtained based on the pixel coordinates of the pixel corresponding to each first motion vector in the previous frame and the pixel coordinates of the next frame in the first interval frame image between two consecutive frames. In this way, the first motion vector with a vector length greater than a first preset length can be selected as the first target motion vector, thus selecting the first motion vector whose displacement exceeds a certain amplitude. The first preset length can be 1.5.

[0097] (3) If the ratio of the total number of first target motion vectors to the first preset number is greater than the first preset ratio, average all the first target motion vectors to obtain the first average vector;

[0098] In a specific implementation, if the ratio of the total number of first target motion vectors to the first preset number is greater than the first preset proportion, all the first target motion vectors can be averaged to obtain the first average vector. The first preset proportion can be 0.2.

[0099] (4) If the vector length of the first average vector is greater than the second preset length, the first average vector is used as the first comprehensive optical loss corresponding to the optical flow region between the previous and next frames.

[0100] In a specific implementation, the vector length of the first average vector can be obtained based on the pixel coordinates of the corresponding pixel in the previous frame and the pixel coordinates in the next frame. Then, the vector length of the first average vector is compared with a second preset length. If the vector length of the first average vector is greater than the second preset length, the first average vector can be used as the first comprehensive optical loss corresponding to the optical flow region between the previous and next frames. The second preset length can be the same as the first preset length, both being 1.5.

[0101] It should be noted that the reason for comparing the length of the first average vector to be greater than the second preset length is that vectors have direction. Although the length of a single first target motion vector is greater than the first preset length, the length of the first average vector may be shortened after averaging all the first target motion vectors. Therefore, if the length of the first average vector is greater than the second preset length, it means that the displacement of the first average vector exceeds a certain range, thereby reducing the first comprehensive optical loss corresponding to the optical flow region.

[0102] Step 105: Based on all the first comprehensive light loss, determine the first behavioral information of the second monitoring target relative to the first monitoring target.

[0103] In a specific implementation, if the ratio of the number of first comprehensive light loss quantities to the total number of frames in N consecutive first interval images is greater than a second preset ratio, it indicates that the second monitored target is active within the optical flow region. In this case, the first angle between each first comprehensive light loss quantity and the orthogonal direction of the first monitored target can be calculated; wherein, the orthogonal direction of the first monitored target is the direction perpendicular to the orientation of the first monitored target. For example... Figure 5 The direction shown by Vc is the orthogonal direction of the first monitored target.

[0104] In a specific implementation, if all the first included angles are less than the first preset included angle, the first behavioral information of the second monitored target relative to the first monitored target is determined to be an approaching state; if all the first included angles are greater than the second preset included angle, the first behavioral information of the second monitored target relative to the first monitored target is determined to be a withdrawing state; if some of the first included angles are greater than or equal to the first preset included angle, and / or some of the first included angles are less than or equal to the second preset included angle, the first behavioral information of the second monitored target relative to the first monitored target is determined to be a stationary state. Wherein, the second preset included angle is greater than the first preset included angle. In some examples, the sum of the first preset included angle and the second preset included angle can be 180°.

[0105] In this embodiment, by utilizing the precise key points of the first monitored target and the foreground region of a preset part of the first monitored target, reliable geometric information such as the orientation of the first monitored target can be provided, thereby greatly improving the accuracy of the first behavioral information of the second monitored target relative to the first monitored target, which is its motion state. For example, it can accurately detect whether the second monitored target is stationary or in motion, and for the motion state, it can accurately detect whether the movement direction of the second monitored target is approaching or moving away from the first monitored target. Furthermore, based on the timestamp of each frame image, it can accurately locate information such as the time from the start to the stop of approaching the second monitored target and the time from the start to the end of moving away.

[0106] The behavior recognition method of this embodiment, after acquiring consecutive first-interval frame images in the current scene, can take screenshots of the first-interval frame images based on the determined optical flow regions to obtain a first screenshot image corresponding to the first-interval frame images; perform optical flow calculation on the first screenshot images of the preceding and following frames to obtain a first optical flow map corresponding to the optical flow regions between the preceding and following frames; obtain a first comprehensive optical loss amount corresponding to the optical flow regions based on the first optical flow map; and determine the first behavior information of the second monitoring target relative to the first monitoring target based on all the first comprehensive optical loss amounts. In this way, under any field of view, the movement direction and displacement amplitude of each pixel within the optical flow region can be accurately determined using the obtained first comprehensive optical loss amount of the optical flow region, thereby accurately determining the first behavior information of the second monitoring target relative to the first monitoring target.

[0107] In a specific implementation process, in outdoor scenarios, factors such as rain, snow, and wind inevitably interfere with the monitoring camera, causing issues like shaking and momentary obstruction. Camera shaking can lead to light loss distortion. Alternatively, the monitored target may repeatedly move forward and backward during its movement, which can cause misjudgments in the first behavioral information of the second monitored target relative to the first monitored target obtained by the above method. Therefore, to solve the above technical problems, the present invention also provides the following technical solution.

[0108] Specifically, regardless of whether the surveillance camera shakes, momentarily obstructs the view, or the second surveillance target is repeatedly adjusted, causing it to move forward and backward repeatedly, the above method will determine that the final behavior information of the second surveillance target relative to the first surveillance target is a motion state. Therefore, if the first behavior information is an approaching state or a retreating state, the behavior recognition method of this embodiment further includes the following steps:

[0109] (11) Obtain the second consecutive interval frame images in the current scene;

[0110] In a specific implementation, due to factors such as camera shake, momentary obstruction, and repeated adjustments to the second monitored target, the first behavioral information of the second monitored target relative to the first monitored target can be obtained as a motion state within a short period. However, if the time interval is long, the final judgment result will be inconsistent with the judgment result when the time interval is short. Therefore, in this embodiment, continuous second interval frame images of the current scene can be acquired. The second interval frame images also include the first and second monitored targets. The first interval time corresponding to the first interval frame image is shorter than the interval time corresponding to the second interval frame image.

[0111] It should be noted that the second interval frame image of the first frame is the same image as the first interval frame image of the first frame.

[0112] (12) Based on the determined optical flow region, a screenshot of the second interval frame image is taken to obtain the second screenshot image corresponding to the second interval frame image;

[0113] (13) Perform optical flow calculation on the second screenshot images of the previous and next frames to obtain the second optical flow map corresponding to the optical flow region between the previous and next frames;

[0114] The implementation process of steps (12) to (13) above can be referred to the implementation process of steps 102 to 103, and will not be repeated here.

[0115] (14) Based on the second optical flow map between the previous and next frames, obtain the second comprehensive optical loss corresponding to the optical flow region between the previous and next frames;

[0116] Specifically, a second preset number of second motion vectors can be extracted from the second optical flow map; second motion vectors with a vector length greater than a third preset length are selected as second target motion vectors; if the ratio of the total number of second target motion vectors to the second preset number is greater than a third preset ratio, all second target motion vectors are averaged to obtain a second average vector; if the vector length of the second average vector is greater than a fourth preset length, the second average vector is used as the second comprehensive optical loss corresponding to the optical flow region between consecutive frames.

[0117] It should be noted that the second preset number can be the same as the first preset number, the third preset length can be the same as the first preset length, the fourth preset length can be the same as the second preset length, and the third preset ratio can be the same as the first preset ratio.

[0118] (15) Based on all the second integrated light loss, determine the second behavioral information of the second monitoring target relative to the first monitoring target;

[0119] In a specific implementation, a second angle can be determined between each second comprehensive optical loss and the orthogonal direction of the first monitoring target; wherein, the orthogonal direction of the first monitoring target is the direction perpendicular to the orientation of the first monitoring target; if all second angles are less than a third preset angle, the second behavioral information of the second monitoring target relative to the first monitoring target is determined to be an approaching state; if all second angles are greater than a fourth preset angle, the second behavioral information of the second monitoring target relative to the first monitoring target is determined to be a withdrawing state; wherein, the fourth preset angle is greater than the third preset angle; if some second angles are greater than or equal to the third preset angle, and / or, some second angles are less than or equal to the fourth preset angle, the second behavioral information of the second monitoring target relative to the first monitoring target is determined to be a stationary state.

[0120] It should be noted that the third preset angle can be the same as the first preset angle. The fourth preset angle can be the same as the second preset angle.

[0121] (16) Based on the first behavior information and the second behavior information, determine the final behavior information of the second monitoring target relative to the first monitoring target.

[0122] In a specific implementation process, if both the first behavioral information and the second behavioral information are in an approaching state, the final behavioral information of the second monitoring target relative to the first monitoring target is determined to be in an approaching state; if both the first behavioral information and the second behavioral information are in a withdrawing state, the final behavioral information of the second monitoring target relative to the first monitoring target is determined to be in a withdrawing state; if the first behavioral information and the second behavioral information are inconsistent, the final behavioral information of the second monitoring target relative to the first monitoring target is determined to be in a stationary state.

[0123] In other words, if the first comprehensive light loss amount obtained from the first interval image with a short time interval and the second comprehensive light loss amount obtained from the second interval image with a longer time interval have the same direction, and the angle between them and the orthogonal vector Vc of the first monitoring target is less than their respective preset angles, then the final behavior information of the second monitoring target relative to the first monitoring target is determined to be an approaching state; otherwise, the final behavior information of the second monitoring target relative to the first monitoring target is determined to be a stationary state. If the first comprehensive light loss amount obtained from the first interval image with a short time interval and the second comprehensive light loss amount obtained from the second interval image with a longer time interval have the same direction, and the angle between them and the orthogonal vector Vc of the aircraft is greater than their respective preset angles, then the final behavior information of the second monitoring target relative to the first monitoring target is determined to be a withdrawing state; otherwise, the final behavior information of the second monitoring target relative to the first monitoring target is determined to be a stationary state.

[0124] The behavior recognition method in this embodiment obtains a second comprehensive light loss based on a second interval image with a longer time interval. Then, based on all the second comprehensive light loss values, it determines the second behavior information of the second monitoring target relative to the first monitoring target. Based on the first behavior information and the second behavior information, it determines the final behavior information of the second monitoring target relative to the first monitoring target. This corrects the first behavior information, thereby avoiding misjudgments caused by the displacement of the monitoring camera or the back-and-forth adjustment of the second monitoring target, and ensuring the recognition result of the behavior information of the second monitoring target relative to the first monitoring target.

[0125] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effects of the present invention, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders, and these variations are all within the scope of protection of the present invention.

[0126] Those skilled in the art will understand that all or part of the processes in the method of the above embodiment of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0127] Furthermore, the present invention also provides a behavior recognition device.

[0128] See appendix Figure 5 , Figure 5 This is a main structural block diagram of a behavior recognition device according to an embodiment of the present invention. ​ As shown, the behavior recognition device of this embodiment may include a processor 51 and a storage device 52. The storage device 52 is adapted to store a plurality of program codes, which are adapted to be loaded and run by the processor 51 to perform the behavior recognition method of the above embodiment.

[0129] For ease of explanation, only the parts relevant to the embodiments of the present invention are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. This behavior recognition device can be a control device comprising various electronic devices.

[0130] Furthermore, the present invention also provides a computer-readable storage medium. In one embodiment of the computer-readable storage medium according to the present invention, the computer-readable storage medium can be configured to store a program that performs the behavior recognition method of the above-described method embodiments. This program can be loaded and run by a processor to implement the above-described behavior recognition method. For ease of explanation, only the parts related to the embodiments of the present invention are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. The computer-readable storage medium can be a storage device comprising various electronic devices. Optionally, in the embodiments of the present invention, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0131] Furthermore, it should be understood that since the various modules are only provided to illustrate the functional units of the device of the present invention, the physical devices corresponding to these modules may be the processor itself, or a part of the processor's software, hardware, or a combination of software and hardware. Therefore, the number of modules shown in the figures is merely illustrative.

[0132] Those skilled in the art will understand that the various modules in the device can be adaptively split or combined. Such splitting or combining of specific modules will not cause the technical solution to deviate from the principles of the present invention; therefore, the technical solutions after splitting or combining will fall within the protection scope of the present invention.

[0133] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A behavior recognition method, characterized in that, include: Acquire consecutive first-interval frame images in the current scene; wherein, the first-interval frame images include a first monitoring target and a second monitoring target; Based on the determined optical flow region, a screenshot is taken of the first interval frame image to obtain the first screenshot image corresponding to the first interval frame image. Optical flow calculation is performed on the first screenshot images of the preceding and following frames to obtain the first optical flow map corresponding to the optical flow region between the preceding and following frames; Based on the first optical flow map, the first comprehensive optical loss corresponding to the optical flow region is obtained; Based on all the first comprehensive light loss, the first behavioral information of the second monitored target relative to the first monitored target is determined through the following steps: If the ratio of the number of first integrated light loss quantities to the total number of image frames is greater than the second preset ratio, determine the first angle between each first integrated light loss quantity and the orthogonal direction of the first monitoring target; wherein, the orthogonal direction of the first monitoring target is the direction perpendicular to the orientation of the first monitoring target; Based on the first included angle, the first behavioral information of the second monitoring target relative to the first monitoring target is determined.

2. The behavior recognition method according to claim 1, characterized in that, Based on the first optical flow map, the first comprehensive optical loss corresponding to the optical flow region is obtained, including: Extract a first preset number of first motion vectors from the first optical flow map; A first motion vector with a length greater than a first preset length is selected as the first target motion vector; If the ratio of the total number of first target motion vectors to the first preset number is greater than the first preset ratio, the first average vector is obtained by averaging all the first target motion vectors. If the length of the first average vector is greater than the second preset length, the first average vector is used as the first comprehensive optical loss corresponding to the optical flow region between the previous and next frames.

3. The behavior recognition method according to claim 1, characterized in that, The step of determining the first behavioral information of the second monitored target relative to the first monitored target based on the first included angle includes: If all the first included angles are smaller than the first preset included angle, the first behavioral information of the second monitored target relative to the first monitored target is determined to be a close proximity state. If all the first included angles are greater than the second preset included angle, the first behavior information of the second monitored target relative to the first monitored target is determined to be a withdrawal state; wherein, the second preset included angle is greater than the first preset angle; If a portion of the first included angle is greater than or equal to the first preset included angle, and / or a portion of the first included angle is less than or equal to the second preset included angle, the first behavioral information of the second monitored target relative to the first monitored target is determined to be in a stationary state.

4. The behavior recognition method according to claim 1, characterized in that, If the first behavioral information indicates an approaching or withdrawing state, the method further includes: Acquire consecutive second-interval frame images in the current scene; wherein, the second-interval frame images include a first monitoring target and a second monitoring target; the first interval time corresponding to the first interval frame image is less than the interval time corresponding to the second interval frame image; Based on the determined optical flow region, a screenshot is taken of the second interval frame image to obtain the second screenshot image corresponding to the second interval frame image; Optical flow calculation is performed on the second screenshot images of the preceding and following frames to obtain the second optical flow map corresponding to the optical flow region between the preceding and following frames; Based on the second optical flow map between consecutive frames, the second comprehensive optical loss corresponding to the optical flow region between consecutive frames is obtained; Based on all the second comprehensive light loss, determine the second behavioral information of the second monitored target relative to the first monitored target; Based on the first behavioral information and the second behavioral information, the final behavioral information of the second monitoring target relative to the first monitoring target is determined.

5. The behavior recognition method according to claim 4, characterized in that, Based on the second optical flow map between consecutive frames, the second comprehensive optical loss corresponding to the optical flow region between consecutive frames is obtained, including: Extract a second preset number of second motion vectors from the second optical flow map; A second motion vector with a length greater than a third preset length is selected as the second target motion vector; If the ratio of the total number of second target motion vectors to the second preset number is greater than the third preset ratio, the second average vector is obtained by averaging all the second target motion vectors. If the length of the second average vector is greater than the fourth preset length, the second average vector is used as the second comprehensive optical loss amount corresponding to the optical flow region between the previous and next frames.

6. The behavior recognition method according to claim 4, characterized in that, Based on all the second comprehensive light loss, determine the second behavioral information of the second monitored target relative to the first monitored target, including: Determine the second angle between each second integrated optical loss and the orthogonal direction of the first monitoring target; wherein, the orthogonal direction of the first monitoring target is the direction perpendicular to the orientation of the first monitoring target; If all the second included angles are less than the third preset included angle, the second behavioral information of the second monitored target relative to the first monitored target is determined to be a close state. If all second included angles are greater than the fourth preset included angle, the second behavior information of the second monitored target relative to the first monitored target is determined to be a withdrawal state; wherein, the fourth preset included angle is greater than the third preset included angle; If a portion of the second included angle is greater than or equal to the third preset included angle, and / or a portion of the second included angle is less than or equal to the fourth preset included angle, the second behavioral information of the second monitored target relative to the first monitored target is determined to be in a stationary state.

7. The behavior recognition method according to claim 4, characterized in that, Based on the first behavioral information and the second behavioral information, the final behavioral information of the second monitored target relative to the first monitored target is determined, including: If both the first behavioral information and the second behavioral information are in a close proximity state, then the final behavioral information of the second monitoring target relative to the first monitoring target is determined to be in a close proximity state. If both the first behavioral information and the second behavioral information are in a evacuation state, then the final behavioral information of the second monitoring target relative to the first monitoring target is determined to be in a evacuation state. If the first behavioral information and the second behavioral information are inconsistent, it is determined that the final behavioral information of the second monitoring target relative to the first monitoring target is a static state.

8. The behavior recognition method according to any one of claims 1-4, characterized in that, Before acquiring the first consecutive interval frame images in the current scene, the following steps are also included: Obtain the third consecutive frame image in the current scene; If the first monitoring target and the second monitoring target are detected in the third interval frame image, a detection box for the first monitoring target and a detection box for the second monitoring target are generated; Based on the detection frame of the first monitoring target, the key points of the first monitoring target and the foreground area of ​​the preset part in the first monitoring target are determined; Based on the key points of the first monitored target, determine the orientation of the first monitored target; Determine the overlap between the detection frame of the second monitored target and the foreground area of ​​the preset location; If the orientation of the first monitored target is within the first field of view and the overlap is greater than the first preset overlap, or if the orientation of the first monitored target is within the second field of view and the overlap is greater than the second preset overlap, the area corresponding to the detection frame of the second monitored target shall be taken as the optical flow area. Wherein, the first preset overlap degree is less than the second preset overlap degree.

9. A behavior recognition device, characterized in that, The method includes a processor and a storage device, the storage device being adapted to store a plurality of program codes, the program codes being adapted to be loaded and run by the processor to perform the behavior recognition method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The system stores multiple lines of program code, which are adapted to be loaded and run by a processor to perform the behavior recognition method according to any one of claims 1 to 8.

Images