Dog walking violation behavior detection method, device and equipment based on deep learning

Abstract

The application discloses a deep learning-based dog walking violation behavior detection method, device and equipment, which comprises the following steps: acquiring monitoring video data and frame extraction to obtain a to-be-detected image; dog and pedestrian detection is performed on the to-be-detected image to obtain corresponding detection frames; the Euclidean distance between the dog and the pedestrian is calculated; the region of interest near the dog is calculated; a rope connector detection is performed on the region of interest to obtain information of the rope connector in the region of interest; when the rope connector is detected, it is judged whether the dog has historical tracking information and corresponding modification is made; whether the dog is walked in violation of regulations is judged according to whether the historical value of the rope state of the dog in the k-frame interval is greater than or equal to a threshold value. The application adopts the historical value of the rope state of the dog and the pedestrian in the k-frame interval to judge whether the dog is walked in violation of regulations, thereby improving the efficiency of the public place manager in managing the dog.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, in particular to a deep learning-based dog walking violation detection method, device and equipment. BACKGROUND

[0002] Pet dog owners will regularly walk their dogs, and the uncivilized behavior of walking dogs without a leash will bring serious safety hazards to the community.

[0003] At present, with the improvement of relevant systems, the relevant departments have strengthened the constraint and control of non-standard dog-keeping behavior. The newly revised Animal Epidemic Prevention Law clearly stipulates that dogs should be taken out with a dog tag and measures such as tying a dog leash; the management of pet dogs in public places mainly relies on manual supervision; and manual supervision often overlooks and cannot notify relevant personnel in time, so the management method of dog walking violations needs to be improved. SUMMARY

[0004] The embodiments of the present application provide a deep learning-based dog walking violation detection method, device and equipment, aiming to solve the problem of low efficiency of manual supervision of dog walking violations.

[0005] In a first aspect, the embodiments of the present application provide a deep learning-based dog walking violation detection method, comprising:

[0006] Obtain monitoring video data and perform frame extraction processing to obtain a to-be-detected image;

[0007] Detect a dog and a pedestrian according to a first target detection algorithm on the to-be-detected image to obtain a detection frame of the dog and a detection frame of the pedestrian, respectively;

[0008] Calculate the Euclidean distance between the dog and the pedestrian by using the detection frame of the dog and the detection frame of the pedestrian; and obtain a region of interest near the dog by calculating the Euclidean distance;

[0009] Detect a rope-like connecting object in the region of interest according to a second target detection algorithm to obtain information of the rope-like connecting object in the region of interest;

[0010] Confirm whether the rope-like connecting object is detected according to the information of the rope-like connecting object, and when the rope-like connecting object is detected, determine whether the dog has historical tracking information; if yes, update the historical value of the leash state of the dog; if no, initialize the leash state of the dog;

[0011] determining whether the leash state history value of the dog in a k-frame interval is greater than or equal to a threshold value; if yes, determining that the dog is normally walked; if no, determining that the dog is walked in violation of rules; wherein when the pedestrian is not detected around the dog, the dog is determined to be a stray dog.

[0012] In a second aspect, an embodiment of the present application provides a deep learning-based dog walking violation behavior detection device, comprising:

[0013] an image acquisition unit configured to acquire monitoring video data and perform frame extraction processing to obtain a to-be-detected image;

[0014] a first detection unit configured to perform dog and pedestrian detection on the to-be-detected image according to a first target detection algorithm to obtain a detection frame of the dog and a detection frame of the pedestrian, respectively;

[0015] a second detection unit configured to calculate an Euclidean distance between the dog and the pedestrian by using the detection frame of the dog and the detection frame of the pedestrian, and calculate a region of interest around the dog by using the Euclidean distance;

[0016] a third detection unit configured to perform rope-like connector detection on the region of interest according to a second target detection algorithm to obtain information of the rope-like connector in the region of interest;

[0017] a first judgment unit configured to determine whether the rope-like connector is detected according to the information of the rope-like connector, and when the rope-like connector is detected, determine whether the dog has historical tracking information; if yes, update a leash state history value of the dog; if no, initialize the leash state of the dog;

[0018] a second judgment unit configured to determine whether the leash state history value of the dog in a k-frame interval is greater than or equal to a threshold value; if yes, determining that the dog is normally walked; if no, determining that the dog is walked in violation of rules; wherein when the pedestrian is not detected around the dog, the dog is determined to be a stray dog.

[0019] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the deep learning-based dog walking violation behavior detection method of the first aspect when executing the computer program.

[0020] The embodiment of the present application provides a kind of based on deep learning's dog walking violation behavior detection method, obtains monitoring video data and frame extraction obtains the image to be detected;Dog and pedestrian detection are carried out to the image to be detected, respectively corresponding detection frame is obtained;The Euclidean distance between the dog and the pedestrian is calculated;The region of interest near the dog is calculated by the Euclidean distance;Rope connector detection is carried out to the region of interest, and the information of the rope connector in the region of interest is obtained;When the rope connector is detected, it is judged whether the dog has historical tracking information and corresponding modification is made;Whether the dog is illegally walked is judged according to whether the leash state historical value of the dog in k frame interval is greater than or equal to threshold value.The present application uses the leash state historical value between dog and pedestrian in k frame interval to judge whether it is illegally walked, which improves the efficiency of dog management in public places.

[0021] The embodiment of the present application also provides a kind of based on deep learning's dog walking violation behavior detection device and computer equipment, also have the beneficial effects described above. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0023] Figure 1 A flowchart of a deep learning-based dog walking violation behavior detection method provided by the embodiment of the present application is shown in the figure.

[0024] Figure 2 Another flowchart of a deep learning-based dog walking violation behavior detection method provided by the embodiment of the present application is shown in the figure.

[0025] Figure 3 A region of interest selection diagram of a deep learning-based dog walking violation behavior detection method provided by the embodiment of the present application is shown in the figure.

[0026] Figure 4 A result diagram of a deep learning-based dog walking violation behavior detection method provided by the embodiment of the present application is shown in the figure.

[0027] Figure 5 A Hungarian algorithm flowchart of a deep learning-based dog walking violation behavior detection method provided by the embodiment of the present application is shown in the figure.

[0028] Figure 6 A schematic block diagram of a deep learning-based dog walking violation behavior detection device provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the scope of protection of the present application.

[0030] It should be understood that the terms "comprising" and "including" as used in the specification and the appended claims indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0031] It should also be understood that the terms used in the present application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0032] It should be further understood that the term "and / or" used in the present application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0033] Please see Figure 1 , Figure 1 A flowchart of a deep learning-based dog walking violation detection method according to an embodiment of the present application is shown in FIG. 1. The method includes the following steps S101-S106.

[0034] S101, acquire monitoring video data and perform frame extraction processing to obtain a to-be-detected image;

[0035] S102, detect a dog and a pedestrian from the to-be-detected image according to a first target detection algorithm to obtain a detection frame of the dog and a detection frame of the pedestrian, respectively;

[0036] S103, calculate the Euclidean distance between the dog and the pedestrian using the detection frame of the dog and the detection frame of the pedestrian, and calculate the region of interest near the dog using the Euclidean distance;

[0037] S104, detect a rope connector in the region of interest according to a second target detection algorithm to obtain information of the rope connector in the region of interest;

[0038] S105, confirming whether the rope connection is detected according to the information of the rope connection, when the rope connection is detected, judging whether the dog has historical tracking information, if yes, updating the leash state historical value of the dog, if no, initializing the leash state of the dog;

[0039] S106, judging whether the leash state historical value of the dog in the k frame interval is greater than or equal to a threshold value, if yes, judging as normal dog walking, if no, judging as illegal dog walking, wherein when the dog around does not detect the pedestrian, judging as a stray dog.

[0040] Combining Figure 1 And Figure 2 As shown in FIGS. 1-4, in step S101, first, video data in a cell public area camera is acquired, and the video data is frame-extracted to obtain a to-be-detected image. Here, frame extraction is performed to facilitate detection of dogs and pedestrians in the image frame. The embodiment of the application can only frame-extract the video data in one camera. Of course, if necessary, the video data of multiple cameras can be frame-extracted at the same time, so that illegal dog walking behaviors in multiple scenes can be detected at the same time. For example, if the video data of five cameras is frame-extracted, five to-be-detected images of different scenes can be obtained, improving the efficiency of detecting illegal dog walking behaviors. In addition, multiple cameras can also be used to take pictures of the same scene from different angles, and the monitoring video data of these cameras can be acquired and frame-extracted, and whether there is an illegal dog walking behavior in the scene can be comprehensively judged by combining the monitoring video data of these different cameras. Step S101 mainly frame-extracts the acquired video data to prepare for the processing of the next step.

[0041] In an embodiment, the step S101 comprises:

[0042] Acquiring monitoring video data and frame-extracting to obtain an original image;

[0043] Adjusting the color and spatial domain of the original image, completing the color difference correction of the original image, and calculating an empty domain reconstruction image according to the following formula:

[0044]

[0045] Wherein, I(g) represents the pixel value of point g in the original image; I(h) represents the pixel value of point h in the original image; (I(g)-I(h)) represents the pixel difference between points g and h; ||g-h|| represents the distance metric; Ω represents the set of all pixel points in the original image; S α (t) represents the slope function; R(g) represents the empty domain reconstruction image;

[0046] The corrected spatial domain reconstruction image is dynamically expanded, and the spatial domain reconstruction image is normalized to [0, 1] according to the following formula to obtain the to-be-detected image:

[0047]

[0048] Wherein, L(g) represents the to-be-detected image; max R represents the maximum value of R(g); min R represents the minimum value of R(g).

[0049] In this embodiment, the camera installed in the scene may be affected by environmental factors such as light, and the brightness of the to-be-detected image after frame extraction processing may be too dark, overexposed, etc. The automatic color balance algorithm is used to enhance the frame extraction image, thereby preventing the to-be-detected image from being too dark, overexposed, etc. First, video data is obtained, and frame extraction processing is performed to obtain an original image. The color and spatial domain of the original image are adjusted, and the color difference correction of the original image is completed to obtain a spatial domain reconstruction image. The spatial domain reconstruction image also needs to be dynamically expanded, and the spatial domain reconstruction image is normalized to [0, 1] to finally obtain an image after global white balance, that is, the to-be-detected image. It should be noted that if there are other alternative image improvement algorithms, they are also applicable to the present application, such as optimizing the clarity of the image, so that the system can capture the target more conveniently.

[0050] In step S102, the to-be-detected image is detected according to a first target detection algorithm. In this embodiment, the first target detection algorithm is a yolov5 model. The yolov5 model mainly consists of a backbone network, a Neck layer and a prediction layer. The backbone network adopts a Focus structure and a CSP structure (cross-stage local fusion network of CSPNet), which enhances the learning ability of the CNN and can maintain accuracy while being lightweight. The Neck layer adopts a PFN+PAN module to fuse features of different dimensions, thereby improving the extraction ability of the network for feature and position information of different size target objects. The prediction layer outputs class information and coordinate information of the target object. It should be noted that if there are other alternative target detection algorithms, they can realize detection of dogs and pedestrians. Finally, the detection frame of the dog and the detection frame of the pedestrian in the to-be-detected image are obtained after detection by the first target detection algorithm.

[0051] In an embodiment, before step S102, the following steps are included:

[0052] The first target detection algorithm is trained and optimized according to the following loss function:

[0053] L loss = λ1Lconf + λ2L cla + λ3L loc

[0054] wherein, L conf represents the confidence loss of whether it is the target, L cla represents the category loss, L loc represents the positioning loss, λ1, λ2 and λ3 all represent the corresponding balance coefficients; L loss represents the total loss.

[0055] In the embodiment, before the dog and pedestrian detection of the to-be-detected image is performed using the first target detection algorithm, the first target detection algorithm used needs to be trained; first, the picture data set of pedestrians and dogs is labeled using a labeling software such as labelImg, and then the labeled pictures are input into an initialized network model (the first target detection algorithm, the same below) for training and optimization using a loss function.

[0056] Specifically, in the inference stage, the to-be-detected image is directly input into the trained network model, so that the detection box parameters of the dog and the detection box parameters of the pedestrian are obtained, wherein i represents the i-th dog, j represents the j-th pedestrian, d and p respectively represent the dog and the pedestrian; in addition, and the detection box parameter formats of and are all unified as (x1, y1, x2, y2), x1 and y1 respectively represent the horizontal coordinate and the vertical coordinate of the left upper corner of the detection box, and x2 and y2 respectively represent the horizontal coordinate and the vertical coordinate of the right lower corner of the detection box.

[0057] In the above step S102, the most common case is that one detection box of a dog and multiple detection boxes of pedestrians are detected in one to-be-detected image; generally, the pedestrian closest to the dog is the owner of the dog, if the dog owner is normally walking the dog, there should be a rope-like connecting object between the dog owner and the dog, so the embodiment can determine whether there is a non-rule-breaking dog walking behavior by detecting whether there is a rope-like connecting object between the dog and the closest pedestrian. In combination with Figure 3 shown, in step S103, the Euclidean distance (Euclidean distance is a distance definition, which refers to the real distance between two points in m-dimensional space, or the natural length of a vector) between the dog and multiple pedestrians is calculated using the detection box of the dog and the detection boxes of the multiple pedestrians; and then the region of interest (ROI) around the dog is calculated through the Euclidean distance, Figure 3The "All" rectangle represents the region of interest near the dog, the "dog" rectangle represents the detection box for the dog, and the "person" rectangle represents the detection box for the pedestrian.

[0058] In one embodiment, step S103 includes:

[0059] The Euclidean distance between the dog and the pedestrian is calculated using the following formula:

[0060]

[0061] Among them, D ij This represents the Euclidean distance between the i-th dog and the j-th pedestrian; and These represent the x-coordinate and y-coordinate of the center point of the i-th dog, respectively; and These represent the x-coordinate and y-coordinate of the center point of the j-th pedestrian, respectively.

[0062] Furthermore, the pedestrian closest to the dog is obtained through Euclidean distance calculation and is designated as the target pedestrian; the region of interest near the dog is calculated using the following formula:

[0063] I R =[min(x1,m1),min(y1,n1),max(x2,m2),max(y2,n2)]

[0064] Among them, I R The region of interest is represented by: min(x1,m1); min(y1,n1); min(y1,n1); max(x2,m2); max(y2,n2); and max(y2,n2).

[0065] In this embodiment, the true distance between all pedestrians and the dog in the image to be detected can be calculated using Euclidean distance. The Euclidean distance formula contains... and These represent the x-coordinate and y-coordinate of the center point of the i-th dog, respectively; and respectively represent the horizontal coordinate of the center point of the jth pedestrian and the vertical coordinate of the center point of the jth pedestrian; wherein the horizontal coordinates of the center points of the dog and the pedestrian can be calculated according to the average values of the horizontal coordinates of the top-left corner points and the bottom-right corner points corresponding to the detection boxes of the dog and the pedestrian; the vertical coordinates of the center points of the dog and the pedestrian can be calculated according to the average values of the vertical coordinates of the top-left corner points and the bottom-right corner points corresponding to the detection boxes of the dog and the pedestrian; if the horizontal coordinate of the center point of the dog needs to be calculated, then represents the horizontal coordinate of the top-left corner of the detection box of the dog, represents the horizontal coordinate of the bottom-right corner of the detection box of the dog.

[0066] Further, by calculating the Euclidean distance between each pedestrian in the to-be-detected image and the dog, the target pedestrian closest to the dog can be obtained wherein M represents the minimum value, and the pedestrian closest to the ith dog can be denoted as It should be noted that the detection box parameters of are represented as (m1, n1, m2, n2), and the rectangular box parameters of the region of interest near the dog can be obtained according to the calculation formula of the region of interest near the dog, so as to facilitate the judgment in the subsequent steps.

[0067] In step S104, the region of interest is detected for the leash (leash rectangular box in Figure 3 ) according to a second target detection algorithm, to obtain information of the leash in the region of interest; similarly, the second target detection algorithm is not limited here, and the yolov5 model can also be used to detect whether the leash exists in the embodiment of the present application; it should be noted that the yolov5 model needs to be trained before use, first, the leash in the image data set is labeled, and the yolov5 model is optimized and trained according to the loss function (the loss function can use the loss function used for training in the first target detection algorithm); in the inference stage, the region of interest of the dog is input into the yolov5 model which has been trained, if the region of interest detects the leash, the detection box parameters of the leash can be denoted as for the convenience of subsequent steps.

[0068] In step S105, it is determined whether the rope connection is detected according to the information of the rope connection (i.e., whether the rope connection exists between the target pedestrian and the dog), and when the rope connection is detected, it is determined whether the dog has historical tracking information (the historical tracking information is the information of the rope connection in the interval frames before the current frame for each frame of the to-be-detected image); if yes, the historical value of the leash state of the dog is updated; if no, the leash state of the dog is initialized; when the historical value of the leash state of the dog is 4, the historical value of the leash state of the dog can be increased by 1 on the basis of 4, which can be set according to actual application; and the leash state of the dog is initialized to 0, and the historical value of the leash state of the dog can be updated when the rope connection is detected again.

[0069] In combination Figure 4 As shown in FIG. 1, in step S106, it is determined whether the historical value of the leash state of the dog in the k-frame interval is greater than or equal to a threshold value; if yes, it is determined that the dog is normally walked; if no, it is determined that the dog is irregularly walked; here, the k-frame interval refers to a plurality of continuous frames of to-be-detected images, for example, 1-10 frames can be set as the k-frame interval, or 5-14 frames can be set as the k-frame interval, and the like. When 1-10 frames are set as the k-frame interval, the rope connection is detected in the first, second and fourth frames, and the rope connection is not detected in other frames, i.e., the historical value of the leash state is 3, and when the threshold value is 3, the historical value of the leash state of the dog is equal to the threshold value, and it is determined that the dog is normally walked; when the threshold value is 4, the historical value of the leash state of the dog is less than the threshold value, and it is determined that the dog is irregularly walked; in addition, when the pedestrian is not detected around the dog, i.e., the detection frame parameter of the pedestrian does not exist, it is determined that the dog is a stray dog; when the stray dog or the irregularly walked dog is detected, the system triggers the voice reminding module to issue a warning, and real-time data is transmitted to the computer terminal of the relevant management personnel for reminding, so that the relevant management personnel can perform subsequent management, and the efficiency of the management personnel in managing the dog in the public place is greatly improved.

[0070] In an embodiment, the dog walking irregular behavior detection method based on deep learning further comprises:

[0071] According to the Kalman filtering algorithm, the dog is tracked and predicted, and a tracking frame of the dog is calculated; the detection frame of the dog is matched with the tracking frame of the dog by using the Hungarian algorithm, and tracking information of the dog is obtained.

[0072] Further, the tracking and prediction of the dog according to the Kalman filtering algorithm, and the calculation of the tracking frame of the dog, comprise:

[0073] The state vector of the dog is represented as:

[0074]

[0075] Wherein, s represents the area of the detection frame; r represents the aspect ratio; All represent the corresponding change rate; X represents the state vector of the dog; T represents the T time;

[0076] The prediction value and the observation value of the Kalman filtering algorithm are fused according to the following formula to obtain the fusion value;

[0077]

[0078] Wherein, The fusion value of the k frame is represented; The prediction value of the k frame is calculated according to the motion state of the dog; K k The Kalman gain is represented; z k The detection frame measurement value of the dog is represented; H represents the conversion matrix of the state vector of the dog to the measurement value; The residual error is represented.

[0079] Further, the detection frame of the dog is matched with the tracking frame of the dog using the Hungarian algorithm to obtain the tracking information of the dog, including:

[0080] The detection frame of the dog is maximally matched with the tracking frame of the dog, and the following three results exist in the maximally matching process:

[0081] When the detection frame of the dog and the tracking frame of the dog are successfully matched with each other, the Kalman filtering weight and parameter update of the tracking result are performed using the detection frame of the dog, and the counting number of the matched tracking frame is recorded;

[0082] When the detection frame of the dog fails to match, the detection frame of the dog is initialized by Kalman filtering and then added to the tracking list;

[0083] When the tracking frame of the dog fails to match, it is judged whether the counting number of the tracking frame of the dog losing association is greater than the maximum threshold value; if yes, the tracking frame of the dog is deleted from the tracking list; otherwise, the counting number of losing association is increased by one;

[0084] According to the three results existing in the maximally matching process, it is judged whether the interval frame number of the dog is less than the minimum hit frame number of the dog; if yes, the display is returned; if not, it is judged whether the continuous hit frame number of the dog is greater than or equal to the minimum hit frame number of the dog, and if greater than or equal to the minimum hit frame number of the dog, the display is returned.

[0085] Combining Figure 5 As shown in the embodiment, in order to reduce the false detection of a single frame image, the application adopts a multi-frame detection method to improve the accuracy and robustness of the detection effect, but it needs to be considered that when multiple dogs exist in the image to be detected, the first target detection algorithm cannot judge the IDs of the multiple dogs, and cannot perform multi-frame judgment according to historical detection results, so a sort multi-target tracking algorithm is adopted to track each dog in the image to be detected; the sort multi-target tracking algorithm includes a Kalman filtering algorithm and a Hungarian algorithm.

[0086] Specifically, in the Kalman filtering algorithm, the state vector of the dog is represented as X, and the fusion value of the predicted value and the observed value of the Kalman filtering algorithm after fusion can be obtained by substituting the formula, that is, the tracking box of the dog; in the formula, the residual error, that is, the difference between the predicted value and the measured value; and the Kalman gain K k is calculated as follows:

[0087]

[0088] where P represents the noise covariance matrix, H represents the observation matrix, K represents the Kalman gain (coefficient), and R represents the observation noise covariance matrix.

[0089] Further, after the tracking box of the dog is calculated according to the Kalman filtering algorithm, the Hungarian algorithm (the Hungarian algorithm is a data association algorithm, which is based on "minimum loss", and the loss is represented by a loss matrix, each C u,v element of the loss matrix is determined by the intersection over union of the tracking box of the u-th dog in the previous frame predicted by the Kalman filtering algorithm and the detection box of the v-th dog in the current frame) is used to maximize the matching between the detection box of the dog and the tracking box of the dog. In the process of maximum matching, the three results respectively represent: the detection box and the tracking box that match each other successfully (dog tracking success), the detection box that fails to match (new dog appears), and the tracking box that fails to match (dog disappears); finally, whether the interval frame number of the dog is less than the minimum hit frame number of the dog is judged according to the three results in the process of maximum matching; if yes, it is returned to display; if not, whether the continuous hit frame number of the dog is greater than or equal to the minimum hit frame number of the dog is judged, and if it is greater than or equal to the minimum hit frame number of the dog, it is returned to display; if the display is successful, it means that the sort multi-target tracking algorithm successfully tracks each dog in the image to be detected, so as to subsequently judge whether the dog is violated according to the leash state of the dog.

[0090] Combining Figure 6As shown, Figure 6 A schematic block diagram of a deep learning-based dog walking violation behavior detection device provided for an embodiment of the present application, the deep learning-based dog walking violation behavior detection device 600 comprises:

[0091] An image acquisition unit 601 is configured to acquire monitoring video data and perform frame extraction processing to obtain a to-be-detected image;

[0092] A first detection unit 602 is configured to perform dog and pedestrian detection on the to-be-detected image according to a first target detection algorithm to obtain a detection frame of the dog and a detection frame of the pedestrian, respectively;

[0093] A second detection unit 603 is configured to calculate the Euclidean distance between the dog and the pedestrian by using the detection frame of the dog and the detection frame of the pedestrian, and obtain a region of interest near the dog by the Euclidean distance calculation;

[0094] A third detection unit 604 is configured to perform rope-like connector detection on the region of interest according to a second target detection algorithm to obtain information of the rope-like connector in the region of interest;

[0095] A first judgment unit 605 is configured to determine whether the rope-like connector is detected according to the information of the rope-like connector, and when the rope-like connector is detected, determine whether the dog has historical tracking information; if yes, update the leash state historical value of the dog; if no, initialize the leash state of the dog;

[0096] A second judgment unit 606 is configured to determine whether the leash state historical value of the dog in a k-frame interval is greater than or equal to a threshold value; if yes, determine that the dog is normally walked; if no, determine that the dog is walked in violation; when the pedestrian is not detected around the dog, determine that the dog is a stray dog.

[0097] In the present embodiment, the image acquisition unit 601 first acquires monitoring video data and performs frame extraction processing to obtain a to-be-detected image; the first detection unit 602 detects the detection frame of the dog and the detection frame of the pedestrian; the second detection unit 603 detects the region of interest near the dog; the third detection unit 604 detects the information of the rope-like connector in the region of interest; the first judgment unit 605 determines whether the rope-like connector is detected and performs corresponding operations; finally, the second judgment unit 606 determines whether the leash state historical value of the dog in a k-frame interval is greater than or equal to a threshold value to determine whether the dog is walked in violation.

[0098] In an embodiment, the image acquisition unit 601 comprises:

[0099] An acquisition unit is configured to acquire monitoring video data and perform frame extraction processing to obtain an original image.

[0100] A reconstruction unit is configured to adjust the color and spatial domain of the original image, complete color difference correction of the original image, and calculate a spatial domain reconstruction image according to the following formula:

[0101]

[0102] wherein I(g) represents a pixel value of a point g in the original image, I(h) represents a pixel value of a point h in the original image, (I(g)-I(h)) represents a pixel difference between the points g and h, ||g-h|| represents a distance metric, Ω represents a set of all pixel points in the original image, S α (t) represents a slope function, and R(g) represents the spatial domain reconstruction image.

[0103] An output unit is configured to perform dynamic expansion on the corrected spatial domain reconstruction image, normalize the spatial domain reconstruction image to [0, 1] according to the following formula, and obtain a to-be-detected image:

[0104]

[0105] wherein L(g) represents the to-be-detected image, max R represents a maximum value of R(g), and min R represents a minimum value of R(g).

[0106] In an embodiment, the first detection unit 602 further includes, before the first detection unit 602:

[0107] A training unit is configured to train the first target detection algorithm and optimize the first target detection algorithm according to the following loss function:

[0108] L loss = λ1L conf + λ2L cla + λ3L loc

[0109] wherein L conf represents a confidence loss of whether it is a target, L cla represents a category loss, L loc represents a positioning loss, λ1, λ2, and λ3 each represent a corresponding balance coefficient, and L loss represents a total loss.

[0110] In an embodiment, the second detection unit 603 is configured to:

[0111] calculate the Euclidean distance between the dog and the pedestrian according to the following formula:

[0112]

[0113] wherein D ij represents the Euclidean distance between the ith dog and the jth pedestrian; and respectively represent the horizontal coordinate of the center point of the ith dog and the vertical coordinate of the center point of the ith dog; and respectively represent the horizontal coordinate of the center point of the jth pedestrian and the vertical coordinate of the center point of the jth pedestrian.

[0114] Further, the pedestrian closest to the dog is calculated as the target pedestrian through the Euclidean distance;

[0115] The region of interest near the dog is calculated according to the following formula:

[0116] I R =[min(x1,m1),min(y1,n1),max(x2,m2),max(y2,n2)]

[0117] wherein I R represents the region of interest near the dog; min(x1,m1) represents the minimum value between the horizontal coordinate of the upper left corner of the detection frame of the dog and the horizontal coordinate of the upper left corner of the detection frame of the target pedestrian; min(y1,n1) represents the minimum value between the vertical coordinate of the upper left corner of the detection frame of the dog and the vertical coordinate of the upper left corner of the detection frame of the target pedestrian; max(x2,m2) represents the maximum value between the horizontal coordinate of the lower right corner of the detection frame of the dog and the horizontal coordinate of the lower right corner of the detection frame of the target pedestrian; and max(y2,n2) represents the maximum value between the vertical coordinate of the lower right corner of the detection frame of the dog and the vertical coordinate of the lower right corner of the detection frame of the target pedestrian.

[0118] In an embodiment, the deep learning-based dog walking violation behavior detection device 600 further comprises:

[0119] A tracking unit configured to perform tracking prediction on the dog according to a Kalman filtering algorithm to calculate a tracking frame of the dog, and match the detection frame of the dog with the tracking frame of the dog by using a Hungarian algorithm to obtain tracking information of the dog.

[0120] In an embodiment, the tracking unit is configured to:

[0121] The state vector of the dog is represented as:

[0122]

[0123] wherein s represents the area of the detection frame; and r represents the aspect ratio. respectively, represent the corresponding change rate; X represents a state vector of the dog; T represents the T time;

[0124] Further, the predicted value of the Kalman filtering algorithm and the observed value are fused by the following formula to obtain a fused value;

[0125]

[0126] wherein, represents the fused value of the kth frame; represents the predicted value of the kth frame calculated according to the motion state of the dog; K k represents the Kalman gain; z k represents the bounding box measurement value of the dog; H represents a conversion matrix from the state vector of the dog to the measurement value; represents the residual error.

[0127] In an embodiment, the tracking unit is further configured to:

[0128] maximizing matching the bounding box of the dog with the tracking box of the dog, wherein the maximizing matching process has the following three results:

[0129] when the bounding box of the dog and the tracking box of the dog are successfully matched with each other, using the bounding box of the dog to update the Kalman filtering weight and parameters of the tracking result, and recording the count number of the matched tracking box;

[0130] when the bounding box of the dog fails to match, initializing the bounding box of the dog by Kalman filtering and then adding it to the tracking list;

[0131] when the tracking box of the dog fails to match, judging whether the count number of the tracking box of the dog losing association is greater than a maximum threshold value; if yes, deleting the tracking box of the dog from the tracking list; otherwise, increasing the count number of the tracking box losing association by one;

[0132] judging whether the interval frame number of the dog is less than the minimum hit frame number of the dog according to the three results in the maximizing matching process; if yes, returning to display; if not, judging whether the continuous hit frame number of the dog is greater than or equal to the minimum hit frame number of the dog; if greater than or equal to the minimum hit frame number of the dog, returning to display.

[0133] Since the embodiments of the device part correspond to the embodiments of the method part, the embodiments of the device part are described in the description of the embodiments of the method part, which will not be described here.

[0134] The embodiment of the present application further provides a computer device, which can comprise a memory and a processor, the memory has a computer program stored therein, and the processor can realize the steps provided by the above embodiment when calling the computer program in the memory. Of course, the computer device can further comprise various network interfaces, power supplies and other components.

[0135] The various embodiments are described in the specification by way of progressive progression, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be mutually referred to. For the system disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part. It should be noted that, for those skilled in the art, some improvements and modifications can be made to the present application without departing from the principles of the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

[0136] It should be further noted that, in the specification, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A deep learning-based dog walking violation detection method, characterized in that, The method comprises the following steps: acquiring monitoring video data and performing frame extraction processing to obtain a to-be-detected image; detecting a dog and a pedestrian in the to-be-detected image according to a first target detection algorithm to obtain a detection frame of the dog and a detection frame of the pedestrian respectively; calculating an Euclidean distance between the dog and the pedestrian by using the detection frame of the dog and the detection frame of the pedestrian, and calculating a region of interest near the dog by using the Euclidean distance; detecting a rope connecting object in the region of interest according to a second target detection algorithm to obtain information of the rope connecting object in the region of interest; confirming whether the rope connecting object is detected according to the information of the rope connecting object, and judging whether the dog has historical tracking information when the rope connecting object is detected, and updating a leash state historical value of the dog if the dog has the historical tracking information, and initializing the leash state of the dog if the dog does not have the historical tracking information; judging whether the leash state historical value of the dog in a k-frame interval is greater than or equal to a threshold value, and judging that the dog is normally walked if the leash state historical value is greater than or equal to the threshold value, and judging that the dog is walked in violation of rules if the leash state historical value is less than the threshold value, and judging that the dog is a stray dog when the dog is not detected with the pedestrian around the dog. 2.The deep learning-based dog walking violation detection method of claim 1, wherein, The method of acquiring monitoring video data and performing frame extraction processing to obtain a to-be-detected image comprises the following steps: acquiring monitoring video data and performing frame extraction processing to obtain an original image; adjusting a color and a spatial domain of the original image, completing color difference correction of the original image, and calculating a spatial domain reconstruction image according to the following formula: wherein I(g) represents a pixel value of a point g in the original image; I(h) represents a pixel value of a point h in the original image; (I(g)-I(h)) represents a pixel difference between the points g and h; ||g-h|| represents a distance metric; Ω represents a set of all pixel points in the original image; S α (t) represents a slope function; R(g) represents the spatially reconstructed image; performing dynamic expansion on the spatial domain reconstruction image after correction, and normalizing the spatial domain reconstruction image to [0, 1] according to the following formula to obtain the to-be-detected image: wherein, L(g) represents the to-be-detected image; max R represents a maximum value of R(g); and min R represents a minimum value of R(g). 3.The deep learning-based dog walking violation detection method of claim 1, wherein, Before the step of detecting a dog and a pedestrian in the to-be-detected image according to a first target detection algorithm to obtain a detection frame of the dog and a detection frame of the pedestrian respectively, the method comprises the following steps: training the first target detection algorithm, and optimizing the first target detection algorithm according to the following loss function: L loss = λ1L conf + λ2L cla + λ3L loc wherein, L conf represents the confidence loss of whether it is the target, L cla represents the category loss, L loc represents the positioning loss, λ1, λ2 and λ3 all represent the corresponding balance coefficients; L loss represents the total loss. 4.The deep learning-based dog walking violation detection method of claim 1, wherein, The method of calculating an Euclidean distance between the dog and the pedestrian by using the detection frame of the dog and the detection frame of the pedestrian comprises the following steps: calculating the Euclidean distance between the dog and the pedestrian according to the following formula: where D ij represents the Euclidean distance between the ith dog and the jth pedestrian; and respectively represent the horizontal coordinate of the center point of the ith dog and the vertical coordinate of the center point of the ith dog; and respectively represent the horizontal coordinate of the center point of the jth pedestrian and the vertical coordinate of the center point of the jth pedestrian. 5.The deep learning-based dog walking violation detection method of claim 1, wherein, The method of calculating a region of interest near the dog by using the Euclidean distance comprises the following steps: calculating a pedestrian closest to the dog as a target pedestrian by using the Euclidean distance; calculating the region of interest near the dog according to the following formula: I R = [min(x1,m1),min(y1,n1),max(x2,m2),max(y2,n2)] where I R represents a region of interest near the dog; min(x1, m1) represents taking the minimum value between the horizontal coordinate of the top-left corner of the detection frame of the dog and the horizontal coordinate of the top-left corner of the detection frame of the target pedestrian; min(y1, n1) represents taking the minimum value between the vertical coordinate of the top-left corner of the detection frame of the dog and the vertical coordinate of the top-left corner of the detection frame of the target pedestrian; max(x2, m2) represents taking the maximum value between the horizontal coordinate of the bottom-right corner of the detection frame of the dog and the horizontal coordinate of the bottom-right corner of the detection frame of the target pedestrian; and max(y2, n2) represents taking the maximum value between the vertical coordinate of the bottom-right corner of the detection frame of the dog and the vertical coordinate of the bottom-right corner of the detection frame of the target pedestrian. 6.The deep learning-based dog walking violation detection method of claim 1, wherein, The method further comprises the following steps: tracking and predicting the dog according to a Kalman filtering algorithm to calculate a tracking frame of the dog, and matching the detection frame of the dog with the tracking frame of the dog by using a Hungarian algorithm to obtain tracking information of the dog. 7.The deep learning-based dog walking violation detection method of claim 6, wherein, The method of tracking and predicting the dog according to a Kalman filtering algorithm to calculate a tracking frame of the dog comprises the following steps: representing a state vector of the dog as: Wherein, s represents the area of the detection frame; r represents the aspect ratio; All represent the corresponding change rate; X represents the state vector of the dog; T represents the T time. performing data fusion on a prediction value and an observation value of the Kalman filtering algorithm according to the following formula to obtain a fusion value; wherein, represents a fusion value of the kth frame; represents a prediction value of the kth frame calculated according to a motion state of the dog; K k represents a Kalman gain; z k represents a bounding box measurement value of the dog; H represents a conversion matrix of a state vector of the dog to a measurement value; represents a residual error. 8.The deep learning-based dog walking violation detection method of claim 6, wherein, The detection frame of the dog is matched with the tracking frame of the dog by using the Hungarian algorithm to obtain tracking information of the dog, including: The detection frame of the dog is maximally matched with the tracking frame of the dog, and the following three results exist in the process of the maximally matching: When the detection frame of the dog and the tracking frame of the dog are successfully matched with each other, the detection frame of the dog is used to update the Kalman filtering weight and parameters of the tracking result, and the count number of the matched tracking frame is recorded; When the detection frame of the dog fails to match, the detection frame of the dog is added to the tracking list after Kalman filtering initialization; When the tracking frame of the dog fails to match, it is judged whether the count number of the tracking frame of the dog losing association is greater than a maximum threshold value; if yes, the tracking frame of the dog is deleted from the tracking list; otherwise, the count number of losing association is increased by one; According to the three results existing in the process of the maximally matching, it is judged whether the interval frame number of the dog is less than the minimum hit frame number of the dog; if yes, the display is returned; if not, it is judged whether the continuous hit frame number of the dog is greater than or equal to the minimum hit frame number of the dog, and if greater than or equal to the minimum hit frame number of the dog, the display is returned. 9.A deep learning-based dog-walking violation behavior detection device, characterized by Comprise: An image acquisition unit is configured to acquire monitoring video data and perform frame extraction processing to obtain a to-be-detected image; A first detection unit is configured to detect a dog and a pedestrian in the to-be-detected image according to a first target detection algorithm to obtain a detection frame of the dog and a detection frame of the pedestrian, respectively; A second detection unit is configured to calculate an Euclidean distance between the dog and the pedestrian by using the detection frame of the dog and the detection frame of the pedestrian, and obtain a region of interest near the dog by the Euclidean distance calculation; A third detection unit is configured to detect a rope-like connecting object in the region of interest according to a second target detection algorithm to obtain information of the rope-like connecting object in the region of interest; A first judgment unit is configured to determine whether the rope-like connecting object is detected according to the information of the rope-like connecting object, and determine whether the dog has historical tracking information when the rope-like connecting object is detected; if yes, update a historical value of a leash state of the dog; if not, initialize the leash state of the dog; A second judgment unit is configured to determine whether the historical value of the leash state of the dog in k-frame interval is greater than or equal to a threshold value; if yes, determine that the dog is normally walked; if not, determine that the dog is walked in violation of rules; when the pedestrian is not detected around the dog, the dog is determined to be a stray dog.

10. A computer device, comprising: The method comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the deep learning-based dog walking violation detection method according to any one of claims 1 to 8 when executing the computer program.

Images