Highway pedestrian intrusion detection method and electronic equipment

Abstract

The invention discloses an expressway pedestrian intrusion detection method and electronic equipment, and the method comprises the steps: firstly, employing a multi-target detection tracking algorithm, determining target detection frames in a plurality of continuous video frames, inputting a target small image corresponding to each target detection frame into a target multi-classification label model, and carrying out the recognition of a target small image corresponding to each target detection frame; and outputting the classification label of each target object, and for the to-be-detected target object corresponding to the classification label representing the off-highway worker, determining whether there is a pedestrian intruding into the highway according to the motion speed, the displacement distance and the object continuous appearance duration of the to-be-detected target object. The target objects on the expressway are classified through the model, the target objects with the classification labels being off-highway workers are screened out, and then whether pedestrians intrude into the expressway or not is detected according to the movement speed, the displacement distance and the object continuous appearance duration of the screened-out target objects. Therefore, the detection accuracy is improved.

Description

Background Art

[0002] Safety control is the key point of highway management. Pedestrian intrusions pose a huge safety hazard. Therefore, it is necessary to detect pedestrians on the highway. If pedestrians are detected on the highway, an alarm will be issued.

[0003] There are usually construction workers and maintenance personnel on highways. Therefore, when detecting pedestrians on highways, it is usually necessary to distinguish and detect construction workers, maintenance personnel and pedestrians. However, due to the wide range of monitoring scenes on highways and the small pedestrian targets, inputting small target samples into the model may detect objects or images with shapes similar to humans in high-speed scenes as pedestrians, such as trees, signs or other similar objects in the distance, and the ghosting images formed due to the influence of network speed, which may cause false alarms in pedestrian detection. Summary of the invention

[0004] The invention provides a method and electronic equipment for detecting pedestrian intrusion on a highway, which are used to solve the problem of low detection accuracy of pedestrian intrusion events on highways in the prior art.

[0005] In a first aspect, the present application provides a method for detecting pedestrian intrusion on a highway, the method comprising:

[0006] A multi-target detection and tracking algorithm is used to determine the target detection frame of each target object in multiple consecutive video frames;

[0007] Input the target image corresponding to each target detection box into the target multi-classification label model, and output the classification label of each target object;

[0008] For each target object to be detected corresponding to a classification label representing off-highway workers, determine the movement speed, displacement distance and duration of continuous appearance of the target object to be detected according to the duration of adjacent frames, each target detection frame corresponding to the target object to be detected and the size of the target object to be detected in the continuous video frames in which the target object to be detected continuously appears;

[0009] According to the moving speed of each target object to be detected, the displacement distance of each target object to be detected, and the continuous appearance time of each target object to be detected, it is determined whether there is a pedestrian intruding on the highway.

[0010] In a possible implementation, determining the movement speed, displacement distance, and continuous appearance duration of the target object to be detected according to the duration of adjacent frames, each target detection frame corresponding to the target object to be detected, and the size of the target object to be detected in the continuous video frames in which the target object to be detected continuously appears, includes:

[0011] Taking the center point of each target detection frame as the position information of the target object to be detected in the target detection frame;

[0012] In the continuous video frames in which the target object to be detected continuously appears, the movement speed, displacement distance and continuous appearance duration of the target object to be detected are determined according to the duration of adjacent frames, the first position information of the target object to be detected in the first video frame, the second position information of the target object to be detected in the last video frame, and the size of the target object to be detected, wherein the first video frame is the first video frame in the continuous video frames in which the target object to be detected continuously appears, and the last video frame is the last video frame in the continuous video frames in which the target object to be detected continuously appears.

[0013] In a possible implementation manner, the displacement distance of the target object to be detected is determined by the following method, including:

[0014] According to the first position information of the target object to be detected and the second position information of the target object to be detected, the frame displacement distance of the target object to be detected is determined; and the product of the frame displacement distance and the coefficient corresponding to the size of the target object to be detected is used as the displacement distance of the target object to be detected, wherein the size of the target object to be detected is negatively correlated with the coefficient corresponding to it.

[0015] In a possible implementation, the duration of continuous appearance of the target object to be detected is determined by:

[0016] According to the first video frame corresponding to the target object to be detected and the second video frame corresponding to the target object to be detected, the number of consecutive frames in which the target object to be detected appears is determined; and the product of the number of consecutive frames and the duration of adjacent frames is used as the duration of continuous appearance of the target object to be detected.

[0017] In a possible implementation manner, the movement speed of the target object to be detected is determined by:

[0018] According to the video frames in which the target object to be detected appears continuously, the adjacent motion speeds of the target object to be detected in each group of adjacent video frames are calculated; and the average value of the calculated adjacent motion speeds is used as the motion speed of the target object to be detected.

[0019] In a possible implementation, calculating the adjacent motion speeds of the target object to be detected in each group of adjacent video frames according to the video frames in which the target object to be detected appears continuously includes:

[0020] For any group of adjacent video frames, determine the previous position information of the target object to be detected in the previous video frame of the adjacent video frames, and the next position information of the target object to be detected in the next video frame of the adjacent video frames; determine the adjacent displacement distance of the target object to be detected according to the next position information and the previous position information; and use the ratio of the adjacent displacement distance to the adjacent frame duration as the adjacent movement speed of the target object in the adjacent video frames.

[0021] In a possible implementation manner, determining the adjacent displacement distance of the target object to be detected according to the subsequent position information and the previous position information includes:

[0022] Determining the displacement distance between adjacent frames according to the latter position information and the former position information;

[0023] The product of the adjacent frame displacement distance and the coefficient corresponding to the size of the target object to be detected is used as the adjacent displacement distance of the target object to be detected.

[0024] In a possible implementation, judging whether there is a pedestrian intruding on the highway according to the movement speed of each target object to be detected, the displacement distance of each target object to be detected, and the duration of continuous appearance of each target object includes:

[0025] For any target object to be detected, if it is determined that the movement speed of the target object to be detected is greater than or equal to the preset speed, the displacement distance of the target object to be detected is greater than or equal to the preset distance, and the duration of the continuous appearance of the target object to be detected is greater than or equal to the preset duration, it is determined that a pedestrian has intruded on the highway;

[0026] Otherwise, make sure there are no pedestrians on the highway.

[0027] In a possible implementation, if the size of the target object to be detected is greater than or equal to a first preset size, the coefficient is the first coefficient;

[0028] If the size of the target object to be detected is smaller than the first preset size and greater than or equal to the second preset size, the coefficient is the second coefficient;

[0029] If the size of the target object to be detected is smaller than the second preset size, the coefficient is the third coefficient;

[0030] The first preset size is greater than the second preset size, and the first coefficient, the second coefficient and the third coefficient are all greater than 1.

[0031] In a second aspect, the present application provides an electronic device, including a communication unit and a processor, wherein:

[0032] The communication unit is used to communicate with the camera and obtain a video stream consisting of a plurality of continuous video frames from the camera;

[0033] The processor is used to use a multi-target detection and tracking algorithm to determine the target detection frame of each target object in multiple continuous video frames; input the target thumbnail corresponding to each target detection frame into the target multi-classification label model, and output the classification label of each target object; for the target object to be detected corresponding to each classification label representing non-highway workers, according to the adjacent frame duration, each target detection frame corresponding to the target object to be detected, and the size of the target object to be detected in the continuous video frames in which the target object to be detected continuously appears, determine the movement speed, displacement distance and continuous appearance duration of the target object to be detected; according to the movement speed of each target object to be detected, the displacement distance of each target object to be detected, and the continuous appearance duration of each target object to be detected, determine whether there is a pedestrian intruding on the highway.

[0034] In a possible implementation manner, the processor is specifically configured to:

[0035] Taking the center point of each target detection frame as the position information of the target object to be detected in the target detection frame;

[0036] In the continuous video frames in which the target object to be detected continuously appears, the movement speed, displacement distance and continuous appearance duration of the target object to be detected are determined according to the duration of adjacent frames, the first position information of the target object to be detected in the first video frame, the second position information of the target object to be detected in the last video frame, and the size of the target object to be detected, wherein the first video frame is the first video frame in the continuous video frames in which the target object to be detected continuously appears, and the last video frame is the last video frame in the continuous video frames in which the target object to be detected continuously appears.

[0037] In a possible implementation manner, the processor determines the displacement distance of the target object to be detected in the following manner, including:

[0038] According to the first position information of the target object to be detected and the second position information of the target object to be detected, the frame displacement distance of the target object to be detected is determined; and the product of the frame displacement distance and the coefficient corresponding to the size of the target object to be detected is used as the displacement distance of the target object to be detected, wherein the size of the target object to be detected is negatively correlated with the coefficient corresponding to it.

[0039] In a possible implementation manner, the processor determines the duration of continuous appearance of the target object to be detected by:

[0040] According to the first video frame corresponding to the target object to be detected and the second video frame corresponding to the target object to be detected, the number of consecutive frames in which the target object to be detected appears is determined; and the product of the number of consecutive frames and the duration of adjacent frames is used as the duration of continuous appearance of the target object to be detected.

[0041] In a possible implementation manner, the processor determines the movement speed of the target object to be detected in the following manner:

[0042] According to the video frames in which the target object to be detected appears continuously, the adjacent motion speeds of the target object to be detected in each group of adjacent video frames are calculated; and the average value of the calculated adjacent motion speeds is used as the motion speed of the target object to be detected.

[0043] In a possible implementation manner, the processor is specifically configured to:

[0044] For any group of adjacent video frames, determine the previous position information of the target object to be detected in the previous video frame of the adjacent video frames, and the next position information of the target object to be detected in the next video frame of the adjacent video frames; determine the adjacent displacement distance of the target object to be detected according to the next position information and the previous position information; and use the ratio of the adjacent displacement distance to the adjacent frame duration as the adjacent movement speed of the target object in the adjacent video frames.

[0045] In a possible implementation manner, the processor is specifically configured to:

[0046] Determining the displacement distance between adjacent frames according to the latter position information and the former position information;

[0047] The product of the adjacent frame displacement distance and the coefficient corresponding to the size of the target object to be detected is used as the adjacent displacement distance of the target object to be detected.

[0048] In a possible implementation manner, the processor is specifically configured to:

[0049] For any target object to be detected, if it is determined that the movement speed of the target object to be detected is greater than or equal to the preset speed, the displacement distance of the target object to be detected is greater than or equal to the preset distance, and the duration of the continuous appearance of the target object to be detected is greater than or equal to the preset duration, it is determined that a pedestrian has intruded on the highway;

[0050] Otherwise, make sure there are no pedestrians on the highway.

[0051] In a possible implementation, if the size of the target object to be detected is greater than or equal to a first preset size, the coefficient is the first coefficient;

[0052] If the size of the target object to be detected is smaller than the first preset size and greater than or equal to the second preset size, the coefficient is the second coefficient;

[0053] If the size of the target object to be detected is smaller than the second preset size, the coefficient is the third coefficient;

[0054] The first preset size is greater than the second preset size, and the first coefficient, the second coefficient and the third coefficient are all greater than 1.

[0055] In a third aspect, the present application further provides a highway pedestrian intrusion detection device, the device comprising:

[0056] A first determination module is used to determine a target detection frame of each target object in a plurality of consecutive video frames by using a multi-target detection and tracking algorithm;

[0057] The classification module is used to input the target image corresponding to each target detection box into the target multi-classification label model and output the classification label of each target object;

[0058] The second determination module is used to determine, for each target object to be detected corresponding to the classification label representing the off-highway worker, the movement speed, displacement distance and continuous appearance duration of the target object to be detected according to the duration of adjacent frames, each target detection frame corresponding to the target object to be detected and the size of the target object to be detected in the continuous video frames in which the target object to be detected continuously appears, and the size of the target object to be detected;

[0059] The detection module is used to determine whether there is a pedestrian intruding on the highway according to the movement speed of each target object to be detected, the displacement distance of each target object to be detected, and the continuous appearance time of each target object to be detected.

[0060] In a possible implementation manner, the second determining module is specifically configured to:

[0061] Taking the center point of each target detection frame as the position information of the target object to be detected in the target detection frame;

[0062] In the continuous video frames in which the target object to be detected continuously appears, the movement speed, displacement distance and continuous appearance duration of the target object to be detected are determined according to the duration of adjacent frames, the first position information of the target object to be detected in the first video frame, the second position information of the target object to be detected in the last video frame, and the size of the target object to be detected, wherein the first video frame is the first video frame in the continuous video frames in which the target object to be detected continuously appears, and the last video frame is the last video frame in the continuous video frames in which the target object to be detected continuously appears.

[0063] In a possible implementation manner, the second determining module is specifically configured to:

[0064] According to the first position information of the target object to be detected and the second position information of the target object to be detected, the frame displacement distance of the target object to be detected is determined; and the product of the frame displacement distance and the coefficient corresponding to the size of the target object to be detected is used as the displacement distance of the target object to be detected, wherein the size of the target object to be detected is negatively correlated with the coefficient corresponding to it.

[0065] In a possible implementation, the second determining module determines the duration of continuous appearance of the target object to be detected by:

[0066] According to the first video frame corresponding to the target object to be detected and the second video frame corresponding to the target object to be detected, the number of consecutive frames in which the target object to be detected appears is determined; and the product of the number of consecutive frames and the duration of adjacent frames is used as the duration of continuous appearance of the target object to be detected.

[0067] In a possible implementation manner, the second determination module determines the movement speed of the target object to be detected in the following manner:

[0068] According to the video frames in which the target object to be detected appears continuously, the adjacent motion speeds of the target object to be detected in each group of adjacent video frames are calculated; and the average value of the calculated adjacent motion speeds is used as the motion speed of the target object to be detected.

[0069] In a possible implementation manner, the second determining module is specifically configured to:

[0070] For any group of adjacent video frames, determine the previous position information of the target object to be detected in the previous video frame of the adjacent video frames, and the next position information of the target object to be detected in the next video frame of the adjacent video frames; determine the adjacent displacement distance of the target object to be detected according to the next position information and the previous position information; and use the ratio of the adjacent displacement distance to the adjacent frame duration as the adjacent movement speed of the target object in the adjacent video frames.

[0071] In a possible implementation manner, the second determining module is specifically configured to:

[0072] Determining the displacement distance between adjacent frames according to the latter position information and the former position information;

[0073] The product of the adjacent frame displacement distance and the coefficient corresponding to the size of the target object to be detected is used as the adjacent displacement distance of the target object to be detected.

[0074] In a possible implementation, the detection module is specifically used to:

[0075] For any target object to be detected, if it is determined that the movement speed of the target object to be detected is greater than or equal to the preset speed, the displacement distance of the target object to be detected is greater than or equal to the preset distance, and the duration of the continuous appearance of the target object to be detected is greater than or equal to the preset duration, it is determined that a pedestrian has intruded on the highway;

[0076] Otherwise, make sure there are no pedestrians on the highway.

[0077] In a possible implementation manner, the second determining module is specifically configured to:

[0078] If the size of the target object to be detected is greater than or equal to a first preset size, the coefficient is the first coefficient;

[0079] If the size of the target object to be detected is smaller than the first preset size and greater than or equal to the second preset size, the coefficient is the second coefficient;

[0080] If the size of the target object to be detected is smaller than the second preset size, the coefficient is the third coefficient;

[0081] The first preset size is greater than the second preset size, and the first coefficient, the second coefficient and the third coefficient are all greater than 1.

[0082] The beneficial effects of the present invention are as follows:

[0083] The present application provides a method and electronic device for detecting pedestrians intruding on a highway. The method comprises: firstly, using a multi-target detection and tracking algorithm to determine a target detection frame in a plurality of continuous video frames, then inputting a target thumbnail corresponding to each target detection frame into a target multi-classification label model, outputting a classification label for each target object, and for each target object to be detected corresponding to a classification label representing a non-highway worker, according to the adjacent frame duration, each target detection frame corresponding to the target object to be detected, and the size of the target object to be detected in the continuous video frames where the target object to be detected appears, the motion speed, displacement distance, and continuous appearance duration of the target object to be detected are determined, and finally, according to the motion speed of each target object to be detected, the displacement distance of each target object to be detected, and the continuous appearance duration of each target object to be detected, it is judged whether there is a pedestrian intruding on the highway. Since the present application first classifies the target objects on the highway through a model, firstly screens out the target objects with the classification label of non-highway workers, and then detects whether there is a pedestrian intruding on the highway according to the motion speed, displacement distance, and continuous appearance duration of the screened target objects, thereby improving the accuracy of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0085] Figure 1 A schematic diagram of a video frame provided in an embodiment of the present application;

[0086] Figure 2 A schematic diagram of an application scenario of a method for detecting pedestrian intrusion on a highway provided in an embodiment of the present application;

[0087] Figure 3 A schematic diagram of a process flow of a method for detecting pedestrian intrusion on a highway provided in an embodiment of the present application;

[0088] Figure 4 A schematic diagram of a video frame provided in an embodiment of the present application;

[0089] Figure 5 A schematic diagram of another video frame provided in an embodiment of the present application;

[0090] Figure 6 A schematic diagram of another video frame provided in an embodiment of the present application;

[0091] Figure 7 A schematic diagram of another video frame provided in an embodiment of the present application;

[0092] Figure 8 A schematic diagram of another video frame provided in an embodiment of the present application;

[0093] Fig. 9 A schematic diagram of a continuous video frame provided in an embodiment of the present application;

[0094] Fig.10 A schematic diagram of another continuous video frame provided in an embodiment of the present application;

[0095] Fig.11 A complete flowchart of a method for detecting pedestrian intrusion on a highway provided by an embodiment of the present application;

[0096] Fig.12 A schematic diagram of the structure of a highway pedestrian intrusion detection device provided in an embodiment of the present application;

[0097] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0098] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0099] In the description of the embodiments of the present application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0100] The application scenarios described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure, and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. It is known to those skilled in the art that with the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems. In the description of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more.

[0101] In related technologies, it is usually necessary to detect whether there are pedestrians on the highway in complex engineering application environments. On the highway, there may be pedestrians, construction workers and maintenance workers. Pedestrians are not allowed to walk on the highway. Therefore, it is necessary to detect pedestrians on the highway and issue an early warning message when pedestrians are detected on the highway. However, since the monitoring scene range of the highway is relatively wide, the detected target object is relatively small, for example, Figure 1 As shown in the figure, the target object in the black solid line frame is a big tree. Since the big tree is far away, that is, the target is small, when the image corresponding to the distant tree is input into the model, the model may output the big tree as a pedestrian, resulting in false positives in pedestrian detection. In summary, when detecting whether there are pedestrians on the highway in the related art, the detection rate is relatively low.

[0102] Based on the above technical problems, the embodiment of the present application provides a method and electronic device for detecting pedestrians intruding on a highway. First, a multi-target detection and tracking algorithm is used to determine the target detection frame of each target object in multiple continuous video frames, and then the target small image corresponding to each target detection frame is input into the target multi-classification label model, and the classification label of each target object is output. For each target object corresponding to the classification label representing non-highway workers, the target object to be detected is continuously detected according to the adjacent frame duration, each target detection frame corresponding to the target object to be detected, and the size of the target object to be detected in the continuous video frames where the target object to be detected continuously appears. The movement speed, displacement distance and continuous appearance time of the target object to be detected are determined, and finally, according to the transportation speed of each target object to be detected, the displacement distance of each target object to be detected, and the continuous appearance time of each target object to be detected, it is judged whether there is a pedestrian intruding on the highway. Since the present application first classifies the target objects on the highway through the model, first screens out the target objects with the classification label of non-highway workers, and then detects whether there is a pedestrian intruding on the highway according to the movement speed, displacement distance and continuous appearance time of the screened target objects, thereby improving the accuracy of detection. Below, the scheme of the present disclosure is described in detail in conjunction with the accompanying drawings.

[0103] like Figure 2 As shown, an application scenario of a method for detecting pedestrian intrusion on a highway provided in an embodiment of the present application, the application scenario includes cameras 110, electronic devices 120 and terminal devices 130 corresponding to multiple traffic sections, wherein the electronic device 120 in this embodiment can be a central processing unit CPU or a graphics processing unit GPU. Figure 2 In the figure, one terminal device 130 is taken as an example, and in fact, the number of terminal devices 130 is not limited. The terminal device 130 may be a mobile phone, a tablet computer, a personal computer, etc.

[0104] In a possible application scenario, the electronic device 120 communicates with the camera 110, and obtains a video stream consisting of multiple continuous video frames from the camera 110; then a multi-target detection and tracking algorithm is used to determine the target detection frame in the multiple continuous video frames; the target thumbnail corresponding to each target detection frame is input into the target multi-classification label model, and the classification label of each target object is output; for each target object to be detected corresponding to the classification label representing the off-highway worker, the movement speed, displacement distance and continuous appearance duration of the target object to be detected are determined according to the adjacent frame duration, each target detection frame corresponding to the target object to be detected, and the size of the target object to be detected in the continuous video frames in which the target object to be detected continuously appears; according to the movement speed of each target object to be detected, the displacement distance of each target object to be detected, and the continuous appearance duration of each target object to be detected, it is determined whether there is a pedestrian intruding on the highway. Finally, the electronic device 120 sends the detection result of whether there is a pedestrian intruding corresponding to each traffic section to the terminal device 130 for display.

[0105] It should be noted that the method provided in the embodiment of the present application is not limited to Figure 2 The application scenario shown can also be used in other possible application scenarios, which are not limited in the embodiments of the present application.

[0106] like Figure 3 FIG. 1 is a flow chart of a method for detecting pedestrian intrusion on a highway provided by an embodiment of the present application, the method comprising the following steps:

[0107] S301, using a multi-target detection and tracking algorithm to determine a target detection frame for each target object in a plurality of consecutive video frames;

[0108] S302, inputting the target small image corresponding to each target detection box into the target multi-classification label model, and outputting the classification label of each target object;

[0109] S303, for each target object to be detected corresponding to the classification label representing off-highway workers, determine the movement speed, displacement distance and continuous appearance duration of the target object to be detected according to the duration of adjacent frames, each target detection frame corresponding to the target object to be detected, and the size of the target object to be detected in the continuous video frames in which the target object to be detected continuously appears;

[0110] S304: Determine whether there is a pedestrian intruding on the highway according to the movement speed of each target object to be detected, the displacement distance of each target object to be detected, and the duration of continuous appearance of each target object to be detected.

[0111] The embodiment of the present application provides a method for detecting pedestrians intruding on a highway. First, a multi-target detection and tracking algorithm is used to determine the target detection frames in multiple continuous video frames. Then, the target thumbnail corresponding to each target detection frame is input into the target multi-classification label model, and the classification label of each target object is output. For each target object to be detected corresponding to the classification label representing non-highway workers, the motion speed, displacement distance and continuous appearance time of the target object to be detected are determined according to the adjacent frame duration, each target detection frame corresponding to the target object to be detected, and the size of the target object to be detected in the continuous video frames where the target object to be detected appears. Finally, according to the motion speed of each target object to be detected, the displacement distance of each target object to be detected, and the continuous appearance time of each target object to be detected, it is judged whether there is a pedestrian intruding on the highway. Since the present application first classifies the target objects on the highway through the model, first screens out the target objects with the classification label of non-highway workers, and then detects whether there is a pedestrian intruding on the highway according to the motion speed, displacement distance and continuous appearance time of the screened target objects, thereby improving the accuracy of detection.

[0112] In a specific implementation, a video stream is first obtained through a camera, and the video stream is a video stream on a highway. Then, for multiple consecutive video frames in the video stream, a multi-target detection and tracking algorithm is used to determine a target detection frame in the multiple consecutive video frames. The target detection frame is a target object, and the target object can be a construction worker, a maintenance worker, an ordinary pedestrian, or an object with a shape similar to that of a person, such as a number tree, a sign, etc. In the embodiment of the present application, ordinary pedestrians on the highway are detected.

[0113] It should be noted that when a multi-target detection and tracking algorithm is used to determine target detection frames in multiple consecutive video frames, multiple target detection frames can be determined for each video frame, for example, Figure 4 As shown in the figure, there are 4 target detection frames, namely target detection frame 31, target detection frame 32, target detection frame 33 and target detection frame 34. In theory, using the multi-target detection and tracking algorithm, the target objects in the target detection frame should all be people, such as maintenance personnel, construction personnel, and pedestrians. However, since the monitoring scene range of the camera on the highway is relatively wide, the detected target objects are relatively small. When using the multi-target tracking algorithm to determine the target detection frame, some objects with shapes similar to people may also be identified as people, such as Figure 4 The target detection box 34 in .

[0114] In a specific implementation, the plurality of continuous video frames may be 25 continuous video frames.

[0115] In one embodiment, after determining the target detection frames in multiple consecutive video frames, a unique identifier is assigned to the target object in each target detection frame, for example, Figure 5 As shown, the identifier of the target object in the target detection frame 31 is ID1, the identifier of the target object in the target detection frame 32 is ID2, the identifier of the target object in the target detection frame 33 is ID3, and the target identifier of the target object in the target detection frame 34 is ID4.

[0116] For different video frames, the target identity of the same target object remains unchanged, for example Figure 6 As shown, the video frame includes a target detection frame 51, a target detection frame 52, a target detection frame 53 and a target detection frame 54, wherein the target object in the target detection frame 51 is identified as ID1, the target object in the target detection frame 52 is identified as ID2, the target object in the target detection frame 53 is identified as ID3, and the target object in the target detection frame 54 is identified as ID4.

[0117] Figure 5 and Figure 6 It can be seen that Figure 5 The target object in the target detection box 31 and Figure 6 The target identifications of the target objects in the target detection frame 51 are the same, both of which are ID1, that is, the target object in the target detection frame 31 and the target object in the target detection frame 51 are the same target object.

[0118] In one embodiment, for each video frame, after the target detection frame in each video frame is determined, the image of each target detection frame in the video frame is used as a target thumbnail corresponding to the target detection frame.

[0119] In one embodiment, each target sub-image is input into a target multi-classification label model, and a classification label for each target object is output, wherein the classification labels may include labels representing ordinary pedestrians, labels representing construction workers, and labels representing maintenance workers. For example, a label "1" is used to represent an ordinary pedestrian, a label "2" is used to represent a construction worker, and a label "3" is used to represent a maintenance worker. Since both construction workers and maintenance workers are highway workers, the labels representing construction workers and the labels representing maintenance workers are both highway worker labels, while the labels representing ordinary pedestrians are labels representing non-highway workers.

[0120] In an embodiment of the present application, the target multi-classification label model is pre-trained. Specifically, pedestrian images, construction worker images and maintenance worker images on the highway are obtained, and the obtained pedestrian images, construction worker images and maintenance worker images are used as training data sets, input into the initial classification model, and the initial classification model is trained. The trained model obtained is the target multi-classification label model. In a specific implementation, the pedestrian images, construction worker images and maintenance worker images in the training data set can be matched in a ratio of 2:3:5.

[0121] It should be noted that the ratio of pedestrian images, construction worker images and maintenance worker images in the training data set is 2:3:5, which can be determined based on historical data.

[0122] After inputting the target image corresponding to each target detection box in each video frame into the target multi-classification label model, the label of the target object in each target image is output. For example, Figure 5 The target images corresponding to the four target detection boxes in the figure are input into the target multi-classification label model, and the classification label corresponding to the output target identification ID1 is "1", the classification label corresponding to the target identification ID2 is "2", the classification label corresponding to the target identification ID3 is "3", and the classification label corresponding to the target identification ID4 is "1". Since label "1" represents the label of ordinary pedestrians, label "2" represents the label of construction workers, and label "3" represents the label of maintenance workers, the target objects corresponding to label "2" and label "3" are highway workers, and the target object corresponding to label "1" is non-highway workers. That is, the target object corresponding to target identification ID1 and the target object corresponding to target identification ID4 are taken as the target objects to be detected.

[0123] In one embodiment, for each target object to be detected corresponding to a classification label representing off-road workers, the movement speed, displacement distance and continuous appearance duration of the target object to be detected are determined according to the duration of adjacent frames, each target detection frame corresponding to the target object to be detected and the size of the target object to be detected in the continuous video frames in which the target object to be detected continuously appears. Specifically, the target objects whose classification labels represent off-road workers, i.e., the target objects to be detected, are first screened out from the classification labels, and then the video frames in which each target object to be detected continuously appears in the continuous video frames are determined. For example, the continuous video frames are 25 frames, and the classification labels of the target object ID1 and the target object ID4 are both classification labels representing off-road workers. Then, according to the video frames in which the target object ID1 and the target object ID4 continuously appear in the 25 video frames, the movement speed, displacement distance and continuous appearance duration of the target object ID1 and the target object ID4 are respectively calculated. How to calculate the movement speed, displacement distance and continuous appearance duration of the target object to be detected is described in detail below.

[0124] In one embodiment, according to the duration of adjacent frames, each target detection frame corresponding to the target object to be detected, and the size of the target object to be detected in the continuous video frames in which the target object to be detected continuously appears, the movement speed, displacement distance, and continuous appearance duration of the target object to be detected are determined. Specifically, for each video frame in the continuous video frames in which the target object to be detected continuously appears, the center point of the target detection frame is used as the position information of the target object to be detected in the target detection frame; in the continuous video frames in which the target object to be detected continuously appears, according to the duration of adjacent frames, the first position information of the target object to be detected in the first video frame, the second position information of the target object to be detected in the last video frame, and the size of the target object to be detected, the movement speed, displacement distance, and continuous appearance duration of the target object to be detected are determined, wherein the first video frame is the first video frame in the continuous video frames in which the target object to be detected continuously appears, and the last video frame is the last video frame in the continuous video frames in which the target object to be detected continuously appears.

[0125] In a specific implementation, firstly, a rectangular coordinate system is established with the lower left corner of the video frame as the coordinate origin (0, 0) to determine the center point position information of the target object to be detected, that is, the position information of the target object to be detected in the video frame.

[0126] like Figure 7 As shown, a rectangular coordinate system is established with the lower left corner of the video frame as the coordinate origin O.

[0127] In the embodiment of the present application, the position information of the target object to be detected in the video frame may be the coordinate information of the center point of the target detection frame corresponding to the target object to be detected.

[0128] After the rectangular coordinate system is established, the coordinates of the center point of the target detection frame are used as the position information of the target object to be detected in the target detection frame, such as Figure 8 As shown, the coordinates (x1, y1) corresponding to O1 are the position information of the target object ID1 in the current video frame, and the coordinates (x2, y2) corresponding to O2 are the position information of the target object ID4 in the current video frame.

[0129] After obtaining the position information of each target object to be detected, the displacement distance, movement speed and continuous appearance time of each target object to be detected are calculated.

[0130] Specifically, the displacement distance of the target object to be detected can be determined in the following manner: according to the first position information and the second position information of the target object to be detected, the frame displacement distance of the target object to be detected is determined; the product of the frame displacement distance and the coefficient corresponding to the size of the target object to be detected is used as the displacement distance of the target object to be detected, wherein the size of the target object to be detected is negatively correlated with the coefficient corresponding to it.

[0131] In one embodiment, the video frame in which the target object to be detected appears for the first time and the video frame in which the target object to be detected appears continuously are determined, and then the displacement distance of the target object to be detected is determined based on the first position information of the target object to be detected in the video frame in which it first appears and the second position information of the target object to be detected in the video frame in which it last appears.

[0132] For example, Fig. 9 As shown, for multiple video frames in which the target object to be detected appears continuously, the first video frame is the video frame in which the target object to be detected ID1 appears for the first time, the twentieth video frame is the video frame in which the target object to be detected ID1 appears for the last time, the position information O1(x1, y1) of the target object to be detected ID1 in the first video frame is the first position information, and the position information O1'(x1', y1') of the target object to be detected ID1 in the twentieth video frame is the second position information.

[0133] After the first position information of the target object to be detected and the second position information of the target object to be detected are determined, the frame displacement distance of the target object to be detected is determined according to the first position information of the target object to be detected and the second position information of the target object to be detected.

[0134] For example, Fig. 9 As shown, the frame displacement distance between the position information O1(x1, y1) of the target object ID1 to be detected in the first frame of the video and the position information O1'(x1', y1') of the target object ID1 to be detected in the twentieth frame of the video is calculated, and the frame displacement distance can be a pixel distance.

[0135] After determining the frame displacement distance of the target object to be detected, the product of the frame displacement distance and the coefficient is taken as the displacement distance of the target object. The coefficient here is the coefficient corresponding to the size of the target object to be detected. The coefficient corresponding to the size of the target object to be detected is described in detail below.

[0136] Specifically, the size of the target object to be detected is negatively correlated with the coefficient. If the size of the target object to be detected is greater than or equal to the first preset size, the coefficient is the first coefficient; if the size of the target object to be detected is smaller than the first preset size and greater than or equal to the second preset size, the coefficient is the second coefficient; if the size of the target object to be detected is smaller than the second preset size, the coefficient is the third coefficient; wherein the first preset size is greater than the second preset size, the first coefficient, the second coefficient and the third coefficient are all greater than 1, and the first coefficient is smaller than the second coefficient, and the second coefficient is smaller than the third coefficient.

[0137] For example, if the size of the target object to be detected is ≥ 30*30px, the coefficient γ is 1; if the size of the target object to be detected is 20*20px≤<30*30px, the coefficient γ is 1.2; if the size of the target object to be detected is <20*20px, the coefficient γ is 1.4.

[0138] In the embodiment of the present application, the size of the target object to be detected may be the height of the target detection frame, or may be the height of the specifically detected target object to be detected, and the present application does not make any specific limitation on this.

[0139] It should be noted that, when calculating the displacement distance of the target object to be detected, the size of the target object to be detected can be the size of the target object to be detected in any video frame in which the target object to be detected appears continuously, or it can be the average size of the target object to be detected in the video frames in which the target object to be detected appears continuously.

[0140] The above is an explanation of how to determine the displacement distance of the target object to be detected. The following is an explanation of how to determine the duration of the continuous appearance of the target object to be detected.

[0141] In one embodiment, the duration of the target object to be detected may be determined by:

[0142] According to the first video frame corresponding to the target object to be detected and the second video frame corresponding to the target object to be detected, since the first video frame is the first video frame in the video frames in which the target object to be detected appears continuously, and the second video frame is the last video frame in the video frames in which the target object to be detected appears continuously, therefore, that is, according to the video frames in which the target object to be detected appears continuously, the number of consecutive frames in which the target object to be detected exists is determined; according to the number of frames and the length of adjacent frames, the duration of the existence of the target object to be detected is determined.

[0143] For example, Fig. 9 As shown, the video frame in which the target object ID1 to be detected first appears is the first frame, and the video frame in which the target object ID1 to be detected last appears is the twentieth frame. Then, the continuous frames in which the target object ID1 to be detected exists are 20 frames, and the 20 video frames are separated by 19 adjacent frames, and the adjacent frame duration is 0.005ms. Therefore, 19*0.005ms=0.095ms is taken as the continuous appearance duration of the target object ID1 to be detected.

[0144] How to determine the moving speed of the target object to be detected is described in detail below.

[0145] In one embodiment, based on the video frames in which the target object to be detected appears continuously, the adjacent motion speeds of the target object to be detected in each group of adjacent video frames are calculated; and the average value of the calculated adjacent motion speeds is used as the motion speed of the target object to be detected.

[0146] For example, the video frame in which the target object ID1 to be detected first appears is the first video frame, and the video frame in which it last appears is the tenth video frame. The adjacent motion speeds of the target object ID1 to be detected in the first video frame and the second video frame are calculated to be v1, the adjacent motion speeds of the target object ID1 to be detected in the second video frame and the third video frame are calculated to be v2, the adjacent motion speeds of the target object ID1 to be detected in the third video frame and the fourth video frame are calculated to be v3, the adjacent motion speeds of the target object ID1 to be detected in the fourth video frame and the fifth video frame are calculated to be v4, and the adjacent motion speeds of the target object ID1 to be detected in the fifth video frame and the sixth video frame are calculated to be v5. The adjacent motion speed of the target object ID1 to be detected is v5, the adjacent motion speed of the target object ID1 to be detected in the sixth video frame and the seventh video frame is v6, the adjacent motion speed of the target object ID1 to be detected in the seventh video frame and the eighth video frame is v7, the adjacent motion speed of the target object ID1 to be detected in the eighth video frame and the ninth video frame is v8, and the adjacent motion speed of the target object ID1 to be detected in the ninth video frame and the tenth video frame is v9, then the motion speed of the target object ID1 to be detected is (v1+v2+v3+v4+v5+v6+v7+v8+v9) / 9.

[0147] In one embodiment, the adjacent motion speeds of the target object to be detected in adjacent video frames are calculated. For any group of adjacent video frames, the previous position information of the target object to be detected in the previous video frame in the adjacent video frames and the next position information of the target object to be detected in the next video frame in the adjacent video frames can be determined; based on the previous position information and the next position information, the adjacent displacement distance of the target object to be detected is determined; and the ratio of the adjacent displacement distance to the adjacent frame duration is used as the adjacent motion speed of the target object to be detected in the adjacent video frames.

[0148] Specifically, in one embodiment, the adjacent displacement distance of the target object to be detected is determined based on the latter position information and the previous position information, and the adjacent frame displacement distance can be determined based on the latter position information and the previous position information, and the product of the adjacent frame displacement distance and the coefficient corresponding to the size of the target object to be detected is used as the adjacent displacement distance of the target object to be detected.

[0149] For example, if the previous position information is (x3, y3) and the next position information is (x4, y4), then the displacement distance between adjacent frames is

[0150] In an embodiment of the present application, for adjacent video frames, the previous position information of the target object ID1 to be detected in the previous video frame in the adjacent video frames, and the subsequent position information of the target object ID1 to be detected in the subsequent video frame in the adjacent video frames are first determined, wherein the subsequent video frame is located after the previous video frame, and then the adjacent frame displacement distance is determined based on the previous position information and the subsequent position information, and the product of the adjacent frame displacement distance and the coefficient corresponding to the size of the target object to be detected is used as the adjacent displacement distance of the target object to be detected, and the ratio of the adjacent displacement distance to the adjacent frame duration is used as the adjacent motion speed of the target object to be detected.

[0151] For example, Fig.10 As shown, video frame 1 and video frame 2 are adjacent video frames, the position information of the target object ID1 to be detected in video frame 1 is O1, and the position information of the target object ID1 to be detected in video frame 2 is O9. The distance between the position information O9 and the position information O1 is calculated, and the distance may be a pixel distance. According to the size of the target object ID1 to be detected, the coefficient corresponding to the size of the target object ID1 to be detected is determined to be the first coefficient. The product of the calculated distance and the first coefficient is taken as the adjacent displacement distance. Assuming that the adjacent frame duration is 0.005ms, the adjacent motion speed of the target object ID1 to be detected is the adjacent displacement distance divided by 0.005ms.

[0152] The above is an explanation of the movement speed, displacement distance and continuous appearance duration of the target object to be detected. The following is how to judge whether there is a pedestrian intruding on the highway based on the movement speed of each target object to be detected, the displacement distance of each target object to be detected, and the continuous appearance duration of each target object to be detected.

[0153] In one embodiment, for any target object to be detected, if the movement speed of the target object to be detected is greater than a preset speed, and the displacement distance of the target object is greater than a preset distance, and the duration of the target object is greater than a preset duration, it is determined that a pedestrian has intruded on the highway and an alarm is performed; otherwise, it is determined that there are no pedestrians intruding on the highway.

[0154] The embodiment of the present application provides a method for detecting pedestrian intrusion on a highway. First, a target object to be detected that represents a non-staff member is identified through a target multi-classification label model, and then the movement speed, displacement distance, and continuous appearance duration of the target object to be detected are determined. If the movement speed, displacement distance, and continuous appearance duration of the target object to be detected all meet the preset conditions, the target object to be detected is determined to be a pedestrian, and an alarm is processed. If at least one of the movement speed, displacement distance, and continuous appearance duration of the target object to be detected does not meet the preset conditions, it is determined that the target object to be detected is not a pedestrian. Since the target object in the target detection frame can also be some objects with shapes similar to people, such as trees, signs, and other objects, it can also be an image generated by video drag, wherein the trees and signs are stationary objects, so the movement speed and displacement distance are 0, and the duration of continuous appearance of the image generated by the video drag is short, so after the embodiment of the present application detects the target object representing the non-highway staff, the movement speed, displacement distance, and continuous appearance duration of the detected target object are determined.

[0155] like Fig.11 FIG. 1 is a complete flow chart of a method for detecting pedestrian intrusion on a highway provided by an embodiment of the present application, which specifically includes the following steps:

[0156] S1101, access high-speed video surveillance resources through GB181 protocol to obtain real-time video stream images;

[0157] S1102, manually selecting the driving lane and emergency lane areas in the video image as video analysis detection areas;

[0158] It should be noted that it is sufficient to manually select the driving lane and emergency lane areas in a video frame, and other video frames are selected based on the previously marked coordinates.

[0159] S1103, for each video frame, determining a target detection frame and an ID of a target object in each target detection frame;

[0160] S1104, inputting the target small image corresponding to each target detection box into the target multi-classification label model, and outputting the classification label of each target object;

[0161] S1105, for each video frame, the classification label represents the target object of the off-road worker, and calculates the displacement distance, movement speed and continuous appearance time of the target object;

[0162] S1106, determining whether the target object's moving speed is greater than a preset speed, if so, executing S1107, otherwise executing S1110;

[0163] S1107, determining whether the displacement distance of the target object is greater than a preset distance, if so, executing S1108, otherwise executing S1110;

[0164] S1108, determining whether the duration of the continuous appearance of the target object is greater than a preset duration, if so, executing S1109, otherwise executing S1110;

[0165] S1109, confirm that there are pedestrians on the highway;

[0166] S1110, confirm that there are no pedestrians on the highway;

[0167] S1111. Obtain the current frame image of a pedestrian intruding on a highway that meets the determination conditions, report a warning message and a video screenshot, and obtain the video 10 seconds before and after the historical moment for archiving and storage.

[0168] Based on the same disclosure concept, the above-mentioned highway pedestrian intrusion detection method of the present disclosure can also be implemented by a highway pedestrian intrusion detection device. The effect of the highway pedestrian intrusion detection device is similar to that of the above-mentioned method, which will not be described in detail here.

[0169] Fig.12 The figure is a schematic diagram of the structure of a highway pedestrian intrusion detection device according to an embodiment of the present disclosure.

[0170] like Fig.12 As shown, the highway pedestrian intrusion detection device 1100 of the present disclosure may include a first determination module 1101, a classification module 1102, a second determination module 1103 and a detection module 1104, wherein:

[0171] A first determination module 1101 is used to determine a target detection frame of each target object in a plurality of consecutive video frames by using a multi-target detection and tracking algorithm;

[0172] The classification module 1102 is used to input the target image corresponding to each target detection box into the target multi-classification label model and output the classification label of each target object;

[0173] The second determination module 1103 is used to determine, for each target object to be detected corresponding to the classification label representing the off-highway worker, the movement speed, displacement distance and continuous appearance duration of the target object to be detected according to the duration of adjacent frames, each target detection frame corresponding to the target object to be detected and the size of the target object to be detected in the continuous video frames in which the target object to be detected continuously appears, and the size of the target object to be detected;

[0174] The detection module 1104 is used to determine whether there is a pedestrian intruding on the highway according to the movement speed of each target object to be detected, the displacement distance of each target object to be detected, and the continuous appearance time of each target object to be detected.

[0175] In one embodiment, the second determining module 1103 is specifically configured to:

[0176] Taking the center point of each target detection frame as the position information of the target object to be detected in the target detection frame;

[0177] In the continuous video frames in which the target object to be detected continuously appears, the movement speed, displacement distance and continuous appearance duration of the target object to be detected are determined according to the duration of adjacent frames, the first position information of the target object to be detected in the first video frame, the second position information of the target object to be detected in the last video frame, and the size of the target object to be detected, wherein the first video frame is the first video frame in the continuous video frames in which the target object to be detected continuously appears, and the last video frame is the last video frame in the continuous video frames in which the target object to be detected continuously appears.

[0178] In one embodiment, the second determining module 1103 is specifically configured to:

[0179] According to the first position information of the target object to be detected and the second position information of the target object to be detected, the frame displacement distance of the target object to be detected is determined; and the product of the frame displacement distance and the coefficient corresponding to the size of the target object to be detected is used as the displacement distance of the target object to be detected, wherein the size of the target object to be detected is negatively correlated with the coefficient corresponding to it.

[0180] In one embodiment, the second determining module 1103 determines the duration of continuous appearance of the target object to be detected by:

[0181] According to the first video frame corresponding to the target object to be detected and the second video frame corresponding to the target object to be detected, the number of consecutive frames in which the target object to be detected appears is determined; and the product of the number of consecutive frames and the duration of adjacent frames is used as the duration of continuous appearance of the target object to be detected.

[0182] In one embodiment, the second determining module 1103 determines the moving speed of the target object to be detected by:

[0183] According to the video frames in which the target object to be detected appears continuously, the adjacent motion speeds of the target object to be detected in each group of adjacent video frames are calculated; and the average value of the calculated adjacent motion speeds is used as the motion speed of the target object to be detected.

[0184] In one embodiment, the second determining module 1103 is specifically configured to:

[0185] For any group of adjacent video frames, determine the previous position information of the target object to be detected in the previous video frame of the adjacent video frames, and the next position information of the target object to be detected in the next video frame of the adjacent video frames; determine the adjacent displacement distance of the target object to be detected according to the next position information and the previous position information; and use the ratio of the adjacent displacement distance to the adjacent frame duration as the adjacent movement speed of the target object in the adjacent video frames.

[0186] In one embodiment, the second determining module 1103 is specifically configured to:

[0187] Determining the displacement distance between adjacent frames according to the latter position information and the former position information;

[0188] The product of the adjacent frame displacement distance and the coefficient corresponding to the size of the target object to be detected is used as the adjacent displacement distance of the target object to be detected.

[0189] In one embodiment, the detection module 1104 is specifically used to:

[0190] For any target object to be detected, if it is determined that the movement speed of the target object to be detected is greater than or equal to the preset speed, the displacement distance of the target object to be detected is greater than or equal to the preset distance, and the duration of the continuous appearance of the target object to be detected is greater than or equal to the preset duration, it is determined that a pedestrian has intruded on the highway;

[0191] Otherwise, make sure there are no pedestrians on the highway.

[0192] In one embodiment, the second determining module 1103 is specifically configured to:

[0193] If the size of the target object to be detected is greater than or equal to a first preset size, the coefficient is the first coefficient;

[0194] If the size of the target object to be detected is smaller than the first preset size and greater than or equal to the second preset size, the coefficient is the second coefficient;

[0195] If the size of the target object to be detected is smaller than the second preset size, the coefficient is the third coefficient;

[0196] The first preset size is greater than the second preset size, and the first coefficient, the second coefficient and the third coefficient are all greater than 1.

[0197] After introducing a method and device for detecting pedestrian intrusion on a highway according to an exemplary embodiment of the present disclosure, next, an electronic device according to another exemplary embodiment of the present disclosure is introduced.

[0198] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0199] In some possible implementations, the electronic device in the present disclosure may be a central processing unit or a graphics processing unit. The electronic device according to the present disclosure may include at least one processor and at least one computer storage medium. The computer storage medium stores program code, and when the program code is executed by the processor, the processor executes the steps of the highway pedestrian intrusion detection method according to various exemplary embodiments of the present disclosure described above in this specification. For example, the processor may execute the following steps: Figure 3 Steps 301-304 shown in .

[0200] Refer to the following Fig.13 1200 according to this embodiment of the present disclosure is described. Fig.13 The electronic device 1200 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0201] like Fig.13 As shown, the electronic device 1200 is in the form of a general electronic device. The components of the electronic device 1200 may include but are not limited to: the at least one processor 1201, the at least one computer storage medium 1202, and a bus 1203 connecting different system components (including the computer storage medium 1202 and the processor 1201).

[0202] Bus 1203 represents one or more of several types of bus structures, including a computer storage media bus or computer storage media controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0203] Computer storage media 1202 may include readable media in the form of volatile computer storage media, such as random access computer storage media (RAM) 1221 and / or cache storage media 1222 , and may further include read-only computer storage media (ROM) 1223 .

[0204] The computer storage medium 1202 may also include a program / utility 1225 having a set (at least one) of program modules 1224, such program modules 1224 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0205] The electronic device 1200 may also communicate with one or more external devices 1204 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1200, and / or communicate with any device that enables the electronic device 1200 to communicate with one or more other electronic devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1205. Furthermore, the electronic device 1200 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1206. As shown, the network adapter 1206 communicates with other modules for the electronic device 1200 via a bus 1203. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1200, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0206] In one embodiment, the electronic device disclosed in the present application may further include a communication unit ( Fig.13 The communication unit is used to communicate with the camera and obtain a video stream consisting of the continuous video frames from the camera.

[0207] Processor 1201 is used to use a multi-target detection and tracking algorithm to determine the target detection frame of each target object in multiple continuous video frames; input the target thumbnail corresponding to each target detection frame into the target multi-classification label model, and output the classification label of each target object; for the target object to be detected corresponding to each classification label representing non-highway workers, according to the adjacent frame duration, each target detection frame corresponding to the target object to be detected, and the size of the target object to be detected in the continuous video frames in which the target object to be detected continuously appears, determine the movement speed, displacement distance and continuous appearance duration of the target object to be detected; according to the movement speed of each target object to be detected, the displacement distance of each target object to be detected, and the continuous appearance duration of each target object to be detected, determine whether there is a pedestrian intruding on the highway.

[0208] In one embodiment, the processor 1201 is specifically configured to:

[0209] The center point of each target detection frame is used as the position information of the target object to be detected in the target detection frame; in the continuous video frames where the target object to be detected continuously appears, the movement speed, displacement distance and continuous appearance duration of the target object to be detected are determined according to the duration of adjacent frames, the first position information of the target object to be detected in the first video frame, the second position information of the target object to be detected in the last video frame, and the size of the target object to be detected, wherein the first video frame is the first video frame in the continuous video frames where the target object to be detected continuously appears, and the last video frame is the last video frame in the continuous video frames where the target object to be detected continuously appears.

[0210] In one embodiment, the processor 1201 is specifically configured to:

[0211] According to the first position information of the target object to be detected and the second position information of the target object to be detected, the frame displacement distance of the target object to be detected is determined; and the product of the frame displacement distance and the coefficient corresponding to the size of the target object to be detected is used as the displacement distance of the target object to be detected, wherein the size of the target object to be detected is negatively correlated with the coefficient corresponding to it.

[0212] In one embodiment, the processor 1201 determines the duration of the continuous appearance of the target object to be detected by:

[0213] According to the first video frame corresponding to the target object to be detected and the second video frame corresponding to the target object to be detected, the number of consecutive frames in which the target object to be detected appears is determined; and the product of the number of consecutive frames and the duration of adjacent frames is used as the duration of continuous appearance of the target object to be detected.

[0214] In one embodiment, the processor 1201 determines the movement speed of the target object to be detected by:

[0215] According to the video frames in which the target object to be detected appears continuously, the adjacent motion speeds of the target object to be detected in each group of adjacent video frames are calculated; and the average value of the calculated adjacent motion speeds is used as the motion speed of the target object to be detected.

[0216] In one embodiment, the processor 1201 is specifically configured to:

[0217] For any group of adjacent video frames, determine the previous position information of the target object to be detected in the previous video frame of the adjacent video frames, and the next position information of the target object to be detected in the next video frame of the adjacent video frames; determine the adjacent displacement distance of the target object to be detected according to the next position information and the previous position information; and use the ratio of the adjacent displacement distance to the adjacent frame duration as the adjacent movement speed of the target object in the adjacent video frames.

[0218] In one embodiment, the processor 1201 is specifically configured to:

[0219] Determining the displacement distance between adjacent frames according to the latter position information and the former position information;

[0220] The product of the adjacent frame displacement distance and the coefficient corresponding to the size of the target object to be detected is used as the adjacent displacement distance of the target object to be detected.

[0221] In one embodiment, the processor 1201 is specifically configured to:

[0222] For any target object to be detected, if it is determined that the movement speed of the target object to be detected is greater than or equal to the preset speed, the displacement distance of the target object to be detected is greater than or equal to the preset distance, and the duration of the continuous appearance of the target object to be detected is greater than or equal to the preset duration, it is determined that a pedestrian has intruded on the highway;

[0223] Otherwise, make sure there are no pedestrians on the highway.

[0224] In one embodiment, the processor 1201 is specifically configured to:

[0225] If the size of the target object to be detected is greater than or equal to a first preset size, the coefficient is the first coefficient;

[0226] If the size of the target object to be detected is smaller than the first preset size and greater than or equal to the second preset size, the coefficient is the second coefficient;

[0227] If the size of the target object to be detected is smaller than the second preset size, the coefficient is the third coefficient;

[0228] The first preset size is greater than the second preset size, and the first coefficient, the second coefficient and the third coefficient are all greater than 1.

[0229] In some possible implementations, various aspects of a highway pedestrian intrusion detection method provided by the present disclosure may also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the highway pedestrian intrusion detection method according to various exemplary embodiments of the present disclosure described above in this specification.

[0230] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access computer storage medium (RAM), a read-only computer storage medium (ROM), an erasable programmable read-only computer storage medium (EPROM or flash memory), an optical fiber, a portable compact disk read-only computer storage medium (CD-ROM), an optical computer storage medium, a magnetic computer storage medium, or any suitable combination of the above.

[0231] The program product of the highway pedestrian intrusion detection of the embodiment of the present disclosure can adopt a portable compact disk read-only computer storage medium (CD-ROM) and include program code, and can be run on an electronic device. However, the program product of the present disclosure is not limited to this. In this document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or device.

[0232] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0233] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0234] Program code for performing the disclosed operations may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user electronic device, partially on the user device, as a separate software package, partially on the user electronic device and partially on a remote electronic device, or entirely on the remote electronic device. In the case of a remote electronic device, the remote electronic device may be connected to the user electronic device via any type of network including a local area network (LAN) or a wide area network (WAN), or may be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0235] It should be noted that although several modules of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules to be embodied.

[0236] In addition, although the operations of the disclosed method are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0237] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk computer storage media, CD-ROM, optical computer storage media, etc.) containing computer-usable program codes.

[0238] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0239] These computer program instructions may also be stored in a computer-readable computer storage medium that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable computer storage medium produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0240] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0241] Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A method for detecting pedestrian intrusion on a highway, characterized in that: The method includes: A multi-target detection and tracking algorithm is used to determine the target detection frame of each target object in multiple consecutive video frames; Input the target image corresponding to each target detection box into the target multi-classification label model, and output the classification label of each target object; For each target object to be detected corresponding to a classification label representing off-highway workers, determine the movement speed, displacement distance and duration of continuous appearance of the target object to be detected according to the duration of adjacent frames, each target detection frame corresponding to the target object to be detected and the size of the target object to be detected in the continuous video frames in which the target object to be detected continuously appears; According to the moving speed of each target object to be detected, the displacement distance of each target object to be detected, and the continuous appearance time of each target object to be detected, it is determined whether there is a pedestrian intruding on the highway.

2. The method according to claim 1, characterized in that The determining of the movement speed, displacement distance and continuous appearance duration of the target object to be detected according to the duration of adjacent frames, each target detection frame corresponding to the target object to be detected and the size of the target object to be detected in the continuous video frames in which the target object to be detected continuously appears, comprises: Taking the center point of each target detection frame as the position information of the target object to be detected in the target detection frame; In the continuous video frames in which the target object to be detected continuously appears, the movement speed, displacement distance and continuous appearance duration of the target object to be detected are determined according to the duration of adjacent frames, the first position information of the target object to be detected in the first video frame, the second position information of the target object to be detected in the last video frame, and the size of the target object to be detected, wherein the first video frame is the first video frame in the continuous video frames in which the target object to be detected continuously appears, and the last video frame is the last video frame in the continuous video frames in which the target object to be detected continuously appears.

3. The method according to claim 2, characterized in that The displacement distance of the target object to be detected is determined by the following methods, including: According to the first position information of the target object to be detected and the second position information of the target object to be detected, the frame displacement distance of the target object to be detected is determined; and the product of the frame displacement distance and the coefficient corresponding to the size of the target object to be detected is used as the displacement distance of the target object to be detected, wherein the size of the target object to be detected is negatively correlated with the coefficient corresponding to it.

4. The method according to claim 3, characterized in that The duration of the object to be detected is determined by: According to the first video frame corresponding to the target object to be detected and the second video frame corresponding to the target object to be detected, the number of consecutive frames in which the target object to be detected appears is determined; and the product of the number of consecutive frames and the duration of adjacent frames is used as the duration of continuous appearance of the target object to be detected.

5. The method according to claim 3, characterized in that The moving speed of the target object to be detected is determined by: According to the video frames in which the target object to be detected appears continuously, the adjacent motion speeds of the target object to be detected in each group of adjacent video frames are calculated; and the average value of the calculated adjacent motion speeds is used as the motion speed of the target object to be detected.

6. The method according to claim 5, characterized in that The step of calculating the adjacent motion speeds of the target object to be detected in each group of adjacent video frames according to the video frames in which the target object to be detected appears continuously includes: For any group of adjacent video frames, determine the previous position information of the target object to be detected in the previous video frame of the adjacent video frames, and the next position information of the target object to be detected in the next video frame of the adjacent video frames; determine the adjacent displacement distance of the target object to be detected according to the next position information and the previous position information; and use the ratio of the adjacent displacement distance to the adjacent frame duration as the adjacent movement speed of the target object in the adjacent video frames.

7. The method according to claim 6, characterized in that The step of determining the adjacent displacement distance of the target object to be detected according to the latter position information and the former position information includes: Determining the displacement distance between adjacent frames according to the latter position information and the former position information; The product of the adjacent frame displacement distance and the coefficient corresponding to the size of the target object to be detected is used as the adjacent displacement distance of the target object to be detected.

8. The method according to any one of claims 1 to 7, characterized in that: The method of judging whether a pedestrian has intruded on the highway according to the movement speed of each target object to be detected, the displacement distance of each target object to be detected, and the duration of continuous appearance of each target object includes: For any target object to be detected, if it is determined that the movement speed of the target object to be detected is greater than or equal to the preset speed, the displacement distance of the target object to be detected is greater than or equal to the preset distance, and the duration of the continuous appearance of the target object to be detected is greater than or equal to the preset duration, it is determined that a pedestrian has intruded on the highway; Otherwise, make sure there are no pedestrians on the highway.

9. The method according to any one of claims 3 to 7, characterized in that: If the size of the target object to be detected is greater than or equal to a first preset size, the coefficient is the first coefficient; If the size of the target object to be detected is smaller than the first preset size and greater than or equal to the second preset size, the coefficient is the second coefficient; If the size of the target object to be detected is smaller than the second preset size, the coefficient is the third coefficient; The first preset size is greater than the second preset size, and the first coefficient, the second coefficient and the third coefficient are all greater than 1.

10. An electronic device, characterized in that: The invention comprises a communication unit and a processor, wherein: The communication unit is used to communicate with the camera and obtain a video stream consisting of a plurality of continuous video frames from the camera; The processor is used to use a multi-target detection and tracking algorithm to determine the target detection frame of each target object in multiple continuous video frames; input the target thumbnail corresponding to each target detection frame into the target multi-classification label model, and output the classification label of each target object; for the target object to be detected corresponding to each classification label representing non-highway workers, according to the adjacent frame duration, each target detection frame corresponding to the target object to be detected, and the size of the target object to be detected in the continuous video frames in which the target object to be detected continuously appears, determine the movement speed, displacement distance and continuous appearance duration of the target object to be detected; according to the movement speed of each target object to be detected, the displacement distance of each target object to be detected, and the continuous appearance duration of each target object to be detected, determine whether there is a pedestrian intruding on the highway.

Images