Method, device and electronic device for detecting leftover items based on deep neural network

Abstract

The present invention provides a method, device and electronic device for detecting leftover items based on a deep neural network. The method uses an expanded detection frame obtained by expanding an original detection frame based on an item to detect pedestrians, and judges whether the item is a leftover item based on whether a pedestrian is detected in the expanded detection frame, and then records the leftover time of the leftover item. When the leftover time exceeds a preset leftover time threshold, it is considered that a leftover event is sent, and an alarm message is generated and sent to a staff member in a timely manner, which can realize full-automatic detection and alarm of leftover items and reduce the burden on the staff.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a method, device and electronic equipment for detecting leftover items based on a deep neural network. Background Art

[0002] With the continuous improvement of surveillance camera networks, more and more places have surveillance cameras, which provides a good guarantee for the safety of personnel, but the huge amount of data also brings a great burden to the staff. Under the premise of ensuring the quality of service, the unit needs to increase more manpower costs to handle the events captured by the surveillance cameras in real time throughout the day. Random events such as items left behind over a long time span require staff to invest more energy, which greatly increases the burden on the staff and reduces their work efficiency.

[0003] At present, there are some schemes for detecting leftover objects in the prior art. For example, in an existing scheme for detecting leftover objects, a corresponding different background model is constructed for each frame of the video to be detected based on different learning rates, and the background difference of the background model is obtained according to the difference in the speed at which the leftover objects are integrated into the background under different learning rates. Then, the connected domain corresponding to the background difference is extracted, and the circumscribed rectangle of the connected domain is obtained. Finally, each circumscribed rectangle of each frame is tracked to obtain a stable static area, i.e., the leftover object area. The scheme uses background information to determine the area of ​​the leftover objects. It can take advantage of the high stability of the background information to avoid interference such as light changes, target occlusion, and camera shaking, and realize the detection of leftover objects in complex environments. It can improve the stability of leftover object detection in complex environments. However, the scheme has problems such as high algorithm complexity, difficulty in model training, and long calculation time. In addition, it cannot achieve fully automatic alarm, which increases the burden on staff. Summary of the invention

[0004] The purpose of the present invention is to provide a method, device and electronic equipment for detecting left-behind items based on a deep neural network, which can realize fully automatic detection and alarm of left-behind items and reduce the burden on staff.

[0005] According to one aspect of an embodiment of the present invention, a method for detecting leftover items based on a deep neural network is provided, comprising the following steps:

[0006] Input a first video, pre-process the first video, and obtain a detection image set;

[0007] Input the detection image set into the trained legacy object detection model, and the legacy object detection model detects and determines the objects contained in the detection image set and the depth features of each object, wherein the depth features include at least the color features, texture features and size features of the objects;

[0008] The left-behind object detection model determines a first multiple corresponding to each object according to the depth feature of each object, and expands the width and height of the original detection frame of each object by the first multiple based on the center point of the original detection frame of each object, so as to obtain an expanded detection frame corresponding to each object;

[0009] The left-behind object detection model detects pedestrians within the range of the enlarged detection frame corresponding to each object. If a pedestrian is detected in the enlarged detection frame of the object, the object is determined to be a non-left-behind object, otherwise the object is determined to be a left-behind object;

[0010] Extracting the deep features of the left-behind items and outputting them to the item tracking model, the item tracking model tracks the left-behind items according to the deep features of the left-behind items and records the time when the left-behind items were left;

[0011] When the time left behind by the left-behind item exceeds the preset left-behind time threshold, an alarm message is generated and sent.

[0012] Optionally, the training process of the abandoned item detection model includes:

[0013] Establishing a left-behind object detection model, wherein the left-behind object detection model is a yolov5 network model, and a parallel first convolution layer for optimizing the first multiple is added to the last convolution layer of the yolov5 network model;

[0014] Set the initial value of the first multiple;

[0015] Providing a training sample set, labeling training samples in the training sample set, and labeling a legacy label of an object in each training sample, wherein the legacy label is used to reflect whether the object is a legacy object;

[0016] Inputting training samples into the left-behind object detection model, and the left-behind object detection model detecting objects in the training samples;

[0017] Taking the center point of the original detection frame of each object as a reference, the width and height of the original detection frame of each object are enlarged by a first multiple to obtain an enlarged detection frame corresponding to each object, and pedestrians are detected within the range of the enlarged detection frame corresponding to each object;

[0018] Mapping the depth features of the detected objects and the depth features of the pedestrians into a multi-dimensional depth space through the first convolutional layer, and comparing the detection results in the depth space;

[0019] When the detection result is that the legacy tag of the item shows that the item is not a legacy item, and a pedestrian is detected in the enlarged detection frame corresponding to the item, the value of the first multiple corresponding to the item is reduced;

[0020] When the detection result is that the legacy label of the item shows that the item is not a legacy item, and no pedestrian is detected in the enlarged detection frame corresponding to the item, the value of the first multiple corresponding to the item is increased;

[0021] When the detection result is that the left-behind label of the item shows that the item is a left-behind item, and no pedestrian is detected in the enlarged detection frame corresponding to the item, the value of the first multiple corresponding to the item is kept unchanged;

[0022] When the detection result is that the left-behind label of the item shows that the item is a left-behind item, and a pedestrian is detected in the enlarged detection frame corresponding to the item, the value of the first multiple corresponding to the item is reduced;

[0023] The left-behind object detection model is continuously trained until the detection result of the left-behind object detection model reaches a preset training standard.

[0024] Optionally, a multiple change difference is also set while setting the initial value of the first multiple;

[0025] When the detection result is that the legacy label of the item shows that the item is not a legacy item, and a pedestrian is detected in the enlarged detection frame corresponding to the item, the Euclidean distance between the detection frame of the pedestrian and the detection frame of the item is calculated, and the Euclidean distance between the detection frame of the pedestrian and the detection frame of the item is divided by the diagonal distance of the detection frame of the item to obtain a multiple base value, and the multiple change difference is subtracted from the multiple base value to obtain the adjusted first multiple corresponding to the item;

[0026] When the detection result is that the legacy label of the item shows that the item is not a legacy item, and no pedestrian is detected in the enlarged detection frame corresponding to the item, the Euclidean distance between the detection frame of the pedestrian and the detection frame of the item is calculated, and the Euclidean distance between the detection frame of the pedestrian and the detection frame of the item is divided by the diagonal distance of the detection frame of the item to obtain a multiple base value, and the multiple base value plus the multiple change difference is used to obtain the adjusted first multiple corresponding to the item;

[0027] When the detection result is that the left-behind label of the item shows that the item is a left-behind item, and a pedestrian is detected in the enlarged detection frame corresponding to the item, the Euclidean distance between the detection frame of the pedestrian and the detection frame of the item is calculated, and the Euclidean distance between the detection frame of the pedestrian and the detection frame of the item is divided by the diagonal distance of the detection frame of the item to obtain a multiple base value, and the multiple change difference is subtracted from the multiple base value to obtain the adjusted first multiple corresponding to the item.

[0028] Optionally, when the detection result is that the item's legacy tag shows that the item is not a legacy item and a pedestrian is detected in the enlarged detection frame corresponding to the item, the item's legacy tag shows that the item is not a legacy item and no pedestrian is detected in the enlarged detection frame corresponding to the item, or the item's legacy tag shows that the item is a legacy item and a pedestrian is detected in the enlarged detection frame corresponding to the item,

[0029] If the base value of the multiple calculated in the current training is within the range of (dm, d+m), the first multiple is kept unchanged, d is the base value of the multiple calculated in the previous training, and m is the multiple change difference;

[0030] If the multiple base value calculated during the current training exceeds the range of (dm, d+m), the multiple base value calculated during the current training is used to replace the multiple base value calculated during the previous training to obtain a new first multiple.

[0031] Optionally, a Bi-FPN module is used to replace the PAN-Net module in the yolov5 network model of the legacy object detection model.

[0032] Optionally, if a pedestrian is detected in the expanded detection frame of the item, it is also identified whether the pedestrian in the expanded detection frame is always the same pedestrian in N consecutive frames of images. If so, the item is determined to be a non-leftover item, otherwise, the item is determined to be a leftover item, and N is a positive integer greater than 1.

[0033] Optionally, the method further includes: when the legacy item detection model detects that an item is transformed from a legacy item to a non-legacy item, stopping outputting the depth features of the item to the item tracking model, and the item tracking model stops tracking the item.

[0034] Optionally, the alarm information includes: monitoring screen information of the left-behind items and initial left-behind time.

[0035] The present invention also provides a device for detecting leftover items based on a deep neural network, comprising:

[0036] A preprocessing module, inputting a first video, preprocessing the first video to obtain a detection image set;

[0037] The object detection module inputs the detection image set into the trained legacy object detection model, and the legacy object detection model detects and determines the objects contained in the detection image set and the depth features of each object, wherein the depth features include the color features, texture features and size features of the objects;

[0038] A multiple determination module, wherein the left-behind object detection model determines a first multiple corresponding to each object according to the depth feature of the object, and expands the width and height of the original detection frame of each object by the first multiple based on the center point of the original detection frame of each object, so as to obtain an expanded detection frame corresponding to each object;

[0039] A left-behind judgment module, wherein the left-behind object detection model detects pedestrians within the range of the enlarged detection frame corresponding to each object. If a pedestrian is detected in the enlarged detection frame of the object, the object is determined to be a non-left-behind object, otherwise the object is determined to be a left-behind object;

[0040] The tracking module extracts the deep features of the left-behind items and outputs them to the item tracking model. The item tracking model tracks the left-behind items according to the deep features of the left-behind items and records the time when the left-behind items were left;

[0041] The alarm module generates and sends an alarm message when the time left behind by the left-behind item exceeds a preset left-behind time threshold.

[0042] The present invention also provides an electronic device, comprising: a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0043] Beneficial effects of the present invention: The present invention provides a method, device and electronic device for detecting leftover items based on a deep neural network. The method uses an expanded detection frame obtained by expanding the original detection frame of the item to detect pedestrians, and judges whether the item is a leftover item based on whether the pedestrian is detected in the expanded detection frame, and then records the leftover time of the leftover item. When the leftover time exceeds the preset leftover time threshold, it is considered that a leftover event is sent, and an alarm message is generated and sent to the staff in a timely manner, which can realize full-automatic detection and alarm of leftover items and reduce the burden on the staff. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to further understand the features and technical contents of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings are only provided for reference and illustration and are not intended to limit the present invention.

[0045] In the accompanying drawings,

[0046] Figure 1 It is a flow chart of the method for detecting leftover items based on deep neural network of the present invention;

[0047] Figure 2 It is a workflow diagram of the object detection model in the method for detecting leftover objects based on a deep neural network of the present invention;

[0048] Figure 3It is a workflow diagram of the item tracking model in the method for detecting leftover items based on a deep neural network of the present invention;

[0049] Figure 4 This is a schematic diagram of the architecture of the existing yolov5 network model;

[0050] Figure 5 It is a schematic diagram of the architecture of the object detection model in the method for detecting leftover objects based on a deep neural network of the present invention;

[0051] Figure 6 and Figure 7 A schematic diagram of non-leftover items in one embodiment of the leftover item detection method based on a deep neural network of the present invention;

[0052] Figure 8 Schematic diagram of a left-behind object in an embodiment of the left-behind object detection method based on a deep neural network of the present invention

[0053] Fig. 9 It is a schematic diagram of a device for detecting leftover items based on a deep neural network according to the present invention;

[0054] Fig.10 Schematic diagram of an electronic device of the present invention. DETAILED DESCRIPTION

[0055] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be more comprehensive and complete and will fully convey the concept of the example embodiments to those skilled in the art.

[0056] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0057] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0058] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0059] See also Figure 1 , Figure 1 The flowchart of a method for detecting leftover items based on a deep neural network provided by an embodiment of the present invention is illustrated. The method for detecting leftover items based on a deep neural network comprises the following steps:

[0060] Step S1: input a first video, pre-process the first video, and obtain a detection image set.

[0061] Specifically, in some embodiments of the present invention, the left-behind item detection is mainly targeted at places with large traffic such as halls or elevators in areas such as stations, squares, and shopping malls. The first video can be a video taken by surveillance cameras in scenes such as halls and elevators. The video can be a real-time video or an offline video.

[0062] Furthermore, in step S1, the data format of the first video should comply with the preprocessing format requirements. If the user inputs a video that does not comply with the preprocessing format requirements, the algorithm will generate a warning message and prompt the user for the correct data format.

[0063] Specifically, in some embodiments of the present invention, preprocessing the first video specifically includes:

[0064] Resampling and decimating the first video to obtain a plurality of first images of a preset size;

[0065] Performing brightness and contrast enhancement on the plurality of first images respectively to obtain a plurality of second images;

[0066] The plurality of first images and the plurality of second images are used to form a detection image set.

[0067] In detail, in some embodiments of the present invention, the user also needs to set the target area when inputting the first video. The specific setting method can be to draw a polygonal area that the user wants to detect on the video screen by drawing a line with the mouse. The polygonal area is the target area. Accordingly, in the subsequent left-behind item detection process, the algorithm will only perform detection in the target area.

[0068] Furthermore, in some embodiments of the present invention, the first video is resampled and framed to obtain multiple first images of preset sizes. Specifically, the first video input by the user is resampled into video data of a size of 640*640 to reduce the occupancy of memory and video memory, and one frame is extracted from every five frames of the first video as a key frame, and these key frames are the multiple first images.

[0069] The step of respectively performing brightness and contrast enhancement on the plurality of first images to obtain the plurality of second images specifically includes performing image sharpening on the plurality of first images, then creating duplicate images, and performing brightness and contrast enhancement on the duplicate images to simulate obtaining images of scenes under different lighting conditions, namely, the plurality of second images;

[0070] Based on the multiple first images, multiple second images of scenes under different lighting conditions are obtained through data enhancement simulation. A detection image set is composed of multiple first images and multiple second images, which ultimately achieves sample expansion of the detection image set, reduces the impact of lighting changes on detection results, and improves the accuracy of detection results.

[0071] Step S2: input the detection image set into a trained abandoned object detection model. The abandoned object detection model detects and determines the objects contained in the detection image set and the depth features of each object. The depth features include at least the color features, texture features and size features of the objects.

[0072] It should be noted that the deep features are extracted from a trained legacy item detection model, which is used to reflect various features of an item, including but not limited to the color features, texture features, and size features of the item. The deep feature legacy item detection model can discover the similarities and differences between various items.

[0073] Step S3, the left-behind object detection model determines a first multiple corresponding to each object according to the depth feature of the object, and expands the width and height of the original detection frame of each object by the first multiple based on the center point of the original detection frame of each object, so as to obtain an expanded detection frame corresponding to each object;

[0074] Step S4, the left-behind item detection model detects pedestrians within the range of the enlarged detection frame corresponding to each item. If a pedestrian is detected in the enlarged detection frame of the item, the item is determined to be a non-left-behind item, otherwise the item is determined to be a left-behind item;

[0075] It should be noted that the prior art generally adopts Figure 4 The yolov5 network model shown in the figure detects object targets, but Figure 4The traditional yolov5 network model shown in FIG. 1 has the problem of insufficient robustness for small targets. Based on this, the object detection model used in the present invention is as follows: Figure 5 The yolov5 network model shown uses the Bi-FPN module to replace the PAN-Net module. Figure 5 The object detection model shown uses Bi-FPN to introduce weights to better balance feature information of different scales, while increasing feature fusion at the same level without generating new computational consumption, effectively improving the problem of insufficient robustness of the traditional yolov5 network model for small targets.

[0076] Specifically, the detection process of the left-behind object detection model is as follows: Figure 2 As shown, first, a detection image set is received, and then feature extraction is performed on the detection image set to identify objects in the detection image set. Then, a legacy determination is performed on the objects in the detection image set to identify legacy objects and non-legacy objects, and the deep features of the legacy objects are sent to the object tracking model for tracking.

[0077] It should be noted that, in order to successfully determine the first multiple of each item, the training process of the left-behind item detection model includes:

[0078] A left-behind object detection model is established, wherein the left-behind object detection model is a yolov5 network model, and a parallel first convolution layer for optimizing the first multiple is added to the last convolution layer of the yolov5 network model, and the PAN-Net module in the yolov5 network model is replaced by a Bi-FPN module.

[0079] Setting an initial value of the first multiple, such as 5 in some embodiments of the present invention;

[0080] Providing a training sample set, labeling training samples in the training sample set, and labeling a legacy label of an object in each training sample, wherein the legacy label is used to reflect whether the object is a legacy object;

[0081] Inputting training samples into the left-behind object detection model, and the left-behind object detection model detecting objects in the training samples;

[0082] Taking the center point of the original detection frame of each object as a reference, the width and height of the original detection frame of each object are enlarged by a first multiple to obtain an enlarged detection frame corresponding to each object, and pedestrians are detected within the range of the enlarged detection frame corresponding to each object;

[0083] Mapping the depth features of the detected objects and the depth features of the pedestrians into a multi-dimensional depth space through the first convolutional layer, and comparing the detection results in the depth space;

[0084] When the detection result is that the legacy tag of the item shows that the item is not a legacy item, and a pedestrian is detected in the enlarged detection frame corresponding to the item, the value of the first multiple corresponding to the item is reduced;

[0085] When the detection result is that the legacy label of the item shows that the item is not a legacy item, and no pedestrian is detected in the enlarged detection frame corresponding to the item, the value of the first multiple corresponding to the item is increased;

[0086] When the detection result is that the left-behind label of the item shows that the item is a left-behind item, and no pedestrian is detected in the enlarged detection frame corresponding to the item, the value of the first multiple corresponding to the item is kept unchanged;

[0087] When the detection result is that the left-behind label of the item shows that the item is a left-behind item, and a pedestrian is detected in the enlarged detection frame corresponding to the item, the value of the first multiple corresponding to the item is reduced;

[0088] The left-behind object detection model is continuously trained until the detection result of the left-behind object detection model reaches a preset training standard.

[0089] Further, for the reduction or increase of the first multiple, the present invention adopts the following method to determine the first multiple after the reduction or increase:

[0090] When setting the initial value of the first multiple, a multiple change difference m is also set. For example, in some embodiments of the present invention, the multiple change difference is 0.5;

[0091] When the detection result is that the legacy label of the item shows that the item is not a legacy item, and a pedestrian is detected in the enlarged detection frame corresponding to the item, the Euclidean distance d1 between the detection frame of the pedestrian and the detection frame of the item is calculated, and the Euclidean distance d1 between the detection frame of the pedestrian and the detection frame of the item is divided by the diagonal distance d2 of the detection frame of the item to obtain a multiple base value d, and the multiple change difference m is subtracted from the multiple base value d to obtain the adjusted first multiple corresponding to the item, that is, dm;

[0092] When the detection result is that the legacy label of the item shows that the item is not a legacy item, and no pedestrian is detected in the enlarged detection frame corresponding to the item, the Euclidean distance d1 between the detection frame of the pedestrian and the detection frame of the item is calculated, and the Euclidean distance d1 between the detection frame of the pedestrian and the detection frame of the item is divided by the diagonal distance d2 of the detection frame of the item to obtain a multiple base value d, and the multiple base value d is added to the multiple change difference m to obtain the adjusted first multiple corresponding to the item, that is, d+m;

[0093] When the detection result is that the item's left-behind label shows that the item is a left-behind item, and a pedestrian is detected in the enlarged detection frame corresponding to the item, the Euclidean distance d1 between the pedestrian's detection frame and the item's detection frame is calculated, and the Euclidean distance d1 between the pedestrian's detection frame and the item's detection frame is divided by the diagonal distance d2 of the item's detection frame to obtain a multiple base value d, and the multiple change difference m is subtracted from the multiple base value d to obtain the adjusted first multiple corresponding to the item, that is, dm.

[0094] In addition, in order to simplify the calculation and reduce the cost, when the difference between the first multiples obtained in the two trainings is small, the present invention will not change the first multiple to reduce the amount of data calculation and storage. Specifically, when the detection result is that the legacy label of the item shows that the item is not a legacy item and a pedestrian is detected in the enlarged detection frame corresponding to the item, the legacy label of the item shows that the item is not a legacy item and no pedestrian is detected in the enlarged detection frame corresponding to the item, or the legacy label of the item shows that the item is a legacy item and a pedestrian is detected in the enlarged detection frame corresponding to the item,

[0095] If the base value of the multiple calculated in the current training is within the range of (dm, d+m), the first multiple is kept unchanged, d is the base value of the multiple calculated in the previous training, and m is the multiple change difference;

[0096] If the multiple base value calculated during the current training exceeds the range of (dm, d+m), the multiple base value calculated during the current training is used to replace the multiple base value calculated during the previous training to obtain a new first multiple.

[0097] For example, in some embodiments of the present invention, the base value of the multiple calculated in the current training is 3.8, the base value of the multiple calculated in the previous training is 4, and the multiple change difference is 0.5. At this time, the range of (dm, d+m) is (3.5, 4.5), and 3.8 belongs to the range of (3.5, 4.5), so there is no need to update the first multiple at this time; in other embodiments of the present invention, the base value of the multiple calculated in the current training is 4.8, the base value of the multiple calculated in the previous training is 4, and the multiple change difference is 0.5. At this time, the range of (dm, d+m) is (3.5, 4.5), and 4.8 does not belong to the range of (3.5, 4.5), so it is necessary to update the first multiple and replace 4 with 4.8.

[0098] Specifically, in some embodiments of the present invention, in order to ensure the accuracy of model judgment and prevent objects appearing within the scope of the expanded detection frame from appearing together with different pedestrians, resulting in misjudgment of left-behind situations, the present invention also adds a pedestrian re-identification function, that is, pedestrians appearing within the scope of the expanded detection frame in N consecutive images are re-identified to determine whether these pedestrians are the same pedestrian. Only when the same pedestrian is detected in N consecutive images, is the item determined to be a left-behind item.

[0099] Specifically, the method further includes: if a pedestrian is detected in the expanded detection frame of the object, identifying whether the pedestrian in the expanded detection frame is always the same pedestrian in N consecutive frames of images, and if so, determining that the object is not a legacy object, otherwise determining that the object is a legacy object, where N is a positive integer greater than 1, for example Figure 7 and Figure 8 The suitcases and handbags shown in the table are non-legacy items. Figure 6 The suitcase and backpack shown are left-behind items. Preferably, in some embodiments of the present invention, N is equal to 5.

[0100] It is worth mentioning that for objects of different types and sizes, the left-over object detection model can obtain different depth features, and then determine different first multiples based on different depth features. Generally speaking, the smaller the size of the object, the larger the first multiple will be, because the smaller the object, the smaller the size of the original detection frame is usually, and in order to achieve effective pedestrian detection, the original detection frame needs to be enlarged to a larger size. In other words, the smaller the object, the larger the first multiple should be, so that the enlarged detection frame generated based on the first multiple and the original detection frame can achieve effective pedestrian detection. The left-over object detection model after training can automatically generate different first multiples corresponding to objects of different types and sizes, and then control the subsequent enlargement ratio of the original detection frame to ensure the accuracy of left-over object recognition.

[0101] Step S5: extracting the depth features of the left-behind items and outputting them to the item tracking model; the item tracking model tracks the left-behind items according to the depth features of the left-behind items and records the time when the left-behind items were left;

[0102] Specifically, in some embodiments of the present invention, the object tracking model uses the DeepSort model to track the target, and its workflow is as follows: Figure 3As shown, first, the parameter file is read, the item tracking model is parsed, and the trained item tracking model is loaded. Then, the depth features of the abandoned items determined by the item detection model are received, and a unique identity tag is assigned to each abandoned item. Then, the depth features are matched with the depth features of the abandoned items that have been received. When the depth features of the abandoned items that have been received match the depth features currently received in real time, the legacy time of the abandoned items corresponding to the depth features continues to be increased. When the depth features of the abandoned items that have been received do not match all the depth features currently received in real time, it is considered that the legacy status of the depth features of the abandoned items that have been received has changed (from a legacy item to a non-legacy item), and the tracking of the depth features of the abandoned items that have been received is stopped.

[0103] Step S4: When the time left behind by the left-behind item exceeds a preset time threshold, an alarm message is generated and sent.

[0104] Specifically, when the time a certain abandoned item has been left behind exceeds a preset threshold of the time it has been left behind, an alarm is issued to the staff, and the time when the item was first left behind is also output.

[0105] If the state of a left-behind item changes within the set time threshold and becomes a non-left-behind item, no alarm is generated and the tracking of the item is stopped. That is, when the item detection model detects that an item has changed from a left-behind item to a non-left-behind item, it stops outputting the deep features of the item to the item tracking model and the item tracking model stops tracking the item.

[0106] Furthermore, the alarm information includes: monitoring screen information of the left-behind items and the initial time of being left-behind, so that the staff who receive the information can find the left-behind items in time.

[0107] like Fig. 9 As shown, the present invention also provides an embodiment of a device for detecting leftover items based on a deep neural network, comprising:

[0108] A preprocessing module 10 inputs a first video, preprocesses the first video, and obtains a detection image set;

[0109] The object detection module 20 inputs the detection image set into the trained legacy object detection model, and the legacy object detection model detects and determines the objects contained in the detection image set and the depth features of each object, wherein the depth features include the color features, texture features and size features of the objects;

[0110] The multiple determination module 30, wherein the left-behind object detection model determines the first multiple corresponding to each object according to the depth feature of each object, and expands the width and height of the original detection frame of each object by the first multiple based on the center point of the original detection frame of each object, so as to obtain the expanded detection frame corresponding to each object;

[0111] A left-behind judgment module 40, wherein the left-behind object detection model detects pedestrians within the range of the enlarged detection frame corresponding to each object, and if a pedestrian is detected in the enlarged detection frame of the object, the object is determined to be a non-left-behind object, otherwise the object is determined to be a left-behind object;

[0112] The tracking module 50 extracts the depth features of the left-behind items and outputs them to the item tracking model. The item tracking model tracks the left-behind items according to the depth features of the left-behind items and records the time when the left-behind items were left;

[0113] The alarm module 60 generates and sends an alarm message when the time left behind by the left-behind item exceeds a preset time threshold.

[0114] like Fig.10 As shown, the present invention also provides an embodiment of an electronic device, including: a memory 100 and a processor 200, wherein the memory 100 stores a computer program, and when the computer program is executed by the processor 200, the processor 200 executes the steps of the above method.

[0115] In summary, the present invention provides a method, device and electronic device for detecting leftover items based on a deep neural network. The method uses an expanded detection frame obtained by expanding the original detection frame based on the item to detect pedestrians, and judges whether the item is a leftover item based on whether the pedestrian is detected in the expanded detection frame, and then records the leftover time of the leftover item. When the leftover time exceeds the preset leftover time threshold, it is considered that a leftover event is sent, and an alarm message is generated and sent to the staff in time, which can realize full-automatic detection and alarm of leftover items and reduce the burden on the staff.

[0116] As described above, for ordinary technicians in this field, various other corresponding changes and modifications can be made based on the technical solutions and technical concepts of the present invention, and all these changes and modifications should fall within the protection scope of the claims of the present invention.

Claims

1. A method for detecting leftover items based on deep neural network, characterized in that: The steps include: Input a first video, pre-process the first video, and obtain a detection image set; Input the detection image set into the trained legacy object detection model, and the legacy object detection model detects and determines the objects contained in the detection image set and the depth features of each object, wherein the depth features include at least the color features, texture features and size features of the objects; The left-behind object detection model determines a first multiple corresponding to each object according to the depth feature of each object, and expands the width and height of the original detection frame of each object by the first multiple based on the center point of the original detection frame of each object, so as to obtain an expanded detection frame corresponding to each object; The left-behind object detection model detects pedestrians within the range of the enlarged detection frame corresponding to each object. If a pedestrian is detected in the enlarged detection frame of the object, the object is determined to be a non-left-behind object, otherwise the object is determined to be a left-behind object; Extracting the deep features of the left-behind items and outputting them to the item tracking model, the item tracking model tracks the left-behind items according to the deep features of the left-behind items and records the time when the left-behind items were left; When the time left behind by the left-behind item exceeds the preset left-behind time threshold, an alarm message is generated and sent.

2. The method for detecting leftover items based on a deep neural network according to claim 1, characterized in that: The training process of the abandoned item detection model includes: Establishing a left-behind object detection model, wherein the left-behind object detection model is a yolov5 network model, and a parallel first convolution layer for optimizing the first multiple is added to the last convolution layer of the yolov5 network model; Set the initial value of the first multiple; Providing a training sample set, labeling training samples in the training sample set, and labeling a legacy label of an object in each training sample, wherein the legacy label is used to reflect whether the object is a legacy object; Inputting training samples into the left-behind object detection model, and the left-behind object detection model detecting objects in the training samples; Taking the center point of the original detection frame of each object as a reference, the width and height of the original detection frame of each object are enlarged by a first multiple to obtain an enlarged detection frame corresponding to each object, and pedestrians are detected within the range of the enlarged detection frame corresponding to each object; Mapping the depth features of the detected objects and the depth features of the pedestrians into a multi-dimensional depth space through the first convolutional layer, and comparing the detection results in the depth space; When the detection result is that the legacy tag of the item shows that the item is not a legacy item, and a pedestrian is detected in the enlarged detection frame corresponding to the item, the value of the first multiple corresponding to the item is reduced; When the detection result is that the legacy label of the item shows that the item is not a legacy item, and no pedestrian is detected in the enlarged detection frame corresponding to the item, the value of the first multiple corresponding to the item is increased; When the detection result is that the left-behind label of the item shows that the item is a left-behind item, and no pedestrian is detected in the enlarged detection frame corresponding to the item, the value of the first multiple corresponding to the item is kept unchanged; When the detection result is that the left-behind label of the item shows that the item is a left-behind item, and a pedestrian is detected in the enlarged detection frame corresponding to the item, the value of the first multiple corresponding to the item is reduced; The left-behind object detection model is continuously trained until the detection result of the left-behind object detection model reaches a preset training standard.

3. The method for detecting leftover items based on a deep neural network as claimed in claim 2, characterized in that: When setting the initial value of the first multiple, a multiple change difference is also set; When the detection result is that the legacy label of the item shows that the item is not a legacy item, and a pedestrian is detected in the enlarged detection frame corresponding to the item, the Euclidean distance between the detection frame of the pedestrian and the detection frame of the item is calculated, and the Euclidean distance between the detection frame of the pedestrian and the detection frame of the item is divided by the diagonal distance of the detection frame of the item to obtain a multiple base value, and the multiple change difference is subtracted from the multiple base value to obtain the adjusted first multiple corresponding to the item; When the detection result is that the legacy label of the item shows that the item is not a legacy item, and no pedestrian is detected in the enlarged detection frame corresponding to the item, the Euclidean distance between the detection frame of the pedestrian and the detection frame of the item is calculated, and the Euclidean distance between the detection frame of the pedestrian and the detection frame of the item is divided by the diagonal distance of the detection frame of the item to obtain a multiple base value, and the multiple base value plus the multiple change difference is used to obtain the adjusted first multiple corresponding to the item; When the detection result is that the left-behind label of the item shows that the item is a left-behind item, and a pedestrian is detected in the enlarged detection frame corresponding to the item, the Euclidean distance between the detection frame of the pedestrian and the detection frame of the item is calculated, and the Euclidean distance between the detection frame of the pedestrian and the detection frame of the item is divided by the diagonal distance of the detection frame of the item to obtain a multiple base value, and the multiple change difference is subtracted from the multiple base value to obtain the adjusted first multiple corresponding to the item.

4. The method for detecting leftover items based on a deep neural network as claimed in claim 3, characterized in that: When the detection result is that the item's legacy tag shows that the item is not a legacy item and a pedestrian is detected in the enlarged detection frame corresponding to the item, the item's legacy tag shows that the item is not a legacy item and no pedestrian is detected in the enlarged detection frame corresponding to the item, or the item's legacy tag shows that the item is a legacy item and a pedestrian is detected in the enlarged detection frame corresponding to the item, If the base value of the multiple calculated in the current training is within the range of (dm, d+m), the first multiple is kept unchanged, d is the base value of the multiple calculated in the previous training, and m is the multiple change difference; If the multiple base value calculated during the current training exceeds the range of (dm, d+m), the multiple base value calculated during the current training is used to replace the multiple base value calculated during the previous training to obtain a new first multiple.

5. The method for detecting leftover items based on a deep neural network as claimed in claim 2, characterized in that: The Bi-FPN module is used to replace the PAN-Net module in the yolov5 network model of the legacy object detection model.

6. The method for detecting leftover items based on a deep neural network according to claim 1, characterized in that: If a pedestrian is detected in the enlarged detection frame of the item, it is also identified whether the pedestrian in the enlarged detection frame is always the same pedestrian in N consecutive frames of images. If so, the item is determined to be a non-leftover item, otherwise, the item is determined to be a leftover item, and N is a positive integer greater than 1.

7. The method for detecting leftover items based on a deep neural network according to claim 1, characterized in that: Also includes: When the legacy item detection model detects that an item is transformed from a legacy item to a non-legacy item, it stops outputting the deep features of the item to the item tracking model, and the item tracking model stops tracking the item.

8. The method for detecting leftover items based on a deep neural network according to claim 1, characterized in that: The alarm information includes: monitoring screen information of the left-behind items and the initial left-behind time.

9. A device for detecting leftover items based on a deep neural network, characterized in that: include: A preprocessing module, inputting a first video, preprocessing the first video to obtain a detection image set; The object detection module inputs the detection image set into the trained legacy object detection model, and the legacy object detection model detects and determines the objects contained in the detection image set and the depth features of each object, wherein the depth features include the color features, texture features and size features of the objects; A multiple determination module, wherein the left-behind object detection model determines a first multiple corresponding to each object according to the depth feature of the object, and expands the width and height of the original detection frame of each object by the first multiple based on the center point of the original detection frame of each object, so as to obtain an expanded detection frame corresponding to each object; A left-behind judgment module, wherein the left-behind object detection model detects pedestrians within the range of the enlarged detection frame corresponding to each object. If a pedestrian is detected in the enlarged detection frame of the object, the object is determined to be a non-left-behind object, otherwise the object is determined to be a left-behind object; The tracking module extracts the deep features of the left-behind items and outputs them to the item tracking model. The item tracking model tracks the left-behind items according to the deep features of the left-behind items and records the time when the left-behind items were left; The alarm module generates and sends an alarm message when the time left behind by the left-behind item exceeds a preset left-behind time threshold.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 8.

Images