Pallet residue detection method based on target tracking detection

Abstract

The present invention discloses a pallet residue detection method based on target tracking detection, comprising: step A: defining a preset detection area range in a camera shooting area, cutting a preset part of a conveyor belt area, and sending the detection area to a detection algorithm to detect the pallet area; step B: using a Kalman filter algorithm and a tracking algorithm based on a maximum IOU to track the pallet area in a video frame; step C: selecting a plurality of designated pallet area images, and judging whether there is a light reflection area in the pallet area image; step D: using a light reflection removal algorithm to remove the light reflection in the pallet area image for the pallet area image with the reflected light area; step E: detecting the residue in the pallet area image after removing the light reflection, and determining whether there is the residue in the pallet.

Description

Technical Field

[0001] The present invention relates to the field of intelligent security inspection technology, and in particular to a method for detecting leftover objects on a pallet based on target tracking detection. Background Art

[0002] When passengers travel by high-speed trains, motor vehicles, or airplanes, they need to check in and transport their carry-on luggage. When it comes to pallet transportation, if the number of items is large or they are relatively fragmented, it is easy for items to be left on the pallet. If they are not discovered and informed to the passengers in time, the items will be lost, causing inconvenience to the passengers and staff in finding them and the risk of missing flights or trains. In addition, most luggage trays in the existing technology are checked by staff, and when there is a large flow of people, it is very easy to miss items. The existing target detection algorithm has difficulty dealing with the problem of self-reflection of the items in the pallet and the reflection of the pallet, and cannot solve the problem of missed detection or false detection. Leftover items also affect the storage and recycling of the pallet. Summary of the Invention

[0003] One object of the present invention is to provide a method for detecting leftover objects on a pallet based on target tracking detection, which combines a traditional image processing algorithm with a light reflection removal algorithm. The traditional algorithm is used to determine whether there are obvious reflective areas in the pallet, and the light reflection algorithm is used to determine whether there are any missed items in the pallet, thereby solving the problems of missed detection and false detection caused by the reflection of the pallet material or the material of the item itself.

[0004] Other advantages and features of the present invention will become more apparent from the following detailed description and will be realized by means of the instrumentalities and combinations particularly pointed out in the appended claims.

[0005] According to one aspect of the present invention, a method for detecting leftover objects on a pallet based on target tracking detection, which can achieve the aforementioned objectives and other objectives and advantages, includes:

[0006] Step A: Define a preset detection area within the camera's shooting area, crop a preset portion of the conveyor belt area, and send the detection area to the detection algorithm for pallet area detection;

[0007] Step B: Use the Kalman filter algorithm and the maximum IOU-based tracking algorithm to track the pallet area in the video frame;

[0008] Step C: selecting a plurality of designated tray area images and determining whether there is a light reflection area in the tray area images;

[0009] Step D: using a light reflection removal algorithm to remove light reflections from the tray area image where the reflected light area exists;

[0010] Step E: Detecting left objects in the tray area image after light reflection is removed to determine whether there are left objects in the tray.

[0011] According to one embodiment of the present invention, step A includes sub-step A1: obtaining a pallet area and four vertex areas of the pallet area through a detection algorithm, obtaining a coordinate frame of the four vertex areas through a target detection algorithm, assuming that the coordinates of the upper left vertex of the coordinate frame of any vertex area are (x0, y0) and the width and height of the coordinate frame of the vertex area are represented by w and h respectively, then the position of the vertex of the pallet area is the middle point of the coordinate frame, and the coordinates of the vertex, i.e., the middle point, are (x0+0.5*w, y0+0.5*h).

[0012] According to one embodiment of the present invention, step B includes sub-step B1: combining a Kalman filter algorithm with a maximum IOU tracking algorithm, wherein each pallet area is represented by a state vector as input to the Kalman filter algorithm. In the first frame image, a corresponding Kalman filter is initialized for each detected pallet area. During the prediction phase, in each frame image, the Kalman filter is used to perform state prediction for each pallet area, providing a predicted position of the pallet area in the next frame image. For each pallet area in the current frame, the IOU value between the pallet area and the target pallet is calculated, wherein the pallet area with the largest IOU value is considered to be the most likely target to be tracked. Based on the result of the maximum IOU value tracking, the detected pallet area is used to update the pallet area state in the Kalman filter. The pallet area state includes but is not limited to: successful tracking, untracked. For pallet areas that have not been tracked, they are regarded as new targets and the Kalman filter is initialized. For pallet areas that have not been successfully tracked multiple times, they are marked as missing targets and tracking is stopped. The above steps are then repeated to continue iterative tracking of the pallet area.

[0013] According to one embodiment of the present invention, step C includes sub-step C1: continuously capturing video frames of a pallet moving on a conveyor belt, and continuously tracking the same pallet; calculating changes in the area of ​​the pallet region image through a target detection algorithm; selecting three complete pallet region images, namely, a complete pallet region image entering the detection area, an image of the pallet region directly under the light, and an image of the complete pallet region about to be shipped out of the detection area; continuously detecting and tracking each pallet region image, and calculating the corresponding pallet region image area.

[0014] According to one embodiment of the present invention, step C includes sub-step C2: performing light reflection detection on the three pallet area images to obtain the light reflection area in the pallet area image: by calculating the dark channels of the three pallet area images, assuming that any pallet area image is V, first finding the minimum value of each pixel (r, g, b) in the three channels, where r, g, and b represent the pixel value sizes of the three channels, respectively, and the minimum value calculation formula is minP = min(r, g, b), generating a grayscale image with the same width and height as the original image through the minimum value, using minimum value filtering for smoothing, and filtering the grayscale image using minimum value filtering, the minimum value filtering step includes: defining the size and shape of the filter; placing the center of the filter at each pixel value of the pallet image; finding the minimum pixel value within the filter range; assigning the minimum pixel value to the current pixel value, and obtaining the filtered result image through the above steps, in the obtained grayscale image, taking a rectangular window of a certain size with each pixel as the center, and taking the minimum grayscale value in the rectangular window instead of the value of the center pixel, thereby obtaining the dark channel image corresponding to the original image.

[0015] According to one embodiment of the present invention, step C includes sub-step C3: calculating the connected domain of the dark channel image and obtaining the light reflection area by image segmentation: using the connectedComponents function in openCV, calculating the connected area on the pallet image, and performing connected domain segmentation, assuming that the reflective area in each pallet area image is an independent area without adhesion, finding a non-zero pixel value and starting to judge until all connected non-zero pixels are traversed and marked, then the position of the pallet area of ​​the reflected light can be determined, wherein the non-zero pixel is the reflective area.

[0016] According to one embodiment of the present invention, step C includes sub-step C4: completing the light reflection area search of the three pallet area images, including: 1> If there is no light reflection area on any two pallet area images, the two pallet area images are sent to step D to detect the left objects in the pallet, which is divided into the following situations: when no objects are detected in any two pallet area images, it is judged that there are no left objects in the pallet; when any one of the two pallet area images does not detect the left objects, but the other detects the left objects, the result is pending, and the third pallet area image where the light reflection is detected needs to be sent to the light reflection removal algorithm for The light reflection is removed, and then the image is sent to step five for pallet left-behind detection to determine whether there is any left-behind item on the corresponding pallet; 2> If any two pallet area images or three pallet area images contain light reflection areas, the pallet area image with light reflection needs to be sent to the light reflection removal algorithm, and then the result image after light reflection removal and the image without light reflection are sent to the target detection algorithm to obtain the target detection result, and finally the voting rule is: when any two pallet area images among the three pallet area images are detected as left-behind items or confirmed to have no left-behind items, it is considered that there is or does not exist left-behind items in the corresponding pallet.

[0017] According to one embodiment of the present invention, step D includes sub-step D1: the light reflection removal algorithm uses the neural network UGG19 network structure to remove the light reflection image calculation, and the image without light reflection damage is recorded as , the light reflection damaged image is recorded as , the light reflection removed image is recorded as , The calculation formula is , where ⨀ represents element-by-element multiplication, M represents the light reflection mask, M is the light reflection detection result of step 3, and f represents the neural network algorithm. Represents the network weight.

[0018] According to one embodiment of the present invention, step D includes sub-step D2: details of the training process for obtaining the light reflection removal image: the input image size in the network structure is 512*512*3, the UGG19 network structure is used to extract features of the light reflection image, the pallet area image is input into VGG-19, and the image features of conv1_2, conv2_2, conv3_2, conv4_2 and conv5_2 are extracted. The five features and the input pallet area image are spliced ​​together to form an image feature with 1475 channels. A convolution with a convolution kernel of 1*1 is used to obtain an image feature with 64 channels. An 8-layer convolution with a convolution kernel of 3*3 is used, and the number of channels in the intermediate feature layer is kept at 64. A convolution with a convolution kernel of 1*1 is used to obtain a convolution feature with a channel number of 3, which is recorded as the light reflection removal image. During the training process, the loss function for ,in, Represents the image with light reflection removed The features extracted by the convolutional layer in VGG19, l represents the five convolution layers conv1_2, conv2_2, conv3_2, conv4_2 and conv5_2, respectively. Represents a matte reflective image The convolution features extracted in conv1_2, conv2_2, conv3_2, conv4_2 and conv5_2 respectively, It represents the absolute value operation in mathematics. VGG19 obtains the network structure of the image with light reflection removed.

[0019] According to one embodiment of the present invention, step E includes sub-step E1: using a target detection algorithm, when inputting the pallet area image into the target detection network structure, based on the placement and shape of the pallet in the actual scene, the input size of the pallet area image is selected as 640*640, and target detection is performed on the left items in the pallet through the target detection algorithm to determine whether there are left items in the pallet area image.

[0020] The beneficial effects of the present invention are: through traditional image processing and light reflection removal algorithm processing, it solves the problem of items being left behind on the pallet during the check-in process, promptly discovers and reminds passengers to put away their own items to avoid causing property loss to passengers, reduces labor costs, and alleviates the burden on staff. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 The present invention is a flowchart of a method for detecting leftover objects on a pallet based on target tracking detection according to an embodiment of the present invention.

[0022] Figure 2Schematic diagram of a pallet usage scenario of the pallet residue detection method based on target tracking detection according to the above embodiment of the present invention.

[0023] Figure 3 Schematic diagram of pallet detection results of the pallet residue detection method based on target tracking detection according to the above embodiment of the present invention.

[0024] Figure 4 Schematic diagram of pallet vertex calculation in the pallet residue detection method based on target tracking detection according to the above embodiment of the present invention.

[0025] Figure 5 This is a diagram of an algorithm framework for predicting the pallet area and the four vertex areas of the pallet in the pallet residue detection method based on target tracking detection according to the above embodiment of the present invention.

[0026] Figure 6 This is an algorithm process diagram of Kalman filtering and maximum IOU tracking in the pallet residue detection method based on target tracking detection in the above embodiment of the present invention.

[0027] Figure 7 This is a curve of the change in the pallet area of ​​the pallet residue detection method based on target tracking detection according to the above embodiment of the present invention.

[0028] Figure 8 Schematic diagram of the areas where a pallet is located in three states of the pallet residue detection method based on target tracking detection according to the above embodiment of the present invention.

[0029] Figure 9 Schematic diagram of the network structure of the method for detecting leftover objects on a pallet based on target tracking detection according to the above embodiment of the present invention, in which light reflection images are removed.

[0030] Figure 10 This is an algorithm framework diagram for detecting left-behind objects in the pallet left-behind detection method based on target tracking detection according to the above embodiment of the present invention.

[0031] Figure 11 This is a before-and-after comparison diagram of the method for detecting pallet residues based on target tracking detection in the above embodiment of the present invention, in which an attention mechanism module for enhancing the performance of the convolutional network is added. DETAILED DESCRIPTION

[0032] The following description is intended to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are for illustrative purposes only, and those skilled in the art will readily appreciate other obvious variations. The basic principles of the present invention defined in the following description may be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the present invention.

[0033] Those skilled in the art should understand that, in the disclosure of the present invention, the terms "longitudinal", "transverse", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like to indicate orientations or positional relationships are based on the orientations or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, the above terms should not be understood as limiting the present invention.

[0034] It is understood that the term "a" should be understood as "at least one" or "one or more." That is, in one embodiment, the number of an element may be one, while in another embodiment, the number of the element may be multiple, and the term "a" should not be understood as limiting the quantity. In addition, in the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0035] In the description of the present invention, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through a medium. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0036] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0037] like Figures 1 to 10As shown, a method for detecting objects left on a tray based on target tracking detection according to an embodiment of the present invention is disclosed. The method can detect objects left on a tray, thereby promptly reminding passengers to take their belongings with them, reducing the workload of staff. Specifically, the method includes the following steps:

[0038] Step 1: Perform pallet detection on the pallets in the designated area. Specifically, in the usage scenario, the pallets transported by the conveyor belt are always in a moving state. In order to more accurately detect the left-behind items in the pallets, a preset detection area is defined in the camera shooting area, and the pallets on the conveyor belt within the detection area are detected for left-behind items.

[0039] Furthermore, if Figure 2 As shown, the conveyor belt area surrounded by the black line is cropped, and only the detection area is sent to the detection algorithm for pallet area detection. The detection algorithm of the pallet area can be but is not limited to using a deep learning detection algorithm to detect the positions of the pallet area and the four vertices of the pallet area. The detection results are shown in Figure 3 As shown in FIG, in detail, the pallet area and the four vertex areas of the pallet area are obtained through the detection algorithm. The calculation process of the four vertex areas of the pallet area is as follows: Figure 4 As shown, the coordinate frames of the four vertex areas are obtained through the target detection algorithm. Suppose the coordinates of the upper left vertex of the coordinate frame of any vertex area is (x0, y0) and the width and height of the coordinate frame of the vertex area are represented by w and h respectively. Then the position of the vertex of the pallet area is the middle point of the coordinate frame, and the coordinates of the vertex, i.e. the middle point, are (x0+0.5*w, y0+0.5*h). The four vertex areas of the pallet area are detected to realize auxiliary detection of the pallet area. This can avoid the situation where the pallet area is missed due to the fact that the items left in the pallet are too large and block part of the pallet area captured by the camera. Furthermore, when the pallet area is missed, the pallet area can be deducted from the coordinates of the four vertex positions of the pallet area, which is beneficial to provide a tracking target for subsequent pallet area tracking. The algorithm framework of this step is as follows: Figure 5 As shown;

[0040] Step 2: Use the Kalman filter algorithm and the maximum IOU tracking algorithm to track the trajectory of the pallet area. In detail, the Kalman filter algorithm and the maximum IOU-based tracking algorithm are used to track the pallet area in the video frame. The maximum IOU tracking algorithm measures the degree of overlap of two bounding boxes by calculating the ratio between the intersection area and the union area. The calculation formula is , represents the intersection of two areas, represents the union of two areas, represents the prediction box, represents the true frame,

[0041] In the present invention, the Kalman filter algorithm and the maximum IOU tracking algorithm are combined and used. The algorithm framework is as follows: Figure 6 The detailed steps are as follows:

[0042] 1) Target state representation: Each pallet area is represented by a state vector, which includes information such as the position, velocity, and acceleration of the pallet area, and serves as the input of the Kalman filter algorithm;

[0043] 2) Initialize the Kalman filter: In the first frame, a corresponding Kalman filter is initialized for each detected pallet area. The Kalman filter is used to predict and update the target state. During the prediction phase, the Kalman filter is used to predict the state of each pallet area in each frame, thereby providing a predicted position of the pallet area in the next frame.

[0044] 3) Tracking using maximum IOU: For each pallet area in the current frame, the IOU value between the pallet area and the target pallet is calculated. The pallet area with the largest IOU value is considered the most likely target to be tracked.

[0045] 4) Update location information: Based on the results of the maximum IOU tracking, the detected pallet area is used to update the pallet area status in the Kalman filter. The pallet area status includes but is not limited to: tracking successful, not tracked (including missing);

[0046] 5) For any pallet area that has not been tracked, treat it as a new target and initialize the Kalman filter. For any pallet area that has not been tracked successfully for multiple times, mark it as a missing target and stop tracking. Preferably, the number of tracking times is three.

[0047] 6) Iterative processing: repeat the above steps to continue iterative tracking of the pallet area;

[0048] Step 3: Select multiple designated pallet area images, use a traditional imaging algorithm to process them, and determine whether there is a light reflection area in the pallet area images. Preferably, the traditional imaging algorithm is a dark channel light reflection area search algorithm. Preferably, the number of pallet area images is three. Specifically, three pallet area images are selected to determine whether light reflection detection is required for the three pallet area images. The detailed steps are as follows:

[0049] 1) Select three pallet area images: For pallets moving on the conveyor belt, continuously capture video frames and continuously track the same pallet. Use the target detection algorithm to calculate the change in the pallet area image. Select three complete pallet area images, namely, the complete pallet area image A entering the detection area, the pallet area image B directly under the light, and the complete pallet area image C about to leave the detection area. Continuously detect and track each pallet area image and calculate the corresponding pallet area image area, so as to Figure 7 The area change curve of the tray area image is shown as Figure 8 The diagram shows the areas where the pallet area images A, B, and C are located in three different states of the same pallet. It is worth noting that the three pallet area images are selected so that when any two pallet area images are detected as having leftover items or confirmed to have no leftover items, it is considered that there is or is not leftover items in the corresponding pallet.

[0050] In actual application scenarios, after a pallet enters the detection area, when the area of ​​the pallet area image begins to stabilize, the first pallet area image A is obtained. When the area of ​​the pallet area image begins to decrease steadily, the previous image where the area begins to decrease is selected as the third pallet area image C, and the middle position of the area stability value is selected as the second pallet area image B. The reason for selecting this is that in the hardware device of the present invention, the lighting source is in the middle area of ​​the conveyor belt. This middle area is the brightest and is most likely to detect leftover items. In addition, this middle area is most likely to produce a light reflection area, which can easily affect the detection of leftover items.

[0051] Furthermore, light reflection detection is performed on the three pallet area images to obtain the light reflection areas in the pallet area images. The detailed steps are as follows:

[0052] a) Calculate the dark channels of the three pallet area images. Assume that any pallet area image is V. First, find the minimum value of each pixel in the three channels (r, g, b), where r, g, and b represent the pixel values ​​of the three channels, respectively. The minimum value calculation formula is minP = min(r, g, b).

[0053] b) generating a grayscale image with the same width and height as the original image through the minimum value, performing smoothing using a minimum value filter, and filtering the grayscale image using the minimum value filter, wherein the minimum value filtering step comprises: defining the size and shape of the filter, preferably using a square or rectangular filter; placing the center of the filter at each pixel value of the tray image; finding the minimum pixel value within the filter area; and assigning the minimum pixel value to the current pixel value;

[0054] The filtered result image is obtained through the above steps. In the obtained grayscale image, a rectangular window of a certain size is taken with each pixel as the center. Preferably, the size of the present invention is 15x15. The minimum value of the grayscale value in the rectangular window is taken instead of the value of the central pixel to obtain the dark channel image corresponding to the original image;

[0055] 2) Calculate the connected domain of the dark channel image and obtain the light reflection area by image segmentation. Specifically, preferably, the seed filling method is used to calculate the connected domain. This paper uses the connectedComponents function in openCV to calculate the connected domain on the pallet image and perform connected domain segmentation. Preferably, the CFS connected domain segmentation method is used. It is assumed that the reflective area in each pallet area image is an independent area without adhesion. Find a non-zero pixel value and start judging until all connected non-zero pixels are traversed and marked. The position of the pallet area that reflects light can be determined. Among them, the non-zero pixel is the reflective area. The algorithm steps are as follows:

[0056] a) Traverse the connected domain graph from left to right and from top to bottom. If a non-zero pixel is encountered and the pixel has not been visited, save the pixel position and mark it as visited.

[0057] b) Continue to explore the surrounding 8 pixels and continue to search for non-zero pixels until all adjacent pixels are 0. After searching all non-zero pixels, it means that a tray light reflection area has been found;

[0058] c) Repeat the above steps until all pixels are detected and the light reflection mask M is obtained;

[0059] d) Finally, determine whether the three pallet area images need to use the light reflection removal algorithm;

[0060] Through the above steps 1) and 2), the light reflection area search of the three pallet area images is completed, including the following situations:

[0061] 1> If there is no light reflection area on any two pallet area images, the two pallet area images are sent to step five for detection of left-behind items on the pallet, which is divided into the following situations: when no items are detected in any two pallet area images, it is determined that there are no left-behind items on the pallet; when no left-behind items are detected in any one of the two pallet area images, but a left-behind item is detected in the other, the result is undetermined, and the third pallet area image with light reflection detected needs to be sent to the light reflection removal algorithm for light reflection removal, and then sent to step five for pallet left-behind item detection, so as to determine whether there are left-behind items on the corresponding pallet;

[0062] 2> If any two or three pallet area images contain light reflection areas, the pallet area images with light reflection need to be fed into the light reflection removal algorithm, and then the resulting images after light reflection removal and the images without light reflection are fed into the target detection algorithm to obtain the target detection results. Finally, the target detection results are determined by voting. The voting rules are as follows: when any two of the three pallet area images are detected as left-behind items or confirmed to be free of left-behind items, it is considered that there are or are not left-behind items in the corresponding pallet. Furthermore, it is confirmed whether there are left-behind items in the pallet. In addition, light reflection removal can also confirm whether the light reflection is caused by the pallet's own material or by the left-behind items in the pallet, thereby avoiding the problems of missed detection and false detection caused by directly using the target detection algorithm, which affects the normal use of the luggage tray.

[0063] Step 4: For the image of the pallet area with light reflection, use the light reflection removal algorithm to remove the light reflection in the pallet area image. Specifically, the light reflection removal algorithm uses the neural network UGG19 network structure to calculate the image without light reflection damage. The image without light reflection damage is recorded as , the light reflection damaged image is recorded as , the light reflection removed image is recorded as , The calculation formula is ,

[0064] Among them, ⨀ represents element-by-element multiplication, M represents the light reflection mask, M is the light reflection detection result of step 3, and f represents the neural network algorithm. Representing the network weights, the training process details of the light reflection removal image acquisition are as follows:

[0065] The input image size in the network structure is 512*512*3. The UGG19 network structure is used to extract the features of the light reflection image. The pallet area image is input into VGG-19, and the image features of conv1_2, conv2_2, conv3_2, conv4_2 and conv5_2 are extracted. The five features and the input pallet area image are spliced ​​together to form an image feature with 1475 channels. The convolution kernel is 1*1, and the image feature with 64 channels is obtained. The convolution kernel is 3*3 for 8 layers, and the number of channels in the middle feature layer is kept at 64. The convolution kernel is 1*1, and the convolution feature with 3 channels is obtained. It is recorded as the light reflection image removed. During the training process, the loss function Recorded as ,

[0066] in, Represents the image with light reflection removed The features extracted by the convolutional layer in VGG19, l represents the five convolution layers conv1_2, conv2_2, conv3_2, conv4_2 and conv5_2, respectively. Represents a matte reflective image The convolution features extracted in conv1_2, conv2_2, conv3_2, conv4_2 and conv5_2 respectively, Indicates the absolute value operation in mathematics. VGG19 obtains the network structure of the image with light reflection removed, such as Figure 9 As shown;

[0067] Step 5: Detect the left objects in the pallet area image after removing the light reflection to determine whether there are any left objects in the pallet. Specifically, use a target detection algorithm, such as yolov5. When inputting the pallet area image into the target detection network structure, the input size of the pallet area image is selected as 640*640 according to the placement and shape of the pallet in the actual scene. The target detection algorithm is used to detect the left objects in the pallet to determine whether there are any left objects in the pallet area image. The algorithm framework of this step is as follows: Figure 10 As shown,

[0068] It is worth noting that the target detection training network structure consists of four parts: backbone, neck, head, and loss function. In this step, since the items left in the tray are mostly small targets that are easily missed, during training, the C3 structure in the head position of yolo5s is added to the CBAM (Convolutional Block Attention Module). CBAM consists of channel attention and spatial attention, which can simulate the human eye to focus on the foreground target that should be paid attention to. By adding this module, the present invention prompts the target detection algorithm to pay more attention to the foreground target, that is, the items left in the tray. The changes in the network structure of some parts of the head before and after the addition of this module are as follows: Figure 11 As shown, Figure 11 The left branch is the network structure of the prior art, and the right branch is the network structure after adding CBAM. Since small targets occupy fewer pixels in the entire image, slight offsets are likely to occur when matching the predicted frame and the target frame, resulting in prediction frame matching failure. Therefore, the present invention uses the NWD loss function to replace the conventional IOU loss function with the NWD loss function, thereby optimizing the missed detection problem of small target detection. The calculation formula of the NWD loss function is: ,in, ,

[0069] in, ,

[0070] Among them, the two bounding boxes a and b are represented as , for two bounding box representations 、 , in the matching process, the bounding box of the bounding box a represents , the bounding box representation of bounding box b , is the second-order Wasserstein distance, is the Wasserstein distance normalization, T is the transpose of the matrix, C represents a constant, and the default value of C is 12.8.

[0071] Furthermore, the NWD loss function is used to train the network structure for target detection, so as to more accurately determine whether there are leftover items in the pallet area image and reduce the missed detection of small targets left on the pallet.

Claims

1. A pallet residue detection method based on target tracking detection, characterized in that: include: Step A: defining a preset detection area range in the camera shooting area, cutting out a preset part of the conveyor belt area, and sending the detection area to the detection algorithm for pallet area detection; Step A includes sub-step A1: obtaining the pallet area and four vertex areas of the pallet area through the detection algorithm, obtaining the coordinate frames of the four vertex areas through the target detection algorithm, assuming that the coordinates of the upper left vertex of the coordinate frame of any vertex area are (x0, y0) and the width and height of the coordinate frame of the vertex area are represented by w and h respectively, then the position of the vertex of the pallet area is the middle point of the coordinate frame, and the coordinates of the vertex, i.e. the middle point, are (x0+0.5*w, y0+0.5*h); Step B: Use the Kalman filter algorithm and the maximum IOU-based tracking algorithm to track the pallet area in the video frame; Step C: selecting a plurality of designated pallet area images, and determining whether there is a light reflection area in the pallet area image; said step C comprises sub-step C1: continuously capturing video frames of a pallet moving on a conveyor belt, and continuously tracking the same pallet, calculating the change in the area of ​​the pallet area image through a target detection algorithm, selecting three complete pallet area images, namely, a complete pallet area image entering the detection area, a pallet area image directly under the light, and a complete pallet area image to be consigned out of the detection area, continuously detecting and tracking each pallet area image, and calculating the corresponding pallet area image area; Step D: for the tray area image where the reflected light area exists, using a light reflection removal algorithm to remove the light reflection in the tray area image; Step E: Use the target detection algorithm to detect the leftover objects in the tray area image with light reflection removed to determine whether there are leftover objects in the tray.

2. The pallet residue detection method according to claim 1, characterized in that: The step B includes sub-step B1: combining the Kalman filter algorithm and the maximum IOU tracking algorithm, each pallet area is represented by a state vector as the input of the Kalman filter algorithm, in the first frame image, for the detected pallet area, a corresponding Kalman filter is initialized, in the prediction stage, in each frame image, for each pallet area, the Kalman filter is used to perform state prediction, providing the predicted position of the pallet area in the next frame image, for each pallet area in the current frame, the IOU value between the pallet area and the target pallet is calculated, wherein the pallet area with the largest IOU value is considered to be the most likely target to be tracked, according to the result of the maximum IOU value tracking, the detected pallet area is used to update the pallet area state in the Kalman filter, the pallet area state includes: successful tracking, untracked, for the pallet area that has not been tracked, it is regarded as a new target and the Kalman filter is initialized, for the pallet area that has not been successfully tracked for multiple times, it is marked as a missing target and tracking is stopped, and then the above steps are repeated to continue iterative tracking of the pallet area.

3. The pallet residue detection method according to claim 2, characterized in that: The step C includes a sub-step C2: performing light reflection detection on the three pallet area images to obtain the light reflection area in the pallet area images: by calculating the dark channels of the three pallet area images, assuming that any pallet area image is V, first finding the minimum value of each pixel Pixel (r, g, b) in the three channels, wherein r, g, and b represent the pixel value sizes of the three channels respectively, and the minimum value calculation formula is minP = min(r, g, b), generating a grayscale image with the same width and height as the original image through the minimum value, using the minimum value filter for smoothing, and using the minimum value filter to filter the grayscale image, the minimum value filtering step includes: defining the size and shape of the filter; placing the center of the filter at each pixel value of the pallet image; finding the minimum pixel value within the filter domain; assigning the minimum pixel value to the current pixel value, and obtaining the filtered result image through the above steps, in the obtained grayscale image, taking a rectangular window of a certain size with each pixel as the center, taking the minimum value of the grayscale value in the rectangular window instead of the value of the central pixel, thereby obtaining the dark channel image corresponding to the original image.

4. The pallet residue detection method according to claim 3, characterized in that: The step C includes sub-step C3: calculating the connected domain of the dark channel image and obtaining the light reflection area by image segmentation: using the connectedComponents function in openCV to calculate the connected area on the pallet image and perform connected domain segmentation, assuming that the reflective area in each pallet area image is an independent area without adhesion, finding a non-zero pixel value and starting to judge until all connected non-zero pixels are traversed and marked, then the position of the pallet area of ​​the reflected light can be judged, wherein the non-zero pixel is the reflective area.

5. The pallet residue detection method according to claim 4, characterized in that: The step C includes a sub-step C4: completing the light reflection area search of the three pallet area images, including: 1> if there is no light reflection area on any two pallet area images, the two pallet area images are sent to step D to detect the leftover items in the pallet, which is divided into the following situations: when no items are detected in any two pallet area images, it is determined that there are no leftover items in the pallet; when any one of the two pallet area images does not detect the leftover items, but the other detects the leftover items, the result is pending, and the third pallet area image with detected light reflection needs to be sent to the light reflection removal algorithm to remove the light reflection, Then proceed to step five to detect items left on the pallet, so as to determine whether there are any items left on the corresponding pallet; 2> If any two pallet area images or three pallet area images contain light reflection areas, the pallet area images with light reflection need to be sent to the light reflection removal algorithm, and then the result image after light reflection removal and the image without light reflection are sent to the target detection algorithm to obtain the target detection result, and finally vote. The voting rules are: when any two of the three pallet area images are detected with leftover items or confirmed to have no leftover items, it is considered that there are or no leftover items in the corresponding pallet.

6. The pallet residue detection method according to claim 5, characterized in that: The step D includes sub-step D1: the light reflection removal algorithm uses the neural network VGG19 network structure to remove the light reflection image calculation, and the image without light reflection damage is recorded as , the light reflection damaged image is recorded as , the light reflection removed image is recorded as , The calculation formula is ， Among them, ⨀ represents element-by-element multiplication, M represents the light reflection mask, M is the light reflection detection result of step 3, and f represents the neural network algorithm. Represents the network weight.

7. The pallet residue detection method according to claim 6, characterized in that: The step D includes sub-step D2: details of the training process for obtaining the light reflection removal image: the input image size in the network structure is 512*512*3, the UGG19 network structure is used to extract features of the light reflection image, the pallet area image is input into VGG-19, the image features of conv1_2, conv2_2, conv3_2, conv4_2 and conv5_2 are extracted, the five features and the input pallet area image are spliced ​​together to obtain image features with 1475 channels, a convolution with a convolution kernel of 1*1 is used to obtain image features with 64 channels, 8 layers of convolution with a convolution kernel of 3*3 are used, the number of channels in the intermediate feature layer is kept at 64, a convolution with a convolution kernel of 1*1 is used to obtain a convolution feature with a channel number of 3, which is recorded as the light reflection removal image, and during the training process, the loss function for ， in, Represents the image with light reflection removed The features extracted by the convolutional layer in VGG19, l represents five layers of convolution, namely conv1_2, conv2_2, conv3_2, conv4_2 and conv5_2. Represents a matte reflective image The convolution features extracted in conv1_2, conv2_2, conv3_2, conv4_2 and conv5_2 respectively, Represents the norm operation in mathematics. VGG19 obtains the network structure of the image with light reflection removed. is the weight coefficient.

8. The pallet residue detection method according to claim 1, characterized in that: The step E includes sub-step E1: when inputting the pallet area image into the target detection network structure, according to the placement and shape of the pallet in the actual scene, the input size of the pallet area image is selected as 640*640, and the target detection algorithm is used to perform target detection on the leftover items in the pallet to obtain whether there are leftover items in the pallet area image.

Images