A visual-based picking and placing goods on a shelf behavior detection method

Abstract

The present application relates to computer vision technical field, especially for a kind of based on vision's take and put shelf commodity behavior detection method, including the following steps: monitoring camera is from oblique downward angle from side video shooting shelf, this angle is most conducive to human and shelf interaction analysis, image is sent to human detection module after decoding, since the process that personnel takes and puts commodity can be short, to reduce the missed detection rate, the present scheme uses higher sampling rate, and sampling rate can be greater than 3 frames / second under the condition of computing resource support.The present application complete system is composed of monitoring camera, human detection module, human tracking module and human pose estimation module, monitoring camera is responsible for the real-time image sequence in supermarket acquisition, human detection module is responsible for locating human in image, human tracking module is responsible for distinguishing different pedestrians and continuously tracking these pedestrians, so that the behavior of each pedestrian can be analyzed.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a vision-based method for detecting the picking and placing of goods on shelves. Background Technology

[0002] Computer vision is the field that studies how to enable computers to acquire, analyze and understand visual information from images or videos. Vision-based methods for detecting the picking and placing of goods on shelves analyze the placement and movement of goods on shelves and use computer vision algorithms to detect and identify picking and placing behaviors.

[0003] Current methods for detecting the handling of goods on shelves primarily rely on surveillance cameras. However, these cameras can only capture simple images of customers and, while helpful in identifying suspects during thefts, have drawbacks. Firstly, they may misinterpret normal behavior as handling or fail to detect actual handling. Secondly, human postures and behaviors are diverse and complex, making it difficult for surveillance devices to accurately deduce specific action details from static images. To address these issues, a vision-based method for detecting the handling of goods on shelves urgently needs development. Summary of the Invention

[0004] The purpose of this invention is to provide a vision-based method for detecting the picking and placing of goods on shelves, which has the advantage of being able to determine the picking and placing behavior of pedestrians and solves the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a vision-based method for detecting the picking and placing of goods on shelves, comprising the following steps:

[0006] S1: The surveillance camera captures video of the shelf from a downward angle on the side. This angle is most conducive to the analysis of human interaction with the shelf. After the image is decoded, it is sent to the human detection module. Since the process of people picking up and putting down goods may be very short, in order to reduce the missed detection rate, this solution adopts a high sampling rate. With the support of computing resources, the sampling rate can be greater than 3 frames / second.

[0007] S2: After receiving the real-time image, the human detector detects all human areas in the image, filters out human bodies with low scores, and removes human bodies obscured by shelves based on the aspect ratio, height, and bottom border position of the human detection box.

[0008] Furthermore, in step S2, the human detection algorithm can employ publicly available algorithms such as YOLOX, Center and ScalePrediction.

[0009] S3: For the selected human body regions, perform human body tracking. First, extract features from the human body region images. Feature extraction can use a publicly available deep learning-based ReID model. Then, create multiple new trajectories with each newly detected human body in each frame. Then, use a clustering algorithm to merge these new trajectories with historical trajectories and remove trajectories that have not been updated for a long time, thereby obtaining all pedestrian tracking trajectories in the current frame.

[0010] S4: Calculate the distance between the human body and the bottom edge of each shelf, and draw a perpendicular line from the midpoint of the human body's bottom edge to the bottom edge of the shelf. If the x and y coordinates of the intersection point are both within the x and y coordinate range of the bottom edge endpoint of the shelf, then the human body is within the range of that shelf. Record the distance between each human body and each shelf.

[0011] S5: Preprocess all human body area images within the shelf area, including normalization and scaling to a fixed size.

[0012] Furthermore, in step S5, in order to ensure that the entire human body is within the detection frame, the width and height of the detection frame can be increased by 1 / 10 each.

[0013] S6: A 3D human pose estimation model is used to generate human pose parameters. The input of the model is a preprocessed 2D human image, and the output is a 3D global human rotation matrix, a human joint rotation matrix, human morphology parameters, and camera parameters. The human pose estimation model can use publicly available 3D reconstruction algorithms such as HMR. Then, the SMPL model is used to generate human skin and joint positions. The input of this model is the global human rotation matrix, the human joint rotation matrix, and human morphology parameters. The output is normalized 3D joint coordinates and normalized 3D skin coordinates. Finally, the human orientation is obtained by multiplying the global human rotation matrix by the 3D normal vector when the human is stationary.

[0014] Furthermore, in step S6, the coordinate system of the human body normal vector, joint points, and skin is as follows: the x-axis is positive to the right, the y-axis is positive downwards, and the z-axis is positive inwards.

[0015] S7: Find the shelf corresponding to the pedestrian, select the two shelves closest to the person. When the normal vectors of the two shelves are similar, select the shelf with the smallest distance from the person. When the normal vectors are opposite, select the shelf with the largest angle with the person's normal vector.

[0016] S8: Determine if the human body has made a picking or putting action. If the human body makes a specific posture, it can be considered as a picking or putting action.

[0017] S9: If a pick-up or put-down occurs once in each trajectory, and then occurs a second time within n frames, it is considered a complete pick-up or put-down. Each trajectory has a buffer to store pick-up or put-down flags. The buffer is cleared after a complete pick-up or put-down is detected, and the buffer will start storing pick-up or put-down flags again when another pick-up or put-down occurs.

[0018] Preferably, in step S2, the human detection algorithm can be a publicly available algorithm such as YOLOX, Center and ScalePrediction.

[0019] Preferably, in step S5, in order to ensure that the entire human body is within the detection frame, the width and height of the detection frame can be increased by 1 / 10 each.

[0020] Preferably, in step S6, the coordinate system of the human body normal vector, joint points and skin is as follows: the x-axis is positive to the right, the y-axis is positive downwards, and the z-axis is positive inwards.

[0021] In this invention: In step S8, the human posture for determining whether the human body is performing a picking-up or putting-down action is as follows:

[0022] S8.1: Use the vector direction from the neck to the knee joint to filter out 3D human figures with obvious pose estimation errors.

[0023] S8.2: When the distance between the human body and the corresponding shelf is less than the width of the human body, and the angle between the normal vector of the human body and the shelf exceeds the threshold, the angle between the arm and the shelf is calculated using 3D joints.

[0024] S8.3: When the angle between the arm and the shelf exceeds the threshold, calculate the camera translation parameters and project the 3D joints using the camera intrinsic and extrinsic parameters to obtain the 2D joints in the image coordinate system.

[0025] S8.4: Use 2D joints to determine whether the wrist position passes through the bottom edge of the shelf. Draw a perpendicular line from the wrist to the bottom edge, and compare the x and y coordinates of the intersection of the perpendicular line and the bottom edge with the x and y coordinates of the wrist. Discuss different cases according to the orientation of the shelf.

[0026] Preferably, in step S8.2, the 2D joint points at this time are points on the preprocessed small image, and the original image needs to be restored before proceeding to the next step.

[0027] Preferably, in step S8.4, if the wrist passes through the bottom edge of the shelf, it is necessary to determine whether the wrist position is within the shelf area corresponding to the human body. If the wrist position is within the shelf area, it can be considered as one take-up and put-down.

[0028] Beneficial effects: The technical solution of this application has the following technical effects:

[0029] The entire system consists of surveillance cameras, a human detection module, a human tracking module, and a human pose estimation module. The surveillance cameras are responsible for acquiring real-time image sequences within the supermarket. The human detection module is responsible for locating human bodies in the images. The human tracking module is responsible for distinguishing different pedestrians and continuously tracking these pedestrians, thereby analyzing the behavior of each pedestrian. The human pose estimation module is responsible for estimating human pose and determining whether there is any picking or putting down behavior.

[0030] By analyzing the video streams from surveillance cameras, the timing and location of customers picking up and putting down goods can be automatically determined. This can assist in screening surveillance videos if theft occurs in the supermarket, or automatically identify which types of goods customers are most interested in.

[0031] It should be understood that all combinations of the foregoing concepts and the additional concepts described in more detail below can be considered part of the inventive subject matter of this disclosure, provided that such concepts do not contradict each other. Attached Figure Description

[0032] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0033] Figure 1 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. To better understand the technical content of the present invention, specific embodiments are provided and described in conjunction with the accompanying drawings. Various aspects of the present invention are described in this disclosure with reference to the accompanying drawings, which illustrate numerous illustrative embodiments. It should be understood that the various concepts and embodiments described above, as well as those described in more detail below, can be implemented in any of many ways. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0035] As attached Figure 1 As shown: This embodiment provides a vision-based method for detecting the picking and placing of goods on shelves, including the following steps:

[0036] S1: The surveillance camera captures video of the shelf from a downward angle on the side. This angle is most conducive to the analysis of human interaction with the shelf. After the image is decoded, it is sent to the human detection module. Since the process of people picking up and putting down goods may be very short, in order to reduce the missed detection rate, this solution adopts a high sampling rate. With the support of computing resources, the sampling rate can be greater than 3 frames / second.

[0037] S2: After receiving the real-time image, the human detector detects all human areas in the image, filters out human bodies with low scores, and removes human bodies obscured by shelves based on the aspect ratio, height, and bottom border position of the human detection box.

[0038] S3: For the selected human body regions, perform human body tracking. First, extract features from the human body region images. Feature extraction can use a publicly available deep learning-based ReID model. Then, create multiple new trajectories with each newly detected human body in each frame. Then, use a clustering algorithm to merge these new trajectories with historical trajectories and remove trajectories that have not been updated for a long time, thereby obtaining all pedestrian tracking trajectories in the current frame.

[0039] S4: Calculate the distance between the human body and the bottom edge of each shelf, and draw a perpendicular line from the midpoint of the human body's bottom edge to the bottom edge of the shelf. If the x and y coordinates of the intersection point are both within the x and y coordinate range of the bottom edge endpoint of the shelf, then the human body is within the range of that shelf. Record the distance between each human body and each shelf.

[0040] S5: Preprocess all human body area images within the shelf area, including normalization and scaling to a fixed size.

[0041] S6: A 3D human pose estimation model is used to generate human pose parameters. The input of the model is a preprocessed 2D human image, and the output is a 3D global human rotation matrix, a human joint rotation matrix, human morphology parameters, and camera parameters. The human pose estimation model can use publicly available 3D reconstruction algorithms such as HMR. Then, the SMPL model is used to generate human skin and joint positions. The input of this model is the global human rotation matrix, the human joint rotation matrix, and human morphology parameters. The output is normalized 3D joint coordinates and normalized 3D skin coordinates. Finally, the human orientation is obtained by multiplying the global human rotation matrix by the 3D normal vector when the human is stationary.

[0042] S7: Find the shelf corresponding to the pedestrian, select the two shelves closest to the person. When the normal vectors of the two shelves are similar, select the shelf with the smallest distance from the person. When the normal vectors are opposite, select the shelf with the largest angle with the person's normal vector.

[0043] S8: Determine if the human body has made a picking or putting action. If the human body makes a specific posture, it can be considered as a picking or putting action.

[0044] S9: If a pick-up or put-down occurs once in each trajectory, and then occurs a second time within n frames, it is considered a complete pick-up or put-down. Each trajectory has a buffer to store pick-up or put-down flags. The buffer is cleared after a complete pick-up or put-down is detected, and the buffer will start storing pick-up or put-down flags again when another pick-up or put-down occurs.

[0045] In this invention: In step S8, the human posture for determining whether the human body is performing a picking or putting-down action is as follows:

[0046] S8.1: Use the vector direction from the neck to the knee joint to filter out 3D human figures with obvious pose estimation errors.

[0047] S8.2: When the distance between the human body and the corresponding shelf is less than the width of the human body, and the angle between the normal vector of the human body and the shelf exceeds the threshold, the angle between the arm and the shelf is calculated using 3D joints.

[0048] Furthermore, in step S8.2, the 2D joints at this point are points on the preprocessed small image, and the original image needs to be restored before proceeding to the next step.

[0049] S8.3: When the angle between the arm and the shelf exceeds the threshold, calculate the camera translation parameters and project the 3D joints using the camera intrinsic and extrinsic parameters to obtain the 2D joints in the image coordinate system.

[0050] S8.4: Use 2D joints to determine whether the wrist position passes through the bottom edge of the shelf. Draw a perpendicular line from the wrist to the bottom edge, and compare the x and y coordinates of the intersection of the perpendicular line and the bottom edge with the x and y coordinates of the wrist. Discuss different cases according to the orientation of the shelf.

[0051] Furthermore, in step S8.4, if the wrist passes through the bottom edge of the shelf, it is necessary to determine whether the wrist position is within the shelf area corresponding to the human body. If the wrist position is within the shelf area, it can be considered as one take-up and put-down.

[0052] It should be noted that in this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0053] While the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A vision-based method for detecting the picking and placing of goods on a shelf, characterized in that, Includes the following steps: S1: The surveillance camera captures video of the shelf from the side at a downward angle. The image is decoded and sent to the human body detector, which uses a sampling rate of 3 frames per second. S2: After receiving the real-time image, the human body detector detects all human body regions in the image, filters out human body regions with confidence scores below the set threshold, and filters out human bodies obscured by shelves based on the aspect ratio, height, and bottom border position of the human body detection box. S3: For the selected human body regions, perform human body tracking. First, extract features from the human body region images using a publicly available deep learning-based ReID model. Then, create multiple new trajectories for each newly detected human body in each frame. Next, use a clustering algorithm to merge these new trajectories with historical trajectories and remove trajectories that have not been updated for a long time, thereby obtaining all pedestrian tracking trajectories for the current frame. S4: Calculate the distance between the human body and the bottom edge of each shelf, and draw a perpendicular line from the midpoint of the human body's bottom edge to the bottom edge of the shelf. If the x and y coordinates of the intersection point are both within the x and y coordinate range of the bottom edge endpoint of the shelf, then the human body is within the range of that shelf. Record the distance between each human body and each shelf. S5: Preprocess all human body area images within the shelf area, including normalization and scaling to a fixed size; S6: A 3D human pose estimation model is used to generate human pose parameters. The input of the model is a preprocessed 2D human image, and the output is a 3D global human rotation matrix, a human joint rotation matrix, human morphology parameters, and camera parameters. The human pose estimation uses the HMR 3D human reconstruction algorithm. Then, the SMPL model is used to generate human skin and joint positions. The input of this model is the global human rotation matrix, the human joint rotation matrix, and human morphology parameters. The output is normalized 3D joint coordinates and normalized 3D skin coordinates. Finally, the human orientation is obtained by multiplying the global human rotation matrix by the 3D normal vector when the human is stationary. S7: Find the shelf corresponding to the pedestrian, select the two shelves closest to the human body. When the normal vectors of the two shelves are similar, select the shelf with the smallest distance from the human body. When the normal vectors are opposite, select the shelf with the largest angle with the human body's normal vector. S8: Determine if the human body has made a picking or putting action. If the human body makes a specific posture, it is considered as a picking or putting action. S9: If a pick-up or put-down occurs once in each trajectory, and then occurs a second time within n frames, it is considered a complete pick-up or put-down. Each trajectory has a buffer to store pick-up or put-down flags. The buffer is cleared after a complete pick-up or put-down is detected, and the buffer will start storing pick-up or put-down flags again when another pick-up or put-down occurs. 2.The vision-based picking and shelving behavior detection method of claim 1, wherein: In step S2, the human body detection uses the open-source YOLOX algorithm. 3.The vision-based picking and shelving behavior detection method of claim 1, wherein: In step S5, to ensure that the entire human body is within the detection frame, the width and height of the detection frame are each increased by 1 / 10. 4.The vision-based picking and shelving behavior detection method of claim 1, wherein: In step S6, the coordinate system of the human body normal vector, joint points and skin is as follows: the x-axis is positive to the right, the y-axis is positive downwards, and the z-axis is positive inwards.

5. The visual-based picking and shelving behavior detection method of claim 1, wherein: In step S8, the human posture for determining whether the human body is performing a picking or putting-down action is as follows: S8.1: Use the vector direction from the neck to the knee joint to filter out 3D human figures with obvious pose estimation errors; S8.2: When the distance between the human body and the corresponding shelf is less than the width of the human body, and the angle between the human body and the shelf normal vector exceeds the threshold, the angle between the arm and the shelf is calculated using 3D joints; S8.3: When the angle between the arm and the shelf exceeds the threshold, calculate the camera translation parameters and project the 3D joints using the camera intrinsic and extrinsic parameters to obtain the 2D joints in the image coordinate system; S8.4: Use 2D joints to determine whether the wrist position passes through the bottom edge of the shelf. Draw a perpendicular line from the wrist to the bottom edge, and compare the x and y coordinates of the intersection of the perpendicular line and the bottom edge with the x and y coordinates of the wrist. Discuss different cases according to the orientation of the shelf.

6. The vision-based method for detecting the picking and placing of goods on shelves according to claim 5, characterized in that: In step S8.2, the 2d joints at this point are points on the preprocessed small image, and the original image needs to be restored before proceeding to the next step.

7. The visual-based picking and shelving behavior detection method of claim 5, wherein: In step S8.4, if the wrist passes through the bottom edge of the shelf, it is necessary to determine whether the wrist position is within the shelf area corresponding to the human body. If the wrist position is within the shelf area, it can be considered as one take-up and put-down.

Images