Method for detecting objects using machine vision

Abstract

FIELD: computer technology.SUBSTANCE: method of object detection using computer vision that uses a depth camera to capture a point cloud of data, which is then converted into a depth map. After that, a 2D image is constructed from the depth map, image pre-processing filters are used, a search for a region of interest (ROI) is performed using machine vision functions. The upper layer of the working range is selected by depth and autoscaling of the upper layer is performed, then a search for the object contour is performed using machine vision functions. The upper layer of the working range is selected by depth. The Sobel operator is applied to the depth map array to search for the contours of objects of the upper layer of the working range by depth at least ten times with different pixel values and an array of images with object boundaries is obtained. A low-pass filter of low or high order is applied to create higher-quality boundaries. Binarization is applied to the boundary image. A pre-prepared template of the location of objects is superimposed on each obtained image and the template value is multiplied by the Sobel operator value, while the obtained value is presented in the form of points, based on which the data processing system makes a decision on the correspondence of the template to the real image.EFFECT: increased accuracy of object detection.5 cl, 14 dwg

Description

[0001] Field of technology to which the invention relates

[0002] The claimed technical solution relates to the field of industrial automation and robotics, and is designed to search for any objects on any pallet using machine vision.

[0003] Technology Level

[0004] Automated palletizing using machine vision systems enables robots and autonomous forklifts to operate in dynamic environments where objects have different shapes or are uneven.

[0005] For example, a method of robotic stacking of objects on pallets is known from the SICK company (Internet resource

[0006] https: / / www.sick.com / pl / ru / catalog / otrasli / logistika / peoizvodstvennaya-logistika / zagruzka-i-vygruzka-materiala / robotizirovannaya-palletizaciya-i-depalletizaciya-s-kamerami-mashinnogo-zreniya / c / p662440. accessed 23.03.2026, [1]), in which, using 3D imaging technology, a 3D machine vision camera recognizes in real time whether there is space for newly incoming objects (containers) on the pallet and where exactly. At the same time, a sensor determines whether the objects on the pallet are correctly aligned. In the second stage, the camera determines the shape and size of the objects on the infeed conveyor to determine the optimal location for their placement. This allows for the optimization of the palletizing process and, at the same time, an increase in throughput without additional transportation costs. In addition to palletizing, the 3D vision camera is also used for depalletizing objects.

[0007] The disadvantage of the analogue [1] is that the objects on the pallet are recognized by a 3D camera with a neural network add-on (Internet resource https: / / www.sick.com / ch / fr / sicks-intelligent-makes-deep-learning-accessible- / w / press-2021-intelligent-inspection#:~:text=Waldkirch%2C%20July%202021%20%2D%20SICK%20has,of%20SICK%20InspectorP6xx%20vision%20sensors. accessed 23.03.2026 [2]). In this case, the requirements for 3D cameras increase, since powerful graphics processors or specialized chips are required. In this case, the neural network is dependent on the training sample. If boxes of a new color arrive at the warehouse, not included in the training, the neural network may fail to recognize them. Neural networks have difficulty finding boundaries for soft objects like bags and sacks, and the system often mistakens the corner of a torn box for the edge of a pallet. Furthermore, the neural network's errors are difficult to debug using logic.

[0008] Methods for segmenting volumetric images using watershed methods are also widely known in the prior art (Internet resource https: / / www.mdpi.com / 2313-433X / 8 / 5 / 127. accessed 23.03.2026 [3]). The general concept of the mentioned method [3] is that a grayscale image is considered as a topographic relief flooded from the source basins.

[0009] The disadvantage of this method [3] is oversegmentation. If the box has bright tape, a large logo, or is dirty, the algorithm mistakes the dark print line for a deep gap. The robot attempts to grasp the "emptiness" or edge of the box and drops it. The watershed method is very sensitive to geometry. Glare from the stretch film creates very bright spots with sharp dark edges.

[0010] The prototype of the proposed invention is a method for detecting objects (US 2025005889 A1, IPC B25J 9 / 16; G06V 10 / 26; G06V 10 / 44; G06V 10 / 54; G06V 10 / 98; G06V 20 / 64; published 02.01.2025, [4]). As in the claimed method, the 3D camera generates an image of the environment in which the camera is located. The environment is, for example, a warehouse, production facility, retail space or other premises. In such cases, the information in the image represents objects located in a given area, such as boxes, crates, containers, boxes, pallets or other containers. In the contour construction operation, the computing system identifies the contour of a stack of objects based on the information obtained from the image captured by the camera. The edge of the image is detected using the Sobel operator.The contour depicted in the image is determined by the computing system by converting the image of a stack of objects into a binary image, pre-processing the binary image to enhance edge features, and extracting the contour. During the process of determining the contour depicted in the image, the computing system applies a filter to smooth the image. The resulting information is transmitted to the robot control system, which, in turn, performs trajectory planning and / or generates commands for the robot to interact with the system based on this information.

[0011] A drawback of the prototype [4] is that the method uses layer separators that are sensitive to object displacement and incomplete layer filling. Moreover, if boxes of different heights are stacked on the same level, even with a difference of 1-2 cm, the rigid gripper will either crush the taller ones or fail to reach the shorter ones. Machine vision is powerless to correct this mechanical limitation.

[0012] The technical problem to be solved is the need to improve the method of detecting objects using machine vision in order to increase efficiency:

[0013] - search for centers of objects with bags and boxes, which are the main containers in production and warehouses;

[0014] - search for soft objects such as packages and sacks;

[0015] - reducing the requirement for computing power compared to using neural networks.

[0016] Disclosure of invention

[0017] The technical result achieved by the claimed technical solution is an increase in the efficiency of the method for detecting objects using machine vision.

[0018] Based on this original observation, the present invention mainly aims to provide a method for detecting objects using machine vision, in which an array of data in the form of a point cloud is obtained from a depth camera, which is then converted into a depth map,

[0019] It is distinguished by the fact that

[0020] - a 2D image is constructed from the depth map;

[0021] - then use image pre-processing filters, and then search for the region of interest (ROI) using machine vision functions;

[0022] - then select the upper layer of the working range by depth and perform autoscaling of the upper layer;

[0023] - then apply the Sobel operator to the depth map array to search for the contours of the objects in the upper layer of the working range in depth at least ten times with different pixel values, and obtain an array of images with the boundaries of the objects;

[0024] - then apply a low-pass or high-order filter to create better edges;

[0025] - apply binarization to the border image;

[0026] - a pre-prepared template of the object arrangement is applied to each received image and the template value is multiplied by the Sobel operator value, and the resulting value is presented in the form of points, based on which the data processing system makes a decision on the correspondence of the template to the real image, and the best multiplication value in terms of points determines the probability of the configuration of the object arrangement;

[0027] - the best solution in coordinates by points is used to calculate the centers of objects for capture by the automatic capture device, while to correct the angle of capture of the object, a search for boundaries on each of the four sides is used;

[0028] - transmits the XYZ coordinates of the centers of objects and the angle A of rotation of the gripper around the Z axis in degrees to the gripper device.

[0029] The above-mentioned entity is a set of essential features of the declared technical solution that ensure the achievement of the declared technical result.

[0030] In special cases, it is permissible to implement the technical solution as follows.

[0031] The depth camera is preferably a 3D camera, a ToF camera, a stereo camera, or a lidar (LiDAR).

[0032] The pre-processing filters used in image processing are mainly a combination of median filter, low-pass filter, particle filter and edge filter.

[0033] If after applying the template to the image, the template does not match the location of the objects, the image is preferably returned to the stage of applying the Sobel operator.

[0034] Finding the contour of an object using machine vision functions preferably involves using Canny detector, or approximation, or circumscribed rectangle, or area and shape filtering.

[0035] The method for detecting objects using machine vision has been implemented by a specialist in practice and, when implemented, ensures the fulfillment of the stated purpose.

[0036] Brief description of drawings

[0037] Figure 1 shows a block diagram of the algorithm for recognizing and localizing objects; Fig. 2 - point cloud, 3D image from a depth camera; Fig. 3 - conversion to 2D image; Fig. 4 - filtering and searching for the zone of interest: pallets; Fig. 5 - the result of filtering and selecting the zone of interest; Fig. 6 - scaling the mask of the working range inside the zone of interest; Fig. 7 - searching for the boundaries of objects using the Sobel operator; Fig. 8 - searching for the boundaries of objects using the Sobel operator; Fig. 9 - the result of applying a low-pass filter; Fig. 10 - binarization; Fig. 11 - template matching; Fig. 12 - determining the centers of objects; Fig. 13 - applying the watershed method.

[0038] Implementation of the invention

[0039] The method for detecting objects (boxes, bags, packages) on a pallet using machine vision uses the following equipment:

[0040] - A depth camera (3D camera, stereo camera, ToF, LIDAR, etc.) positioned so that the object (pallet or other container) falls within its field of view;

[0041] - a computer or other calculator for implementing software;

[0042] - Automatic object gripping device or industrial object picking robot.

[0043] As shown in Fig. 1, the depth camera receives a data array in the form of a point cloud (Fig. 2), where each element has its own spatial coordinates. This array is converted into a depth map, which is an ordered projection of the point cloud onto a plane. A 2D image is then constructed from the depth map (Fig. 3) to improve the performance of subsequent operations.

[0044] Then the image is pre-processed (Fig. 4). To clean the 2D image obtained from the depth map from noise and glare from the stretch film on objects, a combination of image pre-processing filters is used:

[0045] - median filter, which selects the average value from neighboring pixels, while glare and “dead” pixels disappear, and the borders of the boxes remain sharp;

[0046] - low-pass filter (LPF) that passes only low frequencies (smooth changes in color / pitch) and blocks high frequencies (sharp points, noise), while the LPF smooths the surface of the boxes, making it perfectly flat;

[0047] - a particle filter that looks for isolated groups of pixels in an image that are too small to be a box, using morphological operations (e.g. Opening) whose algorithm erases everything that is smaller than a given size (e.g. smaller than 5x5 pixels);

[0048] - An edge filter that makes the entire image black, leaving only the lines where there is a sharp change in height white.

[0049] After this, the object's contour is searched for using machine vision functions (Fig. 5): Canny detector, approximation, circumscribed rectangle, filtering by area and shape, etc.

[0050] The Canny detector detects sharp brightness differences (borders) and transforms them into thin white lines on a black background. Approximation smooths out the jagged edge of the box, smoothing out these irregularities and transforming the curved line into a perfect rectangle. The circumscribed rectangle calculates the minimum rectangle that fits the detected contour. Filtering by area and shape performs standard verification functions. If the contour is too small (garbage) or too long (a gap between pallets), the system simply ignores it.

[0051] After this, the upper layer of the working range is selected by depth (Fig. 6). The working range mask is scaled within the selected region of interest (ROI), i.e., autoscaling of the upper layer is performed.

[0052] Then, the Sobel operator is applied to the depth map array to find the contours of objects in the upper layer of the working range by depth at least ten times with different pixel values, resulting in an array of images with object boundaries (Figs. 7, 8). The main advantage of the Sobel operator over a neural network for the pallet problem is its speed and the lack of training required.

[0053] After this, a low-pass filter (LPF) of low or high order is applied to create higher-quality edges (Fig. 9). A low-pass filter (1st and 2nd order) removes only isolated outliers. Elevation differences remain virtually untouched. The corners of the boxes remain sharp. This is critical for a robot that needs to accurately grasp the edge. An example of such a filter is a Gaussian filter with a small kernel (3x3). A high-pass filter (4th order and higher) makes the surface of objects perfectly smooth, like a mirror. All minor irregularities in the cardboard, inscriptions, and small dents are eliminated. It is used for dusty warehouses where the air contains a lot of suspended matter, creating noise on the sensor. An example of such a filter is a 4th-order Butterworth filter.

[0054] Then, binarization is applied to the edge image (Fig. 10). This allows us to separate objects from the background, speed up image processing, accurately calculate the area and center, and remove visual artifacts such as drawings, tape, or dust, which appear as light gray ripples on the depth map.

[0055] A pre-prepared object layout template (Fig. 11) is superimposed on each acquired image, and the template value is multiplied by the Sobel operator. The resulting value is then presented as a score, based on which the data processing system decides whether the template matches the actual image. The highest score determines the probability of the object layout configuration.

[0056] The best-scoring coordinate solution is used to calculate the centers of objects for pickup by an automated system or industrial robot (Fig. 12). To adjust the pickup angle, a boundary search is performed on each of the four sides. If the template does not match the object's location, the image is returned to the Sobel operator application stage.

[0057] After this, the XYZ coordinates of the centers of the objects and the angle A of rotation of the gripper around the Z axis in degrees are transmitted to the gripper device.

[0058] As a comparative example, (Fig. 13) and (Fig. 14) show the watershed method and the calculation of the object center without using the watershed method.

[0059] This method was tested against the watershed method and demonstrated superior results for finding the centers of objects containing bags and boxes, which are the primary packaging materials in manufacturing and warehouses. This method was also tested against neural networks and demonstrated superior results for soft objects such as bags and sacks, whose boundaries are quite difficult to locate even using neural networks and a large training dataset. Furthermore, this method requires significantly less computing power than neural networks.

[0060] This method has been tested on a real production task and has shown an error probability of 1 incorrect result per 10,000 objects, i.e. 0.01%.

[0061] Industrial applicability

[0062] The method for detecting objects using machine vision is implemented using industrially available devices and materials, can be implemented by a specialist in practice and will find wide application in logistics, warehouse automation and production for systems of: automatic display (palletizing) of goods, automatic unloading of pallets (depalletizing), removal (picking) of objects from pallets or from similar containers (containers, carts), checking the completeness and correctness of pallet stacking.

Claims

1. A method for detecting objects using machine vision, in which a depth camera receives an array of data in the form of a point cloud, which is then converted into a depth map, characterized in that - a 2D image is constructed from the depth map; - then use image pre-processing filters, after which they search for a region of interest (ROI) using machine vision functions; - then the upper layer of the working range is selected by depth and autoscaling of the upper layer is performed; - then apply the Sobel operator to the depth map array to search for the contours of the objects of the upper layer of the working range in depth at least ten times with different pixel values ​​and obtain an array of images with the boundaries of the objects; - then a low-pass or high-order filter is applied to create better boundaries; - apply binarization to the image of boundaries; - a pre-prepared template of the arrangement of objects is applied to each received image and the value of the template is multiplied by the value of the Sobel operator, and the resulting value is presented in the form of points, based on which the data processing system makes a decision on the correspondence of the template to the real image, and the best multiplication value in terms of points determines the probability of the configuration of the layout of objects; - the best solution in coordinates based on points is used to calculate the centers of objects for capture by the automatic capture device, while to correct the angle of capture of the object, a search for boundaries on each of the four sides is used; - transmit the XYZ coordinates of the centers of the objects and the angle A of rotation of the gripper around the Z axis in degrees to the gripper device.

2. The method according to paragraph 1, characterized in that a 3D camera, a ToF camera, a stereo camera, or a lidar is used as a depth camera.

3. The method according to claim 1, characterized in that a combination of a median filter, a low-pass filter, a particle filter, and an edge filter are used as pre-processing filters for images.

4. The method according to claim 1, characterized in that the search for the contour of an object using machine vision functions includes the use of a Canny detector, and / or approximation, and / or a circumscribed rectangle, and / or filtering by area and shape.

5. The method according to paragraph 1, characterized in that if, after superimposing the template on the image, the template does not correspond to the location of the objects, then the image is returned to the stage of applying the Sobel operator.

Images