Overlapped target visual deduplication method, device and equipment
By constructing and transforming the safe area polygon, the problem of accuracy in judging the overlap of irregularly shaped materials is solved, achieving high-precision recognition and safe grasping of overlapping targets, and reducing the collision risk of robot grasping.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN ZMOTION TECH CO LTD
- Filing Date
- 2026-01-16
- Publication Date
- 2026-05-22
AI Technical Summary
In existing technologies, it is difficult to accurately determine the overlap of the true contours of irregularly shaped materials, leading to false overlaps being mistakenly rejected and true overlaps being missed. Furthermore, the lack of safety space for grasping operations increases the risk of collisions during robot grasping.
By acquiring scene images of the target to be captured, the matching template and pose information of the target are identified based on template matching. A reference polygon that encloses the target boundary is constructed, and the polygon is expanded or shrunken based on a preset safety distance to generate a safety area polygon. The pose information is then used to transform the polygon to the same image coordinate system to determine whether there is an overlapping area between the targets.
It effectively reduces false overlaps and false overlaps, improves the accuracy and reliability of overlap judgment, prevents physical collisions during robot grasping, and enhances the safety and practicality of the system.
Smart Images
Figure CN121544895B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of automation control technology, and in particular to a method, apparatus and equipment for visual deduplication of overlapping targets. Background Technology
[0002] In industrial automation production, such as feeding materials onto vibrating discs, sorting logistics, and picking up scattered parts, machine vision systems are commonly used to identify disordered stacked materials and guide robots to perform picking tasks.
[0003] Current mainstream technologies rely on template matching algorithms to locate targets, extract their smallest rotational bounding rectangle, and determine material overlap by judging whether the rectangles overlap. Non-maximum suppression algorithms are then used to retain only targets with the highest confidence. However, the bounding rectangles are difficult to accurately describe the actual contours of irregularly shaped materials, leading to "false overlaps" where a large number of materials are mistakenly rejected due to rectangle intersections even when they are not in contact, and the risk of missed grasping collisions where materials are interlocked but the rectangles do not intersect. Secondly, this approach relies solely on two-dimensional image pixels for judgment, completely ignoring the physical dimensions of the robot's end effector. It fails to reserve a collision-free safety space at the geometric level for grasping operations, resulting in a high risk of collisions during the actual grasping process.
[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention
[0005] The main purpose of this application is to provide a visual deduplication method, apparatus and equipment for overlapping targets, which aims to solve the technical problem that it is difficult to accurately determine the overlap of the true contours of irregularly shaped materials in the prior art.
[0006] To achieve the above objectives, this application proposes a visual deduplication method for overlapping targets, the method comprising:
[0007] Acquire scene images of the target to be captured, and identify the matching template and pose information of each target in the scene image based on template matching;
[0008] Construct a reference polygon for the boundary of the envelope target based on the matching template;
[0009] Based on a preset safety distance, the baseline polygon is expanded or shrunk, and a safety area polygon is generated by combining it with the matching template.
[0010] The associated safe zone polygons are transformed into the same image coordinate system using the pose information of each target;
[0011] In the image coordinate system, it is determined whether any two targets have overlapping regions, so as to perform de-overlap processing.
[0012] In one embodiment, the step of identifying the matching template and pose information of each target in the scene image based on template matching includes:
[0013] Determine the background complexity of the scene image;
[0014] If the background complexity is higher than a preset threshold, the scene image is subjected to high-speed filtering to obtain a processed scene image.
[0015] The scene image is matched with templates in a pre-built template library to obtain the matching template for each target in the scene image and the pose information of each target in the scene image.
[0016] In one embodiment, the step of constructing a reference polygon for the envelope target boundary based on the matching template includes:
[0017] Obtain the target feature point set associated with the matching template;
[0018] Perform concave hull calculation on the target feature point set to generate an initial reference polygon vertex sequence;
[0019] The initial reference polygon vertex sequence is simplified by polygon approximation, and the number of vertices is reduced within a preset fitting error range to generate the final reference polygon vertex sequence.
[0020] In one embodiment, the step of constructing a reference polygon for the envelope target boundary based on the matching template further includes:
[0021] Based on the matching template, obtain and display the associated standard target image;
[0022] In response to the user's continuous plotting operation on the boundary of the standard target image, the coordinates of the boundary points are collected sequentially.
[0023] The coordinates of the boundary points are connected in the order of acquisition to form a vertex sequence of the reference polygon.
[0024] In one embodiment, the step of expanding or shrinking the reference polygon based on a preset safety distance and generating a safety region polygon using the matching template includes:
[0025] Obtain the vertex sequence of the reference polygon and a preset safety distance, wherein a positive safety distance indicates outward expansion and a negative safety distance indicates inward contraction;
[0026] The bias function of the computational geometry library is called to perform bias calculation on the reference polygon vertex sequence with the safety distance as the bias amount, so as to obtain the biased reference polygon vertex sequence.
[0027] The offset reference polygon vertex sequence is connected sequentially to generate a safe region polygon.
[0028] In one embodiment, the step of expanding or shrinking the reference polygon based on a preset safety distance and generating a safety region polygon using the matching template includes:
[0029] Obtain the vertex sequence of the reference polygon and the preset safety distance;
[0030] Construct a polygon resembling a circular disk with the aforementioned safety distance as its radius;
[0031] Calculate the Minkowski sum of the vertex set corresponding to the vertex sequence of the baseline polygon and the polygon to obtain the expanded vertex set;
[0032] The boundary contours of the expanded vertex set are extracted as the base polygon vertex sequence to generate the safe region polygon.
[0033] In one embodiment, the step of expanding or shrinking the reference polygon based on a preset safety distance and generating a safety region polygon using the matching template includes:
[0034] Obtain the vertex sequence of the reference polygon and the preset safety distance;
[0035] Traverse each edge of the reference polygon and calculate the unit normal vector for each edge;
[0036] For each vertex of the reference polygon vertex sequence, the composite movement direction vector of the vertex is calculated based on the unit normal vectors of the two adjacent sides of the vertex;
[0037] Each vertex is translated by the safe distance along the corresponding composite movement direction vector to obtain a new vertex;
[0038] Connect all new vertices in vertex order to obtain the vertex sequence of the base polygon, and generate the safe region polygon.
[0039] In one embodiment, the step of transforming the associated safe area polygon to the same image coordinate system using the pose information of each target includes:
[0040] For each target, the corresponding safe area polygon and pose information are obtained, wherein the pose information includes translation parameters and rotation parameters;
[0041] Based on the translation and rotation parameters, construct a rigid body transformation matrix from the template coordinate system to the image coordinate system;
[0042] The coordinates of each vertex in the safe area polygon are transformed by the rigid body transformation matrix to obtain new vertex coordinates in the image coordinate system;
[0043] Connect all the new vertex coordinates in sequence to generate a safe region polygon located in the same image coordinate system.
[0044] Furthermore, to achieve the above objectives, this application also proposes an overlapping target visual deduplication device, which includes:
[0045] The image recognition module is used to acquire scene images of the target to be captured, and to identify the matching template and pose information of each target in the scene image based on template matching.
[0046] The geometric modeling module is used to construct a reference polygon that encloses the boundary of the target based on the matching template;
[0047] The image processing module is used to expand or shrink the reference polygon based on a preset safety distance, and generate a safety region polygon in combination with the matching template.
[0048] A coordinate transformation module is used to transform the associated safe area polygons to the same image coordinate system using the pose information of each target;
[0049] The overlap determination module is used to determine whether any two targets have overlapping regions in the image coordinate system, so as to perform de-overlap processing.
[0050] In addition, to achieve the above objectives, this application also proposes an overlapping target visual deduplication device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the overlapping target visual deduplication method described above.
[0051] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the visual deduplication method for overlapping targets as described above.
[0052] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the visual deduplication method for overlapping targets as described above.
[0053] One or more technical solutions proposed in this application have at least the following technical effects:
[0054] This application acquires scene images of the target to be grasped and identifies the matching template and pose information of each target in the scene image based on template matching. A reference polygon enclosing the target boundary is constructed according to the matching template. The reference polygon is expanded or shrunk based on a preset safety distance, and a safe area polygon is generated by combining it with the matching template. The associated safe area polygons are transformed to the same image coordinate system using the pose information of each target. In the image coordinate system, it is determined whether any two targets have overlapping areas, and then de-overlap processing is performed. This application effectively reduces the false rejection of "false overlap" caused by irregular shapes and the "missed detection" of true overlap, significantly improving the accuracy and reliability of overlap judgment. By introducing a quantized safety buffer, physical collisions during robot grasping are fundamentally prevented, enhancing the safety of the entire operating system. Simultaneously, by adapting to diverse material shapes and grasping tool requirements, the overall practicality and robustness of the system are improved. Attached Figure Description
[0055] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0056] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0057] Figure 1 A flowchart illustrating an embodiment of the visual deduplication method for overlapping targets in this application;
[0058] Figure 2 This is a flowchart illustrating Embodiment 2 of the visual deduplication method for overlapping targets in this application.
[0059] Figure 3 This is a flowchart illustrating Embodiment 3 of the visual deduplication method for overlapping targets in this application.
[0060] Figure 4 This is a flowchart illustrating Embodiment 4 of the visual deduplication method for overlapping targets in this application.
[0061] Figure 5 This is a flowchart illustrating Embodiment 5 of the visual deduplication method for overlapping targets in this application.
[0062] Figure 6 This is a flowchart illustrating Embodiment Six of the visual deduplication method for overlapping targets in this application.
[0063] Figure 7This is a schematic diagram of the module structure of the visual deduplication device for overlapping targets according to an embodiment of this application;
[0064] Figure 8 This is a schematic diagram of the device structure of the hardware operating environment involved in the visual deduplication method for overlapping targets in the embodiments of this application.
[0065] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0066] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.
[0067] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.
[0068] Due to the technical problem that existing technologies make it difficult to accurately determine the overlap of irregularly shaped materials based on their true outlines.
[0069] This application provides a solution that involves acquiring scene images of the target to be captured, and identifying the matching template and pose information of each target in the scene image based on template matching; constructing a reference polygon that encloses the target boundary according to the matching template; expanding or shrinking the reference polygon based on a preset safety distance, and generating a safety area polygon by combining it with the matching template; transforming the associated safety area polygons to the same image coordinate system using the pose information of each target; and determining whether any two targets have overlapping areas in the image coordinate system for de-overlap processing.
[0070] Based on this, embodiments of this application provide a visual deduplication method for overlapping targets, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the visual deduplication method for overlapping targets in this application.
[0071] In this embodiment, the overlapping target visual deduplication method includes steps S10 to S50:
[0072] Step S10: Acquire scene images of the target to be captured, and identify the matching template and pose information of each target in the scene image based on template matching;
[0073] It should be noted that, in this embodiment, the target to be grasped refers to materials or workpieces that are stacked disorderly in an industrial setting and need to be grasped by a robot. The scene image refers to a digital image obtained by capturing the entire working area containing the target to be grasped using a vision sensor (such as an industrial camera). Template matching is a machine vision technique that compares a predefined template image containing target features (such as contours and textures) with the scene image to find similar regions. The matching template refers to the predefined template that has the highest similarity to a certain region in the scene image during the template matching process and is thus determined to correspond to the same type of target. Pose information refers to the position and orientation of the target in the two-dimensional image plane, typically including at least the two-dimensional coordinates (x, y) of its center point and the rotation angle (θ) relative to the template.
[0074] This embodiment acquires images and preprocesses them to improve the signal-to-noise ratio. Then, using a pre-trained template library, it searches and locates targets in the entire image through algorithms such as correlation matching and feature matching. This allows the template type and pose of each instantiated target to be obtained, enabling rapid and batch detection of disordered targets in complex environments. The system also outputs a unique identifier for each target and its precise spatial location and orientation.
[0075] Step S20: Construct a reference polygon for the envelope target boundary based on the matching template;
[0076] It should be noted that, in this embodiment, the enveloping target boundary refers to the constructed geometry that closely fits or surrounds the actual outer contour of the target. The reference polygon refers to a closed polyline composed of a series of ordered vertices, the shape of which is designed to closely fit the target boundary, serving as the basic model for describing the target's geometry.
[0077] This embodiment generates a polygon surrounding the target outline based on template information, either automatically calculated by a preset algorithm or manually defined through human-computer interaction. This polygon can more realistically reflect the actual shape of the target than a traditional rotating rectangle.
[0078] Alternatively, the construction of the reference polygon can have different sources and methods. It can be pre-created offline and stored with a template, or it can be calculated and generated online in real time after the target is identified.
[0079] Step S30: Expand or shrink the reference polygon based on a preset safety distance, and generate a safety area polygon by combining it with the matching template;
[0080] It should be noted that in this embodiment, the preset safety distance refers to a length value pre-set or calculated based on the physical safety requirements of robot grasping, used to add a buffer area outside the geometric contour of the target. Expansion or contraction is a geometric transformation operation that involves translating each edge of a polygon outward or inward a specified distance along its normal direction, thereby generating a larger or smaller similar polygon. The safety area polygon refers to the new polygon obtained after expanding or contracting the base polygon, defining the boundary of the safety space that the robot's end effector needs to occupy or avoid when grasping the target.
[0081] This embodiment generates a safe operating area by superimposing an offset amount defined by a safety distance on the precise contour of a reference polygon and generating a safe operating area through geometric calculation. This can effectively avoid grasping collisions caused by visual positioning errors, robot repetitive positioning errors, and the size of the gripper itself.
[0082] In one possible implementation, the preset safety distance d is determined by the formula... ,in, This represents the maximum physical width of the gripper. The overall positioning error of the vision system For an additional safety margin, The pixel equivalent of the image (mm / pixel) is obtained by the system through camera calibration.
[0083] Optionally, the reference polygon can be expanded or shrunk based on a preset safety distance. This can be achieved by calling the bias function of the computational geometry library, or by using the Minkowski method, or by direct translation.
[0084] Step S40: Transform the associated safe area polygons to the same image coordinate system using the pose information of each target;
[0085] It should be noted that in this embodiment, the associated safe area polygon refers to the generated safe area polygon corresponding to a specific target and its template. Transformation here refers to coordinate transformation, that is, converting the polygon's vertex coordinate representation from one coordinate system to another through mathematical operations. The same image coordinate system refers to the inherent two-dimensional pixel coordinate system of the acquired scene image; all target polygons are transformed to this coordinate system. The purpose is to accurately place the safe area polygon of each target, defined offline and located in its own template coordinates, to the target's position and orientation in the actual scene image, achieving a mapping from model space to the real observation space.
[0086] In one specific implementation, assume an "L"-shaped part is identified in the scene image, with pose information of (x=100, y=200, θ=30°). The system already possesses a safe zone polygon associated with this part template (defined in the template coordinate system). The system first constructs a rigid body transformation matrix based on the pose information, which includes a composite transformation of a 30° rotation around the origin and a translation to (100, 200). Then, the system multiplies the coordinates of each vertex of the safe zone polygon with this transformation matrix to obtain the new coordinates of each vertex in the current scene image pixel coordinate system. All transformed vertices are connected in their original order to obtain a safe zone polygon instance of the "L"-shaped part in the current scene, correctly positioned and oriented. This process is repeated for all identified targets to unify the safe zone polygons of all targets into the scene image coordinate system.
[0087] Step S50: In the image coordinate system, determine whether any two targets have overlapping regions, so as to perform de-overlap processing.
[0088] It should be noted that in this embodiment, the image coordinate system refers to the scene image pixel coordinate system. "Determining whether any two targets have overlapping areas" means performing geometric calculations on the safe area polygons corresponding to any two targets in the image coordinate system to detect whether they have area intersections or interference. The de-overlap processing refers to executing corresponding logic based on the overlap determination result to remove overlapping targets from the candidate target set, ultimately selecting suitable and valid targets for capture.
[0089] This embodiment traverses all target pairs in a unified coordinate system, uses computational geometry to determine whether the safe zones of each pair of targets intersect, and performs collision detection and conflict resolution on all targets.
[0090] Optionally, if it is determined that the safe area polygons of two targets overlap, both targets are considered ungrabable and are removed.
[0091] In one possible implementation, determining whether any two targets have overlapping regions can be achieved using polygon Boolean operations (such as intersection operations) provided in the computational geometry library, which is efficient and accurate. Specifically, if the safe zone polygons of two targets overlap, both targets are marked as invalid; after traversing all target pairs, all targets not marked as invalid are identified as valid graspable targets, and their pose information list is output.
[0092] In one specific implementation, all target combinations are traversed. For each pair of targets, geometric calculations are performed to determine whether there is an intersection between the two transformed safe area polygons. If the two targets are determined to have an intersection, both targets are marked as invalid overlapping targets. All targets that are not marked as invalid overlapping targets are determined as valid grabbable targets, and their pose information list is output.
[0093] In one possible implementation, the step of identifying the matching template and pose information of each target in the scene image based on template matching includes:
[0094] Determine the background complexity of the scene image;
[0095] If the background complexity is higher than a preset threshold, the scene image is subjected to high-speed filtering to obtain a processed scene image.
[0096] The scene image is matched with templates in a pre-built template library to obtain the matching template for each target in the scene image and the pose information of each target in the scene image.
[0097] It should be noted that in this embodiment, background complexity is used as an indicator to quantify the degree of visual clutter in non-target areas of an image, and can be calculated based on features such as the richness of image texture, the density of edges, or the contrast with the target area. The preset threshold is a threshold value pre-set based on historical experience or experimental data, used to decide whether image filtering needs to be enabled to improve robustness. High-speed filtering refers to a digital image processing technique that can effectively smooth images and suppress high-frequency noise, aiming to reduce the interference of background details or random noise on the template matching process while preserving the main contour features of the target. The pre-built template library is an offline created and stored database containing one or more templates. Each template defines unique visual features (such as contours and textures) of a certain type of target product and serves as a benchmark for online recognition and matching. The matching template refers to the template found successfully from the template library during the template matching process that has the highest similarity to a certain area in the current image, identifying the target's category. Pose information describes the spatial state of the target in the two-dimensional image plane, including at least its position coordinates (x, y) and rotation angle (θ).
[0098] This embodiment performs intelligent evaluation on the acquired raw images, determining whether to apply filtering and noise reduction based on the complexity of the background. Then, it performs a global search and similarity comparison with a known template library to locate and identify all target instances and calculate their precise poses. In other words, it improves the stability and accuracy of template matching through its own preprocessing strategy.
[0099] In one possible implementation, the background complexity can be determined based on the statistical characteristics of the image gradient, such as calculating the gradient magnitude variance of the entire image (or after removing preliminary candidate target regions). If the variance is higher than a threshold, the background texture is considered complex and filtering is required. In addition, the high-speed filtering process can specifically adopt the Gaussian filtering algorithm to smooth the image through convolution operations.
[0100] Furthermore, referring to Figure 2 The second embodiment of the visual deduplication method for overlapping targets in this application provides a flowchart, based on the above. Figure 2 The embodiment shown further refines the step of "constructing a reference polygon for the envelope target boundary according to the matching template" in step S20, including steps A201 to A203:
[0101] Step A201: Obtain the target feature point set associated with the matching template;
[0102] It should be noted that, in this embodiment, the target feature point set refers to the set of coordinates of a series of key pixels extracted from a typical target image represented by the matching template, which can effectively characterize its outline. These feature points are usually extracted from the target outline during the template creation stage using algorithms such as edge detection and corner detection, and are saved as one of the geometric attribute information of the template. Specifically, after identifying a matching template for a target, the system retrieves the pre-calculated and stored list of feature point coordinates bound to it from the pre-built template feature database based on the template's unique identifier (such as template ID).
[0103] Step A202: Perform concave hull calculation on the target feature point set to generate an initial reference polygon vertex sequence;
[0104] It should be noted that in this embodiment, concave hull calculation is a computational geometry algorithm used to find the boundary of the "minimum concave polygon" of a set of planar scattered points. Unlike convex hull, concave hull allows the generated boundary polygon to be concave inward, thus enabling a tighter enclosure of the target point set with irregular shapes or concave structures. The initial reference polygon vertex sequence refers to a series of polygon vertex coordinates arranged in clockwise or counterclockwise order, selected from the input target feature point set through concave hull calculation. This sequence defines a preliminary polygon that can tightly enclose most of the feature points.
[0105] This embodiment automatically transforms a discrete set of feature points that may be unevenly distributed into a continuous, closed polygonal outline that can better reflect the actual shape of the target.
[0106] Step A203: Perform polygon approximation simplification on the initial reference polygon vertex sequence, reduce the number of vertices within a preset fitting error range, and generate the final reference polygon vertex sequence.
[0107] It should be noted that in this embodiment, polygon approximation simplification is an algorithm that optimizes the polygon contour. Its purpose is to approximate the original polygon using fewer vertices while meeting certain accuracy requirements. The preset fitting error range defines the maximum allowable geometric deviation between the simplified polygon and the original polygon. The final baseline polygon vertex sequence refers to the ordered list of polygon vertices obtained after simplification and optimization, which has fewer vertices but retains key shape features.
[0108] This embodiment employs a polygon simplification algorithm. Taking the initial vertex sequence and the preset fitting error as input, it iteratively removes the vertices that contribute the least to the overall shape until it is no longer possible to remove vertices while satisfying the error constraints. This eliminates redundant vertices that may be generated by feature point noise or concave hull calculation, making the polygon contour smoother.
[0109] In one possible implementation, the Douglas-Peucker algorithm is used for simplification. This algorithm achieves simplification by retaining key inflection points and removing midpoints that fall within the tolerance range of the approximate line segment.
[0110] This embodiment achieves the effect of building a high-precision and high-efficiency geometric model for visual recognition targets through a progressive process of acquiring associated feature points, generating concave hulls, and simplifying and optimizing polygons.
[0111] In one possible implementation, the step of constructing the reference polygon of the envelope target boundary based on the matching template further includes:
[0112] Based on the matching template, obtain and display the associated standard target image;
[0113] In response to the user's continuous plotting operation on the boundary of the standard target image, the coordinates of the boundary points are collected sequentially.
[0114] The coordinates of the boundary points are connected in the order of acquisition to form a vertex sequence of the reference polygon.
[0115] It should be noted that in this embodiment, the associated standard target image refers to the image of the target product corresponding to the matching template. It is typically stored as a sample when the template library is created, used to visually demonstrate the complete appearance of the target. Continuous point-tracking is a human-computer interaction action, referring to the user continuously and sequentially clicking along the visible outer boundary of the target object on the displayed image using input devices such as a mouse or touchscreen. Boundary point coordinates refer to the position coordinates (x, y) of the input device cursor in the pixel coordinate system of the standard target image at each point-tracking operation. The reference polygon vertex sequence refers to an ordered list formed by using the sequentially collected boundary point coordinates as the vertices of the polygon and arranging them according to the click order; this sequence defines a closed polygon.
[0116] This embodiment provides a flexible, intuitive, and accurate manual assistance method for defining the geometric contour of a target. It is suitable for scenarios where the template is created only based on the local features of the target, or where the target shape is extremely complex and irregular, resulting in poor automatic hull generation.
[0117] Furthermore, referring to Figure 3 The third embodiment of the visual deduplication method for overlapping targets in this application provides a flowchart, based on the above. Figure 3 The embodiment shown further refines the step S30, "expanding or shrinking the reference polygon based on a preset safety distance, and generating a safety area polygon in conjunction with the matching template," including steps A301 to A303:
[0118] Step A301: Obtain the vertex sequence of the reference polygon and the preset safety distance, wherein a positive safety distance indicates outward expansion and a negative safety distance indicates inward contraction;
[0119] It should be noted that, in this embodiment, the preset safety distance represents the size of the buffer distance that needs to be increased or decreased around the target geometric contour. In this embodiment, the safety distance has a directional attribute: when it is a positive number, it indicates that the reference polygon needs to be expanded outward to reserve collision avoidance space for actuators such as clamps; when it is a negative number, it indicates that the reference polygon needs to be shrunk inward, for example, to define a central safety area for the suction nozzle to grasp.
[0120] Step A302: Call the bias function of the computational geometry library, use the safety distance as the bias amount, and perform bias calculation on the reference polygon vertex sequence to obtain the biased reference polygon vertex sequence.
[0121] It should be noted that in this embodiment, the bias function can accept a polygon (defined by its vertex sequence) and a distance value as input, and output a new polygon, in which each edge of the new polygon is offset by a given safe distance relative to the corresponding edge of the original polygon. The "biased reference polygon vertex sequence" refers to the ordered list of vertex coordinates of the new polygon obtained after calculation by the bias function.
[0122] This embodiment generates a new contour that is geometrically offset by a specified distance. Furthermore, computational geometry library functions typically have built-in robust processing mechanisms that can automatically handle complex situations that may occur during the offset process (such as self-intersection, sharp corner degeneration, etc.), thereby ensuring the geometric validity and computational stability of the output results.
[0123] In one possible implementation, the computational geometry library is the Clipper library, which provides bias functions that can automatically identify and process self-intersecting loops generated during polygon biasing and output clean, self-intersecting result polygons.
[0124] Step A303: Connect the offset reference polygon vertex sequence in order to generate a safe region polygon.
[0125] It should be noted that in this embodiment, the safe area polygon refers to a closed two-dimensional geometric figure formed by sequential connection operations. It represents the projection of the space area that the robot needs to avoid or can safely enter when grasping the target after the safe distance has been expanded (or contracted).
[0126] Furthermore, referring to Figure 4 The fourth embodiment of the visual deduplication method for overlapping targets in this application provides a flowchart, based on the above. Figure 4 The embodiment shown further refines the step S30 of "expanding or shrinking the reference polygon based on a preset safety distance, and generating a safety area polygon in combination with the matching template", including steps A401 to A404:
[0127] Step A401: Obtain the vertex sequence of the reference polygon and the preset safety distance;
[0128] Step A402: Construct a polygon resembling a circular disk with the safety distance as the radius;
[0129] It should be noted that in this embodiment, "using the safety distance as the radius" means using the preset safety distance value as the radius of the circular disk in the geometric construction. "Constructing a polygon similar to a circular disk" means generating a regular polygon to approximate a circular disk with a radius of the safety distance d. The reason for using a polygon approximation instead of a perfect circle is to simplify the subsequent Minkowski calculations and make the calculations between polygons more efficient and stable.
[0130] In this embodiment, a regular polygon with N sides (N can be preset, such as 16 or 32) is generated with the origin of the coordinate system as the center and the safety distance d as the radius of the circumscribed circle. The vertex sequence of this polygon is the desired "circular disk-like polygon", so as to discretize the continuous distance expansion requirement into a calculable geometric figure.
[0131] Optionally, the number of sides N of the regular polygon can be dynamically adjusted according to the size of the safety distance d. The larger d is, the more N can be increased to improve the accuracy of the approximation. For example, the system sets the number of sides N=16. For a safety distance d=180 pixels, the system calculates a regular hexagon, whose coordinates of 16 vertices can be calculated using the formulas (d * cos(2π*i / N), d * sin(2π*i / N)) (i=0,1,…,15). This regular hexagon serves as the approximate "circular disk" structural element polygon.
[0132] Step A403: Calculate the vertex set corresponding to the vertex sequence of the reference polygon and the Minkowski sum of the polygon to obtain the expanded vertex set;
[0133] It should be noted that in this embodiment, the Minkowski sum is a binary operation in computational geometry. For two point sets A and B, their Minkowski sum A⊕B is defined as {a+b | a∈A, b∈B}, where + denotes vector addition. For example, if we consider point set B as a "paintbrush," and use each point in point set A as the starting point for translation, then the union of all these translation trajectories is A⊕B. In this application, this is equivalent to "scanning" the area covered by the structuring element polygon along the boundary of the base polygon A. The expanded vertex set refers to the large number of discrete points contained in the resulting region obtained by calculating the Minkowski sum.
[0134] This embodiment takes the vertex set of the base polygon and the vertex set of the structuring element polygon as input, and generates a point set representing the extended region by calculating all possible vector sums, thus obtaining an accurate extended region.
[0135] Step A404: Extract the boundary contour of the expanded vertex set as the reference polygon vertex sequence to generate the safe area polygon.
[0136] It should be noted that in this embodiment, the points located on the outermost edge of the discrete point set that constitute the convex hull (for convex regions) or more generally the boundary (for potentially concave regions) of the point set are found, and then connected in sequence to generate a safe region polygon.
[0137] Furthermore, referring to Figure 5 The fifth embodiment of the visual deduplication method for overlapping targets in this application provides a flowchart, based on the above. Figure 5 The embodiment shown further refines the step S30 of "expanding or shrinking the reference polygon based on a preset safety distance, and generating a safety area polygon in combination with the matching template", including steps A501 to A505:
[0138] Step A501: Obtain the vertex sequence of the reference polygon and the preset safety distance;
[0139] Step A502: Traverse each edge of the reference polygon and calculate the unit normal vector of each edge;
[0140] In this embodiment, a standardized direction reference is calculated for each edge of the reference polygon. This direction will be used to determine the direction of movement of the edge during the offset process.
[0141] Step A503: For each vertex of the reference polygon vertex sequence, calculate the composite movement direction vector of the vertex based on the unit normal vectors of the two adjacent sides of the vertex.
[0142] It should be noted that, in this embodiment, the synthesized movement direction vector refers to a new unit vector obtained by synthesizing (usually summing and normalizing) the unit normal vectors of the two adjacent edges of a vertex (i.e., the intersection of two edges) during the polygon offset process. This vector indicates in which direction the vertex should move during the offset process, ensuring that the new edge after expansion or contraction can be smoothly connected to the translated adjacent edge, thus solving the problem that simply moving the vertex along the normal vectors of each edge will cause gaps or overlaps in the new polygon at the vertex.
[0143] Step A504: Translate each vertex along the corresponding composite movement direction vector by the safe distance to obtain a new vertex;
[0144] It should be noted that in this embodiment, the coordinates of all new vertices of the safe area polygon are actually calculated based on the calculated vertex movement direction and the preset safe distance.
[0145] Step A505: Connect all new vertices in vertex order to obtain the vertex sequence of the base polygon, and generate the safe region polygon.
[0146] Furthermore, referring to Figure 6 The sixth embodiment of the visual deduplication method for overlapping targets in this application provides a flowchart, based on the above. Figure 6 The embodiment shown further refines the step S40 of "transforming the associated safe area polygon to the same image coordinate system using the pose information of each target", including steps A601 to A604:
[0147] Step A601: For each target, obtain the safe area polygon and pose information corresponding to the target, wherein the pose information includes translation parameters and rotation parameters;
[0148] Step A602: Based on the translation and rotation parameters, construct the rigid body transformation matrix from the template coordinate system to the image coordinate system;
[0149] It should be noted that in this embodiment, the template coordinate system is a local coordinate system associated with the template image (or polygon definition), typically with the template feature point or the image center as the origin. A rigid body transformation matrix is a mathematical matrix used to describe a combination of rotation and translation transformations in a two-dimensional plane (such as a 3x3 homogeneous coordinate transformation matrix), capable of mapping point coordinates in one coordinate system to another.
[0150] In one possible implementation, the rigid body transformation matrix is constructed in the form of a two-dimensional homogeneous coordinate transformation. The upper 2x2 submatrix of this matrix is calculated from the rotation angle θ using trigonometric functions and is responsible for rotation; the first two elements of the third column are composed of translation parameters (x, y) and are responsible for translation.
[0151] Step A603: Transform the coordinates of each vertex in the safe area polygon using the rigid body transformation matrix to obtain new vertex coordinates in the image coordinate system;
[0152] It should be noted that in this embodiment, the geometric definition of the safe area polygon is precisely transformed from the abstract template space to the image space that reflects real physical observation, thereby generating the precise pixel position of the safe area boundary of the target instance in the current scene.
[0153] Step A604: Connect all the new vertex coordinates in sequence to generate a safe region polygon located in the same image coordinate system.
[0154] This embodiment accurately registers the safe area polygons defined offline in their respective template coordinate systems to their actual positions and directions in the real scene image, thereby reconstructing the safe space distribution map of all targets in a unified image coordinate system. This provides a geometric reference benchmark for making unified spatial relationship (such as overlap and interference) judgments on all targets, ensuring the accuracy and consistency of the judgments.
[0155] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the visual deduplication method for overlapping targets in this application. Any simple transformations based on this technical concept are within the protection scope of this application.
[0156] This application also provides a visual deduplication device for overlapping targets; please refer to [reference needed]. Figure 7 The overlapping target visual deduplication device includes:
[0157] The image recognition module 10 is used to acquire scene images of the target to be captured, and to identify the matching template and pose information of each target in the scene image based on template matching.
[0158] Geometric modeling module 20 is used to construct a reference polygon that encloses the target boundary based on the matching template;
[0159] Image processing module 30 is used to expand or shrink the reference polygon based on a preset safety distance, and generate a safety area polygon in combination with the matching template;
[0160] The coordinate transformation module 40 is used to transform the associated safe area polygon to the same image coordinate system using the pose information of each target;
[0161] The overlap determination module 50 is used to determine whether any two targets have an overlapping region in the image coordinate system, so as to perform de-overlap processing.
[0162] The overlapping target visual deduplication device provided in this application, employing the overlapping target visual deduplication method in the above embodiments, can solve the technical problem that it is difficult to accurately determine the overlap of the true contours of irregularly shaped materials. Compared with the prior art, the beneficial effects of the overlapping target visual deduplication device provided in this application are the same as those of the overlapping target visual deduplication method provided in the above embodiments, and other technical features in the overlapping target visual deduplication device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.
[0163] This application provides a visual deduplication device for overlapping targets, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the visual deduplication method for overlapping targets in the first embodiment described above.
[0164] The following is for reference. Figure 8 The diagram illustrates a structural schematic suitable for implementing the visual deduplication device for overlapping targets in the embodiments of this application. The visual deduplication device for overlapping targets in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The visual deduplication device for overlapping targets shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.
[0165] like Figure 8 As shown, the overlapping target visual deduplication device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the overlapping target visual deduplication device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the overlapping target visual deduplication device to communicate wirelessly or wiredly with other devices to exchange data. Although overlapping target visual deduplication devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.
[0166] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.
[0167] The overlapping target visual deduplication device provided in this application, employing the overlapping target visual deduplication method in the above embodiments, can solve the technical problem that it is difficult to accurately determine the overlap of the true contours of irregularly shaped materials. Compared with the prior art, the beneficial effects of the overlapping target visual deduplication device provided in this application are the same as those of the overlapping target visual deduplication method provided in the above embodiments, and other technical features in this overlapping target visual deduplication device are the same as those disclosed in the previous embodiment method, and will not be repeated here.
[0168] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.
[0169] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
[0170] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the overlapping target visual deduplication method in the above embodiments.
[0171] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.
[0172] The aforementioned computer-readable storage medium may be included in the overlapping target visual deduplication device; or it may exist independently and not assembled into the overlapping target visual deduplication device.
[0173] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the overlapping target visual deduplication device, the overlapping target visual deduplication device performs the following actions: acquires scene images of the targets to be captured, and identifies the matching template and pose information of each target in the scene image based on template matching; constructs a reference polygon that encloses the target boundary according to the matching template; performs outward or inward expansion processing on the reference polygon based on a preset safety distance, and generates a safety region polygon in combination with the matching template; transforms the associated safety region polygon to the same image coordinate system using the pose information of each target; and determines whether any two targets have an overlapping region in the image coordinate system for deduplication processing.
[0174] Computer program code for performing the operations of this application can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0175] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0176] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.
[0177] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described visual deduplication method for overlapping targets. This solves the technical problem that the true contours of irregularly shaped materials are difficult to accurately determine for overlap. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the visual deduplication method for overlapping targets provided in the above embodiments, and will not be repeated here.
[0178] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the visual deduplication method for overlapping targets as described above.
[0179] The computer program product provided in this application can solve the technical problem that it is difficult to accurately determine the overlap of the true contours of irregularly shaped materials. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the overlapping target visual deduplication method provided in the above embodiments, and will not be repeated here.
[0180] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.
Claims
1. A visual deduplication method for overlapping targets, characterized in that, The overlapping target visual deduplication method includes: Acquire scene images of the target to be captured, and identify the matching template and pose information of each target in the scene image based on template matching; Construct a reference polygon for the boundary of the envelope target based on the matching template; The baseline polygon is expanded or shrunk based on a preset safety distance, and a safe area polygon is generated by combining it with the matching template; the preset safety distance refers to the length value that is preset or calculated based on the size of the robot's end effector and / or the physical safety requirements of grasping. The associated safe zone polygons are transformed into the same image coordinate system using the pose information of each target; In the image coordinate system, it is determined whether any two targets have overlapping regions, so as to perform de-overlap processing.
2. The visual deduplication method for overlapping targets as described in claim 1, characterized in that, The steps for identifying the matching template and pose information of each target in the scene image based on template matching include: Determine the background complexity of the scene image; If the background complexity is higher than a preset threshold, the scene image is subjected to high-speed filtering to obtain a processed scene image. The scene image is matched with templates in a pre-built template library to obtain the matching template for each target in the scene image and the pose information of each target in the scene image.
3. The visual deduplication method for overlapping targets as described in claim 2, characterized in that, The step of constructing the reference polygon enclosing the boundary of the target based on the matching template includes: Obtain the target feature point set associated with the matching template; Perform concave hull calculation on the target feature point set to generate an initial reference polygon vertex sequence; The initial reference polygon vertex sequence is simplified by polygon approximation, and the number of vertices is reduced within a preset fitting error range to generate the final reference polygon vertex sequence.
4. The visual deduplication method for overlapping targets as described in claim 2, characterized in that, The step of constructing the reference polygon enclosing the boundary of the target based on the matching template further includes: Based on the matching template, obtain and display the associated standard target image; In response to the user's continuous plotting operation on the boundary of the standard target image, the coordinates of the boundary points are collected sequentially. The coordinates of the boundary points are connected in the order of acquisition to form a vertex sequence of the reference polygon.
5. The visual deduplication method for overlapping targets as described in any one of claims 3-4, characterized in that, The step of expanding or shrinking the reference polygon based on a preset safety distance and generating a safe region polygon using the matching template includes: Obtain the vertex sequence of the reference polygon and a preset safety distance, wherein a positive safety distance indicates outward expansion and a negative safety distance indicates inward contraction; The bias function of the computational geometry library is called to perform bias calculation on the reference polygon vertex sequence with the safety distance as the bias amount, so as to obtain the biased reference polygon vertex sequence. The offset reference polygon vertex sequence is connected sequentially to generate a safe region polygon.
6. The visual deduplication method for overlapping targets as described in any one of claims 3-4, characterized in that, The step of expanding or shrinking the reference polygon based on a preset safety distance and generating a safe region polygon using the matching template includes: Obtain the vertex sequence of the reference polygon and the preset safety distance; Construct a polygon resembling a circular disk with the aforementioned safety distance as its radius; Calculate the Minkowski sum of the vertex set corresponding to the vertex sequence of the baseline polygon and the polygon to obtain the expanded vertex set; The boundary contours of the expanded vertex set are extracted as the base polygon vertex sequence to generate the safe region polygon.
7. The visual deduplication method for overlapping targets as described in any one of claims 3-4, characterized in that, The step of expanding or shrinking the reference polygon based on a preset safety distance and generating a safe region polygon using the matching template includes: Obtain the vertex sequence of the reference polygon and the preset safety distance; Traverse each edge of the reference polygon and calculate the unit normal vector for each edge; For each vertex of the reference polygon vertex sequence, the composite movement direction vector of the vertex is calculated based on the unit normal vectors of the two adjacent sides of the vertex; Each vertex is translated by the safe distance along the corresponding composite movement direction vector to obtain a new vertex; Connect all new vertices in vertex order to obtain the vertex sequence of the base polygon, and generate the safe region polygon.
8. The visual deduplication method for overlapping targets as described in claim 1, characterized in that, The step of transforming the associated safe area polygon to the same image coordinate system using the pose information of each target includes: For each target, the corresponding safe area polygon and pose information are obtained, wherein the pose information includes translation parameters and rotation parameters; Based on the translation and rotation parameters, construct a rigid body transformation matrix from the template coordinate system to the image coordinate system; The coordinates of each vertex in the safe area polygon are transformed by the rigid body transformation matrix to obtain new vertex coordinates in the image coordinate system; Connect all the new vertex coordinates in sequence to generate a safe region polygon located in the same image coordinate system.
9. A visual deduplication device for overlapping targets, characterized in that, The overlapping target visual deduplication device includes: The image recognition module is used to acquire scene images of the target to be captured, and to identify the matching template and pose information of each target in the scene image based on template matching. The geometric modeling module is used to construct a reference polygon that encloses the boundary of the target based on the matching template; The image processing module is used to expand or shrink the reference polygon based on a preset safety distance, and generate a safe area polygon by combining it with the matching template; the preset safety distance refers to a length value that is preset or calculated according to the size of the robot's end effector and / or the physical safety requirements of grasping. A coordinate transformation module is used to transform the associated safe area polygons to the same image coordinate system using the pose information of each target; The overlap determination module is used to determine whether any two targets have overlapping regions in the image coordinate system, so as to perform de-overlap processing.
10. A visual deduplication device for overlapping targets, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the visual deduplication method for overlapping targets as described in any one of claims 1 to 8.