Full-automatic production of body intelligent robot cargo identification and grabbing method and system

Abstract

The application provides a full-automatic production body intelligent robot cargo identification and grabbing method and system, and relates to the technical field of robot grabbing control. The method determines a target compartment according to a production task, bin positioning information and compartment material mapping information, collects material taking perception data and identifies the main body area, the lock protruding area, the compartment boundary and the adjacent constraint object of the target connector assembly; determines the lock direction according to the relative attitude of the main body area and the lock protruding area, and generates the local constraint relationship of the lock protruding area; when the local constraint can be released, the robot is controlled to move the target connector assembly along the release direction first, and then is grabbed. The method can reduce the risk of jamming, material carrying and lock damage when the connector assembly is directly lifted.

Description

Technical Field

[0001] This application relates to the field of robot grasping and control technology, and in particular to a method and system for cargo recognition and grasping of a fully automated production-integrated intelligent robot. Background Technology

[0002] In the automated production of electronic products, industrial controllers, home appliances, and wire harnesses, production lines often utilize stacker cranes, conveyor lines, and robotic picking stations to collaboratively supply small materials. Stacker cranes are suitable for high-speed storage and retrieval of multi-compartment bins in automated storage and retrieval systems or line-side buffer zones, while embodied intelligent robots are suitable for material identification and grasping at picking stations based on visual, depth, or tactile information. Therefore, the combination of the two can achieve high-density buffering, rapid connection, and flexible end-point picking.

[0003] For connector assemblies with locking structures, existing production lines typically determine the target grid based on the task order, bin code, or grid material mapping information. A camera then captures an image of the target grid and estimates the overall pose of the connector assembly. Subsequently, grippers are controlled to hold the visible main body area of ​​the connector assembly and extract it. This approach is well-suited for small materials with regular shapes, no obvious overhanging structures, and relatively large surrounding gaps.

[0004] However, when multiple connector assemblies are stored in batches within multi-compartment bins, multiple assemblies of the same model often exist in the same compartment. The latches, buckles, or elastic clips on these connector assemblies may protrude outwards relative to the main body. When the target connector assembly approaches the compartment partition, bin wall, or adjacent connector assembly, the protruding area of ​​the latch may be partially blocked, jammed, or hooked by the side of the partition, the inner wall of the bin, or the edge of adjacent material. If the robot directly lifts the assembly based solely on its overall position and the gripping area of ​​the main body, it may be able to grip the main body, but the protruding area of ​​the latch may scrape, get stuck, pull out adjacent material, or damage the latch during the removal process. Existing solutions for stacker crane operation, general robot navigation, or general stable gripping typically do not consider the relationship between the protruding area of ​​the latch and the local constraints of the compartment, and it is difficult to provide a releaseable lateral movement before gripping. Summary of the Invention

[0005] The purpose of this application is to provide a method and system for cargo identification and grasping of a fully automated production embodied intelligent robot, in order to solve the problem in the prior art of how to identify the local constraint state of a locking connector assembly in a multi-compartment material picking scenario, and generate a control action of releasing before grasping when it is releasable.

[0006] To address the aforementioned technical problems, in a first aspect, this application provides a method for cargo identification and grasping using a fully automated production-integrated intelligent robot, the method comprising: Based on the target cargo information corresponding to the current production task, the positioning information of the multi-compartment bin at the robot material handling station, and the material mapping information of the multi-compartment bin, the target compartment is determined; Collect material handling perception data within the target compartment, identify the target connector component to be grasped based on the material handling perception data, and identify the main body area, latch protrusion area, compartment boundary of the target compartment, and adjacent constraint objects located around the target connector component. The latch orientation of the target connector assembly is determined based on the relative posture between the main body area and the latch protruding area. Based on the latch orientation, the grid boundary, and the adjacent constraint object, determine whether the latch protrusion area on the target connector assembly forms a local constraint with the grid boundary or the adjacent constraint object, so as to generate a local constraint relationship; When the local constraint relationship indicates that the latch protrusion area forms a releasable local constraint, the release direction and release displacement are determined according to the local constraint relationship, and the embodied intelligent robot is controlled to move the target connector assembly along the release direction so that the latch protrusion area is detached from the corresponding constraint object, and then the target connector assembly is grasped.

[0007] Optionally, the step of determining whether the protruding area of ​​the latch on the target connector assembly forms a local constraint with the grid boundary or the adjacent constraint object based on the latch orientation, the grid boundary, and the adjacent constraint object, in order to generate a local constraint relationship, includes: Establish a grid coordinate system based on the grid boundary of the target grid, determine the rib side boundary and the extraction direction inside the grid in the grid coordinate system, and determine the box wall side boundary when the target grid is adjacent to the outer periphery of the material box. Transform the main body area of ​​the target connector assembly, the latch protrusion area on the target connector assembly, and the adjacent constraint object into the grid coordinate system; After the conversion is completed, the protruding area of ​​the latch is swept along the extraction direction to obtain the space area swept by the latch extraction, and the projection overlap between the space area and the side boundary of the rib, the side boundary of the box wall, and the adjacent constraint object is calculated respectively. Calculate the releasable displacement in the direction opposite to the direction of the latch, and determine the local constraint relationship based on the projected overlap and the releasable displacement.

[0008] Optionally, the sweeping of the protruding area of ​​the latch along the removal direction to obtain the space area swept during latch removal includes: In the grid coordinate system, determine the set of hook-sensitive points within the buckle protrusion area; Based on the depth error of the material picking perception data, the recognition confidence of the buckle protrusion area, and the positioning error of the multi-compartment bin at the robot picking station, the outward expansion of the hook sensitive point set in the buckle orientation, the picking direction, and the lateral direction perpendicular to the buckle orientation is determined respectively. According to the buckle orientation, the extraction direction, and the lateral expansion amount, the hook sensitive point set is anisotropically expanded to obtain the buckle expansion envelope; Based on the lateral clearance of the main body region within the target slot and the grasping and positioning error of the embodied intelligent robot, the allowable attitude deviation range relative to the removal direction when the target connector assembly is removed is determined; Within the allowable attitude sway range, multiple height layers are set along the extraction direction, and the latch outer envelope is copied on each height layer according to the corresponding allowable attitude. The latch outer envelopes on each height layer are merged to obtain the spatial region.

[0009] Optionally, the projection overlap between the spatial region and the side boundary of the rib, the side boundary of the box wall, and the adjacent constraint objects is calculated, including: Based on the grid boundary, construct the rib blocking zone corresponding to the rib side boundary and the box wall blocking zone corresponding to the box wall side boundary in the grid coordinate system, and construct the adjacent object occupancy envelope based on the outer contour of the adjacent constraint object and the recognition confidence of the adjacent constraint object. The spatial region, the partition strip, the box wall barrier, and the adjacent object's occupancy envelope are projected onto a detection plane perpendicular to the extraction direction, and the overlapping areas of the spatial region with the partition strip, the box wall barrier, and the adjacent object's occupancy envelope are determined in the detection plane. The overlapping areas are continuously filtered according to the corresponding height layers. Overlapping areas that span multiple consecutive height layers are taken as effective overlapping areas. The corresponding projected overlap amount is calculated based on the coverage length of the effective overlapping areas in the latch direction and the continuous height in the extraction direction.

[0010] Optionally, the step of calculating the releasable displacement along a direction opposite to the latch orientation, and determining the local constraint relationship based on the projected overlap and the releasable displacement, includes: Calculate the releasable displacement of the main body region within the target compartment in a direction opposite to the direction of the latch; For each of the rib side boundary, the box wall side boundary, and the adjacent constraint objects, obtain the corresponding projection overlap and locking gap respectively; When the projected overlap of any object is greater than a preset overlap and the corresponding locking gap is less than a preset locking gap, the object that meets the conditions is determined as a constraint object, and a local constraint relationship is generated based on the releasable displacement and the constraint object; or, when the side boundary of the rib, the side boundary of the box wall, and the adjacent constraint object do not meet the conditions that the projected overlap is greater than a preset overlap and the locking gap is less than a preset locking gap, a local constraint relationship indicating that no local constraint has been formed is generated.

[0011] Optionally, generating local constraint relationships based on the releasable displacement and the constraint object includes: Determine whether the releasable displacement is greater than the release displacement required for the latch protrusion area to cross the constrained object; If the releasable displacement is greater than the release displacement, then the direction opposite to the direction of the latch and avoiding the constrained object is determined as the release direction, and the displacement that is not less than the release displacement and not greater than the releasable displacement is determined as the release displacement, and a local constraint relationship containing the constrained object, the release direction and the release displacement is generated, indicating that it is releasable. Alternatively, if the releasable displacement is not greater than the release displacement, a local constraint relationship is generated that includes the constraint object and a prohibition on direct lifting marker, indicating that it cannot be directly released.

[0012] Optionally, after generating local constraint relations representing those that cannot be directly released, the method further includes: The embodied intelligent robot is restricted from directly grasping the target connector assembly along the extraction direction.

[0013] Optionally, determining the latch orientation of the target connector assembly based on the relative posture between the main body region and the latch protruding region includes: Determine the connection position between the latch protrusion area and the main body area, and the outward extension direction of the latch protrusion area relative to the main body area; Based on the connection position and the outward direction, an initial orientation of the latch is generated, and the initial orientation of the latch is verified based on at least one of the angle between the latch protruding area and the main body area, the protrusion distance, and the height relationship. When the verification is successful, the initial orientation of the latch is determined as the latch orientation of the target connector assembly.

[0014] Optionally, identifying the target connector component to be grasped based on the material handling perception data includes: Based on the material handling sensing data, multiple candidate connector components are identified from within the target slot; Based on the degree of exposure of the main body, the identifiability of the latch protrusion area, and the reachability of the clamping, one of several candidate connector components is selected as the target connector component to be grasped in this operation.

[0015] Secondly, this application provides a cargo identification and grasping system for a fully automated production embodied intelligent robot, the system comprising: The first determining module is used to determine the target compartment based on the target goods information corresponding to the current production task, the positioning information of the multi-compartment bin at the robot picking station, and the compartment material mapping information of the multi-compartment bin. The identification module is used to collect material picking perception data in the target compartment, identify the target connector component to be picked up this time based on the material picking perception data, and identify the main body area, the latch protrusion area, the compartment boundary of the target compartment, and the adjacent constraint objects located around the target connector component. The second determining module is used to determine the latch orientation of the target connector assembly based on the relative posture between the main body area and the latch protrusion area. The judgment module is used to determine, based on the buckle orientation, the grid boundary, and the adjacent constraint object, whether the buckle protrusion area on the target connector assembly forms a local constraint with the grid boundary or the adjacent constraint object, so as to generate a local constraint relationship. The control module is used to determine the release direction and release displacement according to the local constraint relationship when the local constraint relationship indicates that the latch protrusion area forms a releasable local constraint, and to control the embodied intelligent robot to move the target connector assembly along the release direction so that the latch protrusion area is detached from the corresponding constraint object, and then grasp the target connector assembly.

[0016] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor is configured to execute the computer program to implement the steps of the cargo identification and grasping method for a fully automated production embodied intelligent robot as described in the first aspect above.

[0017] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps of the cargo identification and grasping method of the fully automated production embodied intelligent robot described in the first aspect above.

[0018] The cargo identification and grasping method of the fully automated production robot provided in this application has the following beneficial effects: This application does not only identify the overall pose of the connector assembly, nor does it directly lift the assembly based on the first visual grasping result after the material box is in place. Instead, for connector assemblies with locking structures, the protruding area of ​​the locking mechanism is identified as a local structure distinct from the main body area. The method combines the grid boundary, ribs, box walls, and adjacent constraint objects to determine whether a local constraint is formed. Under releasable local constraints, the release direction and release displacement are first determined, and the target connector assembly is controlled to move along the release direction before grasping. Therefore, the risk of jamming, material snagging, and damage caused by directly lifting the assembly when the locking mechanism is blocked, stuck, or hooked by ribs, box walls, or adjacent materials can be reduced.

[0019] Furthermore, this application uses a grid coordinate system to express the positional relationship between the target connector assembly, the latch protruding area, and the constrained object, and forms a spatial area swept by the latch during removal along the removal direction. This, combined with projection overlap, locking gap, and releasable displacement, generates local constraint relationships, making local constraint judgment no longer dependent on the two-dimensional contact impression in a single frame image. By using a hook-sensitive point set, anisotropic expansion, allowable attitude sway range, and height layer continuity filtering, depth error, bin positioning error, and robot grasping positioning error can be incorporated into the judgment process, thereby improving the accuracy and reliability of constraint identification before grasping small connector assemblies in multi-grid bins. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a fully automated production robot's cargo identification and grasping method disclosed in this application. Figure 2 This is a flowchart illustrating a local constraint relationship generation process disclosed in this application; Figure 3 This is a schematic diagram showing the positional relationship between a target connector assembly, a latching protrusion area, and a local constraint object within a multi-compartment bin, as disclosed in this application. Figure 4 This is a schematic diagram of the process of grasping a target connector assembly after releasing local constraints, as disclosed in this application. Figure 5 This is a schematic diagram of the structure of a fully automated production-integrated intelligent robot cargo recognition and grasping system disclosed in this application. Detailed Implementation

[0022] In fully automated production lines, connector assemblies with locking structures can be used for assembling products such as industrial controllers, home appliance control boards, power modules, sensor harnesses, and communication interface modules. These connector assemblies typically include a plastic body, a metal terminal housing, and a locking structure, which can be a resilient snap, a snap-fit, or a limiting boss located on the side or top of the body. For small 6-pin locking connectors, the connector assembly can have a length of 18mm to 32mm, a width of 8mm to 18mm, and a height of 6mm to 15mm, with the protrusion distance of the locking protrusion relative to the outer surface of the body ranging from 0.8mm to 3mm. These dimensional ranges correspond to the physical quantities of common small industrial connectors and are used to illustrate the application scenarios of this application, but do not constitute a limitation on the scope of protection of this application.

[0023] The production line can use multi-compartment bins to hold connector assemblies. For example, the external dimensions of the multi-compartment bin can be 400mm×300mm×120mm or 600mm×400mm×150mm. The compartments can be arranged in 3×4, 4×6, or other matrix configurations. The internal dimensions of a single compartment can be 60mm×50mm to 120mm×90mm, and the rib thickness can be 1.5mm to 4mm. The bottom of the compartment can be chamfered or rounded to facilitate the sliding of small parts. Multiple connector assemblies of the same model can be placed in one compartment, for example, 5 to 30. The robot grasps only one connector assembly at a time as the target connector assembly to be grasped, and the remaining connector assemblies may form adjacent constraint objects.

[0024] In this scenario, the bin-type stacker crane is mainly responsible for high-speed conveying of bins in buffer areas, automated warehouses, or line-side racks. Its advantage lies in handling entire boxes, pallets, or compartments of material. The embodied intelligent robot is mainly responsible for identifying individual materials, judging their local state, and performing grasping at the picking station. Stacker cranes typically cannot select a specific small connector component within a bin and avoid latches. If the robot directly undertakes long-distance bin retrieval, buffering, and line-side material supply, the equipment cycle time, space occupation, and energy consumption are all uneconomical. Therefore, this application addresses the scenario where the stacker crane delivers multiple bins to the picking station, and the embodied intelligent robot completes the precise picking of small parts.

[0025] The core idea of ​​this application is as follows: After determining the target slot, the target connector component to be grasped is selected from multiple candidate connector components within the target slot. Then, the main body area and the latch protrusion area of ​​the target connector component are identified separately. Subsequently, the latch orientation is determined based on the outward extension direction of the latch protrusion area relative to the main body area, and it is determined whether the latch protrusion area will cause local constraints with the partition, box wall or adjacent connector components during the removal process. If the local constraints can be released by a small displacement, the release movement is executed first, and then the grasping is executed, so that the robot's grasping action matches the actual local constraint state of the connector component.

[0026] Therefore, the technical closed loop formed by this application is as follows: the risk in the background technology comes from the fact that the main body can be clamped but the latch is partially restricted. The identification step of this application first divides the target connector assembly into the main body area and the latch protruding area. The judgment step then converts the local spatial relationship between the latch protruding area and the grid boundary, partition, box wall and adjacent materials into a local constraint relationship. The control step then outputs the release direction, release displacement or prohibition of direct lifting mark according to the local constraint relationship. In this way, there is a direct correspondence between the technical problem, the technical solution and the technical effect. That is, by identifying the local restriction state of the latch before gripping, the jamming, material smuggling and latch damage caused by direct lifting are reduced.

[0027] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] Example 1: Figure 1 This is a flowchart illustrating a fully automated production robot's cargo identification and grasping method disclosed in this application. Figure 1 This is used to illustrate the overall process from S101 to S105. From the data flow perspective, S101 outputs the target grid and its spatial range, which serves as the region of interest for S102 to collect material sensing data; S102 outputs the main body area of ​​the target connector assembly, the latch protrusion area, the grid boundary, and adjacent constraint objects, which serve as the geometric input for S103 and S104; S103 outputs the latch orientation, which serves as the directional reference for the locking gap, releasable displacement, and release direction in S104; S104 outputs the local constraint relationship, which serves as the control basis for S105 to generate release movement or restrict direct lifting action.

[0029] like Figure 1 As shown, the method includes: S101. Determine the target compartment based on the target cargo information corresponding to the current production task, the positioning information of the multi-compartment bin at the robot picking station, and the material mapping information of the compartment of the multi-compartment bin.

[0030] The current production task can be issued by the Manufacturing Execution System, Warehouse Management System, or Production Line Control System. The target goods information may include the material code, material name, number of connector bits, whether it has a locking structure, batch number, and quantity to be picked up at one time. For example, if the current production task requires assembling a 6-pin locking connector assembly for an industrial controller, the target goods information may include the material code of the connector assembly and its corresponding locking structure marking.

[0031] The positioning information of multi-compartment bins at the robot's material handling station can be determined by the bin's barcode, RFID tag, station stop mechanism, positioning pin, visual positioning marker, or the bin's outer contour identified by a depth camera. The material mapping information for each compartment can be pre-attached to the bin's code or generated by the production line control system based on loading records. For example, in a 4×6 multi-compartment bin, if the compartment in row 2, column 5 is mapped to a 6-pin latching connector assembly, and the compartment in row 3, column 1 is mapped to a 2-pin connector assembly, then the robot can determine the spatial range of the target compartment in the robot's coordinate system by combining the target goods information and the bin's positioning information.

[0032] In S101, the target cargo information is used to determine which material to retrieve, the multi-compartment bin positioning information is used to determine the current location of the bin at the robot's retrieval station, and the compartment material mapping information is used to determine which compartment the material is placed in. The acquisition of these three types of information can be achieved using conventional techniques, such as reading the task order through a manufacturing execution system interface, reading the bin code through a QR code or RFID tag, and determining the bin's outer contour position through positioning pins and visual calibration. This application does not consider these information acquisition methods themselves as unconventional inventive points, but rather as entry points for subsequent local constraint judgment of the locking mechanism, preventing the robot from performing precise gripping on incorrect compartments or incorrect materials.

[0033] Once the target grid is determined, the system output can include the row and column number of the target grid, the three-dimensional boundary of the target grid in the robot coordinate system, the material code corresponding to the target grid, and a mark indicating whether the material has a locking structure. If the bin code and the grid material mapping information are inconsistent, or if the positioning deviation of the target grid at the picking station exceeds the allowable deviation, for example, exceeding 2mm to 5mm, the current picking can be stopped and a request can be made to reposition the bin to avoid subsequent mismatches between the identified object and the production task.

[0034] The output of S101 does not directly determine the grasping posture, but rather defines the spatial range for subsequent visual recognition. For example, the three-dimensional boundary of the target compartment can be projected onto the camera image to form a clipping region for material handling perception data, and other materials outside the target compartment can be filtered out in the point cloud. This reduces the interference of materials outside the target compartment on instance segmentation and also allows the subsequent compartment boundaries, rib side boundaries, and box wall side boundaries to be obtained from the same spatial reference.

[0035] S102. Collect material picking perception data within the target compartment, identify the target connector component to be picked up this time based on the material picking perception data, and identify the main body area, latch protrusion area, compartment boundary of the target compartment, and adjacent constraint objects located around the target connector component.

[0036] Material handling perception data can include RGB images of the target grid, depth images, structured light point clouds, binocular parallax data, camera extrinsic parameters, bin positioning mark positions, and gripper workspace data. The acquisition device can be installed at the robot's end effector or above the material handling station. For small connector assemblies with a size of approximately 20mm, the working distance of the depth camera can be set to 300mm to 800mm, with depth resolution or repeatability error in the range of 0.3mm to 1.5mm. If a structured light camera is used, multi-frame fusion or localized supplemental lighting can reduce recognition instability caused by black plastic parts and reflective terminals.

[0037] When identifying the target connector component to be grasped, multiple candidate connector components can first be segmented from the target slot based on material handling perception data. Candidate connector components can be fully exposed connectors or connectors partially obscured by adjacent materials but still clampable. Subsequently, the candidate connector components are evaluated based on the degree of exposure of the main body, the identifiability of the latch protrusion area, and clamping accessibility. The degree of exposure of the main body indicates the matching ratio between the visible main body area and the preset main body template; the identifiability of the latch protrusion area indicates whether the latch structure can be stably distinguished from the point cloud or image; and clamping accessibility indicates whether the gripper can enter the main body clamping area without colliding with the slot boundary. The candidate connector component with the highest overall evaluation and meeting the minimum grasping conditions is determined as the target connector component to be grasped.

[0038] The main body area refers to the area within the target connector assembly to be grasped that carries terminals, wire harness insertion ports, or the main contour of the housing. The latching protrusion area refers to the snap-fit, buckle, or boss area located on the target connector assembly and extending outward relative to the main body area. Adjacent constraint objects refer to adjacent materials within the target compartment other than the target connector assembly, such as adjacent connector assemblies or the edges of scattered similar materials.

[0039] In S102, identifying the main body area of ​​the target connector assembly can be achieved using conventional techniques such as instance segmentation, template matching, point cloud clustering, or edge contour fitting; identifying the grid boundary can be achieved using conventional techniques such as bin outer contour fitting, rib segment detection, or pre-calibrated bin model projection. Unlike conventional overall grasping, this application further decomposes the same target connector assembly into two local objects: the main body area and the latching protrusion area, and treats the remaining candidate connector assemblies, ribs, and bin walls as adjacent constraint objects in subsequent judgments. This is because whether the target connector assembly can be clamped is mainly determined by the main body area, while whether it can be successfully removed often depends on whether the latching protrusion area is partially blocked; the two cannot be simply replaced by the same overall outer frame.

[0040] The recognition output can be represented as a point set, a 2D contour, a 3D envelope, or a semantic mask. To avoid confusion between "the target connector component to be captured" and "all connector components within the grid," the system can assign a candidate number to each candidate connector component and output the main body area and latch protrusion area only for the selected target connector component; unselected connector components are treated only as adjacent constraint objects. If the main body exposure level of a candidate connector component is less than 0.55, or the recognizability of the latch protrusion area is less than 0.50, it can be excluded from the current capture target to ensure reliable input for subsequent latch orientation and local constraint judgments.

[0041] The output of S102 includes at least four types of data: the main area for selecting the gripping zone, the latch protruding area for judging the risk of hooking, the grid boundary for establishing the grid coordinate system, and the adjacent constraint objects for judging blocking or hooking. These four types of data have different functions: the main area primarily serves gripping stability, the latch protruding area primarily serves retrieval safety, and the grid boundary and adjacent constraint objects constitute the environmental boundary in the local constraint judgment.

[0042] S103. Determine the latch orientation of the target connector assembly based on the relative posture between the main body area and the latch protrusion area.

[0043] The relative orientation can include the connection position and outward direction of the latch protrusion relative to the main body area, and may further include one or more of the following: the angle between the latch protrusion and the main body area, the protrusion distance, and the height relationship. The latch orientation is preferably the outward direction of the latch protrusion relative to the main body area. For example, when the latch protrusion is connected to the right side wall of the main body area and extends outward to the right, the latch orientation is the direction from the main body area to the outside of the right side wall; when the latch is located above the main body and extends obliquely upward to the outside, its outward component in the horizontal plane of the grid coordinate system can be used as the latch orientation for subsequent local constraint determination.

[0044] In one specific implementation, the connection position between the latch protrusion and the main body area, and the outward extension direction of the latch protrusion relative to the main body area, can be determined first. An initial latch orientation is generated based on the connection position and outward extension direction, and then verified using at least one of the following: the angle between the latch protrusion and the main body area, the protrusion distance, and the height relationship. When the verification passes, the initial latch orientation is determined as the latch orientation of the target connector assembly. When the verification fails, it indicates that the latch area may be obstructed, misidentified due to reflection, or confused with the edge of adjacent materials. In this case, local sensing data of the target grid can be re-acquired, or the grasping priority of the candidate connector assembly can be reduced and another candidate connector assembly selected. Before the latch orientation can be reliably determined, the robot is not controlled to directly lift the target connector assembly.

[0045] In S103, the connection position can be determined by the contact boundary between the main body area and the latching protrusion area, and the outward direction can be determined by the direction from the center point of the outer contour of the latching protrusion area to the opposite side of the connection position. The included angle, protrusion distance, and height relationship are used for verification, not necessarily all of them. For example, if the included angle between the outward direction of the latching protrusion area and the normal to the main body sidewall is greater than 45°, and the latch of this type of connector should typically be approximately perpendicular to the main body sidewall, then there is a risk of misidentification of the initial latch orientation. If the latch protrusion distance is less than 0.3mm or greater than 5mm, it can also be considered to not conform to the common structural size range of small latching connectors, requiring re-identification or switching of candidate connector assemblies.

[0046] The latch orientation output by S103 can be represented as a unit direction vector in the grid coordinate system, such as +X, -X, +Y, -Y directions or combinations thereof. This unit direction vector is then used in three different stages: first, in S104, to determine the key outward expansion direction of the spatial area swept by the latch during removal; second, to calculate the locking clearance from the latch protrusion area to the constrained object; and third, to determine the release direction opposite to the latch orientation when there are releasable local constraints.

[0047] S104. Based on the latch orientation, the grid boundary, and the adjacent constraint object, determine whether the latch protrusion area on the target connector assembly forms a local constraint with the grid boundary or the adjacent constraint object, so as to generate a local constraint relationship.

[0048] Local constraint refers to the state where the latching protrusion area is partially blocked, jammed, or hooked by the grid boundary or adjacent constraint objects when the target connector assembly is removed or released, preventing the target connector assembly from simply leaving the target grid in the removal direction. This local constraint does not necessarily manifest as the entire main area being clamped; in many cases, the main area still has a visible clamping surface, but the corner where the latching protrusion is located has entered the narrow gap formed by the rib or adjacent material.

[0049] For example, the grid boundary, such as the side of the rib, the inside of the box wall, etc.; the constraint object can be at least one of the rib of the target grid, the box wall of the target grid, or an adjacent constraint object.

[0050] In S104, the grid boundary is used to define the movable spatial boundary of the target connector assembly, the latch orientation is used to determine the direction in which the latch protrusion area is most likely to jam or hook, and the adjacent constraint object is used to represent objects other than the target connector assembly that may form an obstruction. If only conventional collision detection is used, it can detect whether the overall outer frame overlaps with an obstacle, but it is difficult to distinguish cases where the main body can pass but the latch cannot pass in certain areas; if only inertia or the bin conveying direction is used to infer the restricted edge, it cannot handle cases where the connector assembly randomly flips within the grid, adjacent materials are squeezed, or the latch orientation is inconsistent with the stacker crane's movement direction. Therefore, this application uses the space area swept by the latch removal, the projected overlap, the locking gap, and the releasable displacement to jointly generate local constraint relationships, which is an unconventional local structure judgment process for connector assemblies with latches.

[0051] Local constraint relationships can include at least three types: no local constraint, releasable local constraint, and non-releasable local constraint. No local constraint means the latch protruding area is not effectively blocked during removal; a releasable local constraint means that although the latch protruding area is blocked by the constrained object, the target connector assembly still has sufficient release space in the direction opposite to the latch's orientation; a non-releasable local constraint means the latch protruding area is blocked by the constrained object and there is insufficient space for release in the opposite direction, making direct lifting risky.

[0052] Local constraint relationships can be represented using structured records. These structured records may include at least one of the following: relationship type, constraint object identifier, latch orientation, projection overlap, latch gap, releasable displacement, release displacement, release direction, release displacement, and a "do not lift directly" flag. For cases where no local constraint is formed, the structured record may not include the release direction and release displacement; for releasable local constraints, the structured record must at least include the constraint object, release direction, and release displacement; for local constraints that cannot be directly released, the structured record must at least include the constraint object and a "do not lift directly" flag.

[0053] S105. When the local constraint relationship indicates that the latch protrusion area forms a releasable local constraint, the release direction and release displacement are determined according to the local constraint relationship, and the embodied intelligent robot is controlled to move the target connector assembly along the release direction so that the latch protrusion area is detached from the corresponding constraint object, and then the target connector assembly is grasped.

[0054] The release direction can be opposite to the direction of the latch and avoid the constrained object. The release displacement can be the minimum displacement required for the target connector assembly to move so that the latch protrusion area passes the constrained object, plus a safety margin. For example, if the disengagement displacement of the latch protrusion area relative to the side of the rib is 2.2 mm, and the releasable displacement of the main body area within the target slot in the opposite direction is 5.0 mm, then the release displacement can be 2.8 mm to 3.5 mm. The robot can first clamp the main body clamping area under low speed and low acceleration conditions and translate the release displacement along the release direction, and then lift it along the removal direction. The above displacement values ​​are matched with the latch protrusion amount and slot gap of the small plastic connector, and are used to illustrate the implementation process, without limiting the specific parameters of this application.

[0055] In S105, controlling the embodied intelligent robot to move the target connector assembly along the release direction can be achieved through robot end-effector pose control, gripper closing force control, and speed limiting. Conventional robot trajectory interpolation, inverse kinematics solving, and end-effector closed-loop control can be used to execute this movement. The unconventional aspect of this application is that this release movement is not a preset fixed jittering motion, nor is it an arbitrary directional adjustment motion, but rather it is determined by the constraint object, latch orientation, and disengagement displacement in the local constraint relationship. After the release movement is completed, the system can re-acquire one or more frames of local perception data to confirm that the locking gap between the latch protrusion area and the constraint object has increased, and then perform an upward lift along the extraction direction to reduce secondary jamming caused by insufficient release.

[0056] Example 2: Figure 2 This is a flowchart illustrating a local constraint relationship generation process disclosed in this application.

[0057] Figure 3 This is a schematic diagram illustrating the positional relationship between a target connector assembly, a latching protrusion, and a locally constrained object within a multi-compartment bin, as disclosed in this application. Figure 3As shown, target compartment 110 is one compartment in a multi-compartment bin. The target connector assembly is located within target compartment 110. The target connector assembly includes a main body region 10 and a latching protrusion region 20 located on one side of the main body region 10. The latching protrusion region 20 extends outward relative to the main body region 10 toward one side of the target compartment 110, thus forming the latching orientation shown in the figure. Adjacent connector assemblies 30 are also placed within the target compartment 110. The adjacent connector assemblies 30 are located around the target connector assembly and can participate in local constraint judgment as adjacent constraint objects. The compartment boundary around the target compartment 110 includes a rib side boundary 120 formed by ribs between adjacent compartments. A bin wall 130 is also formed around the outer periphery of the bin. The rib side boundary 120 and the bin wall 130 are used to characterize the boundary types that may correspond to compartments at different locations. Therefore, in the subsequent judgment process, it can be determined whether the target connector assembly needs to be moved in the release direction before lifting and grabbing the target connector assembly in the extraction direction, based on the latch orientation of the latch protrusion area 20, the side boundary 120 of the partition rib, the box wall 130 and the relative position between the adjacent connector assembly 30, so as to release the local constraint between the latch protrusion area 20 and the partition rib, the box wall or the adjacent connector assembly 30.

[0058] Figure 4 This is a schematic diagram illustrating the process of grasping a target connector assembly after release under a releasable local constraint, as disclosed in this application. When the constraint object is a rib, the latch protruding area of ​​the target connector assembly is close to the corresponding rib side boundary 120. If the embodied intelligent robot directly lifts and grasps along the extraction direction, the latch protruding area may get stuck with the constraint object. Therefore, when the local constraint relationship indicates that the latch protruding area forms a releasable local constraint, the embodied intelligent robot first acts on the main body area of ​​the target connector assembly and moves the target connector assembly along the release direction by a preset release displacement, so that the latch protruding area is detached from the corresponding constraint object. After the local constraint is released, the embodied intelligent robot then lifts and grasps the target connector assembly along the extraction direction, thereby reducing the risk of jamming, bringing out adjacent materials, or damaging the latch when directly lifting and grasping.

[0059] like Figures 2 to 4 As shown below, the generation process of the local constraint relationship in S104 will be further explained. Since the outward extension direction and removal direction of the latch protruding area relative to the main body area, as well as the ribs, box walls and adjacent materials in the grid, all affect the grasping risk, this application unifies the above objects into the grid coordinate system for judgment, rather than judging whether there is contact based solely on the two-dimensional distance in the camera image.

[0060] S201. Establish a grid coordinate system based on the grid boundary of the target grid, determine the rib side boundary of the target grid and the extraction direction within the grid in the grid coordinate system, and determine the box wall side boundary when the target grid is adjacent to the outer periphery of the material box.

[0061] Specifically, a corner point on the bottom surface of the target compartment can be used as the origin of the coordinate system. The width direction of the compartment can be set as the X-axis, the length direction as the Y-axis, and the upward lifting direction perpendicular to the bottom surface of the compartment as the Z-axis, i.e., the removal direction. The side boundary of the ribs can be determined by the inner contour of the ribs between the target compartment and adjacent compartments, and the side boundary of the box wall can be determined by the inner contour of the target compartment near the outer wall of the box. When the target compartment is located in the middle of a multi-compartment box, its surrounding area can mainly be the side boundary of the ribs; when the target compartment is located at the edge of the box, one or both sides can be the side boundary of the box wall.

[0062] S202. Transform the main body area of ​​the target connector assembly, the latch protrusion area on the target connector assembly, and the adjacent constraint objects into the grid coordinate system.

[0063] This transformation can be performed based on camera extrinsic parameters, robot end-effector pose, and bin positioning information. After the transformation, the main body area, the latch protrusion area, and adjacent constraint objects can all be represented as point sets, envelopes, or segmented contours in the grid coordinate system. This allows for direct calculation of the distances along specific directions between the latch protrusion area and the side boundaries of the ribs, the side boundaries of the bin walls, and adjacent constraint objects, without being affected by the camera mounting angle.

[0064] S203. After the conversion is completed, sweep the protruding area of ​​the latch along the extraction direction to obtain the space area swept by the latch extraction, and calculate the projection overlap between the space area and the side boundary of the rib, the side boundary of the box wall, and the adjacent constraint objects.

[0065] When defining the space area swept by the latch during removal, the set of hook-sensitive points within the latch protrusion area can be determined first in the grid coordinate system. This set of hook-sensitive points may include the outer edge points, lower lip edge points, corner points, and height abrupt change points of the latch protrusion area. For a 6-pin connector assembly with a latch protrusion distance of approximately 1.6mm, the set of hook-sensitive points is typically concentrated at the outer edge of the latch and the corner of the latch, rather than considering the entire body area as the hook-sensitive region.

[0066] Next, based on the depth error of the material handling perception data, the recognition confidence of the latch protruding area, and the positioning error of the multi-compartment bins at the robot's material handling station, the outward expansion of the hook-sensitive point set in the latch orientation, the extraction direction, and the lateral direction perpendicular to the latch orientation is determined. For example, when the depth error is 0.8 mm, the bin positioning error is 0.5 mm, and the recognition confidence of the latch protruding area is 0.75, an outward expansion of approximately 1.4 mm to 2.2 mm can be set in the latch orientation, approximately 1.0 mm to 1.8 mm in the extraction direction, and approximately 0.8 mm to 1.5 mm in the lateral direction. Since the latch is most likely to hook along its outward extension direction, the outward expansion in the latch orientation can be greater than the lateral expansion, thus forming anisotropic expansion.

[0067] Subsequently, the set of hook-sensitive points is anisotropically expanded according to the latch orientation, removal direction, and lateral expansion amount to obtain the latch expansion envelope. This latch expansion envelope is not a simple proportional enlargement of the entire connector assembly, but rather a directional expansion around the edge points in the latch protrusion area most prone to hooking, thus more closely resembling the actual risk area of ​​a small latch structure.

[0068] Based on the lateral clearance of the main body area within the target compartment and the grasping and positioning error of the embodied intelligent robot, the allowable attitude deviation range relative to the removal direction when the target connector assembly is removed is determined. For example, if the lateral clearance between the left side of the main body area and the adjacent connector assembly is 4mm, the lateral clearance between the right side and the partition is 3mm, and the gripper positioning error is 0.5mm, the allowable attitude deviation range can be limited to approximately ±2° to ±5° relative to the Z-axis; if the lateral clearance is larger, the allowable attitude deviation range can be widened accordingly.

[0069] Then, within the allowable attitude sway range, multiple height layers are set along the removal direction, and the latch expansion envelope is replicated on each height layer according to the corresponding allowable attitude. The latch expansion envelopes on each height layer are merged to obtain the space area swept by the latch during removal. The spacing between height layers can be 1mm to 5mm. For example, for a connector assembly with a height of approximately 10mm, several height layers can be set from the current position to a height of 20mm to 40mm above the grid rib. This space area swept by the latch during removal represents the space that the latch protrusion area may pass through during the release and removal of the target connector assembly.

[0070] In one possible embodiment, when calculating the projection overlap, a rib blocking band corresponding to the rib side boundary and a box wall blocking band corresponding to the box wall side boundary can be constructed in the grid coordinate system according to the grid boundary. The adjacent object occupancy envelope is then constructed based on the outer contours of adjacent constraint objects and their identification confidence levels. The widths of the rib blocking band and the box wall blocking band may include the projection of the actual rib thickness or box wall thickness inside the grid, as well as a safety margin for covering positioning errors. The adjacent object occupancy envelope can be formed by expanding the point cloud contours of adjacent connector assemblies.

[0071] Then, the space swept by the latch removal, the rib barrier, the box wall barrier, and the envelope occupied by adjacent objects are projected onto a detection plane perpendicular to the removal direction. Within this detection plane, the overlapping areas of the space swept by the latch removal with the rib barrier, the box wall barrier, and the envelope occupied by adjacent objects are determined. Subsequently, the overlapping areas are continuously filtered according to their corresponding height layers, with overlapping areas spanning multiple consecutive height layers considered valid overlapping areas. For example, if an overlap only occurs in a single height layer and its continuous height length is less than 2mm, it may be point cloud noise or edge error; if the overlap spans more than three consecutive height layers and its coverage length in the latch orientation exceeds 1mm, it is more likely to correspond to actual blocking or snagging risks.

[0072] The projected overlap can be calculated based on the coverage length of the effective overlap area in the latch orientation and the continuous height in the removal direction. For example, the coverage length and continuous height can be weighted and summed, or the coverage length, continuous height, and overlap area can be normalized to form the projected overlap. This application does not limit the specific mathematical expression of the projected overlap, as long as it can reflect the degree of overlap between the spatial area swept by the latch removal and the constrained object on the removal path.

[0073] S204. Calculate the releasable displacement in the direction opposite to the latch orientation, and determine the local constraint relationship based on the projected overlap and the releasable displacement.

[0074] The releasable displacement refers to the maximum displacement that the main body area of ​​the target connector assembly can move in the direction opposite to the latch orientation without causing a new collision with the grid boundary or adjacent materials. Since when the connector assembly is stuck, the real issue is whether the latch can disengage from the constrained object through a small reverse displacement, the releasable displacement should primarily be based on the remaining reverse space of the main body area within the target grid. For example, the releasable displacement can be obtained by subtracting the gripper positioning error, the hopper positioning error, and the safety clearance from the minimum distance from the main body area's outward expansion envelope along the release direction to the grid boundary or the envelope occupied by adjacent materials. The calculation method for the releasable displacement can refer to relevant displacement calculation formulas, which will not be elaborated here.

[0075] For each object among the rib side boundary, box wall side boundary, and adjacent constraint objects, the corresponding projected overlap and locking gap can be obtained respectively. When the projected overlap of any object is greater than the preset overlap and the corresponding locking gap is less than the preset locking gap, the object that meets the condition is determined as a constraint object. In other words, the condition is that the projected overlap is greater than the preset overlap and the corresponding locking gap is less than the preset locking gap.

[0076] It should be noted that the locking clearance refers to the minimum available gap between the protruding area of ​​the latch and the side boundary of the rib, the side boundary of the enclosure wall, or adjacent constrained objects along the latch direction. The preset overlap and preset locking clearance can be set according to the connector size, gripper positioning accuracy, and production cycle time. For example, for a connector assembly with a latch protrusion distance of approximately 1.5mm, the preset locking clearance can be between 0.5mm and 1.2mm, and the preset overlap can correspond to an overlap relationship where the latch direction covers a length of more than 1mm and a continuous height of more than 3mm.

[0077] If the side boundaries of the ribs, the side boundaries of the box walls, and the adjacent constraint objects do not meet the conditions that the projected overlap is greater than the preset overlap and the locking gap is less than the preset locking gap, then a local constraint relationship representing the absence of local constraints can be generated. In this case, the robot can determine the main gripping area in the main body region that avoids the protruding area of ​​the latch, and grasp the target connector assembly along the extraction direction.

[0078] Once a constraint is identified, it is further determined whether the release displacement is greater than the release displacement required for the latch protrusion area to cross the constraint. The release displacement can be calculated based on the overlap width between the latch protrusion area and the constraint, the latch protrusion distance, and a safety margin. For example, if the effective overlap width between the latch protrusion area and the rib barrier along the latch direction is 1.6mm, and a safety margin of 0.6mm is set, then the release displacement can be approximately 2.2mm. If there is still 5mm of movable space in the reverse direction within the main area of ​​the target grid, a releaseable local constraint relationship is generated; if the remaining space in the reverse direction is only 1mm, a non-releaseable local constraint relationship is generated.

[0079] When the releasable displacement is greater than the release displacement, the direction opposite to the latch and avoiding the constraint object is determined as the release direction, and a displacement not less than the release displacement and not greater than the releasable displacement is determined as the release displacement. This generates a local constraint relationship that includes the constraint object, the release direction, and the release displacement, indicating that it can be released. When the releasable displacement is not greater than the release displacement, a local constraint relationship that includes the constraint object and a "prohibit direct lifting" marker is generated, indicating that it cannot be directly released. This "prohibit direct lifting" marker can be transmitted to the robot control module, preventing the robot from performing the action of directly grasping the target connector assembly along the extraction direction.

[0080] In a complete numerical example, the multi-compartment bin has 4×6 compartments, with external dimensions of 600mm×400mm×150mm. The target compartment's internal dimensions are 90mm×70mm×70mm, and the rib thickness is 3mm. Twelve identical 6-pin locking connector assemblies are placed within the target compartment. Each connector assembly has a body length of 24mm, a width of 12mm, and a height of 9mm, with a locking protrusion distance of approximately 1.6mm. Based on the compartment material mapping information, the task identifies the 2nd row and 5th column as the target compartment. The robot identifies four candidate connector assemblies within this compartment. One candidate connector assembly has a body exposure level of 0.82, a locking protrusion area identifiability of 0.78, and a clamping accessibility of 0.85, and is therefore identified as the target connector assembly.

[0081] Continuing the example above, the latching orientation of the target connector assembly is the +X direction in the grid coordinate system, and its latching protrusion area is close to the right side boundary of the partition rib. The depth error of the material handling sensing data is taken as 0.8mm, the bin positioning error as 0.5mm, the robot gripping positioning error as 0.5mm, and the latching protrusion area recognition confidence level as 0.76. Accordingly, the outward expansion of the hook-sensitive point set in the latching orientation is taken as 1.8mm, in the extraction direction as 1.2mm, and in the lateral direction as 1.0mm. The allowable attitude sway range is ±3° relative to the extraction direction, the height layer spacing is 3mm, and the sweep height is 30mm. After projection and height layer continuity filtering, the effective overlap between the space swept by the latch extraction and the right side partition rib blocking area has a coverage length of 1.4mm in the latching orientation, a continuous height of 12mm in the extraction direction, and a locking gap of 0.4mm.

[0082] In the example above, if the preset overlap corresponds to a coverage length of not less than 1.0 mm and a continuous height of not less than 6 mm, and the preset locking gap is 0.8 mm, then the right side boundary of the rib is determined as the constraint object. The disengagement displacement required for the latch protruding area to cross this constraint object can be 2.4 mm, and the releaseable displacement of the main body area along the -X direction is 4.5 mm. Therefore, a releaseable local constraint relationship is generated, with the release direction being the -X direction and the release displacement being 3.0 mm. After the robot grips the main body gripping area, it first translates 3.0 mm along the -X direction, and then lifts 35 mm along the extraction direction to above the grid rib, thereby completing the extraction of the target connector assembly. If the same disengagement displacement is 2.4 mm but the releaseable displacement is only 1.2 mm, then a local constraint relationship that cannot be directly released is generated. The robot does not perform a direct lift, but can switch to gripping another candidate connector assembly in the same grid or trigger an exception handling.

[0083] Example 3: The control process in S105 is described below. When the local constraint relationship indicates that the latch protrusion area forms a releasable local constraint, the robot can preferentially select a gripping area in the main body region that is far away from the latch protrusion area. The gripper can be a parallel gripper, a flexible finger gripper, or a force-controlled end effector. The gripping area should avoid the latch protrusion area and the terminal opening to prevent damage to the latch or pin during release movement.

[0084] After gripping the target connector assembly, the robot can first move along the release direction at a low speed, ranging from 20 mm / s to 80 mm / s, with the end effector acceleration controlled between 200 mm / s² and 800 mm / s², to reduce the risk of small connectors jumping or flipping within the slot. After the release movement is complete, the robot can lift the target connector assembly along the removal direction and deliver it to the assembly position, buffer fixture, or transfer positioning stage as needed.

[0085] When the local constraint relationship indicates that the latch protrusion area does not form a local constraint, the robot can directly determine the main gripping area in the main body area to avoid the latch protrusion area and grasp the target connector assembly along the extraction direction. When the local constraint relationship indicates that the latch protrusion area forms a local constraint that cannot be directly released, the robot is restricted from directly grasping the target connector assembly along the extraction direction. It can reselect another candidate connector assembly in the same target slot, or mark the target slot as an abnormal slot that requires secondary sorting, vibration sorting, or manual verification. This application does not require forced grasping when it cannot be directly released, thereby avoiding latch breakage, material flying out, or multiple connectors being carried out due to erroneous actions.

[0086] Continuing with the aforementioned numerical example, if the robot directly lifts the latch in the extraction direction when it is close to the right-side rib and the locking gap is only 0.4mm, the outer edge of the latch will remain close to the rib's blocking strip during the first few millimeters of lifting, posing a risk of jamming or scratching. However, with the control method of this application, the robot first releases 3.0mm in the -X direction, allowing the latch protrusion to move away from the right-side rib's blocking strip before lifting 35mm. This action is not a fixed avoidance action set based on human experience, but is triggered and determined by the locking gap, projected overlap, releaseable displacement, and disengagement displacement, thus adapting to the actual posture of the target connector assembly in the current slot.

[0087] The millimeter-level distance, angle, speed, and confidence values ​​in the above embodiments are used to illustrate a feasible engineering implementation, and their magnitudes match the common precision of small plastic connectors, multi-compartment plastic bins, industrial depth cameras, and small grippers. For connector assemblies of different sizes, the outward expansion, preset overlap, preset locking gap, disengagement displacement, and release displacement can be adjusted according to the latch protrusion distance, compartment gap, gripper positioning error, and production cycle. As long as the control result of releasing before gripping or prohibiting direct lifting is still based on the local constraint relationship of the latch protrusion area, it falls within the scope of this application.

[0088] Example 4: Figure 5 This is a schematic diagram of the cargo recognition and grasping system of a fully automated production embodied intelligent robot disclosed in this application, with reference to... Figure 5 The system may include: The first determining module 51 is used to determine the target compartment based on the target goods information corresponding to the current production task, the positioning information of the multi-compartment bin at the robot picking station, and the compartment material mapping information of the multi-compartment bin. The identification module 52 is used to collect material picking perception data in the target compartment, identify the target connector component to be picked up this time based on the material picking perception data, and identify the main body area, the latch protrusion area, the compartment boundary of the target compartment, and the adjacent constraint objects located around the target connector component. The second determining module 53 is used to determine the latch orientation of the target connector assembly based on the relative posture between the main body area and the latch protrusion area. The judgment module 54 is used to determine whether the protruding area of ​​the latch on the target connector assembly forms a local constraint with the grid boundary or the adjacent constraint object based on the latch orientation, the grid boundary and the adjacent constraint object, so as to generate a local constraint relationship. The control module 55 is used to determine the release direction and release displacement according to the local constraint relationship when the local constraint relationship indicates that the latch protrusion area forms a releasable local constraint, and to control the embodied intelligent robot to move the target connector assembly along the release direction so that the latch protrusion area is detached from the corresponding constraint object, and then grasp the target connector assembly.

[0089] The cargo identification and grasping system of the fully automated production embodied intelligent robot in this application embodiment is used to implement the aforementioned cargo identification and grasping method of the fully automated production embodied intelligent robot. Therefore, the specific implementation of the cargo identification and grasping system of the fully automated production embodied intelligent robot can be seen in the embodiment section of the cargo identification and grasping method of the fully automated production embodied intelligent robot mentioned above. The specific implementation can be referred to the description of the corresponding embodiments, and will not be repeated here.

[0090] This application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the cargo identification and grasping method of any of the above-described fully automated production embodied intelligent robots.

[0091] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the cargo identification and grasping method of any of the above-described fully automated production embodied intelligent robots.

[0092] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory, random access memory, portable hard drives, magnetic disks, or optical disks.

[0093] This application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps in any of the above embodiments of the fully automated production embodied intelligent robot's cargo identification and grasping method.

[0094] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0095] The foregoing has provided a detailed description of the cargo identification and grasping method and system for a fully automated production embodied intelligent robot provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for cargo identification and grasping using a fully automated production-integrated intelligent robot, characterized in that, include: Based on the target cargo information corresponding to the current production task, the positioning information of the multi-compartment bin at the robot material handling station, and the material mapping information of the multi-compartment bin, the target compartment is determined; Collect material handling perception data within the target compartment, identify the target connector component to be grasped based on the material handling perception data, and identify the main body area, latch protrusion area, compartment boundary of the target compartment, and adjacent constraint objects located around the target connector component. The latch orientation of the target connector assembly is determined based on the relative posture between the main body area and the latch protruding area. Based on the latch orientation, the grid boundary, and the adjacent constraint object, determine whether the latch protrusion area on the target connector assembly forms a local constraint with the grid boundary or the adjacent constraint object, so as to generate a local constraint relationship; When the local constraint relationship indicates that the latch protrusion area forms a releasable local constraint, the release direction and release displacement are determined according to the local constraint relationship, and the embodied intelligent robot is controlled to move the target connector assembly along the release direction so that the latch protrusion area is detached from the corresponding constraint object, and then the target connector assembly is grasped.

2. The method according to claim 1, characterized in that, The step of determining whether the protruding area of ​​the latch on the target connector assembly forms a local constraint with the grid boundary or the adjacent constraint object based on the latch orientation, the grid boundary, and the adjacent constraint object, in order to generate a local constraint relationship, includes: Establish a grid coordinate system based on the grid boundary of the target grid, determine the rib side boundary and the extraction direction inside the grid in the grid coordinate system, and determine the box wall side boundary when the target grid is adjacent to the outer periphery of the material box. Transform the main body area of ​​the target connector assembly, the latch protrusion area on the target connector assembly, and the adjacent constraint object into the grid coordinate system; After the conversion is completed, the protruding area of ​​the latch is swept along the extraction direction to obtain the space area swept by the latch extraction, and the projection overlap between the space area and the side boundary of the rib, the side boundary of the box wall, and the adjacent constraint object is calculated respectively. Calculate the releasable displacement in the direction opposite to the direction of the latch, and determine the local constraint relationship based on the projected overlap and the releasable displacement.

3. The method according to claim 2, characterized in that, The sweeping of the protruding area of ​​the latch along the removal direction to obtain the space area swept during latch removal includes: In the grid coordinate system, determine the set of hook-sensitive points within the buckle protrusion area; Based on the depth error of the material picking perception data, the recognition confidence of the buckle protrusion area, and the positioning error of the multi-compartment bin at the robot picking station, the outward expansion of the hook sensitive point set in the buckle orientation, the picking direction, and the lateral direction perpendicular to the buckle orientation is determined respectively. According to the buckle orientation, the extraction direction, and the lateral expansion amount, the hook sensitive point set is anisotropically expanded to obtain the buckle expansion envelope; Based on the lateral clearance of the main body region within the target slot and the grasping and positioning error of the embodied intelligent robot, the allowable attitude deviation range relative to the removal direction when the target connector assembly is removed is determined; Within the allowable attitude sway range, multiple height layers are set along the extraction direction, and the latch outer envelope is copied on each height layer according to the corresponding allowable attitude. The latch outer envelopes on each height layer are merged to obtain the spatial region.

4. The method according to claim 3, characterized in that, Calculate the projection overlap between the spatial region and the side boundary of the rib, the side boundary of the box wall, and the adjacent constraint objects, including: Based on the grid boundary, construct the rib blocking zone corresponding to the rib side boundary and the box wall blocking zone corresponding to the box wall side boundary in the grid coordinate system, and construct the adjacent object occupancy envelope based on the outer contour of the adjacent constraint object and the recognition confidence of the adjacent constraint object. The spatial region, the partition strip, the box wall barrier, and the adjacent object's occupancy envelope are projected onto a detection plane perpendicular to the extraction direction, and the overlapping areas of the spatial region with the partition strip, the box wall barrier, and the adjacent object's occupancy envelope are determined in the detection plane. The overlapping areas are continuously filtered according to the corresponding height layers. Overlapping areas that span multiple consecutive height layers are taken as effective overlapping areas. The corresponding projected overlap amount is calculated based on the coverage length of the effective overlapping areas in the latch direction and the continuous height in the extraction direction.

5. The method according to claim 2, characterized in that, The calculation of the releasable displacement along a direction opposite to the latch orientation, and the determination of local constraint relationships based on the projected overlap and the releasable displacement, include: Calculate the releasable displacement of the main body region within the target compartment in a direction opposite to the direction of the latch; For each of the rib side boundary, the box wall side boundary, and the adjacent constraint objects, obtain the corresponding projection overlap and locking gap respectively; When the projected overlap of any object is greater than a preset overlap and the corresponding locking gap is less than a preset locking gap, the object that meets the conditions is determined as a constraint object, and a local constraint relationship is generated based on the releasable displacement and the constraint object; or, when the side boundary of the rib, the side boundary of the box wall, and the adjacent constraint object do not meet the conditions that the projected overlap is greater than a preset overlap and the locking gap is less than a preset locking gap, a local constraint relationship indicating that no local constraint has been formed is generated.

6. The method according to claim 5, characterized in that, The generation of local constraint relationships based on the releasable displacement and the constraint object includes: Determine whether the releasable displacement is greater than the release displacement required for the latch protrusion area to cross the constrained object; If the releasable displacement is greater than the release displacement, then the direction opposite to the direction of the latch and avoiding the constrained object is determined as the release direction, and the displacement that is not less than the release displacement and not greater than the releasable displacement is determined as the release displacement, and a local constraint relationship containing the constrained object, the release direction and the release displacement is generated, indicating that it is releasable. Alternatively, if the releasable displacement is not greater than the release displacement, a local constraint relationship is generated that includes the constraint object and a prohibition on direct lifting marker, indicating that it cannot be directly released.

7. The method according to claim 6, characterized in that, After generating local constraint relations that cannot be directly released, the method further includes: The embodied intelligent robot is restricted from directly grasping the target connector assembly along the extraction direction.

8. The method according to claim 1, characterized in that, Determining the latch orientation of the target connector assembly based on the relative posture between the main body region and the latch protrusion region includes: Determine the connection position between the latch protrusion area and the main body area, and the outward extension direction of the latch protrusion area relative to the main body area; Based on the connection position and the outward direction, an initial orientation of the latch is generated, and the initial orientation of the latch is verified based on at least one of the angle between the latch protruding area and the main body area, the protrusion distance, and the height relationship. When the verification is successful, the initial orientation of the latch is determined as the latch orientation of the target connector assembly.

9. The method according to claim 1, characterized in that, The step of identifying the target connector component to be grasped based on the material sensing data includes: Based on the material handling sensing data, multiple candidate connector components are identified from within the target slot; Based on the degree of exposure of the main body, the identifiability of the latch protrusion area, and the reachability of the clamping, one of several candidate connector components is selected as the target connector component to be grasped in this operation.

10. A cargo identification and grasping system for a fully automated production-integrated intelligent robot, characterized in that, include: The first determining module is used to determine the target compartment based on the target goods information corresponding to the current production task, the positioning information of the multi-compartment bin at the robot picking station, and the compartment material mapping information of the multi-compartment bin. The identification module is used to collect material picking perception data in the target compartment, identify the target connector component to be picked up this time based on the material picking perception data, and identify the main body area, the latch protrusion area, the compartment boundary of the target compartment, and the adjacent constraint objects located around the target connector component. The second determining module is used to determine the latch orientation of the target connector assembly based on the relative posture between the main body area and the latch protrusion area. The judgment module is used to determine, based on the buckle orientation, the grid boundary, and the adjacent constraint object, whether the buckle protrusion area on the target connector assembly forms a local constraint with the grid boundary or the adjacent constraint object, so as to generate a local constraint relationship. The control module is used to determine the release direction and release displacement according to the local constraint relationship when the local constraint relationship indicates that the latch protrusion area forms a releasable local constraint, and to control the embodied intelligent robot to move the target connector assembly along the release direction so that the latch protrusion area is detached from the corresponding constraint object, and then grasp the target connector assembly.

Images