Intelligent dynamic replenishment method and device based on machine vision

Abstract

The present application relates to the technical field of intelligent manufacturing automation equipment, in particular to a kind of intelligent dynamic replenishment method and device based on machine vision, method includes: obtaining the carrier image of material carrier moving with conveying mechanism at the time of photographing, determine the moving carrier coordinate system at the time of photographing;With the preset standard accommodation site model and carrier image are registered, identify empty accommodation site and obtain empty accommodation site standard position;Select natural reference feature, construct local error field according to the offset between the actual position and standard reference position of natural reference feature, determine local correction and correct to obtain empty accommodation site reconstruction coordinates;According to carrier motion information, photographing time, the expected replenishment time of actuating mechanism and the coordinate system calibration relationship, obtain the replenishment time coordinates, control actuating mechanism to empty accommodation site and replenish material.The present application can dynamically determine the replenishment opportunity and replenishment amount according to the material state, improve the accuracy and real-time of replenishment.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing automation equipment technology, and in particular to an intelligent dynamic feeding method and device based on machine vision. Background Technology

[0002] In scenarios such as intelligent manufacturing, automated assembly, packaging, sorting, and logistics turnover, materials typically need to be placed into pallets, jigs, trays, turnover boxes, packaging carriers, or other material carriers with multiple storage positions. To improve production efficiency, the material carrier usually moves continuously or intermittently with the conveyor mechanism. The vision system collects images of the material carrier during its movement and identifies whether each storage position is already filled with material based on the images. This determines the vacant storage positions that need replenishment, and then controls robots, robotic arms, suction mechanisms, gripping mechanisms, pushing mechanisms, or other actuators to replenish the corresponding vacant storage positions with material. This dynamic replenishment method can be applied to replenishment operations for materials such as 3C electronic components, connectors, display panel accessories, food units, pharmaceuticals, packaging parts, and logistics sorting parts.

[0003] In existing technologies, the positions of vacant ...

[0004] However, in actual dynamic feeding processes, the material carrier may experience slight posture deviations, local deformations, errors in the processing or assembly of the receiving positions, image acquisition errors, lateral offsets during the conveying process, and uneven distribution of overall registration errors within local areas. In such cases, relying solely on a preset standard receiving position model or the theoretical receiving position center obtained from overall image registration results is insufficient to fully reflect the actual positional deviations near vacant receiving positions. Especially when there is no directly identifiable material entity within the vacant receiving position, the feeding coordinates of the actuator are prone to deviating from the actual vacant receiving position center, thus affecting the accuracy, stability, and reliability of dynamic feeding. Therefore, it is necessary to provide an intelligent dynamic feeding scheme that can improve the positioning accuracy of vacant receiving positions and the reliability of feeding coordinates during movement. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide an intelligent dynamic feeding method and device based on machine vision, so as to at least solve the problem that in the existing dynamic feeding process, when the position of the vacant ...

[0006] To address the aforementioned technical problems, this invention provides an intelligent dynamic feeding method based on machine vision, comprising: The image of the material carrier moving with the conveying mechanism at the moment of taking the picture is obtained, wherein the material carrier has multiple receiving positions for accommodating materials; Determine the target coordinate system, which is the coordinate system of the moving carrier at the time of the photograph; The preset standard accommodating position model is registered with the carrier image, and the theoretical accommodating position area and theoretical accommodating position center position of each accommodating position in the carrier image are determined based on the registration result. The presence or absence of material is identified within each theoretically defined storage area, and storage areas where no material is identified are designated as vacant storage areas. Based on the transformation relationship between the image coordinate system and the target coordinate system, the center position of the theoretical accommodating position corresponding to the vacant accommodating position is transformed to the target coordinate system to obtain the standard position of the vacant accommodating position; Within a preset local area containing the vacant accommodating space, multiple natural reference features that are identifiable in the image of the carrier and have standard reference positions are selected; wherein, the preset local area is a preset distance range centered on the vacant accommodating space, an adjacent accommodating space area containing the vacant accommodating space, or a local accommodating space area determined according to the block division rules of the preset standard accommodating space model; Identify the image position of each of the natural reference features in the image of the carrier, and convert the image position of each of the natural reference features into the actual position in the target coordinate system according to the transformation relationship between the image coordinate system and the target coordinate system; Obtain the standard reference position of each of the natural reference features in the target coordinate system; Based on the offset of the actual position of each natural reference feature relative to its standard reference position, multiple local deviation vectors are determined; Using the standard reference position of each of the natural reference features in the target coordinate system as the input position, the corresponding local deviation vector as the output deviation, and the weights determined according to the recognition confidence of each of the natural reference features and the distance between each of the natural reference features and the vacant accommodating position, a local error field is constructed to characterize the mapping relationship between the position and the deviation vector in the preset local area. The local correction amount at the standard position of the vacant accommodating position is determined based on the local error field. The standard position of the vacant accommodating position is corrected according to the local correction amount to obtain the reconstructed coordinates of the vacant accommodating position in the target coordinate system; Based on the carrier motion information, the time of taking the picture, the expected replenishment time of the actuator, and the calibration relationship between the target coordinate system and the actuator coordinate system, the reconstructed coordinates of the vacant accommodating position are converted into the replenishment time coordinates in the actuator coordinate system; The actuator is controlled to replenish materials into the empty receiving position according to the replenishment time coordinate.

[0007] The present invention also provides an intelligent dynamic feeding device based on machine vision, comprising: A conveying mechanism for conveying a material carrier with multiple receiving positions; A material supply mechanism, used to provide materials that need to be replenished; A feeding vision component is used to capture an image of the material carrier moving with the conveying mechanism at the moment of taking the picture. Motion information acquisition component, used to acquire motion information of the carrier; An actuator is used to take out or receive the material to be replenished from the feeding mechanism and replenish the material to be replenished into the empty accommodating position of the material carrier. The controller is communicatively connected to the feeding vision component, the motion information acquisition component, and the actuator, respectively. The controller stores a preset standard accommodating position model, the transformation relationship between the image coordinate system and the target coordinate system, and the calibration relationship between the target coordinate system and the actuator coordinate system. The controller is configured to: The image of the carrier is acquired by the feeding vision component; The target coordinate system is defined as the coordinate system of the moving carrier at the moment of taking the picture; The preset standard accommodating position model is registered with the carrier image, and the theoretical accommodating position area and theoretical accommodating position center position of each accommodating position in the carrier image are determined based on the registration result. The presence or absence of material is identified within each theoretically defined storage area, and storage areas where no material is identified are designated as vacant storage areas. Based on the transformation relationship between the image coordinate system and the target coordinate system, the center position of the theoretical accommodating position corresponding to the vacant accommodating position is transformed to the target coordinate system to obtain the standard position of the vacant accommodating position; Within a preset local area containing the vacant accommodating space, multiple natural reference features that are identifiable in the image of the carrier and have standard reference positions are selected; wherein, the preset local area is a preset distance range centered on the vacant accommodating space, an adjacent accommodating space area containing the vacant accommodating space, or a local accommodating space area determined according to the block division rules of the preset standard accommodating space model; Identify the image position of each of the natural reference features in the image of the carrier, and convert the image position of each of the natural reference features into the actual position in the target coordinate system according to the transformation relationship between the image coordinate system and the target coordinate system; Obtain the standard reference position of each of the natural reference features in the target coordinate system; Based on the offset of the actual position of each natural reference feature relative to its standard reference position, multiple local deviation vectors are determined; Using the standard reference position of each of the natural reference features in the target coordinate system as the input position, the corresponding local deviation vector as the output deviation, and the weights determined according to the recognition confidence of each of the natural reference features and the distance between each of the natural reference features and the vacant accommodating position, a local error field is constructed to characterize the mapping relationship between the position and the deviation vector in the preset local area. The local correction amount at the standard position of the vacant accommodating position is determined based on the local error field. The standard position of the vacant accommodating position is corrected according to the local correction amount to obtain the reconstructed coordinates of the vacant accommodating position in the target coordinate system; Based on the carrier motion information, the time of taking the picture, the expected replenishment time of the actuator, and the calibration relationship between the target coordinate system and the actuator coordinate system, the reconstructed coordinates of the vacant accommodating position are converted into the replenishment time coordinates in the actuator coordinate system; The actuator is controlled to replenish materials into the empty receiving position according to the replenishment time coordinate.

[0008] The present invention has the following beneficial effects: (1) By registering the preset standard accommodating position model with the image of the carrier and identifying the presence or absence of material in the theoretical accommodating position area, the present invention can determine the empty accommodating position and its standard position in the moving material carrier, providing a unified position reference for subsequent empty accommodating position coordinate reconstruction and actuator feeding control. (2) The present invention selects multiple natural reference features with standard reference positions within a preset local area containing the vacant accommodating position, and determines the local deviation vector based on the offset of the actual position of each natural reference feature relative to the standard reference position. This allows the local reference information in the neighborhood of the vacant accommodating position to characterize the position deviation near the vacant accommodating position, thereby avoiding the need to determine the position of the vacant accommodating position solely based on the overall registration result. (3) The present invention uses the standard reference position of the natural reference feature as the input position and the corresponding local deviation vector as the output deviation, and combines the recognition confidence of the natural reference feature and the distance between it and the vacant accommodating position to determine the weight to construct a local error field. This enables the reference feature that is reliable in recognition and close to the vacant accommodating position to have a more reasonable influence on the local correction result, thereby improving the stability of the vacant accommodating position coordinate correction result. (4) The present invention determines the local correction amount at the standard position of the vacant accommodating position based on the local error field, and corrects the standard position of the vacant accommodating position to obtain the reconstructed coordinates of the vacant accommodating position. Then, it combines the carrier motion information, the time of taking pictures, the expected feeding time of the actuator, and the coordinate system calibration relationship to generate the feeding time coordinates in the coordinate system of the actuator. This can reduce the overall registration error, local position deviation, and the influence of conveying motion on the feeding coordinates, and improve the matching between the feeding position of the actuator and the actual position of the vacant accommodating position during the dynamic feeding process of the moving material carrier. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a schematic diagram of the structure of the intelligent dynamic feeding device based on machine vision provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a material carrier, a receiving position, an empty receiving position, and a preset local area provided in an embodiment of the present invention. Figure 3 A flowchart of the method for determining the reconstructed coordinates of an empty accommodating position provided in an embodiment of the present invention; Figure 4 This is a flowchart of the method for determining the timing coordinates of material replenishment and controlling material replenishment provided in an embodiment of the present invention.

[0011] Explanation of reference numerals in the attached figures: 10. Conveying mechanism; 20. Feeding mechanism; 30. Material replenishment vision component; 40. Motion information acquisition component; 50. Actuator; 60. Controller; 101. Material carrier; 102. Accommodation position; 103. Empty accommodation position; 104. Theoretical accommodation area; 105. Theoretical accommodation center position; 106. Preset local area; 107. Placed material; 125. Carrier movement direction. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments of the present invention, and not all of them. Other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are all within the protection scope of the present invention.

[0013] In the description of this invention, unless otherwise explicitly stated, the orientations or positional relationships indicated by terms such as "lateral," "longitudinal," "upper," "lower," "left," and "right" are based on the orientations or positional relationships shown in the accompanying drawings. Specifically, when referring to the material carrier 101, the conveying direction and the arrangement direction of the accommodating positions of the material carrier 101 are used as references; when referring to visual positioning and execution control, the target coordinate system and the actuator coordinate system are used as references, respectively. The above terms are used only for the convenience of describing and simplifying the invention and do not constitute a limitation on the scope of protection of this invention. The term "connection" should be interpreted broadly, and can refer to a direct connection or an indirect connection through intermediate components; the term "communication connection" can refer to a wired communication connection or a wireless communication connection, or can be indirectly established through a communication module or communication network. Unless otherwise stated, "multiple" refers to two or more.

[0014] Figure 1 This is a schematic diagram of the structure of the intelligent dynamic feeding device based on machine vision provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of a material carrier, a receiving position, an empty receiving position, and a preset local area provided in an embodiment of the present invention. Figure 3 A flowchart of the method for determining the reconstructed coordinates of an empty accommodating position provided in an embodiment of the present invention; Figure 4 This is a flowchart of the method for determining the timing coordinates of material replenishment and controlling material replenishment provided in an embodiment of the present invention.

[0015] In this invention, the material carrier 101 can be a pallet, jig, tray, turnover box, packaging carrier, connector carrier plate, food carrier box, sorting rack, blister pack, blister tray, medicine carrier plate, medicine box, or other carrier structures with multiple functions for accommodating, positioning, or supporting materials. The accommodating position 102 can be a groove, accommodating slot, limiting slot, supporting slot, accommodating hole, cavity, grid, blister cavity, or other structure capable of accommodating, positioning, or supporting materials. The materials can be 3C electronic components, connectors, screws, structural parts, display panel accessories, gaskets, protective sheets, food units, medicines, packaging, logistics sorting parts, or other objects that can be picked up, placed, pushed, transferred, or replenished by the actuator. The conveying mechanism 10 can include a conveyor line, belt line, chain plate line, synchronous belt, roller line, or other conveying structure capable of driving the material carrier 101 to move. The material carrier 101 moves with the conveying mechanism 10, which can be understood as the material carrier 101 moving with the conveyor line or equivalent conveying structure in the conveying mechanism 10.

[0016] In practical applications, the intelligent dynamic material replenishment method and device based on machine vision of this invention can be applied to various dynamic material replenishment or dynamic delivery scenarios. For example, in the 3C electronics assembly scenario, the material carrier 101 can be an electronic component tray, jig, or material tray, and the material can be a connector, screw, structural component, chip auxiliary material, or other electronic component; in the display panel manufacturing or assembly scenario, the material carrier 101 can be a panel carrier or auxiliary material carrying jig, and the material can be a gasket, protective sheet, label, connector, or other panel auxiliary material; in the food packaging scenario, the material carrier 101 can be a food tray, packaging box, or turnover carrier, and the material can be a food unit to be packaged, packaging accessory, or dispensing unit; in the pharmaceutical packaging scenario, the material carrier 101 can be a medicine box, blister pack, blister tray, or pharmaceutical carrier plate, and the material can be tablets, capsules, pills, soft capsules, or other pharmaceuticals; in the logistics sorting or packaging turnover scenario, the material carrier 101 can be a turnover box, sorting rack, packaging carrier, or sorting tray, and the material can be a package, sorting piece, label, or item to be transferred.

[0017] The shape, size, material, color, and arrangement of the material carrier 101, the receiving position 102, and the material in the above-mentioned different application scenarios may be different. However, as long as it is possible to identify the empty receiving position 103 through the image of the carrier and to construct a local error field 109 based on the natural reference features near the empty receiving position 103 in order to correct the standard position of the empty receiving position and generate the material replenishment time coordinates, the technical solution of the present invention can be applied.

[0018] In this invention, the carrier image refers to an image captured by the replenishment vision component 30 at the time of image capture, containing at least a portion of the accommodating area of ​​the material carrier 101. The replenishment vision component 30 may include an industrial camera, an area scan camera, a line scan camera, a depth camera, a lens, a light source, an image acquisition card, a vision processing unit, or a combination of the above components. The time of image capture may be determined by a camera trigger signal, a conveyor encoder signal, a programmable logic controller 60 trigger signal, an internal clock of the controller 60, or a combination of the above signals. The image coordinate system is the pixel coordinate system where the carrier image is located. The model coordinate system is the coordinate system where the preset standard accommodating position model is located. The target coordinate system is the moving carrier coordinate system at the time of image capture, used to represent the positional relationship between the material carrier 101 body, accommodating position 102, vacant accommodating position 103, natural reference features, local deviation vector, and reconstructed coordinates of vacant accommodating positions at the time of image capture. The actuator coordinate system is the coordinate system used by the actuator 50 when performing material picking, receiving, transferring, or replenishing actions. The transformation relationships between the model coordinate system, image coordinate system, target coordinate system, and actuator coordinate system can be determined through calibration, image registration, positioning feature matching, coordinate transformation matrix, or kinematic relationships.

[0019] In one embodiment, the origin of the target coordinate system is set at the center of the first row and first column of the material carrier 101 at the moment of taking the picture. The X-axis of the target coordinate system is set along the row direction of the material carrier 101, and the Y-axis is set along the column direction of the material carrier 101. In other embodiments, the origin of the target coordinate system may also be set at a corner of the material carrier 101, the center of the material carrier 101, the center of the positioning hole, the intersection of the positioning lines, or other reference points in the preset standard accommodating position model. When the material carrier 101 has a rotation angle on the conveying mechanism 10, the controller 60 can determine the posture of the material carrier 101 at the moment of taking the picture based on the edge of the material carrier 101, the positioning hole, the positioning line, the corner of the material carrier 101, or the intersection of the accommodating position array, and make the target coordinate system rotate synchronously with the posture of the material carrier 101.

[0020] In this invention, an empty accommodating position 103 refers to an accommodating position that should theoretically accommodate material but is not identified in the current carrier image. A natural reference feature refers to a feature in the carrier image that is identifiable and has a standard reference position. A local deviation vector refers to the offset of the actual position of a natural reference feature relative to its standard reference position. A local error field 109 refers to the mapping relationship between position and deviation vector within a preset local region 106. In this embodiment, the local error field 109 is not a pre-fixed global camera error compensation field, but rather is constructed during the current feeding process of the material carrier 101 moving with the conveying mechanism 10, specifically for the preset local region 106 where the empty accommodating position 103 to be replenished is located, based on the actual position and standard reference position of the identifiable natural reference feature in the current carrier image. The local error field 109 is used to characterize the local positional offset state of the current material carrier 101 near the empty accommodating position 103. Even when the camera has completed global calibration or the image has undergone distortion correction, the controller 60 can still construct a local error field 109 based on the natural reference features in the current carrier image to compensate for the positioning error of the empty carrier 103 caused by local deformation, local offset, local manufacturing error of the accommodating position array, lateral micro-offset during transportation, or model registration residual.

[0021] Please see Figures 1 to 4This embodiment provides an intelligent dynamic material replenishment method based on machine vision. This method can be executed by a combination of a controller 60, an industrial computer, a vision controller, an actuator controller, a programmable logic controller, and a vision processing unit, or other devices with image processing and motion control capabilities. The material carrier 101 moves with the conveying mechanism 10, and the replenishment vision component 30 acquires an image of the carrier at the moment of image capture. For empty receiving positions 103 without material, since there is no directly identifiable material entity within these positions, determining the coordinates for material replenishment by the actuator solely based on the image edges of the empty receiving position 103 itself or the registration results of a preset standard receiving position model with the carrier image is easily affected by local deformation of the material carrier 101, reflections at the receiving position edges, camera field-of-view edge errors, slight lateral offsets of the material carrier 101, and local deviations between the preset standard receiving position model and the current material carrier 101. To this end, the present invention selects multiple currently identifiable natural reference features with standard reference positions within a preset local region 106 containing the vacant accommodating position 103. Based on the offset of the actual positions of these natural reference features relative to the standard reference positions, multiple local deviation vectors are determined, and a local error field 109 is constructed to characterize the local position offset state near the vacant accommodating position 103. The controller 60 uses this local error field 109 to correct the standard position of the vacant accommodating position, obtaining the reconstructed coordinates of the vacant accommodating position. Then, it combines the carrier motion information, the photographing time, and the expected replenishment time of the actuator to generate the replenishment time coordinates in the actuator coordinate system.

[0022] Specifically, the controller 60 first acquires an image of the material carrier 101, which moves with the conveyor mechanism 10, at the moment of image capture. The material carrier 101 has multiple receiving positions 102 for receiving materials. The carrier image can be a full-page image of the material carrier 101 or a partial image of a region of the material carrier 101. When a material carrier 101 is captured by multiple cameras in sections, the images captured by the multiple cameras can be used as different sub-images of the carrier image. The carrier image may include receiving position areas, material areas, edges of the material carrier 101, edges of receiving grooves, edges of limiting grooves, edges of supporting grooves, positioning holes, positioning lines, marking features, and other image features that can be used for positioning or identification.

[0023] The controller 60 determines the target coordinate system, which is the coordinate system of the moving carrier at the moment of image capture. The controller 60 can identify at least two positioning points, at least one positioning line and one positioning point, or at least three non-collinear positioning points in the carrier image, and match them with corresponding standard points in a preset standard accommodating position model, thereby determining the transformation relationship from the image coordinate system to the target coordinate system. The transformation relationship from the image coordinate system to the target coordinate system is denoted as T. img_carThe transformation relationship can be an affine transformation matrix, a perspective transformation matrix, a similarity transformation matrix, or a partitioning mapping relationship.

[0024] For example, the pixel coordinates in the image coordinate system are (u, v), and the coordinates in the target coordinate system are (X, v). c Y c ), can then be represented as: [X c Y c ,1]T=T img_car ·[u, v, 1]T. When T img_car When the perspective transformation matrix is ​​obtained, the controller 60 normalizes the results of the next coordinate transformation to obtain the two-dimensional coordinates in the target coordinate system.

[0025] The controller 60 registers the preset standard accommodating position model with the carrier image, and determines the theoretical accommodating position region 104 and the theoretical accommodating position center position 105 of each accommodating position in the carrier image based on the registration result. The preset standard accommodating position model is used to describe the accommodating position layout of the material carrier 101 in a standard state, and may include at least one of the following: material carrier 101 specifications, number of accommodating position rows, number of accommodating position columns, accommodating position spacing, standard center position of each accommodating position, standard contour of each accommodating position, standard edge of the accommodating groove, standard edge of the limiting groove, standard edge of the supporting groove, standard position of the edge of the material carrier 101, standard position of the positioning hole, standard position of the positioning line, or standard position of the intersection of the accommodating position array. The preset standard accommodating position model can be stored in the form of a table, array, parameter file, database record, or JSON format data. For each accommodating position, the preset standard accommodating position model can record the accommodating position number, standard center coordinates of the accommodating position, standard contour parameters of the accommodating position, the local region number to which it belongs, and the relationship between adjacent accommodating positions. For each standard reference feature, the preset standard accommodating bit model or its associated standard reference information can record the reference feature type, standard reference location, accommodating bit number, and priority of participation in error field construction.

[0026] In one embodiment, the controller 60 identifies the carrier positioning features from the carrier image and determines the model-image coordinate transformation relationship between the preset standard accommodating position model and the carrier image based on the carrier positioning features. The carrier positioning features may include at least one of the following: the edge of the material carrier 101, corner points of the material carrier 101, positioning holes, positioning lines, intersection points of the accommodating position array, boundary of the accommodating position array, outer contour of the material carrier 101, coded markers, color block markers, or graphic markers. The model-image coordinate transformation relationship may be at least one of the following: translation transformation, rotation transformation, scaling transformation, similarity transformation, affine transformation, perspective transformation, or partition mapping relationship. Based on this model-image coordinate transformation relationship, the controller 60 maps the standard layout information of multiple accommodating positions in the preset standard accommodating position model to the carrier image, obtaining the theoretical accommodating position region 104 and the theoretical accommodating position center position 105 of each accommodating position in the carrier image. It should be noted that the registration of the preset standard accommodating position model and the carrier image is used to obtain the basic standard position of the vacant accommodating position 103 and the standard reference position of the natural reference feature. The registration result is not directly used as the feeding time coordinate in the actuator coordinate system.

[0027] The controller 60 identifies the presence or absence of material in each theoretical accommodating area 104, and designates accommodating areas where no material is detected as vacant accommodating areas 103. Material presence or absence can be identified using contour recognition, color recognition, grayscale recognition, edge recognition, template matching, target detection model recognition, or a combination of these methods. For example, if a candidate contour exists within a theoretical accommodating area 104 that meets a preset area condition and a material shape condition, it is determined that material exists within that theoretical accommodating area 104; otherwise, it is determined that no material is detected in that theoretical accommodating area 104. For accommodating areas where no material is detected, the controller 60 records the accommodating area number, the theoretical accommodating area 104, and the center position 105 of the theoretical accommodating area 103.

[0028] The controller 60 transforms the theoretical accommodating position center position 105 corresponding to the vacant accommodating position 103 to the target coordinate system according to the transformation relationship between the image coordinate system and the target coordinate system, thus obtaining the standard position of the vacant accommodating position. If the theoretical accommodating position center position 105 corresponding to the vacant accommodating position 103 is represented as P in the image coordinate system... img0 Then the standard position P0 of the vacant storage space can be expressed as: P0 = T img_car (P) img0 ), where T img_car (·) indicates the transformation relationship of converting the position in the image coordinate system to the position in the target coordinate system. The standard position of the empty accommodating position represents the theoretical accommodating position center position 105 of the empty accommodating position 103 obtained according to the preset standard accommodating position model and the current carrier image registration result, which is located in the moving carrier coordinate system at the time of shooting.

[0029] Within a preset local area 106 containing the vacant receiving position 103, the controller 60 selects multiple natural reference features that are identifiable in the carrier image and have standard reference positions. The preset local area 106 is a preset distance range centered on the vacant receiving position 103, an adjacent receiving position area containing the vacant receiving position 103, or a local receiving position area determined according to the block division rules of a preset standard receiving position model. The natural reference features should meet two conditions: first, they should be identifiable in the carrier image; second, they should have an obtainable standard reference position. The natural reference features can be at least one of the following: carrier structure type natural reference features, marker type natural reference features, or material center type natural reference features. Carrier structure type natural reference features can include at least one of the following: receiving position contour center, receiving groove edge center, limiting groove edge center, support groove edge center, cavity array intersection, material carrier 101 edge midpoint, material carrier 101 edge endpoint, material carrier 101 corner point, positioning hole center, positioning line midpoint, or positioning line intersection. Natural reference features of the marking type may include at least one of the following: text markings, color block markings, graphic markings, coded markings, QR code markings, barcode markings, or silkscreen markings. Natural reference features of the center type of placed material may include at least one of the following: the outline center of placed material 107, the center of the circumscribed circle of placed material 107, the center of the circumscribed rectangle of placed material 107, the grayscale centroid of placed material 107, the centroid of the color area of ​​placed material 107, or the geometric center of placed material 107.

[0030] When selecting multiple natural reference features with standard reference positions, the controller 60 can select them according to the priority of the natural reference feature type. Among them, the priority of natural reference features of the carrier structure type is higher than that of natural reference features of the marker type, and the priority of natural reference features of the marker type is higher than that of natural reference features of the center of the placed material type. When the placed material 107 has edge-to-edge, tilt, overlap, offset, abnormal posture, incomplete outline or occlusion, or when the deviation of the natural reference feature of the center of the placed material type in the target coordinate system relative to the theoretical center position 105, the center position of the center ...

[0031] The controller 60 identifies the image position of each natural reference feature in the image of the carrier, and converts the image position of each natural reference feature into its actual position in the target coordinate system according to the transformation relationship between the image coordinate system and the target coordinate system. If the image position of the k-th natural reference feature in the image coordinate system is Q... img,k Then its actual position Q in the target coordinate system act,k It can be represented as: Q act,k =T img_car (Q) img,k The controller 60 also acquires the standard reference position of each natural reference feature in the target coordinate system. The standard reference position of the natural reference feature can be derived from a preset standard accommodating position model or from standard reference information associated with the preset standard accommodating position model. When the corresponding standard position of the natural reference feature is located in the model coordinate system of the preset standard accommodating position model, the controller 60 maps the corresponding standard position to the image coordinate system according to the model-image coordinate transformation relationship, and transforms it to the target coordinate system according to the transformation relationship between the image coordinate system and the target coordinate system, thereby obtaining the standard reference position of the natural reference feature in the target coordinate system. For example, the standard reference position for the center of the receiving position contour, the center of the receiving groove edge, the center of the limiting groove edge, or the center of the supporting groove edge can be the standard center position of the corresponding receiving position or the corresponding structure in the preset standard receiving position model; for the center of the positioning hole, the intersection of the positioning line, or the mark-type feature, the standard reference position can be the corresponding standard position in the standard reference information; for the natural reference feature of the center of the placed material, the standard reference position can be the theoretical receiving position center position 105, the receiving position contour center position, the receiving groove center position, the limiting groove center position, or the supporting groove center position corresponding to the receiving position where the material is located.

[0032] The controller 60 determines multiple local deviation vectors based on the offset of the actual position of each natural reference feature relative to its standard reference position. For the k-th natural reference feature, its actual position in the target coordinate system is Q. act,k The standard reference position is Q. std,k Then the local deviation vector Ek corresponding to this natural reference feature can be expressed as: E k =Q act,k -Q std,k Local deviation vector E k It can include lateral deviation components and longitudinal deviation components, used to characterize the local offset state of the actual position of the natural reference feature in the current carrier image relative to its standard reference position.

[0033] The controller 60 takes the standard reference position of each natural reference feature in the target coordinate system as the input position and the corresponding local deviation vector as the output deviation. It determines the weights based on the recognition confidence of each natural reference feature and the distance between each natural reference feature and the vacant accommodating position 103, constructing a local error field 109 to characterize the mapping relationship between the position and the deviation vector within a preset local region 106. The local error field 109 can be expressed as: F(P) → ΔP. Here, F represents the mapping relationship of the local error field 109, P represents the position within the preset local region 106, and ΔP represents the deviation vector corresponding to that position. The local error field 109 can be constructed individually for each vacant accommodating position 103 to be filled. For different vacant accommodating positions 103, the controller 60 can select different preset local regions 106, different sets of natural reference features, and different weight combinations to obtain a local error field 109 that matches the local state of the corresponding vacant accommodating position 103.

[0034] In one implementation, the weight w of the k-th natural reference feature k It can be represented as: w k =c k ×w dist,k Among them, c k w represents the recognition confidence of the k-th natural reference feature. dist,k This represents the distance weight between the k-th natural reference feature and the vacant location 103. The distance weight can be an inverse distance function or a Gaussian decay function, such as w. dist,k =1 / (d) k +ε), or w dist,k =exp(-dk² / σ²). Where d k Let be the distance between the k-th natural reference feature and the standard position of the vacant accommodating space, ε be a constant to prevent the denominator from being zero, and σ be a preset influence radius. In a further embodiment, the weight can be expressed as: w k =w type,k ×c k ×w dist,k ×w state,k Among them, w type,k For the weight of the natural reference feature type, w state,k Adjust the weights for the natural reference feature state.

[0035] When constructing the local error field 109, the controller 60 can also perform anomaly removal on multiple local deviation vectors. Anomaly removal can be based on at least one of the following: the magnitude of the local deviation vector, the directional consistency of the local deviation vector, the median deviation of the neighborhood, the neighborhood variance, the confidence level of natural reference feature recognition, or the state of natural reference feature. The state of natural reference feature includes at least one of the following: edge-fitting state, tilting state, overlapping state, offset state, pose anomaly state, contour integrity state, occlusion state, or recognition stability state. Based on the anomaly-removed local deviation vectors and their corresponding weights, the controller 60 estimates the mapping relationship between the position and the deviation vector within the preset local region 106 using weighted averaging, inverse distance weighted interpolation, local polynomial fitting, or radial basis function interpolation. When using the weighted averaging method, the local correction ΔP0 at the standard position of the vacant accommodation position can be expressed as: ΔP0 = Σk(w k ×E k ) / (Σkw k ), where E k Let w be the k-th local deviation vector. k The corresponding weights are assigned. In this way, the controller 60 does not directly use the single global offset of the entire material carrier 101 as the correction amount of the vacant accommodating position 103, nor does it directly use the deviation of a single fixed reference feature as the correction amount of the vacant accommodating position 103. Instead, it determines the local correction amount based on multiple natural reference features located near the vacant accommodating position 103 and having standard reference positions.

[0036] The controller 60 determines the local correction amount at the standard position of the vacant accommodating position based on the local error field 109, and corrects the standard position of the vacant accommodating position according to the local correction amount to obtain the reconstructed coordinates of the vacant accommodating position in the target coordinate system. If the standard position of the vacant accommodating position is P0 and the local correction amount is ΔP0, then the reconstructed coordinates of the vacant accommodating position are P... rec It can be represented as: P rec =P0 + ΔP0. Reconstructing coordinates P from vacant storage space. rec This indicates the position of the vacant accommodating position 103 in the target coordinate system at the moment of photography, after correction by the local error field 109. This position, relative to the standard position of the vacant accommodating position, can compensate for positional errors caused by local offset of the material carrier 101, model registration residuals, local camera distortion, or local deformation of the material carrier 101.

[0037] In one implementation, when the recognition confidence of the features of the vacant accommodating position 103 itself, such as its contour, the edge of the accommodating groove, the edge of the limiting groove, the edge of the supporting groove, the edge of the acupoint, the local positioning edge of the carrier, or the edge of the carrier structure, is higher than a preset threshold, the controller 60 can also obtain the center of the contour of the vacant accommodating position 103 itself. Based on the transformation relationship between the image coordinate system and the target coordinate system, the center of the contour of the vacant accommodating position 103 itself is transformed to the target coordinate system. The transformed center of the contour of the vacant accommodating position 103 itself is then weighted and fused with the coordinates obtained by correcting the standard position of the vacant accommodating position according to the local correction amount, to obtain the reconstructed coordinates of the vacant accommodating position. For example, the center of the contour of the vacant accommodating position 103 itself after transformation to the target coordinate system is P. self The coordinates obtained by correcting the standard position of the vacant accommodating space based on the local correction amount are P. field Then the empty space to accommodate the reconstructed coordinates P rec It can be represented as: P rec =β×P self + (1-β)×P field Wherein, β is the fusion weight, 0≤β≤1. The fusion weight can be determined based on at least one of the following: the recognition confidence of the contour of the vacant accommodating position 103 itself, the number of natural reference features involved in constructing the local error field 109, the recognition confidence of each natural reference feature, the consistency of the local deviation vector, or the fitting residual of the local error field 109.

[0038] The controller 60 converts the reconstructed coordinates of the vacant accommodating position into the replenishment time coordinates in the actuator coordinate system based on the carrier motion information, the image capture time, the expected replenishment time of the actuator, and the coordinate system calibration relationship between the target coordinate system and the actuator coordinate system. The carrier image is captured at the image capture time, while the actual replenishment action performed by the actuator 50 is later than the image capture time. The reconstructed coordinates P of the vacant accommodating position are... rec This is represented in the target coordinate system at the time of the photograph. Since the target coordinate system corresponds to the coordinate system of the moving carrier that moves with the material carrier 101, the pose of the moving carrier coordinate system relative to the actuator coordinate system may be different from that at the actual time of material replenishment by the actuator 50. In this embodiment, the controller 60 determines the pose change of the moving carrier coordinate system corresponding to the target coordinate system relative to the actuator coordinate system from the time of the photograph to the expected time of material replenishment by the actuator, based on the carrier motion information, the time of the photograph, and the expected time of material replenishment by the actuator. Based on the transformation relationship between the moving carrier coordinate system corresponding to the target coordinate system and the actuator coordinate system at the expected time of material replenishment by the actuator, the controller 60 converts the reconstructed coordinates of the vacant accommodating position into the replenishment time coordinates in the actuator coordinate system. Since the reconstructed coordinates P of the vacant accommodating position are... recThe coordinates are represented in the target coordinate system at the time of taking the picture. However, the actual time when the actuator 50 performs the material replenishment action is later than the time of taking the picture. Therefore, by converting the above-mentioned pose change, the coordinates of the material replenishment time can be matched with the position of the material carrier 101 at the actual material replenishment time of the actuator 50.

[0039] In one implementation, the controller 60 determines the coordinate system of the moving carrier corresponding to the target coordinate system from the time t of the photographing based on the carrier motion information. img The time to replenishment by the executing agency is t drop The pose changes relative to the actuator coordinate system are calculated, and the expected refueling time t of the actuator is obtained. drop The transformation relationship T from the moving carrier coordinate system corresponding to the target coordinate system to the actuator coordinate system. box_robot (t) drop The coordinates of the feeding time P in the actuator coordinate system. drop It can be represented as: P drop =T box_robot (t) drop P rec Among them, P rec Reconstruct the coordinates for the empty space, T box_robot (t) drop () indicates the time t is the expected refueling time for the executing agency. drop Below, the transformation relationship from the moving carrier coordinate system corresponding to the target coordinate system to the actuator coordinate system, P drop The coordinates for the replenishment time in the actuator coordinate system. The carrier motion information may include at least one of the following: carrier motion speed, carrier displacement, carrier motion direction 125°, or carrier lateral offset. The carrier motion speed or carrier displacement can be obtained from the conveyor speed, encoder signal, or changes in the carrier reference feature position in multiple consecutive frames of carrier images. The expected replenishment time of the actuator can be determined based on at least one of the following: actuator motion path, actuator current state, material pick-up position, target replenishment position, actuator motion cycle, or replenishment task timing issued by controller 60. When the deviation between the updated expected replenishment time and the original expected replenishment time of the actuator 50 exceeds a preset threshold before the actuator 50 performs replenishment, controller 60 recalculates the replenishment time coordinates based on the updated expected replenishment time.

[0040] The controller 60 determines the confidence level of the reconstructed coordinates of the vacant container position based on at least one of the following: the number of natural reference features involved in constructing the local error field 109, the recognition confidence level of the natural reference features, the consistency of the local deviation vector, and the fitting residual of the local error field 109. When the confidence level of the reconstructed coordinates of the vacant container position meets the preset replenishment conditions, the controller 60 controls the actuator 50 to replenish material into the vacant container position 103 according to the replenishment time coordinates. The preset replenishment conditions may include at least one of the following: the number of effective natural reference features is not less than a quantity threshold, the average recognition confidence level of the natural reference features is not less than a confidence threshold, the consistency of the local deviation vector direction meets the preset consistency condition, the fitting residual of the local error field 109 is less than the residual threshold, the reconstructed coordinates of the vacant container position are within the replenishment range of the actuator 50, and the expected replenishment time is within the replenishment time window. When the confidence level of the reconstructed coordinates of the vacant receiving position does not meet the preset replenishment conditions, the controller 60 may select and execute at least one of the following processes according to the preset processing priority: expand the selection range of natural reference features; call the edge of the carrier, positioning hole, positioning line, receiving position array, cavity array, limiting groove array, or outer contour of the carrier as a supplementary reference; call the local error field 109 of the same material carrier 101 in the previous frame, or when the specifications of the material carrier 101, the receiving position layout, the camera field of view area, and the conveying direction are all consistent, and the confidence level of the historical error field meets the preset calling conditions, call the historical error field of the same area of ​​the previous material carrier 101; trigger a reshoot; do not generate a replenishment execution command for the vacant receiving position 103; or output alarm information.

[0041] After the actuator 50 replenishes material into the empty receiving position 103 according to the replenishment time coordinate, the controller 60 can obtain replenishment result information. The replenishment result information may include whether there is material in the receiving position in subsequent images, the deviation between the center of the replenished material and the theoretical receiving position center 105 of the corresponding receiving position, whether the outline of the replenished material is complete, whether the material is attached to the edge or overlaps, whether the pick-up and put-down status of the actuator 50 is normal, whether the release action of the actuator 50 is completed, and whether the programmable logic controller feeds back an action completion signal. When the replenishment is successful, the position deviation of the replenished material is less than a preset threshold, and is confirmed by subsequent images, the controller 60 uses at least one of the following as subsequent natural reference features: the center of the material outline in the receiving position after replenishment, the centroid of the material color area, the center of the material's outer outline, the center of the receiving position outline, the center of the receiving groove edge, the center of the limiting groove edge, or the center of the supporting groove edge. When material replenishment fails, the positional deviation of the replenished material exceeds a preset threshold, the actuator 50 operates abnormally, or no subsequent image confirmation is obtained, the controller 60 prohibits the use of the natural reference feature of the center of the placed material in the receiving position after replenishment as a natural reference feature in the subsequent construction of the local error field 109. This prevents abnormal material replenishment results from contaminating the subsequent local error field 109.

[0042] Please continue reading Figure 1 This embodiment also provides a machine vision-based intelligent dynamic replenishment device. This device can be used to execute the machine vision-based intelligent dynamic replenishment method of any of the above embodiments. The machine vision-based intelligent dynamic replenishment device includes a conveying mechanism 10, a feeding mechanism 20, a replenishment vision component 30, a motion information acquisition component 40, an execution mechanism 50, and a controller 60. The conveying mechanism 10 is used to convey a material carrier 101 having multiple receiving positions 102. The feeding mechanism 20 is used to provide the material to be replenished. The replenishment vision component 30 is used to acquire images of the material carrier 101 moving with the conveying mechanism 10 at the moment of image capture. The motion information acquisition component 40 is used to acquire motion information of the carrier. The execution mechanism 50 is used to remove or receive the material to be replenished from the feeding mechanism 20 and replenish the material to be replenished into the empty receiving positions 103 of the material carrier 101. The controller 60 is communicatively connected to the replenishment vision component 30, the motion information acquisition component 40, and the execution mechanism 50. The controller 60 stores a preset standard receiving position model, the transformation relationship between the image coordinate system and the target coordinate system, and the coordinate system calibration relationship between the target coordinate system and the actuator coordinate system. The controller 60 is configured to execute the steps in the above-mentioned intelligent dynamic feeding method based on machine vision, including acquiring a carrier image, completing the registration of the preset standard receiving position model and the carrier image, identifying vacant receiving positions 103, constructing a local error field 109, obtaining the reconstructed coordinates of the vacant receiving positions, converting the reconstructed coordinates of the vacant receiving positions into feeding time coordinates in the actuator coordinate system, and controlling the actuator 50 to feed material into the vacant receiving positions 103.

[0043] In one embodiment, the controller 60 may include an image acquisition unit, a model registration unit, an vacant accommodating position identification unit, a natural reference feature identification unit, a local deviation vector calculation unit, a local error field construction unit, a coordinate reconstruction unit, a dynamic coordinate prediction unit, and an execution control unit. The image acquisition unit acquires images of the carrier body collected by the feeding vision component 30. The model registration unit registers a preset standard accommodating position model with the carrier body image and determines the theoretical accommodating position region 104 and the theoretical accommodating position center position 105 of each accommodating position in the carrier body image. The vacant accommodating position identification unit identifies the presence or absence of material within each theoretical accommodating position region 104 and identifies accommodating positions where no material is identified as vacant accommodating positions 103. The natural reference feature identification unit selects and identifies multiple natural reference features that are identifiable in the carrier body image and have standard reference positions within a preset local region 106 containing vacant accommodating positions 103. The local deviation vector calculation unit determines multiple local deviation vectors based on the offset of the actual position of each natural reference feature relative to the standard reference position. The local error field construction unit is used to construct a local error field 109 by taking the standard reference position of each natural reference feature as the input position, the corresponding local deviation vector as the output deviation, and determining the weights based on the recognition confidence of each natural reference feature and the distance between each natural reference feature and the vacant receiving position 103. The coordinate reconstruction unit is used to determine the local correction amount at the standard position of the vacant receiving position based on the local error field 109, and correct the standard position of the vacant receiving position based on the local correction amount to obtain the reconstructed coordinates of the vacant receiving position in the target coordinate system. The dynamic coordinate prediction unit is used to determine the pose of the moving carrier coordinate system corresponding to the target coordinate system relative to the actuator coordinate system at the actuator's expected replenishment time based on the carrier motion information, the time of taking the picture, the expected replenishment time of the actuator, and the coordinate system calibration relationship between the target coordinate system and the actuator coordinate system, and converts the reconstructed coordinates of the vacant receiving position into the replenishment time coordinates in the actuator coordinate system. The execution control unit is used to generate a replenishment execution command based on the replenishment time coordinates and control the actuator 50 to replenish material into the vacant receiving position 103 according to the replenishment time coordinates.

[0044] In some embodiments, the replenishment vision component 30 may include multiple cameras. The multiple cameras respectively acquire images of different areas of the material carrier 101. The controller 60 can unify the detection results in the images of different areas to the same target time and the same target coordinate system based on the calibration relationship between each camera and the target coordinate system and the capture time of each camera. When the fields of view of two replenishment vision components 30 overlap, the controller 60 can calculate the reconstructed coordinates corresponding to the same accommodating position within the overlapping area, and perform fusion or selection based on at least one of the following: calibration error, image sharpness, field of view edge distance, number of natural reference features, fitting residual of local error field 109, or confidence of reconstructed coordinates of vacant accommodating positions. A material-retrieving vision component may also be provided at the feeding mechanism 20 to identify the position and orientation of the material to be replenished within the feeding mechanism 20. The actuator 50 may be a robot, a manipulator, a suction mechanism, a gripping mechanism, a pushing mechanism, a material-discharging mechanism, a material-dropping mechanism, or a combination of the above mechanisms; the actuator 50 can actively retrieve the material to be replenished from the feeding mechanism 20, or receive the material to be replenished conveyed or released by the feeding mechanism 20. The replenishment result information can be obtained by the downstream re-inspection camera. The controller 60 can also store the local error field 109, the reconstructed coordinates of the empty accommodating position, the coordinates of the replenishment time, the replenishment result information, and the alarm information to the database for use in subsequent model updates, process traceability, or equipment maintenance.

[0045] According to the intelligent dynamic feeding method and apparatus based on machine vision provided in the embodiments of the present invention, the controller 60 first obtains the standard position of the vacant ...

[0046] Furthermore, the local error field 109 in this invention is constructed based on multiple natural reference features near the empty accommodating position 103 in the current carrier image, rather than relying solely on the registration results of the entire material carrier 101. The controller 60 determines the weights based on the recognition confidence, distance, type, and state of the natural reference features, and can eliminate abnormal local deviation vectors, thereby reducing the impact of individual recognition errors, material edge contact, occlusion, or material replenishment misalignment on the coordinate reconstruction results of the empty accommodating position 103.

[0047] Furthermore, this invention combines the reconstructed coordinates of the vacant accommodating position with the carrier's motion information, the time of image capture, the expected replenishment time of the actuator, and the coordinate system calibration relationship between the target coordinate system and the actuator coordinate system to generate replenishment time coordinates in the actuator coordinate system. This reduces dynamic replenishment errors caused by the discrepancy between the time of image capture and the actual replenishment time of the actuator 50 when the material carrier 101 moves with the conveying mechanism 10.

[0048] When the confidence level of the reconstructed coordinates of the vacant storage position is insufficient or the replenishment result is abnormal, the controller 60 can also perform actions such as expanding the selection range of natural reference features, calling supplementary references, calling historical error fields, triggering re-shooting, not generating replenishment execution instructions, outputting alarm information, or prohibiting abnormal replenishment results from participating in the construction of subsequent local error fields 109. This reduces the risk of erroneous replenishment under low confidence conditions and prevents abnormal replenishment results from contaminating subsequent local error fields 109, thereby improving the stability of the continuous dynamic replenishment process.

Claims

1. A machine vision-based intelligent dynamic feeding method, characterized in that, include: The image of the material carrier moving with the conveying mechanism at the moment of taking the picture is obtained, wherein the material carrier has multiple receiving positions for accommodating materials; Determine the target coordinate system, which is the coordinate system of the moving carrier at the time of the photograph; The preset standard accommodating position model is registered with the carrier image, and the theoretical accommodating position area and theoretical accommodating position center position of each accommodating position in the carrier image are determined based on the registration result. The presence or absence of material is identified within each theoretically defined storage area, and storage areas where no material is identified are designated as vacant storage areas. Based on the transformation relationship between the image coordinate system and the target coordinate system, the center position of the theoretical accommodating position corresponding to the vacant accommodating position is transformed to the target coordinate system to obtain the standard position of the vacant accommodating position; Within a preset local area containing the vacant accommodating space, multiple natural reference features that are identifiable in the image of the carrier and have standard reference positions are selected; wherein, the preset local area is a preset distance range centered on the vacant accommodating space, an adjacent accommodating space area containing the vacant accommodating space, or a local accommodating space area determined according to the block division rules of the preset standard accommodating space model; Identify the image position of each of the natural reference features in the image of the carrier, and convert the image position of each of the natural reference features into the actual position in the target coordinate system according to the transformation relationship between the image coordinate system and the target coordinate system; Obtain the standard reference position of each of the natural reference features in the target coordinate system; Based on the offset of the actual position of each natural reference feature relative to its standard reference position, multiple local deviation vectors are determined; Using the standard reference position of each of the natural reference features in the target coordinate system as the input position, the corresponding local deviation vector as the output deviation, and the weights determined according to the recognition confidence of each of the natural reference features and the distance between each of the natural reference features and the vacant accommodating position, a local error field is constructed to characterize the mapping relationship between the position and the deviation vector in the preset local area. The local correction amount at the standard position of the vacant accommodating position is determined based on the local error field. The standard position of the vacant accommodating position is corrected according to the local correction amount to obtain the reconstructed coordinates of the vacant accommodating position in the target coordinate system; Based on the carrier motion information, the time of taking the picture, the expected replenishment time of the actuator, and the calibration relationship between the target coordinate system and the actuator coordinate system, the reconstructed coordinates of the vacant accommodating position are converted into the replenishment time coordinates in the actuator coordinate system; The actuator is controlled to replenish materials into the empty receiving position according to the replenishment time coordinate.

2. The intelligent dynamic feeding method based on machine vision according to claim 1, characterized in that, The step of registering the preset standard accommodating position model with the carrier image includes: Identify the carrier's positioning features from the carrier image; The model-image coordinate transformation relationship between the preset standard accommodating position model and the carrier image is determined based on the carrier positioning characteristics. Based on the model-image coordinate transformation relationship, the standard layout information of multiple accommodating positions in the preset standard accommodating position model is mapped to the carrier image to obtain the theoretical accommodating position area and theoretical accommodating position center position of each accommodating position in the carrier image; The positioning features of the carrier include at least one of the following: carrier edge, carrier corner, positioning hole, positioning line, intersection of accommodating bit array, accommodating bit array boundary, outer contour of carrier, coding mark, color block mark or graphic mark.

3. The intelligent dynamic feeding method based on machine vision according to claim 2, characterized in that, The standard reference position of the natural reference feature is derived from the preset standard accommodating position model or the standard reference information associated with the preset standard accommodating position model; The step of obtaining the standard reference position of each of the natural reference features in the target coordinate system includes: Obtain the corresponding standard position of the natural reference feature in the preset standard accommodating position model, or obtain the corresponding standard position of the natural reference feature in the standard reference information; Based on the model-image coordinate transformation relationship and the transformation relationship between the image coordinate system and the target coordinate system, the corresponding standard position is transformed to the target coordinate system to obtain the standard reference position of the natural reference feature in the target coordinate system.

4. The intelligent dynamic feeding method based on machine vision according to claim 1, characterized in that, The natural reference features include at least one of the following: natural reference features of carrier structure, natural reference features of marker, or natural reference features of the center of placed material; The natural reference features of the carrier structure include at least one of the following: the center of the receiving position contour, the center of the receiving groove edge, the center of the limiting groove edge, the center of the supporting groove edge, the intersection of the acupoint array, the midpoint of the carrier edge, the end point of the carrier edge, the corner point of the carrier, the center of the positioning hole, the midpoint of the positioning line, or the intersection of the positioning lines. The marker-type natural reference features include at least one of the following: text markers, color block markers, graphic markers, coded markers, QR code markers, barcode markers, or silkscreen markers; The natural reference features of the center of the placed material include at least one of the following: the outline center of the placed material, the circumcircle center of the placed material, the circumrectangle center of the placed material, the grayscale centroid of the placed material, the centroid of the color region of the placed material, or the geometric center of the placed material.

5. The intelligent dynamic feeding method based on machine vision according to claim 4, characterized in that, When selecting multiple natural reference features with standard reference positions, the selection is made according to the type priority of the natural reference features. Among them, the priority of the carrier structure type natural reference feature is higher than the priority of the marker type natural reference feature, and the priority of the marker type natural reference feature is higher than the priority of the placed material center type natural reference feature; When the placed material has issues such as being stuck to the edge, tilted, overlapping, offset, having an abnormal posture, incomplete outline, or being obscured, or when the deviation of the center-type natural reference feature of the placed material relative to the theoretical center position, the center position of the outline of the container, the center position of the container groove, the center position of the limiting groove, or the center position of the supporting groove in the target coordinate system exceeds a preset threshold, the weight of the local deviation vector corresponding to the center-type natural reference feature of the placed material is reduced, or the center-type natural reference feature of the placed material is removed from the natural reference features participating in the construction of the local error field.

6. The intelligent dynamic feeding method based on machine vision according to claim 1, characterized in that, The construction of the local error field, used to characterize the mapping relationship between position and deviation vector within the preset local region, includes: Anomaly removal is performed on the multiple local deviation vectors; Based on the type of the natural reference feature, the identification confidence level, the distance between it and the empty accommodation position, and the state of the natural reference feature, determine the weights corresponding to each of the local deviation vectors after anomaly removal; Based on the local deviation vector and its corresponding weight after anomaly removal, the mapping relationship between the position and the deviation vector in the preset local area is estimated by means of weighted average, inverse distance weighted interpolation, local polynomial fitting or radial basis function interpolation, and the local correction amount at the standard position of the vacant accommodation space is determined by the estimated mapping relationship. The anomaly removal is based on at least one of the following: the magnitude of the local deviation vector, the directional consistency of the local deviation vector, the neighborhood median deviation, the neighborhood variance, the identification confidence of the natural reference feature, or the state of the natural reference feature. The natural reference feature states include at least one of the following: edge-fitting state, tilted state, overlapping state, offset state, abnormal posture state, contour integrity state, occlusion state, or recognition stability state.

7. The intelligent dynamic feeding method based on machine vision according to claim 1, characterized in that, When the confidence level of the identification of the features of the vacant accommodating position itself, such as the accommodating position contour, the accommodating groove edge, the limiting groove edge, the supporting groove edge, the acupoint edge, the local positioning edge of the carrier, or the structural edge of the carrier, is higher than a preset threshold, the center of the vacant accommodating position itself is obtained. Based on the transformation relationship between the image coordinate system and the target coordinate system, the center of the contour of the vacant accommodating space is transformed to the target coordinate system; The center of the contour of the vacant accommodating position after transformation to the target coordinate system is weighted and fused with the coordinates obtained by correcting the standard position of the vacant accommodating position according to the local correction amount to obtain the reconstructed coordinates of the vacant accommodating position. The weights of the weighted fusion are determined based on at least one of the following: the recognition confidence of the contour of the vacant accommodating position itself, the number of natural reference features involved in constructing the local error field, the recognition confidence of each natural reference feature, the consistency of the local deviation vector, or the fitting residual of the local error field.

8. The intelligent dynamic feeding method based on machine vision according to claim 1, characterized in that, The confidence level of the reconstructed coordinates of the vacant accommodating position is determined based on at least one of the following: the number of natural reference features involved in constructing the local error field, the identification confidence level of the natural reference features, the consistency of the local deviation vector, and the fitting residual of the local error field. When the confidence level of the reconstructed coordinates of the vacant storage position meets the preset replenishment conditions, the actuator is controlled to replenish the material to the vacant storage position according to the replenishment time coordinates; When the confidence level of the reconstructed coordinates of the vacant storage position does not meet the preset replenishment condition, at least one of the following processes is selected and executed according to the preset processing priority: Expand the selection range of the natural reference features; Use the edge of the carrier, positioning holes, positioning lines, receiving position array, acupoint array, limiting groove array, or the outer contour of the carrier as supplementary references; The local error field of the same material carrier in the previous frame is invoked, or when the specifications of the material carrier, the layout of the accommodating position, the camera field of view and the conveying direction are consistent, and the confidence of the historical error field meets the preset invocation conditions, the historical error field of the same area of ​​the previous material carrier is invoked. Trigger a repeat shot; No replenishment command will be generated for this empty compartment. Alternatively, it can output alarm information.

9. The intelligent dynamic feeding method based on machine vision according to claim 1, characterized in that, The carrier motion information includes at least one of the carrier motion speed, carrier displacement, carrier motion direction, or carrier lateral offset. The speed or displacement of the carrier is obtained by the speed of the conveying mechanism, the encoder signal, or the change in the position of the carrier's reference feature in multiple consecutive frames of carrier images; The expected replenishment time of the actuator is determined based on at least one of the following: the movement path of the actuator, the current state of the actuator, the material picking position, the target replenishment position, the movement rhythm of the actuator, or the replenishment task sequence issued by the controller. If the deviation between the updated expected replenishment time and the expected replenishment time of the actuator exceeds a preset threshold before the actuator performs replenishment, the replenishment time coordinates are recalculated based on the updated expected replenishment time. After the actuator replenishes material into the empty receiving position, the replenishment result information is obtained; When the replenishment is successful, the position deviation of the replenished material is less than the preset threshold and is confirmed by subsequent images, at least one of the following in the replenished accommodating position is used as the subsequent natural reference feature: the center of the material outline, the centroid of the material color area, the center of the material outer outline, the center of the accommodating position outline, the center of the accommodating groove edge, the center of the limiting groove edge, or the center of the supporting groove edge. When feeding fails, the position deviation of the fed material exceeds a preset threshold, the actuator's operation is abnormal, or no subsequent image confirmation is obtained, the material center-type features in the feeding position are prohibited from being used as natural reference features in the subsequent local error field construction.

10. A machine vision-based intelligent dynamic feeding device, characterized in that, include: A conveying mechanism for conveying a material carrier with multiple receiving positions; A material supply mechanism, used to provide materials that need to be replenished; The feeding vision component is used to capture images of the material carrier moving with the conveying mechanism at the moment of taking the picture. Motion information acquisition component, used to acquire motion information of the carrier; An actuator is used to take out or receive materials to be replenished from the feeding mechanism and replenish the materials to be replenished into the empty accommodating position of the material carrier. The controller is communicatively connected to the feeding vision component, the motion information acquisition component, and the actuator, respectively. The controller stores a preset standard accommodating position model, the transformation relationship between the image coordinate system and the target coordinate system, and the calibration relationship between the target coordinate system and the actuator coordinate system. The controller is configured to: Acquire the image of the carrier captured by the feeding vision component; The preset standard accommodating position model is registered with the carrier image, and the theoretical accommodating position area and theoretical accommodating position center position of each accommodating position in the carrier image are determined. The presence or absence of material is identified within each theoretically defined storage area, and storage areas where no material is identified are designated as vacant storage areas. Transform the center position of the theoretical accommodating position corresponding to the vacant accommodating position to the target coordinate system to obtain the standard position of the vacant accommodating position; Within a preset local area containing the vacant accommodating space, multiple natural reference features that are identifiable in the image of the carrier and have standard reference positions are selected; wherein, the preset local area is a preset distance range centered on the vacant accommodating space, an adjacent accommodating space area containing the vacant accommodating space, or a local accommodating space area determined according to the block division rules of the preset standard accommodating space model; Identify the image position of each of the natural reference features in the image of the carrier, and convert the image position of each of the natural reference features into the actual position in the target coordinate system according to the transformation relationship between the image coordinate system and the target coordinate system; Obtain the standard reference position of each of the natural reference features in the target coordinate system; Based on the offset of the actual position of each natural reference feature relative to its standard reference position, multiple local deviation vectors are determined; Using the standard reference position of each of the natural reference features in the target coordinate system as the input position, the corresponding local deviation vector as the output deviation, and the weights determined according to the recognition confidence of each of the natural reference features and the distance between each of the natural reference features and the vacant accommodating position, a local error field is constructed to characterize the mapping relationship between the position and the deviation vector in the preset local area. The local correction amount at the standard position of the vacant accommodating position is determined based on the local error field. The standard position of the vacant accommodating position is corrected according to the local correction amount to obtain the reconstructed coordinates of the vacant accommodating position in the target coordinate system; Based on the carrier motion information, the time of taking the picture, the expected replenishment time of the actuator, and the calibration relationship between the target coordinate system and the actuator coordinate system, the reconstructed coordinates of the vacant accommodating position are converted into the replenishment time coordinates in the actuator coordinate system; The actuator is controlled to replenish materials into the empty receiving position according to the replenishment time coordinate.

Images