Nut feeding control method and device and computer readable storage medium

Abstract

The invention discloses a nut feeding control method and device and a computer readable storage medium. The method comprises the following steps: acquiring a nut image of a target nut in a target container under the condition of determining to start loading operation on the nut; comparing and analyzing the nut image and the template nut image by using a visual proofreading model to obtain a deviation value of the target nut relative to a preset template nut; according to the deviation value, the grabbing assembly is controlled to conduct posture adjustment, and the target grabbing posture of the grabbing assembly is obtained; and the grabbing assembly is controlled to grab the target nut in the target grabbing posture, the target nut is placed at the target assembling position so as to complete feeding operation, and the target assembling position is the position where the target nut needs to be assembled. The technical problems that in the related technology, the accuracy of feeding of copper nut joints through a traditional method is difficult to meet the accuracy requirement, consequently, the placement position of nuts has deviation, and potential safety hazards are likely to be caused are solved.

Description

Technical Field

[0001] The present invention relates to the field of robot vision control technology, and in particular to a nut feeding control method and device, and a computer-readable storage medium. Background Art

[0002] In the production process of air-conditioning liquid inlet pipes and gas pipes, there is a process that requires the copper nut joints to be accurately fed so that the liquid and gas pipes are in a coaxial position with the copper nut joints, which is conducive to the precise insertion of the liquid and gas pipes into the copper nut joints, and facilitates the subsequent welding production of the liquid inlet pipe and the gas collecting pipe. Due to the high precision requirements, the feeding of copper nut joints is a relatively difficult problem. In traditional production, there are also some equipment for feeding, but the degree of automation is low, and manual operation or monitoring is still required. It is necessary to stare at the equipment for a long time, which is labor-intensive and has a poor on-site working environment. In view of the above situation and in combination with the existing technology, the automatic feeding equipment for pipe joints was born. In this equipment, the feeding of copper nuts is very important. This workstation is composed of a four-axis robot equipped with a visual camera. Combined with the visual scheme planned in the early stage, the "eyes outside the hand" method is adopted to pick up materials, which means that the camera is fixed just above the material tray and does not move with the robot arm. However, it is found in the production debugging that although this method is relatively simple to calculate and the system design is simple, in this scenario, on the one hand, due to the fixed viewing angle, the flexibility is too low and there are blind spots, and the material tray often cannot be fully taken. On the other hand, when grabbing the nut, it needs to fix a template angle so that it can be accurately fed. However, due to the limitation of camera resolution, the angle value of real-time material picking feedback has large differences and cannot be satisfied.

[0003] With regard to the problem that the accuracy of feeding copper nut joints using traditional methods in the above-mentioned related technologies is difficult to meet the accuracy requirements, resulting in deviations in the nut placement position and easily causing safety hazards, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present invention provide a nut feeding control method and device, and a computer-readable storage medium, so as to at least solve the technical problem in the related art that the accuracy of feeding copper nut joints using traditional methods is difficult to meet the accuracy requirements, resulting in deviations in the nut placement position and easily causing safety hazards.

[0005] According to one aspect of an embodiment of the present invention, a method for controlling nut feeding is provided, comprising: when determining to start a feeding operation on the nut, obtaining a nut image of a target nut in a target container, wherein the target container is used to hold the nut, and the target nut is any one of the nuts in the target container; using a visual calibration model to compare and analyze the nut image and a template nut image to obtain an offset value of the target nut compared to a preset template nut, wherein the visual calibration model is a model constructed by performing image calibration and feature matching on multiple groups of sample data using an image calibration method, and each of the multiple groups of sample data includes : a sample template nut image, and multiple sample nut images that need to be calibrated based on the sample template nut image, wherein the template nut image is an image captured when a grasping component grasps a preset template nut with a preset reference posture, and the grasping component is a component that grasps the nut to perform the feeding operation; controlling the grasping component to adjust its posture according to the offset value to obtain a target grasping posture of the grasping component; controlling the grasping component to grasp the target nut with the target grasping posture, and placing the target nut at a target assembly position to complete the feeding operation, wherein the target assembly position is a position where the target nut needs to be assembled.

[0006] Optionally, determining to start the feeding operation on the nuts includes: receiving position information transmitted by a grasping device based on a first communication channel, wherein the first communication channel is used for data transmission with the grasping device, the grasping device is used to perform the feeding operation, and the grasping component is a component in the grasping device for grasping the nuts; receiving image information transmitted by an image acquisition device based on a second communication channel, wherein the second communication channel is used for data transmission with the grasping device, the image acquisition device is installed within a predetermined range of the grasping component and is used to capture images of the nuts in the target container; determining to start the feeding operation on the nuts when the position information indicates that the grasping device is in a predetermined position and the image information indicates that there is at least one nut in the target container.

[0007] Optionally, before using the visual proofreading model to compare and analyze the nut image and the template nut image to obtain the offset value of the target nut compared to the preset template nut, the nut feeding control method also includes: determining that the current posture of the grasping component is the preset reference posture; determining that the nut within the current grasping range of the grasping component is the preset template nut, wherein the current grasping range refers to the grasping range reached when the grasping component grasps with the preset reference posture; obtaining the first relative grasping point when the grasping component grasps the preset template nut with the preset reference posture, wherein the first relative grasping point refers to the contact point between the grasping component and the preset template nut when the grasping component grasps the preset template nut with the preset reference posture; when the first relative grasping point meets the preset grasping requirements, obtaining the current image of the preset template nut; and inputting the current image as the template nut image into the visual proofreading model.

[0008] Optionally, a visual proofreading model is used to compare and analyze the nut image and the template nut image to obtain an offset value of the target nut compared to the preset template nut, including: comparing and analyzing the nut image and the template nut image using a visual proofreading model to obtain an analysis result; based on the analysis result, determining the position deviation between the first center point of the target nut in the nut image and the second center point of the preset template nut in the template nut image as the position offset value of the target nut compared to the preset template nut; based on the analysis result, determining the angle deviation between the target nut in the nut image and the preset template nut in the template nut image as the angle offset value of the target nut compared to the preset template nut; integrating the position offset value and the angle offset value to obtain the offset value of the target nut compared to the preset template nut.

[0009] Optionally, the position offset value and the angle offset value are integrated to obtain the offset value of the target nut compared to the preset template nut, including: integrating the position offset value and the angle offset value to obtain the target offset value; and converting the format of the target offset value using a format conversion program to obtain the offset value.

[0010] Optionally, the grasping component is controlled to adjust its posture according to the offset value to obtain a target grasping posture of the grasping component, including: generating a posture adjustment strategy for adjusting the posture of the grasping component using a preset algorithm according to the offset value; in the process of controlling the grasping component to adjust its posture according to the posture adjustment strategy, obtaining a second relative grasping point when the grasping component grasps the target nut with the current grasping posture, wherein the current grasping posture is the grasping posture of the grasping component at any moment in the posture adjustment process, and the second relative grasping point refers to the contact point between the grasping component and the target nut when the grasping component grasps the target nut with the current grasping posture; when the second relative grasping point is the same as the first relative grasping point when the grasping component grasps the preset template nut with the preset reference posture, the current grasping posture corresponding to the second relative grasping point is determined to be the target grasping posture of the grasping component.

[0011] Optionally, the grasping component includes a first grasping component and a second grasping component, the target nut includes a first target nut and a second target nut, the target container includes a first target container and a second target container, the preset reference posture includes a first preset reference posture and a second preset reference posture, the target grasping posture includes a first target grasping posture and a second target grasping posture, the target assembly position includes a first target assembly position and a second target assembly position, and the grasping component is controlled to grasp the target nut with the target grasping posture and place the target nut at the target assembly position to complete the feeding operation, including: determining that the target grasping posture of the first grasping component grasping the first target nut is the first target grasping posture, and determining that the target grasping posture of the second grasping component grasping the second target nut is the second target grasping posture, wherein the first grasping component and the second grasping component are two different grasping components in the grasping device, the first target nut is any one of the nuts contained in the first target container, and the second target nut is any one of the nuts contained in the second target container. any one of the nuts contained in the target container, the first target container is used to hold the nuts that the first grasping component needs to grasp, and the second target container is used to hold the nuts that the second grasping component needs to grasp; control the grasping device to adjust its posture, and control the first grasping component to grasp the first target nut after adjusting the first grasping component to the first target grasping posture; after grasping the first target nut, control the grasping device to restore to a reference posture, wherein the reference posture is the posture of the grasping device when the first grasping component is in a first preset reference posture and the second grasping component is in a second preset reference posture; control the grasping device to adjust its posture, and control the second grasping component to grasp the second target nut after adjusting the second grasping component to the second target grasping posture; after grasping the first target nut and the second target nut, control the grasping device to place the first target nut in the first target assembly position and the second target nut in the second target assembly position to complete the feeding operation.

[0012] According to another aspect of an embodiment of the present invention, a nut feeding control device is further provided, comprising: a first acquisition unit, for acquiring a nut image of a target nut in a target container when it is determined that a feeding operation is to be started on the nut, wherein the target container is used to hold the nut, and the target nut is any one of the nuts in the target container; a second acquisition unit, for comparing and analyzing the nut image and the template nut image using a visual calibration model to obtain an offset value of the target nut compared to a preset template nut, wherein the visual calibration model is a model constructed by performing image calibration and feature matching on multiple groups of sample data using an image calibration method, and each of the multiple groups of sample data is It includes: a sample template nut image, and multiple sample nut images that need to be calibrated based on the sample template nut image, wherein the template nut image is an image captured when a grasping component grasps a preset template nut with a preset reference posture, and the grasping component is a component that grasps the nut to perform the feeding operation; a third acquisition unit, used to control the grasping component to adjust its posture according to the offset value to obtain a target grasping posture of the grasping component; a control unit, used to control the grasping component to grasp the target nut with the target grasping posture, and place the target nut at a target assembly position to complete the feeding operation, wherein the target assembly position is the position where the target nut needs to be assembled.

[0013] Optionally, the first acquisition unit includes: a first receiving module, used to receive position information transmitted by a grasping device based on a first communication channel, wherein the first communication channel is used to transmit data with the grasping device, the grasping device is used to perform the feeding operation, and the grasping component is a component in the grasping device used to grasp the nut; a second receiving module, used to receive image information transmitted by an image acquisition device based on a second communication channel, wherein the second communication channel is used to transmit data with the grasping device, the image acquisition device is installed within a predetermined range of the grasping component and is used to capture the image of the nut in the target container; a first determination module, used to determine to start the feeding operation on the nut when the position information indicates that the grasping device is in a predetermined position and the image information indicates that there is at least one nut in the target container.

[0014] Optionally, the nut feeding control device also includes: a first determination unit, which is used to determine that the current posture of the grasping component is the preset reference posture before comparing and analyzing the nut image and the template nut image using the visual proofreading model to obtain the offset value of the target nut compared to the preset template nut; a second determination unit, which is used to determine that the nut within the current grasping range of the grasping component is the preset template nut, wherein the current grasping range refers to the grasping range reached when the grasping component grasps with the preset reference posture; a fourth acquisition unit, which is used to acquire the first relative grasping point when the grasping component grasps the preset template nut with the preset reference posture, wherein the first relative grasping point refers to the contact point between the grasping component and the preset template nut when the grasping component grasps the preset template nut with the preset reference posture; a fifth acquisition unit, which is used to acquire the current image of the preset template nut when the first relative grasping point meets the preset grasping requirements; an input unit, which is used to input the current image as the template nut image into the visual proofreading model.

[0015] Optionally, the second acquisition unit includes: a first acquisition module, used to compare and analyze the nut image and the template nut image using a visual proofreading model to obtain an analysis result; a second determination module, used to determine, based on the analysis result, a position deviation between a first center point of the target nut in the nut image and a second center point of the preset template nut in the template nut image as a position offset value of the target nut compared to the preset template nut; a third determination module, used to determine, based on the analysis result, an angle deviation between the target nut in the nut image and the preset template nut in the template nut image as an angle offset value of the target nut compared to the preset template nut; and a fourth acquisition module, used to integrate the position offset value and the angle offset value to obtain the offset value of the target nut compared to the preset template nut.

[0016] Optionally, the fourth acquisition module includes: a first acquisition submodule, used to integrate the position offset value and the angle offset value to obtain a target offset value; and a second acquisition submodule, used to perform format conversion on the target offset value using a format conversion program to obtain the offset value.

[0017] Optionally, the third acquisition unit includes: a generation module, which is used to generate a posture adjustment strategy for adjusting the posture of the grasping component by using a preset algorithm according to the offset value; a fifth acquisition module, which is used to obtain a second relative grasping point when the grasping component grasps the target nut with a current grasping posture during the process of controlling the grasping component to adjust the posture according to the posture adjustment strategy, wherein the current grasping posture is the grasping posture of the grasping component at any moment during the posture adjustment process, and the second relative grasping point refers to the contact point between the grasping component and the target nut when the grasping component grasps the target nut with the current grasping posture; a fourth determination module, which is used to determine that the current grasping posture corresponding to the second relative grasping point is the target grasping posture of the grasping component when the second relative grasping point is the same as the first relative grasping point when the grasping component grasps the preset template nut with the preset reference posture.

[0018] Optionally, the grasping component includes a first grasping component and a second grasping component, the target nut includes a first target nut and a second target nut, the target container includes a first target container and a second target container, the preset reference posture includes a first preset reference posture and a second preset reference posture, the target grasping posture includes a first target grasping posture and a second target grasping posture, the target assembly position includes a first target assembly position and a second target assembly position, and the control unit includes: a fifth determination module, used to determine that the target grasping posture of the first grasping component grasping the first target nut is the first target grasping posture, and determine that the target grasping posture of the second grasping component grasping the second target nut is the second target grasping posture, wherein the first grasping component and the second grasping component are two different grasping components in the grasping device, the first target nut is any one of the nuts contained in the first target container, the second target nut is any one of the nuts contained in the second target container, and the first target container is used to hold the first grasping component required The nut to be grasped, the second target container is used to hold the nut that the second grasping component needs to grasp; a first control module is used to control the grasping device to adjust its posture, and control the first grasping component to grasp the first target nut after adjusting the first grasping component to the first target grasping posture; a second control module is used to control the grasping device to restore to a reference posture after grasping the first target nut, wherein the reference posture is the posture of the grasping device when the first grasping component is in a first preset reference posture and the second grasping component is in a second preset reference posture; a third control module is used to control the grasping device to adjust its posture, and control the second grasping component to grasp the second target nut after adjusting the second grasping component to the second target grasping posture; a fourth control module is used to control the grasping device to place the first target nut in the first target assembly position and the second target nut in the second target assembly position after grasping the first target nut and the second target nut, so as to complete the loading operation.

[0019] According to another aspect of an embodiment of the present invention, a nut feeding control system is further provided, and the nut feeding control system uses any one of the nut feeding control methods described above.

[0020] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium includes a stored program, wherein the program executes any one of the above-mentioned nut feeding control methods.

[0021] According to another aspect of an embodiment of the present invention, a processor is further provided, wherein the processor is used to run a program, wherein when the program is run, any one of the above-mentioned nut feeding control methods is executed.

[0022] According to another aspect of an embodiment of the present invention, a computer program product is provided, comprising computer instructions, wherein when the computer instructions are executed by a processor, any one of the above-mentioned nut feeding control methods is executed.

[0023] In an embodiment of the present invention, when it is determined that a feeding operation is to be started on a nut, a nut image of a target nut in a target container is obtained, wherein the target container is used to hold nuts, and the target nut is any nut in the target container; the nut image and the template nut image are compared and analyzed using a visual proofreading model to obtain an offset value of the target nut compared to a preset template nut, wherein the visual proofreading model is a model constructed by image calibration and feature matching of multiple groups of sample data using an image calibration method, and each of the multiple groups of sample data includes: a sample template nut image, and multiple sample nut images that need to be calibrated based on the sample template nut image, the template nut image is an image captured when a grasping component grasps a preset template nut with a preset reference posture, and the grasping component is a component that grasps the nut to perform a feeding operation; the grasping component is controlled to adjust its posture according to the offset value to obtain a target grasping posture of the grasping component; the grasping component is controlled to grasp the target nut with the target grasping posture, and the target nut is placed at a target assembly position to complete the feeding operation, wherein the target assembly position is a position where the target nut needs to be assembled. Through the above technical scheme, the purpose of adjusting the grasping posture of the grasping component by comparing the deviation between the current placement image of the nut to be grasped and the image of the standard template nut is achieved, so that the grasping state of the grasping component grasping the nut is the same as the state of grasping the standard template nut. The technical effect of accurately adjusting the posture of the grasping component when grasping the nut to avoid deviation in the placement position of the nut is achieved, and the accuracy of loading the copper nut joint is improved, thereby solving the technical problem in the related technology that the accuracy of loading the copper nut joint by the traditional method is difficult to meet the precision requirements, resulting in deviation in the nut placement position and easy safety hazards. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0025] Figure 1 It is a hardware structure block diagram of a mobile terminal of a nut feeding control method according to an embodiment of the present invention;

[0026] Figure 2 is a flow chart of a nut feeding control method according to an embodiment of the present invention;

[0027] Figure 3 is a flow chart of an optional nut feeding control method according to an embodiment of the present invention;

[0028] Figure 4 Schematic diagram of a nut feeding control device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] As introduced in the background technology, the accuracy of feeding copper nut joints using traditional methods in the related art is difficult to meet the accuracy requirements, resulting in deviations in the placement of the nut and easily causing safety hazards. In view of the above defects, a nut feeding control method and device, and a computer-readable storage medium are provided in the embodiments of the present invention.

[0032] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0033] The method embodiments provided in the embodiments of the present invention can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal of a nut feeding control method according to an embodiment of the present invention. Figure 1As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0034] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the feeding control method of nuts in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific network examples may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0035] According to an embodiment of the present invention, a method embodiment of a nut feeding control method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0036] Figure 2 is a flow chart of a nut feeding control method according to an embodiment of the present invention. Figure 2 As shown, the method comprises the following steps:

[0037] Step S202, when it is determined to start the feeding operation on the nuts, a nut image of the target nut in the target container is obtained, wherein the target container is used to hold nuts, and the target nut is any nut in the target container.

[0038] In this embodiment, by acquiring an image of any nut in the material tray (i.e., the target container), the nut is the nut that the gripper (i.e., the gripping component in the embodiment of the present invention, including the first gripping component and the second gripping component) of the robot (i.e., the gripping device in the embodiment of the present invention) is ready to grasp. By acquiring the image of the nut, its placement position can be analyzed so that the robot can grasp the nut in accordance with the preset gripping requirements when grasping the nut, thereby ensuring the grasping accuracy and improving the accuracy of nut feeding.

[0039] According to the above embodiment of the present invention, in the above step S202, it is determined to start the feeding operation on the nuts, including: receiving position information transmitted by a grasping device based on a first communication channel, wherein the first communication channel is used for data transmission with the grasping device, the grasping device is used to perform the feeding operation, and the grasping component is a component in the grasping device used to grasp the nuts; receiving image information transmitted by an image acquisition device based on a second communication channel, wherein the second communication channel is used for data transmission with the grasping device, the image acquisition device is installed within a predetermined range of the grasping component and is used to capture images of nuts in the target container; when the position information indicates that the grasping device is in a predetermined position and the image information indicates that there is at least one nut in the target container, it is determined to start the feeding operation on the nuts.

[0040] Optionally, the above-mentioned image acquisition device may include but is not limited to: a camera, a webcam, an infrared sensor and other devices that can be used to capture images.

[0041] In this embodiment, before the nuts are loaded, the material tray and the robot need to be placed in their respective positions. Here, a control center can be used to communicate with the visual camera (i.e., image acquisition device, which is described in detail using the visual camera as an example in the embodiment of the present invention) and the robot, so as to control the visual camera and the robot through the control center to achieve accurate loading of the nuts.

[0042] Specifically, firstly, the visual camera can be controlled to capture the image of the material tray, so as to determine whether the material tray has been placed in the required position (such as under the robot gripper, or on a workbench used to prevent the material tray, etc.), and whether nuts are placed on the material tray; the robot's own position information can also be received to determine whether the robot has been placed in the required position. After confirming that all these preparations are done, the loading operation can be started.

[0043] It should be noted that the visual camera is installed at the gripper of the robot or connected to the gripper, so that the visual camera can move along with the movement of the gripper, thereby facilitating the collection of more accurate image data.

[0044] In addition, when loading nuts, the robot needs to grab the nuts in the tray multiple times, and due to the limitation of visual field, it needs to change the photographing position multiple times until all the nuts in the tray are taken out; in order to avoid the clutter of the robot switching the photographing position, the tray can be used as a stacking tray to establish a predetermined number of stacks (such as 5*9, which is just an example given here, and other data can also be used to establish a specific stacking, which is not specifically limited here), and a nut is placed in the range of each stacking, and the robot's photographing program can be written according to the established stacking. For example, the stacking can be numbered, and the robot photographing program under each number can be numbered and stored, so that when the robot grabs the nut, it can control the gripper to move to the corresponding photographing position according to the stored number to facilitate the capture of the corresponding nut image.

[0045] Step S204, using the visual proofreading model to compare and analyze the nut image and the template nut image to obtain the offset value of the target nut compared to the preset template nut, wherein the visual proofreading model is a model constructed by image calibration and feature matching of multiple groups of sample data using an image calibration method, and each of the multiple groups of sample data includes: a sample template nut image, and multiple sample nut images that need to be calibrated based on the sample template nut image. The template nut image is an image captured when the grasping component grasps the preset template nut with a preset reference posture, and the grasping component is a component that grasps the nut to perform a feeding operation.

[0046] In this embodiment, the nut image captured by the visual camera can be input into the visual proofreading model, and the input nut image can be compared and analyzed with the template nut image using the visual proofreading model to analyze the offset value of the nut in the nut image compared to the preset template nut in the template nut image; the offset value here may include but is not limited to the position offset value and the angle offset value.

[0047] According to the above embodiment of the present invention, before the above step S204, that is, before comparing and analyzing the nut image and the template nut image using the visual proofreading model to obtain the offset value of the target nut compared to the preset template nut, the nut feeding control method also includes: determining that the current posture of the grasping component is the preset reference posture; determining that the nut within the current grasping range of the grasping component is the preset template nut, wherein the current grasping range refers to the grasping range reached when the grasping component grasps with the preset reference posture; obtaining the first relative grasping point when the grasping component grasps the preset template nut with the preset reference posture, wherein the first relative grasping point refers to the contact point between the grasping component and the preset template nut when the grasping component grasps the preset template nut with the preset reference posture; when the first relative grasping point meets the preset grasping requirements, obtaining the current image of the preset template nut; and inputting the current image as the template nut image into the visual proofreading model.

[0048] Specifically, an angle of the robot can be arbitrarily selected as a reference angle. Assuming that at the current reference angle, the posture of the robot is the reference posture, and the posture of the gripper in the robot is the preset reference posture, a nut can be placed under the current gripper as a preset template nut, and it can be adjusted to a placement state that meets the preset gripping requirements. Then, a visual camera is used to capture the current image of the preset template nut in the preset reference posture of the gripper, and the captured image is input into the visual proofreading model as a template nut image to provide a reference for subsequent image analysis of the required gripper nuts.

[0049] It should be noted that the preset grasping requirement here means that the relative grasping point between the grasping clamp and the nut meets the grasping requirements. For example, the contact point between the grasping clamp and the nut is exactly the midpoint of the contact edge in the nut to ensure the accuracy of grasping the nut. Of course, other conditions that can ensure grasping accuracy can also be selected as preset grasping requirements without specific restrictions.

[0050] According to the above embodiment of the present invention, in the above step S204, the nut image and the template nut image are compared and analyzed using a visual proofreading model to obtain an offset value of the target nut compared to the preset template nut, including: comparing and analyzing the nut image and the template nut image using a visual proofreading model to obtain an analysis result; based on the analysis result, determining the position deviation between the first center point of the target nut in the nut image and the second center point of the preset template nut in the template nut image as the position offset value of the target nut compared to the preset template nut; based on the analysis result, determining the angle deviation between the target nut in the nut image and the preset template nut in the template nut image as the angle offset value of the target nut compared to the preset template nut; integrating the position offset value and the angle offset value to obtain the offset value of the target nut compared to the preset template nut.

[0051] In this embodiment, the nut in the captured nut image can be calibrated by using the nine-point calibration and rotational calibration methods, and then the nut image is compared and analyzed with the template nut image using a visual proofreading image to calculate the position offset value and angle offset value of the nut in the nut image (i.e., the nut that the robot is ready to grasp) compared to the preset template nut.

[0052] In the above embodiment of the present invention, the position offset value and the angle offset value are integrated to obtain the offset value of the target nut compared to the preset template nut, including: integrating the position offset value and the angle offset value to obtain the target offset value; using a format conversion program to convert the format of the target offset value to obtain the offset value.

[0053] Specifically, since the image data taken by the visual camera and the data obtained by the preliminary analysis are both in string format, they need to be formatted to convert them into a data format that the robot can understand (such as real numbers). That is, the format conversion program can be used to convert the offset value obtained by the preliminary analysis to obtain an offset value in a format that the robot can understand.

[0054] Step S206, controlling the grasping component to adjust its posture according to the offset value to obtain a target grasping posture of the grasping component.

[0055] In this embodiment, the posture of the robot or the robot's gripper can be adjusted based on the offset value obtained in the above steps, so that the contact point when the adjusted gripper grasps the target nut (i.e., the nut that the robot is ready to grasp) in the current posture (i.e., the target grasping posture) meets the preset grasping requirements.

[0056] According to the above embodiment of the present invention, in the above step S206, the grasping component is controlled to adjust its posture according to the offset value to obtain the target grasping posture of the grasping component, including: generating a posture adjustment strategy for adjusting the posture of the grasping component according to the offset value using a preset algorithm; in the process of controlling the grasping component to adjust its posture according to the posture adjustment strategy, obtaining a second relative grasping point when the grasping component grasps the target nut with the current grasping posture, wherein the current grasping posture is the grasping posture of the grasping component at any moment in the posture adjustment process, and the second relative grasping point refers to the contact point between the grasping component and the target nut when the grasping component grasps the target nut with the current grasping posture; when the second relative grasping point is the same as the first relative grasping point when the grasping component grasps the preset template nut with the preset reference posture, the current grasping posture corresponding to the second relative grasping point is determined as the target grasping posture of the grasping component.

[0057] In this embodiment, a posture adjustment strategy for adjusting the posture of the robot can be generated according to the offset value, and the robot can be directly adjusted to the posture calculated by the posture adjustment strategy, or the posture of the robot and its gripper can be slowly adjusted according to the posture adjustment strategy, and during the adjustment process, the contact points of the gripper when grasping the target nut with the current posture are continuously monitored to see whether they meet the preset grasping requirements. If so, it can be considered that the gripper has currently been adjusted to the posture required to grasp the target nut (i.e., the target grasping posture).

[0058] It should be noted that when adjusting the posture of the gripper, it is generally not limited to adjusting the gripper. If necessary, the posture of the robot will also be adjusted to drive the adjustment of the gripper.

[0059] Step S208, controlling the grasping component to grasp the target nut in a target grasping posture, and placing the target nut at a target assembly position to complete the loading operation, wherein the target assembly position is a position where the target nut needs to be assembled.

[0060] In this embodiment, after adjusting the posture of the robot gripper, the robot gripper can be controlled to grab the target nut and place it at the position where the nut needs to be assembled to complete the precise loading of the nut.

[0061] According to the above embodiment of the present invention, in the above step S208, the grasping component includes a first grasping component and a second grasping component, the target nut includes a first target nut and a second target nut, the target container includes a first target container and a second target container, the preset reference posture includes a first preset reference posture and a second preset reference posture, the target grasping posture includes a first target grasping posture and a second target grasping posture, the target assembly position includes a first target assembly position and a second target assembly position, and the grasping component is controlled to grasp the target nut with the target grasping posture, and the target nut is placed at the target assembly position to complete the loading operation, including: determining that the target grasping posture of the first grasping component grasping the first target nut is the first target grasping posture, and determining that the target grasping posture of the second grasping component grasping the second target nut is the second target grasping posture, wherein the first grasping component and the second grasping component are two different grasping components in the grasping device, and the first target nut is any one contained in the first target container. nut, the second target nut is any nut contained in the second target container, the first target container is used to contain the nut that the first grasping component needs to grasp, and the second target container is used to contain the nut that the second grasping component needs to grasp; control the grasping device to adjust its posture, and control the first grasping component to grasp the first target nut after adjusting the first grasping component to the first target grasping posture; after grasping the first target nut, control the grasping device to restore to the reference posture, wherein the reference posture is the posture of the grasping device when the first grasping component is in the first preset reference posture and the second grasping component is in the second preset reference posture; control the grasping device to adjust its posture, and control the second grasping component to grasp the second target nut after adjusting the second grasping component to the second target grasping posture; after grasping the first target nut and the second target nut, control the grasping device to place the first target nut in the first target assembly position and the second target nut in the second target assembly position to complete the loading operation.

[0062] Specifically, since the robot generally has two grippers, the robot generally grabs two nuts (i.e., one gripper grabs one) each time before assembling them. Of course, each gripper of the robot corresponds to its own material tray, and each gripper is applicable to the gripping method provided in the above-mentioned embodiment of the present invention when grabbing the nuts in the material tray. The robot is generally in a preset reference posture. When grabbing the nuts, the posture of the robot and the gripper is generally adjusted according to the posture adjustment strategy of one of the grippers. After the gripper grabs the nut, the robot is first restored to the reference posture, and then the posture is adjusted according to the posture adjustment group strategy of the other gripper to grab the second nut. After that, the two nuts are placed at the corresponding assembly positions to complete the precise loading of the nuts.

[0063] It should be noted that the two nuts grasped by the robot can be placed in the same assembly position or in different assembly positions, which can be adjusted according to actual conditions and needs.

[0064] It can be seen from the above that, through the technical solution provided by the above embodiment of the present invention, when it is determined that the feeding operation on the nut has started, a nut image of the target nut in the target container can be obtained, wherein the target container is used to hold the nuts, and the target nut is any nut in the target container; the nut image and the template nut image are compared and analyzed by using a visual proofreading model to obtain an offset value of the target nut compared to the preset template nut, wherein the visual proofreading model is a model constructed by image calibration and feature matching of multiple groups of sample data using an image calibration method, and each of the multiple groups of sample data includes: a sample template nut image, and multiple sample nut images that need to be calibrated based on the sample template nut image, and the template nut image is collected when the grasping component grasps the preset template nut with a preset reference posture. image, the grasping component is a component for grasping nuts to perform feeding operations; the grasping component is controlled to adjust its posture according to the offset value to obtain a target grasping posture of the grasping component; the grasping component is controlled to grasp the target nut with the target grasping posture, and the target nut is placed in the target assembly position to complete the feeding operation, wherein the target assembly position is the position where the target nut needs to be assembled, thereby achieving the purpose of adjusting the grasping posture of the grasping component by comparing the deviation between the current placement image of the nut to be grasped and the image of the standard template nut, so that the grasping state of the grasping component grasping the nut is the same as the state of grasping the standard template nut, achieving the technical effect of accurately adjusting the posture of the grasping component when grasping the nut to avoid deviation in the nut placement position, and improving the accuracy of feeding the copper nut joint.

[0065] Therefore, the technical solution provided by the above-mentioned embodiment of the present invention solves the technical problem in the related art that the accuracy of feeding copper nut joints using traditional methods is difficult to meet the accuracy requirements, resulting in deviations in the nut placement position and easily causing safety hazards.

[0066] It should be noted that in the embodiments provided by the present invention, a control center can be used to communicate with the visual camera and the robot respectively, so as to control the visual camera and the robot through the control center to achieve precise loading of the nuts; it is also possible to use only the communication between the visual camera and the robot to collaboratively complete the precise loading of the nuts. The image taken by the visual camera can be transmitted to the corresponding visual software or processed using its own embedded image processing program, and then can be transmitted to the robot by the visual camera through a certain communication method (such as SOCKET communication method). After receiving the corresponding data, the robot uses its own embedded algorithm program to control the gripper to adjust its posture to achieve precise loading of the nuts; no specific restrictions are made here.

[0067] Combine the following Figure 3 Another embodiment of the present invention is described in detail. Figure 3 Detailed description of the invention The figure is a flow chart of an optional nut feeding control method according to an embodiment of the present invention.

[0068] like Figure 3As shown in the figure, after the material tray arrives at the fixed photographing station and nuts are placed on the pallet of the material tray, the robot can go to the first photographing station to use the visual camera to photograph the nuts at the current position. If the current photographing station is empty, or only one nut is photographed (when both grippers of the robot need to grab nuts, the visual camera should capture two nuts each time), or the nut at the current photographing station is abnormal, the visual camera can send an NG signal (abnormal signal), and the robot can move to the next photographing station after receiving the signal; if the visual camera sends an OK signal (normal signal), the visual camera can be used to transmit the captured image to the corresponding visual software or use its own embedded image processing program to process the captured image to obtain the position offset value and angle offset value of the target nut to be grabbed compared to the preset template nut. The robot and the visual camera communicate using SOCKET (socket) communication, and the machine As a client, a person sends a trigger command to the camera and defines a character as a communication command. Here, the communication is defined to use the letter 'A' as the command to trigger the visual computing process. The visual camera acts as a server to provide the robot with the required target value and target angle. Because the data sent by the visual camera is in string format, the robot must first establish three character variables to receive and store the data sent by the visual camera, and then convert them into real numbers as the required target values ​​through program instructions; then, the robot can adjust its posture based on the target value, first adjust from the current reference posture to the first target grasping posture of the first grasping nut for grasping, then lift it up and rotate it back to the reference posture, then adjust from the current reference posture to the second target grasping posture of the second grasping nut for grasping, and finally rotate it back to the reference posture to place the two grasped nuts in the required assembly position to complete the loading, thus ensuring that the two grasped nuts match the preset template nut state.

[0069] The overall final process is implemented in which the robot obtains the angle deviation value and position deviation value fed back by the visual camera, and changes the offset value of the photographing position each time to pick up and load the copper nuts. After verification under production conditions, the angle deviation of the nut loading is basically maintained within 1 degree, and all of them can be loaded into the transplanting clamp with high compatibility.

[0070] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0071] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0072] According to an embodiment of the present invention, a nut feeding control device for implementing the above-mentioned nut feeding control method is also provided. Figure 4 is a schematic diagram of a nut feeding control device according to an embodiment of the present invention, Figure 4 As shown, the device comprises: a first acquisition unit 41, a second acquisition unit 43, a third acquisition unit 45 and a control unit 47. The nut feeding control device is described in detail below.

[0073] The first acquisition unit 41 is used to acquire a nut image of a target nut in a target container when it is determined to start a feeding operation on the nut, wherein the target container is used to hold nuts, and the target nut is any nut in the target container.

[0074] The second acquisition unit 43 is used to compare and analyze the nut image and the template nut image by using the visual proofreading model to obtain the offset value of the target nut compared to the preset template nut, wherein the visual proofreading model is a model constructed by image calibration and feature matching of multiple groups of sample data using an image calibration method, and each of the multiple groups of sample data includes: a sample template nut image, and multiple sample nut images that need to be calibrated based on the sample template nut image. The template nut image is an image collected when the grasping component grasps the preset template nut with a preset reference posture, and the grasping component is a component that grasps the nut to perform a feeding operation.

[0075] The third acquisition unit 45 is used to control the grasping component to adjust its posture according to the offset value to obtain a target grasping posture of the grasping component.

[0076] The control unit 47 is used to control the grasping component to grasp the target nut in a target grasping posture and place the target nut at a target assembly position to complete the loading operation, wherein the target assembly position is a position where the target nut needs to be assembled.

[0077] It should be noted here that the above-mentioned first acquisition unit 41, second acquisition unit 43, third acquisition unit 45 and control unit 47 correspond to steps S202 to S208 in the above-mentioned embodiments, and the four units and the corresponding steps implement the same instances and application scenarios, but are not limited to the contents disclosed in the above-mentioned embodiments.

[0078] As can be seen from the above, in the scheme recorded in the above-mentioned embodiment of the present invention, the first acquisition unit can be used to acquire a nut image of a target nut in a target container when it is determined that a feeding operation is started on the nut, wherein the target container is used to hold nuts, and the target nut is any nut in the target container; then the second acquisition unit is used to compare and analyze the nut image and the template nut image using a visual calibration model to obtain an offset value of the target nut compared to a preset template nut, wherein the visual calibration model is a model constructed by image calibration and feature matching of multiple groups of sample data using an image calibration method, and each of the multiple groups of sample data includes: a sample template nut image, and multiple sample nut images that need to be calibrated based on the sample template nut image, and the template nut image is an image captured when the grasping component grasps the preset template nut with a preset reference posture, The grasping component is a component that grasps the nut to perform the feeding operation; then the third acquisition unit is used to control the grasping component to adjust its posture according to the offset value to obtain the target grasping posture of the grasping component; finally, the control unit is used to control the grasping component to grasp the target nut with the target grasping posture, and place the target nut at the target assembly position to complete the feeding operation, wherein the target assembly position is the position where the target nut needs to be assembled, thereby achieving the purpose of adjusting the grasping posture of the grasping component by comparing the deviation between the current placement image of the nut to be grasped and the image of the standard template nut, so that the grasping state of the grasping component grasping the nut is the same as the state of grasping the standard template nut, achieving the technical effect of accurately adjusting the posture of the grasping component when grasping the nut to avoid deviation in the nut placement position, and improving the accuracy of feeding the copper nut joint.

[0079] Therefore, the technical solution provided by the above-mentioned embodiment of the present invention solves the technical problem in the related art that the accuracy of feeding copper nut joints using traditional methods is difficult to meet the accuracy requirements, resulting in deviations in the nut placement position and easily causing safety hazards.

[0080] In an optional embodiment, the first acquisition unit includes: a first receiving module, used to receive position information transmitted by a grasping device based on a first communication channel, wherein the first communication channel is used for data transmission with the grasping device, the grasping device is used to perform a loading operation, and the grasping component is a component in the grasping device used to grasp nuts; a second receiving module, used to receive image information transmitted by an image acquisition device based on a second communication channel, wherein the second communication channel is used for data transmission with the grasping device, the image acquisition device is installed within a predetermined range of the grasping component and is used to capture images of nuts in a target container; a first determination module, used to determine to start a loading operation on the nuts when the position information indicates that the grasping device is in a predetermined position and the image information indicates that there is at least one nut in the target container.

[0081] In an optional embodiment, the nut feeding control device also includes: a first determination unit, which is used to determine that the current posture of the grasping component is a preset reference posture before comparing and analyzing the nut image and the template nut image using a visual proofreading model to obtain the offset value of the target nut compared to the preset template nut; a second determination unit, which is used to determine that the nut within the current grasping range of the grasping component is a preset template nut, wherein the current grasping range refers to the grasping range reached when the grasping component grasps with a preset reference posture; a fourth acquisition unit, which is used to acquire the first relative grasping point when the grasping component grasps the preset template nut with a preset reference posture, wherein the first relative grasping point refers to the contact point between the grasping component and the preset template nut when the grasping component grasps the preset template nut with a preset reference posture; a fifth acquisition unit, which is used to acquire the current image of the preset template nut when the first relative grasping point meets the preset grasping requirements; an input unit, which is used to input the current image as a template nut image into the visual proofreading model.

[0082] In an optional embodiment, the second acquisition unit includes: a first acquisition module, which is used to compare and analyze the nut image and the template nut image using a visual proofreading model to obtain an analysis result; a second determination module, which is used to determine, based on the analysis result, a position deviation between a first center point of a target nut in the nut image and a second center point of a preset template nut in the template nut image as a position offset value of the target nut compared to the preset template nut; a third determination module, which is used to determine, based on the analysis result, an angle deviation between a target nut in the nut image and a preset template nut in the template nut image as an angle offset value of the target nut compared to the preset template nut; and a fourth acquisition module, which is used to integrate the position offset value and the angle offset value to obtain an offset value of the target nut compared to the preset template nut.

[0083] In an optional embodiment, the fourth acquisition module includes: a first acquisition submodule, used to integrate the position offset value and the angle offset value to obtain a target offset value; and a second acquisition submodule, used to convert the format of the target offset value using a format conversion program to obtain an offset value.

[0084] In an optional embodiment, the third acquisition unit includes: a generation module, which is used to generate a posture adjustment strategy for adjusting the posture of the grasping component by using a preset algorithm according to the offset value; a fifth acquisition module, which is used to obtain the second relative grasping point when the grasping component grasps the target nut with the current grasping posture in the process of controlling the grasping component to adjust the posture according to the posture adjustment strategy, wherein the current grasping posture is the grasping posture of the grasping component at any moment in the posture adjustment process, and the second relative grasping point refers to the contact point between the grasping component and the target nut when the grasping component grasps the target nut with the current grasping posture; a fourth determination module, which is used to determine that the current grasping posture corresponding to the second relative grasping point is the target grasping posture of the grasping component when the second relative grasping point is the same as the first relative grasping point when the grasping component grasps the preset template nut with a preset reference posture.

[0085] In an optional embodiment, the grasping component includes a first grasping component and a second grasping component, the target nut includes a first target nut and a second target nut, the target container includes a first target container and a second target container, the preset reference posture includes a first preset reference posture and a second preset reference posture, the target grasping posture includes a first target grasping posture and a second target grasping posture, the target assembly position includes a first target assembly position and a second target assembly position, and the control unit includes: a fifth determination module, used to determine that the target grasping posture of the first grasping component grasping the first target nut is the first target grasping posture, and determine that the target grasping posture of the second grasping component grasping the second target nut is the second target grasping posture, wherein the first grasping component and the second grasping component are two different grasping components in the grasping device, the first target nut is any nut contained in the first target container, the second target nut is any nut contained in the second target container, and the first target container is used to contain the first grasping component. The first control module is used to control the grasping device to adjust its posture, and control the first grasping component to grasp the first target nut after adjusting the first grasping component to the first target grasping posture; the second control module is used to control the grasping device to restore to the reference posture after grasping the first target nut, wherein the reference posture is the posture of the grasping device when the first grasping component is in the first preset reference posture and the second grasping component is in the second preset reference posture; the third control module is used to control the grasping device to adjust its posture, and control the second grasping component to grasp the second target nut after adjusting the second grasping component to the second target grasping posture; the fourth control module is used to control the grasping device to place the first target nut in the first target assembly position and the second target nut in the second target assembly position after grasping the first target nut and the second target nut, so as to complete the loading operation.

[0086] According to another aspect of an embodiment of the present invention, a nut feeding control system is further provided, and the nut feeding control system uses any one of the above-mentioned nut feeding control methods.

[0087] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, the computer-readable storage medium comprising a stored program, wherein the program executes any one of the above-mentioned nut feeding control methods.

[0088] Optionally, in this embodiment, the computer-readable storage medium may be located in any one of the computer terminals in a computer terminal group in a computer network, or in any one of the communication devices in a communication device group.

[0089] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: when it is determined that a feeding operation is to be started on the nuts, a nut image of a target nut in a target container is obtained, wherein the target container is used to hold nuts, and the target nut is any nut in the target container; a visual calibration model is used to compare and analyze the nut image and the template nut image to obtain an offset value of the target nut compared to a preset template nut, wherein the visual calibration model is a model constructed by performing image calibration and feature matching on multiple sets of sample data using an image calibration method, and multiple sets of sample data are Each group of data includes: a sample template nut image, and multiple sample nut images that need to be calibrated based on the sample template nut image. The template nut image is an image captured when the grasping component grasps the preset template nut with a preset reference posture. The grasping component is a component that grasps the nut to perform a feeding operation; the grasping component is controlled to adjust its posture according to the offset value to obtain a target grasping posture of the grasping component; the grasping component is controlled to grasp the target nut with the target grasping posture, and the target nut is placed in the target assembly position to complete the feeding operation, wherein the target assembly position is the position where the target nut needs to be assembled.

[0090] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: receiving position information transmitted by a grasping device based on a first communication channel, wherein the first communication channel is used for data transmission with the grasping device, the grasping device is used for performing a loading operation, and the grasping component is a component in the grasping device for grasping nuts; receiving image information transmitted by an image acquisition device based on a second communication channel, wherein the second communication channel is used for data transmission with the grasping device, the image acquisition device is installed within a predetermined range of the grasping component and is used to capture images of nuts in a target container; when the position information indicates that the grasping device is in a predetermined position and the image information indicates that there is at least one nut in the target container, determining to start a loading operation on the nuts.

[0091] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: determining that the current posture of the grasping component is a preset reference posture; determining that the nut within the current grasping range of the grasping component is a preset template nut, wherein the current grasping range refers to the grasping range reached when the grasping component grasps with a preset reference posture; obtaining the first relative grasping point when the grasping component grasps the preset template nut with a preset reference posture, wherein the first relative grasping point refers to the contact point between the grasping component and the preset template nut when the grasping component grasps the preset template nut with a preset reference posture; obtaining the current image of the preset template nut when the first relative grasping point meets the preset grasping requirements; and inputting the current image as the template nut image into the visual proofreading model.

[0092] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: using a visual proofreading model to compare and analyze the nut image and the template nut image to obtain an analysis result; based on the analysis result, determining the position deviation between the first center point of the target nut in the nut image and the second center point of the preset template nut in the template nut image as the position offset value of the target nut compared to the preset template nut; based on the analysis result, determining the angle deviation between the target nut in the nut image and the preset template nut in the template nut image as the angle offset value of the target nut compared to the preset template nut; integrating the position offset value and the angle offset value to obtain the offset value of the target nut compared to the preset template nut.

[0093] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: integrating the position offset value and the angle offset value to obtain a target offset value; and converting the format of the target offset value using a format conversion program to obtain an offset value.

[0094] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: generating a posture adjustment strategy for adjusting the posture of the grasping component using a preset algorithm according to the offset value; in the process of controlling the grasping component to adjust the posture according to the posture adjustment strategy, obtaining a second relative grasping point when the grasping component grasps the target nut with the current grasping posture, wherein the current grasping posture is the grasping posture of the grasping component at any moment in the posture adjustment process, and the second relative grasping point refers to the contact point between the grasping component and the target nut when the grasping component grasps the target nut with the current grasping posture; when the second relative grasping point is the same as the first relative grasping point when the grasping component grasps the preset template nut with the preset reference posture, determine the current grasping posture corresponding to the second relative grasping point as the target grasping posture of the grasping component.

[0095] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: determining that the target grasping posture of the first grasping component grasping the first target nut is the first target grasping posture, and determining that the target grasping posture of the second grasping component grasping the second target nut is the second target grasping posture, wherein the first grasping component and the second grasping component are two different grasping components in the grasping device, the first target nut is any nut contained in the first target container, the second target nut is any nut contained in the second target container, the first target container is used to hold the nut that the first grasping component needs to grasp, and the second target container is used to hold the nut that the second grasping component needs to grasp; controlling the grasping device to perform posture Adjust, and after adjusting the first grasping component to the first target grasping posture, control the first grasping component to grasp the first target nut; after grasping the first target nut, control the grasping device to return to the reference posture, wherein the reference posture is the posture of the grasping device when the first grasping component is in the first preset reference posture and the second grasping component is in the second preset reference posture; control the grasping device to adjust the posture, and after adjusting the second grasping component to the second target grasping posture, control the second grasping component to grasp the second target nut; after grasping the first target nut and the second target nut, control the grasping device to place the first target nut in the first target assembly position and the second target nut in the second target assembly position to complete the loading operation.

[0096] According to another aspect of an embodiment of the present invention, a processor is further provided, and the processor is used to run a program, wherein when the program is run, any one of the above-mentioned nut feeding control methods is executed.

[0097] According to another aspect of an embodiment of the present invention, there is further provided a computer program product, comprising computer instructions, wherein when the computer instructions are executed by a processor, any one of the above-mentioned nut feeding control methods is executed.

[0098] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0099] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0100] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0101] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0102] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0103] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0104] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for controlling the feeding of nuts, characterized in that: include: In the case of determining to start performing a feeding operation on nuts, obtaining a nut image of a target nut in a target container, wherein the target container is used to hold the nuts, and the target nut is any one of the nuts in the target container; The nut image and the template nut image are compared and analyzed by using a visual proofreading model to obtain an offset value of the target nut compared to the preset template nut, wherein the visual proofreading model is a model constructed by performing image calibration and feature matching on multiple groups of sample data using an image calibration method, and each of the multiple groups of sample data includes: a sample template nut image, and multiple sample nut images that need to be calibrated based on the sample template nut image, and the template nut image is an image captured when a grasping component grasps a preset template nut with a preset reference posture, and the grasping component is a component that grasps the nut to perform the feeding operation; Controlling the grasping component to adjust its posture according to the offset value to obtain a target grasping posture of the grasping component; The grasping assembly is controlled to grasp the target nut in the target grasping posture, and the target nut is placed in a target assembly position to complete the feeding operation, wherein the target assembly position is a position where the target nut needs to be assembled.

2. The nut feeding control method according to claim 1, characterized in that: Confirm to start the feeding operation on the nut, including: Receiving position information transmitted by a gripping device based on a first communication channel, wherein the first communication channel is used to transmit data with the gripping device, the gripping device is used to perform the feeding operation, and the gripping component is a component in the gripping device used to grip the nut; receiving image information transmitted by an image acquisition device based on a second communication channel, wherein the second communication channel is used to transmit data with the grasping device, the image acquisition device is installed within a predetermined range of the grasping component and is used to acquire an image of the nut in the target container; When the position information indicates that the gripping device is at a predetermined position and the image information indicates that at least one nut exists in the target container, it is determined to start the feeding operation on the nut.

3. The nut feeding control method according to claim 1, characterized in that: Before comparing and analyzing the nut image and the template nut image using the visual proofreading model to obtain the offset value of the target nut compared to the preset template nut, the method further includes: Determining that the current posture of the grasping component is the preset reference posture; Determine that the nut within the current grasping range of the grasping component is the preset template nut, wherein the current grasping range refers to the grasping range reached when the grasping component grasps in the preset reference posture; Obtaining a first relative grasping point when the grasping component grasps the preset template nut in the preset reference posture, wherein the first relative grasping point refers to a contact point between the grasping component and the preset template nut when the grasping component grasps the preset template nut in the preset reference posture; When the first relative grasping point position meets the preset grasping requirement, acquiring the current image of the preset template nut; The current image is input into the visual proofreading model as the template nut image.

4. The nut feeding control method according to claim 1, characterized in that: The nut image and the template nut image are compared and analyzed by using a visual proofreading model to obtain an offset value of the target nut compared to a preset template nut, including: Using a visual proofreading model to compare and analyze the nut image and the template nut image to obtain an analysis result; Determine, based on the analysis result, a position deviation between a first center point of the target nut in the nut image and a second center point of the preset template nut in the template nut image as a position offset value of the target nut compared to the preset template nut; Determine, based on the analysis result, the angle deviation between the target nut in the nut image and the preset template nut in the template nut image as the angle offset value of the target nut compared to the preset template nut; The position offset value and the angle offset value are integrated to obtain the offset value of the target nut compared to the preset template nut.

5. The nut feeding control method according to claim 4, characterized in that: Integrating the position offset value and the angle offset value to obtain the offset value of the target nut compared to the preset template nut includes: Integrate the position offset value and the angle offset value to obtain a target offset value; The target offset value is format-converted using a format conversion program to obtain the offset value.

6. The nut feeding control method according to claim 1, characterized in that: Controlling the grasping component to adjust its posture according to the offset value to obtain a target grasping posture of the grasping component includes: Generate a posture adjustment strategy for adjusting the posture of the grasping component by using a preset algorithm according to the offset value; In the process of controlling the grasping component to adjust its posture according to the posture adjustment strategy, a second relative grasping point position when the grasping component grasps the target nut with the current grasping posture is obtained, wherein the current grasping posture is the grasping posture of the grasping component at any moment in the posture adjustment process, and the second relative grasping point position refers to the contact point between the grasping component and the target nut when the grasping component grasps the target nut with the current grasping posture; When the second relative grasping point is the same as the first relative grasping point when the grasping component grasps the preset template nut in the preset reference posture, the current grasping posture corresponding to the second relative grasping point is determined as the target grasping posture of the grasping component.

7. The nut feeding control method according to claim 1, characterized in that: The grasping assembly includes a first grasping assembly and a second grasping assembly, the target nut includes a first target nut and a second target nut, the target container includes a first target container and a second target container, the preset reference posture includes a first preset reference posture and a second preset reference posture, the target grasping posture includes a first target grasping posture and a second target grasping posture, the target assembly position includes a first target assembly position and a second target assembly position, and the grasping assembly is controlled to grasp the target nut with the target grasping posture and place the target nut at the target assembly position to complete the feeding operation, including: Determine that the target grasping posture of the first grasping component grasping the first target nut is the first target grasping posture, and determine that the target grasping posture of the second grasping component grasping the second target nut is the second target grasping posture, wherein the first grasping component and the second grasping component are two different grasping components in the grasping device, the first target nut is any one of the nuts contained in the first target container, the second target nut is any one of the nuts contained in the second target container, the first target container is used to contain the nuts that the first grasping component needs to grasp, and the second target container is used to contain the nuts that the second grasping component needs to grasp; Controlling the grasping device to adjust the posture, and controlling the first grasping component to grasp the first target nut after adjusting the first grasping component to the first target grasping posture; After grasping the first target nut, controlling the grasping device to return to a reference posture, wherein the reference posture is the posture of the grasping device when the first grasping component is in a first preset reference posture and the second grasping component is in a second preset reference posture; Controlling the grasping device to adjust the posture, and controlling the second grasping component to grasp the second target nut after adjusting the second grasping component to the second target grasping posture; After grasping the first target nut and the second target nut, the grasping device is controlled to place the first target nut at the first target assembly position and the second target nut at the second target assembly position to complete the loading operation.

8. A nut feeding control device, characterized in that: include: A first acquisition unit is used to acquire a nut image of a target nut in a target container when it is determined that a feeding operation is to be started on the nut, wherein the target container is used to hold the nut, and the target nut is any one of the nuts in the target container; A second acquisition unit is used to compare and analyze the nut image and the template nut image by using a visual proofreading model to obtain an offset value of the target nut compared to a preset template nut, wherein the visual proofreading model is a model constructed by performing image calibration and feature matching on multiple groups of sample data using an image calibration method, and each of the multiple groups of sample data includes: a sample template nut image, and multiple sample nut images that need to be calibrated based on the sample template nut image, the template nut image is an image captured when a grasping component grasps a preset template nut with a preset reference posture, and the grasping component is a component that grasps the nut to perform the feeding operation; A third acquisition unit, configured to control the grasping component to adjust its posture according to the offset value, so as to obtain a target grasping posture of the grasping component; A control unit is used to control the grasping assembly to grasp the target nut in the target grasping posture and place the target nut at a target assembly position to complete the loading operation, wherein the target assembly position is a position where the target nut needs to be assembled.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the nut feeding control method according to any one of claims 1 to 7.

10. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by the processor, the nut feeding control method described in any one of claims 1 to 7 is performed.

Images