Punching system and method and electronic equipment

Abstract

The invention discloses a punching system and method and electronic equipment, relates to the technical field of machine vision, and aims to solve the problem of low punching efficiency of materials. The system comprises a first camera used for photographing materials on a transmission device when the materials pass through a first acquisition range corresponding to the first camera to obtain a first image; the second camera is used for taking a picture when the material on the transmission equipment passes through a second acquisition range corresponding to the second camera to obtain a second image; the processing unit is in communication connection with the first camera and the second camera and used for obtaining the first image and the second image, and the first punching position and the second punching position of the target material are obtained by processing the first image and the second image correspondingly; and the processing unit is further connected with the punching device and used for determining the first punching position or the second punching position as a target punching position and controlling the punching device to punch the target material according to the target punching position.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a punching system, method and electronic device. Background Art

[0002] In the production process of materials, the materials need to be hoisted and loaded after painting, cleaning and other processes. Before hoisting, the materials need to be punched for hanging or carrying. The existing punching method supports the punching of a single type of material, which cannot meet the inspection requirements of the material shapes that are varied, complex and mixed at the operation site. Therefore, manual punching is generally used to meet the punching requirements of materials of different shapes. However, manual punching is time-consuming and labor-intensive, and the punching efficiency is low. Summary of the invention

[0003] The present application provides a punching system, method and electronic device for solving the problem of low punching efficiency for materials.

[0004] In order to achieve the above technical objectives, this application adopts the following technical solutions:

[0005] In a first aspect, an embodiment of the present application provides a punching system, including a processing unit, a punching device, and a first camera and a second camera sequentially arranged along a transmission direction of a transmission device;

[0006] The first camera is used to take a picture of the material on the transmission device when it passes through a first acquisition range corresponding to the first camera to obtain a first image;

[0007] The second camera is used to take a picture of the material on the transmission device when it passes through a second acquisition range corresponding to the second camera to obtain a second image;

[0008] a processing unit, which is in communication with the first camera and the second camera, and is used to acquire the first image and the second image, and obtain the first punching position and the second punching position of the target material by processing the first image and the second image respectively; the target material is the material indicated in the second image;

[0009] The processing unit is also connected to the punching device, and is used to determine the first punching position or the second punching position as the target punching position, and control the punching device to punch the target material according to the target punching position.

[0010] In one possible implementation, the first acquisition range is smaller than the second acquisition range; the processing unit is specifically used to determine the first punching position as the target punching position when the difference between the first punching position and the second punching position is smaller than a preset threshold; otherwise, determine the second punching position as the target punching position.

[0011] In a possible implementation, the first camera and the second camera are used to take pictures of the cross section of the material on the transmission device; the distance from the second camera to the transmission device is greater than the distance from the first camera to the transmission device.

[0012] In a possible implementation, the punching system further includes a first sensing unit and / or a second sensing unit;

[0013] The first sensing unit is used to trigger the first camera to take pictures again if it detects the X+1th material passing by after the first camera was triggered to take pictures last time; X is the number of materials contained in the first acquisition range; the second sensing unit is used to trigger the second camera to take pictures again if it detects the Y+1th material passing by after the second camera was triggered to take pictures last time; Y is the number of materials contained in the second acquisition range.

[0014] In a possible implementation, X and Y are preset values; or, X is the number of materials within the first acquisition range determined when the first camera took a photo last time, and Y is the number of materials within the second acquisition range determined when the second camera took a photo last time.

[0015] In one possible implementation, a data queue is provided in the processing unit; the data queue is used to store the first punching position of each material in the first image in sequence according to the order in which the materials pass through the first acquisition range; the processing device is also used to obtain the same number of first punching positions from the data queue according to the number of materials in the second image, and determine the target punching position of each material in sequence according to the rule that the first punching position taken out first matches the material that passes through the second acquisition range first.

[0016] In one possible implementation, the processing unit is specifically used to determine a first center of gravity position of the material in the vertical direction based on pixel values ​​obtained by projecting the material in the target image in the horizontal direction; the target image is the first image or the second image; based on pixel values ​​obtained by projecting the material in the target image in the vertical direction, determine a second center of gravity position of the material in the horizontal direction; based on the first center of gravity position and the second center of gravity position, obtain the center of gravity position of the material as the punching position of the material in the target image.

[0017] In a possible implementation, the processing unit is specifically used to identify the area where the holes of the material are located in the target image; the target image is the first image or the second image;

[0018] Based on the region, the center of gravity of the hole is determined as the punching position of the material in the target image.

[0019] The embodiment of the present application provides a punching system, which includes a processing unit, a punching device, and a first camera and a second camera arranged in sequence along the transmission direction of a transmission device. The transmission device is used to transmit materials, and the first camera is used to take pictures of materials on the transmission device that pass through a first acquisition range to obtain a first image. The second camera is used to take pictures of materials on the transmission device that pass through a second acquisition range to obtain a second image. The processing unit is used to process the first image and the second image respectively to obtain a target punching position of the target material, and control the punching device to punch the target material according to the target punching position. It can be seen that the technical solution of the embodiment of the present application improves the punching efficiency by automatically identifying the punching position of the material through the mutual interaction of the above-mentioned components and realizing automatic punching. In the present application, the material is photographed by a camera to determine the punching position based on the image obtained by taking the picture. The materials are not required to have the same shape, and can be applied to scenes where the material shapes are varied, the types are complex, and the materials are mixed. In addition, by setting two cameras along the transmission direction, the same material can be photographed separately to obtain two sets of punching position data, and then the material can be punched based on the comparison of the two sets of punching position data, avoiding the problem of possible inaccuracy of a single punching position data and improving the fault tolerance rate of material punching.

[0020] In a second aspect, the present application provides a punching method, which is applied to a processing unit of a punching system, wherein the punching system also includes a punching device, and a first camera and a second camera sequentially arranged along a transmission direction of a transmission device; the first camera is used to take a picture of a material on the transmission device when it passes through a first acquisition range corresponding to the first camera, obtain a first image, and send it to the processing unit; the second camera is used to take a picture of a material on the transmission device when it passes through a second acquisition range corresponding to the second camera, obtain a second image, and send it to the processing unit; the method includes: receiving the first image and extracting a first punching position of the material in the first image; receiving the second image and extracting a second punching position of the target material in the second image; obtaining a target first punching position corresponding to the target material; determining the target first punching position or the second punching position as the target punching position, and controlling the punching device to punch the target material according to the target punching position.

[0021] In one possible implementation, the first acquisition range is smaller than the second acquisition range; the target first punching position or the second punching position is determined as the target punching position, including: when the difference between the target first punching position and the second punching position is smaller than a preset threshold, the first punching position is determined as the target punching position; otherwise, the second punching position is determined as the target punching position.

[0022] In a possible implementation, the first camera and the second camera are used to take pictures of the cross section of the material on the transmission device; the distance from the second camera to the transmission device is greater than the distance from the first camera to the transmission device.

[0023] In a possible implementation, the punching system also includes a first sensing unit and / or a second sensing unit; the first sensing unit is used to trigger the first camera to take pictures again if it detects that the X+1th material has passed after the first camera was triggered to take pictures last time; X is the number of materials accommodated in the first acquisition range; the second sensing unit is used to trigger the second camera to take pictures again if it detects that the Y+1th material has passed after the second camera was triggered to take pictures last time; Y is the number of materials accommodated in the second acquisition range.

[0024] In a possible implementation, X and Y are preset values; or, X is the number of materials within the first acquisition range determined when the first camera took a photo last time, and Y is the number of materials within the second acquisition range determined when the second camera took a photo last time.

[0025] In one possible implementation, the method also includes: storing the first punching positions of the materials in the first image in the data queue in sequence according to the order in which the materials pass through the first acquisition range; obtaining the target first punching positions corresponding to the target materials, including: obtaining the same number of first punching positions from the data queue according to the number of materials in the second image, and obtaining the target first punching positions corresponding to each target material in sequence according to the rule that the first punching positions taken out first match the materials that pass through the second acquisition range first.

[0026] In one possible implementation, extracting the punching position of a material in a target image includes: determining a first center of gravity position of the material in a vertical direction based on pixel values ​​obtained by projecting the material in the target image in a horizontal direction; the target image is a first image or a second image; determining a second center of gravity position of the material in a horizontal direction based on pixel values ​​obtained by projecting the material in the target image in a vertical direction; and obtaining the center of gravity position of the material as the punching position of the material in the target image based on the first center of gravity position and the second center of gravity position.

[0027] In a possible implementation, extracting the punching position of the material in the target image includes: identifying the area where the hole of the material in the target image is located; the target image is the first image or the second image; based on the area, determining the center of gravity position of the hole as the punching position of the material in the target image.

[0028] In a third aspect, the present application provides an electronic device comprising: one or more processors; one or more memories; wherein the one or more memories are used to store computer program codes, the computer program codes include computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes any one of the punching methods provided in the second aspect above.

[0029] In a fourth aspect, the present application provides a computer-readable storage medium, which stores computer execution instructions. When the computer execution instructions are executed on a computer, the computer executes any one of the punching methods provided in the second aspect above.

[0030] In a fifth aspect, the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the punching method as described in the second aspect and any possible design thereof.

[0031] For the specific description of the second to fifth aspects and their various implementations in the present application, reference may be made to the detailed description in the first aspect and its various implementations; and for the beneficial effects of the second to sixth aspects and their various implementations, reference may be made to the analysis of the beneficial effects in the first aspect and its various implementations, which will not be repeated here.

[0032] These and other aspects of the present application will become more apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of the composition of a punching system provided in an embodiment of the present application;

[0034] Figure 2 A schematic diagram of another punching system provided in an embodiment of the present application;

[0035] Figure 3 A schematic diagram of punching positions on a material provided in an embodiment of the present application;

[0036] Figure 4 A schematic diagram of a projection method provided in an embodiment of the present application;

[0037] Figure 5 A schematic diagram of a hole method provided in an embodiment of the present application;

[0038] Figure 6 A schematic diagram of a punching result provided in an embodiment of the present application;

[0039] Figure 7 A schematic diagram of a process of a punching method provided in an embodiment of the present application;

[0040] Figure 8 A schematic diagram of a projection method provided in an embodiment of the present application;

[0041] Fig. 9 A schematic diagram of a process of a hole method provided in an embodiment of the present application;

[0042] Fig.10A schematic diagram of a processing flow of a first workstation provided in an embodiment of the present application;

[0043] Fig.11 A schematic diagram of a processing flow of a second workstation provided in an embodiment of the present application;

[0044] Fig.12 A schematic diagram of the composition of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0046] It should be noted that, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way. The terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise specified, "multiple" means two or more.

[0047] As described in the background art, the current punching method is only applicable to the punching of fixed-shaped materials (such as steel), and is not applicable to the punching scene of mixed placement of materials with variable shapes. If the manual punching method is used, it is time-consuming and labor-intensive, and the punching efficiency is not high.

[0048] In this regard, an embodiment of the present application provides a punching system, which includes a processing unit, a punching device, and a first camera and a second camera arranged in sequence along the transmission direction of a transmission device. Among them, the transmission device is used to transmit materials, and the first camera is used to take pictures of materials passing through a first acquisition range on the transmission device to obtain a first image. The second camera is used to take pictures of materials passing through a second acquisition range on the transmission device to obtain a second image. The processing unit is used to process the first image and the second image respectively to obtain a target punching position of the target material, and control the punching device to punch the target material according to the target punching position. It can be seen that the technical solution of the embodiment of the present application improves the punching efficiency by automatically identifying the punching position of the material through the mutual interaction of the above-mentioned components and realizing automatic punching. In the present application, the material is photographed by a camera to determine the punching position based on the image obtained by taking the picture. The material is not required to have the same shape, and can be applied to scenes with variable material shapes, complex types and mixed placement. In addition, by setting two cameras along the transmission direction, the same material can be photographed separately to obtain two sets of punching position data, and then the material can be punched based on the comparison of the two sets of punching position data, avoiding the problem of possible inaccuracy of a single punching position data and improving the fault tolerance rate of material punching.

[0049] refer to Figure 1 , which shows a schematic diagram of the composition of a punching system provided in an embodiment of the present application. Figure 1 As shown, it includes a transmission device 101 (also referred to as a carrier), a first camera 1021 and a second camera 1022, and a punching device 103, which are sequentially arranged along the transmission direction of the transmission device 101. Figure 1 The setting position of the first camera can be called the first station, and the setting position of the second camera can be called the second station.

[0050] In addition, the punching system may further include a processing unit 104 (not shown in the figure). The processing unit may communicate with the Figure 1 The processing unit is connected to other components in the punching system (such as the first camera 1021, the second camera 1022, etc.) to control the cooperation between the components in the punching system. The processing unit mentioned here can be a server, or other equipment with data processing capabilities. Among them, the server mentioned here can be a server cluster composed of multiple servers, or a single server, or a computer. The processing unit can specifically be a processor in a server, etc. The embodiment of the present application does not limit the specific device form of the above-mentioned server.

[0051] The first camera 1021 is used to take a picture of the material on the transmission device when it passes through a first acquisition range corresponding to the first camera 1021 to obtain a first image.

[0052] The second camera 1022 is used to take a picture of the material on the transmission device when it passes through a second acquisition range corresponding to the second camera 1022 to obtain a second image.

[0053] Among them, the first camera mentioned here can be a camera deployed at the first station, or it can be multiple cameras deployed at the first station. Similarly, the second camera mentioned here can be a camera deployed at the second station, or it can be multiple cameras deployed at the second station. Taking the first station as an example, when multiple cameras are deployed at the first station, one image can be selected from the images obtained by taking pictures with multiple cameras as the above-mentioned first image, or the images obtained by taking pictures with multiple cameras can be fused to obtain a higher quality first image. The embodiment of the present application does not limit this specific implementation method, and the following description will be taken as an example in which the first camera and the second camera are both one camera.

[0054] The processing unit 104 is connected to the first camera 1021 and the second camera 1022 for acquiring the first image and the second image, and obtaining the first punching position and the second punching position of the target material by processing the first image and the second image respectively; the target material is the material indicated in the second image;

[0055] The processing unit 104 is also connected to the punching device 103, and is further used to determine the first punching position or the second punching position as the target punching position, and control the punching device 103 to punch the target material according to the target punching position.

[0056] The working range of the punching device 103 can cover the second collection range, so that the punching device 103 can punch holes in the target material. Figure 2 As shown, the punching device 103 can be arranged in the second station.

[0057] In some embodiments, the punching device 103 may be a robotic arm.

[0058] It should be understood that the technical solution of the embodiment of the present application can realize automatic identification of the punching position of the material and automatic punching through the interaction of the above components, thereby improving the punching efficiency. In addition, in the present application, the material is photographed by a camera to determine the punching position based on the image obtained by the photograph, and the material is not required to have the same shape, which can be applied to scenes with variable material shapes, complex types and mixed placement.

[0059] In addition, by setting up two cameras to take pictures of the materials transported on the transmission equipment, two sets of punching positions (a first punching position and a second punching position) can be generated for the same material for comparison, thereby improving the fault tolerance of punching.

[0060] In some embodiments, Figure 2As shown, the punching system further includes a first sensing unit 1051 and / or a second sensing unit 1502; the sensing unit is used to detect the passage of the material and trigger the camera to take a picture.

[0061] The first sensor unit 1051 is used to trigger the first camera 1021 to take photos again if it detects that the X+1th material has passed after the first camera was triggered to take photos last time; X is the number of materials contained in the first acquisition range.

[0062] The second sensor unit 1052 is used to trigger the second camera 1022 to take pictures again if it detects that the Y+1th material has passed after the second camera was triggered to take pictures last time; Y is the number of materials contained in the second acquisition range.

[0063] It should be noted that the embodiment of the present application is to take photos of materials in batches and punch holes, that is, after taking photos to obtain the second image, the materials indicated in the second image are punched in batches. Therefore, in order to ensure the correct execution of the solution, the sensor unit does not trigger the camera to take photos every time it detects that a material has passed by, but based on certain rules, it will trigger the camera to take photos again after determining that the current batch of materials has passed by.

[0064] For example, for the second sensing unit, there are 4 materials (e.g., marked as materials ad) within the current second acquisition range. After the second camera takes pictures, the second punching positions of materials ad can be obtained respectively. In this way, it is necessary to wait for these 4 materials to pass through before the second camera takes pictures of the next batch of materials. That is, after the second sensing unit has cumulatively detected 4 materials passing through, when it detects the 5th (4+1) material, it means that material ad has been processed, and the camera is triggered again to take pictures.

[0065] In a possible implementation, X and Y are preset values. In this case, there are certain requirements for the rules for placing the materials on the transmission device, and it is necessary to ensure that the number of materials within the collection range is fixed each time a photo is taken.

[0066] In another possible implementation, X is the number of materials within the first acquisition range determined when the first camera took a photo last time, and Y is the number of materials within the second acquisition range determined when the second camera took a photo last time. In this case, even if the number of materials within the acquisition range is not fixed each time, the camera can automatically identify the number of materials within the current acquisition range by taking photos to determine the specific values ​​of X and Y. It can be seen that in this way, there are not too many restrictions on the rules for users to place materials, thereby improving the user experience.

[0067] In some embodiments, the above-mentioned sensing unit can be a photoelectric sensor or an infrared sensor, and the embodiments of the present application do not impose specific limitations on this.

[0068] It should be understood that by providing a sensor unit, automatic photo triggering of the cameras at the two workstations can be achieved without manual intervention, thereby ensuring the automated operation of the punching system.

[0069] In some embodiments, after acquiring the image taken by the first camera 1021 or the second camera 1022, the processing unit 104 can pre-process the image (for example, locating the position of the material in the image, correcting the image size, cropping, binarization, etc.), remove irrelevant information in the image, and retain key information, so that the processing unit 104 can determine the punching position of the material based on the preprocessed image.

[0070] In some embodiments, the punching position may be represented by coordinate data. The coordinate data may include: a horizontal coordinate, a vertical coordinate, and an angle at which the punching direction deviates from a preset direction. The preset direction may be a preset direction, such as a horizontal direction or a vertical direction.

[0071] For example, the coordinate data can be expressed as (x, y, r), where x is the horizontal coordinate, used to represent the horizontal distance from the punching position to the preset origin. y is the vertical coordinate, used to represent the vertical distance from the punching position to the preset origin. r is the angle at which the punching direction deviates from the preset direction. For example, Figure 3 A schematic diagram of punching positions on a material provided in an embodiment of the present application.

[0072] It should be noted that the coordinate data determined by the processing unit 104 based on the image may be relative coordinates, such as coordinates relative to the lower left corner of the material.

[0073] In some embodiments, before the processing unit 104 controls the punching device 103 to punch holes based on the target punching position, the coordinate data of each material can be converted from relative coordinates to world coordinates, and then the coordinate data is sent to the punching device 103, so that the punching device 103 punches holes based on the world coordinates. In this way, the punching device does not need to determine the reference position of each material (for example, the lower left corner), and can also accurately determine the punching position, thereby achieving accurate punching and improving efficiency.

[0074] It should be understood that the specific process of converting relative coordinates into world coordinates is not limited in the embodiments of the present application, such as realizing the conversion of relative coordinates to world coordinates by a nine-point calibration matrix or a twelve-point calibration matrix. Among them, nine-point calibration and twelve-point calibration are commonly used camera calibration methods in the field of machine vision and image processing, which use specific dot patterns to determine the internal parameters (such as focal length, principal point coordinates) and external parameters (such as the position and direction of the camera relative to the world coordinate system) of the camera, respectively. Nine-point calibration generally refers to the use of a 3x3 checkerboard pattern, which consists of 9 inner points (i.e., the intersection of the checkerboard). By taking multiple pictures containing this checkerboard from different angles and positions, sufficient information can be obtained to calculate the parameters of the camera. Twelve-point calibration generally refers to the use of a 4x3 checkerboard pattern, which consists of 12 inner points (i.e., the intersection of the checkerboard). Similar to nine-point calibration, twelve-point calibration also requires taking multiple pictures from different angles and positions to obtain camera parameters.

[0075] In some embodiments, the first acquisition range is smaller than the second acquisition range; the processing unit 104 is specifically used to determine the first punching position as the target punching position when the difference between the first punching position and the second punching position is smaller than a preset threshold; otherwise, determine the second punching position as the target punching position.

[0076] Specifically, when the punching position is represented by coordinates, the difference can be determined by calculating the Euclidean distance between two coordinate points.

[0077] It should be noted that for the single-camera / station punching solution, a camera is used to take a picture of the material on the transmission device and then punch holes. If you want to improve the punching efficiency, you need to set the camera farther away from the transmission device. The camera field of view becomes larger, and more materials can be photographed at one time, so more materials can be punched. If you want to improve the punching accuracy, you need to set the camera closer to the transmission device to improve the accuracy of the photo, but this will take less material at one time, which will reduce the punching efficiency.

[0078] In the embodiment of the present application, by setting the first acquisition range to be smaller than the second acquisition range, it means that the first image obtained by the first camera has a higher display accuracy of the material, and therefore the accuracy of the first punching position obtained based on the first image is also higher. By comparing the first punching position and the second punching position, if the difference between the two is small, it means that the first punching position and the material to be punched are matched, so the first punching position with higher accuracy is preferentially used for punching. If the difference between the two is large, it means that the first camera may have missed the material in the process of moving the material from the first acquisition range to the second acquisition range, resulting in the mismatch between the first punching position and the material to be punched. In this case, the second punching position obtained from the second image is used to punch the material indicated in the second image, so that correct punching can be ensured.

[0079] In addition, the punching device punches the indicated material in the second image. Since the second acquisition range is large, the number of materials in the second image obtained by photographing the second acquisition range is also large, so the punching positions of more materials can be obtained in one photograph, and then more materials can be batch-punched, thereby improving the punching efficiency. In other words, the present application provides a double-station / camera punching solution that can improve the punching accuracy while ensuring the punching efficiency.

[0080] In some embodiments, the first camera 1021 and the second camera 1022 are used to take pictures of the cross section of the material on the transmission device; the distance from the second camera 1022 to the transmission device is greater than the distance from the first camera 1021 to the transmission device.

[0081] It should be understood that if Figure 1 As shown, the first camera 1021 is located closer to the transmission device, so the field of view is smaller, but the accuracy of the first image captured at the first acquisition position is higher, and the punching coordinates determined based on the first image are more accurate. The second camera 1022 is located farther from the transmission device, so the field of view is larger, and more materials can be captured at the same time in one shot. Compared with the first camera 1021, the possibility of missing materials can be effectively reduced.

[0082] In some embodiments, a data queue is provided in the processing unit 104. The data queue is used to sequentially store the first punching position of each material in the first image according to the order in which the materials pass through the first acquisition range.

[0083] The processing device is also used to obtain the same number of first punching positions from the data queue according to the number of materials in the second image, and determine the target punching position of each material in turn according to the rule that the first punching position taken out first matches the material that passes through the second acquisition range first.

[0084] The data queue is a data format with a first-in-first-out feature, that is, the data first stored in the data queue is taken out and used first.

[0085] Exemplarily, if there are materials 1 to 6 that have passed through the first acquisition range in sequence, the processing unit 104 determines the first punching position of each material and stores it in the data queue, which is represented as [coordinate 1, coordinate 2, coordinate 3, coordinate 4, coordinate 5, coordinate 6]. When these six materials pass through the second acquisition range, the processing unit 104 determines the second punching position of each material in the order of materials 1 to 6, and takes out 6 groups of first punching positions from the data queue in sequence according to the quantity. The first punching position (coordinate 1) taken out first matches the material that passes through the second acquisition range first (material 1 on the far right of the transmission device). Similarly, coordinate 2 matches material 2, and so on. The first punching position and the second punching position of each material in the second image are compared, and finally the target punching position of each material is determined.

[0086] It is understandable that in the process of comparing the first punching position and the second punching position, in order to ensure that the two sets of coordinates correspond to the same material, the conventional solution is to configure the material identification in advance (such as the identified material shape as the identification) to determine the correct matching of the first punching position and the second punching position of the material based on the identification. However, the number and shape of materials are in the tens of thousands. If an identification is set for each material, the storage resources and computing resources of the processing unit 104 will be greatly demanded. Therefore, the present application configures the data queue to store the first punching position, and by utilizing the first-in-first-out feature of the data queue, according to the rule that the first punching position taken out first matches the material that passes through the second acquisition range first, the first punching position and the second punching position can be matched to the same material. In this way, the data queue only needs to store the first punching position of the material, and there is no need to configure the material identification, which can greatly save the storage resources and computing resources of the processing unit 104.

[0087] In some scenarios, if the first camera 1021 of the first station misses to identify the material, such as only identifying material 1, material 3 and material 4, but missing to identify material 2. In this case, the processing unit 104 will mistakenly match the coordinate 3 corresponding to material 3 to material 2, and the coordinate 4 corresponding to material 4 to material 3, and so on, there will be a problem of data confusion, affecting the accuracy of subsequent punching. Therefore, in the scheme of the embodiment of the present application, the second camera 1022 of the second station will take pictures again to obtain the second punching position, and compare it with the first punching position obtained by the first station to comprehensively determine the target punching position, rather than directly punching according to the first punching position of the first station, so as to ensure that even if the first camera 1021 misses to identify the material and causes data confusion, the second camera 1022 can determine the second punching position again to ensure the accuracy of punching.

[0088] In addition, in the process of comparing the first punching position and the second punching position, if it is determined that the difference between the two is greater than the preset threshold, it means that a data confusion problem has occurred. Therefore, the processing unit 104 can generate a prompt message to prompt the user to remove the unpunched material on the transmission device, clear the data queue, and then put the material back on the transmission device for punching to ensure the consistency of the data generated by the first workstation and the second workstation.

[0089] The following is an explanation of the process of determining the punching position (coordinates), which is specifically divided into the projection method and the hole method.

[0090] In some embodiments, the processing unit 104 is specifically configured to determine a first center of gravity position of the material in the vertical direction based on pixel values ​​obtained by projecting the material in the target image in the horizontal direction. The target image is the first image or the second image.

[0091] Determine the second center of gravity position of the material in the horizontal direction based on the pixel value obtained by projecting the material in the target image in the vertical direction;

[0092] Based on the first center of gravity position and the second center of gravity position, the center of gravity position of the material is obtained as the punching position of the material in the target image.

[0093] Among them, Figure 4 As shown, the projection method performs a mask in the height direction according to the projection of the image to be processed in the Y direction, and then performs a mask in the length direction according to the projection in the X direction. The total pixel value of the material in the X direction and the total pixel value in the Y direction are determined by projection, and the point with the largest pixel value (the center of gravity of the material) is determined in the X direction and the Y direction as the punchable position. In addition, the projection method can also eliminate the interference of irrelevant parts in the material (such as reinforcing ribs).

[0094] In some embodiments, the processing unit 104 is specifically configured to identify an area where holes of a material are located in a target image. The target image is the first image or the second image.

[0095] Based on the region, the center of gravity of the hole is determined as the punching position of the material in the target image.

[0096] In other words, the hole method finds the center of gravity of the material based on the original hole position of the material. If the material has a hole, the center of gravity position of the hole can be selected as the drilling position, such as Figure 5 Determining the center of gravity by the hole method can solve the problem that when there are holes in the material, the drilling position determined by the projection method is not at the center of gravity of the material, and the material is prone to flipping during hoisting after drilling.

[0097] In some embodiments, the processing unit 104 can also determine the most accurate punching position by combining the projection method and the hole method. If the material has holes, and the holes meet the punching width, and the punching plane is low, the hole method is preferred to determine the punching coordinates, otherwise the projection method is selected (some materials have no holes, so the projection method is used).

[0098] For example, Figure 6 A schematic diagram of a punching result provided in an embodiment of the present application. In the figure, a double parallel line is used in each material to simulate the punching position, and the punching device 103 can punch the material from bottom to top (in the direction shown in the figure).

[0099] In some embodiments, the punching direction of the punching device 103 is perpendicular to the direction in which the camera takes pictures, that is, the camera takes pictures of the cross section of the material, and the punching device 103 punches holes in a direction perpendicular to the cross section of the material.

[0100] Compared with the related art, the punching system provided by the embodiment of the present application brings at least the following beneficial effects. On the one hand, compared with the manual punching solution, it can improve the punching efficiency and reduce the waste of human resources. On the other hand, for the single-station punching solution, if the camera's shooting field of view is small, the punching positions obtained each time are photographed are relatively small, and the punching efficiency is relatively low. If the camera's shooting field of view is large, the accuracy of the photos obtained by taking pictures is low, and the accuracy of the punching positions determined is also relatively low. However, the present application adopts a dual-station method, and the second acquisition range of the second camera 1022 is set to be larger than the first acquisition range of the first camera 1021. On the one hand, the data with higher accuracy in the first station can be used. On the other hand, by utilizing the larger acquisition range of the second station, a larger number of materials can be punched at the same time by taking a photo, thereby taking into account both ensuring the punching accuracy and improving the punching efficiency.

[0101] like Figure 7 As shown, the embodiment of the present application provides a punching method, which is applied to a processing unit of a punching system, wherein the punching system is as follows Figure 2 As shown, the punching system also includes a punching device, and a first camera and a second camera sequentially arranged along the transmission direction of the transmission device; the first camera is used to take a picture of the material on the transmission device when it passes through the first acquisition range corresponding to the first camera, obtain a first image, and send it to the processing unit; the second camera is used to take a picture of the material on the transmission device when it passes through the second acquisition range corresponding to the second camera, obtain a second image, and send it to the processing unit. The punching method of the embodiment of the present application includes the following steps:

[0102] S701, receiving a first image and extracting a first punching position of a material in the first image.

[0103] In the embodiment of the present application, the processing unit is communicatively connected to the first camera, and can obtain a first image from the first camera, and then identify the material in the first image to obtain a first punching position of the material.

[0104] S702, receiving a second image and extracting a second punching position of the target material in the second image.

[0105] In the embodiment of the present application, the processing unit is also connected to the second camera for communication, and can obtain a second image from the second camera, and then identify the target material in the second image to obtain a second punching position of the target material.

[0106] In some embodiments, Figure 2 As shown, the punching system further includes a first sensing unit and / or a second sensing unit.

[0107] The first sensor unit is used to trigger the first camera to take pictures again if it detects that the X+1th material has passed after the first camera was triggered to take pictures last time; X is the number of materials contained in the first acquisition range;

[0108] The second sensing unit is used to trigger the second camera to take photos again if it detects that the Y+1th material has passed after the second camera was triggered to take photos last time; Y is the number of materials contained in the second acquisition range.

[0109] It should be noted that the embodiment of the present application is to take photos of materials in batches and punch holes, that is, after taking photos to obtain the second image, the materials indicated in the second image are punched in batches. Therefore, in order to ensure the correct execution of the solution, the sensor unit does not trigger the camera to take photos every time it detects that a material has passed by, but based on certain rules, it will trigger the camera to take photos again after determining that the current batch of materials has passed by.

[0110] For example, for the second sensing unit, there are 4 materials (e.g., marked as materials ad) within the current second acquisition range. After the second camera takes pictures, the second punching positions of materials ad can be obtained respectively. In this way, it is necessary to wait for these 4 materials to pass through before the second camera takes pictures of the next batch of materials. That is, after the second sensing unit has cumulatively detected 4 materials passing through, when it detects the 5th (4+1) material, it means that material ad has been processed, and the camera is triggered again to take pictures.

[0111] In some embodiments, X and Y are preset values. Alternatively, X is the number of materials within the first acquisition range determined when the first camera took a photo last time, and Y is the number of materials within the second acquisition range determined when the second camera took a photo last time.

[0112] S703, obtaining a target first punching position corresponding to the target material.

[0113] In some embodiments, after S701, the punching method of the embodiment of the present application further includes: storing the first punching positions of the material in the first image in the data queue in sequence according to the order in which the material passes through the first acquisition range.

[0114] In the case where the first punching positions are stored through a data queue, the above S703 can be specifically implemented as follows: the processing unit obtains the same number of first punching positions from the data queue according to the number of materials in the second image, and obtains the target first punching positions corresponding to the target material in accordance with the rule that the first punching positions taken out first match the materials that first pass through the second acquisition range.

[0115] Exemplarily, if there are materials 1-6 that have passed through the first acquisition range in sequence, the processing unit 104 determines the first punching position of each material and stores it in the data queue, which is represented as [coordinate 1, coordinate 2, coordinate 3, coordinate 4, coordinate 5, coordinate 6]. When these six materials pass through the second acquisition range, the second camera takes pictures of these six materials to obtain a second image. The processing unit determines the second punching position of each material from the second image in the order of materials 1-6, and takes out 6 groups of first punching positions from the data queue in sequence according to the quantity. The first punching position (coordinate 1) taken out first matches the material that passes through the second acquisition range first (material 1 on the far right of the transmission device). Similarly, coordinate 2 matches material 2, and so on, to determine the target first punching position corresponding to the target material in the second image. For example, if the target material is material 1, the target first punching position is the above-mentioned coordinate 1.

[0116] S704: Determine the target first punching position or the second punching position as the target punching position, and control the punching device to punch the target material according to the target punching position.

[0117] For example, the punching position can be represented by coordinate data. Figure 3 As shown, the coordinate data may include: a horizontal coordinate x, a vertical coordinate y, and an angle r at which the punching direction deviates from a preset direction. The preset direction may be a preset direction, such as a horizontal direction or a vertical direction.

[0118] It should be noted that the coordinate data representing the punching position may be a relative coordinate, such as the coordinate of the punching position relative to the lower left corner of the material. Before controlling the punching device to punch holes, the processing unit may convert the target punching position from a relative coordinate to a world coordinate, so that the punching device punches holes based on the world coordinate. In this way, during the punching process, the punching device does not need to determine the reference position (such as the lower left corner) of each material, but can also accurately determine the punching position, thereby achieving accurate punching and improving efficiency.

[0119] In some embodiments, the first acquisition range is smaller than the second acquisition range. In the above S704, determining the target first punching position or the second punching position as the target punching position can be specifically implemented as follows: if the difference between the target first punching position and the second punching position is smaller than a preset threshold, determining the target first punching position as the target punching position; otherwise, determining the second punching position as the target punching position.

[0120] It should be noted that, in the embodiment of the present application, by setting the first acquisition range to be smaller than the second acquisition range, it means that the first image obtained by the first camera has a high display accuracy of the material, so the accuracy of the first punching position obtained based on the first image is also high. By comparing the target first punching position and the second punching position of the target material, if the difference between the two is small, it means that the target first punching position and the target material to be punched are matched, so the target first punching position with higher accuracy is preferentially used for punching. If the difference between the two is large, it means that the first camera may miss the material in the process of moving the material from the first acquisition range to the second acquisition range, resulting in the mismatch between the target first punching position and the target material to be punched. In this case, the second punching position is used to punch the target material, which can ensure correct punching. In addition, the punching device punches the indicated material in the second image. Since the second acquisition range is large, the number of materials in the second image obtained by taking a photo of the second acquisition range is also large, so the punching positions of more materials can be obtained by taking a photo at one time, and then more materials can be punched in batches, thereby improving the punching efficiency. That is to say, the present application provides a dual-station / camera punching solution, which can improve the punching accuracy while ensuring the punching efficiency compared to the single-station punching solution.

[0121] In some embodiments, the first camera and the second camera are used to take pictures of the cross section of the material on the transmission device; the distance from the second camera to the transmission device is greater than the distance from the first camera to the transmission device. It can be understood that by providing a camera setting method, the first acquisition range corresponding to the first camera is smaller than the second acquisition range corresponding to the second camera, thereby improving the feasibility of the solution provided in the embodiment of the present application.

[0122] The following is an explanation of the process of determining the punching position (coordinate) in the above S701 or S702, which is specifically divided into a projection method and a hole method.

[0123] Figure 8 A schematic diagram of a projection method provided in an embodiment of the present application. Figure 8 As shown, including:

[0124] S801: Determine a first center of gravity position of the material in the vertical direction based on pixel values ​​obtained by projecting the material in the horizontal direction in the target image, wherein the target image is the first image or the second image.

[0125] S802, determining a second center of gravity position of the material in the horizontal direction based on a pixel value obtained by projecting the material in the target image in the vertical direction;

[0126] S803: Based on the first center of gravity position and the second center of gravity position, obtain the center of gravity position of the material as the punching position of the material in the target image.

[0127] For the above S801-S803, for each material in the image, the processing unit determines the sum of the pixel values ​​of the material in the X direction in the image through projection, determines the point with the largest pixel value in the X direction, and obtains the first center of gravity position of the material in the vertical direction. Similarly, the sum of the pixel values ​​of the material in the Y direction in the image is determined through projection, and the point with the largest pixel value in the Y direction is determined to obtain the second center of gravity position of the material in the horizontal direction. The intersection of the straight lines where the first center of gravity position and the second center of gravity position are located is the center of gravity position of the material, and then the punching position is determined based on the center of gravity position.

[0128] Fig. 9 A schematic diagram of a hole method provided in an embodiment of the present application. Fig. 9 As shown, including:

[0129] S901, identifying the area where the holes of the material are located in the target image. The target image is the first image or the second image.

[0130] S902: Based on the region, determine the center of gravity position of the hole as the punching position of the material in the target image.

[0131] For the above S901-S902, for each material in the image, it is determined by image recognition whether the material has a hole. If the material has a hole, the center of gravity position of the hole (the specific method of determining the center of gravity is not limited) can be selected as the punching position.

[0132] Fig.10 A processing flow chart of a first workstation provided in an embodiment of the present application. Fig.10 As shown, at the beginning, the first image of the first workstation is acquired, and after image preprocessing to obtain a binary image, the projection method and the hole method are used to determine the punching position, and the first punching position is optimized based on the two results, stored in the data queue, and ended.

[0133] Fig.11 A processing flow chart of a second workstation provided in an embodiment of the present application. Fig.11As shown, at the beginning, the second image of the second station is obtained, and after image preprocessing, it is determined whether the material number is 0. If so, the coordinates are sent to enable the punching device to punch. If not, the second punching position is determined, and the first punching position is obtained from the data queue, and then the final target punching position is determined by comparison, and the coordinates are sent after world coordinate conversion so that the punching device can punch, and the process ends.

[0134] The embodiment of the present application provides a punching method, which uses a camera to take a picture of a material to obtain an image, identifies the punching position of the material from the image, and punches the hole through a punching device, thereby realizing automated punching and improving the punching efficiency. In the present application, the material is photographed by a camera to determine the punching position based on the image obtained by taking the picture, and the materials are not required to have the same shape. It can be applied to scenes where the materials have variable shapes, complex types, and mixed placement. In addition, two cameras are set along the transmission direction, and the same material can be photographed separately to obtain two sets of punching position data. Then, the material can be punched based on the comparison of the two sets of punching position data, avoiding the problem that a single punching position data may be inaccurate, and the fault tolerance rate of punching the material can be improved.

[0135] In addition, the embodiment of the present application sets up a double station, compares the first punching position of the first station with the second punching position of the second station, determines the target punching position, and punches the hole through the punching device of the second station, thereby achieving double guarantees of punching efficiency and punching accuracy. By configuring the data queue, the amount of data storage can be reduced, ensuring the stable operation of the punching system in high-throughput scenarios.

[0136] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to achieve the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. It should be easy to realize that the technical goals in this field are combined with the units and algorithm steps of each example described in the embodiments disclosed in this article, and the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical goals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0137] Another embodiment of the present application further provides an electronic device, such as Fig.12As shown, the electronic device 1200 includes a memory 1201 and a processor 1202; the memory 1201 and the processor 1202 are coupled; the memory 1201 is used to store computer program codes, and the computer program codes include computer instructions. When the processor 1202 executes the computer instructions, the electronic device 1200 executes each step executed by the electronic device in the method flow shown in the above method embodiment.

[0138] Another embodiment of the present application further provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes each step executed by the electronic device in the method flow shown in the above method embodiment.

[0139] In another embodiment of the present application, a computer program product is provided. The computer program product includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes each step executed by the electronic device in the method flow shown in the above method embodiment.

[0140] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer execution instruction is loaded and executed on the computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instruction can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that can be accessed by the computer or a data storage device such as a server, data center, etc. that contains one or more servers that can be integrated with the medium. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), etc.

[0141] The above is only a specific implementation of the present application. Those skilled in the art may conceive of changes or substitutions based on the specific implementation provided by the present application, which should all be included in the protection scope of the present application.

Claims

1. A punching system, characterized in that: It includes a processing unit, a punching device, and a first camera and a second camera which are sequentially arranged along the transmission direction of the transmission device; The first camera is used to take a picture of the material on the transmission device when it passes through a first acquisition range corresponding to the first camera to obtain a first image; The second camera is used to take a picture of the material on the transmission device when it passes through a second acquisition range corresponding to the second camera to obtain a second image; The processing unit is connected to the first camera and the second camera for acquiring the first image and the second image, and obtaining a first punching position and a second punching position of a target material by processing the first image and the second image respectively; the target material is the material indicated in the second image; The processing unit is also connected to the punching device, and is further used to determine the first punching position or the second punching position as a target punching position, and control the punching device to punch the target material according to the target punching position.

2. The punching system according to claim 1, characterized in that: The first acquisition range is smaller than the second acquisition range; The processing unit is specifically configured to, when a difference between the first punching position and the second punching position is less than a preset threshold, determine the first punching position as the target punching position; otherwise, determine the second punching position as the target punching position.

3. The punching system according to claim 2, characterized in that: The first camera and the second camera are used to take pictures of the cross section of the material on the transmission device; The distance from the second camera to the transmission device is greater than the distance from the first camera to the transmission device.

4. The punching system according to claim 1, characterized in that: The punching system further comprises a first sensing unit, and / or a second sensing unit; The first sensing unit is used to trigger the first camera to take pictures again if it detects that the X+1th material has passed after the first camera was triggered to take pictures last time; X is the number of materials contained in the first acquisition range; The second sensing unit is used to trigger the second camera to take pictures again if it detects that the Y+1th material has passed after the second camera was triggered to take pictures last time; Y is the number of materials contained in the second acquisition range.

5. The punching system according to claim 4, characterized in that: X and Y are preset values; Alternatively, X is the number of materials within the first acquisition range determined when the first camera took a photo last time, and Y is the number of materials within the second acquisition range determined when the second camera took a photo last time.

6. The punching system according to claim 1, characterized in that: The processing unit is provided with a data queue; the data queue is used to sequentially store the first punching position of each material in the first image according to the order in which the materials pass through the first acquisition range; The processing device is also used to obtain the same number of first punching positions from the data queue according to the number of materials in the second image, and determine the target punching position of each material in turn according to the rule that the first punching position taken out first matches the material that passes through the second acquisition range first.

7. The punching system according to claim 1, characterized in that: The processing unit is specifically used to determine a first center of gravity position of the material in the vertical direction based on a pixel value obtained by projecting the material in the horizontal direction in a target image; the target image is the first image or the second image; Determine the second center of gravity position of the material in the horizontal direction based on the pixel value obtained by projecting the material in the target image in the vertical direction; Based on the first center of gravity position and the second center of gravity position, the center of gravity position of the material is obtained as the punching position of the material in the target image.

8. The punching system according to claim 1, characterized in that: The processing unit is specifically used to identify the area where the holes of the material are located in the target image; the target image is the first image or the second image; Based on the area, the center of gravity position of the hole is determined as the punching position of the material in the target image.

9. A punching method, applied to a processing unit of a punching system, characterized in that: The punching system further includes a punching device, and a first camera and a second camera sequentially arranged along the transmission direction of the transmission device; the first camera is used to take a picture when the material on the transmission device passes through a first acquisition range corresponding to the first camera, obtain a first image, and send it to the processing unit; the second camera is used to take a picture when the material on the transmission device passes through a second acquisition range corresponding to the second camera, obtain a second image, and send it to the processing unit; The method comprises: receiving the first image and extracting a first punch hole location of a material within the first image; receiving the second image and extracting a second punch hole location of the target material within the second image; Acquire a target first punching position corresponding to the target material; The target first punching position or the second punching position is determined as a target punching position, and a punching device is controlled to punch holes in the target material according to the target punching position.

10. The method according to claim 9, characterized in that The first acquisition range is smaller than the second acquisition range; and determining the target first punching position or the second punching position as the target punching position includes: When the difference between the target first punching position and the second punching position is less than a preset threshold, the target first punching position is determined as the target punching position; otherwise, the second punching position is determined as the target punching position.

11. The method according to claim 10, characterized in that The first camera and the second camera are used to take pictures of the cross section of the material on the transmission device; The distance from the second camera to the transmission device is greater than the distance from the first camera to the transmission device.

12. The method according to claim 9, characterized in that The punching system further comprises a first sensing unit, and / or a second sensing unit; The first sensing unit is used to trigger the first camera to take pictures again if it detects that the X+1th material has passed after the first camera was triggered to take pictures last time; X is the number of materials contained in the first acquisition range; The second sensing unit is used to trigger the second camera to take pictures again if it detects that the Y+1th material has passed after the second camera was triggered to take pictures last time; Y is the number of materials contained in the second acquisition range.

13. The method according to claim 12, characterized in that X and Y are preset values; Alternatively, X is the number of materials within the first acquisition range determined when the first camera took a photo last time, and Y is the number of materials within the second acquisition range determined when the second camera took a photo last time.

14. The method according to claim 9, characterized in that The method further comprises: According to the order in which the materials pass through the first acquisition range, the first punching positions of the materials in the first image are sequentially stored in the data queue; The step of obtaining a target first punching position corresponding to the target material includes: According to the number of materials in the second image, the same number of first punching positions are obtained from the data queue, and the target first punching positions corresponding to the target material are obtained according to the rule that the first punching positions taken out first match the materials that pass through the second acquisition range first.

15. The method according to claim 9, characterized in that Extract the punch hole positions of the material in the target image, including: Determine a first center of gravity position of the material in the vertical direction based on pixel values ​​obtained by projecting the material in the horizontal direction in the target image; the target image is the first image or the second image; Determine the second center of gravity position of the material in the horizontal direction based on the pixel value obtained by projecting the material in the target image in the vertical direction; Based on the first center of gravity position and the second center of gravity position, the center of gravity position of the material is obtained as the punching position of the material in the target image.

16. The method according to claim 9, characterized in that Extract the punch hole positions of the material in the target image, including: Identify the area where the holes of the material are located in the target image; the target image is the first image or the second image; Based on the area, the center of gravity position of the hole is determined as the punching position of the material in the target image.

17. An electronic device, characterized in that: The electronic device comprises: a processor and a memory; The memory stores instructions executable by the processor; When the processor is configured to execute the instructions, the electronic device implements the method according to any one of claims 9 to 16.

Images