Image processing method and device

Abstract

The present application proposes an image processing method and device, which relates to the technical fields of image processing, image matching, and image positioning. The method includes: collecting a target debris image; obtaining the minimum closed contour of the target debris based on a known template image and a target debris image, and determining the eigenvector of the HU invariant moment based on the minimum closed contour; using the eigenvector to perform image matching on the target debris image and the template image to obtain the target debris contour; based on the minimum circumscribed rectangle and the target debris contour, obtaining the coordinate position of the target debris in the pixel coordinate system; converting the coordinate position of the target debris position in the pixel coordinate system to the robot coordinate system, determining the expected arrival position of the robot arm, and driving the robot arm to move to the expected arrival position, so as to control the robot arm to grab the target debris on the de-mediation screen and return to the stop position. As a result, this scheme improves the efficiency of identifying the contour of the target debris and can achieve accurate positioning of the target debris.

Description

Technical Field

[0001] The present application relates to the technical fields of image processing, image matching, and image positioning, and in particular to an image processing method and device. Background Art

[0002] In the production and safety management of coal preparation plants, image matching and positioning technology, as an important auxiliary means, can improve production safety, increase production efficiency, assist in accident investigation and analysis, and optimize equipment asset management. However, due to the harsh environment of coal preparation plants, traditional image matching technology has problems such as poor adaptability to complex environments, limited real-time performance, insufficient accuracy and robustness, and strong data dependence, which makes it impossible to achieve the expected results. Summary of the invention

[0003] The present application provides an image processing method and device.

[0004] The first purpose of the present application is to propose an image processing method, which improves the efficiency of identifying the contour of target debris and can achieve accurate positioning of the target debris.

[0005] The second objective of the present application is to provide an image processing device.

[0006] To achieve the above-mentioned purpose, the first aspect of the present application proposes an image processing method, including: performing image acquisition on a target debris on a de-mediating screen to obtain a target debris image; obtaining a minimum closed contour of the target debris based on a known template image and the target debris image, and determining a feature vector of an HU invariant moment based on the minimum closed contour, wherein the HU invariant moment includes E feature vectors, where E is a positive integer greater than 1 or equal to 1; performing image matching on the target debris image and the template image using the feature vector of the HU invariant moment to obtain a target debris contour; processing the target debris contour using a minimum circumscribed rectangle to obtain a coordinate position of the target debris in a pixel coordinate system; converting the coordinate position of the target debris in the pixel coordinate system to a robot coordinate system, determining an expected arrival position of a robot arm, and driving the robot arm to move from a current stop position to the expected arrival position; in response to the robot arm moving to the expected arrival position, controlling the robot arm to grab the target debris on the de-mediating screen and return to the stop position.

[0007] To achieve the above-mentioned purpose, the second aspect of the present application proposes an image processing device, including: an acquisition module, used to acquire an image of a target debris on a de-mesh screen to obtain a target debris image; a first determination module, used to obtain a minimum closed contour of the target debris based on a known template image and the target debris image, and determine a feature vector of an HU invariant moment based on the minimum closed contour, wherein the HU invariant moment includes E feature vectors, where E is a positive integer greater than 1 or equal to 1; a matching module, used to match the target debris image and the template image using the feature vector of the HU invariant moment. The first module is used to perform image matching on the target debris image to obtain the contour of the target debris; the second determination module is used to process the contour of the target debris using the minimum circumscribed rectangle to obtain the coordinate position of the target debris in the pixel coordinate system; the third determination module is used to convert the coordinate position of the target debris in the pixel coordinate system to the robot coordinate system, determine the expected arrival position of the robot arm, and drive the robot arm to move from the current stop position to the expected arrival position; the grasping module is used to control the robot arm to grasp the target debris on the de-mediating screen and return to the stop position in response to the robot arm moving to the expected arrival position.

[0008] The image processing method and device provided in the present application collect the target debris image, determine the minimum closed contour of the target debris based on the known template image, and determine the eigenvector of the HU invariant moment based on the minimum closed contour. Then, the eigenvector of the HU invariant moment can be used to determine the target debris contour. By obtaining the coordinate position of the target debris in the pixel coordinate system from the target debris contour and converting the coordinate position to the robot coordinate system, the expected arrival position of the robot arm can be obtained, and the robot arm can be driven from the current stop position to the expected arrival position to grab the target debris and realize the precise positioning of the target debris. In the present application, since the eigenvector of the HU invariant moment has the characteristics of translation invariance, rotation invariance, scale invariance, etc., the target debris contour can be accurately identified when the target debris moves or rotates, thereby improving the efficiency of identifying the target debris contour.

[0009] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0011] Figure 1 A flowchart of an image processing method provided in an embodiment of the present application;

[0012] Figure 2A flowchart of another image processing method provided in an embodiment of the present application;

[0013] Figure 3 A schematic flow chart of a process for determining a feature vector of an HU invariant moment in an image processing method provided in an embodiment of the present application;

[0014] Figure 4 A flowchart of another image processing method provided in an embodiment of the present application;

[0015] Figure 5 A flowchart of another image processing method provided in an embodiment of the present application;

[0016] Figure 6 A schematic diagram of a process for determining the coordinate position of a target object in a robot coordinate system provided in an embodiment of the present application;

[0017] Figure 7 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0019] The image processing method and apparatus according to the embodiments of the present application are described below with reference to the accompanying drawings.

[0020] Figure 1 is a flowchart of an image processing method according to an exemplary embodiment. Figure 1 As shown, the image processing method of the embodiment of the present application includes but is not limited to the following steps:

[0021] S101, collecting images of target debris on the de-mediating screen to obtain images of the target debris.

[0022] It should be noted that the execution subject of the image processing method provided in the embodiment of the present application is an electronic device, and the electronic device may be a terminal device. Optionally, the terminal device may be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device may be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc., and the non-mobile electronic device may be a personal computer (PC), etc. The embodiment of the present application does not make specific limitations.

[0023] Optionally, the target debris on the de-mesh screen may be imaged based on a binocular vision system to obtain an image of the target debris, that is, a target debris image. The binocular vision system may be a binocular stereo camera.

[0024] S102, according to the known template image and the target debris image, the minimum closed contour of the target debris is obtained, and the eigenvector of the HU invariant moment is determined according to the minimum closed contour, where the HU invariant moment includes E eigenvectors, where E is a positive integer greater than 1 or equal to 1.

[0025] In some implementations, in order to enhance the image recognition capability and improve the image matching accuracy, it is possible to determine whether to enhance the target debris image based on the known template image and the sampling rate of the target debris image, and extract the minimum closed contour of the target debris from the enhanced target debris image. Optionally, an edge detection algorithm can be used to determine the minimum closed contour of the target debris from the target debris image.

[0026] In some implementations, by obtaining a first sampling rate of the template image and a second sampling rate of the target debris image, if the second sampling rate is less than the first sampling rate, a centroid image preprocessing algorithm can be performed on the target debris image to enhance the target debris image, and a minimum closed contour can be extracted from the enhanced target debris image.

[0027] Optionally, if the second sampling rate is greater than or equal to the first sampling rate, the minimum closed contour may be directly extracted from the target debris image.

[0028] Furthermore, the eigenvector of the HU invariant moment of the minimum closed contour may be calculated. For example, multiple moments of different orders of the minimum closed contour may be calculated, and the moments of different orders may be normalized to obtain the eigenvector of the HU invariant moment.

[0029] It can be understood that the HU invariant moment is an image feature extraction method with translation, rotation and scale invariance. HU stands for Hu, which is the name of the person who constructed the invariant moment. HU realizes the construction of the invariant moment based on the second-order and third-order central moments, hence the name HU invariant moment.

[0030] S103, using the eigenvector of the HU invariant moment to perform image matching on the target debris image and the template image to obtain the target debris contour.

[0031] In some implementations, the similarity between the target debris image and the template image can be calculated by acquiring feature data of each of the template image and the target debris image, and based on the feature vector and feature data of the HU invariant moment, the target debris contour can be determined based on the similarity.

[0032] Optionally, the similarities between the target debris images and the template images in multiple search windows may be determined, and the similarities corresponding to different search windows may be compared, and the search window with the greatest similarity may be determined, and the image area in the search window is the target debris contour.

[0033] In some implementations, the Euclidean distance corresponding to the search window can be calculated based on the feature data of the template image and the target debris image in different search windows, and the feature vector of the HU invariant moment, and the similarity between the target debris image and the template image in the search window can be determined based on the Euclidean distance. The smaller the Euclidean distance, the greater the similarity between the target debris image and the template image in the corresponding search window.

[0034] Optionally, use E eigenvectors of the HU invariant moment to perform image matching on the target debris image and the template image in different search windows, obtain a first feature of the template image and a second feature of the target debris image in the search window for each search window, perform operations on the first feature and the second feature with the e-th eigenvector of the HU invariant moment, respectively, to obtain a first operation result and a second operation result corresponding to the e-th eigenvector, determine the Euclidean distance corresponding to the search window according to the first operation results and the second operation results corresponding to each of the E eigenvectors, and obtain the target debris contour according to the Euclidean distance corresponding to the search window, where e is a positive integer and 1≤e≤E.

[0035] S104, using a minimum circumscribed rectangle to process the target debris contour to obtain the coordinate position of the target debris in the pixel coordinate system.

[0036] In some implementations, the minimum bounding rectangle of the target debris outline may be calculated and the target debris outline may be calibrated using the minimum bounding rectangle, so as to determine the coordinate position of the target debris in the pixel coordinate system according to the calibrated position.

[0037] Optionally, the minimum bounding rectangle of the target debris contour may be calculated based on a function for calculating the minimum bounding rectangle. Further, the coordinate position of the target debris may be calculated based on the pixel coordinates of the minimum bounding rectangle. For example, the coordinate position of the target debris may be calculated based on the pixel coordinates of the vertices of the minimum bounding rectangle.

[0038] S105, converting the coordinate position of the target debris in the pixel coordinate system into the robot coordinate system, determining the expected arrival position of the robot arm, and driving the robot arm to move from the current stop position to the expected arrival position.

[0039] In some implementations, in the process of converting the coordinate position of the target debris from the pixel coordinate system to the robot coordinate system, the conversion can be completed based on the world coordinate system. That is, the coordinate position can be first converted from the pixel coordinate system to the world coordinate system to obtain the position of the target debris in the world coordinate system, and then the position of the target debris in the world coordinate system can be converted to the robot coordinate system to obtain the position of the target debris in the robot coordinate system, and the position is used as the expected arrival position of the robot arm.

[0040] Furthermore, the robot arm can be driven to move from the current stop position to the desired arrival position. Optionally, the movement of the robot arm can be path planned according to the stop position and the desired arrival position of the robot arm to obtain a path planning result, and the robot arm can be controlled to move from the stop position to the desired arrival position based on the path planning result.

[0041] Optionally, the robot arm may be driven to move based on the driving signal. The driving signal for driving the robot arm may be generated according to the path planning result.

[0042] S106, in response to the robot arm moving to the desired arrival position, the robot arm is controlled to grab the target debris on the de-mediating screen and return to the rest position.

[0043] In some implementations, when the robot arm moves to a desired position, the gripping mechanism of the robot arm is activated to grip the target debris on the de-mediating screen, and after successfully gripping the target debris, the robot arm can return to the rest position according to the planned path.

[0044] Optionally, after the robotic arm reaches the rest position, the robotic arm may release the target debris in the gripping mechanism to a designated collection area.

[0045] In the image processing method provided in the embodiment of the present application, by collecting the target debris image, and determining the minimum closed contour of the target debris according to the known template image, and determining the eigenvector of the HU invariant moment according to the minimum closed contour. Then, the eigenvector of the HU invariant moment can be used to determine the target debris contour. By obtaining the coordinate position of the target debris in the pixel coordinate system from the target debris contour, and converting the coordinate position to the robot coordinate system, the expected arrival position of the robot arm can be obtained, and the robot arm can be driven from the current stop position to the expected arrival position to grab the target debris and realize the precise positioning of the target debris. In the present application, since the eigenvector of the HU invariant moment has the characteristics of translation invariance, rotation invariance, scale invariance, etc., the target debris contour can be accurately identified when the target debris moves and rotates, thereby improving the efficiency of identifying the target debris contour.

[0046] Figure 2 is a flowchart of an image processing method according to an exemplary embodiment. Figure 2 As shown, the image processing method of the embodiment of the present application includes but is not limited to the following steps:

[0047] S201, collecting images of target debris on the de-mediating screen to obtain images of the target debris.

[0048] In the embodiment of the present application, the implementation method of step S201 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0049] S202, performing image preprocessing on the template image and the target debris image to obtain a first image and a second image.

[0050] In some implementations, preprocessing may include grayscale conversion, denoising and other operations, that is, the template image and the target debris image may be grayscale converted to obtain a first grayscale image corresponding to the template image and a second grayscale image corresponding to the target debris image.

[0051] Furthermore, the first grayscale image and the second grayscale image are filtered and denoised, and the denoised images are binarized to separate the target and the background, and the separated images are morphologically processed to fill the holes in the images and remove isolated noise points to obtain the first image and the second image with smooth boundaries. In other words, the first image can be obtained after image preprocessing the template image, and the second image can be obtained after image preprocessing the target debris image.

[0052] S203: Obtain the sampling rates of the first image and the second image respectively.

[0053] As you can understand, sampling rate usually refers to dots per inch or pixels per inch, which describes the density of pixels in an image. For digital images, the sampling rate is usually related to the resolution and physical size of the image.

[0054] That is, the sampling rate of the first image may be determined based on the resolution and physical size of the first image; and the sampling rate of the second image may be determined based on the resolution and physical size of the second image.

[0055] S204: Process the second image according to the respective sampling rates of the first image and the second image to obtain a minimum closed contour of the target debris.

[0056] In some implementations, the second image may be processed based on a sampling threshold to obtain a minimum closed contour of the target debris. If the sampling rate of the second image is greater than the sampling threshold, the minimum closed contour of the target debris may be extracted from the second image. Optionally, the sampling rate of the first image may be determined as the sampling threshold.

[0057] Furthermore, the sampling rate of the second image is compared with the sampling threshold. In response to the sampling rate of the second image being greater than the sampling threshold, an edge detection operator is used to extract edges from the second image, and the edges are tracked to obtain an outer contour image as the minimum closed contour of the target debris.

[0058] Optionally, in response to a sampling rate of the second image being less than or equal to a sampling threshold, a centroid image preprocessing algorithm is performed on the second image to obtain a minimum closed contour of the target debris.

[0059] As an example, the second image is grayed and inverted, and a threshold is selected to determine the spot feature area in the second image based on the spot threshold. The spot feature area is denoised and the threshold is selected again to obtain a clearer spot feature area. Further, the spot feature area is filled to obtain an image of a regular shape, and the edge is extracted from the image using an edge detection operator to obtain an outer contour image. Further, a weighted centroid algorithm is applied to the outer contour image, and the weighted outer contour image is used as the minimum closed contour of the target debris.

[0060] Among them, the formula of the imposed weighted centroid algorithm is as follows:

[0061]

[0062] in,( x c , y c ) represents the coordinates of the center point of the light spot, ( x 0 , y 0) represents the coordinates of the starting point, ( x , y )and( x i , y j ) represents the coordinates of the current pixel being measured, W 0 represents the weight value, I ij Indicates the pixel value of the current pixel. w Indicates the enhanced light intensity value, w 0 Indicates the light intensity value before enhancement.

[0063] It can be understood that the principle of the weighted centroid algorithm is to enhance the pixel values ​​of some areas near the center of the light spot, so that the influence of the pixels that have a greater impact on the light spot is further increased to improve the accuracy of centroid detection. Even if the shape of the light spot is not close to the Gaussian distribution, this method can still detect the centroid of the light spot more accurately.

[0064] S205, determining a eigenvector of the HU invariant moment according to the minimum closed contour, and using the eigenvector of the HU invariant moment to perform image matching on the target debris image and the template image to obtain a target debris contour.

[0065] In the embodiment of the present application, the implementation method of step S205 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0066] S206, using a minimum circumscribed rectangle to process the target debris contour to obtain the coordinate position of the target debris in the pixel coordinate system.

[0067] In the embodiment of the present application, the implementation method of step S206 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0068] S207, converting the coordinate position of the target debris in the pixel coordinate system into the robot coordinate system, determining the expected arrival position of the robot arm, and driving the robot arm to move from the current stop position to the expected arrival position.

[0069] In the embodiment of the present application, the implementation method of step S207 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0070] S208, in response to the robot arm moving to the desired arrival position, the robot arm is controlled to grab the target debris on the de-mediating screen and return to the rest position.

[0071] In the embodiment of the present application, the implementation method of step S208 can be implemented by any method in the various embodiments of the present application, which is not limited here and will not be repeated.

[0072] In the image processing method provided in the embodiment of the present application, by collecting the target debris image, and according to the known template image and the sampling rate of the target debris image, the minimum closed contour of the target debris is determined, and the eigenvector of the HU invariant moment is determined according to the minimum closed contour. Then, the eigenvector of the HU invariant moment can be used to determine the target debris contour. By obtaining the coordinate position of the target debris in the pixel coordinate system from the target debris contour, and converting the coordinate position to the robot coordinate system, the expected arrival position of the robot arm can be obtained, and the robot arm can be driven from the current stop position to the expected arrival position to grab the target debris and realize the precise positioning of the target debris. In the present application, since the eigenvector of the HU invariant moment has the characteristics of translation invariance, rotation invariance, scale invariance, etc., the target debris contour can be accurately identified when the target debris moves and rotates, thereby improving the efficiency of identifying the target debris contour.

[0073] Based on the above embodiments, the present application embodiment can explain the process of determining the eigenvector of the HU invariant moment, such as: Figure 3 As shown, the process of determining the eigenvector of the HU invariant moment includes but is not limited to the following steps:

[0074] S301, determine a p+q order central moment expression, and normalize the p+q order central moment expression to obtain a p+q order normalized central moment expression, where p and q are positive integers greater than or equal to 1.

[0075] In some implementations, the p+q order central moment expression can be determined based on the p+q order geometric moment expression of the image, and then the p+q order central moment expression can be normalized to obtain the p+q order normalized central moment expression. Optionally, the p+q order geometric moment expression can be determined based on the pixel information of the image and the discrete image parameters of the image. The pixel information includes but is not limited to: pixel size and pixel point coordinates.

[0076] Alternatively, the expression for the p+q order geometric moment is as follows:

[0077]

[0078] in, H pq represents the p+q order geometric moment expression, G , B represents the pixel size, ( x , y ) represents the pixel coordinates,g ( x , y ) represents discrete image parameters.

[0079] S302, determining the area of ​​the target debris and the centroid of the target debris according to the minimum closed contour, and determining the image centroid coordinates according to the area and centroid of the target debris.

[0080] Optionally, the area enclosed by the minimum closed contour may be calculated as the area of ​​the target debris.

[0081] Alternatively, the center of mass of the target debris can be calculated based on the p+q order geometric moment expression. Alternatively, the center of mass of the target debris can be calculated based on the first order geometric moment. That is, p+q=1. By substituting p=0 and q=1 into formula (2), we can determine H 01 is the center of mass of the target debris. By substituting p=1, q=0 into formula (2), we can determine H 10 is the centroid of the target debris.

[0082] Furthermore, the image center of gravity coordinates can be calculated based on the area and center of mass of the target debris. Suppose the image center of gravity coordinates are ( x 0 , y 0 ), the area of ​​the target debris is H 00 ,but , .

[0083] S303, determining discrete image parameters according to the pixel coordinates within the minimum closed contour, and determining the second-order normalized central moment and the third-order normalized central moment according to the discrete image parameters, pixel coordinates and image centroid coordinates in combination with the p+q-order normalized central moment expression.

[0084] Optionally, the coordinates of all pixels within the minimum closed contour may be counted, and discrete image parameters may be calculated based on the coordinates of the pixels. For example, shape parameters, position parameters, and the like may be calculated as discrete image parameters.

[0085] In some implementations, the p+q order central moment expression can be first determined based on discrete image parameters, pixel point coordinates and image centroid coordinates, as well as the p+q order geometric moment expression, and then the p+q order central moment expression is normalized to obtain the p+q order normalized central moment expression.

[0086] Optionally, the formula for determining the expression of the p+q order central moment is as follows:

[0087]

[0088] in, η pq represents the expression of the p+q order central moment, G , B represents the pixel size, ( x , y ) represents the pixel coordinates, g ( x , y ) represents discrete image parameters, ( x 0 , y 0 ) represents the coordinates of the image center of gravity.

[0089] Optionally, the formula for normalizing the p+q order central moment expression is as follows:

[0090]

[0091] in, μ pq represents the p+q order normalized central moment expression, η pq represents the expression of the p+q order central moment, η 00 Represents the zero-order center distance.

[0092] It can be understood that the order corresponding to the second-order normalized central moment is p+q=2. By substituting p=0, q=2, p=1, q=1, p=2, q=0 into formula (4), the second-order normalized central moment can be calculated: μ 02 , μ 11 , μ 20 The order corresponding to the third-order normalized central moment is p+q=3. By substituting p=0, q=3, p=1, q=2, p=2, q=1, p=3, q=0 into formula (4), the third-order normalized central moment can be calculated: μ 03 , μ 12 , μ 21 , μ 30 .

[0093] S304: Determine the eigenvector of the HU invariant moment according to the second-order normalized central moment and the third-order normalized central moment.

[0094] Optionally, based on the second-order normalized central moment and the third-order normalized central moment, the formula for calculating the eigenvector of the HU invariant moment is as follows:

[0095]

[0096] in, I 1 , I 2 , I 3 , I 4 , I 5 , I 6 , I 7 is the eigenvector of HU invariant moment, μ 20 , μ 02 , μ 11 is the second-order normalized central moment, μ 30 , μ 21 , μ 03 , μ 12 is the third-order normalized central moment.

[0097] In the image processing method provided in the embodiment of the present application, the characteristic vector of the HU invariant moment can be calculated based on the p+q order normalized central moment expression and the minimum closed contour. Since the characteristic vector of the HU invariant moment has the characteristics of translation invariance, rotation invariance, scale invariance, etc., the contour of the target debris can be accurately identified when the target debris moves or rotates, and the efficiency of identifying the contour of the target debris is improved.

[0098] Figure 4 is a flowchart of an image processing method according to an exemplary embodiment. Figure 4 As shown, the image processing method of the embodiment of the present application includes but is not limited to the following steps:

[0099] S401, collecting images of target debris on the de-mediating screen to obtain images of the target debris.

[0100] In the embodiment of the present application, the implementation method of step S401 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0101] S402, obtaining the minimum closed contour of the target debris according to the known template image and the target debris image, and determining the eigenvector of the HU invariant moment according to the minimum closed contour.

[0102] In the embodiment of the present application, the implementation method of step S402 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0103] S403, based on the feature vector of the HU invariant moment, feature matching is performed on the target debris image and the template image in the search window to obtain the Euclidean distance corresponding to the search window.

[0104] In some implementations, E eigenvectors of the HU invariant moment can be used to perform image matching between the target debris image and the template image in different search windows. For the nth search window, a first feature of the template image and a second feature of the target debris image in the nth search window are obtained, where n is a positive integer and 1≤n≤N, and the first feature and the eth eigenvector of the HU invariant moment are operated to obtain a first operation result corresponding to the eth eigenvector, where e is a positive integer and 1≤e≤E, and E is the number of eigenvectors of the HU invariant moment.

[0105] Furthermore, the second feature and the e-th eigenvector of the HU invariant moment are operated to obtain a second operation result corresponding to the e-th eigenvector, and then the Euclidean distance of the n-th search window is determined according to the first operation result and the second operation result corresponding to the E eigenvectors.

[0106] Optionally, by obtaining the square difference between the first operation result and the second operation result corresponding to the e-th eigenvector, and summing the square differences corresponding to the E eigenvectors, a sum value is obtained, and the Euclidean distance of the search window is obtained according to the sum value. The formula for calculating the Euclidean distance corresponding to the search window is as follows:

[0107]

[0108] in, D n represents the Euclidean distance, F ( e ) represents the first operation result, F n ( e ) represents the second operation result. There are 7 eigenvectors, that is, E=7.

[0109] S404, obtaining the minimum Euclidean distance from the Euclidean distances corresponding to N search windows, and determining the search window corresponding to the minimum Euclidean distance as the search window with the greatest similarity, wherein N is a positive integer greater than 1.

[0110] S405: The image area in the search window with the greatest similarity is used as the target debris contour.

[0111] It can be understood that the search window corresponding to the minimum Euclidean distance is the search window with the greatest similarity. That is, the minimum Euclidean distance can be obtained from the Euclidean distances corresponding to the N search windows, and the search window corresponding to the Euclidean distance is used as the search window with the greatest similarity. For example, assuming that the minimum Euclidean distance is D min ,but .

[0112] That is to say, the minimum Euclidean distance is obtained from the Euclidean distances corresponding to N different search windows, and in response to the minimum Euclidean distance being less than the set distance threshold, the search window corresponding to the minimum Euclidean distance is determined to be the search window with the greatest similarity, and the image area within the search window with the greatest similarity is used as the target debris contour, where N is a positive integer greater than 1.

[0113] Optionally, in response to the minimum Euclidean distance being greater than or equal to a set distance threshold, the search window is adjusted, and the Euclidean distances corresponding to N different search windows are re-acquired and subsequent operations are performed until the target debris contour is obtained.

[0114] S406, using a minimum circumscribed rectangle to process the target debris contour to obtain the coordinate position of the target debris in the pixel coordinate system.

[0115] In the embodiment of the present application, the implementation method of step S406 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0116] S407, converting the coordinate position of the target debris in the pixel coordinate system into the robot coordinate system, determining the expected arrival position of the robot arm, and driving the robot arm to move from the current stop position to the expected arrival position.

[0117] In the embodiment of the present application, the implementation method of step S407 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0118] S408, in response to the robot arm moving to the desired arrival position, the robot arm is controlled to grab the target debris on the de-mediating screen and return to the rest position.

[0119] In the embodiment of the present application, the implementation method of step S408 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0120] In the image processing method provided in the embodiment of the present application, by collecting the target debris image, and determining the minimum closed contour of the target debris according to the known template image, and determining the eigenvector of the HU invariant moment according to the minimum closed contour. Then, the eigenvector of the HU invariant moment can be used to calculate the Euclidean distance of the eigenvector in the search window, and based on the Euclidean distance, the target debris contour can be determined. By obtaining the coordinate position of the target debris in the pixel coordinate system from the target debris contour, and converting the coordinate position to the robot coordinate system, the desired arrival position of the manipulator can be obtained, and the manipulator is driven to move from the current stop position to the desired arrival position to grab the target debris and realize the precise positioning of the target debris. In the present application, since the eigenvector of the HU invariant moment has the characteristics of translation invariance, rotation invariance, scale invariance, etc., the target debris contour can be accurately identified when the target debris moves and rotates, and the efficiency of identifying the target debris contour is improved.

[0121] Figure 5 is a flowchart of an image processing method according to an exemplary embodiment. Figure 5 As shown, the image processing method of the embodiment of the present application includes but is not limited to the following steps:

[0122] S501, collecting images of target debris on the de-mediating screen to obtain images of the target debris.

[0123] In the embodiment of the present application, the implementation method of step S501 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0124] S502, obtaining the minimum closed contour of the target debris according to the known template image and the target debris image, and determining the eigenvector of the HU invariant moment according to the minimum closed contour.

[0125] In the embodiment of the present application, the implementation method of step S502 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0126] S503, using the eigenvector of the HU invariant moment to perform image matching on the target debris image and the template image to obtain the target debris contour.

[0127] In the embodiment of the present application, the implementation method of step S503 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0128] S504: Process the contour of the target debris using a minimum circumscribed rectangle to obtain the coordinate position of the target debris in the pixel coordinate system.

[0129] Optionally, the coordinate value of the minimum circumscribed rectangle in the pixel coordinate system may be determined, and based on the coordinate value, the coordinate position of the target debris in the pixel coordinate system may be determined.

[0130] Optionally, by determining the minimum bounding rectangle in pixel coordinates u The maximum and minimum coordinate values ​​in the axis direction and determine the minimum enclosing rectangle in the pixel coordinate system v The maximum and minimum coordinate values ​​in the axis direction.

[0131] Furthermore, according to u The maximum and minimum coordinate values ​​in the axis direction, and v The maximum and minimum coordinate values ​​in the axial direction determine the centroid coordinate position of the target debris in the pixel coordinate system.

[0132] Further, the coordinate position of the target debris in the pixel coordinate system is determined according to the centroid coordinate position of the target debris in the pixel coordinate system and the size of the minimum bounding rectangle. Optionally, the centroid coordinate position can be scaled based on the size of the minimum bounding rectangle to obtain the coordinate position of the target debris in the pixel coordinate system.

[0133] Optionally, the formula for calculating the centroid coordinate position of the target debris in the pixel coordinate system is as follows:

[0134]

[0135] in,( X , Y ) represents the centroid coordinate position, u max , u min Respectively u The maximum and minimum coordinate values ​​in the axis direction. v max , v min Respectively v The maximum and minimum coordinate values ​​in the axis direction.

[0136] S505: Convert the coordinate position of the target debris in the pixel coordinate system to the world coordinate system to obtain the coordinate position of the target debris in the world coordinate system.

[0137] In some implementations, the camera coordinate system can be transformed into the world coordinate system based on the offset vector and rotation matrix between the camera coordinate system and the world coordinate system, and then the coordinate position of the pixel coordinate system can be transformed into the world coordinate system according to the mapping relationship between the camera coordinate system and the pixel coordinate system to obtain the coordinate position of the target debris in the world coordinate system.

[0138] For example, let the rotation matrix be R 1 , the offset vector is T 1 , which can be based on the rotation matrix R 1 and the offset vector T 1 , realize the conversion between the camera coordinate system and the world coordinate system, the conversion formula is as follows:

[0139]

[0140] in,( X w , Y w , Z w ) is the coordinate of the world coordinate system, ( X c , Y c , Z c ) is the coordinate of the camera coordinate system. Among them, the rotation matrix , the offset vector .

[0141] Furthermore, the spatial coordinates can be converted into plane coordinates according to the mapping relationship between the camera coordinate system and the pixel coordinate system. The calculation formula is as follows:

[0142]

[0143] in,( x , y ) are plane coordinates, f is the focal length of the camera, ( X c , Y c , Z c ) are spatial coordinates.

[0144] Furthermore, since there is no rotation matrix between the pixel coordinate system and the world coordinate system during the conversion process, the coordinate system conversion can be achieved based on the proportional relationship and translation transformation of the plane coordinates. The conversion process is as follows:

[0145]

[0146] in,( u , v ) represents pixel coordinates, ( x , y ) represents the plane coordinates, ( u0 , v 0 ) represents the coordinates of the origin of the camera coordinate system in the pixel coordinate system.

[0147] Furthermore, according to formula (9) and formula (10), the conversion formula of the coordinate position of the target debris in the pixel coordinate system can be obtained, and the conversion formula is used to convert it to the world coordinate system to obtain the coordinate position of the target debris in the world coordinate system. The conversion formula is as follows:

[0148]

[0149] in,( X c , Y c , Z c ) represents the coordinates of the camera coordinate system, ( u , v ) represents pixel coordinates, ( x , y ) represents the plane coordinates, ( u 0 , v 0 ) represents the coordinates of the origin of the camera coordinate system in the pixel coordinate system, f Indicates the focal length of the camera.

[0150] S506 , combining the position information of the robot, converting the coordinate position of the target debris in the world coordinate system into the robot coordinate system to obtain the coordinate position of the target debris in the robot coordinate system.

[0151] Optionally, the coordinate position of the world coordinate system can be converted to the coordinate position of the robot coordinate system based on the conversion matrix between the robot coordinate system and the world coordinate system. Taking point P as an example, the conversion matrix is ​​as follows:

[0152]

[0153] in, c u , c v For each unit pixel, u Axis and v The physical size of the axis, ( u 0 , v 0 ) is the coordinate of the origin of the camera coordinate system in the pixel coordinate system, is the 3×3 camera extrinsic parameter rotation matrix, is the 3×1 camera extrinsic parameter translation matrix, ( XW , Y W , Z W ,1) is the homogeneous coordinate of point P in the robot coordinate system, f u , f v They are u Axis and v The scale factor on the axis, M 1 , M 2 are the camera internal and external parameter matrices respectively, M 1 Obtained by camera calibration, P W is the coordinate of point P in the robot coordinate system.

[0154] Furthermore, the coordinates of point P in the robot coordinate system can be calculated based on formula (12). Optionally, the coordinates of point P in the camera coordinate system can be calculated based on formula (12), and then the coordinates of point P in the robot coordinate system can be calculated based on the coordinates of point P in the camera coordinate system. The calculation formula is as follows:

[0155]

[0156] in, P C is the coordinate of point P in the camera coordinate system, P W is the coordinate of point P in the robot coordinate system, E is the robot coordinate system, T is the translation matrix, W is the camera coordinate system, R is the rotation matrix, that is is the rotation matrix of the robot coordinate system and the camera coordinate system, is the robot coordinate system rotation matrix, is the translation matrix of the robot coordinate system, is the translation matrix between the robot coordinate system and the camera coordinate system.

[0157] S507, determining the forward kinematics equation of the manipulator end coordinate system relative to the robot coordinate system.

[0158] S508, obtaining the expected arrival position of the robot arm according to the forward kinematics equation and the coordinate position of the target debris in the robot coordinate system.

[0159] Alternatively, the forward kinematics equations are formulated as follows:

[0160]

[0161] in, represents the forward kinematics equation of the robotic arm, , , represents the matrix of the three rotational joints of the robot arm, , Indicates the original position distance, θ 3 Represents the rotation angle of the robot arm joint, , , Indicates the displacement size.

[0162] In some implementations, in order to achieve parameter conversion of the motion structure of the manipulator at the desired arrival position, the desired arrival position of the manipulator can also be determined based on the inverse kinematics equation. Optionally, the inverse kinematics equation can be established by algebraic method based on the forward kinematics equation, and the rotation matrix and translation matrix of the manipulator end coordinate system relative to the robot coordinate system can be determined based on the inverse kinematics equation.

[0163] Optionally, the translation matrix of the manipulator end coordinate system relative to the robot coordinate system can be determined based on the position coordinates of the manipulator end coordinate system relative to the robot coordinate system and the rotation matrix. The formula for calculating the translation matrix is ​​as follows:

[0164]

[0165] in, T represents the rotation matrix, ( b X , b Y , b Z ) represents the position coordinates of the manipulator end coordinate system relative to the robot coordinate system, and the rotation matrix is .

[0166] Furthermore, the coordinate position of the target debris in the robot coordinate system can be transformed according to the rotation matrix and the translation matrix to obtain the coordinate position of the target debris in the robot arm end coordinate system as the expected arrival position of the robot arm.

[0167] Optionally, the formula for calculating the expected arrival position is as follows:

[0168]

[0169] in,( d 1 , d 2 , d3 , θ 3 ) represents the expected arrival position, ( b X , b Y , b Z ) represents the position coordinates of the manipulator end coordinate system relative to the robot coordinate system. h 13 , h 32 , h 33 Represents an element in a rotation matrix.

[0170] S509: Drive the robot arm to move from the current stop position to the desired arrival position.

[0171] In the embodiment of the present application, the implementation method of step S509 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0172] S510, in response to the robot arm moving to the desired arrival position, the robot arm is controlled to grab the target debris on the de-mediating screen and return to the rest position.

[0173] In the embodiment of the present application, the implementation method of step S510 can be implemented by any method in the embodiments of the present application, which is not limited here and will not be repeated.

[0174] In the image processing method provided in the embodiment of the present application, by collecting the target debris image, and determining the minimum closed contour of the target debris according to the known template image, and determining the eigenvector of the HU invariant moment according to the minimum closed contour. Then the eigenvector of the HU invariant moment can be used to determine the target debris contour. By obtaining the coordinate position of the target debris in the pixel coordinate system from the target debris contour, and converting the coordinate position to the world coordinate system, and then converting from the world coordinate system to the robot coordinate system. Further, based on the forward kinematics equation and the inverse kinematics equation, and according to the coordinate position of the target debris in the robot coordinate system, the expected arrival position of the manipulator is calculated, and the manipulator is driven from the current stop position to the expected arrival position to grab the target debris, so as to achieve accurate positioning of the target debris. In the present application, since the eigenvector of the HU invariant moment has the characteristics of translation invariance, rotation invariance, scale invariance, etc., the target debris contour can be accurately identified when the target debris moves and rotates, and the efficiency of identifying the target debris contour is improved.

[0175] Figure 6The flowchart for determining the coordinate position of the target debris in the robot coordinate system is shown. The target debris image is obtained by image acquisition of the target debris on the de-mediation screen, and the known template image and the target debris image are preprocessed to obtain the first image and the second image. The sampling rates of the first image and the second image are obtained, and the sampling rate of the first image is set as the sampling threshold. By determining whether the sampling rate of the second image is greater than the sampling threshold, when the sampling rate of the second image is greater than the sampling threshold, the edge of the second image is extracted to obtain the minimum closed contour of the target debris; in response to the sampling rate of the second image being less than or equal to the sampling threshold, the second image is subjected to the centroid image preprocessing algorithm to obtain the minimum closed contour of the target debris.

[0176] Furthermore, according to the minimum closed contour of the target debris, the eigenvector of the HU invariant moment can be determined, and the eigenvector of the HU invariant moment can be used for image matching and recognition to obtain the contour of the target debris, and then the coordinate position of the target debris in the pixel coordinate system can be determined according to the contour of the target debris. By converting the coordinate position of the target debris position in the pixel coordinate system to the robot coordinate system, the coordinate position of the target debris in the robot coordinate system can be determined.

[0177] In order to implement the above embodiments, the present application also proposes an image processing device.

[0178] Figure 7 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application.

[0179] like Figure 7 As shown, the image processing device 700 includes:

[0180] The acquisition module 701 is used to acquire an image of the target debris on the de-mediating screen to obtain an image of the target debris;

[0181] A first determination module 702 is used to obtain a minimum closed contour of the target debris according to a known template image and the target debris image, and determine a feature vector of the HU invariant moment according to the minimum closed contour, wherein the HU invariant moment includes E feature vectors, where E is a positive integer greater than 1 or equal to 1;

[0182] A matching module 703 is used to perform image matching between the target debris image and the template image in different search windows using the E feature vectors of the HU invariant moment, obtain a first feature of the template image and a second feature of the target debris image in the search window for each search window, respectively operate the first feature and the second feature with the e-th feature vector of the HU invariant moment to obtain a first operation result and a second operation result corresponding to the e-th feature vector, determine the Euclidean distance corresponding to the search window according to the first operation result and the second operation result corresponding to each of the E feature vectors, and obtain the target debris contour according to the Euclidean distance corresponding to the search window, wherein e is a positive integer and 1≤e≤E;

[0183] The second determination module 704 is used to process the outline of the target debris using a minimum circumscribed rectangle to obtain the coordinate position of the target debris in a pixel coordinate system;

[0184] The third determination module 705 is used to convert the coordinate position of the target debris in the pixel coordinate system to the robot coordinate system, determine the expected arrival position of the robot arm, and drive the robot arm to move from the current stop position to the expected arrival position;

[0185] The grabbing module 706 is used for controlling the robotic arm to grab the target debris on the de-mediating screen and return to the stop position in response to the robotic arm moving to the expected arrival position.

[0186] In a possible implementation manner of the embodiment of the present application, the first determination module 702 is further used to: perform image preprocessing on the template image and the target debris image to obtain a first image and a second image; obtain a sampling rate of each of the first image and the second image; and process the second image according to the sampling rates of each of the first image and the second image to obtain a minimum closed contour of the target debris.

[0187] In a possible implementation manner of the embodiment of the present application, the first determination module 702 is further used to: determine the sampling rate of the first image as a sampling threshold; in response to the sampling rate of the second image being greater than the sampling threshold, extract edges from the second image using an edge detection operator, and track the edges to obtain an outer contour image as the minimum closed contour of the target debris; in response to the sampling rate of the second image being less than or equal to the sampling threshold, perform a centroid image preprocessing algorithm on the second image to obtain the minimum closed contour of the target debris.

[0188] In a possible implementation manner of the embodiment of the present application, the first determination module 702 is further used to: determine a p+q order central moment expression, and normalize the p+q order central moment expression to obtain a p+q order normalized central moment expression, where p and q are positive integers greater than or equal to 1; determine the area of ​​the target debris and the center of mass of the target debris according to the minimum closed contour, and determine the image center of gravity coordinates according to the area and center of mass of the target debris; determine discrete image parameters according to the pixel point coordinates within the minimum closed contour, and determine the second-order normalized central moment and the third-order normalized central moment according to the discrete image parameters, the pixel point coordinates and the image center of gravity coordinates in combination with the p+q order normalized central moment expression; determine the eigenvector of the HU invariant moment according to the second-order normalized central moment and the third-order normalized central moment.

[0189] In a possible implementation manner of the embodiment of the present application, the matching module 703 is further used to: obtain a minimum Euclidean distance from the Euclidean distances corresponding to N different search windows; in response to the minimum Euclidean distance being less than a set distance threshold, determine that the search window corresponding to the minimum Euclidean distance is the search window with the greatest similarity, and use the image area within the search window with the greatest similarity as the target debris contour, where N is a positive integer greater than 1; in response to the minimum Euclidean distance being greater than or equal to the set distance threshold, adjust the search window, and re-acquire the Euclidean distances corresponding to N different search windows and perform subsequent operations until the target debris contour is obtained.

[0190] In a possible implementation method of an embodiment of the present application, the matching module 703 is also used to: obtain the square difference between the first operation result and the second operation result corresponding to the e-th eigenvector; sum the square differences corresponding to the E eigenvectors to obtain the sum value, and obtain the Euclidean distance of the search window based on the sum value.

[0191] In a possible implementation of the embodiment of the present application, the second determining module 704 is further configured to: determine the minimum bounding rectangle in the pixel coordinate system. u The maximum and minimum coordinate values ​​in the axis direction; Determine the minimum bounding rectangle in the pixel coordinate system v The maximum and minimum coordinate values ​​in the axis direction; according to the u The maximum and minimum coordinate values ​​in the axis direction, as well as the v The maximum coordinate value and the minimum coordinate value in the axis direction are used to determine the centroid coordinate position of the target debris in the pixel coordinate system; the coordinate position of the target debris in the pixel coordinate system is determined according to the centroid coordinate position of the target debris in the pixel coordinate system and the size of the minimum circumscribed rectangle.

[0192] In a possible implementation manner of the embodiment of the present application, the third determination module 705 is further used to: transform the coordinate position of the target debris in the pixel coordinate system into the world coordinate system to obtain the coordinate position of the target debris in the world coordinate system; transform the coordinate position of the target debris in the world coordinate system into the robot coordinate system in combination with the posture information of the robot to obtain the coordinate position of the target debris in the robot coordinate system; determine the forward kinematics equation of the robot arm end coordinate system relative to the robot coordinate system; and obtain the expected arrival position of the robot arm according to the forward kinematics equation and the coordinate position of the target debris in the robot coordinate system.

[0193] In a possible implementation manner of the embodiment of the present application, the third determination module 705 is further used to: establish an inverse kinematics equation based on the forward kinematics equation using an algebraic method, and determine a rotation matrix and a translation matrix of the manipulator end coordinate system relative to the robot coordinate system according to the inverse kinematics equation; and transform the coordinate position of the target debris in the robot coordinate system according to the rotation matrix and translation matrix to obtain the coordinate position of the target debris in the manipulator end coordinate system as the expected arrival position of the manipulator.

[0194] In the image processing device provided in the embodiment of the present application, by collecting the target debris image, and determining the minimum closed contour of the target debris according to the known template image, and determining the eigenvector of the HU invariant moment according to the minimum closed contour. Then, the eigenvector of the HU invariant moment can be used to determine the target debris contour. By obtaining the coordinate position of the target debris in the pixel coordinate system from the target debris contour, and converting the coordinate position to the robot coordinate system, the expected arrival position of the robot arm can be obtained, and the robot arm can be driven from the current stop position to the expected arrival position to grab the target debris and realize the precise positioning of the target debris. In the present application, since the eigenvector of the HU invariant moment has the characteristics of translation invariance, rotation invariance, scale invariance, etc., the target debris contour can be accurately identified when the target debris moves and rotates, and the efficiency of identifying the target debris contour is improved.

[0195] It should be noted that the above explanation of the embodiment of the image processing method is also applicable to the image processing device of this embodiment, and will not be repeated here.

[0196] The collection, storage, use, processing, transmission, provision and application of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0197] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0198] This application is expected to provide an implementation scheme for users to selectively block the use or access of personal information data. That is, this application is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, risks can be minimized by limiting data collection and deleting data. In addition, when applicable, personal identifiers are removed from such personal information to protect the privacy of users.

[0199] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they contradict each other.

[0200] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0201] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0202] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0203] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0204] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0205] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0206] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. An image processing method, characterized in that: Applied to a coal preparation plant, the method comprises: Capturing images of target debris on the de-mediating sieve to obtain images of the target debris; Performing image preprocessing on a known template image and the target debris image to obtain a first image and a second image, determining a sampling rate of the first image as a sampling threshold, using an edge detection operator to extract edges from the second image when the sampling rate of the second image is greater than the sampling threshold, and tracking the edges to obtain an outer contour image, using the outer contour image as the minimum closed contour of the target debris, and determining the minimum closed contour of the target debris by performing a centroid image preprocessing algorithm on the second image when the sampling rate of the second image is not greater than the sampling threshold, and determining a eigenvector of the HU invariant moment according to the minimum closed contour, wherein the HU invariant moment includes E eigenvectors, where E is a positive integer greater than 1 or equal to 1; Use the E feature vectors of the HU invariant moment to perform image matching on the target debris image and the template image in different search windows, obtain a first feature of the template image and a second feature of the target debris image in the search window for each search window, respectively operate the first feature and the second feature with the e-th feature vector of the HU invariant moment to obtain a first operation result and a second operation result corresponding to the e-th feature vector, determine the Euclidean distance corresponding to the search window according to the first operation result and the second operation result corresponding to each of the E feature vectors, obtain the minimum Euclidean distance from the Euclidean distances corresponding to N different search windows, and when the minimum Euclidean distance is less than a set distance threshold, use the search window corresponding to the minimum Euclidean distance as the search window with the greatest similarity, and use the image area in the search window with the greatest similarity as the target debris contour, wherein e is a positive integer, 1≤e≤E, and N is a positive integer greater than 1; Determine the minimum enclosing rectangle in pixel coordinates u The maximum and minimum coordinate values ​​in the axis direction and the minimum bounding rectangle in the pixel coordinate system v The maximum and minimum coordinate values ​​in the axis direction; according to the u The maximum and minimum coordinate values ​​in the axis direction are consistent with the v The maximum coordinate value and the minimum coordinate value in the axial direction are used to determine the centroid coordinate position of the target debris in the pixel coordinate system; the coordinate position of the target debris in the pixel coordinate system is determined according to the centroid coordinate position of the target debris in the pixel coordinate system and the size of the minimum circumscribed rectangle; The coordinate position of the target debris in the pixel coordinate system is converted into the robot coordinate system, the expected arrival position of the robot arm is determined, and the robot arm is driven to move from the current stop position to the expected arrival position; In response to the mechanical arm moving to the desired arrival position, controlling the mechanical arm to grab the target debris on the de-mediating screen and return to the stop position; Determining the Euclidean distance corresponding to the search window according to the first operation result and the second operation result corresponding to each of the E feature vectors, including: obtaining the square difference between the first operation result and the second operation result corresponding to the e-th feature vector; summing the square differences corresponding to the E feature vectors to obtain a sum value, and obtaining the Euclidean distance of the search window according to the sum value; The coordinate position of the target debris in the pixel coordinate system is converted into the robot coordinate system to determine the expected arrival position of the robot arm, including: converting the coordinate position of the target debris in the pixel coordinate system into the world coordinate system to obtain the coordinate position of the target debris in the world coordinate system; converting the coordinate position of the target debris in the world coordinate system into the robot coordinate system in combination with the posture information of the robot to obtain the coordinate position of the target debris in the robot coordinate system; determining the forward kinematics equation of the coordinate system of the end of the robot arm relative to the robot coordinate system; and obtaining the expected arrival position of the robot arm according to the forward kinematics equation and the coordinate position of the target debris in the robot coordinate system.

2. The method according to claim 1, characterized in that Determining the characteristic vector of the HU invariant moment according to the minimum closed contour includes: Determine a p+q order central moment expression, and normalize the p+q order central moment expression to obtain a p+q order normalized central moment expression, where p and q are positive integers greater than or equal to 1; Determine the area of ​​the target debris and the center of mass of the target debris according to the minimum closed contour, and determine the coordinates of the center of gravity of the image according to the area and the center of mass of the target debris; Determine discrete image parameters according to the pixel coordinates within the minimum closed contour, and determine the second-order normalized central moment and the third-order normalized central moment according to the discrete image parameters, the pixel coordinates and the image centroid coordinates in combination with the p+q-order normalized central moment expression; The eigenvector of the HU invariant moment is determined according to the second-order normalized central moment and the third-order normalized central moment.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: When the minimum Euclidean distance is greater than or equal to the set distance threshold, the search window is adjusted, and the Euclidean distances corresponding to N different search windows are re-acquired and subsequent operations are performed until the target debris contour is obtained.

4. The method according to claim 1, characterized in that: The step of obtaining the expected arrival position of the robot arm according to the forward kinematics equation and the coordinate position of the target debris in the robot coordinate system includes: Based on the forward kinematics equation, an inverse kinematics equation is established by an algebraic method, and a rotation matrix and a translation matrix of the manipulator end coordinate system relative to the robot coordinate system are determined according to the inverse kinematics equation; According to the rotation matrix and the translation matrix, the coordinate position of the target debris in the robot coordinate system is transformed to obtain the coordinate position of the target debris in the robot arm end coordinate system as the expected arrival position of the robot arm.

5. An image processing device, characterized in that: Applied to a coal preparation plant, the device comprises: The acquisition module is used to acquire images of target debris on the de-mediation screen to obtain images of the target debris; A first determination module is used to perform image preprocessing on a known template image and the target debris image to obtain a first image and a second image, determine a sampling rate of the first image as a sampling threshold, and when the sampling rate of the second image is greater than the sampling threshold, use an edge detection operator to extract edges from the second image, and track the edges to obtain an outer contour image, and use the outer contour image as the minimum closed contour of the target debris. When the sampling rate of the second image is not greater than the sampling threshold, determine the minimum closed contour of the target debris by performing a centroid image preprocessing algorithm on the second image, and determine a eigenvector of the HU invariant moment based on the minimum closed contour, wherein the HU invariant moment includes E eigenvectors, wherein E is a positive integer greater than 1 or equal to 1, and N is a positive integer greater than 1; a matching module, configured to use the E feature vectors of the HU invariant moment to perform image matching on the target debris image and the template image in different search windows, obtain a first feature of the template image and a second feature of the target debris image in the search window for each search window, respectively operate the first feature and the second feature with the e-th feature vector of the HU invariant moment to obtain a first operation result and a second operation result corresponding to the e-th feature vector, determine the Euclidean distance corresponding to the search window according to the first operation result and the second operation result corresponding to each of the E feature vectors, and obtain a target debris contour according to the Euclidean distance corresponding to the search window, wherein e is a positive integer and 1≤e≤E; The second determination module is used to determine the minimum bounding rectangle in the pixel coordinate system. u The maximum and minimum coordinate values ​​in the axis direction and the minimum bounding rectangle in the pixel coordinate system v The maximum and minimum coordinate values ​​in the axis direction; according to the u The maximum and minimum coordinate values ​​in the axis direction are consistent with the v The maximum coordinate value and the minimum coordinate value in the axial direction are used to determine the centroid coordinate position of the target debris in the pixel coordinate system; the coordinate position of the target debris in the pixel coordinate system is determined according to the centroid coordinate position of the target debris in the pixel coordinate system and the size of the minimum circumscribed rectangle; A third determination module is used to convert the coordinate position of the target debris in the pixel coordinate system into the robot coordinate system, determine the expected arrival position of the robot arm, and drive the robot arm to move from the current stop position to the expected arrival position; A grabbing module, for controlling the robotic arm to grab the target debris on the de-mediating screen and return to the stop position in response to the robotic arm moving to the desired arrival position; Determining the Euclidean distance corresponding to the search window according to the first operation result and the second operation result corresponding to each of the E feature vectors, including: obtaining the square difference between the first operation result and the second operation result corresponding to the e-th feature vector; summing the square differences corresponding to the E feature vectors to obtain a sum value, and obtaining the Euclidean distance of the search window according to the sum value; The coordinate position of the target debris in the pixel coordinate system is converted into the robot coordinate system to determine the expected arrival position of the robot arm, including: converting the coordinate position of the target debris in the pixel coordinate system into the world coordinate system to obtain the coordinate position of the target debris in the world coordinate system; converting the coordinate position of the target debris in the world coordinate system into the robot coordinate system in combination with the posture information of the robot to obtain the coordinate position of the target debris in the robot coordinate system; determining the forward kinematics equation of the coordinate system of the end of the robot arm relative to the robot coordinate system; and obtaining the expected arrival position of the robot arm according to the forward kinematics equation and the coordinate position of the target debris in the robot coordinate system.

Images