Automatic feeding and discharging man-machine interaction method and system based on image recognition

Abstract

The invention discloses an automatic feeding and discharging man-machine interaction method and system based on image recognition. Comprising the following steps: acquiring blanking image data of a product in a blanking process; extracting feature information of the product according to the blanking image data to obtain product feature point data; obtaining the maximum feature point and the minimum feature point of the product according to the product feature point data; obtaining a limit position of the product according to the maximum feature point and the minimum feature point of the product; obtaining a first weight and a second weight of the blanking image data according to the limit position of the product; obtaining a product abnormal coefficient according to the first weight and the second weight of the blanking image data; according to the abnormal coefficient of the product, the abnormal condition of the discharging position is obtained; and according to the abnormal condition of the discharging position, the posture of the product at the discharging position is corrected. The problem that positioning and correction cannot be carried out due to the fact that the product position condition and the product position abnormal condition in the discharging process of the intelligent robot cannot be effectively detected in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to an automatic loading and unloading human-computer interaction method and system based on image recognition. Background Art

[0002] In one prior art, an intelligent robot loading and unloading system based on machine vision is connected to three conveyor lines, an intelligent mobile device and a visual device through a PLC controller to achieve automated operation. The first conveyor line delivers the workpiece to the second conveyor line, which transfers the workpiece to the third conveyor line for processing through a robot gripping device. The visual device consists of three visual cameras and a processing system, which is responsible for collecting and processing workpiece position information to ensure that the robot can accurately locate and grasp the workpiece, thereby achieving automated loading and unloading.

[0003] During the automatic loading and unloading process of the intelligent robot, the existing technology cannot effectively detect the product position during the unloading process of the intelligent robot, and cannot locate, correct and alarm for abnormal product position caused by erroneous operation. For example, the operator is unable to place the product at an angle deviation in the unloading position. Therefore, the unloading process of the intelligent robot cannot be effectively controlled, thereby greatly reducing the safety and reliability of the unloading process. Summary of the invention

[0004] The present invention provides a human-machine interaction method and system for automatic loading and unloading based on image recognition, so as to solve the problems in the prior art that the product position cannot be effectively detected during the unloading process of the intelligent robot, and the abnormal product position caused by erroneous operation cannot be located, corrected and alarmed.

[0005] In the first aspect, in order to solve the above-mentioned technical problems, the present invention provides an automatic loading and unloading human-computer interaction method based on image recognition, comprising: acquiring unloading image data of a product during the unloading process, wherein the unloading image data is obtained by preprocessing a product image collected by a machine vision module; extracting feature information of the product according to the unloading image data to obtain product feature point data; obtaining the maximum feature point of the product and the minimum feature point of the product according to the product feature point data; obtaining the extreme position of the product according to the maximum feature point of the product and the minimum feature point of the product; obtaining a first weight of the unloading image data and a second weight of the image data according to the extreme position of the product; obtaining a product abnormality coefficient according to the first weight of the unloading image data and the second weight of the unloading image data; obtaining an abnormality of the unloading position during the unloading process according to the product abnormality coefficient; and correcting the posture of the product at the unloading position according to the abnormality of the unloading position.

[0006] As an optional implementation, the obtaining of the blanking image data of the product during the blanking process includes: placing a machine vision module directly above the product to obtain a product image, and the product image is a plan view of the product; performing analog-to-digital conversion according to the plan view of the product to obtain RGB three color channel information; calculating grayscale information according to the RGB three color channel information to obtain grayscale information of the product image; performing two-dimensional Gaussian filtering according to the grayscale information of the product image to obtain blanking image data; wherein the two-dimensional Gaussian filtering process includes calculating the Gaussian filtering function corresponding to the pixels on the product image: In the formula, Represents the horizontal coordinate value of the pixel in the two-dimensional coordinate system, Represents the vertical coordinate value of the pixel in the two-dimensional coordinate system, is the preset Gaussian standard deviation, express Gaussian filter function value at the coordinate point; wherein, according to the Gaussian filter function value, a convolution operation is performed with the image to obtain image filtering information; the convolution operation process is as follows: In the formula, for Image filtering information under coordinates, for The Gaussian filter function value under the coordinate, for Grayscale information at the coordinates, To establish the coordinate system, the maximum value of the horizontal coordinate is minus half of the minimum value.

[0007] As an optional implementation, extracting the characteristic information of the product according to the cutting image data to obtain product characteristic point data includes: obtaining pixel digitization information according to the cutting image data; performing information comparison according to the pixel digitization information to obtain the characteristic information of the product; obtaining product characteristic point data according to the characteristic information of the product; wherein, obtaining pixel digitization information according to the cutting image data includes: dividing the captured cutting image data into pixels according to the resolution of the machine vision module to obtain matrix information; obtaining a corresponding voltage signal according to the matrix information; performing digital-to-analog conversion according to the voltage signal to obtain pixel digitization information.

[0008] As an optional implementation, the method of obtaining the extreme position of the product based on the maximum feature point of the product and the minimum feature point of the product includes: performing regional position judgment based on the maximum feature point of the product and the minimum feature point of the product to obtain position area information; performing numerical processing based on the position area information to obtain numerical information; and obtaining the extreme position of the product based on the numerical information.

[0009] As an optional implementation, the obtaining of the first weight of the blanking image data and the second weight of the blanking image data according to the extreme position of the product includes: obtaining a product coordinate position set according to the extreme position of the product; obtaining the first weight of the blanking image data according to the product coordinate position set; calculating the absolute value of the difference between the horizontal coordinates according to the product coordinate position set to obtain the maximum horizontal coordinate difference; calculating the absolute value of the difference between the vertical coordinates according to the product coordinate position set to obtain the maximum vertical coordinate difference; calculating the product of the maximum horizontal coordinate difference and the maximum vertical coordinate difference according to the maximum horizontal coordinate difference and the maximum vertical coordinate difference to obtain product information; performing a reciprocal operation according to the product information to obtain the second weight of the blanking image data; wherein, the first weight of the blanking image data according to the product coordinate position set is determined by: In the formula, Indicates The maximum value of the horizontal coordinate in the coordinate set of products, Indicates The minimum value of the horizontal coordinate in the coordinate set of products, Indicates The maximum value of the vertical coordinate in the coordinate set of products, Indicates The minimum value of the ordinate in the coordinate set of products, Indicates the product serial data The first weight of the product.

[0010] As an optional implementation, the product abnormality coefficient is obtained according to the first weight of the cutting image data and the second weight of the cutting image data, including: obtaining weight-corrected product feature point data according to the first weight of the cutting image data and the second weight of the cutting image data; comparing the weight-corrected product feature point data with the standard position to obtain the correctness of the product cutting position; comparing the correctness of the product cutting position with the correctness threshold to obtain the product abnormality coefficient.

[0011] As an optional implementation, the abnormality of the material unloading position is obtained according to the product abnormality coefficient, including: performing numerical judgment according to the product abnormality coefficient to obtain the maximum allowable error of the product; judging according to the maximum allowable error of the product and the actual placement of the product to obtain the abnormality of the material unloading position; wherein, performing numerical judgment according to the product abnormality coefficient to obtain the maximum allowable error of the product includes: In the formula, Indicates the minimum allowable error. Indicates the maximum allowable error. Indicates the average value of the blanking image data information, Indicates the standard deviation of the blanking image data information; according to the minimum allowable error value and the maximum allowable error value, the abnormal situation of the blanking position is obtained, including: At this time, it is judged as abnormal; where, represents the anomaly coefficient; At this time, it is judged to be normal.

[0012] As an optional implementation, the method of correcting the posture of the product at the unloading position according to the abnormal situation at the unloading position includes: according to the abnormal situation at the unloading position, when the abnormal situation at the unloading position is normal, the display screen displays a normal image; according to the abnormal situation at the unloading position, when the abnormal situation at the unloading position is abnormal, the display screen displays a prompt message.

[0013] In a second aspect, the present invention provides an automatic loading and unloading human-computer interaction system based on image recognition, comprising: an image acquisition module, used to acquire the unloading image data of the product during the unloading process; a feature point acquisition module, used to extract the feature information of the product according to the unloading image data, and obtain the product feature point data; a data comparison module, used to obtain the maximum feature point of the product and the minimum feature point of the product according to the product feature point data; an extreme feature acquisition module, used to obtain the extreme position of the product according to the maximum feature point of the product and the minimum feature point of the product; a data correction module, used to obtain the first weight of the unloading image data and the second weight of the unloading image data according to the extreme position of the product; an abnormality coefficient acquisition module, used to obtain the product abnormality coefficient according to the first weight of the unloading image data and the second weight of the unloading image data; an abnormality judgment module, used to obtain the abnormal situation of the unloading position according to the product abnormality coefficient; a posture correction module, used to correct the posture of the product at the unloading position according to the abnormal situation of the unloading position.

[0014] In a third aspect, the present invention further provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned human-computer interaction methods for automatic loading and unloading based on image recognition.

[0015] Compared with the prior art, the present invention has the following beneficial effects: the present invention provides an automatic loading and unloading human-computer interaction method and system based on image recognition, the method is executed by a controller, and includes: obtaining the unloading image data of the product in the unloading process, wherein the unloading image data is obtained by preprocessing the product image collected by a machine vision module; according to the unloading image data, extracting the feature information of the product to obtain the product feature point data; according to the product feature point data, obtaining the maximum feature point of the product and the minimum feature point of the product; according to the maximum feature point of the product and the minimum feature point of the product, obtaining the extreme position of the product; according to the extreme position of the product, obtaining the first weight of the unloading image data and the second weight of the unloading image data; according to the first weight of the unloading image data and the second weight of the unloading image data, obtaining the product abnormality coefficient; according to the product abnormality coefficient, obtaining the abnormality of the unloading position in the unloading process; according to the abnormality of the unloading position, correcting the posture of the product at the unloading position.

[0016] In the process of product unloading, first obtain the unloading image data, then extract the feature information of the product from these data, and determine the product feature point data. Next, identify the maximum and minimum feature points of the product based on these feature point data, and then determine the limit position of the product. Based on the limit position, calculate the first weight and the second weight of the unloading image data, and use these weights to calculate the product abnormality coefficient. Finally, determine whether there is an abnormality in the unloading position based on the abnormality coefficient, and correct the posture of the product at the unloading position accordingly to ensure the accuracy and efficiency of the unloading process. The above method solves the problem that the prior art cannot effectively detect the product position during the unloading process of the intelligent robot, and cannot locate, correct and alarm the abnormal product position caused by erroneous operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of a human-machine interaction method for automatic loading and unloading based on image recognition provided by the first embodiment of the present invention; Figure 2 It is a structural schematic diagram of a human-computer interaction method for automatic loading and unloading based on image recognition provided by the second embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] At present, intelligent robot technology is a highly advanced artificial intelligence technology used to simulate and realize various human thinking activities, such as perception, learning, understanding, thinking, planning, execution, operation, etc. Intelligent robot is a highly autonomous intelligent machine with a semi-human body, similar to humans in movement and behavior, and can replace humans to perform complex, arduous and even dangerous heavy physical labor. Intelligent robots can accept human commands, run pre-programmed programs, or act according to the principles and guidelines formulated by artificial intelligence technology. Its mission is to assist or replace human work.

[0020] In one prior art, an intelligent robot loading and unloading system based on machine vision is connected to three conveyor lines, an intelligent mobile device and a visual device through a PLC controller to achieve automated operation. The first conveyor line delivers the workpiece to the second conveyor line, which transfers the workpiece to the third conveyor line for processing through a robot gripping device. The visual device consists of three visual cameras and a processing system, which is responsible for collecting and processing workpiece position information to ensure that the robot can accurately locate and grasp the workpiece, thereby achieving automated loading and unloading.

[0021] During the automatic loading and unloading process of the intelligent robot, the existing technology cannot effectively detect the product position during the unloading process of the intelligent robot, and cannot locate, correct and alarm for abnormal product position caused by erroneous operation. For example, the operator is unable to place the product at an angle deviation in the unloading position. Therefore, the unloading process of the intelligent robot cannot be effectively controlled, thereby greatly reducing the safety and reliability of the unloading process.

[0022] Reference Figure 1 The first embodiment of the present invention provides a human-machine interaction method for automatic loading and unloading based on image recognition, comprising the following steps: S11, obtaining blanking image data of the product during the blanking process, wherein the blanking image data is obtained by preprocessing the product image collected by the machine vision module; S12, extracting feature information of the product according to the blanking image data to obtain product feature point data; S13, obtaining the maximum feature point and the minimum feature point of the product according to the product feature point data; S14, obtaining a limit position of the product according to the maximum feature point of the product and the minimum feature point of the product; S15, obtaining a first weight of the blanking image data and a second weight of the blanking image data according to the extreme position of the product; S16, obtaining a product abnormality coefficient according to the first weight of the blanking image data and the second weight of the blanking image data; S17, obtaining the abnormal situation of the material unloading position during the material unloading process according to the product abnormality coefficient; S18, correcting the posture of the product at the unloading position according to the abnormal situation of the unloading position.

[0023] It should be noted that the acquisition of the image data of the product during the blanking process is collected by the machine vision module, which is a key component in modern industrial automation and intelligent manufacturing. It captures and analyzes images through high-precision visual sensors to provide the necessary visual information for the automation system. This module usually includes different types of cameras, such as CCD (charge-coupled device) cameras, CMOS (complementary metal oxide semiconductor) cameras and line array cameras, each of which has unique advantages and application scenarios. CCD cameras are known for their high resolution, high sensitivity and excellent image quality, and are particularly suitable for applications that require high-quality image capture. CMOS cameras are favored for their fast imaging capabilities, low power consumption and cost-effectiveness, and are widely used in situations that require fast response. Line array cameras are used for continuous object detection and measurement tasks with their expertise in high-speed scanning and precise measurement.

[0024] Exemplarily, in this embodiment, the machine vision module uses a CMOS camera, which captures product images during the product unloading process and performs preprocessing. The obtained unloading image data contains detailed visual information of the product images during the product unloading process, which can be used for further image processing, analysis and decision making.

[0025] In the process of product unloading, first obtain the unloading image data, then extract the feature information of the product from these data, and determine the product feature point data. Next, identify the maximum and minimum feature points of the product based on these feature point data, and then determine the limit position of the product. Based on the limit position, calculate the first weight and the second weight of the unloading image data, and use these weights to calculate the product abnormality coefficient. Finally, determine whether there is an abnormality in the unloading position based on the abnormality coefficient, and correct the posture of the product at the unloading position accordingly to ensure the accuracy and efficiency of the product unloading process. The above method can solve the problem that the prior art cannot effectively detect the product position during the unloading process of the intelligent robot, and cannot locate, correct and alarm the abnormal product position caused by erroneous operation.

[0026] In step S11, the obtaining of the blanking image data of the product during the blanking process includes: placing a machine vision module directly above the product to obtain a plan view of the product; performing analog-to-digital conversion according to the plan view of the product to obtain RGB three color channel information; calculating grayscale information according to the RGB three color channel information to obtain grayscale information of the product image; performing two-dimensional Gaussian filtering according to the grayscale information of the product image to obtain blanking image data; wherein the two-dimensional Gaussian filtering process includes calculating the Gaussian filtering function corresponding to the pixels on the image information: In the formula, Represents the horizontal coordinate value of the pixel in the two-dimensional coordinate system, Indicates the vertical coordinate value of the pixel coordinate in the two-dimensional coordinate system. is the Gaussian standard deviation preset by the system, express Gaussian filter function value at the coordinate point; wherein, according to the Gaussian filter function value, a convolution operation is performed with the image to obtain image filtering information; the convolution operation process is as follows: In the formula, for Image filtering information under coordinates, for The Gaussian filter function value under the coordinate, for Grayscale information at the coordinates, To establish the coordinate system, the maximum value of the horizontal coordinate is minus half of the minimum value.

[0027] It should be noted that the grayscale information is calculated based on the three RGB color channel information to obtain the grayscale information of the product image. The grayscale information of the product is calculated by the following formula: In the formula, represents the intensity of the red channel, Represent the intensity of the green channel, represents the intensity of the blue channel, Represents the grayscale information of the product.

[0028] In step S12, the extracting of feature information of the product according to the cutting image data to obtain feature point data of the product includes: obtaining pixel numerical information according to the cutting image data; performing information comparison according to the pixel numerical information to obtain feature information of the product; obtaining feature point data of the product according to the feature information of the product; wherein, obtaining pixel numerical information according to the cutting image data includes: dividing the captured cutting image data into pixels according to the resolution of the machine vision module to obtain matrix information; obtaining a corresponding voltage signal according to the matrix information; performing digital-to-analog conversion according to the voltage signal to obtain pixel numerical information.

[0029] It should be noted that according to the resolution of the machine vision module, the captured image is pixelated to obtain matrix information, including: the resolution of the machine vision module varies according to the selected product. The example of the present invention uses a machine vision module with a resolution of 1080×1080. The specific resolution is determined according to the selected product and is not specifically described here.

[0030] It should be further explained that the pixelation of the captured image specifically includes decomposing it into small modules at the pixel level according to the resolution of the visual sensor used in the experiment, each module transmits a unique voltage, and subsequent conversion is performed according to the transmitted voltage.

[0031] In step S13, the maximum feature point of the product and the minimum feature point of the product are obtained according to the product feature point data, including: sorting and integrating information according to the product feature point data to obtain the maximum value and the minimum value of the feature point data, and recording the position information corresponding to the maximum feature point and the minimum feature point, and marking the maximum feature point and the minimum feature point as the maximum feature point of the product and the minimum feature point of the product.

[0032] In step S14, the extreme position of the product is obtained according to the maximum feature point of the product and the minimum feature point of the product, including: performing regional position judgment according to the maximum feature point of the product and the minimum feature point of the product to obtain position area information; performing numerical processing according to the position area information to obtain numerical information; and obtaining the extreme position of the product according to the numerical information.

[0033] It should be noted that the numerical processing is performed based on the location area information to obtain numerical information, including: marking the location area information in an established coordinate system, and determining the position based on the horizontal and vertical coordinate points on the coordinate system.

[0034] It should be further explained that the coordinate system is established with the geometric midpoint of the product image captured by the machine vision module as the zero point, the horizontal direction towards the robot arm as the positive direction of the X-axis, and the vertical direction towards the robot arm as the positive direction of the Y-axis.

[0035] In step S15, the first weight of the blanking image data and the second weight of the blanking image data are obtained according to the extreme position of the product, including: obtaining a product coordinate position set according to the extreme position of the product; obtaining the first weight of the blanking image data according to the product coordinate position set; calculating the absolute value of the difference between the horizontal coordinates according to the product coordinate position set to obtain the maximum horizontal coordinate difference; calculating the absolute value of the difference between the vertical coordinates according to the product coordinate position set to obtain the maximum vertical coordinate difference; calculating the product of the maximum horizontal coordinate difference and the maximum vertical coordinate difference according to the maximum horizontal coordinate difference and the maximum vertical coordinate difference to obtain product information; performing a reciprocal operation according to the product information to obtain the second weight of the blanking image data; wherein, the first weight of the blanking image data is obtained according to the product coordinate position set, and the determination method is as follows: In the formula, Indicates The maximum value of the horizontal coordinate in the coordinate set of products, Indicates The minimum value of the horizontal coordinate in the coordinate set of products, Indicates The maximum value of the vertical coordinate in the coordinate set of products, Indicates The minimum value of the ordinate in the coordinate set of products, Indicates the product serial data The first weight of the product.

[0036] It should be noted that obtaining a set of product coordinate positions according to the extreme positions of the product includes: obtaining a set of coordinate positions of the extreme positions of products of the same shape to obtain a set of product coordinate positions.

[0037] In step S16, the product abnormality coefficient is obtained according to the first weight of the cutting image data and the second weight of the cutting image data, including: obtaining weight-corrected product feature point data according to the first weight of the cutting image data and the second weight of the cutting image data; comparing the weight-corrected product feature point data with the standard position to obtain the correctness of the product position; comparing the correctness of the product position with the correctness threshold to obtain the product abnormality coefficient.

[0038] It should be noted that the weight-corrected product feature point data is obtained based on the first weight of the cutting image data and the second weight of the cutting image data, including: multiplying the feature information of the product by the first weight information corresponding to the cutting image data to obtain first correction information; multiplying the feature information of the product by the second weight information corresponding to the cutting image data to obtain second correction information; adding the first correction information and the second correction information to obtain the weight-corrected product feature point data.

[0039] In step S17, the abnormality of the material unloading position is obtained according to the abnormality coefficient of the product, including: performing numerical judgment according to the abnormality coefficient of the product to obtain the maximum allowable error of the product; judging according to the maximum allowable error of the product and the actual placement of the product to obtain the abnormality of the material unloading position; wherein, performing numerical judgment according to the abnormality coefficient of the product to obtain the maximum allowable error of the product includes: In the formula, Indicates the minimum allowable error. Indicates the maximum allowable error. Indicates the average value of the blanking image data information, Indicates the standard deviation of the blanking image data information; according to the minimum allowable error value and the maximum allowable error value, the abnormal situation of the blanking position is obtained, including: At this time, it is judged as abnormal; where, represents the anomaly coefficient; At this time, it is judged to be normal.

[0040] It should be noted that when it is judged as abnormal, Marked as 0, when judged as normal, Marked as 1.

[0041] In step S18, according to the abnormal situation of the unloading position, the posture of the product at the unloading position is corrected, including: according to the abnormal situation of the unloading position, when the abnormal situation of the unloading position is normal, the display screen displays a normal image; according to the abnormal situation of the unloading position, when the abnormal situation of the unloading position is abnormal, the display screen displays a prompt message.

[0042] It should be noted that when the abnormal situation of the unloading position is abnormal, the display screen displays prompt information, including: the display screen prompts to adjust the unloading position of the product to the left or right to eliminate the abnormal situation.

[0043] In order to facilitate the understanding of the present invention, some preferred embodiments of the present invention are further described below.

[0044] In this embodiment, an automatic loading and unloading human-machine device based on image recognition is proposed, and the device consists of the following key parts: an image acquisition device, an image processing device, a feature point analysis device, a weight calculation device, an abnormality coefficient calculation device, an abnormality judgment device and a posture correction device. Among them, the image acquisition device is used to collect the unloading image data of the product during the unloading process, and then the image processing device performs image preprocessing on the collected image data, extracts the feature information of the product, and obtains the product feature point data; according to the product feature point data, the maximum feature point of the product and the minimum feature point of the product are obtained; according to the maximum feature point of the product and the minimum feature point of the product, the extreme position of the product is obtained; according to the extreme position of the product, the first weight of the unloading image data and the second weight of the unloading image data are obtained; according to the first weight of the unloading image data and the second weight of the unloading image data, the product abnormality coefficient is obtained; according to the product abnormality coefficient, the abnormal situation of the unloading position is obtained; according to the abnormal situation of the unloading position, the posture of the product at the unloading position is corrected. The specific working steps are as follows: Step 1: Place the machine vision module directly above the product to obtain a plan view of the product. Through analog-to-digital conversion, extract the RGB three color channel information from the plan view, and calculate the grayscale information to obtain the grayscale data of the product image. Subsequently, use a two-dimensional Gaussian filter to process the grayscale information to eliminate noise and enhance image features, and finally obtain clear blanking image data; Step 2: Get pixel numerical information from the blanking image data, which includes dividing the image into pixels, obtaining the corresponding voltage signal, and obtaining pixel numerical information through digital-to-analog conversion. Then, based on the pixel numerical information, extract the product feature information through information comparison, and obtain the product feature point data accordingly; Step 3: Based on the product feature point data, identify the maximum and minimum feature points of the product, and then make regional position judgments to obtain position area information. By numerically processing these position area information, the ultimate position of the product is finally obtained, that is, the maximum and minimum boundaries of the product during the blanking process; Step 4: According to the extreme position of the product, obtain the product coordinate position set, and calculate the first weight of the blanking image data accordingly. At the same time, calculate the absolute value of the difference between the horizontal and vertical coordinates, obtain the maximum horizontal coordinate difference and the maximum vertical coordinate difference, and calculate their product, and obtain the second weight by performing the inverse operation on the product; Step 5: According to the first weight and the second weight of the blanking image data, the product feature point data is weighted and compared with the standard position to obtain the correctness of the product position. Then, the correctness of the product position is compared with the preset correctness threshold, and finally the product abnormality coefficient is obtained; Step 6: According to the product abnormality coefficient, make a numerical judgment to obtain the maximum allowable error of the product, and compare it with the actual placement of the product to obtain the abnormal situation of the unloading position. According to the abnormal situation, the display screen will display a normal image or prompt information to guide the operator or the automation system to make corresponding corrections to the posture of the product at the unloading position.

[0045] In summary, the present invention provides an automatic loading and unloading human-computer interaction method and system based on image recognition. The method is executed by a controller, including: obtaining the unloading image data of the product during the unloading process; extracting the characteristic information of the product according to the unloading image data, and obtaining the product characteristic point data; obtaining the maximum characteristic point and the minimum characteristic point of the product according to the product characteristic point data; obtaining the limit position of the product according to the maximum characteristic point and the minimum characteristic point of the product; obtaining the first weight of the unloading image data and the second weight of the unloading image data according to the limit position of the product; obtaining the product abnormality coefficient according to the first weight of the unloading image data and the second weight of the unloading image data; obtaining the abnormality of the unloading position according to the product abnormality coefficient; and correcting the posture of the product at the unloading position according to the abnormality of the unloading position. In the unloading process of the product, firstly obtain the unloading image data, then extract the characteristic information of the product from these data, and determine the product characteristic point data. Then, identify the maximum and minimum characteristic points of the product according to these characteristic point data, and then determine the limit position of the product. Based on the limit position, calculate the first weight and the second weight of the unloading image data, and use these weights to calculate the product abnormality coefficient. Finally, the abnormality coefficient is used to determine whether there is an abnormality in the unloading position, and the posture of the product at the unloading position is corrected accordingly to ensure the accuracy and efficiency of the unloading process. The above method solves the problem that the prior art cannot effectively detect the product position during the unloading process of the intelligent robot, and cannot locate, correct and alarm the abnormal product position caused by erroneous operation.

[0046] Reference Figure 2The second embodiment of the present invention provides an automatic loading and unloading human-machine interaction device based on image recognition, comprising: an image acquisition module, used to acquire the unloading image data of the product during the unloading process, wherein the unloading image data is obtained by preprocessing the product image collected by the machine vision module; a feature point acquisition module, used to extract the feature information of the product according to the unloading image data, and obtain the product feature point data; a data comparison module, used to obtain the maximum feature point of the product and the minimum feature point of the product according to the product feature point data; an extreme feature acquisition module, used to obtain the extreme position of the product according to the maximum feature point of the product and the minimum feature point of the product; a data correction module, used to obtain the first weight of the unloading image data and the second weight of the image data according to the extreme position of the product; an abnormality coefficient acquisition module, used to obtain the product abnormality coefficient according to the first weight of the unloading image data and the second weight of the unloading image data; an abnormality judgment module, used to obtain the abnormal situation of the unloading position according to the product abnormality coefficient; a posture correction module, used to correct the posture of the product at the unloading position according to the abnormal situation of the unloading position.

[0047] In one embodiment, the image acquisition module is used to acquire the blanking image data of the product during the blanking process, including: placing a machine vision module directly above the product to obtain a product image, and the product image is a plan view of the product; performing analog-to-digital conversion according to the plan view of the product to obtain RGB three color channel information; calculating grayscale information according to the RGB three color channel information to obtain grayscale information of the product image; performing two-dimensional Gaussian filtering according to the grayscale information of the product image to obtain blanking image data; wherein the two-dimensional Gaussian filtering process includes calculating the Gaussian filtering function corresponding to the pixels on the product image information: In the formula, Represents the horizontal coordinate value of the pixel in the two-dimensional coordinate system, Represents the vertical coordinate value of the pixel in the two-dimensional coordinate system, is the preset Gaussian standard deviation, express Gaussian filter function value at the coordinate point; wherein, according to the Gaussian filter function value, a convolution operation is performed with the image to obtain image filtering information; the convolution operation process is as follows: In the formula, for Image filtering information under coordinates, for The Gaussian filter function value under the coordinate, for Grayscale information at the coordinates, To establish the coordinate system, the maximum value of the horizontal coordinate is minus half of the minimum value.

[0048] In one embodiment, the feature point acquisition module is used to extract feature information of the product based on the cutting image data to obtain product feature point data, including: obtaining pixel digitization information based on the cutting image data; performing information comparison based on the pixel digitization information to obtain feature information of the product; obtaining product feature point data based on the feature information of the product; wherein, obtaining pixel digitization information based on the cutting image data includes: pixelating the captured image according to the resolution of the machine vision module to obtain matrix information; obtaining a corresponding voltage signal based on the matrix information; performing digital-to-analog conversion based on the voltage signal to obtain pixel digitization information.

[0049] In one implementation, the data comparison module is used to obtain the maximum feature point and the minimum feature point of the product based on the product feature point data.

[0050] In one embodiment, the extreme feature acquisition module is used to obtain the extreme position of the product based on the maximum feature point of the product and the minimum feature point of the product, including: performing regional position judgment based on the maximum feature point of the product and the minimum feature point of the product to obtain position area information; performing numerical processing based on the position area information to obtain numerical information; and obtaining the extreme position of the product based on the numerical information.

[0051] In one embodiment, the data correction module is used to obtain the first weight of the blanking image data and the second weight of the blanking image data according to the extreme position of the product, including: obtaining a product coordinate position set according to the extreme position of the product; obtaining the first weight of the blanking image data according to the product coordinate position set; calculating the absolute value of the difference between the horizontal coordinates according to the product coordinate position set to obtain the maximum horizontal coordinate difference; calculating the absolute value of the vertical coordinate difference according to the product coordinate position set to obtain the maximum vertical coordinate difference; calculating the product of the maximum horizontal coordinate difference and the maximum vertical coordinate difference according to the maximum horizontal coordinate difference and the maximum vertical coordinate difference to obtain product information; performing a reciprocal operation according to the product information to obtain the second weight of the blanking image data; wherein, the first weight of the blanking image data obtained according to the product coordinate position set is determined by: In the formula, Indicates The maximum value of the horizontal coordinate in the coordinate set of products, Indicates The minimum value of the horizontal coordinate in the coordinate set of products, Indicates The maximum value of the vertical coordinate in the coordinate set of products, Indicates The minimum value of the ordinate in the coordinate set of products, Indicates the product serial data The first weight of the product.

[0052] In one embodiment, the abnormal data acquisition module is used to obtain a product abnormality coefficient based on a first weight of the cutting image data and a second weight of the cutting image data, including: obtaining weight-corrected product feature point data based on the first weight of the cutting image data and the second weight of the cutting image data; comparing the weight-corrected product feature point data with the standard position to obtain the degree of correctness of the product position; comparing the degree of correctness of the product position with a correctness threshold to obtain the product abnormality coefficient.

[0053] In one embodiment, the abnormality determination module is used to obtain the abnormality of the material unloading position according to the product abnormality coefficient, including: performing numerical judgment according to the product abnormality coefficient to obtain the maximum allowable error of the product; performing judgment according to the maximum allowable error of the product and the actual placement of the product to obtain the abnormality of the material unloading position; wherein, performing numerical judgment according to the product abnormality coefficient to obtain the maximum allowable error of the product includes: In the formula, Indicates the minimum allowable error. Indicates the maximum allowable error. Indicates the average value of the blanking image data information, Indicates the standard deviation of the blanking image data information; according to the minimum allowable error value and the maximum allowable error value, the abnormal situation of the blanking position is obtained, including: At this time, it is judged as abnormal; where, represents the anomaly coefficient; At this time, it is judged to be normal.

[0054] In one embodiment, the posture correction module is used to correct the posture of the product at the unloading position according to the abnormal situation of the unloading position, including: according to the abnormal situation of the unloading position, when the abnormal situation of the unloading position is normal, the display screen displays a normal image; according to the abnormal situation of the unloading position, when the abnormal situation of the unloading position is abnormal, the display screen displays a prompt message.

[0055] In summary, the present invention provides a human-machine interaction method and system for automatic loading and unloading based on image recognition, the method is executed by a controller, including: The present invention provides a human-machine interaction method and system for automatic loading and unloading based on image recognition, the method is executed by a controller, including: obtaining the unloading image data of the product in the unloading process, wherein the unloading image data is obtained by preprocessing the product image collected by the machine vision module; extracting the characteristic information of the product according to the unloading image data, and obtaining the product characteristic point data; obtaining the maximum characteristic point of the product and the minimum characteristic point of the product according to the product characteristic point data; obtaining the extreme position of the product according to the maximum characteristic point of the product and the minimum characteristic point of the product; obtaining the first weight of the unloading image data and the second weight of the unloading image data according to the extreme position of the product; obtaining the product abnormality coefficient according to the first weight of the unloading image data and the second weight of the unloading image data; obtaining the abnormality of the unloading position in the unloading process according to the product abnormality coefficient; and correcting the posture of the product at the unloading position according to the abnormality of the unloading position. In the unloading process of the product, firstly obtain the unloading image data, then extract the characteristic information of the product from these data, and determine the product characteristic point data. Next, the maximum and minimum feature points of the product are identified based on these feature point data, and then the extreme position of the product is determined. Based on the extreme position, the first weight and the second weight of the blanking image data are calculated, and the product abnormality coefficient is calculated using these weights. Finally, it is determined whether there is an abnormality in the blanking position based on the abnormality coefficient, and the posture of the product at the blanking position is corrected accordingly to ensure the accuracy and efficiency of the blanking process. The above method solves the problem that the prior art cannot effectively detect the product position during the blanking process of the intelligent robot, and cannot locate, correct and alarm the abnormal product position caused by erroneous operation.

[0056] It should be noted that an automatic loading and unloading human-computer interaction device based on image recognition provided in an embodiment of the present invention is used to execute all the process steps of an automatic loading and unloading human-computer interaction method based on image recognition in the above embodiment. The working principles and beneficial effects of the two correspond one to one, and therefore will not be repeated here.

[0057] The embodiment of the present invention further provides an electronic device. The electronic device includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, such as a data preprocessing program. When the processor executes the computer program, the steps in the above-mentioned embodiments of the automatic loading and unloading human-computer interaction method based on image recognition are implemented, such as Figure 1 Alternatively, when the processor executes the computer program, the functions of the modules / units in the above-mentioned device embodiments are realized, such as the image acquisition module.

[0058] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the electronic device.

[0059] The electronic device may be a computing device such as a desktop computer, a notebook, a PDA, and a smart tablet. The electronic device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the above components are merely examples of electronic devices and do not constitute a limitation on the electronic device. The electronic device may include more or fewer components than the above components, or may combine certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0060] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device.

[0061] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the electronic device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0062] Wherein, if the module / unit integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0063] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0064] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. The human-computer interaction method for automatic loading and unloading based on image recognition is characterized in that: include: Obtaining the blanking image data of the product during the blanking process, wherein the blanking image data is obtained by preprocessing the product image collected by the machine vision module; extracting the feature information of the product according to the blanking image data to obtain the feature point data of the product; obtaining the maximum feature point of the product and the minimum feature point of the product according to the feature point data of the product; obtaining the extreme position of the product according to the maximum feature point of the product and the minimum feature point of the product; obtaining the first weight of the blanking image data and the second weight of the blanking image data according to the extreme position of the product; obtaining the product abnormality coefficient according to the first weight of the blanking image data and the second weight of the blanking image data; obtaining the abnormality of the blanking position during the blanking process according to the product abnormality coefficient; correcting the posture of the product at the blanking position according to the abnormality of the blanking position.

2. The human-computer interaction method for automatic loading and unloading based on image recognition according to claim 1 is characterized in that: The method of obtaining the blanking image data of the product during the blanking process includes: placing a machine vision module directly above the product to obtain a product image, wherein the product image is a plan view of the product; performing analog-to-digital conversion according to the plan view of the product to obtain RGB three color channel information; calculating grayscale information according to the RGB three color channel information to obtain grayscale information of the product image; performing two-dimensional Gaussian filtering according to the grayscale information of the product image to obtain blanking image data; wherein the two-dimensional Gaussian filtering process includes calculating the Gaussian filtering function corresponding to the pixels on the product image: In the formula, Represents the horizontal coordinate value of the pixel in the two-dimensional coordinate system, Represents the vertical coordinate value of the pixel in the two-dimensional coordinate system, is the preset Gaussian standard deviation, express Gaussian filter function value at the coordinate point; wherein, according to the Gaussian filter function value, a convolution operation is performed with the image to obtain image filtering information; the convolution operation process is as follows: In the formula, for Image filtering information under coordinates, for The Gaussian filter function value under the coordinate, for Grayscale information at the coordinates, To establish the coordinate system, the maximum value of the horizontal coordinate is minus half of the minimum value.

3. The human-computer interaction method for automatic loading and unloading based on image recognition according to claim 1 is characterized in that: The extracting of product feature information according to the blanking image data to obtain product feature point data includes: obtaining pixel digitization information according to the blanking image data; performing information comparison according to the pixel digitization information to obtain product feature information; obtaining product feature point data according to the product feature information; wherein, obtaining pixel digitization information according to the blanking image data includes: dividing the captured blanking image data into pixels according to the resolution of a machine vision module to obtain matrix information; obtaining a corresponding voltage signal according to the matrix information; and performing digital-to-analog conversion according to the voltage signal to obtain pixel digitization information.

4. The human-computer interaction method for automatic loading and unloading based on image recognition according to claim 1 is characterized in that: The method of obtaining the extreme position of the product based on the maximum feature point of the product and the minimum feature point of the product includes: performing regional position judgment based on the maximum feature point of the product and the minimum feature point of the product to obtain position area information; performing numerical processing based on the position area information to obtain numerical information; and obtaining the extreme position of the product based on the numerical information.

5. The human-computer interaction method for automatic loading and unloading based on image recognition according to claim 1 is characterized in that: The method of obtaining the first weight of the blanking image data and the second weight of the blanking image data according to the extreme position of the product includes: obtaining a product coordinate position set according to the extreme position of the product; obtaining the first weight of the blanking image data according to the product coordinate position set; calculating the absolute value of the difference between the horizontal coordinates according to the product coordinate position set to obtain the maximum horizontal coordinate difference; calculating the absolute value of the difference between the vertical coordinates according to the product coordinate position set to obtain the maximum vertical coordinate difference; calculating the product of the maximum horizontal coordinate difference and the maximum vertical coordinate difference according to the maximum horizontal coordinate difference and the maximum vertical coordinate difference to obtain product information; performing a reciprocal operation according to the product information to obtain the second weight of the blanking image data; wherein, the method of obtaining the first weight of the blanking image data according to the product coordinate position set is as follows: In the formula, Indicates The maximum value of the horizontal coordinate in the coordinate set of products, Indicates The minimum value of the horizontal coordinate in the coordinate set of products, Indicates The maximum value of the vertical coordinate in the coordinate set of products, Indicates The minimum value of the ordinate in the coordinate set of products, Indicates the product serial data The first weight of the product.

6. The human-computer interaction method for automatic loading and unloading based on image recognition according to claim 1 is characterized in that: The method of obtaining a product abnormality coefficient according to the first weight of the cutting image data and the second weight of the cutting image data includes: obtaining weight-corrected product feature point data according to the first weight of the cutting image data and the second weight of the cutting image data; comparing the weight-corrected product feature point data with the standard position to obtain the correctness of the product position; and comparing the correctness of the product position with a correctness threshold to obtain the product abnormality coefficient.

7. The human-computer interaction method for automatic loading and unloading based on image recognition according to claim 1 is characterized in that: The method of obtaining the abnormality of the material unloading position according to the abnormality coefficient of the product includes: performing numerical judgment according to the abnormality coefficient of the product to obtain the maximum allowable error of the product; performing judgment according to the maximum allowable error of the product and the actual placement of the product to obtain the abnormality of the material unloading position; wherein, performing numerical judgment according to the abnormality coefficient of the product to obtain the maximum allowable error of the product includes: In the formula, Indicates the minimum allowable error. Indicates the maximum allowable error. Indicates the average value of the blanking image data information, Indicates the standard deviation of the blanking image data information; according to the minimum allowable error value and the maximum allowable error value, the abnormal situation of the blanking position is obtained, including: At this time, it is judged as abnormal; where, represents the anomaly coefficient; At this time, it is judged to be normal.

8. The human-computer interaction method for automatic loading and unloading based on image recognition according to claim 1 is characterized in that: The method of correcting the posture of the product at the unloading position according to the abnormal situation at the unloading position includes: according to the abnormal situation at the unloading position, when the abnormal situation at the unloading position is normal, the display screen displays a normal image; according to the abnormal situation at the unloading position, when the abnormal situation at the unloading position is abnormal, the display screen displays a prompt message.

9. An automatic loading and unloading human-computer interaction system based on image recognition, characterized in that: include: An image acquisition module is used to acquire the blanking image data of the product during the blanking process, wherein the blanking image data is obtained by preprocessing the product image collected by the machine vision module; a feature point acquisition module is used to extract the feature information of the product according to the blanking image data to obtain the product feature point data; a data comparison module is used to obtain the maximum feature point and the minimum feature point of the product according to the product feature point data; an extreme feature acquisition module is used to obtain the extreme position of the product according to the maximum feature point and the minimum feature point of the product; a data correction module is used to obtain the first weight of the blanking image data and the second weight of the image data according to the extreme position of the product; an abnormality coefficient acquisition module is used to obtain the product abnormality coefficient according to the first weight of the blanking image data and the second weight of the blanking image data; an abnormality judgment module is used to obtain the abnormal situation of the blanking position according to the product abnormality coefficient; a posture correction module is used to correct the posture of the product at the blanking position according to the abnormal situation of the blanking position.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the automatic loading and unloading human-computer interaction method based on image recognition as described in any one of claims 1 to 8.

Images