Intelligent overturning control method and device based on machine vision

Abstract

The invention relates to the technical field of automatic control, and discloses an intelligent turnover control method and device based on machine vision, and the method comprises the steps: obtaining the physical parameters of an object, a multi-view original image group, an ideal three-dimensional model, and the motion constraint and movement range of a turnover machine; the method comprises the following steps: firstly, preprocessing an image to generate a uniform image group, and identifying key feature points for three-dimensional reconstruction to obtain an initial three-dimensional model; and identifying and complementing the shielded region by using a deep learning algorithm to generate a complete three-dimensional model. Comparing the complete model with the ideal model, adjusting the posture of the object until the difference value is smaller than a threshold value, and sending into a turnover machine. Updating the three-dimensional model in real time, calculating overturning torque by combining physical parameters and motion constraints of an object, determining an optimal path and generating a smooth track; and finally, generating a control instruction in combination with the moving range. By calibrating the posture of the object, the object can be overturned more firmly, and accurate overturning of the object is achieved.

Description

Technical Field

[0001] The present invention relates to the field of automatic control technology, and in particular to an intelligent flipping control method and device based on machine vision. Background Art

[0002] Object flipping is a key link in scenarios such as industrial production and logistics. In industrial production, many process flows require processing or assembly of objects at different angles, such as welding and painting of parts in automobile manufacturing, and double-sided patching of circuit boards in electronic equipment production. Accurate and fast flipping can improve production efficiency and ensure product quality. Traditional flipping methods rely on manual operation of clamps to flip the product over, and then turn it back and forth after flipping, or use simple mechanical devices to flip the product over. The flipping action for complex workpieces is cumbersome, and there are problems such as insufficient precision and high labor intensity. With the development of automation technology, a variety of automatic flipping equipment and technologies have emerged, such as flipping mechanisms based on PLC control, flipping devices in automated production lines, and flipping platforms with angle adjustment functions. These technologies have improved the automation and flipping accuracy of flipping operations to a certain extent.

[0003] However, the existing technology still has some shortcomings. The existing automatic flipping system does not check and calibrate the posture of the object before it enters the flipping machine. During the object flipping process, the object posture is not detected in real time, resulting in object shaking or inaccurate positioning, causing damage to the object and causing unnecessary machine restarts and inspections, affecting the accuracy of the flipping. Summary of the invention

[0004] The present invention provides an intelligent flipping control method and device based on machine vision, which solves the shaking problem of the object by calibrating the posture of the object before entering a flipping machine and during the flipping process, thereby improving the accuracy and efficiency of the flipping.

[0005] In a first aspect, the present invention provides an intelligent flipping control method based on machine vision, which mainly includes: Obtain the physical parameters of the object to be processed, the original image group of multiple perspectives, obtain the ideal 3D model, the moving range and motion constraints of the flip machine; pre-process the original image group to obtain a uniform image group; identify the key feature points of the object in the uniform image group, and perform 3D reconstruction on the key feature points to generate an initial 3D model of the object; predict the position of the feature points of the occluded area of ​​the initial 3D model based on the uniform image group based on the deep learning algorithm, complete the occluded area 3D model of the initial 3D model, and obtain a complete 3D model; compare the complete 3D model with the ideal 3D model to obtain the The static posture difference value of the complete three-dimensional model; adjusting the object until the static posture difference value is less than a preset difference threshold, and sending the object to a flipping machine for flipping, updating the complete three-dimensional model in real time, and obtaining a real-time three-dimensional model; calculating the torque generated during the flipping process according to the physical parameters and the real-time three-dimensional model, and determining the optimal flipping path in combination with the motion constraints of the flipping machine; smoothing the optimal flipping path to generate a smooth flipping trajectory from the current position to the target position; combining the moving range, the motion constraints and the smooth flipping trajectory, generating a control instruction to control the flipping machine to flip the object.

[0006] In an optional implementation, preprocessing the original image group to obtain a uniform image group includes: Performing denoising processing on the original image group to obtain a denoised image group; Adjusting the overall brightness distribution of the denoised image group to obtain a balanced image group; Dividing the equalized image group into a plurality of regions, and determining an initial texture pattern of each region; Completing the initial texture pattern and eliminating edge traces between regions to generate a complete texture pattern; The complete texture pattern is merged with the balanced image group to generate a final uniform image group.

[0007] In an optional implementation, the identifying key feature points of the object in the uniform image group and performing three-dimensional reconstruction on the key feature points to generate an initial three-dimensional model of the object includes: Identify key feature points of an object from the uniform image group, and obtain two-dimensional coordinates of the key feature points; Based on the two-dimensional coordinates, calculating the three-dimensional spatial position of the key feature point; Constructing an initial three-dimensional point cloud model of the object according to the three-dimensional spatial position; Performing surface fitting on the initial three-dimensional point cloud model to generate an initial three-dimensional surface model of the object; For the initial three-dimensional surface model, adjusting the density and structure of the grid to obtain an optimized three-dimensional grid model; According to the optimized three-dimensional mesh model, the texture information of the uniform image group is mapped to the model surface to generate an initial three-dimensional model with texture.

[0008] In an optional implementation, the predicting the positions of feature points of the occluded area of ​​the initial three-dimensional model based on the uniform image group and based on a deep learning algorithm, completing the three-dimensional model of the occluded area of ​​the initial three-dimensional model, and obtaining a complete three-dimensional model includes: Using a convolutional neural network to identify the occluded area of ​​the uniform image group, and obtaining the three-dimensional occluded feature point position of the occluded area by triangulation; Processing the feature point positions to generate a three-dimensional point cloud model of the occluded area; Reconstructing the three-dimensional point cloud model of the occluded area and constructing a three-dimensional surface model of the occluded area; Processing the three-dimensional surface model of the occluded area, and then obtaining the three-dimensional model of the occluded area through texture completion; The occluded area three-dimensional model and the initial three-dimensional model are combined to obtain a complete three-dimensional model.

[0009] In an optional implementation, comparing the complete three-dimensional model with the ideal three-dimensional model to obtain a static posture difference value of the complete three-dimensional model includes: Extracting model feature points of the complete three-dimensional model and the ideal three-dimensional model; According to the model feature points, a static spatial transformation relationship between the complete three-dimensional model and the ideal three-dimensional model is calculated, and a static posture difference value of the complete three-dimensional model is obtained through error calculation.

[0010] In an optional implementation, the calculating the torque generated during the flipping process according to the physical parameters and the real-time three-dimensional model, and determining the optimal flipping path in combination with the motion constraints of the flipping machine, includes: Calculating the real-time torque generated by the object during the flipping process according to the physical parameters and the real-time three-dimensional model; Combining the real-time torque and the motion constraint, a torque balance algorithm is used to perform path planning to obtain a flipping path set; Discretizing the flipping path set into a plurality of key points, wherein each of the key points corresponds to a curvature radius and a path length; According to the key point and the motion constraint, the next key point of the flipping path is determined according to a preset comprehensive trajectory index, and the optimal flipping path is obtained by continuous iteration.

[0011] In an optional implementation, the step of smoothing the optimal flipping path to generate a smooth flipping trajectory from the current position to the target position includes: Smoothing the optimal flipping path to obtain a continuous preliminary smooth trajectory; According to the preliminary smooth trajectory and motion constraints, a smooth curvature radius of each point in the trajectory is calculated; It is determined whether the smooth curvature radius is less than a preset curvature radius threshold. If so, the point corresponding to the smooth curvature radius and the trajectory around it are smoothed. If not, no additional operation is performed to obtain a smooth flip trajectory.

[0012] In an optional implementation, the motion constraint and the smooth flipping trajectory generate a control instruction to control a flipping machine to flip the object, and further include: Combining the smooth flipping trajectory with the ideal three-dimensional model to generate an ideal real-time three-dimensional model; Extract effective feature points of the real-time 3D model and the ideal real-time 3D model; calculate the real-time spatial transformation relationship between the real-time 3D model and the ideal real-time 3D model based on the effective feature points to obtain the dynamic posture difference value of the real-time 3D model; determine whether the dynamic posture difference value exceeds a preset difference threshold, if so, trigger the flip machine posture correction mechanism to generate a posture adjustment instruction, if not, do nothing; the flip machine control system receives the posture adjustment instruction to control the angle and displacement of the object's movement; the flip machine accurately controls the flipping of the object and adjusts the object until the dynamic posture difference value is less than the preset difference threshold.

[0013] In a second aspect, the present invention provides an intelligent flip control device based on machine vision, comprising: A data acquisition module is used to obtain the physical parameters of the object to be processed, a multi-view original image group, an ideal three-dimensional model, and the movement range and motion constraints of the flip machine; an image preprocessing module is used to preprocess the original image group to obtain a uniform image group; a feature recognition and reconstruction module is used to identify the key feature points of the object in the uniform image group, and perform three-dimensional reconstruction on the key feature points to generate an initial three-dimensional model of the object; an occluded area completion module is used to predict the position of the feature points of the occluded area of ​​the initial three-dimensional model based on the uniform image group and a deep learning algorithm, and complete the three-dimensional model of the occluded area of ​​the initial three-dimensional model to obtain a complete three-dimensional model; a posture difference calculation module is used to compare the complete three-dimensional model with the ideal three-dimensional model to obtain the complete three-dimensional model. Static posture difference value; an object adjustment module, used to adjust the object until the static posture difference value is less than a preset difference threshold, and send the object to a flipping machine for flipping, update the complete three-dimensional model in real time, and obtain a real-time three-dimensional model; a torque calculation and path determination module, used to calculate the torque generated during the flipping process according to the physical parameters and the real-time three-dimensional model, and determine the optimal flipping path in combination with the motion constraints of the flipping machine; a path smoothing processing module, used to smooth the optimal flipping path, and generate a smooth flipping trajectory from the current position to the target position; a control instruction generation module, used to compare the smooth flipping trajectory with the ideal three-dimensional model, generate an ideal real-time three-dimensional model, and generate a control instruction to control the flipping machine to flip the object in combination with the moving range and the motion constraints.

[0014] In a third aspect, the present invention further provides a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned intelligent flipping control methods based on machine vision. The technical solution provided by the embodiment of the present invention may include the following beneficial effects: The invention discloses a method for accurately flipping an object based on three-dimensional modeling and path planning.

[0015] This method obtains multi-angle images and physical parameters of the object, uses deep learning and 3D reconstruction technology to generate a complete 3D model, and adjusts the object's posture by comparing it with the ideal model. Through the comprehensive use of multi-angle images and physical parameters, the 3D model of the object can be reconstructed more accurately, errors can be reduced, and the realism and detail performance of the model can be improved; after comparing with the ideal model, the posture of the object can be adjusted more accurately to ensure that its position and direction in actual application are in line with expectations and operational errors can be reduced; through automated deep learning and 3D reconstruction technology, manual intervention and measurement costs are reduced, while the risks caused by inaccurate models or posture errors are reduced.

[0016] During the flipping process, the present invention updates the three-dimensional model of the object in real time, calculates the optimal flipping path based on physical parameters and flipping machine constraints, and performs smoothing to generate a smooth trajectory, thereby optimizing path planning and reducing collision risks and energy consumption during the flipping process; enhancing operation smoothness and avoiding mechanical vibration or model damage caused by sudden path changes, thereby improving the reliability and safety of the overall operation.

[0017] The present invention combines a smooth trajectory with an ideal model to generate control instructions to achieve precise and rapid flipping of objects. It can effectively solve the posture control problem in the flipping process of objects with complex shapes and improve the flipping accuracy. It is suitable for scenarios that require precise posture adjustment in industrial production. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 The present invention is a flowchart of an intelligent flipping control method based on machine vision.

[0019] Figure 2 The present invention is a schematic structural diagram of an intelligent flipping control method and device based on machine vision. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0021] like Figure 1 In this embodiment, a method and device for intelligent flipping control based on machine vision may specifically include: Step S101, obtaining the physical parameters of the object to be processed, the original image group of different viewing angles, the ideal three-dimensional model, the moving range and motion constraints of the flip machine.

[0022] It should be noted that the physical parameters of an object refer to the size, volume, and mass distribution of the object; the original image group of different perspectives is the image of the object taken by an industrial camera at different perspectives, and for example, includes images of the object from the front, back, top, and bottom perspectives; the ideal three-dimensional model refers to the model data of the object to be processed that is pre-stored in the database, including the ideal size, volume, and mass distribution data of the object; the moving range refers to the maximum rotation angle, moving distance, or operating space that the flip machine can achieve if the physical structure and design allow. For example, a flip machine may be designed to achieve a full rotation from 0 to 360 degrees, or move a certain distance in the horizontal direction; the motion constraints include speed, acceleration, torque, minimum radius of curvature, and other restrictions, which ensure that the flip machine does not exceed its design capacity during operation, thereby ensuring the safety and reliability of the equipment. For example, the maximum rotation speed of the flip machine may be limited to 180 degrees per minute, and the maximum acceleration is 30 degrees / second² to prevent damage to the object or the equipment itself during rapid movement.

[0023] Step S102, preprocessing the original image group to obtain a uniform image group.

[0024] The original image group is subjected to denoising to obtain a denoised image group. The overall brightness distribution of the denoised image group is adjusted to obtain a balanced image group. The balanced image group is divided into a plurality of regions, and an initial texture pattern of each region is determined. The initial texture pattern is completed, and edge traces between regions are eliminated to generate a complete texture pattern. The complete texture pattern is merged with the balanced image group to generate a final uniform image group.

[0025] It should be noted that the texture pattern refers to the pattern formed by the texture characteristics of the surface of an object. It reflects the structure, texture and visual effect of the surface of the object. For example, there will be the texture of annual rings on wood and the texture of veins on leaves. The texture pattern can be naturally formed or artificially designed, and it is widely present on the surfaces of various materials and objects.

[0026] It should be noted that denoising is a combination of one or several algorithms, and image denoising is a basic step to improve image quality. Exemplarily, the present invention uses a serial combination filtering algorithm of Gaussian filtering, median filtering, and bilateral filtering to remove Gaussian noise and salt and pepper noise from the original image group and perform smoothing while keeping edges clear.

[0027] It should be noted that adjusting the overall brightness distribution of the denoised image group is to improve image contrast by redistributing pixel grayscale values ​​through image equalization. Exemplarily, an adaptive equalization algorithm is used to map the pixel value range of the denoised image group to a larger range to improve the contrast of the denoised image group.

[0028] In an embodiment of the present invention, the equalized image group is divided into multiple regions based on a threshold segmentation method. By setting multiple thresholds, the equalized image group is divided into objects and backgrounds, and the objects are divided into multiple regions, and different regions have different texture patterns. For example, in the detection of surface defects of industrial products, the surface image of metal parts is divided into multiple regions, and the texture features of different types of defects such as surface scratches and pits can be extracted respectively. The segmentation process also needs to consider regional connectivity and boundary continuity. The texture synthesis algorithm generates a new texture pattern based on the existing texture pattern to complete the missing part.

[0029] In an embodiment of the present invention, the fusion of the complete texture pattern with the balanced image group is based on a pixel-level fusion method, and two pixel values ​​are directly selectively fused. Exemplarily, the complete texture pattern of each region is directly replaced into the corresponding object region of the balanced image group.

[0030] Step S103 , identifying key feature points of the object in the uniform image group, and performing three-dimensional reconstruction on the key feature points to generate an initial three-dimensional model of the object.

[0031] Identify key feature points of the object from the uniform image group and obtain the two-dimensional coordinates of the key feature points. Based on the two-dimensional coordinates, calculate the three-dimensional spatial position of the key feature points. According to the three-dimensional spatial position, construct an initial three-dimensional point cloud model of the object. Perform surface fitting on the initial three-dimensional point cloud model to generate an initial three-dimensional surface model of the object. Adjust the density and structure of the grid for the initial three-dimensional surface model to obtain an optimized three-dimensional grid model. According to the optimized three-dimensional grid model, map the texture information of the uniform image group to the model surface to generate an initial three-dimensional model with texture.

[0032] It should be noted that the two-dimensional coordinates are the coordinate positions of the key feature points in the corresponding uniform image, the three-dimensional spatial position refers to the position in a pre-set three-dimensional spatial coordinate system, and the key feature points are the positions of significant shapes on the surface of an object, which are crucial for identifying and reconstructing the shape of an object, such as the corners and grooves of electronic devices; a three-dimensional point cloud model is a data model used to represent the three-dimensional structure of an object, consisting of a large number of points, each of which has three-dimensional coordinates (x, y, z), which describes the shape and surface features of the object; a three-dimensional surface model is a mathematical model used to represent the surface shape of an object, and the surface of the object is constructed by a surface reconstruction algorithm to describe and reconstruct the object; a three-dimensional mesh model is a model used to represent the surface of an object, which consists of multiple polygons (triangles or quadrilaterals), and the polygons are connected together by vertices to form a continuous surface; the initial three-dimensional model has the shape of the object, and the surface has the texture of the object.

[0033] In an embodiment of the present invention, the key feature points of the object are identified from the uniform image group by extracting the two-dimensional coordinate information of the key feature points from the uniform image group using a SIFT (Scale Invariant Feature Transform) feature extraction algorithm. The triangulation algorithm calculates the three-dimensional spatial position of the key feature points based on known camera parameters and the two-dimensional coordinates of the key feature points in different images. The camera parameters refer to camera calibration data, including focal length, principal point, and rotation range and translation range of the camera.

[0034] It should be noted that constructing the initial 3D point cloud model of the object is to convert the 3D spatial position of the discrete key feature points into a dense point cloud. The density and uniformity of the initial 3D point cloud model will affect the accuracy of subsequent surface reconstruction. The surface reconstruction algorithm generates a continuous model surface by performing surface fitting on the point cloud data to obtain the initial 3D surface model.

[0035] In an embodiment of the present invention, the surface fitting of the initial three-dimensional point cloud model is based on the Poisson equation, and a continuous and smooth three-dimensional surface model is generated by solving the Poisson equation.

[0036] It should be noted that the optimized 3D mesh model is obtained by adjusting the mesh structure of the initial 3D surface model to simplify the data volume while maintaining the model accuracy. The texture mapping algorithm maps the 2D image texture to the 3D mesh model.

[0037] It should be noted that the texture information of the uniform image group is mapped to the model surface by using the UV mapping method. By assigning UV coordinates (two-dimensional coordinates) to each vertex of the model, the color or details in the texture image are mapped to the model surface, thereby enhancing the visual effect and realism of the model.

[0038] Step S104, predicting the positions of feature points of the occluded area of ​​the initial three-dimensional model based on the uniform image group and the deep learning algorithm, completing the three-dimensional model of the occluded area of ​​the initial three-dimensional model, and obtaining a complete three-dimensional model.

[0039] A convolutional neural network is used to identify the occluded area of ​​the uniform image group, and the position of the three-dimensional occluded feature points of the occluded area is obtained by triangulation. The feature point positions are processed to generate a three-dimensional point cloud model of the occluded area. The three-dimensional point cloud model of the occluded area is reconstructed to construct a three-dimensional surface model of the occluded area. The three-dimensional surface model of the occluded area is processed, and then the three-dimensional model of the occluded area is obtained by texture completion. The three-dimensional model of the occluded area is combined with the initial three-dimensional model to obtain a complete three-dimensional model.

[0040] It should be noted that the convolutional neural network is a deep learning model that extracts image features through multi-layer convolution and pooling operations. When identifying occluded areas of objects, the network structure includes a feature extraction layer and a classification layer.

[0041] In an embodiment of the present invention, R-CNN is used to extract features of the initial uniform image group and predict the occluded area through a classification layer, and the two-dimensional coordinates of the feature points of the occluded area are obtained, and the three-dimensional coordinates of the feature points of the occluded area are processed through a triangulation algorithm.

[0042] It should be noted that the three-dimensional surface model of the occluded area is constructed by converting the three-dimensional coordinates of the discrete feature points of the occluded area into a three-dimensional point cloud model of the occluded area. The surface reconstruction algorithm generates a continuous model surface by performing surface fitting on the three-dimensional point cloud model of the occluded area to obtain a three-dimensional surface model of the occluded area.

[0043] In an embodiment of the present invention, the reconstruction of the three-dimensional point cloud model of the occluded area is based on the Poisson equation, and a continuous and smooth three-dimensional surface model is generated by solving the Poisson equation. The grid structure of the three-dimensional surface model of the occluded area is adjusted to simplify the data volume while maintaining the model accuracy. Texture completion is based on the texture information of the uniform image group, and the nearest texture information is used to complete the surface corresponding to the three-dimensional model of the occluded area.

[0044] Step S105 : comparing the complete three-dimensional model with the ideal three-dimensional model to obtain a static posture difference value of the complete three-dimensional model.

[0045] Extract model feature points of the complete 3D model and the ideal 3D model. Calculate the static spatial transformation relationship between the complete 3D model and the ideal 3D model based on the model feature points, and obtain the static posture difference value of the complete 3D model through error calculation.

[0046] It should be noted that a model feature point refers to a geometric feature point with significant features on the surface of a three-dimensional model. In an embodiment of the present invention, the significant features are measured by the curvature value of the three-dimensional model surface. When the curvature value of a point on the three-dimensional model surface is greater than a preset curvature threshold, the point is determined to be a model feature point on the three-dimensional model surface.

[0047] It should be noted that the extraction of model feature points of the complete three-dimensional model and the ideal three-dimensional model is based on the curvature value of the three-dimensional model surface, and points with curvature values ​​greater than a preset curvature threshold are determined as model feature points.

[0048] It should be noted that both the complete three-dimensional model and the ideal three-dimensional model are established in a unified three-dimensional space coordinate system, which is the basis for comparing the two.

[0049] It should be noted that the static spatial transformation relationship describes the spatial rotation and spatial translation relationship between the complete 3D model and the ideal 3D model, and reflects the way in which the complete 3D model is transformed into the ideal 3D model. For example, in an XYZ 3D spatial coordinate system, assuming that the center coordinates of a complete 3D model are , the center coordinates of an ideal three-dimensional model The static spatial transformation relationship describes the complete three-dimensional model that needs to pass through the X-axis. , on the Y axis , on the Z axis The center point can coincide with the center point of the ideal 3D model only after the translation; assuming that the key point coordinates of the complete 3D model and the ideal 3D model after the translation are , On the other hand, the static spatial transformation relationship describes how the complete 3D model after translation is transformed. Do a three-dimensional rotation with point as the center point, so that and coincide.

[0050] It should be noted that the calculation and The distance difference and and by The spatial angle difference of three-dimensional rotation with the point as the center is the static posture difference between the complete three-dimensional model and the ideal three-dimensional model.

[0051] It should be noted that accurate model comparison and analysis can promptly detect product defects, construction errors or positioning deviations, thereby ensuring product quality and production efficiency.

[0052] Step S106, adjusting the object until the static posture difference value is less than a preset difference threshold, and sending the object to a flipping machine for flipping, updating the complete three-dimensional model in real time, and obtaining a real-time three-dimensional model.

[0053] It should be noted that the difference threshold is a key parameter, which determines the stability of the object during the flipping process. If the difference threshold is set too large, the object is likely to be damaged due to incorrect posture when entering the flipping machine, affecting efficiency; if the difference threshold is set, it will lead to frequent adjustments, affecting flipping efficiency; the real-time three-dimensional model is a three-dimensional model obtained in real time through the method described in steps S101-S104 during the flipping process of the object, which reflects the posture of the object during the flipping process.

[0054] Step S107, calculating the torque generated during the flipping process according to the physical parameters and the real-time three-dimensional model, and determining the optimal flipping path in combination with the motion constraints of the flipping machine.

[0055] According to the mass distribution in the physical parameters and the position of the real-time three-dimensional model in the flipping machine, the force analysis of the object is performed through the relationship between gravity and mass to determine the resultant force F of the object during the flipping process. Then, the flipping axis of the flipping machine is used as the rotation axis to calculate the distance d from the object to the rotation axis. The torque calculation formula is as follows: in, represents the torque, F represents the resultant force of the object during the flipping process, and d represents the distance from the object to the rotation axis.

[0056] The real-time torque generated by the object during the flipping process is calculated. The real-time torque and the motion constraint are combined to perform path planning using a torque balance algorithm to obtain a flipping path set. The flipping path set is discretized into a plurality of key points, wherein each key point corresponds to a curvature radius and a path length. According to the key points and the motion constraint, the next key point of the flipping path is determined according to the preset comprehensive trajectory index, and the optimal flipping path is obtained by continuous iteration.

[0057] In an embodiment of the present invention, the physical parameters refer to the mass distribution, geometric shape, and maximum torque that the object can withstand; the flipping path set is a set of flipping paths predicted by the torque balance algorithm, reflecting the possible flipping paths of the object. The flipping of the object along the path specified by the flipping path set must satisfy the motion constraints, and the torque borne by the object must be less than the maximum torque that it can withstand; the key point is obtained by sampling the paths in the flipping path set. For example, one point is sampled every certain distance of the path, and the radius of curvature of the point is recorded at the same time.

[0058] It should be noted that the comprehensive trajectory index is a key parameter, which determines the selection criteria of the optimal flip path. In the embodiment of the present invention, the comprehensive trajectory index formula is as follows: in, represents the path length, represents the maximum flipping path length; represents the length weight, represents the curvature radius weight, represents the radius of curvature, represents the minimum radius of curvature; represents the comprehensive trajectory index, where The larger it is, the more importance is attached to the impact of path length on the optimal flipping path. The larger it is, the more important the effect of the curvature radius on the optimal flipping path is. The smaller it is, the more likely the key point is to be selected as a point on the optimal flipping path.

[0059] Take a rectangular workpiece as an example. Its dimensions are 2 meters long, 1 meter wide, and 0.5 meters high. Its center of gravity is located at the geometric center. Its mass is 200 kg, and its maximum torque is 50 N·m. During the flipping process, the real-time 3D model posture of the workpiece is acquired by the sensor in real time. Combined with the mass and size parameters, the torque change under different postures can be calculated. Assume that the flip machine is driven by two sets of hydraulic cylinders, and its movement range is limited to 90 degrees in the vertical direction and 45 degrees in the horizontal direction. According to the principle of the torque balance algorithm, under the action of the gravity of the workpiece, the hydraulic cylinder is required to provide the corresponding balance torque, so as to plan a smooth flipping path. These paths form a path set, which contains multiple feasible flipping trajectories. Path discretization converts the continuous flipping path into a sequence of key points. For the above workpiece, the 90-degree flipping process can be divided into 9 key points, each with a 10-degree interval. Each key point corresponds to a specific curvature radius and path length. For example, the key point at the 45-degree position has a curvature radius of 1.5 meters and a corresponding path length of 0.8 meters. In the above workpiece flipping process, by evaluating the state parameters of each key point, such as angular velocity not exceeding 10 degrees per second and acceleration not exceeding 5 degrees per square second, the next key point that meets the constraints and has the best comprehensive indicators is selected. The key point set of the complete path trajectory is determined by iteration, and finally an optimal flipping path that takes into account both safety and efficiency is obtained.

[0060] Step S108, smoothing the optimal flipping path to generate a smooth flipping trajectory from the current position to the target position.

[0061] The optimal flipping path is smoothed to obtain a continuous preliminary smooth trajectory. According to the preliminary smooth trajectory and motion constraints, the smooth curvature radius of each point in the trajectory is calculated. It is determined whether the smooth curvature radius is less than a preset curvature radius threshold. If so, the point corresponding to the smooth curvature radius and the trajectory around it are smoothed. If not, no additional operation is performed to obtain a smooth flipping trajectory.

[0062] It should be noted that the preliminary smooth trajectory is obtained by completing and smoothing the discrete key points of the optimal flipping path, and is a continuous trajectory line; the smooth curvature radius of each point refers to the curvature radius calculated by sampling at a smaller interval on the preliminary smooth trajectory. For example, the curvature radius is calculated every 1 cm; the preset curvature radius threshold is a key parameter, which determines the minimum curvature radius of the smooth flipping trajectory. If the threshold is too small, the object may be damaged due to excessive force during the flipping process. The core of the path smoothing algorithm is to connect discrete key points into a smooth continuous trajectory, which is crucial to the stability and safety of the workpiece flipping process. Taking the flipping of the workpiece as an example, assuming that a flipping trajectory consists of multiple key points, there may be mutations or corners in the connection between each key point, which will cause the workpiece to jitter or be unstable during movement.

[0063] In the embodiment of the present invention, a cubic spline interpolation smoothing algorithm is used to generate a smooth curve passing through these key points to form a preliminary smooth trajectory. In another feasible embodiment, taking the flipping of a rectangular workpiece weighing 200 kilograms as an example, the path between its key points is smoothed using a Bezier curve. The trajectory after preliminary smoothing needs to be further checked to see if its curvature radius meets the requirements.

[0064] Exemplarily, the preset curvature radius threshold is set to 1.5 meters. If the curvature radius of a certain point is less than this threshold, it means that the turn is too sharp, which may cause excessive force on the workpiece or excessive load on the flipping mechanism. For points that are less than the curvature radius threshold, local smoothing needs to be re-performed. The specific method is to take this point as the center, take three points forward and backward to form a seven-point sequence, and re-apply the smoothing algorithm to this section of the trajectory. For example, in the process of flipping the workpiece from a horizontal position to a vertical position, if the curvature radius is found to be one meter at a forty-five-degree position, which is less than the threshold, it is necessary to perform secondary smoothing on this place to make the transition smoother. The smooth flipping trajectory obtained after repeated optimization not only ensures the continuity of the trajectory, but also ensures that the curvature radius of each point is within a safe range. This smoothing process can effectively reduce the impact force during the flipping of the workpiece, reduce equipment wear, and extend the service life of the equipment.

[0065] Step S109, combining the smooth flipping trajectory with the ideal three-dimensional model to generate an ideal real-time three-dimensional model, combining the moving range and the motion constraint to generate a control instruction to control the flipping machine to flip the object.

[0066] Detect the dynamic posture difference value between the real-time three-dimensional model and the ideal real-time three-dimensional model, and judge whether the dynamic posture difference value exceeds the preset difference threshold. If so, trigger the posture correction mechanism of the flip machine, and generate a control instruction to control the flip machine to correct the posture of the object. Extract the effective feature points of the real-time three-dimensional model and the ideal real-time three-dimensional model. According to the effective feature points, calculate the real-time spatial transformation relationship between the real-time three-dimensional model and the ideal real-time three-dimensional model to obtain the dynamic posture difference value of the real-time three-dimensional model. Judge whether the dynamic posture difference value exceeds the preset difference threshold. If so, trigger the posture correction mechanism of the flip machine and generate a posture adjustment instruction. If not, do nothing. The flip machine control system receives the posture adjustment instruction and calculates the angle and displacement that the object needs to adjust. The flip machine accurately controls the flipping of the object and adjusts the object until the dynamic posture difference value is less than the preset difference threshold.

[0067] It should be noted that a valid feature point refers to a geometric feature point with significant features on the surface of a three-dimensional model. In an embodiment of the present invention, the significant features are measured by the curvature value of the three-dimensional model surface. When the curvature value of a point on the three-dimensional model surface is greater than a preset curvature threshold, the point is determined to be a valid feature point on the three-dimensional model surface.

[0068] It should be noted that the extraction of effective feature points of the two is based on the curvature value of the three-dimensional model surface, and points with curvature values ​​greater than a preset curvature threshold are determined as effective feature points.

[0069] It should be noted that the real-time spatial transformation relationship describes the spatial rotation relationship and spatial translation relationship between the real-time 3D model and the ideal real-time 3D model. For example, in an XYZ 3D space coordinate system, assuming that the center coordinates of a real-time 3D model are , an ideal real-time 3D model center coordinate The dynamic spatial transformation relationship describes the real-time 3D model needs to go through the X-axis , on the Y axis , on the Z axis The center point can coincide with the center point of the ideal 3D model only after translation; assuming that the key point coordinates of the real-time 3D model and the ideal real-time 3D model after translation are , On the other hand, the dynamic space transformation relationship describes how the complete 3D model after translation is moved along The point is rotated in three dimensions so that and coincide.

[0070] It should be noted that and The distance difference and and by The spatial angle difference of the rotation center point is the dynamic posture difference value between the real-time three-dimensional model and the ideal real-time three-dimensional model. The difference threshold is a key parameter, which determines the sensitivity of the flip machine to the deviation of the object. If it is set too high, it is easy to cause the object to deviate from the predetermined trajectory during the flipping process. If it is set too low, it is easy to cause the flip machine to frequently adjust the object, affecting efficiency. For example, the difference threshold is set to a translation distance of no more than 0.1m and a spatial rotation angle of no more than 5°. When the distance difference exceeds 0.1m or the spatial angle difference exceeds 5°, the flip machine posture correction mechanism is triggered to generate a posture adjustment instruction.

[0071] It should be noted that the precise posture adjustment adopted by the flipping mechanism is based on closed-loop control. When it is detected that the object needs to be adjusted, the target angle and direction of the adjustment are first calculated. For example, when it is necessary to adjust forty-five degrees, the system will dynamically adjust the motor output torque according to the deviation between the current angle and the target angle. In the initial stage, a large force is used for rapid adjustment, and the adjustment force is reduced when approaching the target position to avoid overshoot. Fine-tuning is performed through real-time feedback until the object posture returns to the allowable error range. During the entire adjustment process, the system continues to monitor and feedback in real time. Taking the adjustment of industrial parts as an example, when it is detected that the part is gradually adjusted from the initial thirty degrees to twenty degrees, ten degrees, and finally reaches the target vertical state, the system will calculate the remaining deviation value in real time. If the deviation after adjustment still exceeds the preset threshold, a new round of fine-tuning will be initiated until the deviation is reduced to an acceptable range. This closed-loop feedback mechanism ensures the accuracy and reliability of posture adjustment.

[0072] like Figure 2As shown, the present invention provides an intelligent flip control device based on machine vision, which mainly includes: a data acquisition module, which is used to acquire the physical parameters of the object to be processed, a multi-perspective original image group, an ideal three-dimensional model, a moving range and motion constraints of the flip machine; an image preprocessing module, which is used to preprocess the original image group to obtain a uniform image group; a feature recognition and reconstruction module, which is used to identify the key feature points of the object in the uniform image group, and perform three-dimensional reconstruction on the key feature points to generate an initial three-dimensional model of the object; an occluded area completion module, which is used to predict the position of the feature points of the occluded area of ​​the initial three-dimensional model based on the uniform image group and a deep learning algorithm, and complete the three-dimensional model of the occluded area of ​​the initial three-dimensional model to obtain a complete three-dimensional model; a posture difference calculation module, which is used to compare the complete three-dimensional model with the ideal three-dimensional model Compare and obtain the static posture difference value of the complete three-dimensional model; an object adjustment module, used to adjust the object until the static posture difference value is less than the preset difference threshold, and send the object to the flip machine for flipping, update the complete three-dimensional model in real time, and obtain a real-time three-dimensional model; a torque calculation and path determination module, used to calculate the torque generated during the flipping process according to the physical parameters and the real-time three-dimensional model, and determine the optimal flipping path in combination with the motion constraints of the flip machine; a path smoothing module, used to smooth the optimal flipping path and generate a smooth flipping trajectory from the current position to the target position; a control instruction generation module, used to compare the smooth flipping trajectory with the ideal three-dimensional model, generate an ideal real-time three-dimensional model, and generate a control instruction to control the flip machine to flip the object in combination with the moving range and the motion constraints. Although the present invention has been described in detail above with general descriptions and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to it on the basis of the present invention. Therefore, these modifications or improvements made on the basis of not departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

[0073] The embodiment of the present invention further provides an electronic device. The electronic device includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, such as a machine vision-based intelligent flip control program. When the processor executes the computer program, the steps in the above-mentioned embodiments of the machine vision-based intelligent flip control method are implemented, such as Figure 1 Alternatively, when the processor executes the computer program, the functions of the modules in the above-mentioned device embodiments are implemented, such as the torque calculation and path determination modules.

[0074] Exemplarily, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the electronic device.

[0075] The electronic device may be a computing device such as a desktop computer, a notebook, a PDA, and a smart tablet. The electronic device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the above components are merely examples of electronic devices and do not constitute a limitation on the electronic device. The electronic device may include more or fewer components than the above components, or may combine certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0076] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device.

[0077] The memory can be used to store the computer programs and modules, and the processor realizes various functions of the electronic device by running or executing the computer programs and modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0078] Wherein, if the module integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0079] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0080] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. An intelligent flipping control method based on machine vision, characterized in that: The method comprises: Obtain the physical parameters of the object to be processed, a group of original images from multiple perspectives, an ideal three-dimensional model, a moving range and motion constraints of a flipping machine; pre-process the group of original images to obtain a uniform image group; identify the key feature points of the object in the uniform image group, and perform three-dimensional reconstruction on the key feature points to generate an initial three-dimensional model of the object; predict the positions of feature points of the occluded area of ​​the initial three-dimensional model based on the uniform image group based on a deep learning algorithm, complete the occluded area of ​​the initial three-dimensional model, and obtain a complete three-dimensional model; compare the complete three-dimensional model with the ideal three-dimensional model to obtain a static posture difference value between the two models; adjust the object until the static posture difference value is less than a preset difference threshold, and send the object into a flipping machine for flipping, update the complete three-dimensional model in real time, and obtain a real-time three-dimensional model; calculate the torque generated during the flipping process based on the physical parameters and the real-time three-dimensional model, and determine the optimal flipping path in combination with the motion constraints of the flipping machine; The optimal flipping path is smoothed to generate a smooth flipping trajectory from the current position to the target position; and a control instruction is generated to control the flipping machine to flip the object by combining the moving range, the motion constraint and the smooth flipping trajectory.

2. The method according to claim 1, characterized in that The preprocessing of the original image group to obtain a uniform image group includes: Performing denoising processing on the original image group to obtain a denoised image group; Adjusting the overall brightness distribution of the denoised image group to obtain a balanced image group; Dividing the equalized image group into a plurality of regions, and determining an initial texture pattern of each region; Completing the initial texture pattern and eliminating edge traces between regions to generate a complete texture pattern; The complete texture pattern is merged with the balanced image group to generate the uniform image group.

3. The method according to claim 1, characterized in that The step of identifying key feature points of the object in the uniform image group and performing three-dimensional reconstruction on the key feature points to generate an initial three-dimensional model of the object includes: Identify key feature points of an object from the uniform image group, and obtain two-dimensional coordinates of the key feature points; Based on the two-dimensional coordinates, calculating the three-dimensional spatial position of the key feature point; Constructing an initial three-dimensional point cloud model of the object according to the three-dimensional spatial position; Performing surface fitting on the initial three-dimensional point cloud model to generate an initial three-dimensional surface model of the object.

4. The method according to claim 1, characterized in that: The method of predicting the positions of feature points of the blocked area of ​​the initial three-dimensional model based on the uniform image group and completing the three-dimensional model of the blocked area of ​​the initial three-dimensional model to obtain a complete three-dimensional model includes: Using a convolutional neural network to identify the occluded area of ​​the uniform image group, and obtaining the three-dimensional occluded feature point position of the occluded area by triangulation; Processing the feature point positions to generate a three-dimensional point cloud model of the occluded area; Reconstructing the three-dimensional point cloud model of the occluded area and constructing a three-dimensional surface model of the occluded area; Processing the three-dimensional surface model of the occluded area, and then obtaining the three-dimensional model of the occluded area through texture completion; The occluded area three-dimensional model and the initial three-dimensional model are combined to obtain the complete three-dimensional model.

5. The method according to claim 1, characterized in that The step of comparing the complete three-dimensional model with the ideal three-dimensional model to obtain a static posture difference value of the complete three-dimensional model includes: Extracting model feature points of the complete three-dimensional model and the ideal three-dimensional model; According to the model feature points, a static spatial transformation relationship between the complete three-dimensional model and the ideal three-dimensional model is calculated, and a static posture difference value of the complete three-dimensional model is obtained through error calculation.

6. The method according to claim 1, characterized in that The calculating the torque generated during the flipping process according to the physical parameters and the real-time three-dimensional model, and determining the optimal flipping path in combination with the motion constraints of the flipping machine, includes: Calculating the real-time torque generated by the object during the flipping process according to the physical parameters and the real-time three-dimensional model; Combining the real-time torque and the motion constraint, a torque balance algorithm is used to perform path planning to obtain a flipping path set; Discretizing the flipping path set into a plurality of key points, wherein each of the key points corresponds to a curvature radius and a path length; According to the key point and the motion constraint, the next key point of the flipping path is determined according to a preset comprehensive trajectory index, and the optimal flipping path is obtained by continuous iteration.

7. The method according to claim 1, characterized in that The step of smoothing the optimal flipping path to generate a smooth flipping trajectory from the current position to the target position includes: Smoothing the optimal flipping path to obtain a continuous preliminary smooth trajectory; According to the preliminary smooth trajectory and the motion constraint, a smooth curvature radius of each point in the trajectory is calculated; It is determined whether the smooth curvature radius is less than a preset curvature radius threshold. If so, the point corresponding to the smooth curvature radius and the trajectory around it are smoothed. If not, no additional operation is performed to obtain a smooth flip trajectory.

8. The method according to claim 1, characterized in that Combining the moving range, the motion constraint and the smooth flipping trajectory, generating a control instruction to control the flipping machine to flip the object, including: Combining the smooth flipping trajectory with the ideal three-dimensional model to generate an ideal real-time three-dimensional model; Extract effective feature points of the real-time 3D model and the ideal real-time 3D model; calculate the real-time spatial transformation relationship between the real-time 3D model and the ideal real-time 3D model based on the effective feature points to obtain the dynamic posture difference value of the real-time 3D model; determine whether the dynamic posture difference value exceeds a preset difference threshold, if so, trigger the flip machine posture correction mechanism to generate a posture adjustment instruction, if not, do nothing; the flip machine control system receives the posture adjustment instruction to control the angle and displacement of the object's movement; the flip machine accurately controls the flipping of the object and adjusts the object until the dynamic posture difference value is less than the preset difference threshold.

9. An intelligent flip control device based on machine vision, characterized in that: The device comprises: a data acquisition module, which is used to acquire physical parameters of the object to be processed, a multi-view original image group, an ideal three-dimensional model, a moving range and motion constraints of a flip machine; an image preprocessing module, which is used to preprocess the original image group to obtain a uniform image group; a feature recognition and reconstruction module, which is used to identify key feature points of the object in the uniform image group, and perform three-dimensional reconstruction on the key feature points to generate an initial three-dimensional model of the object; an occluded area completion module, which is used to predict the position of feature points of the occluded area of ​​the initial three-dimensional model based on the uniform image group and a deep learning algorithm, and complete the three-dimensional model of the occluded area of ​​the initial three-dimensional model to obtain a complete three-dimensional model; a posture difference calculation module, which is used to compare the complete three-dimensional model with the ideal three-dimensional model to obtain a static posture difference value of the complete three-dimensional model; An object adjustment module is used to adjust the object until the static posture difference value is less than a preset difference threshold, and send the object to a flipping machine for flipping, and update the complete three-dimensional model in real time to obtain a real-time three-dimensional model; a torque calculation and path determination module is used to calculate the torque generated during the flipping process according to the physical parameters and the real-time three-dimensional model, and determine the optimal flipping path in combination with the motion constraints of the flipping machine; a path smoothing module is used to smooth the optimal flipping path and generate a smooth flipping trajectory from the current position to the target position; a control instruction generation module is used to combine the smooth flipping trajectory with the ideal three-dimensional model to generate an ideal real-time three-dimensional model, and generate a control instruction in combination with the moving range and the motion constraints to control the flipping machine to flip the object.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the intelligent flipping control method based on machine vision as described in any one of claims 1 to 8.

Images