An intelligent flipping control method and device based on machine vision

Abstract

The present invention relates to the field of automatic control technology, and discloses an intelligent flipping control method and device based on machine vision. The method includes: obtaining the physical parameters of an object, the original image group of multiple perspectives and the ideal three-dimensional model, as well as the motion constraints and moving range of the flipping machine; first, preprocessing the images to generate a uniform image group, identifying key feature points for three-dimensional reconstruction to obtain an initial three-dimensional model. Using a deep learning algorithm to identify and complete the occluded areas to generate a complete three-dimensional model. Comparing the complete model with the ideal model, adjusting the object's posture until the difference value is less than the threshold and then sending it into the flipping machine. Updating the three-dimensional model in real time, calculating the flipping torque in combination with the object's physical parameters and motion constraints, determining the optimal path and generating a smooth trajectory; finally, generating a control instruction in combination with the moving range. The present invention makes the object flip more securely by calibrating the object's posture, and realizes precise flipping of the object.

Description

Technical Field

[0001] The present invention relates to the field of automatic control technology, and in particular to an intelligent flipping control method and device based on machine vision. Background Art

[0002] Object flipping is a key link in scenarios such as industrial production and logistics. In industrial production, many process flows require processing or assembly of objects at different angles, such as welding and painting of parts in automobile manufacturing, and double-sided patching of circuit boards in electronic equipment production. Accurate and fast flipping can improve production efficiency and ensure product quality. Traditional flipping methods rely on manual operation of clamps to flip the product over, and then turn it back and forth after flipping, or use simple mechanical devices to flip the product over. The flipping action for complex workpieces is cumbersome, and there are problems such as insufficient precision and high labor intensity. With the development of automation technology, a variety of automatic flipping equipment and technologies have emerged, such as flipping mechanisms based on PLC control, flipping devices in automated production lines, and flipping platforms with angle adjustment functions. These technologies have improved the automation and flipping accuracy of flipping operations to a certain extent.

[0003] However, the existing technology still has some shortcomings. The existing automatic flipping system does not check and calibrate the posture of the object before it enters the flipping machine. During the object flipping process, the object posture is not detected in real time, resulting in object shaking or inaccurate positioning, causing damage to the object and causing unnecessary machine restarts and inspections, affecting the accuracy of the flipping. Summary of the invention

[0004] The present invention provides an intelligent flipping control method and device based on machine vision, which solves the shaking problem of the object by calibrating the posture of the object before entering a flipping machine and during the flipping process, thereby improving the accuracy and efficiency of the flipping.

[0005] In a first aspect, the present invention provides an intelligent flipping control method based on machine vision, which mainly includes:

[0006] Obtain the physical parameters of the object to be processed, the original image group from multiple perspectives, the ideal three-dimensional model, the moving range and motion constraints of the flipping machine; preprocess the original image group to obtain a uniform image group; identify the key feature points of the object in the uniform image group, and perform three-dimensional reconstruction on the key feature points to generate an initial three-dimensional model of the object; based on the uniform image group, predict the positions of the feature points in the occluded area of the initial three-dimensional model using a deep learning algorithm, and complete the three-dimensional model of the occluded area of the initial three-dimensional model to obtain a complete three-dimensional model; compare the complete three-dimensional model with the ideal three-dimensional model to obtain the static attitude difference value of the complete three-dimensional model; adjust the object until the static attitude difference value is less than the preset difference threshold, and send the object into the flipping machine for flipping, and update the complete three-dimensional model in real time to obtain a real-time three-dimensional model; calculate the torque generated during the flipping process according to the physical parameters and the real-time three-dimensional model, and combine the motion constraints of the flipping machine to determine the optimal flipping path; smooth the optimal flipping path to generate a smooth flipping trajectory from the current position to the target position; combine the moving range, the motion constraints and the smooth flipping trajectory to generate a control instruction to control the flipping machine to flip the object.

[0007] In an alternative embodiment, the preprocessing the original image group to obtain a uniform image group includes:

[0008] Perform denoising processing on the original image group to obtain a denoised image group;

[0009] Adjust the overall brightness distribution of the denoised image group to obtain a balanced image group;

[0010] Divide the balanced image group into multiple regions, and determine the initial texture pattern of each region;

[0011] Complete the initial texture pattern and eliminate the edge traces between regions to generate a complete texture pattern;

[0012] Fuse the complete texture pattern with the balanced image group to generate a final uniform image group.

[0013] In an alternative embodiment, the identifying the key feature points of the object in the uniform image group and performing three-dimensional reconstruction on the key feature points to generate an initial three-dimensional model of the object includes:

[0014] Identify the key feature points of the object from the uniform image group and obtain the two-dimensional coordinates of the key feature points;

[0015] Based on the two-dimensional coordinates, calculate the three-dimensional spatial positions of the key feature points;

[0016] Construct an initial three-dimensional point cloud model of the object according to the three-dimensional spatial position;

[0017] Perform surface fitting on the initial three-dimensional point cloud model to generate an initial three-dimensional surface model of the object;

[0018] For the initial three-dimensional surface model, adjust the density and structure of the mesh to obtain an optimized three-dimensional mesh model;

[0019] According to the optimized three-dimensional mesh model, map the texture information of the uniform image group to the model surface to generate an initial three-dimensional model with texture.

[0020] In an optional implementation manner, the method of predicting the position of the feature points in the occluded area of the initial three-dimensional model based on the deep learning algorithm according to the uniform image group, and completing the three-dimensional model of the occluded area of the initial three-dimensional model to obtain a complete three-dimensional model includes:

[0021] Use a convolutional neural network to identify the occluded area of the uniform image group, and obtain the three-dimensional occlusion feature point positions of the occluded area through triangulation;

[0022] Process the feature point positions to generate a three-dimensional point cloud model of the occluded area;

[0023] Reconstruct the three-dimensional point cloud model of the occluded area to construct a three-dimensional surface model of the occluded area;

[0024] Process the three-dimensional surface model of the occluded area, and then obtain a three-dimensional model of the occluded area through texture completion;

[0025] Combine the three-dimensional model of the occluded area and the initial three-dimensional model to obtain a complete three-dimensional model.

[0026] In an optional implementation manner, the method of comparing the complete three-dimensional model with the ideal three-dimensional model to obtain the static pose difference value of the complete three-dimensional model includes:

[0027] Extract the model feature points of the complete three-dimensional model and the ideal three-dimensional model;

[0028] According to the model feature points, calculate the static spatial transformation relationship between the complete three-dimensional model and the ideal three-dimensional model, and obtain the static pose difference value of the complete three-dimensional model through error calculation.

[0029] In an optional implementation manner, the method of calculating the torque generated during the flipping process according to the physical parameters and the real-time three-dimensional model, and determining the optimal flipping path in combination with the motion constraints of the flipper includes:

[0030] Calculate the real-time torque generated by the object during the flipping process according to the physical parameters and the real-time three-dimensional model;

[0031] Combine the real-time torque and the motion constraints, and use the torque balance algorithm for path planning to obtain a set of flipping paths;

[0032] Discretize the set of flipping paths into multiple key points, where each key point corresponds to a curvature radius and a path length;

[0033] According to the key points and the motion constraints, determine the next key point of the flipping path according to a preset comprehensive trajectory index, and continuously iterate to obtain the optimal flipping path.

[0034] In an alternative embodiment, the smoothing the optimal flipping path to generate a smooth flipping trajectory from the current position to the target position includes:

[0035] Smooth the optimal flipping path to obtain a continuous preliminary smooth trajectory;

[0036] According to the preliminary smooth trajectory and the motion constraints, calculate the smooth curvature radius of each point in the trajectory;

[0037] Judge whether the smooth curvature radius is less than a preset curvature radius threshold. If so, smooth the point corresponding to the smooth curvature radius and the surrounding trajectory. If not, do not perform any additional operations to obtain a smooth flipping trajectory.

[0038] In an alternative embodiment, the motion constraints and the smooth flipping trajectory to generate a control command to control the flipping machine to flip the object further includes:

[0039] Combine the smooth flipping trajectory with the ideal three-dimensional model to generate an ideal real-time three-dimensional model;

[0040] Extract the effective feature points of the real-time three-dimensional model and the ideal real-time three-dimensional model; according to the effective feature points, calculate the real-time spatial transformation relationship between the real-time three-dimensional model and the ideal real-time three-dimensional model to obtain the dynamic pose difference value of the real-time three-dimensional model; judge whether the dynamic pose difference value exceeds a preset difference threshold. If so, trigger the flipping machine pose correction mechanism to generate a pose adjustment command. If not, do not perform any processing; the flipping machine control system receives the pose adjustment command and controls the angle and displacement of the object's movement; the flipping machine precisely controls the flipping of the object to adjust the object so that the dynamic pose difference value is less than the preset difference threshold.

[0041] In a second aspect, the present invention provides an intelligent flipping control device based on machine vision, including:

[0042] A data acquisition module, configured to acquire physical parameters of an object to be processed, an original image group from multiple perspectives, an ideal three-dimensional model, the moving range and motion constraints of a turning machine; an image preprocessing module, configured to preprocess the original image group to obtain a uniform image group; a feature recognition and reconstruction module, configured to recognize key feature points of the object in the uniform image group and perform three-dimensional reconstruction on the key feature points to generate an initial three-dimensional model of the object; an occluded area completion module, configured to predict the positions of feature points in the occluded area of the initial three-dimensional model based on a deep learning algorithm according to the uniform image group, and complete the three-dimensional model of the occluded area of the initial three-dimensional model to obtain a complete three-dimensional model; a posture difference calculation module, configured to compare the complete three-dimensional model with the ideal three-dimensional model to obtain a static posture difference value of the complete three-dimensional model; an object adjustment module, configured to adjust the object until the static posture difference value is less than a preset difference threshold, and send the object into the turning machine for turning, and update the complete three-dimensional model in real time to obtain a real-time three-dimensional model; a torque calculation and path determination module, configured to calculate the torque generated during the turning process according to the physical parameters and the real-time three-dimensional model, and determine an optimal turning path in combination with the motion constraints of the turning machine; a path smoothing processing module, configured to perform smoothing processing on the optimal turning path to generate a smooth turning trajectory from the current position to the target position; a control instruction generation module, configured to generate an ideal real-time three-dimensional model by combining the smooth turning trajectory with the ideal three-dimensional model, and generate a control instruction to control the turning machine to turn the object in combination with the moving range and the motion constraints.

[0043] In a third aspect, the present invention further provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program runs, it controls the device where the computer-readable storage medium is located to execute any one of the above-mentioned intelligent turning control methods based on machine vision. The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0044] The present invention discloses an object precise turning method based on three-dimensional modeling and path planning.

[0045] This method generates a complete 3D model by obtaining multi - angle images and physical parameters of an object, using deep learning and 3D reconstruction technologies, and compares it with an ideal model to adjust the object's pose. Through the comprehensive utilization of multi - angle images and physical parameters, it can more accurately reconstruct the 3D model of the object, reduce errors, and enhance the realism and detail performance of the model. After comparing with the ideal model, it can more precisely adjust the object's pose to ensure that its position and orientation in actual applications meet expectations and reduce operation errors. Through automated deep learning and 3D reconstruction technologies, it reduces manual intervention and measurement costs, while reducing the risks brought by inaccurate models or incorrect poses.

[0046] During the flipping process, the present invention updates the 3D model of the object in real - time, calculates the optimal flipping path by combining physical parameters and the constraints of the flipping machine, and performs smoothing processing to generate a smooth trajectory, optimizing the path planning, reducing the collision risk and energy consumption during the flipping process; enhancing the operation smoothness, avoiding mechanical vibrations or model damage caused by sudden path changes, thereby improving the reliability and safety of the overall operation.

[0047] The present invention combines the smooth trajectory with the ideal model to generate control instructions to achieve precise and rapid flipping of the object, which can effectively solve the problem of attitude control during the flipping process of complex - shaped objects, improve the flipping accuracy, and is applicable to scenarios that require precise attitude adjustment in industrial production. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flowchart of an intelligent flipping control method based on machine vision of the present invention.

[0049] Figure 2 It is a schematic structural diagram of an intelligent flipping control method and device based on machine vision of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0050] In order to enable those skilled in the art to better understand the technical solutions in this specification, the following will clearly and completely describe the technical solutions in the embodiments of this specification in conjunction with the accompanying drawings in the embodiments of this specification. Obviously, the described embodiments are only a part of the embodiments of this specification, rather than all the embodiments. Based on the embodiments in this specification, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this specification.

[0051] Such as Figure 1 , a specific intelligent flipping control method and device based on machine vision in this embodiment may specifically include:

[0052] Step S101, obtain the physical parameters of the object to be processed, the original image group from different perspectives, obtain the ideal 3D model, the moving range and motion constraints of the flipping machine.

[0053] It should be noted that the physical parameters of an object refer to the size, volume, and mass distribution of the object; the original image group from different perspectives is the object images captured by industrial cameras from different perspectives. Exemplarily, it includes pictures of the object from the front, back, top, and bottom perspectives; the ideal three-dimensional model refers to the model data of the object pre-stored in the database to be processed, including the ideal size, volume, and mass distribution data of the object; the movement range refers to the maximum rotation angle, movement distance, or operating space that the flipping machine can achieve under the allowable physical structure and design. For example, a flipping machine may be designed to be able to achieve a full rotation of 0 to 360 degrees, or move a certain distance in the horizontal direction; the motion constraints include limiting conditions such as speed, acceleration, torque, and minimum curvature radius, which ensure that the flipping machine does not exceed its design capabilities during operation, thereby ensuring the safety and reliability of the equipment. For example, the maximum rotation speed of the flipping machine may be limited to 180 degrees per minute, and the maximum acceleration is 30 degrees / second² to prevent damage to the object or the equipment itself during rapid movement.

[0054] Step S102: Preprocess the original image group to obtain a uniform image group.

[0055] Perform denoising processing on the original image group to obtain a denoised image group. Adjust the overall brightness distribution of the denoised image group to obtain an equalized image group. Divide the equalized image group into multiple regions and determine the initial texture pattern of each region. Complete the initial texture pattern and eliminate the edge traces between regions to generate a complete texture pattern. Fuse the complete texture pattern with the equalized image group to generate the final uniform image group.

[0056] It should be noted that the texture pattern refers to the pattern formed by the texture features on the surface of an object, which reflects the structure, texture, and visual effect of the object surface. For example, there are texture patterns of annual rings on wood and vein textures on leaves. The texture pattern can be naturally formed or artificially designed and widely exists on the surfaces of various materials and objects.

[0057] It should be noted that the denoising processing is a combination of one or several algorithms, and image denoising processing is a basic step to improve image quality. Exemplarily, the present invention uses a serial combined filtering algorithm of Gaussian filtering, median filtering, and bilateral filtering to remove Gaussian noise and salt-and-pepper noise from the original image group and perform smoothing processing while keeping the edges clear.

[0058] It should be noted that adjusting the overall brightness distribution of the denoised image group is to improve the image contrast by redistributing pixel gray values through image equalization. Exemplarily, an adaptive equalization algorithm is adopted to map the pixel value range of the denoised image group to a larger range, enhancing the contrast of the denoised image group.

[0059] In the embodiment of the present invention, dividing the equalized image group into multiple regions is realized based on the method of threshold segmentation. By setting multiple thresholds, the equalized image group is divided into objects and the background, and the objects are divided into multiple regions, with different texture patterns in different regions. For example, in the detection of surface defects of industrial products, the surface image of a metal part is divided into multiple regions, and the texture features of different types of defects such as surface scratches and pits can be extracted respectively. The segmentation process also needs to consider regional connectivity and boundary continuity. The texture synthesis algorithm generates new texture patterns based on the existing texture patterns to complete the missing parts.

[0060] In the embodiment of the present invention, fusing the complete texture pattern with the equalized image group is based on the pixel-level fusion method, directly performing selective fusion on the two pixel values. Exemplarily, the complete texture pattern of each region is directly replaced into the corresponding object region of the equalized image group.

[0061] Step S103, identify the key feature points of the object in the uniform image group, and perform three-dimensional reconstruction on the key feature points to generate an initial three-dimensional model of the object.

[0062] Identify the key feature points of the object from the uniform image group, and obtain the two-dimensional coordinates of the key feature points. Based on the two-dimensional coordinates, calculate the three-dimensional spatial positions of the key feature points. According to the three-dimensional spatial positions, construct an initial three-dimensional point cloud model of the object. Perform surface fitting on the initial three-dimensional point cloud model to generate an initial three-dimensional surface model of the object. For the initial three-dimensional surface model, adjust the density and structure of the mesh to obtain an optimized three-dimensional mesh model. According to the optimized three-dimensional mesh model, map the texture information of the uniform image group to the model surface to generate an initial three-dimensional model with texture.

[0063] It should be noted that the two-dimensional coordinates are the coordinate positions of the key feature points in the corresponding uniform image, the three-dimensional spatial position refers to the position in a pre-set three-dimensional spatial coordinate system, and the key feature points are the positions of the prominent shapes on the surface of the object, which are crucial for identifying and reconstructing the shape of the object, such as the corners and grooves of an electronic device; the three-dimensional point cloud model is a data model used to represent the three-dimensional structure of an object, consisting of a large number of points, each with three-dimensional coordinates (x, y, z), describing the shape and surface features of the object; the three-dimensional surface model is a mathematical model used to represent the surface shape of an object, constructing the surface of the object through a surface reconstruction algorithm to describe and reconstruct the object; the three-dimensional mesh model is a model used to represent the surface of an object, which consists of multiple polygons (triangles or quadrilaterals), and the polygons are connected together through vertices to form a continuous surface; the initial three-dimensional model has the shape of the object, and the surface has the texture of the object.

[0064] In the embodiment of the present invention, identifying the key feature points of the object from the uniform image group is to extract the two-dimensional coordinate information of the key feature points from the uniform image group using the SIFT (Scale-Invariant Feature Transform) feature extraction algorithm. The triangulation algorithm calculates the three-dimensional spatial position of the key feature points based on the known camera parameters and the two-dimensional coordinates of the key feature points in different images. The camera parameters refer to the camera calibration data, including the focal length, the principal point, and the rotation range and translation range of the camera.

[0065] It should be noted that constructing the initial three-dimensional point cloud model of the object is to convert the three-dimensional spatial positions of the discrete key feature points into a dense point cloud. The density and uniformity of the initial three-dimensional point cloud model will affect the accuracy of subsequent surface reconstruction. The surface reconstruction algorithm generates a continuous model surface curve by performing surface fitting on the point cloud data to obtain the initial three-dimensional surface model.

[0066] In the embodiment of the present invention, performing surface fitting on the initial three-dimensional point cloud model is based on the Poisson equation, and a continuous and smooth three-dimensional surface model is generated by solving the Poisson equation.

[0067] It should be noted that the optimized three-dimensional mesh model is obtained by adjusting the mesh structure of the initial three-dimensional surface model, simplifying the data volume while maintaining the model accuracy. The texture mapping algorithm maps the two-dimensional image texture onto the three-dimensional mesh model.

[0068] It should be noted that the UV mapping method is used to map the texture information of the uniform image group onto the model surface. By assigning UV coordinates (two-dimensional coordinates) to each vertex of the model, the colors or details in the texture image are mapped onto the model surface, thereby enhancing the visual effect and realism of the model.

[0069] Step S104: Based on the uniform image group, predict the positions of the feature points in the occluded area of the initial three-dimensional model according to the deep learning algorithm, and complete the three-dimensional model of the occluded area of the initial three-dimensional model to obtain a complete three-dimensional model.

[0070] Use a convolutional neural network to identify the occluded area of the uniform image group, and obtain the three-dimensional occlusion feature point positions of the occluded area through triangulation. Process the feature point positions to generate a three-dimensional point cloud model of the occluded area. Reconstruct the three-dimensional point cloud model of the occluded area to construct a three-dimensional surface model of the occluded area. Process the three-dimensional surface model of the occluded area, and then complete the texture to obtain a three-dimensional model of the occluded area. Combine the three-dimensional model of the occluded area and the initial three-dimensional model to obtain a complete three-dimensional model.

[0071] It should be noted that a convolutional neural network is a deep learning model that extracts image features through multiple convolutional and pooling operations. When identifying the occluded area of an object, the network structure includes a feature extraction layer and a classification layer.

[0072] In the embodiment of the present invention, R-CNN is used to extract the features of the initial uniform image group and predict the occluded area through the classification layer, and obtain the two-dimensional coordinates of the feature points in the occluded area. Process the three-dimensional coordinates of the feature points in the occluded area through the triangulation algorithm.

[0073] It should be noted that constructing a three-dimensional surface model of the occluded area is to convert the three-dimensional coordinates of the discrete feature points in the occluded area into a three-dimensional point cloud model of the occluded area. The surface reconstruction algorithm generates a continuous model surface by performing surface fitting on the three-dimensional point cloud model of the occluded area to obtain a three-dimensional surface model of the occluded area.

[0074] In the embodiment of the present invention, reconstructing the three-dimensional point cloud model of the occluded area is based on the Poisson equation, and a continuous and smooth three-dimensional surface model is generated by solving the Poisson equation. Adjust the mesh structure of the three-dimensional surface model of the occluded area to simplify the data volume while maintaining the model accuracy. Texture completion is based on the texture information of the uniform image group, and the nearest texture information is used to complete the corresponding surface of the three-dimensional model of the occluded area.

[0075] Step S105: Compare the complete three-dimensional model with the ideal three-dimensional model to obtain the static pose difference value of the complete three-dimensional model.

[0076] Extract the model feature points of the complete three-dimensional model and the ideal three-dimensional model. According to the model feature points, calculate the static spatial transformation relationship between the complete three-dimensional model and the ideal three-dimensional model, and obtain the static pose difference value of the complete three-dimensional model through error calculation.

[0077] It should be noted that the model feature points refer to the geometric feature points with significant features on the surface of the three-dimensional model. In the embodiments of the present invention, the significant features are measured by the curvature values on the surface of the three-dimensional model. When the curvature value of a certain point on the surface of the three-dimensional model is greater than the preset curvature threshold, it is determined that this point is the model feature point on the surface of the three-dimensional model.

[0078] It should be noted that the extraction of the model feature points of the complete three-dimensional model and the ideal three-dimensional model is based on the curvature values on the surface of the three-dimensional model, and the points with curvature values greater than the preset curvature threshold are determined as model feature points.

[0079] It should be noted that both the complete three-dimensional model and the ideal three-dimensional model are established in a unified three-dimensional space coordinate system, which is the basis for comparing the two.

[0080] It should be noted that the static spatial transformation relationship describes the spatial rotation and spatial translation relationships between the complete three-dimensional model and the ideal three-dimensional model, and reflects the way of transforming the complete three-dimensional model into the ideal three-dimensional model. For example: in an X-Y-Z three-dimensional space coordinate system, assume that the center coordinates of a complete three-dimensional model , and the center coordinates of an ideal three-dimensional model . On the one hand, the static spatial transformation relationship describes that the complete three-dimensional model needs to undergo a translation of on the X-axis , on the Y-axis, and on the Z-axis so that the center points can coincide with the center point of the ideal three-dimensional model; assume that the key point coordinates of the translated complete three-dimensional model and the ideal three-dimensional model are , , respectively. On the other hand, the static spatial transformation relationship describes how the translated complete three-dimensional model rotates three-dimensionally with as the center point so that coincides with .

[0081] It should be noted that calculating the distance difference between and and the spatial angle difference between and rotating three-dimensionally with as the center point is the static pose difference value between the complete three-dimensional model and the ideal three-dimensional model.

[0082] It should be noted that through precise model comparison and analysis, product defects, construction errors or positioning deviations can be discovered in a timely manner, thereby ensuring product quality and production efficiency.

[0083] Step S106: Adjust the object until the static pose difference value is less than a preset difference threshold, and send the object into a turning machine for turning. Update the complete three-dimensional model in real time to obtain a real-time three-dimensional model.

[0084] It should be noted that the difference threshold is a key parameter, which determines the stability of the object during the turning process. If the difference threshold is set too large, the object is likely to be damaged due to incorrect pose when entering the turning machine, affecting the efficiency; if the difference threshold is set too small, it will lead to frequent adjustments, affecting the turning efficiency. The real-time three-dimensional model is a three-dimensional model obtained in real time through the methods described in Steps S101 - S104 during the turning process of the object, which reflects the pose of the object during the turning process.

[0085] Step S107: Calculate the torque generated during the turning process based on the physical parameters and the real-time three-dimensional model, and determine the optimal turning path in combination with the motion constraints of the turning machine.

[0086] Based on the mass distribution in the physical parameters and the position of the real-time three-dimensional model in the turning machine, perform a force analysis on the object through the relationship between gravity and mass to determine the resultant force F of the object during the turning process. Then, taking the turning axis of the turning machine as the rotation axis, calculate the distance d from the object to the rotation axis. The torque calculation formula is as follows:

[0087]

[0088] Among them, represents the torque, F represents the resultant force of the object during the turning process, and d represents the distance from the object to the rotation axis.

[0089] Calculate the real-time torque generated by the object during the turning process. Combine the real-time torque and the motion constraints, and use the torque balance algorithm for path planning to obtain a set of turning paths. Discretize the set of turning paths into multiple key points. Among them, each key point corresponds to a curvature radius and a path length. According to the key points and the motion constraints, determine the next key point of the turning path according to a preset comprehensive trajectory index, and iterate continuously to obtain the optimal turning path.

[0090] In the embodiments of the present invention, the physical parameters refer to the mass distribution, geometric shape, and the maximum torque that the object can withstand; the set of turning paths is a set of turning paths predicted by the torque balance algorithm, which reflects the possible turning paths of the object. The object turning along the paths specified in the set of turning paths will surely meet the motion constraints, and the torque borne by the object will also be necessarily less than the maximum torque that it can withstand; the key points are sampled from the paths in the set of turning paths. Exemplarily, a point is sampled every certain path distance, and the curvature radius of this point is recorded at the same time.

[0091] It should be noted that the comprehensive trajectory index is a key parameter, which determines the selection criteria for the optimal flipping path. In the embodiment of the present invention, the formula of the comprehensive trajectory index is as follows:

[0092]

[0093] Wherein, represents the path length, represents the maximum flipping path length; represents the length weight, represents the curvature radius weight, represents the curvature radius, represents the minimum curvature radius; represents the comprehensive trajectory index, where the larger it is, the more the influence of the path length on the optimal flipping path is emphasized, the larger it is, the more the influence of the curvature radius on the optimal flipping path is emphasized, the smaller it is, the more likely the key point is to be selected as a point on the optimal flipping path.

[0094] Taking a cuboid workpiece as an example, its dimensions are 2 meters in length, 1 meter in width, and 0.5 meter in height. The center of gravity is located at the geometric center, the mass is 200 kilograms, and the maximum torque it can withstand is 50 N·m. During the flipping process, the attitude of the real-time three-dimensional model of the workpiece is obtained in real time through sensors. Combining with the mass and dimension parameters, the torque change under different attitudes can be calculated. Assume that the flipping machine is driven by two sets of hydraulic cylinders, and its movement range is limited within 90 degrees in the vertical direction and 45 degrees in the horizontal direction. According to the principle of the torque balance algorithm, under the action of the workpiece gravity, the hydraulic cylinders need to provide corresponding balancing torques, so as to plan a smooth flipping path. These paths form a path set, including multiple feasible flipping trajectories. The path discretization process converts the continuous flipping path into a key point sequence. For the above workpiece, the 90-degree flipping process can be divided into 9 key points, with each point spaced 10 degrees apart. Each key point corresponds to a specific curvature radius and path length. For example, at the key point at the 45-degree position, its curvature radius is 1.5 meters, and the corresponding path length is 0.8 meters. During the flipping process of the above workpiece, by evaluating the state parameters of each key point, such as the angular velocity not exceeding 10 degrees per second and the acceleration not exceeding 5 degrees per square second, the next key point that meets the constraint conditions and has the optimal comprehensive index is selected. By iteratively determining the set of key points of the complete path trajectory, an optimal flipping path that takes into account both safety and efficiency is finally obtained.

[0095] Step S108, perform smoothing processing on the optimal flipping path to generate a smooth flipping trajectory from the current position to the target position.

[0096] Smooth the optimal flipping path to obtain a continuous preliminary smoothed trajectory. According to the preliminary smoothed trajectory and motion constraints, calculate the smoothed curvature radius of each point in the trajectory. Determine whether the smoothed curvature radius is less than a preset curvature radius threshold. If so, smooth the point corresponding to the smoothed curvature radius and the surrounding trajectory. If not, no additional operation is performed to obtain the smoothed flipping trajectory.

[0097] It should be noted that the preliminary smoothed trajectory is obtained by complementing and smoothing the discrete key points of the optimal flipping path and is a continuous trajectory line; the smoothed curvature radius of each point refers to the curvature radius calculated by sampling at a small interval on the preliminary smoothed trajectory. Exemplarily, the curvature radius is calculated every 1 cm; the preset curvature radius threshold is a key parameter that determines the minimum curvature radius of the smoothed flipping trajectory. If the threshold is too small, the object may be damaged due to excessive force during the flipping process. The core of the path smoothing algorithm is to connect discrete key points into a smooth continuous trajectory, which is crucial for the stability and safety of the workpiece flipping process. Taking the workpiece flipping as an example, assume that a flipping trajectory consists of multiple key points, and the connection lines between each key point may have sudden changes or corners, which will cause the workpiece to jitter or be unstable during the movement.

[0098] In the embodiment of the present invention, a cubic spline interpolation smoothing algorithm is adopted to generate a smooth curve passing through these key points to form a preliminary smoothed trajectory. In another feasible embodiment, taking the flipping of a cuboid workpiece with a weight of two hundred kilograms as an example, the path between key points is smoothed using a Bezier curve. The curvature radius of the trajectory after preliminary smoothing needs to be further checked to see if it meets the requirements.

[0099] Exemplarily, set the preset curvature radius threshold to 1.5 meters. If the curvature radius of a certain point is less than this threshold, it means that the turning at this point is too rapid, which may cause excessive force on the workpiece or excessive load on the flipping mechanism. For the points with a curvature radius less than the threshold, local smoothing needs to be performed again. The specific method is to take three points before and after this point to form a seven-point sequence centered on this point, and reapply the smoothing algorithm to process this section of the trajectory. For example, during the flipping of the workpiece from the horizontal position to the vertical position, if it is found that the curvature radius is 1 meter at the 45-degree position, which is less than the threshold, secondary smoothing needs to be performed at this point to make the transition smoother. The smoothed flipping trajectory obtained after repeated optimization not only ensures the continuity of the trajectory but also ensures that the curvature radius of each point is within the safe range. This kind of smoothing can effectively reduce the impact force during the workpiece flipping process, reduce equipment wear, and extend the service life of the equipment.

[0100] Step S109: Generate an ideal real-time three-dimensional model by combining the smooth flipping trajectory with the ideal three-dimensional model, and generate a control instruction to control the flipper to flip the object in combination with the moving range and the motion constraint.

[0101] Detect the dynamic attitude difference value between the real-time three-dimensional model and the ideal real-time three-dimensional model, and determine whether the dynamic attitude difference value exceeds a preset difference threshold. If so, trigger the flipper attitude correction mechanism and generate a control instruction to control the flipper to correct the object's attitude. For the real-time three-dimensional model and the ideal real-time three-dimensional model, extract the valid feature points of both. According to the valid feature points, calculate the real-time spatial transformation relationship between the real-time three-dimensional model and the ideal real-time three-dimensional model to obtain the dynamic attitude difference value of the real-time three-dimensional model. Determine whether the dynamic attitude difference value exceeds the preset difference threshold. If so, trigger the flipper attitude correction mechanism and generate an attitude adjustment instruction. If not, do nothing. The flipper control system receives the attitude adjustment instruction and calculates the angle and displacement that the object needs to be adjusted. The flipper precisely controls the flipping of the object to adjust the object until the dynamic attitude difference value is less than the preset difference threshold.

[0102] It should be noted that the valid feature points refer to the geometric feature points with significant features on the surface of the three-dimensional model. In the embodiment of the present invention, the significant features are measured by the curvature value on the surface of the three-dimensional model. When the curvature value of a certain point on the surface of the three-dimensional model is greater than the preset curvature threshold, it is determined that this point is a valid feature point on the surface of the three-dimensional model.

[0103] It should be noted that the extraction of the valid feature points of both is based on the curvature value on the surface of the three-dimensional model, and the points with curvature values greater than the preset curvature threshold are determined as valid feature points.

[0104] It should be noted that the real-time spatial transformation relationship describes the spatial rotation relationship and spatial translation relationship between the real-time three-dimensional model and the ideal real-time three-dimensional model. For example: in an X-Y-Z three-dimensional space coordinate system, assume the center coordinate of a real-time three-dimensional model , and the center coordinate of an ideal real-time three-dimensional model . The dynamic spatial transformation relationship describes on the one hand that the real-time three-dimensional model needs to be translated by on the X-axis, on the Y-axis, and on the Z-axis so that the center points can coincide with the center point of the ideal three-dimensional model; assume that the key point coordinates of the translated real-time three-dimensional model and the ideal real-time three-dimensional model are , , respectively. The dynamic spatial transformation relationship describes on the other hand how the translated complete three-dimensional model rotates three-dimensionally along points so that coincides with Coincide.

[0105] It should be noted that and the distance difference from and with as the rotation center point, the spatial angle difference is the dynamic attitude difference value between the real-time three-dimensional model and the ideal real-time three-dimensional model. The difference threshold is a key parameter, which determines the sensitivity of the turning machine to the offset of the object. If it is set too high, it is easy to cause the object to deviate from the predetermined trajectory during the turning process. If it is set too low, it is easy to cause the turning machine to frequently adjust the object, affecting the efficiency. For example, the difference threshold is set such that the translation distance does not exceed 0.1 m and the spatial rotation angle does not exceed 5°. When the distance difference exceeds 0.1 m or the spatial angle difference exceeds 5°, the attitude correction mechanism of the turning machine is triggered to generate an attitude adjustment instruction.

[0106] It should be noted that the precise attitude adjustment of the turning mechanism is based on closed-loop control. When it is detected that an item needs to be adjusted, the target angle and direction of adjustment are first calculated. For example, when a forty-five-degree adjustment is required, the system dynamically adjusts the motor output torque according to the deviation between the current angle and the target angle. A larger force is used for rapid adjustment in the initial stage, and the adjustment force is reduced when approaching the target position to avoid overshoot. Fine-tuning is performed through real-time feedback until the object attitude returns to the allowable error range. During the entire adjustment process, the system continuously performs real-time monitoring and feedback. Taking the adjustment of industrial parts as an example, when it is detected that the parts are gradually adjusted from the initial thirty degrees to twenty degrees, ten degrees, and finally reach the target vertical state, the system will calculate the remaining deviation value in real time. If the deviation after adjustment still exceeds the preset threshold, a new round of fine-tuning will be started until the deviation is reduced to an acceptable range. This closed-loop feedback mechanism ensures the accuracy and reliability of the attitude adjustment.

[0107] Such as Figure 2As shown in the figure, the present invention provides an intelligent flipping control device based on machine vision, mainly including: a data acquisition module, which is used to acquire the physical parameters of the object to be processed, the original image group of multiple perspectives, obtain the ideal three-dimensional model, the moving range and motion constraints of the flipping machine; an image preprocessing module, which is used to preprocess the original image group to obtain a uniform image group; a feature recognition and reconstruction module, which is used to identify the key feature points of the object in the uniform image group and perform three-dimensional reconstruction on the key feature points to generate the initial three-dimensional model of the object; an occluded area complement module, which is used to predict the position of the feature points in the occluded area of the initial three-dimensional model based on the uniform image group and the deep learning algorithm, and complement the three-dimensional model of the occluded area of the initial three-dimensional model to obtain a complete three-dimensional model; a posture difference calculation module, which is used to compare the complete three-dimensional model with the ideal three-dimensional model to obtain the static posture difference value of the complete three-dimensional model; an object adjustment module, which is used to adjust the object until the static posture difference value is less than the preset difference threshold, and send the object into the flipping machine for flipping, and update the complete three-dimensional model in real time to obtain a real-time three-dimensional model; a torque calculation and path determination module, which is used to calculate the torque generated during the flipping process according to the physical parameters and the real-time three-dimensional model, and combine the motion constraints of the flipping machine to determine the optimal flipping path; a path smoothing processing module, which is used to smooth the optimal flipping path to generate a smooth flipping trajectory from the current position to the target position; a control instruction generation module, which is used to generate an ideal real-time three-dimensional model by combining the smooth flipping trajectory with the ideal three-dimensional model, and generate a control instruction to control the flipping machine to flip the object in combination with the moving range and the motion constraints. Although the present invention has been described in detail above with general descriptions and specific embodiments, on the basis of the present invention, some modifications or improvements can be made, which are obvious to those skilled in the art. Therefore, these modifications or improvements made without departing from the spirit of the present invention all fall within the scope of protection required by the present invention.

[0108] An embodiment of the present invention also provides an electronic device. The electronic device includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, such as an intelligent flipping control program based on machine vision. When the processor executes the computer program, it implements the steps in each embodiment of the above-mentioned intelligent flipping control method based on machine vision, such as Figure 1 the step S101 shown in the figure. Alternatively, when the processor executes the computer program, it implements the functions of each module in each device embodiment above, such as the torque calculation and path determination module.

[0109] Exemplarily, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the computer program in the electronic device.

[0110] The electronic device may be a computing device such as a desktop computer, a notebook, a palm computer, and a smart tablet. The electronic device may include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the above components are only examples of the electronic device and do not constitute a limitation on the electronic device. It may include more or fewer components than the above, or combine certain components, or different components. For example, the electronic device may further include input / output devices, network access devices, a bus, etc.

[0111] The so-called processor may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The processor is the control center of the electronic device and connects various parts of the entire electronic device through various interfaces and lines.

[0112] The memory may be used to store the computer program and modules. The processor realizes various functions of the electronic device by running or executing the computer program and modules stored in the memory, and by calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area. Among them, the program storage area may store an operating system, applications required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area may store data created according to the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a FlashCard, at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0113] Among them, if the modules integrated in the electronic device are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above-mentioned embodiment methods of the present invention, it can also be completed by a computer program instructing relevant hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0114] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the attached drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines. Those of ordinary skill in the art can understand and implement it without creative work.

[0115] The specific embodiments described above have further elaborated on the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only the specific embodiments of the present invention and is not used to limit the protection scope of the present invention. It is particularly pointed out that for those skilled in the art, any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An intelligent flipping control method based on machine vision, characterized in that The method includes: Obtaining the physical parameters of the object to be processed, the original image group from multiple perspectives, obtaining the ideal three-dimensional model, the moving range and motion constraints of the flipping machine; Preprocessing the original image group to obtain a uniform image group; Identifying the key feature points of the object in the uniform image group and performing three-dimensional reconstruction on the key feature points to generate an initial three-dimensional model of the object; Based on the uniform image group, predicting the positions of the occluded region feature points of the initial three-dimensional model using a deep learning algorithm, and complementing the occluded region of the initial three-dimensional model to obtain a complete three-dimensional model; Comparing the complete three-dimensional model with the ideal three-dimensional model to obtain the static pose difference value between the two models; Adjusting the object until the static pose difference value is less than a preset difference threshold, and sending the object into the flipping machine for flipping, and updating the complete three-dimensional model in real time to obtain a real-time three-dimensional model; According to the physical parameters and the real-time three-dimensional model, calculating the torque generated during the flipping process, and combining the motion constraints of the flipping machine to determine the optimal flipping path, including: According to the physical parameters and the real-time three-dimensional model, calculating the real-time torque generated by the object during the flipping process; Combining the real-time torque and the motion constraints, and using a torque balance algorithm for path planning to obtain a set of flipping paths; Discretizing the set of flipping paths into multiple key points, where each key point corresponds to a curvature radius and a path length; According to the key points and the motion constraints, determining the next key point of the flipping path according to a preset comprehensive trajectory index, and continuously iterating to obtain the optimal flipping path; Smoothing the optimal flipping path to generate a smooth flipping trajectory from the current position to the target position; Combining the moving range, the motion constraints and the smooth flipping trajectory to generate a control instruction to control the flipping machine to flip the object.

2. The method according to claim 1, wherein The preprocessing of the original image group to obtain a uniform image group includes: Performing denoising processing on the original image group to obtain a denoised image group; Adjusting the overall brightness distribution of the denoised image group to obtain an equalized image group; Dividing the equalized image group into multiple regions and determining the initial texture pattern of each region; Completing the initial texture pattern and eliminating the edge traces between regions to generate a complete texture pattern; Fusing the complete texture pattern with the equalized image group to generate the uniform image group.

3. The method according to claim 1, characterized in that The identifying the key feature points of the object in the uniform image group and performing three-dimensional reconstruction on the key feature points to generate an initial three-dimensional model of the object includes: Identifying the key feature points of the object from the uniform image group and obtaining the two-dimensional coordinates of the key feature points; Based on the two-dimensional coordinates, calculating the three-dimensional spatial positions of the key feature points; According to the three-dimensional spatial positions, constructing an initial three-dimensional point cloud model of the object; Performing surface fitting on the initial three-dimensional point cloud model to generate an initial three-dimensional surface model of the object.

4. The method according to claim 1, characterized in that, Based on the uniform image group, predicting the positions of the feature points in the occluded area of the initial three-dimensional model according to a deep learning algorithm, and complementing the three-dimensional model of the occluded area of the initial three-dimensional model to obtain a complete three-dimensional model, including: Using a convolutional neural network to identify the occluded areas of the uniform image group, and obtaining the three-dimensional occlusion feature point positions of the occluded areas through triangulation; Processing the feature point positions to generate a three-dimensional point cloud model of the occluded area; Reconstructing the three-dimensional point cloud model of the occluded area to construct a three-dimensional surface model of the occluded area; Processing the three-dimensional surface model of the occluded area, and then obtaining the three-dimensional model of the occluded area through texture completion; Combining the three-dimensional model of the occluded area and the initial three-dimensional model to obtain the complete three-dimensional model.

5. The method according to claim 1, wherein Comparing the complete three-dimensional model with the ideal three-dimensional model to obtain the static attitude difference value of the complete three-dimensional model, including: Extracting the model feature points of the complete three-dimensional model and the ideal three-dimensional model; According to the model feature points, calculating the static spatial transformation relationship between the complete three-dimensional model and the ideal three-dimensional model, and obtaining the static attitude difference value of the complete three-dimensional model through error calculation.

6. The method according to claim 1, wherein Smoothing the optimal flipping path to generate a smooth flipping trajectory from the current position to the target position, including: Smoothing the optimal flipping path to obtain a continuous preliminary smooth trajectory; According to the preliminary smooth trajectory and the motion constraints, calculating the smooth curvature radius of each point in the trajectory; Judging whether the smooth curvature radius is less than a preset curvature radius threshold. If so, smoothing the points corresponding to the smooth curvature radius and the surrounding trajectories. If not, no additional operation is performed to obtain a smooth flipping trajectory.

7. The method according to claim 1, characterized in that, Combining the moving range, the motion constraints and the smooth flipping trajectory to generate a control instruction to control the flipper to flip the object, including: Combining the smooth flipping trajectory with the ideal three-dimensional model to generate an ideal real-time three-dimensional model; Extracting the effective feature points of the real-time three-dimensional model and the ideal real-time three-dimensional model; According to the effective feature points, calculating the real-time spatial transformation relationship between the real-time three-dimensional model and the ideal real-time three-dimensional model to obtain the dynamic attitude difference value of the real-time three-dimensional model; Judging whether the dynamic attitude difference value exceeds a preset difference threshold. If so, triggering the attitude correction mechanism of the flipper to generate an attitude adjustment instruction. If not, no processing is performed; The flipper control system receives the attitude adjustment instruction and controls the angle and displacement of the object movement; The flipper precisely controls the flipping of the object and adjusts the object until the dynamic attitude difference value is less than the preset difference threshold.

8. An intelligent flipping control device based on machine vision, characterized in that, A device for implementing the intelligent flipping control method based on machine vision according to any one of claims 1 to 7, the device includes: A data acquisition module, configured to acquire the physical parameters of the object to be processed, the original image group of multiple perspectives, acquire the ideal three-dimensional model, the moving range and motion constraints of the flipper; An image preprocessing module, configured to preprocess the original image group to obtain a uniform image group; A feature recognition and reconstruction module, which is used to recognize the key feature points of the object in the uniform image group, perform three-dimensional reconstruction on the key feature points, and generate an initial three-dimensional model of the object; An occlusion area completion module, which is used to predict the position of the feature points in the occluded area of the initial three-dimensional model based on the deep learning algorithm according to the uniform image group, and complete the three-dimensional model of the occluded area of the initial three-dimensional model to obtain a complete three-dimensional model; A posture difference calculation module, which is used to compare the complete three-dimensional model with the ideal three-dimensional model to obtain the static posture difference value of the complete three-dimensional model; An object adjustment module, which is used to adjust the object until the static posture difference value is less than a preset difference threshold, and send the object into a flipper for flipping, and update the complete three-dimensional model in real time to obtain a real-time three-dimensional model; A torque calculation and path determination module, which is used to calculate the torque generated during the flipping process according to the physical parameters and the real-time three-dimensional model, and determine the optimal flipping path in combination with the motion constraints of the flipper; A path smoothing processing module, which is used to smooth the optimal flipping path to generate a smooth flipping trajectory from the current position to the target position; A control instruction generation module, which is used to generate an ideal real-time three-dimensional model by combining the smooth flipping trajectory with the ideal three-dimensional model, and generate a control instruction to control the flipper to flip the object in combination with the moving range and the motion constraints.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein when the computer program runs, it controls the device where the computer-readable storage medium is located to execute a machine vision-based intelligent flipping control method according to any one of claims 1 to 7.

Images