Intelligent factory voice and vision cooperative control method and system

Abstract

The invention relates to the technical field of industrial automation and machine vision, in particular to an intelligent factory voice and vision cooperative control method and system. According to the intelligent factory voice and vision cooperative control method and system, a multi-target recognition model is combined with an object feature point extraction algorithm to determine the grabbing difficulty tendency type of a target object; according to the grabbing difficulty tendency type of the target object and the shape complexity degree of the target object, the calculation sub-parameters for the grabbing posture of the target object are determined, then the grabbing posture of the mechanical arm is determined according to the calculation sub-parameters, and non-contact control is achieved through the preset wake-up word. Operators can quickly start the grabbing process through natural language processing instructions, time delay of traditional button or touch screen operation is remarkably reduced, voice instructions and visual collection are seamlessly connected, the manual confirmation link is avoided, the system response speed is increased, and the system reliability is improved. And for a target object with relatively high grabbing difficulty, the grabbing success rate is improved through a multi-angle coordinate calculation mode.

Description

Technical Field

[0001] The present invention relates to the field of industrial automation and machine vision technology, and in particular to a method and system for collaborative voice and vision control of an intelligent factory. Background Art

[0002] In the robotic arm grasping system of traditional smart factories, fixed programming paths or single visual guidance are usually used to control the movement of the robotic arm. The existing technology dynamically adjusts the lighting intensity around the products on the production line based on the comparison results by dynamically adjusting the illuminator, and acquires and analyzes image data from multispectral cameras in real time to identify defects on the product surface and eliminate defective products. However, traditional methods mostly use single-view multi-point coordinate calculations, which are difficult to handle complex objects or interference scenes, resulting in low grasping efficiency.

[0003] Chinese patent application number: CN202411173365.4 discloses a smart factory production line visual inspection system and method, including: an environmental perception module to obtain the light intensity around the products on the production line in real time; a system integration control module to receive the light intensity around the products in real time and compare it with a preset threshold; a dynamic adjustment illuminator to dynamically adjust the light intensity around the products on the production line according to the comparison result; a multi-spectral camera module to obtain images of products that meet the light intensity in real time; an intelligent recognition and processing module to obtain and analyze image data from the multi-spectral camera in real time to identify defects on the product surface; if there are defects on the surface of the identified product, the defective product will be rejected by the automatic rejection module and placed in the defective product placement area; if the number of products placed in the defective product placement area exceeds two-thirds, feedback will be given to the remote interactive control panel module, and the early warning module will notify the production personnel to inspect the products in the defective product placement area to determine whether there is any false detection.

[0004] However, the prior art still has the following problems: Traditional methods mostly use single-view multi-point coordinate calculation, which is difficult to handle complex objects or interference scenes, resulting in low grasping efficiency. Summary of the Invention

[0005] To this end, the present invention provides a smart factory voice and vision collaborative control method and system to overcome the problem that traditional methods in the prior art mostly use single-view multi-point coordinate calculation, which is difficult to handle complex objects or interference scenes, resulting in low grasping efficiency.

[0006] To achieve the above objectives, the present invention provides a method for voice and visual collaborative control of smart factories. The method comprises: Step S1: wake up the device according to the preset wake-up word, recognize the voice command through the preset command word, and capture the target object image when the voice recognition is successful; Step S2, determining feature points of the target object based on the target object image, and determining a grasping difficulty tendency type according to a ratio of the number of feature points located in the feature area to the total number of feature points; Step S3, determining calculation parameters for the grasping posture of the target object based on the determined grasping difficulty tendency type and the shape complexity of the target object, including: The three-dimensional coordinate parameters of the single-angle target object space are used as the calculation sub-parameters of the three-dimensional coordinates of the manipulator target posture; Using the multi-angle target space 3D coordinate parameters as the calculation sub-parameters of the robot arm's target posture 3D coordinates, analyze whether there is a conflict in the target space 3D coordinate parameters at different angles, and analyze whether there is a local 3D coordinate distortion phenomenon of the target space. Based on the analysis results, determine whether to eliminate interfering coordinates or correct the shooting conditions; Step S4: determining the grasping posture of the robot arm according to the determined grasping posture calculation sub-parameters, and grasping the target object.

[0007] Furthermore, in step S2, determining the grasping difficulty tendency type according to the ratio of the number of feature points located in the feature area to the total number of feature points includes: Calculate the ratio of the number of feature points in the feature area to the total number of feature points to obtain the representation of the grasping difficulty. If the grasping difficulty representation is less than or equal to the preset grasping difficulty representation, determining that the grasping difficulty tendency type of the target object is a strong grasping difficulty tendency; If the grasping difficulty representation is greater than the preset grasping difficulty representation, it is determined that the grasping difficulty tendency type of the target object is a weak grasping difficulty tendency.

[0008] Furthermore, in step S3, the calculation parameters for the grasping posture of the target object are determined based on the determined grasping difficulty tendency type and the shape complexity of the target object, including: If the grasping difficulty tendency type of the target object is weak grasping difficulty tendency and the shape complexity of the target object is low, the single-angle target object space three-dimensional coordinate parameters are used as calculation sub-parameters for the three-dimensional coordinates of the manipulator target posture; If the grasping difficulty tendency type of the target object is a strong grasping difficulty tendency or the shape of the target object is highly complex, the multi-angle target space three-dimensional coordinate parameters are used as the calculation sub-parameters of the three-dimensional coordinates of the robot arm target posture, and it is analyzed whether there is a conflict in the target space three-dimensional coordinate parameters at different angles, and whether there is a local space three-dimensional coordinate distortion phenomenon of the target object. Based on the analysis results, it is determined to eliminate the interfering coordinates or correct the shooting conditions.

[0009] Furthermore, in step S3, contour features of the target object are extracted based on the target object image, and contour curvature is calculated to determine the shape complexity of the target object based on the contour curvature.

[0010] Furthermore, the analysis of whether there is a conflict in the three-dimensional coordinate parameters of the target object at different angles includes: Convert the coordinates to the same reference frame, Use hand-eye calibration to align the data. Select the key points of the target object, Calculate the coordinates of the same key point at each angle respectively. Calculate the Euclidean distance deviation, Determine the key points whose Euclidean distance deviation is greater than or equal to the preset Euclidean distance deviation, and obtain abnormal key points. Count the number of abnormal key points, If the number of abnormal key points is greater than or equal to the preset number, it is determined that there is a conflict in the three-dimensional coordinate parameters of the target space at different angles; If the number of the abnormal key points is less than the preset number, it is determined that there is no conflict in the three-dimensional coordinate parameters of the target object space at different angles.

[0011] Furthermore, the analysis of whether there is a local three-dimensional coordinate distortion phenomenon of the target object includes: Divide the target object into several local areas, Calculate the variance of the coordinates of a single local region, Calculate the average variance of the coordinates of each local area, Calculate the difference between the variance of the coordinates of a single local area and the average variance to obtain the distortion representation. Count the number of local areas whose distortion representation is greater than or equal to the preset distortion representation to obtain the number of distorted areas. If the number of distorted regions is greater than or equal to the preset number of distorted regions, it is determined that there is a local three-dimensional coordinate distortion phenomenon of the target object; If the number of the distorted regions is less than the preset number of distorted regions, it is determined that there is no local spatial three-dimensional coordinate distortion of the target object.

[0012] Furthermore, the determining of eliminating interference coordinates or correcting shooting conditions based on the analysis results includes: If there is a conflict in the three-dimensional coordinate parameters of the target object at different angles and there is a distortion phenomenon in the three-dimensional coordinates of the local space of the target object, the interference coordinates are determined to be excluded; If only the three-dimensional coordinate parameters of the target object at different angles conflict or only the three-dimensional coordinate distortion of the local space of the target object exists, the correction shooting condition is determined.

[0013] Furthermore, the eliminating interference coordinates includes: The abnormal key points and local areas whose distortion representation is greater than or equal to the preset distortion representation are eliminated.

[0014] Furthermore, the correction of shooting conditions includes: The shooting angle of view is increased to increase the three-dimensional coordinate parameters of the target object space, wherein the increase in the shooting angle of view is positively correlated with the number of the distortion areas.

[0015] The present invention provides a smart factory voice and vision collaborative control system, comprising: Furthermore, a voice command recognition module is used to recognize a wake-up command based on a preset wake-up word and recognize a voice command based on a preset command word; An image acquisition module, connected to the voice command recognition module, for acquiring an image of a target object; A feature point recognition module, connected to the voice command recognition module and the image acquisition module, is used to determine the feature points of the target object and determine the grasping difficulty tendency type of the target object based on the feature points; an analysis module, connected to the image acquisition module and the feature point recognition module, respectively, for determining, based on the target object's grasping difficulty tendency type, whether to use single-angle target object spatial three-dimensional coordinate parameters or multi-angle target object spatial three-dimensional coordinate parameters as calculation sub-parameters for the robot arm's target posture three-dimensional coordinates, analyzing whether there is a conflict between the target object spatial three-dimensional coordinate parameters at different angles, analyzing whether there is a local three-dimensional coordinate distortion phenomenon of the target object, and determining, based on the analysis results, to eliminate interfering coordinates or correct shooting conditions; A calculation module is connected to the image acquisition module, the feature point recognition module and the analysis module, and is used to calculate the grasping posture of the manipulator based on the calculation sub-parameters of the determined three-dimensional coordinates of the manipulator target posture.

[0016] Compared with the existing technology, the beneficial effect of the present invention is that the present invention realizes contactless control through preset wake-up words, and the operator can quickly start the grasping process through natural language processing instructions, which significantly reduces the time delay of traditional button or touch screen operations. The voice instructions are seamlessly connected with visual acquisition, avoiding manual confirmation links, and improving the system response speed. The grasping difficulty is classified based on the distribution ratio of feature points (proportion of feature areas), and the single-angle or multi-angle coordinate calculation mode is automatically selected according to the characteristics of the object. For targets that are more difficult to grasp, the multi-angle coordinate calculation mode is used to improve the grasping success rate.

[0017] Furthermore, the present invention realizes the transformation of difficulty assessment from qualitative to quantitative by calculating the ratio of the number of feature points in the feature area to the total number of feature points (characterization of grasping difficulty). For low-texture objects, even if the total number of feature points is small, they are still judged as weak difficulty if they are evenly distributed, avoiding overly conservative strategies. For local high-feature objects, strong difficulty is accurately identified when feature points are concentrated, triggering high-precision calculation, automatically shielding abnormal point clouds in reflective areas, and improving the success rate of grasping curved objects.

[0018] Furthermore, by extracting the contour and calculating the curvature, the present invention can convert the shape complexity of the target object into a quantitative indicator (such as curvature distribution, the proportion of high curvature areas, etc.), avoiding subjective judgment errors. The present invention does not rely on the specific type of the target object (such as regular geometric bodies, irregular objects), and can distinguish simple shapes (such as cylinders) from complex shapes (such as gears, blades) through curvature distribution, providing a basis for subsequent classification, matching or path planning. Through hand-eye calibration and coordinate unification, the spatial alignment problem of multi-sensor (such as camera) or multi-angle measurement data is solved, avoiding positioning deviations caused by inconsistent coordinate systems.

[0019] Furthermore, the present invention can accurately locate local distortions in three-dimensional coordinates (such as local density or sparseness of point clouds, and deformation of local geometric structures) by dividing the target object into local areas and calculating variance differences. Compared with global analysis, it is more sensitive to subtle anomalies. Through distortion detection and intelligent decision-making, it can significantly reduce the impact of noise, outliers and coordinate conflicts on the three-dimensional model, thereby improving reconstruction accuracy and stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of the intelligent factory voice and vision collaborative control method of the present invention; Figure 2 A flow chart for determining crawling difficulty tendency types according to the present invention; Figure 3 A flow chart for determining calculation parameters for grasping a target object according to the present invention; Figure 4 This is a flow chart for analyzing whether there is a conflict in the three-dimensional coordinate parameters of a target object at different angles in the present invention. DETAILED DESCRIPTION

[0021] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below with reference to embodiments. It should be understood that the specific embodiments described herein are merely used to explain the present invention and are not intended to limit the present invention.

[0022] It should be noted that the data in this embodiment are obtained by comprehensive analysis and evaluation of the historical data of the six months before the current determination and the corresponding historical determination results by the system of the present invention. It can be understood by those skilled in the art that the system of the present invention can determine the above parameters for each of the above parameters by selecting the value with the highest proportion as the preset standard parameter based on the data distribution, using weighted summation to use the obtained value as the preset standard parameter, substituting each historical data into a specific formula and using the value obtained by the formula as the preset standard parameter, or other selection methods, as long as the system of the present invention can clearly define the different specific situations in the single determination process through the obtained values.

[0023] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0024] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside", and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings. This is only for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0025] Furthermore, it should be noted that, in the description of the present invention, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0026] See also Figure 1 As shown, it is the intelligent factory voice and vision collaborative control method of the present invention, The smart factory voice and vision collaborative control method provided in this embodiment includes: Step S1: wake up the device according to the preset wake-up word, recognize the voice command through the preset command word, and capture the target object image when the voice recognition is successful; Step S2, determining feature points of the target object based on the target object image, and determining a grasping difficulty tendency type according to a ratio of the number of feature points located in the feature area to the total number of feature points; Step S3, determining calculation parameters for the grasping posture of the target object based on the determined grasping difficulty tendency type and the shape complexity of the target object, including: The three-dimensional coordinate parameters of the single-angle target object space are used as the calculation sub-parameters of the three-dimensional coordinates of the manipulator target posture; Using the multi-angle target space 3D coordinate parameters as the calculation sub-parameters of the robot arm's target posture 3D coordinates, analyze whether there is a conflict in the target space 3D coordinate parameters at different angles, and analyze whether there is a local 3D coordinate distortion phenomenon of the target space. Based on the analysis results, determine whether to eliminate interfering coordinates or correct the shooting conditions; Step S4: determining the grasping posture of the robot arm according to the determined grasping posture calculation sub-parameters, and grasping the target object.

[0027] Specifically, in this embodiment, feature points refer to unique pixel points extracted from the target object image through a feature point detection algorithm, which can characterize the local texture, contour, edge or geometric structure of the target object. This type of feature point has local uniqueness and can realize the morphological recognition and spatial positioning of the target object through its characteristic parameters such as position, grayscale change, gradient direction, etc., and is the basic unit for describing the surface characteristics of the target object.

[0028] Specifically, in this embodiment, the feature area refers to a specific area on the surface of the target object that is suitable for stable grasping by the robotic arm gripper. This area must meet the characteristics of structural stability (such as a flat surface, no significant protrusions or depressions), clamping reliability (such as having a certain friction or geometric constraints to prevent the target object from slipping during the grasping process), etc. It is a key area for evaluating the feasibility of grasping the target object. The distribution of feature points in the feature area directly reflects the grasping adaptability of the area, and its proportion to the total number of feature points of the target object is used to quantitatively characterize the difficulty of grasping the target object.

[0029] In the present invention, contactless control is achieved through a preset wake-up word. The operator can quickly start the grasping process through natural language processing instructions, significantly reducing the time delay of traditional button or touch screen operations. Voice commands are seamlessly connected with visual acquisition, avoiding manual confirmation links and improving the system response speed. The grasping difficulty is classified based on the distribution ratio of feature points (proportion of feature areas), and the single-angle or multi-angle coordinate calculation mode is automatically selected according to the characteristics of the object. For targets that are more difficult to grasp, the multi-angle coordinate calculation mode is used to improve the grasping success rate.

[0030] Specifically, in this embodiment, the gripping posture of the robotic arm can be determined by the following methods: Using OpenCV and the RealSense SDK to call camera resources, we collected 25 sets of calibration plate data and recorded the robotic arm position corresponding to each set of calibration plate data (including: robotic arm position parameters (X, Y, Z) and robotic arm posture parameters (RX, RY, RZ)). Adjust the size of the checkerboard, the number of checkerboard grids on the horizontal coordinate axis, and the number of checkerboard grids on the vertical coordinate axis according to the number and size of the checkerboard grids in the calibration plate, obtain the camera intrinsic parameters and distortion coefficients, run the calibration program, and obtain the hand-eye matrix between the current camera and the end of the robotic arm; Adjust the robotic arm's photo-taking posture and record the position and posture parameters (X, Y, Z, RX, RY, RZ) of the robotic arm when taking photos. Run the YOLO visual recognition program and combine it with the depth camera 3D coordinate calculation API to obtain the target object type and the 3D coordinates of the target object under the current camera. Calculate the position parameters (X, Y, Z) of the robotic arm based on the hand-eye matrix, the three-dimensional spatial coordinates of the target object, and the posture parameters (RX, RY, RZ) of the robotic arm, and finally obtain the complete motion parameters (X, Y, Z, RX, RY, RZ) of the robotic arm. For objects with irregular shapes or difficult-to-grasp positions, GraspNet is used to locate and estimate the object's pose. The robot's motion parameters (X, Y, Z, RX, RY, RZ) are calculated based on the hand-eye matrix, the object's three-dimensional spatial coordinates, and GraspNet's pose estimation results. Control the robot arm to move to the grasping posture according to the determined motion data (X, Y, Z, RX, RY, RZ) of the robot arm; Drive the gripper to grab the target and place it at the designated location.

[0031] Specifically, in this embodiment, the conversion from pixel coordinates in the camera to three-dimensional spatial coordinates is obtained through the camera SDK.

[0032] Specifically, in this embodiment, relevant parameters are adjusted according to the number and size of the checkerboard grids in the calibration plate, wherein the parameters include: checkerboard grid size, the number of checkerboard grids on the horizontal coordinate axis, and the number of checkerboard grids on the vertical coordinate axis.

[0033] Specifically, in step S2, determining the capture difficulty tendency type according to the ratio of the number of feature points located in the feature area to the total number of feature points includes: Use the multi-target recognition model to complete the recognition of targets within the camera's field of view. The feature extraction algorithm is used to complete the feature extraction of the target object, and the ratio of the number of feature points located in the feature area to the total number of feature points is calculated to obtain the representation of the grasping difficulty. If the grasping difficulty representation is less than or equal to the preset grasping difficulty representation, determining that the grasping difficulty tendency type of the target object is a strong grasping difficulty tendency; If the grasping difficulty representation is greater than the preset grasping difficulty representation, it is determined that the grasping difficulty tendency type of the target object is a weak grasping difficulty tendency.

[0034] Specifically, in this embodiment, the preset grasping difficulty representation can be determined by the following method: collect 100 typical part samples and divide them into two categories: easy-to-grasp samples (50): parts with a successful grasping rate >95% (such as square parts with a flat surface); difficult-to-grasp samples (50): parts with a grasping failure rate >30% (such as small cylindrical parts, flexible objects); use feature point detection algorithms such as SIFT / SURF to extract feature points of the target object image, count the number of feature points located in the feature area and the total number of feature points, calculate the ratio of the number of feature points to the total number of feature points, and analyze the R value distribution of the samples: the R values ​​of the easy-to-grasp samples are concentrated in the range of 0.65~0.95, with an average value of 0.82; the R values ​​of the difficult-to-grasp samples are concentrated in the range of 0.20~0.55, with an average value of 0.38; select the dividing point of the R value distribution of the two types of samples (such as 0.55) as the preset grasping difficulty representation.

[0035] Specifically, in step S3, the calculation parameters for the grasping posture of the target object are determined based on the determined grasping difficulty tendency type and the shape complexity of the target object, including: If the grasping difficulty tendency type of the target object is weak grasping difficulty tendency and the shape complexity of the target object is low, the single-angle target object space three-dimensional coordinate parameters are used as calculation sub-parameters for the three-dimensional coordinates of the manipulator target posture; If the grasping difficulty tendency type of the target object is a strong grasping difficulty tendency or the shape of the target object is highly complex, the multi-angle target space three-dimensional coordinate parameters are used as the calculation sub-parameters of the three-dimensional coordinates of the robot arm target posture, and it is analyzed whether there is a conflict in the target space three-dimensional coordinate parameters at different angles, and whether there is a local space three-dimensional coordinate distortion phenomenon of the target object. Based on the analysis results, it is determined to eliminate the interfering coordinates or correct the shooting conditions.

[0036] The present invention realizes the transformation of difficulty assessment from qualitative to quantitative by calculating the ratio of the number of feature points in the feature area to the total number of feature points (characterization of grasping difficulty). For low-texture objects, even if the total number of feature points is small, they are still judged as weak difficulty if they are evenly distributed, avoiding overly conservative strategies. For local high-feature objects, strong difficulty is accurately identified when feature points are concentrated, triggering high-precision calculation, automatically shielding abnormal point clouds in reflective areas, and improving the success rate of grasping curved objects.

[0037] Specifically, in step S3, the contour features of the target object are extracted based on the target object image, and the contour curvature is calculated to determine the shape complexity of the target object based on the contour curvature.

[0038] Specifically, the analysis of whether there is a conflict in the three-dimensional coordinate parameters of the target object at different angles includes: Convert the coordinates to the same reference frame, Use hand-eye calibration to align the data. Select the key points of the target object, Calculate the coordinates of the same key point at each angle respectively. Calculate the Euclidean distance deviation, Determine the key points whose Euclidean distance deviation is greater than or equal to the preset Euclidean distance deviation, and obtain abnormal key points. Count the number of abnormal key points, If the number of abnormal key points is greater than or equal to the preset number, it is determined that there is a conflict in the three-dimensional coordinate parameters of the target space at different angles; If the number of the abnormal key points is less than the preset number, it is determined that there is no conflict in the three-dimensional coordinate parameters of the target object space at different angles.

[0039] Specifically, in this embodiment, the preset Euclidean distance deviation can be determined by the following method to calibrate the sensor accuracy: binocular camera: measure a calibration plate of known size in the workspace, statistically analyze the measurement error distribution, and determine its standard deviation to be ±0.15mm; depth camera: measure the fixed point multiple times and calculate the standard deviation of its depth error to be ±0.2mm; comprehensive error: according to the error propagation law, the theoretical error after the fusion of the two sensors is: A= ≈0.25mm; Determine the positioning accuracy of the robotic arm: the robot's repeatability is ±0.08mm (manufacturer specification), and in actual tests, the positioning error within the working area is distributed within ±0.1mm; Considering the superposition of sensor error and robotic arm error, the theoretically allowable maximum deviation is: B = A + robotic arm error = 0.25 + 0.1 = 0.35mm; Perform 100 key point coordinate measurements in a collision-free scenario, calculate the Euclidean distance deviation between the two sensors, and obtain a distribution: 95% of the deviation values ​​are <0.3mm, and the maximum value is 0.32mm; In a test where a 0.5mm artificial offset is known to exist, the system successfully identifies all abnormal points. Combining theoretical calculations with measured data, the preset Euclidean distance deviation is set to 0.3mm.

[0040] Specifically, in this embodiment, the number of key points directly affects the ratio threshold of abnormal points. Usually, a preset number is set in proportion (such as 10%-20%) to avoid misjudgment due to differences in the total number of key points.

[0041] In the present invention, by extracting the contour and calculating the curvature, the shape complexity of the target object can be converted into quantitative indicators (such as curvature distribution, proportion of high curvature areas, etc.), avoiding subjective judgment errors. The present invention does not rely on the specific type of the target object (such as regular geometric bodies, irregular objects). The curvature distribution can distinguish simple shapes (such as cylinders) from complex shapes (such as gears, blades), providing a basis for subsequent classification, matching or path planning. Through hand-eye calibration and coordinate unification, the spatial alignment problem of multi-sensor (such as camera) or multi-angle measurement data is solved, avoiding positioning deviations caused by inconsistent coordinate systems.

[0042] Specifically, the analysis of whether there is a local three-dimensional coordinate distortion phenomenon of the target object includes: Divide the target object into several local areas, Calculate the variance of the coordinates of a single local region, Calculate the average variance of the coordinates of each local area, Calculate the difference between the variance of the coordinates of a single local area and the average variance to obtain the distortion representation. Count the number of local areas whose distortion representation is greater than or equal to the preset distortion representation to obtain the number of distorted areas. If the number of distorted regions is greater than or equal to the preset number of distorted regions, it is determined that there is a local three-dimensional coordinate distortion phenomenon of the target object; If the number of the distorted regions is less than the preset number of distorted regions, it is determined that there is no local spatial three-dimensional coordinate distortion of the target object.

[0043] Specifically, in this embodiment, the number of preset distortion areas can be determined by the following method: set the target object to be a cube with a side length of 10 cm, divide it into 20 local areas, shoot it with a binocular camera, and obtain 100 three-dimensional coordinate points in each local area; divide the 6 faces of the cube into 4 rectangular areas (such as each face is divided into 2×2 grids), the total number of local areas = 6×4 = 24; assuming that the grasping task allows a maximum of 10% of the area to be distorted, the number of preset distortion areas = 4×10% = 2.4, rounded up to 3; if the number of distortion areas obtained by statistics ≥3, it is determined that three-dimensional coordinate distortion exists; calculate the variance and average variance of each region: assume that the coordinate variance of each local region is: [0.12, 0.08, 0.56, 0.23, 0.15, ..., 0.31] (a total of 24 data); calculate the average variance: σ = 0.25, standard deviation: σ = 0.15; take the average variance + 1 times the standard deviation as the distortion characterization critical value: 0.25 + 0.15 = 0.4; assume that there are 5 regions with variance ≥ 0.4, then the number of preset distorted regions can be set to 5 (that is, the threshold is dynamically determined according to the actual distribution).

[0044] Specifically, determining the interference elimination coordinates or correcting the shooting conditions based on the analysis results includes: If there is a conflict in the three-dimensional coordinate parameters of the target object at different angles and there is a distortion phenomenon in the three-dimensional coordinates of the local space of the target object, the interference coordinates are determined to be excluded; If only the three-dimensional coordinate parameters of the target object at different angles conflict or only the three-dimensional coordinate distortion of the local space of the target object exists, the correction shooting condition is determined.

[0045] By dividing the target object into local areas and calculating the variance difference, the present invention can accurately locate local distortions in three-dimensional coordinates (such as local density or sparseness of point clouds, and deformation of local geometric structures). Compared with global analysis, it is more sensitive to subtle anomalies. Through distortion detection and intelligent decision-making, it significantly reduces the impact of noise, outliers and coordinate conflicts on the three-dimensional model, thereby improving reconstruction accuracy and stability.

[0046] Specifically, eliminating interference coordinates includes: The abnormal key points and local areas whose distortion representation is greater than or equal to the preset distortion representation are eliminated.

[0047] Specifically, the correction of shooting conditions includes: The shooting angle of view is increased to increase the three-dimensional coordinate parameters of the target object space, wherein the increase in the shooting angle of view is positively correlated with the number of the distortion areas.

[0048] Specifically, in this embodiment, increasing the shooting angle to increase the three-dimensional coordinate parameters of the target object space can be adjusted in the following manner: dividing the surface into 30 local areas according to the curvature distribution (12 high curvature areas and 18 low curvature areas); calculating the coordinate variance of each local area, the average variance is 0.35mm2, and the standard deviation is 0.18mm2; the distortion characterization threshold is 0.35+0.18=0.53mm2; the number of distorted areas is counted as 8 (exceeding the preset number by 3, the preset number is set at 10%: 30×10%= 3); Select 20 key points and calculate the coordinate deviation between the binocular camera and the depth camera; the number of abnormal key points with a Euclidean distance deviation ≥ 0.3mm is 5 (exceeding the preset number by 4, and the preset number is set at 20%: 20 × 20% = 4); Due to the simultaneous existence of coordinate conflict and distortion (8 distorted areas + 5 abnormal key points), it is determined that the shooting conditions need to be corrected and the shooting angle of view needs to be increased; the increase in shooting angle of view = the number of distorted areas × the angle of view coefficient (k = 0.5, which can be adjusted according to the task accuracy); the number of angles required to be increased = 8 × 0.5 = 4.

[0049] The smart factory voice and vision collaborative control system provided in this embodiment includes: A voice command recognition module, which is used to recognize voice commands based on preset wake-up words; An image acquisition module, connected to the voice command recognition module, for acquiring an image of a target object; A feature point recognition module, connected to the voice command recognition module and the image acquisition module, is used to determine the feature points of the target object and determine the grasping difficulty tendency type of the target object based on the feature points; an analysis module, connected to the image acquisition module and the feature point recognition module, respectively, for determining, based on the target object's grasping difficulty tendency type, whether to use single-angle target object spatial three-dimensional coordinate parameters or multi-angle target object spatial three-dimensional coordinate parameters as calculation sub-parameters for the robot arm's target posture three-dimensional coordinates, analyzing whether there is a conflict between the target object spatial three-dimensional coordinate parameters at different angles, analyzing whether there is a local three-dimensional coordinate distortion phenomenon of the target object, and determining, based on the analysis results, to eliminate interfering coordinates or correct shooting conditions; A calculation module is connected to the image acquisition module, the feature point recognition module and the analysis module, and is used to calculate the grasping posture of the manipulator based on the calculation sub-parameters of the determined three-dimensional coordinates of the manipulator target posture.

[0050] Specifically, there is no limitation on the specific structures of the voice command recognition module and the image acquisition module, as long as they can realize the corresponding functions.

[0051] Specifically, there is no limitation on the specific structures of the feature point recognition module, the analysis module and the calculation module, which may be composed of logic components including a field programmable processor, a computer and a microprocessor in the computer.

[0052] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

[0053] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for collaborative control of voice and vision in smart factories, characterized in that: include: Step S1: wake up the device according to the preset wake-up word, recognize the voice command through the preset command word, and capture the target object image when the voice recognition is successful; Step S2, determining feature points of the target object based on the target object image, and determining a grasping difficulty tendency type according to a ratio of the number of feature points located in the feature area to the total number of feature points; Step S3, determining calculation parameters for the grasping posture of the target object based on the determined grasping difficulty tendency type and the shape complexity of the target object, including: The three-dimensional coordinate parameters of the single-angle target object space are used as the calculation sub-parameters of the three-dimensional coordinates of the manipulator target posture; Using the multi-angle target space 3D coordinate parameters as the calculation sub-parameters of the robot arm's target posture 3D coordinates, analyze whether there is a conflict in the target space 3D coordinate parameters at different angles, and analyze whether there is a local 3D coordinate distortion phenomenon of the target space. Based on the analysis results, determine whether to eliminate interfering coordinates or correct shooting conditions; Step S4: determining the grasping posture of the robot arm according to the determined grasping posture calculation sub-parameters, and grasping the target object.

2. The intelligent factory voice and vision collaborative control method according to claim 1 is characterized in that: In step S2, determining the grasping difficulty tendency type according to the ratio of the number of feature points located in the feature area to the total number of feature points includes: Calculate the ratio of the number of feature points in the feature area to the total number of feature points to obtain the representation of the grasping difficulty. If the grasping difficulty representation is less than or equal to the preset grasping difficulty representation, determining that the grasping difficulty tendency type of the target object is a strong grasping difficulty tendency; If the grasping difficulty representation is greater than the preset grasping difficulty representation, it is determined that the grasping difficulty tendency type of the target object is a weak grasping difficulty tendency.

3. The intelligent factory voice and vision collaborative control method according to claim 2 is characterized in that: In step S3, the calculation parameters for the grasping posture of the target object are determined based on the determined grasping difficulty tendency type and the shape complexity of the target object, including: If the grasping difficulty tendency type of the target object is weak grasping difficulty tendency and the shape complexity of the target object is low, the single-angle target object space three-dimensional coordinate parameters are used as calculation sub-parameters for the three-dimensional coordinates of the manipulator target posture; If the grasping difficulty tendency type of the target object is a strong grasping difficulty tendency or the shape of the target object is highly complex, the multi-angle target space three-dimensional coordinate parameters are used as the calculation sub-parameters of the three-dimensional coordinates of the robot arm target posture, and it is analyzed whether there is a conflict in the target space three-dimensional coordinate parameters at different angles, and whether there is a local space three-dimensional coordinate distortion phenomenon of the target object. Based on the analysis results, it is determined to eliminate the interfering coordinates or correct the shooting conditions.

4. The intelligent factory voice and vision collaborative control method according to claim 1 is characterized in that: In step S3, the contour features of the target object are extracted based on the target object image, and the contour curvature is calculated to determine the shape complexity of the target object based on the contour curvature.

5. The intelligent factory voice and vision collaborative control method according to claim 3 is characterized in that: The analysis of whether there is a conflict in the three-dimensional coordinate parameters of the target object at different angles includes: Convert the coordinates to the same reference frame, Use hand-eye calibration to align the data. Select the key points of the target object, Calculate the coordinates of the same key point at each angle respectively. Calculate the Euclidean distance deviation, Determine the key points whose Euclidean distance deviation is greater than or equal to the preset Euclidean distance deviation, and obtain abnormal key points. Count the number of abnormal key points, If the number of abnormal key points is greater than or equal to the preset number, it is determined that there is a conflict in the three-dimensional coordinate parameters of the target space at different angles; If the number of the abnormal key points is less than the preset number, it is determined that there is no conflict in the three-dimensional coordinate parameters of the target object space at different angles.

6. The intelligent factory voice and vision collaborative control method according to claim 3 is characterized in that: The analysis of whether there is a local three-dimensional coordinate distortion phenomenon of the target object includes: Divide the target object into several local areas, Calculate the variance of the coordinates of a single local region, Calculate the average variance of the coordinates of each local area, Calculate the difference between the variance of the coordinates of a single local area and the average variance to obtain the distortion representation. Count the number of local areas whose distortion representation is greater than or equal to the preset distortion representation to obtain the number of distorted areas. If the number of distorted regions is greater than or equal to the preset number of distorted regions, it is determined that there is a local three-dimensional coordinate distortion phenomenon of the target object; If the number of the distorted regions is less than the preset number of distorted regions, it is determined that there is no local spatial three-dimensional coordinate distortion of the target object.

7. The intelligent factory voice and vision collaborative control method according to claim 3 is characterized in that: The determining of eliminating interference coordinates or correcting shooting conditions based on the analysis results includes: If there is a conflict in the three-dimensional coordinate parameters of the target object at different angles and there is a distortion phenomenon in the three-dimensional coordinates of the local space of the target object, the interference coordinates are determined to be excluded; If only the three-dimensional coordinate parameters of the target object at different angles conflict or only the three-dimensional coordinate distortion of the local space of the target object exists, the correction shooting condition is determined.

8. The intelligent factory voice and vision collaborative control method according to claim 5 is characterized in that: The elimination of interference coordinates includes: The abnormal key points and local areas whose distortion representation is greater than or equal to the preset distortion representation are eliminated.

9. The intelligent factory voice and vision collaborative control method according to claim 7 is characterized in that: The correction of shooting conditions includes: The shooting angle of view is increased to increase the three-dimensional coordinate parameters of the target object space, wherein the increase in the shooting angle of view is positively correlated with the number of the distortion areas.

10. A system using the intelligent factory voice and vision collaborative control method according to any one of claims 1 to 9, characterized in that: include: A voice command recognition module, which is used to recognize wake-up commands based on preset wake-up words and recognize voice commands based on preset command words; An image acquisition module, connected to the voice command recognition module, for acquiring an image of a target object; A feature point recognition module, connected to the voice command recognition module and the image acquisition module, is used to determine the feature points of the target object and determine the grasping difficulty tendency type of the target object based on the feature points; an analysis module, connected to the image acquisition module and the feature point recognition module, respectively, for determining, based on the target object's grasping difficulty tendency type, whether to use single-angle target object spatial three-dimensional coordinate parameters or multi-angle target object spatial three-dimensional coordinate parameters as calculation sub-parameters for the robot arm's target posture three-dimensional coordinates, analyzing whether there is a conflict between the target object spatial three-dimensional coordinate parameters at different angles, analyzing whether there is a local three-dimensional coordinate distortion phenomenon of the target object, and determining, based on the analysis results, to eliminate interfering coordinates or correct shooting conditions; A calculation module is connected to the image acquisition module, the feature point recognition module and the analysis module, and is used to calculate the grasping posture of the manipulator based on the calculation sub-parameters of the determined three-dimensional coordinates of the manipulator target posture.

Images