A multimodal data-driven fuzzy decision-making robot scheduling method

Abstract

The present application relates to a multimodal data-driven fuzzy decision robot scheduling method, including: obtaining image data and acoustic wave data of the workpiece surface; calculating the area information, the longest side length information and the category information of the defect on the workpiece surface based on the image data; performing a first preprocessing on the three types of information, and performing a first-level multidimensional stereo fuzzy decision to obtain a first decision result; calculating the depth information and position information of the defect based on the acoustic wave data; obtaining a stereo matrix based on the depth information, the position information and the first decision result; performing a second-level multidimensional stereo fuzzy decision on the stereo matrix to obtain a grinding result; determining the robot's grinding time based on the workpiece's grinding result, and obtaining the optimal robot grinding scheduling plan based on the grinding time and using a genetic algorithm. The method integrates multiple data of the workpiece to achieve intelligent monitoring and optimal scheduling of the robot's grinding process.

Description

Technical Field

[0001] The present application relates to the technical field of robot operation scheduling, and in particular to a multimodal data-driven fuzzy decision-making robot scheduling method. Background Art

[0002] At present, the application of robotic technology in modern manufacturing is becoming more and more widespread, especially in surface treatment processes such as grinding and polishing. Traditional robotic grinding usually relies on preset programs and fixed operation paths, and lacks the ability to perceive and adapt to the surface conditions of the workpiece in real time. However, the surface conditions of the workpiece often vary due to factors such as material properties, processing errors, and heat treatment processes. This requires the robotic grinding process to have higher intelligence and flexibility to ensure the consistency and efficiency of processing quality. Summary of the invention

[0003] Based on this, it is necessary to provide a multimodal data-driven fuzzy decision-making robot scheduling method, which includes:

[0004] S1: Acquire image data and acoustic wave data of the workpiece surface;

[0005] S2: Calculate the area information, the longest side length information and the defect category information of the workpiece surface based on the image data; perform a first preprocessing on the three types of information, and perform a first layer of multi-dimensional stereo fuzzy decision to obtain a first decision result;

[0006] S3: Calculating the depth information and position information of the defect based on the sound wave data; obtaining a stereo matrix based on the depth information, the position information and the first decision result; performing a second-layer multi-dimensional stereo fuzzy decision on the stereo matrix to obtain a polishing result;

[0007] S4: Determine the grinding time of the robot according to the grinding result of the workpiece, and obtain the optimal scheduling plan for the robot grinding based on the grinding time and using a genetic algorithm.

[0008] Beneficial effects: This method integrates multiple data of the workpiece, realizes intelligent monitoring and optimal scheduling of the robot grinding process, and provides strong technical support for the automation and intelligent upgrading of the manufacturing industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0010] Figure 1 This is a flowchart of a multimodal data-driven fuzzy decision-making robot scheduling method according to an embodiment of the present application. DETAILED DESCRIPTION

[0011] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present application, so the present application is not limited by the specific embodiments disclosed below.

[0012] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0013] like Figure 1 As shown, this embodiment provides a multimodal data-driven fuzzy decision-making robot scheduling method, the method comprising:

[0014] S1: Acquire image data and acoustic wave data of the workpiece surface.

[0015] In this embodiment, image data of the workpiece surface is acquired by a visual sensor, and acoustic wave data of the workpiece surface is acquired by an acoustic wave sensor.

[0016] In this embodiment, the vision sensor is installed above the path of the metal workpiece to ensure direct observation and capture of the image of the metal surface. For round or curved workpieces, polar coordinate transformation technology is used to make special adjustments to adapt to the shape of the workpiece.

[0017] The specific steps for using polar coordinate transformation technology to make special adjustments are as follows:

[0018] For visual inspection of circular or curved workpieces, polar coordinate conversion technology can convert images from Cartesian coordinates to polar coordinates to accommodate circular or curved features of the workpiece. In the polar coordinate system, the position of each point is determined by the radius. and angle Sure.

[0019] Here is the mathematical expression for converting from Cartesian coordinates to polar coordinates:

[0020] Determine the coordinates of a point in the Cartesian coordinate system , we want to find the corresponding polar coordinates Assuming the origin is the center of the image, the transformation formula is as follows:

[0021] ;

[0022] ;

[0023] in, From the origin to the point Straight-line distance; It is from the positive Axis to point The connection and positive The angle between the axes, in radians. Use function to calculate the angle, because Considering four quadrants, it is possible to and The sign of the angle is returned correctly.

[0024] In practical applications, the installation position of the vision sensor is determined based on the image features after polar coordinate conversion.

[0025] The acoustic wave sensor is installed on a mechanical device that moves along the metal surface in a non-contact manner to collect signals from multiple positions, making it suitable for complex detection environments. The distance between the installation position and the metal surface needs to consider the acoustic wave impedance matching to maximize the transmission efficiency of the acoustic wave and reduce energy loss.

[0026] The installation distance calculation process is as follows:

[0027] Determining the best installation location for an acoustic sensor involves multiple factors, including the type of sensor, the target to be detected, the desired degree of acoustic focusing, and ease of operation. In practical applications, this is usually determined through experimentation and optimization. The following key parameters can be considered:

[0028] 1. Distance of sound waves : Distance from sensor to metal surface.

[0029] 2. Wavelength of sound waves : The propagation length in the medium, given by the speed of sound and frequency Decision, that is .

[0030] 3. Attenuation of sound waves : Related to the properties of the medium, the frequency of the sound waves and the propagation distance.

[0031] 4. Acoustic impedance :The acoustic wave impedance of the sensor and metal affects the reflection and transmission of sound waves.

[0032] Based on the above parameters, the following factors can be considered:

[0033] Acoustic wave focusing: The focusing ability of the sensor determines the optimal working distance. Focusing distance The focal length of the sensor and the desired focus point position To determine, .

[0034] Sound wave attenuation: The energy loss of sound waves when propagating in a medium can be expressed as the attenuation constant. To describe the intensity of the sound wave It decreases with the increase of distance, that is ,in is the initial strength.

[0035] Acoustic wave impedance matching: Ideally, it is hoped that the reflection of sound waves between the sensor and the metal surface is minimized. This usually involves selecting an appropriate coupling agent or adjusting the material of the sensor to make the acoustic wave impedance close, which needs to be selected according to the actual application site.

[0036] Taking the above factors into consideration, the mathematical expression for determining the installation position of the acoustic wave sensor can be expressed as:

[0037] ;

[0038] in, It is the optimal sensor installation distance; The sound waves travel at a distance The energy at can be expressed as , The sound waves travel at a distance The cross-sectional area at is the detection threshold, which is the minimum energy that can produce an acceptable signal.

[0039] Please note that the above expression is a simplified model, and practical applications need to consider factors such as the geometry of the sensor, the shape and material of the workpiece, and environmental conditions. In addition, experimental adjustment and empirical optimization play a vital role in determining the optimal installation position. In practical applications, the installation position of the sensor needs to be determined through experiments and optimization to ensure that the best detection signal and minimum attenuation are obtained while satisfying the acoustic wave impedance matching.

[0040] S2: Calculate the workpiece surface area information, the longest side length information and the defect category information based on the image data; perform a first preprocessing on the three types of information, and perform a first layer of multi-dimensional stereo fuzzy decision to obtain a first decision result.

[0041] Specifically, the area information of the workpiece surface calculated based on the image data includes:

[0042] Step 1: Design an object calibration frame on the image data. The expression of the object calibration frame is:

[0043] ;

[0044] Among them, B represents the image in the object calibration frame; Represents the moving coordinates of the object calibration frame; represents the coordinates of the center point of the object calibration frame; w represents the width of the object calibration frame; h represents the height of the object calibration frame; represents the set of real numbers;

[0045] Step 2: Calculate the ratio between the pixel unit in the image data and the actual length unit of the corresponding workpiece to obtain a proportional factor; calculate the calculated width and calculated height of the workpiece surface based on the proportional factor and the width and height of the object calibration frame; the calculation formula is:

[0046] ;

[0047] ;

[0048] Wherein, W represents the calculated width of the workpiece surface; H represents the calculated height of the workpiece surface; b represents the scale factor;

[0049] The calculated width and height of the workpiece surface are optimized by minimizing the error function; the calculation formula is:

[0050] ;

[0051] ;

[0052] in, represents the calculated width and height of the workpiece surface after optimization; E(·) represents the error function; Indicates the actual width of the workpiece surface; Indicates the actual height of the workpiece surface;

[0053] Step 3: Based on the scale factor and the calculated width and height of the workpiece surface, calculate the area information of the workpiece surface; the calculation formula is:

[0054] ;

[0055] Wherein, a represents the area information of the workpiece surface.

[0056] The longest side length information of the workpiece surface includes:

[0057] Step 1: Get the rotation angle of the object calibration frame relative to the horizontal axis;

[0058] Step 2: Based on the rotation angle and the width and height of the object calibration frame, calculate the longest side length information; the calculation formula is:

[0059] ;

[0060] Among them, l represents the longest side length information; w represents the width of the object calibration frame; h represents the height of the object calibration frame; Indicates the rotation angle.

[0061] The type of defect information obtained on the workpiece surface includes:

[0062] Step 1: The image in the object calibration frame is subjected to a convolutional neural network to obtain a feature vector, and the calculation formula is:

[0063] ;

[0064] Where F represents the feature vector; f ( · ) represents a convolutional neural network; B represents the image in the object calibration frame;

[0065] Step 2: Use a pre-trained random forest classification model to classify the feature vector to obtain a defect category;

[0066] Step 3: Set a threshold value, and determine whether the feature vector belongs to a specific defect category based on the threshold value; the calculation formula is:

[0067] ;

[0068] in, represents a pre-trained random forest classification model; Indicates the threshold value; Indicates m a specific defect category;

[0069] This embodiment provides a training method for a random forest classification model, including:

[0070] Get the labeled training data set, recorded as: ;in, Represents the feature vector of the i-th sample; Represents the category label of the defect of the i-th sample;

[0071] Minimize the loss function to train the random forest classification model. The calculation formula is:

[0072] ;

[0073] ;

[0074] in, represents the random forest classification model before training; N is the total number of training samples; K is the number of categories; is a binary indicator (0 / 1) if sample i belongs to class k, then =1, otherwise =0; Representation Model Predict the probability that sample i belongs to category k.

[0075] In this expression, the goal is to find the model parameters The value of The loss function on minimize.

[0076] Step 4: De-noise the defect category and merge similar results to obtain category information of the defect.

[0077] Further, the first preprocessing of the three types of information includes:

[0078] Step 1: define the first index of area information, longest side length information and defect category information; the range of the first index is [1,5];

[0079] In this embodiment, for five categories, an index range from 1 to 5 is defined, and each category corresponds to an index. For example, the index IC of category C can be expressed as:

[0080] ;

[0081] Where Cmin is the minimum value of the category, Crange is the range of the category, and [ ] means rounding down.

[0082] For area and length, a similar method is used to define their indexes. Assuming that the minimum values ​​of area A and length L are Amin and Lmin respectively, and the ranges are Arange and Lrange respectively, the index IA of area and the index IL of length can be expressed as:

[0083] .

[0084] Step 2: construct a first three-dimensional matrix based on the first index of the three types of information; the first three-dimensional matrix includes three dimensions, which are category, area and length; each dimension includes five scales;

[0085] The specific index calculation method and value allocation method can be adjusted according to the characteristics of actual data and the needs of subsequent applications.

[0086] In this embodiment, each element M(i, j, k) in the first three-dimensional matrix can be expressed as:

[0087] ;

[0088] in, is a function that indexes the class I C , Area Index I A and length index I L to assign or calculate the corresponding value.

[0089] Step 3: assigning a first weight to each of the scales;

[0090] In this embodiment, for the first three-dimensional matrix, we have three dimensions: category (C), area (A), and length (L). Each dimension has five scales, and each scale is assigned a weight wC,k, wA,k, and wL,k, where k=1,2,3,4,5. The weights are based on prior knowledge or learned.

[0091] Step 4: Construct a three-dimensional convolution kernel K; the size of the three-dimensional convolution kernel is k C ×k A ×k L , k C is the size of the convolution kernel in the category dimension, k A is the size of the convolution kernel in the area dimension, k L is the size of the convolution kernel in the length dimension;

[0092] Step 5: Perform a convolution operation on the first three-dimensional matrix based on the three-dimensional convolution kernel, and the calculation formula is:

[0093] ;

[0094] Wherein, M represents the first three-dimensional matrix; (i, j, k) represents the position in the first three-dimensional matrix after convolution; (p, q, r) represents the position in the convolution kernel; K(p, q, r) represents the first weight of the convolution kernel at the position (p, q, r);

[0095] Step 6: Repeat step 5 to find out the position with the largest first weight in the convolution kernel. The calculation formula is:

[0096] ;

[0097] Determine the position coordinates corresponding to the first position with the largest weight in the convolution kernel in the first three-dimensional matrix ( , , ), the calculation formula is:

[0098] .

[0099] Furthermore, the process of performing the first-layer multi-dimensional stereo fuzzy decision to obtain the first decision result includes:

[0100] Step 1: define a fuzzy set, wherein the fuzzy set includes the membership of the output result; the output result includes five types, and each output result defines a membership function;

[0101] Step 2: defining a fuzzy rule based on the determined position coordinates in the first three-dimensional matrix, the fuzzy rule comprising:

[0102] When the position coordinates are on the top two inclined planes parallel to the main diagonal plane in the first three-dimensional matrix, the output result R 1 The degree of membership is the largest;

[0103] When the position coordinates are in the output result R 1 And the output result R 3 The output result is R 2 The degree of membership is the largest;

[0104] When the position coordinates are on the main diagonal plane and its two adjacent inclined planes in the first three-dimensional matrix, the output result R 3 The degree of membership is the largest;

[0105] When the position coordinates are in the output result R 3 And the output result R 5 The output result is R 4 The degree of membership is the largest;

[0106] When the position coordinates are on the two lowest inclined planes parallel to the main diagonal plane in the first three-dimensional matrix, the output result R 5 The degree of membership is the largest;

[0107] Step 3: Perform fuzzy reasoning based on the position coordinates and the fuzzy rules to obtain the total membership of each output result;

[0108] Step 4: The output result with the highest total membership is taken as the first decision result. The calculation formula is:

[0109] ;

[0110] Among them, μ Ri(max weight point) is the position of the maximum weight point for the output result R i The degree of membership.

[0111] S3: Calculate the depth information and position information of the defect based on the sound wave data; obtain a stereo matrix based on the depth information, the position information and the first decision result; perform a second-layer multi-dimensional stereo fuzzy decision on the stereo matrix to obtain a polishing result.

[0112] Specifically, the depth information and position information of the defect calculated based on the sound wave data includes:

[0113] Construct the wave equation, the expression is:

[0114] ;

[0115] in, represents the displacement of the sound wave at position x and time t; v represents the speed of sound in the medium; represents partial derivative; represents the vector calculus operator, used to represent the gradient, which is a vector pointing in the direction of the fastest growth of the function; in the wave equation, (Laplacian operator) is the divergence of the gradient, which indicates the magnitude of the gradient and measures the sum of the changes of the function in all directions of space.

[0116] The specific form of the Laplace operator in a three-dimensional rectangular coordinate system is:

[0117] ;

[0118] The wave equation is one of the basic equations that describes the propagation of sound waves, electromagnetic waves, and other types of waves in a medium. In acoustic testing, by solving this equation, the propagation behavior of sound waves in a medium can be predicted, and then the defect information inside the workpiece can be inferred.

[0119] Step 1: Calculate the sound wave data obtained at time t. The calculation formula is:

[0120] ;

[0121] in, Represents the sound wave data acquired at time t; represents the amplitude of the i-th reflected wave; represents the time function of the reflected wave; It represents the time delay of the reflected wave reaching the acoustic wave sensor; N is the number of reflected waves;

[0122] Step 2: Perform Fourier transform on the sound wave data to obtain the features; the calculation formula is:

[0123] ;

[0124] in, Indicates characteristics; represents the Fourier transform operation; f represents the frequency; j represents the imaginary unit;

[0125] Step 3: Determine the depth and position information of the defect based on the features.

[0126] In this embodiment, the position information P and depth information D of the defect are solved according to the features. This is an inverse problem and can be solved by optimization methods. Assuming that Φ(P,D) is an objective function that measures the difference between the reflection wave predicted by the model and the reflection wave actually observed, the optimization problem can be expressed as:

[0127] ;

[0128] ;

[0129] in, and is the optimal defect location and depth; N is the total number of observed data points; Represents the actual value of the i-th observation data point; It represents the predicted value of the i-th observation data point calculated by the prediction model at the defect position P and depth D.

[0130] In order to improve the accuracy of the solution, it is necessary to calibrate the model parameters such as the speed of sound v and the sensor response according to the experimental data. This is achieved by minimizing the difference between the predicted and observed data:

[0131] ;

[0132] in, and is the calibrated sound velocity and sensor response parameter, Φ obs is the observation data.

[0133] Further, a three-dimensional matrix is ​​obtained; including:

[0134] Step 1: Define 5 scales for depth information and position information respectively, and assign a second weight to each scale;

[0135] Step 2: Construct a stereo matrix based on the scale of the first decision result, the scale of the depth information, and the scale of the position information. Each element M'(i, j, k) in the stereo matrix can be expressed as:

[0136] ;

[0137] Among them, w C,i is the weight of the previous output, w D,j is the weight of the depth information, w P,k is the weight of the location information.

[0138] Furthermore, the stereo matrix is ​​subjected to a second-layer multi-dimensional stereo fuzzy decision, and the polishing results obtained include:

[0139] Step 1: subjecting the stereo matrix to the first preprocessing to determine the second position coordinates corresponding to the position with the second largest weight in the convolution kernel in the stereo matrix; the polishing results include light polishing, default polishing, and deep polishing;

[0140] Step 2: define a mapping rule, wherein the mapping rule is used to map the second position coordinate to the polishing result; the mapping rule includes:

[0141] When the second position coordinate is on the top three inclined planes parallel to the main diagonal plane in the three-dimensional matrix, the degree of membership of light grinding is the largest;

[0142] When the second position coordinate is on the main diagonal surface and its two adjacent inclined surfaces in the three-dimensional matrix, the default grinding membership is the largest;

[0143] When the second position coordinate is on the three lowest inclined planes parallel to the main diagonal plane in the three-dimensional matrix, the degree of membership of deep grinding is the largest;

[0144] Step 3: performing reasoning based on the second position coordinates and the mapping rule to obtain the total membership of each polishing result;

[0145] Step 4: The polishing result with the highest total membership is taken as the final polishing result.

[0146] S4: Determine the grinding time of the robot according to the grinding result of the workpiece, and obtain the optimal scheduling plan for the robot grinding based on the grinding time and using a genetic algorithm.

[0147] Specifically, the optimal scheduling solution for robot polishing includes:

[0148] Step 1: Define a function , map each polishing result to a non-negative real number, representing the corresponding polishing time;

[0149] ;

[0150] ;

[0151] ;

[0152] in, , and is the corresponding grinding time;

[0153] Step 2: Accumulate grinding time,

[0154] For each robot, the grinding time of all its grinding tasks is accumulated to determine the total working time of the robot. N is the number of robots, and a vector is defined , where ti is the total polishing time of the ith robot.

[0155] ;

[0156] in, is the output result of the jth polishing task of the ith robot, and M is the number of polishing tasks of the ith robot;

[0157] Step 3: Production scheduling optimization:

[0158] The production scheduling problem can be regarded as a scheduling problem, hoping to minimize the total working time of all robots. Define an optimization objective function J and use a genetic algorithm to find the optimal solution.

[0159] ;

[0160] Taking into account factors such as robot capabilities, task priorities, and inter-robot collaboration constraints, the optimization objective function of the production scheduling problem can be expressed as follows:

[0161] set up is the total working time of all robots, where ti is the working time of the i-th robot.

[0162] Objective function:

[0163] ;

[0164] Constraints:

[0165] 1. Robot capability constraints:

[0166] ;

[0167] in, is the capability of the ith robot, It is the minimum capability requirement of the robot.

[0168] 2. Task priority constraints:

[0169] ;

[0170] in, is the priority of task j.

[0171] 3. Constraints on collaboration between robots:

[0172] ;

[0173] in, is the degree of collaboration between robots i and j, is the maximum degree of collaboration.

[0174] 4. Path planning constraints:

[0175] ;

[0176] ;

[0177] in, is the distance that robot i moves to robot j is the maximum moving distance, It is the upper limit of the total moving distance.

[0178] 5. Time constraints:

[0179] ;

[0180] ;

[0181] in, and are the minimum and maximum time limits for robot i to complete the task.

[0182] 6. Non-negative constraints:

[0183] ;

[0184] With the above objective functions and constraints, a complex scheduling optimization model can be constructed, which aims to minimize the total working time of all robots while satisfying constraints such as robot capabilities, task priorities, collaboration capabilities, path planning, and time limits.

[0185] This embodiment uses an improved genetic algorithm for optimization. The expression of the improved genetic algorithm includes:

[0186] Fitness function improvement:

[0187] Design a fitness function , considering the robot’s capabilities, task priorities, collaboration capabilities, and path planning constraints:

[0188] ;

[0189] in, are weight coefficients, is the total working time, is the maximum allowed total working time, is the task priority satisfaction, is the robot collaboration efficiency, is the degree of violation of the path planning constraints.

[0190] Select Action Improvement:

[0191] Introduce a tournament selection based on fitness, select k individuals for comparison, and select the individual with the highest fitness as the candidate individual for the next generation:

[0192] .

[0193] Crossover operation improvements:

[0194] Use the uniform crossover operator to maintain the continuity and order of task allocation:

[0195] .

[0196] Mutation operation improvements:

[0197] Introducing local search mutation not only randomly changes certain genes, but also explores the neighborhood of the solution space through local search:

[0198] .

[0199] Elite Strategy:

[0200] In order to ensure that excellent individuals can be passed on to the next generation, an elite strategy is introduced to directly copy the individuals with the highest fitness in the current population to the next generation:

[0201] ;

[0202] in, It is the individual with the highest fitness in the current population.

[0203] Iteration update and termination conditions:

[0204] In each iteration, a new population X' is generated through selection, crossover and mutation operations, and the new fitness f(X') is calculated. If the termination condition is met (the maximum number of iterations N is reached), iter Or find a solution X that satisfies the condition target ), then output the current optimal solution X best ; Otherwise, update the population and continue iterating.

[0205] Through the above improvements, the new genetic algorithm will be more suitable for solving scheduling problems. It can effectively search the solution space and find better robot polishing scheduling solutions while satisfying complex constraints.

[0206] The method provided in this embodiment integrates multiple data of the workpiece, realizes intelligent monitoring and optimal scheduling of the robot grinding process, and provides strong technical support for the automation and intelligent upgrading of the manufacturing industry. With the continuous advancement and improvement of related technologies, this method will play an important role in more fields in the future and promote the development of the manufacturing industry to a higher level.

[0207] The method also has adaptive learning capabilities; as the number of workpieces processed increases, its grinding strategy becomes more and more precise and can adapt to more complex and changing working environments; this adaptive capability not only improves grinding quality, but also significantly improves production efficiency and reduces labor costs and material waste.

[0208] In practical applications, this method can be widely used in high-precision processing fields such as aerospace, automobile manufacturing, and medical equipment. It can not only improve product quality, but also shorten the production cycle, bringing significant economic benefits to the enterprise.

[0209] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0210] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

Claims

1. A multimodal data driven fuzzy decision robot scheduling method, characterized in that: include: S1: Acquire image data and acoustic wave data of the workpiece surface; S2: Calculating the area information, the longest side length information and the defect category information of the workpiece surface based on the image data; Performing a first preprocessing on the three types of information, and performing a first-layer multi-dimensional stereoscopic fuzzy decision, to obtain a first decision result; The first preprocessing of the three types of information includes: Step 1: define the first index of area information, longest side length information and defect category information; the range of the first index is [1,5]; Step 2: construct a first three-dimensional matrix based on the first index of the three types of information; the first three-dimensional matrix includes three dimensions, which are category, area and length; each dimension includes five scales; Step 3: assigning a first weight to each of the scales; Step 4: Construct a three-dimensional convolution kernel K; the size of the three-dimensional convolution kernel is k C ×k A ×k L , k C is the size of the convolution kernel in the category dimension, k A is the size of the convolution kernel in the area dimension, k L is the size of the convolution kernel in the length dimension; Step 5: Perform a convolution operation on the first three-dimensional matrix based on the three-dimensional convolution kernel, and the calculation formula is: ； Wherein, M represents the first three-dimensional matrix; (i, j, k) represents the position in the first three-dimensional matrix after convolution; (p, q, r) represents the position in the convolution kernel; K(p, q, r) represents the first weight of the convolution kernel at the position (p, q, r); Step 6: Repeat step 5 to query the position with the largest first weight in the convolution kernel, and determine its corresponding position coordinates in the first three-dimensional matrix; The process of performing the first-layer multi-dimensional stereo fuzzy decision to obtain the first decision result includes: Step 1: define a fuzzy set, wherein the fuzzy set includes the membership of the output result; the output result includes five types, and each output result defines a membership function; Step 2: defining a fuzzy rule based on the determined position coordinates in the first three-dimensional matrix, the fuzzy rule comprising: When the position coordinates are on the two uppermost inclined planes parallel to the main diagonal planes in the first three-dimensional matrix, the membership degree of the output result R1 is the largest; When the position coordinate is on the slope between the determination slopes of the output result R1 and the output result R3, the membership degree of the output result R2 is the largest; When the position coordinates are on the main diagonal plane and its two adjacent inclined planes in the first three-dimensional matrix, the membership degree of the output result R3 is the largest; When the position coordinate is on the slope between the determination slopes of the output result R3 and the output result R5, the membership degree of the output result R4 is the largest; When the position coordinates are on the two lowest inclined planes parallel to the main diagonal planes in the first three-dimensional matrix, the membership degree of the output result R5 is the largest; Step 3: Perform fuzzy reasoning based on the position coordinates and the fuzzy rules to obtain the total membership of each output result; Step 4: taking the output result with the highest total membership as the first decision result; S3: Calculating the depth information and position information of the defect based on the sound wave data; obtaining a stereo matrix based on the depth information, the position information and the first decision result; performing a second-layer multi-dimensional stereo fuzzy decision on the stereo matrix to obtain a polishing result; Get the stereo matrix, including: Step 1: Define 5 scales for depth information and position information respectively, and assign a second weight to each scale; Step 2: constructing a stereo matrix based on the scale of the first decision result, the scale of the depth information, and the scale of the position information; The stereo matrix is ​​subjected to a second-layer multi-dimensional stereo fuzzy decision, and the polishing results obtained include: Step 1: preprocessing the stereo matrix to determine the second position coordinates corresponding to the position with the second largest weight in the convolution kernel in the stereo matrix; the polishing results include light polishing, default polishing, and deep polishing; Step 2: define a mapping rule, wherein the mapping rule is used to map the second position coordinate to the polishing result; the mapping rule includes: When the second position coordinate is on the top three inclined planes parallel to the main diagonal plane in the three-dimensional matrix, the degree of membership of light grinding is the largest; When the second position coordinate is on the main diagonal surface and its two adjacent inclined surfaces in the three-dimensional matrix, the default grinding membership is the largest; When the second position coordinate is on the three lowest inclined planes parallel to the main diagonal plane in the three-dimensional matrix, the degree of membership of deep grinding is the largest; Step 3: performing reasoning based on the second position coordinates and the mapping rule to obtain the total membership of each polishing result; Step 4: The polishing result with the highest total membership degree is taken as the final polishing result; S4: Determine the grinding time of the robot according to the grinding result of the workpiece, and obtain the optimal scheduling plan for the robot grinding based on the grinding time and using a genetic algorithm.

2. The fuzzy decision robot scheduling method according to claim 1, characterized in that: In S1, image data of the workpiece surface is obtained by a visual sensor, and acoustic wave data of the workpiece surface is obtained by an acoustic wave sensor.

3. The fuzzy decision robot scheduling method according to claim 1, characterized in that: In S2, the area information of the workpiece surface calculated based on the image data includes: Step 1: Design an object calibration frame on the image data. The expression of the object calibration frame is: ； Among them, B represents the image in the object calibration frame; Represents the moving coordinates of the object calibration frame; represents the coordinates of the center point of the object calibration frame; w represents the width of the object calibration frame; h represents the height of the object calibration frame; represents the set of real numbers; Step 2: Calculate the ratio between the pixel unit in the image data and the actual length unit of the corresponding workpiece to obtain a proportional factor; calculate the calculated width and calculated height of the workpiece surface based on the proportional factor and the width and height of the object calibration frame; the calculation formula is: ； ； Wherein, W represents the calculated width of the workpiece surface; H represents the calculated height of the workpiece surface; b represents the scale factor; The calculated width and height of the workpiece surface are optimized by minimizing the error function; the calculation formula is: ； ； in, represents the calculated width and height of the workpiece surface after optimization; E(·) represents the error function; Indicates the actual width of the workpiece surface; Indicates the actual height of the workpiece surface; Step 3: Based on the scale factor and the calculated width and height of the workpiece surface, calculate the area information of the workpiece surface; the calculation formula is: ； Wherein, a represents the area information of the workpiece surface.

4. The fuzzy decision robot scheduling method according to claim 3 is characterized in that: In S2, the longest side length information of the workpiece surface is obtained including: Step 1: Get the rotation angle of the object calibration frame relative to the horizontal axis; Step 2: Based on the rotation angle and the width and height of the object calibration frame, calculate the longest side length information; the calculation formula is: ； Among them, l represents the longest side length information; w represents the width of the object calibration frame; h represents the height of the object calibration frame; Indicates the rotation angle.

5. The fuzzy decision robot scheduling method according to claim 3, characterized in that: In S2, the category information of the defects on the workpiece surface is obtained including: Step 1: The image in the object calibration frame is subjected to a convolutional neural network to obtain a feature vector, and the calculation formula is: ； Where F represents the feature vector; f ( · ) represents a convolutional neural network; B represents the image in the object calibration frame; Step 2: Use a pre-trained random forest classification model to classify the feature vector to obtain a defect category; Step 3: Set a threshold value, and determine whether the feature vector belongs to a specific defect category based on the threshold value; the calculation formula is: ； in, represents a pre-trained random forest classification model; Indicates the threshold value; Indicates m a specific defect category; Step 4: De-noise the defect category and merge similar results to obtain category information of the defect.

6. The fuzzy decision robot scheduling method according to claim 1, characterized in that: The depth information and position information of the defect calculated based on the sound wave data includes: Step 1: Calculate the sound wave data obtained at time t. The calculation formula is: ； in, Represents the sound wave data acquired at time t; represents the amplitude of the i-th reflected wave; represents the time function of the reflected wave; It represents the time delay of the reflected wave reaching the acoustic wave sensor; N is the number of reflected waves; Step 2: Perform Fourier transform on the sound wave data to obtain the features; the calculation formula is: ； in, Indicates characteristics; represents the Fourier transform operation; f represents the frequency; j represents the imaginary unit; Step 3: Determine the depth and position information of the defect based on the features.

Images