Automatic hole searching positioning method and system based on machine vision

Abstract

The invention discloses an automatic hole searching and positioning method and system based on machine vision, and particularly relates to the technical field of automation control, and the method comprises the steps: S101, obtaining and preprocessing the image or video data of a surrounding environment, and carrying out the preliminary analysis of the image through a simulation model; s102, performing cooperative training on the robots, and performing hole site detection and segmentation on the preprocessed image data through the trained robots to obtain hole site candidate areas; according to the method, the influence of noise and shielding on a detection result can be reduced, different scenes can be adapted, the adaptability of the whole system is improved, data privacy and safety are effectively guaranteed, model parameters can be more quickly converged to a better solution, and the training time is shortened; the decision accuracy is effectively improved, invalid or low-efficiency actions are reduced, the execution speed of hole searching, charging and blasting tasks is increased, the positioning deviation caused by error accumulation is reduced, the operation precision is improved, and the uncertainty caused by emergencies is avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic control, and particularly to an automatic hole-searching and positioning method and system based on machine vision. Background Art

[0002] With the rapid development of fields such as intelligent manufacturing, engineering blasting, and drilling operations, the demand for precise positioning and autonomous navigation of robots is increasing day by day. In application scenarios such as mine blasting, engineering drilling, and automatic charging, accurately positioning the target hole is the core link of the entire operation process, directly affecting the safety and operation efficiency of subsequent operations. However, due to the complex environment, variable target hole morphology, and the influence of lighting conditions, traditional target detection methods are difficult to meet the requirements of high precision and real-time performance.

[0003] In the existing automatic hole-searching and positioning methods and systems, the influence of noise and occlusion on the detection results is relatively large, and the adaptability of the overall system is relatively low, and data privacy and security cannot be guaranteed; in addition, the decision-making accuracy of the existing automatic hole-searching and positioning methods and systems is low, there are many invalid or inefficient actions, reducing the execution speed of hole-searching, charging, and blasting tasks, and easily leading to positioning deviations caused by error accumulation; for this reason, we propose an automatic hole-searching and positioning method and system based on machine vision. Summary of the Invention

[0004] The object of the present invention is to solve the problems that the influence of noise and occlusion on the detection results is relatively large, the adaptability of the overall system is relatively low, data privacy and security cannot be guaranteed, the decision-making accuracy is low, there are many invalid or inefficient actions, reducing the execution speed of hole-searching, charging, and blasting tasks, and easily leading to positioning deviations caused by error accumulation, and to propose an automatic hole-searching and positioning method and system based on machine vision.

[0005] In the first aspect of the implementation of the present invention, firstly, an automatic hole-searching and positioning method and system based on machine vision are proposed, and the method includes:

[0006] S101: Obtain and preprocess the image or video data of the surrounding environment, and perform a preliminary analysis of the image using a simulation model;

[0007] S102: Co-train each robot, and use each trained robot to perform hole position detection and segmentation on the preprocessed image data to obtain hole position candidate regions;

[0008] S103: Perform automatic hole-searching and positioning according to the obtained groups of hole position candidate regions, and optimize the search path generated during the hole-searching and positioning process;

[0009] S104: Based on the positioning information of the current target hole position and the optimized search path, make decision optimization according to the current environmental state and historical data to guide the blasting execution.

[0010] As a further solution of the present invention, the specific steps of acquiring and preprocessing the image or video data of the surrounding environment in S101 include:

[0011] P1.1: Select a vision sensor according to the actual application scenario, calibrate the selected vision sensor to determine its internal parameters and external parameters, then control the robot to move along a predetermined path, and simultaneously start the vision sensor to collect images, and store the collected image data and video data, and associate them with the pose information of the robot. The vision sensor specifically includes a laser scanner, a high-definition camera, a depth camera, and a multispectral camera;

[0012] P1.2: Decompose the collected video data into image data frame by frame, remove the noise in each group of image data through Gaussian filtering, then use the histogram equalization method to correct the brightness and contrast in each group of image data. After the correction is completed, use the Laplace filtering algorithm to enhance the edge information in the image data, and then scale the pixel values in each preprocessed image data to a preset range through the MAX-MIN normalization method.

[0013] As a further solution of the present invention, the specific steps of using the simulation model to perform a preliminary analysis on the image in S101 include:

[0014] P2.1: Based on the calibrated sensor data and environmental parameters, determine the current environmental parameters, the shape and size of the target object, and perform geometric modeling on the target object according to each group of data, and then simulate the illumination influence of the illumination in the image data on the display effect of the target object through the illumination model;

[0015] P2.2: Simulate the sensor response according to the resolution, field of view angle, focal length of the used vision sensor and the characteristics of the sensor noise, and then integrate the simulation results of various environmental factors, sensor characteristics and target geometric characteristics to establish a complete simulation model, and generate a target simulation image through this simulation model;

[0016] P2.3: Compare the target simulation image generated by the simulation model with the actually collected image data, calculate the structural similarity index between the target simulation image and the actual image data, and count the target objects in the actual image data whose structural similarity index is lower than the preset threshold as the candidate areas for target preliminary positioning.

[0017] Among them, the specific calculation formula of the geometric modeling in P2.1 is as follows:

[0018] TG(xTG , y TG ) = f TG (x TG , y TG , R TG , D TG , α TG )

[0019] Among them, TG(x TG , y TG ) represents the geometric shape of the target object at the position (x TG , y TG ); R TG represents the radius of the target object; D TG represents the depth of the target object; α TG represents the rotation angle of the target;

[0020] The specific calculation formula of the structural similarity index described in P2.3 is as follows:

[0021]

[0022] Among them, SSIM(x SSIM , y SSIM ) represents the structural similarity index between the actual image I(x SSIM , y SSIM ) and the target simulation image G(x SSIM , y SSIM ); μ I represents the average value of the actual image I(x SSIM , y SSIM ); μ G represents the average value of the target simulation image G(x SSIM , y SSIM ); represents the variance of the actual image I(x SSIM , y SSIM ); represents the variance of the target simulation image G(x SSIM , y SSIM ); σ IG represents the covariance between the actual image I(x SSIM , y SSIM ) and the target simulation image G(x SSIM , y SSIM ); c1 and c2 respectively represent constants.

[0023] As a further solution of the present invention, the specific steps of the collaborative training of each robot in S102 include:

[0024] P3.1: After each robot preprocesses the image data collected by the vision sensor, it stores the preprocessed groups of image data at the local end of each robot, and adds corresponding target positions and category labels to each group of image data to generate independent local datasets. The central server constructs an initial hybrid object detection model based on the WGAN model and the SSD model, and uses it as the global model;

[0025] P3.2: The central server sends the hybrid object detection model to each robot end. The robot end inputs the local dataset into the hybrid object detection model, and transmits the local data to the WGAN layer through forward propagation. The generator in the WGAN layer generates target images based on the input noise and environmental data. Then, the discriminator in the WGAN layer compares the received local data with the target boxes generated by the generator, and determines whether the generated target boxes match the real data in the training set. Then, the loss values of the generator and the discriminator are calculated respectively according to the matching results;

[0026] P3.3: Then, the local data is transmitted to the SSD layer. The SSD layer classifies and regresses the target boxes in the local data through multi-scale convolutional feature maps, and calculates the classification loss and the regression loss through the cross-entropy loss and the smooth L1 loss respectively. Then, the total loss of the SSD object detection is obtained based on the classification loss and the regression loss;

[0027] P3.4: Combine the generator loss value, the discriminator loss value, and the total loss of the SSD object detection to obtain the total loss of the hybrid object detection model, and input it from the output layer of the hybrid object detection model. Propagate backward according to the chain rule, and at the same time calculate the gradient values of the total loss for the parameters of each network layer of the hybrid object detection model. Use the Adam optimizer to optimize the parameters of each network layer;

[0028] P3.5: After the training of the hybrid object detection model is completed on each robot end, use the local data that has not participated in the training as verification data to calculate various indicators such as the accuracy and recall rate of the model. If the indicators of the hybrid object detection model do not reach the preset threshold, retrain, and repeatedly train and verify the hybrid object detection model until the total loss of the model converges to the preset threshold;

[0029] P3.6: After a round of training, each robot end sends the updated model parameters to the central server. The central server performs weighted averaging on the model parameters of all robot ends to obtain a global model, and re-optimizes the parameters of the obtained global model, and then sends the model back to each device for the next round of training. Repeatedly perform local training and global training of the hybrid object detection model until the loss value in the global training of the hybrid object detection model converges to the preset threshold. Then, send the trained hybrid object detection model to each robot end.

[0030] As a further solution of the present invention, the specific steps of re-optimizing the parameters of the obtained global model described in P3.6 include:

[0031] P4.1: Use the aggregated global model as the initial model parameters, collect all combinations of model parameters, use the initial model parameters as the initial state of optimization, then initialize the qubit state, that is, the uniform superposition state of all parameter combinations, and construct a quantum Hamiltonian according to the set optimization objective;

[0032] P4.2: Control the state change of the initial qubit based on the time-dependent Schrödinger equation, generate multiple groups of new parameter combinations, calculate the quantum Hamiltonian of each group of new parameter combinations to obtain the performance of the global model under the new parameter combinations, and at the same time, based on the model performance under the new parameter combinations, calculate the difference in model performance between the new parameter combinations and the current parameter combinations;

[0033] P4.3: According to the calculated performance difference, use the Metropolis criterion to select whether to accept the new parameter combination. If the performance difference is less than 0, automatically accept the new solution. Otherwise, calculate the acceptance probability of the new parameter. If it is greater than the random number, accept it. Otherwise, keep the current parameter combination;

[0034] P4.4: Repeatedly generate new parameter combinations, calculate the performance difference, and update the parameter combinations until the preset termination time is reached, and output the current parameter combination. At the same time, update the parameters of the global model and wait for the next round of global aggregation to re-optimize the parameters of the global model again.

[0035] Among them, the specific manifestation of initializing the qubit state described in P4.1 is as follows:

[0036]

[0037] In the formula, |ψ(0)> represents the initial quantum state, indicating the uniform superposition of all parameter combinations; N Q represents the number of qubits; x Q represents the model parameter combination, that is, the parameter space of the global model;

[0038] The specific calculation formula of the quantum Hamiltonian described in P4.1 is as follows:

[0039]

[0040]

[0041] H(t QH ) = A(t QH )H B +B(t QH )H P

[0042] In the formula, H(t QH ) represents the quantum Hamiltonian; H B represents the driving Hamiltonian, represents the i QH -th qubit corresponding to the Pauli matrix of the model parameter combination x QH ; H P represents the target Hamiltonian; represents the loss value of the global model at the parameter ; A(t QH ) and B(t QH ) represent time factors. When A(0) >> B(0) at the initial moment, the driving Hamiltonian is dominant, and during the iteration process, A(t QH ) gradually decreases, and B(t QH ) gradually increases.

[0043] In the second aspect of the implementation of the present invention, an automatic hole-finding and positioning system based on machine vision is proposed, including:

[0044] The acquisition and processing module is used to obtain real-time image data from the scene and perform image processing after image acquisition;

[0045] The feature extraction module is used to extract key features from the preprocessed image and generate an image feature map;

[0046] The detection and positioning module is used to identify the holes in the image and determine their specific positions, and generate information such as the coordinates, sizes, and angles of the holes;

[0047] The 3D reconstruction module is used to convert the positions and shapes of the holes into coordinates in a 3D model according to the detected 2D image information;

[0048] The motion planning module is used to plan the movement path for the robot or automation equipment, and at the same time, in combination with the kinematic model of the robot, calculate the optimal motion trajectory;

[0049] The execution control module controls the robot to perform precise positioning and task execution according to the instructions received from the motion planning module;

[0050] The feedback correction module is used to monitor the positioning error and execution process of the robot in real time and dynamically adjust the motion trajectory and operation strategy of the robot;

[0051] The storage and analysis module is responsible for all data storage, management, and analysis tasks during the automatic hole-finding and positioning process, and at the same time uses data analysis technology to evaluate the operating conditions of the entire system;

[0052] The anomaly detection module is used to monitor various indicators of the automatic hole-searching and positioning system, and at the same time detect and identify possible faults or abnormal conditions.

[0053] As a further solution of the present invention, the specific steps for the motion planning module to calculate the optimal motion trajectory in combination with the kinematic model of the robot are as follows:

[0054] P5.1: The motion planning module receives the output data of the detection and positioning module, discretizes the three-dimensional environmental map model into grids of a preset size, each grid cell represents a position, and at the same time makes the starting position and the target position of the robot correspond to a grid cell, and marks the immovable grids according to the environmental information;

[0055] P5.2: Initialize the open list and the closed list, where the open list includes all nodes to be investigated, and the closed list includes all nodes that have been evaluated. When initializing, the open list contains the starting point and the closed list is empty. Calculate the actual cost, estimated cost, and total cost of each node in the open list. The specific calculation formula for the actual cost is as follows:

[0056]

[0057] g(x neighbor ) = g(x current ) + d(x current , x neighbor )

[0058]

[0059] f(x TC ) = g(x neighbor ) + h(x heuristics )

[0060] In the formula, d(x current , x neighbor ) represents the cost from the current node x current to the adjacent node x neighbor ; represents the coordinates of the current node x current ; represents the coordinates of the adjacent node x neighbor ; g(x neighbor ) represents the actual cost from the current node x current to the adjacent node x neighbor ; g(x current ) represents the actual cost from the starting point to the current node x current ; h(x heuristics ) represents the estimated cost from the current node x current to the target node; represent the coordinates of the target node; f(x TC ) represents the total cost;

[0061] P5.3: Check the 8 nodes adjacent to the current node. If the adjacent node is an obstacle or already in the closed list, ignore the node. If the adjacent node is not in the open list, add it to the open list and calculate its various cost values. After the check is completed, recalculate the total cost of each feasible adjacent node of the current node;

[0062] P5.4: If the total cost of the recalculated adjacent node is less than the total cost of the original node in the open list, update the various cost values of the corresponding node in the open list, and use the adjacent node with the minimum total cost as the next moving node. Then update the current node, and add the node before the movement to the closed list. Gradually select until the robot reaches the target node, and then backtrack the node information in the closed list to obtain the complete execution path of the robot;

[0063] P5.5: Take the obtained initial path as a group of individuals in the population. Perform mutation and crossover operations on the initial path to generate multiple new path individuals. Then calculate the fitness values of each group of paths in the population according to the path length, motion smoothness, and obstacle avoidance ability. Use the roulette wheel selection method to calculate the selection probabilities of each path individual in the population, and use the path individuals with selection probabilities higher than the preset threshold as the parent path individuals;

[0064] P5.6: Copy the selected parent path individuals, and randomly select two groups of parent path individuals. Through the crossover operation, randomly exchange one or two groups of nodes in the two groups of parent path individuals to generate new path individuals. Then adjust the copied parent path individuals by randomly changing to produce new path individuals. Then calculate the fitness values of the new path individuals, screen out the path individuals below the preset threshold, and form a new population with the remaining path individuals and the parent path individuals;

[0065] P5.7: Record the paths, fitness values, and interactions with the environment in each iteration process. Then repeat the steps of parent path individual selection, crossover, mutation, and new population generation until the change of the population fitness value converges to the preset range. Traverse the fitness values of each group of path individuals in the final population, and select the path individual with the highest fitness value as the final execution path. Send the execution path to the corresponding robot, and control the robot to perform positioning and task execution through the execution control module.

[0066] As a further solution of the present invention, the specific steps for the feedback correction module to dynamically adjust the motion trajectory and operation strategy of the robot are as follows:

[0067] P6.1: The feedback correction module collects various state information when the robot performs operations and establishes a state space Among them, s i represents the state of the robot, i robot ∈N robot , and according to the environmental data obtained by the robot at different times, an observation space is established Based on the state space and the observation space, a transition probability matrix and an observation probability matrix are established respectively;

[0068] P6.2: Collect the historical environmental data of the robot. Each set of data includes the observed values corresponding to each state. The feedback correction module initializes the initial state probability, the state transition matrix, and the observation probability matrix according to the historical data. Then, at the initial time t robot = 1, calculate the forward probability according to the initial state probability and the observation probability, and obtain the forward probability of subsequent times through recursive calculation to get the probability distribution of the robot in different states at each time. The specific calculation formula for forward propagation is as follows:

[0069]

[0070] In the formula, α1(i FP ) represents the probability that the robot is in state and observes o1 at time 1; represents the initial state probability; represents the probability that the robot observes o1 in state ; represents the probability that the robot is in state robot at time t and observes the first t robot observed values o1, o2, K, ; represents the probability that the robot is in state and observes the first t robot -1 observed values o1, o2, K, ; represents the probability that the robot transfers from state to ; represents the probability that the robot observes in state ;

[0071] P6.3: According to the probability distribution of the robot in different states at each time, use the Viterbi algorithm to find the state robot in which the robot is at time t and observes the observed values o1, o2, K, The maximum probability is used to obtain the optimal state path at each moment. After calculating the optimal path probabilities for all moments, it continues until a preset termination moment is reached;

[0072] P6.4: Starting from the termination moment, trace back to find the optimal state sequence to obtain the optimal operations at each moment. Then, according to the inference result, control the robot to select the optimal operation, and adjust the robot's next operation plan in real time based on the path planning result, historical operation selection, and current environmental state.

[0073] Advantages of the present invention:

[0074] The present invention proposes an automatic hole-searching and positioning method and system based on machine vision. The image is trained through a target recognition method. Then, each robot locally optimizes the model using its own collected data and periodically sends the training results to the central node. After the central node receives the information provided by multiple robots, it integrates their respective optimization results to form a more adaptable global model. Subsequently, the global model is further adjusted. Finally, the optimized model is distributed to each robot, reducing the influence of noise and occlusion on the detection result, ensuring adaptability to different scenarios, improving the adaptability of the overall system, effectively protecting data privacy and security, enabling the model parameters to converge to a better solution faster, and reducing the training time.

[0075] The present invention proposes an automatic hole-searching and positioning method and system based on machine vision. After accurately positioning the target hole and planning the path, it comprehensively analyzes the possibilities of different operation options based on the current environmental information and past execution experience, and predicts the optimal next action plan by calculating the influence and success rate of each option. Then, corresponding control instructions are sent to the robot through the execution control model, effectively improving the decision-making accuracy, reducing ineffective or inefficient actions, accelerating the execution speed of hole-searching, charging, and blasting tasks, reducing the positioning deviation caused by error accumulation, improving the operation accuracy, and avoiding the uncertainty brought by unexpected situations. Description of the Drawings

[0076] The following further describes the present invention with reference to the drawings.

[0077] Figure 1 It is a flowchart of an automatic hole-searching and positioning method based on machine vision provided by an embodiment of the present invention;

[0078] Figure 2 It is a framework diagram of an automatic hole-searching and positioning system based on machine vision provided by an embodiment of the present invention. Detailed Embodiments

[0079] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0080] All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0081] The embodiments of the present invention provide an automatic hole searching and positioning method and system based on machine vision. Refer to Figure 1 , Figure 1 which is a flowchart of an automatic hole searching and positioning method based on machine vision provided by the embodiments of the present invention. The method includes the following steps:

[0082] Obtain and preprocess the image or video data of the surrounding environment, and perform a preliminary analysis of the image using a simulation model.

[0083] Specifically, select a vision sensor according to the actual application scenario, calibrate the selected vision sensor to determine its internal parameters and external parameters, then control the robot to move along a predetermined path, and synchronously start the vision sensor to collect images. At the same time, store the collected image data and video data, and associate them with the pose information of the robot. The vision sensor specifically includes a laser scanner, a high-definition camera, a depth camera, and a multispectral camera. Frame by frame, decompose the collected video data into image data, remove the noise in each group of image data through Gaussian filtering, and then use the histogram equalization method to correct the brightness and contrast in each group of image data. After the correction is completed, use the Laplacian filtering algorithm to enhance the edge information in the image data. Then, scale the pixel values in each preprocessed image data to a preset range through the MAX-MIN normalization method.

[0084] Specifically, based on the calibrated sensor data and environmental parameters, determine the current environmental parameters, the shape and size of the target object, and perform geometric modeling on the target object according to each group of data. Then, simulate the lighting effect of the lighting in the image data on the display effect of the target object through the lighting model. According to the resolution, field of view angle, focal length, and noise characteristics of the used vision sensor, simulate the sensor response. After that, integrate the simulation results of various environmental factors, sensor characteristics, and target geometric characteristics to establish a complete simulation model, and generate a target simulation image through this simulation model. Compare the target simulation image generated by the simulation model with the actually collected image data, calculate the structural similarity index between the target simulation image and the actual image data, and count the target objects in the actual image data whose structural similarity index is lower than the preset threshold as the candidate areas for target preliminary positioning.

[0085] In this embodiment, the specific calculation formula for geometric modeling is as follows:

[0086] TG(x TG ,y TG )=f TG (x TG ,y TG ,R TG ,D TG ,α TG )

[0087] Among them, TG(x TG ,y TG ) represents the geometric shape of the target object at the position (x TG ,y TG ); R TG represents the radius of the target object; D TG represents the depth of the target object; α TG represents the rotation angle of the target;

[0088] The specific calculation formula for the structural similarity index is as follows:

[0089]

[0090] Among them, SSIM(x SSIM ,y SSIM ) represents the structural similarity index between the actual image I(x SSIM ,y SSIM ) and the target simulation image G(x SSIM ,y SSIM ); μ I represents the average value of the actual image I(x SSIM ,y SSIM ); μ G represents the average value of the target simulation image G(x SSIM ,ySSIM )'s average value; represents the actual image I(x SSIM , y SSIM )'s variance; represents the target simulation image G(x SSIM , y SSIM )'s variance; σ IG represents the actual image I(x SSIM , y SSIM ) and the target simulation image G(x SSIM , y SSIM )'s covariance; c1 and c2 respectively represent constants.

[0091] Conduct collaborative training on each robot, and use the trained robots to perform hole position detection and segmentation on the preprocessed image data to obtain hole position candidate regions.

[0092] Specifically, after each robot preprocesses the image data collected by the vision sensor, it stores the preprocessed groups of image data at the local end of each robot, and adds corresponding target positions and class labels to each group of image data to generate independent local datasets. The central server constructs an initial hybrid object detection model based on the WGAN model and the SSD model, and uses it as the global model. The central server distributes the hybrid object detection model to each robot end. The robot end inputs the local dataset into the hybrid object detection model, and transmits the local data to the WGAN layer through forward propagation. The generator in the WGAN layer generates a target image based on the input noise and environmental data. Then, the discriminator in the WGAN layer compares the received local data with the target boxes generated by the generator, and determines whether the generated target boxes match the real data in the training set. Then, according to the matching results, the loss values of the generator and the discriminator are calculated respectively. Then, the local data is transmitted to the SSD layer. The SSD layer classifies and regresses the target boxes in the local data through multi-scale convolutional feature maps, and calculates the classification loss and the regression loss through the cross-entropy loss and the smooth L1 loss respectively. Then, based on the classification loss and the regression loss, the total loss of the SSD object detection is obtained. The generator loss value, the discriminator loss value, and the total loss of the SSD object detection are combined to obtain the total loss of the hybrid object detection model, and it is input from the output layer of the hybrid object detection model. According to the chain rule, backpropagation is performed, and at the same time, the gradient values of the total loss with respect to the parameters of each network layer of the hybrid object detection model are calculated. The Adam optimizer is used to optimize the parameters of each network layer. After the training of the hybrid object detection model is completed on each robot end, the data that was not involved in the training locally is used as validation data to calculate various indicators such as the accuracy and recall rate of the model. If the indicators of the hybrid object detection model do not reach the preset threshold, training is restarted, and the hybrid object detection model is repeatedly trained and verified until the total loss of the model converges to the preset threshold. After one round of training, each robot end sends the updated model parameters to the central server. The central server performs weighted averaging on the model parameters of all robot ends to obtain a global model, and re-optimizes the parameters of the obtained global model, and then sends the model back to each device for the next round of training. The local training and global training of the hybrid object detection model are repeatedly performed until the loss value in the global training of the hybrid object detection model converges to the preset threshold. Then, the trained hybrid object detection model is distributed to each robot end.

[0093] It should be noted that the specific steps for re-optimizing the parameters of the obtained global model include: using the aggregated global model as the initial model parameters, collecting all combinations of model parameters, taking the initial model parameters as the initial state of optimization, then initializing the qubit state, that is, the uniform superposition state of all parameter combinations, constructing a quantum Hamiltonian according to the set optimization objective, controlling the state change of the initial qubits based on the time-dependent Schrödinger equation, generating multiple groups of new parameter combinations, calculating the quantum Hamiltonian of each group of new parameter combinations to obtain the performance of the global model under the new parameter combinations, and at the same time, based on the model performance under the new parameter combinations, calculating the difference in model performance between the new parameter combinations and the current parameter combinations. According to the calculated performance difference, use the Metropolis criterion to select whether to accept the new parameter combination. If the performance difference is less than 0, automatically accept the new solution; otherwise, calculate the acceptance probability of the new parameter. If it is greater than the random number, accept it; otherwise, keep the current parameter combination. Repeatedly generate new parameter combinations, calculate the performance difference, and update the parameter combinations until the preset termination time is reached, output the current parameter combination, and at the same time update the parameters of the global model, and wait for the next round of global aggregation, and re-optimize the parameters of the global model again.

[0094] In addition, it should be noted that the specific manifestation of initializing the qubit state is as follows:

[0095]

[0096] In the formula, |ψ(0)> represents the initial quantum state, indicating the uniform superposition of all parameter combinations; N Q represents the number of qubits; x Q represents the model parameter combination, that is, the parameter space of the global model;

[0097] The specific calculation formula of the quantum Hamiltonian is as follows:

[0098]

[0099]

[0100] H(t QH ) = A(t QH )H B + B(t QH )H P

[0101] In the formula, H(t QH ) represents the quantum Hamiltonian; H B represents the driving Hamiltonian, represents the Pauli matrix corresponding to the model parameter combination x QH of the i QH -th qubit; H Prepresents the target Hamiltonian; represents the loss value of the global model at the parameter ; A(t QH ) and B(t QH ) represent time factors. When A(0) >> B(0) at the initial moment, the driving Hamiltonian is dominant, and during the iteration process, A(t QH ) gradually decreases, while B(t QH ) gradually increases.

[0102] Automatically search for and locate holes based on the obtained groups of hole position candidate regions, and optimize the search path generated during the hole search and location process.

[0103] Based on the positioning information of the current target hole and the optimized search path, make decision optimization according to the current environmental state and historical data to guide the blasting execution.

[0104] Based on the same inventive concept, an embodiment of the present invention also provides an automatic hole search and location system based on machine vision. Refer to Figure 2 , Figure 2 which is a schematic structural diagram of an automatic hole search and location system based on machine vision provided by an embodiment of the present invention, including:

[0105] The acquisition and processing module is used to obtain real-time image data from the site and perform image preprocessing after image acquisition; the feature extraction module is used to extract key features from the preprocessed image and generate an image feature map; the detection and location module is used to identify holes in the image and determine their specific positions, and generate information such as the coordinates, dimensions, and angles of the holes; the 3D reconstruction module is used to convert the position and shape of the holes into coordinates in a 3D model according to the detected 2D image information.

[0106] The motion planning module is used to plan the movement path for the robot or automation equipment, and at the same time, combined with the kinematic model of the robot, calculate the optimal motion trajectory.

[0107] Specifically, the motion planning module receives the output data of the detection and positioning module, discretizes the three-dimensional environmental map model into grids of a preset size, where each grid cell represents a position, and at the same time makes the starting position and the target position of the robot correspond to a grid cell. Mark the immovable grids according to the environmental information, initialize the open list and the closed list, where the open list includes all nodes to be examined, and the closed list includes all nodes that have been evaluated. When initializing, the open list contains the starting point and the closed list is empty. Calculate the actual cost, estimated cost, and total cost of each node in the open list, check the 8 nodes adjacent to the current node. If the adjacent node is an obstacle or is already in the closed list, ignore the node. If the adjacent node is not in the open list, add it to the open list and calculate its various cost values. After the check is completed, recalculate the total cost of each feasible adjacent node of the current node. If the total cost of the recalculated adjacent node is less than the total cost of the original node in the open list, update the various cost values of the corresponding node in the open list, and use the adjacent node with the minimum total cost as the next moving node. Then update the current node, and add the node before the movement to the closed list. Gradually select until the robot reaches the target node, then backtrack the node information in the closed list to obtain the complete execution path of the robot. Take the obtained initial path as a group of individuals in the population, perform mutation and crossover operations on the initial path to generate multiple new path individuals. Then calculate the fitness values of each group of paths in the population according to the path length, motion smoothness, and obstacle avoidance ability, use the roulette wheel selection method to calculate the selection probability of each path individual in the population, and use the path individuals with a selection probability higher than the preset threshold as the parent path individuals. Copy the selected parent path individuals, randomly select two groups of parent path individuals, and randomly exchange one or two groups of nodes in the two groups of parent path individuals through the crossover operation to generate new path individuals. Then adjust the copied parent path individuals by randomly changing them to produce new path individuals. Then calculate the fitness values of the new path individuals, screen out the path individuals below the preset threshold, and form a new population with the remaining path individuals and the parent path individuals. Record the paths, fitness, and interactions with the environment in each iteration process. Then repeat the steps of parent path individual selection, crossover, mutation, and new population generation until the change in the population fitness value converges to the preset range. Traverse the fitness values of each group of path individuals in the final population, and select the path individual with the highest fitness value as the final execution path. Send the execution path to the corresponding robot, and control the robot to perform positioning and task execution through the execution control module.

[0108] In this embodiment, the specific calculation formula of the actual cost is as follows:

[0109]

[0110] g(x neighbor )=g(xcurrent ) + d(x current , x neighbor )

[0111]

[0112] f(x TC ) = g(x neighbor ) + h(x heuristics )

[0113] Wherein, d(x current , x neighbor ) represents the cost from the current node x current to the adjacent node x neighbor ; represents the coordinates of the current node x current ; represents the coordinates of the adjacent node x neighbor ; g(x neighbor ) represents the actual cost from the current node x current to the adjacent node x neighbor ; g(x current ) represents the actual cost from the starting point to the current node x current ; h(x heuristics ) represents the estimated cost from the current node x current to the target node; represents the coordinates of the target node; f(x TC ) represents the total cost.

[0114] The execution control module controls the robot to perform precise positioning and task execution according to the instructions received from the motion planning module; the feedback correction module is used to monitor the positioning error and execution process of the robot in real time, and dynamically adjust the motion trajectory and operation strategy of the robot.

[0115] Specifically, the feedback correction module collects various state information when the robot executes operations, and establishes a state space wherein, s i represents the state of the robot, i robot ∈ N robot , obtains various environmental data at different times of the robot, and establishes an observation space Based on the state space and the observation space, a transition probability matrix and an observation probability matrix are respectively established, and the historical environmental data of the robot are collected, and each group of data includes the observation value corresponding to each state. The feedback correction module initializes the initial state probability, the state transition matrix, and the observation probability matrix according to the historical data, and then at the initial time t robotWhen \(t = 1\), calculate the forward probability based on the initial state probability and the observation probability, and through recursive calculation, obtain the forward probability at subsequent times to obtain the probability distribution of the robot being in different states at each time. According to the probability distribution of the robot being in different states at each time, use the Viterbi algorithm for each time \(t\) robot when the robot is in state and observes the observation values \(o_1, o_2, \cdots\) with the maximum probability to obtain the optimal state path at each time. After calculating the optimal path probabilities at all times until reaching the preset termination time, start backtracking from the termination time to find the optimal state sequence to obtain the optimal operation at each time. Then, according to the inference result, control the robot to select the optimal operation and adjust the robot's next operation plan in real time based on the path planning result, historical operation selection, and current environmental state.

[0116] It should be further noted that the specific calculation formula for forward propagation is as follows:

[0117]

[0118] In the formula, \(\alpha_1(i\) FP ) represents the probability that the robot is in state and observes \(o_1\) at time 1; represents the initial state probability; represents the probability that the robot observes \(o_1\) in state ; represents the probability that the robot is in state robot at time \(t\) and observes the first \(t\) robot observation values \(o_1, o_2, \cdots\) ; represents the probability that the robot is in state and observes the first \(t\) robot - 1 observation values \(o_1, o_2, \cdots\) ; represents the probability that the robot transfers from state to ; represents the probability that the robot observes in state .

[0119] The storage and analysis module is responsible for all data storage, management, and analysis tasks during the automatic hole-finding and positioning process, and at the same time uses data analysis technology to evaluate the operation status of the entire system; the anomaly detection module is used to monitor various indicators of the automatic hole-finding and positioning system and detect and identify possible faults or abnormal situations.

[0120] The above has described in detail an embodiment of the present invention, but the above content is only a preferred embodiment of the present invention and cannot be considered as defining the scope of implementation of the present invention. All equivalent changes and improvements made in accordance with the scope of the application of the present invention shall still fall within the scope covered by the patent of the present invention.

Claims

1. An automatic hole finding and positioning method based on machine vision, characterized in that: The following steps are involved: S101: Acquire and pre-process image or video data of the surrounding environment, and perform preliminary analysis on the image using a simulation model; S102: performing collaborative training on each robot, and performing hole position detection and segmentation on the preprocessed image data by each trained robot to obtain a hole position candidate area; S103: Automatically searching for holes according to each group of hole position candidate regions obtained, and optimizing the search path generated during the hole searching and positioning process; S104: Based on the positioning information of the current target hole position and the optimized search path, decision optimization is performed according to the current environmental status and historical data to guide blasting execution.

2. The automatic hole finding and positioning method based on machine vision according to claim 1 is characterized in that: The specific steps of acquiring and preprocessing the image or video data of the surrounding environment in S101 include: P1.1: Select visual sensors according to the actual application scenario, calibrate the selected visual sensors, determine their internal and external parameters, then control the robot to move along the predetermined path, and synchronously start the visual sensors to collect images, store the collected image data and video data, and associate them with the robot's posture information. The visual sensors specifically include laser scanners, high-definition cameras, depth cameras, and multispectral cameras; P1.2: Decompose the collected video data into image data frame by frame, remove the noise in each group of image data by Gaussian filtering, and then use the histogram equalization method to correct the brightness and contrast in each group of image data. After the correction is completed, use the Laplace filter algorithm to enhance the edge information in the image data, and then use the MAX-MIN normalization method to scale the pixel values ​​in each preprocessed image data to a preset range.

3. The automatic hole finding and positioning method based on machine vision according to claim 2 is characterized in that: The specific steps of using the simulation model to perform preliminary analysis on the image in S101 include: P2.1: Based on the calibrated sensor data and environmental parameters, determine the current environmental parameters, the shape and size of the target object, and perform geometric modeling on the target object based on each set of data. Then, use the illumination model to simulate the illumination effect of the illumination in the image data on the display effect of the target object. The specific calculation formula for geometric modeling is as follows: TG(x TG ,y TG )=f TG (x TG ,y TG ,R TG ,D TG ,α TG ) Among them, TG(x TG ,y TG ) represents the position (x TG ,y TG ) is the geometric shape of the target object; R TG Represents the radius of the target object; D TG Represents the depth of the target object; α TG Represents the rotation angle of the target; P2.2: Simulate the sensor response according to the resolution, field of view, focal length and noise characteristics of the visual sensor used, then integrate the simulation results of various environmental factors, sensor characteristics and target geometric characteristics to establish a complete simulation model, and generate a target simulation image through the simulation model; P2.3: Compare the target simulation image generated by the simulation model with the actual collected image data, calculate the structural similarity index between the target simulation image and the actual image data, and count the target objects in the actual image data whose structural similarity index is lower than the preset threshold as the candidate areas for preliminary positioning of the target. The specific calculation formula of the structural similarity index is as follows: Among them, SSIM(x SSIM ,y SSIM ) represents the actual image I(x SSIM ,y SSIM ) and the target simulation image G(x SSIM ,y SSIM )'s structural similarity index; μ I Represents the actual image I(x SSIM ,y SSIM )’s average value; μ G Represents the target simulation image G(x SSIM ,y SSIM )’s average value; Represents the actual image I(x SSIM ,y SSIM )’s variance; Represents the target simulation image G(x SSIM ,y SSIM )’s variance; σ IG Represents the actual image I(x SSIM ,y SSIM ) and the target simulation image G(x SSIM ,y SSIM ); c1 and c2 represent constants.

4. The automatic hole finding and positioning method based on machine vision according to claim 3 is characterized in that: The specific steps of S102 for performing collaborative training on each robot include: P3.1: After each robot preprocesses the image data collected by the visual sensor, it stores each set of preprocessed image data locally on each robot, and adds the corresponding target location and category label to each set of image data to generate an independent local data set. The central server builds an initial hybrid target detection model based on the WGAN model and the SSD model, and uses it as the global model. P3.2: The central server sends the hybrid target detection model to each robot. The robot inputs the local data set into the hybrid target detection model and transmits the local data to the WGAN layer through forward propagation. The generator in the WGAN layer generates the target image based on the input noise and environmental data. The discriminator in the WGAN layer then compares the received local data with the target box generated by the generator and determines whether the generated target box matches the real data in the training set. The generator and discriminator loss values ​​are calculated based on the matching results. P3.3: The local data is then transferred to the SSD layer. The SSD layer classifies and regresses the target boxes in the local data through multi-scale convolutional feature maps, and calculates the classification loss and regression loss respectively through cross entropy loss and smooth L1 loss. Then, the total loss of SSD target detection is obtained based on the classification loss and regression loss. P3.4: Combine the generator loss value, the discriminator loss value, and the total loss of SSD target detection to obtain the total loss of the hybrid target detection model, and input it from the output layer of the hybrid target detection model. Backpropagate according to the chain rule, and calculate the gradient value of the total loss for the parameters of each network layer of the hybrid target detection model. Use the Adam optimizer to optimize the parameters of each network layer. P3.5: After the hybrid target detection model training is completed, each robot uses the local data that is not involved in the training as the verification data to calculate the model's accuracy and recall rate indicators. If the hybrid target detection model indicators do not reach the preset threshold, retraining is performed and the hybrid target detection model is repeatedly trained and verified until the total loss of the model converges to the preset threshold. P3.6: After a round of training, each robot sends the updated model parameters to the central server. The central server performs weighted averaging on the model parameters of all robot terminals to obtain a global model, and re-optimizes the parameters of the obtained global model. The model is then sent back to each device for the next round of training. The local training and global training of the hybrid target detection model are repeated until the loss value in the global training of the hybrid target detection model converges to the preset threshold. After that, the trained hybrid target detection model is sent to each robot terminal.

5. The automatic hole finding and positioning method based on machine vision according to claim 4 is characterized in that: The specific steps of reoptimizing the parameters of the acquired global model described in P3.6 include: P4.1: Take the aggregated global model as the initial model parameters, collect all combinations of model parameters, take the initial model parameters as the initial state of optimization, initialize the quantum bit state, that is, the uniform superposition state of all parameter combinations, and construct the quantum Hamiltonian according to the set optimization goal. The specific form of the initialized quantum bit state is as follows: In the formula, |ψ(0)> represents the initial quantum state, which means the uniform superposition of all parameter combinations; N Q represents the number of quantum bits; x Q Represents the model parameter combination, that is, the parameter space of the global model; The specific calculation formula of quantum Hamiltonian is as follows: H(t QH )=A(t QH )H B +B(t QH )H P In the formula, H(t QH ) represents the quantum Hamiltonian; H B represents the driving Hamiltonian, Represents the i QH The quantum bits correspond to the model parameter combination x QH The Pauli matrix of P represents the target Hamiltonian; Represents the global model in parameters The loss value under QH ) and B(t QH ) represents the time factor. At the initial moment A(0)>>B(0), the driving Hamiltonian is the main factor, and in the iterative process A(t QH ) gradually decreases, B(t QH ) gradually increases; P4.2: Control the state change of the initial quantum bit based on the time-dependent Schrödinger equation, generate multiple sets of new parameter combinations, and calculate the quantum Hamiltonian of each set of new parameter combinations to obtain the performance of the global model under the new parameter combination. At the same time, based on the model performance under the new parameter combination, calculate the model performance difference between the new parameter combination and the current parameter combination; P4.3: Based on the calculated performance difference, use the Metropolis criterion to choose whether to accept the new parameter combination. If the performance difference is less than 0, the new solution is automatically accepted. Otherwise, the acceptance probability of the new parameters is calculated. If it is greater than the random number, it is accepted. Otherwise, the current parameter combination is maintained. P4.4: Repeatedly generate new parameter combinations, calculate performance differences, and update parameter combinations until the preset termination time is reached, and output the current parameter combination. At the same time, update the parameters of the global model, and wait for the next round of global aggregation to re-optimize the global model parameters again.

6. An automatic hole finding and positioning system based on machine vision, used to implement an automatic hole finding and positioning method based on machine vision according to any one of claims 1 to 5, characterized in that: include: The acquisition and processing module is used to obtain real-time image data from the scene and process the image after image acquisition; The feature extraction module is used to extract key features from the preprocessed image and generate an image feature map; The detection and positioning module is used to identify the hole in the image and determine its specific position, and generate the coordinates, size, angle and other information of the hole; The three-dimensional reconstruction module is used to convert the position and shape of the hole into coordinates in the three-dimensional model according to the detected two-dimensional image information; The motion planning module is used to plan the movement path for the robot or automation equipment, and at the same time, calculate the optimal motion trajectory in combination with the robot's kinematic model; The execution control module controls the robot to perform precise positioning and task execution according to the instructions received from the motion planning module; The feedback correction module is used to monitor the positioning error and execution process of the robot in real time, and dynamically adjust the movement trajectory and operation strategy of the robot; The storage and analysis module is responsible for all data storage, management and analysis tasks during the automatic hole finding and positioning process, and uses data analysis technology to evaluate the operating status of the entire system; The abnormality detection module is used to monitor various indicators of the automatic hole finding and positioning system, and detect and identify possible faults or abnormal situations.

7. The automatic hole-finding and positioning system based on machine vision according to claim 6, characterized in that: The specific steps of calculating the optimal motion trajectory by combining the kinematic model of the robot with the motion planning module are as follows: P5.1: The motion planning module receives the output data of the detection and positioning module, and discretizes the 3D environment map model into grids of preset size. Each grid unit represents a position. At the same time, the robot's starting position and target position correspond to a grid unit, and the grids that cannot be moved are marked according to the environmental information. P5.2: Initialize the open list and the closed list. The open list includes all nodes to be examined, and the closed list includes all nodes that have been evaluated. When initialized, the open list contains the starting point and the closed list is empty. Calculate the actual cost, estimated cost, and total cost of each node in the open list. The specific calculation formula for the actual cost is as follows: g(x neighbor )=g(x current )+d(x current ,x neighbor ) f(x TC )=g(x neighbor )+h(x heuristics ) In the formula, d(x current ,x neighbor ) represents the current To the adjacent node x neighbor the cost; Represents the current node x current The coordinates of Represents the adjacent node x neighbor The coordinates of g(x neighbor ) represents the current To the adjacent node x neighbor The actual cost of g(x current ) represents the distance from the starting point to the current node x current the actual cost of h(x heuristics ) represents the current node x current The estimated cost to the target node; represents the coordinates of the target node; f(x TC ) represents the total consideration; P5.3: Check the eight nodes adjacent to the current node. If the adjacent node is an obstacle or is already in the closed list, ignore the node. If the adjacent node is not in the open list, add it to the open list and calculate its generation value. After the check is completed, recalculate the total cost of each feasible adjacent node of the current node. P5.4: If the total cost of the recalculated adjacent nodes is less than the original total cost of the nodes in the open list, then update the values ​​of each generation of the corresponding nodes in the open list, and use the adjacent node with the smallest total cost as the next moving node. Then update the current node and add the node before the move to the closed list. Select step by step until the robot reaches the target node, and then backtrack the node information in the closed list to obtain the complete robot execution path. P5.5: The obtained initial path is used as a group of individuals in the population, and the initial path is mutated and crossed to generate multiple new path individuals. Then, the fitness value of each group of paths in the population is calculated according to the path length, motion smoothness and obstacle avoidance ability. The selection probability of each path individual in the population is calculated using the roulette wheel selection method, and the path individual with a selection probability higher than the preset threshold is taken as the parent path individual; P5.6: Copy the screened parent path individuals, and randomly select two groups of parent path individuals. Through the crossover operation, randomly exchange one or two groups of nodes in the two groups of parent path individuals to generate new path individuals. Then adjust the copied parent path individuals through random changes to produce new path individuals. Then calculate the fitness value of the new path individuals, filter out the path individuals below the preset threshold, and form a new population with the remaining path individuals and the parent path individuals. P5.7: Record the path, fitness and interaction with the environment in each iteration, then repeat the steps of parent path individual selection, crossover, mutation and new population generation until the population fitness value changes converge to the preset range, traverse the fitness values ​​of each group of path individuals in the final population, and select the path individual with the highest fitness value as the final execution path, send the execution path to the corresponding robot, and control the robot for positioning and task execution through the execution control module.

8. The automatic hole-finding and positioning system based on machine vision according to claim 7, characterized in that: The specific steps of the feedback correction module dynamically adjusting the robot's motion trajectory and operation strategy are as follows: P6.1: The feedback correction module collects the state information of the robot when performing operations and establishes the state space Among them, s i Represents the state of the robot. i robot ∈N robot , according to the robot to obtain environmental data at different times, establish an observation space Based on the state space and observation space, the transition probability matrix and observation probability matrix are established respectively; P6.2: Collect the robot's historical environmental data, where each set of data includes the observation value corresponding to each state. The feedback correction module initializes the initial state probability, state transition matrix, and observation probability matrix based on the historical data. Then, at the initial time t robot =1, the forward probability is calculated according to the initial state probability and the observation probability, and the forward probability at subsequent moments is obtained through recursive calculation to obtain the probability distribution of the robot in different states at each moment. The specific calculation formula for forward propagation is as follows: In the formula, α1(i FP ) means that at time 1, the robot is in state And the probability of observing o1; represents the initial state probability; Represents the robot in state The probability of observing o1 under Represents at time t robot When the robot is in state And observe that the previous t robot Observations o1, o2, K, probability; Represents the robot in state And observe that the previous t robot -1 observation o1, o2, K, probability; Represents the robot from the state Transfer to probability; Represents the robot in state The following observation probability; P6.3: According to the probability distribution of the robot in different states at each moment, the Viterbi algorithm is used to calculate the probability distribution of the robot in different states at each moment. robot When the robot is in state And observed observations o1, o2, K, The maximum probability of is used to obtain the optimal state path at each moment. After calculating the optimal path probability at all moments, the preset termination moment is reached. P6.4: Start backtracking from the termination moment to find the optimal state sequence to obtain the optimal operation at each moment. Then, based on the reasoning results, control the robot to select the optimal operation and adjust the robot's next operation plan in real time based on the path planning results, historical operation selections, and current environmental status.

Images