A method and system for intelligent unmanned operation of facility environment based on machine vision

Abstract

The present application discloses a method and system for intelligent unmanned operation of a facility environment based on machine vision, which relates to the field of unmanned driving technology. The method includes: based on an ultrasonic sensor, scanning obstacles inside the target facility environment to determine obstacle information; based on a binocular camera, using an adaptive image enhancement algorithm to collect the label code in front of the operating equipment to obtain a label code image; based on the label code image and obstacle information, constructing a global operation map of the target facility environment; based on an extended Kalman filter, multi-source data fusion is performed on the label code image and the information obtained by the gyroscope, accelerometer and speed sensor to obtain the operating equipment positioning information and the operating equipment status; based on the operating equipment positioning information and the operating equipment status, based on the operating requirements of the operating equipment, the improved A algorithm is used to plan the path of the operating equipment. The present application can ensure accurate positioning of the operating equipment and intelligent unmanned operation.

Description

Technical Field

[0001] The present application relates to the field of unmanned driving technology, and in particular to an intelligent unmanned operation method and system for a facility environment based on machine vision. Background Art

[0002] In modern production, facility agriculture has become a key means to ensure continuous production throughout the year, and the application of facility environments such as breeding workshops and greenhouse farms is becoming increasingly popular. In these facility environments, the driving accuracy and stability of operating machinery play a vital role in crop planting and animal breeding. Although indoor navigation technologies such as the Beidou positioning system, mobile hotspots, Bluetooth, and inertial navigation have been applied, the accuracy and reliability of these technologies are often limited under the complex conditions inside the facilities. For example, GPS signals may be blocked in closed or semi-closed structures, resulting in weakened or distorted signal strength; Wi-Fi and Bluetooth signals are susceptible to interference in metal structures and dense vegetation, affecting the accuracy and stability of positioning; and the accuracy of inertial navigation systems will also decrease due to long-term accumulated errors. Therefore, in complex and changeable facility environments, the application effect of traditional positioning technologies is significantly limited, and it is difficult to meet the needs of high-precision positioning of operating machinery.

[0003] At present, most operating machines usually use manual correction to position, relying mainly on manual decision-making and past experience to guide the positioning and operation of the operating machine. This method cannot ensure the consistency of positioning and the accuracy of the machine running on the predetermined trajectory. Summary of the invention

[0004] The purpose of this application is to provide a method and system for intelligent unmanned operation in a facility environment based on machine vision, which can assist equipment in the facility environment to achieve autonomous navigation and path planning, and realize precise positioning of the operating equipment.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In the first aspect, the present application provides a method for intelligent unmanned operation of a facility environment based on machine vision, based on an intelligent unmanned operation system for a facility environment based on machine vision, the system includes: operation equipment and environmental equipment; the operation equipment includes: a core controller, a binocular camera, a pan / tilt control system, a voltage reduction system, a communication system, an ultrasonic sensor, a gyroscope, an accelerometer and a speed sensor, the environmental equipment includes a tag code and a temperature and humidity sensor; the method for intelligent unmanned operation of a facility environment includes:

[0007] Based on ultrasonic sensors, obstacles inside the target facility environment are scanned to determine obstacle information.

[0008] Based on a binocular camera, an adaptive image enhancement algorithm is used to collect the label code in front of the operating equipment to obtain a label code image; the label code image includes several label codes, each of which has a mark; the operating equipment is an intelligent unmanned operating vehicle.

[0009] Based on the label code image and obstacle information, a global operation map of the target facility environment is constructed.

[0010] Based on the extended Kalman filter, multi-source data fusion is performed on the label code image and the information obtained by the gyroscope, accelerometer and speed sensor to obtain the positioning information and status of the operating equipment.

[0011] According to the positioning information and status of the operating equipment, based on the operating requirements of the operating equipment, the improved A algorithm to plan the path of the operating equipment; the improved A The algorithm is based on the traditional A Based on the algorithm, a sliding average algorithm is used to create a neighborhood for each point to achieve path smoothing.

[0012] On the second aspect, the present application provides an intelligent unmanned operation system for facility environment based on machine vision, including: operation equipment and environmental equipment.

[0013] The operating equipment includes: a core controller, a binocular camera, a pan / tilt control system, a voltage reduction system, a communication system, an ultrasonic sensor, a gyroscope, an accelerometer and a rotation speed sensor.

[0014] The environmental equipment includes a tag code and a temperature and humidity sensor.

[0015] The tag code is arranged inside the target facility environment to provide location information; the temperature and humidity sensor is used to monitor the temperature information and humidity information of the target facility environment, and to correct the sound velocity data of the ultrasonic sensor.

[0016] The core controller is communicatively connected with the step-down system, the rotation speed sensor and the binocular camera; and the system carried on the core controller is a Raspberry Pi system.

[0017] The binocular camera is used to collect and identify images of each label code in the target facility environment.

[0018] The pan / tilt control system is connected to the binocular camera to increase the stability of the binocular camera.

[0019] The step-down system is connected to the input power supply of the operating equipment and is used to output a DC voltage to provide a stable constant voltage source for the system.

[0020] The communication system is embedded in the core controller and is used for communication between the core controller and the operating equipment.

[0021] The ultrasonic sensor is arranged on the front side of the working equipment and is used to detect obstacles in the area in front of the working equipment.

[0022] The gyroscope is used to output the posture information of the operating equipment.

[0023] The accelerometer is used to output acceleration information of the operating equipment.

[0024] The rotation speed sensor is used to collect the wheel rotation speed of the working equipment and determine the speed information of the working equipment according to the wheel circumference and the wheel rotation speed.

[0025] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0026] The present application provides a method and system for intelligent unmanned operation of a facility environment based on machine vision, the method comprising: based on an ultrasonic sensor, scanning obstacles inside a target facility environment to determine obstacle information; based on a binocular camera, using an adaptive image enhancement algorithm to collect label codes in front of an operation device to obtain a label code image; based on the label code image and obstacle information, constructing a global operation map of the target facility environment; based on an extended Kalman filter, performing multi-source data fusion on the label code image and information obtained by a gyroscope, an accelerometer, and a speed sensor to obtain operation equipment positioning information and operation equipment status; based on the operation equipment positioning information and operation equipment status, using an improved A This application aims to assist the equipment in the facility to achieve autonomous navigation and path planning, ensuring the precise positioning of the operating equipment and the realization of unmanned operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 A flowchart of an intelligent unmanned operation system for a facility environment based on machine vision is provided in accordance with an embodiment of the present application.

[0029] Figure 2 A top view of the relative positions of a camera and an ultrasonic sensor provided in one embodiment of the present application.

[0030] Figure 3 An improved A provided in one embodiment of the present application Flowchart of the path planning algorithm.

[0031] Figure 4 A schematic diagram of the structure of an intelligent unmanned operation system for a facility environment based on machine vision provided in one embodiment of the present application.

[0032] Figure 5 A technical roadmap for an intelligent unmanned operation system for a facility environment based on machine vision is provided for one embodiment of the present application. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0034] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0035] Embodiment 1

[0036] like Figure 1 As shown, this embodiment provides a facility environment intelligent unmanned operation method based on machine vision, based on a facility environment intelligent unmanned operation system based on machine vision, the system includes: operation equipment and environmental equipment; the operation equipment includes: a core controller, a binocular camera, a pan / tilt control system, a voltage reduction system, a communication system, an ultrasonic sensor, a gyroscope, an accelerometer and a speed sensor, the environmental equipment includes a tag code and a temperature and humidity sensor; the facility environment intelligent unmanned operation method includes:

[0037] Step 101: Scan obstacles inside the target facility environment based on ultrasonic sensors to determine obstacle information.

[0038] Step 102: Based on a binocular camera, an adaptive image enhancement algorithm is used to collect the label code in front of the operating equipment to obtain a label code image; the label code image includes a plurality of label codes, each of which has a mark; the operating equipment is an intelligent unmanned operating vehicle.

[0039] Step 103: Construct a global operation map of the target facility environment based on the label code image and obstacle information.

[0040] Step 104: Based on the extended Kalman filter, multi-source data fusion is performed on the label code image and the information obtained by the gyroscope, accelerometer and speed sensor to obtain the operating equipment positioning information and the operating equipment status.

[0041] Step 105: Based on the positioning information and status of the operating equipment and the operating requirements of the operating equipment, the improved A algorithm to plan the path of the operating equipment; the improved A The algorithm is based on the traditional A Based on the algorithm, a sliding average algorithm is used to create a neighborhood for each point to achieve path smoothing.

[0042] Before executing step 101, the method further includes:

[0043] Arrange Aruco tags on indoor walls. In order to build a work map, the traditional approach is to arrange label codes in an orderly manner indoors according to strict standards and specifications. However, this method is time-consuming and laborious, and its portability is poor. This embodiment is based on the Raspberry Pi system, and there is no need to deliberately arrange label codes. It only needs to be arranged on different planes of the wall, and the spacing and number of arrangements do not need to be specially planned.

[0044] Specifically, when executing step 101, it can be as follows:

[0045] This embodiment takes into account the temperature and humidity The impact on the speed of sound, the data detected by the temperature and humidity sensor on the speed of sound To make a correction:

[0046] .

[0047] The ultrasonic sensor is used to scan obstacles inside the facility. , using the time difference between transmission and reception Calculate the distance between the obstacle and the sensor :

[0048] .

[0049] In the formula, is the time difference between the ultrasonic sensor transmitting and receiving signals, , is the temperature, H For humidity.

[0050] In some embodiments, when executing step 102, the specific steps may be as follows:

[0051] Get the image captured by the binocular camera.

[0052] Performing histogram equalization on the image to obtain an image with adjusted brightness.

[0053] Adaptive threshold technology is used to perform binarization processing on the brightness-adjusted image to obtain an image with enhanced contour clarity.

[0054] Based on the contour detection method, label code recognition is performed on the image with enhanced contour clarity to obtain a label code image.

[0055] Specifically, when the binocular camera acquires the first frame of image Finally, considering the actual environment in the facility, an adaptive image enhancement algorithm is developed to automatically adjust the contrast and brightness according to the actual lighting conditions to ensure the best image quality at different times and under different lighting conditions; in order to automatically optimize the brightness and contrast of the image and improve the visibility and recognition rate of the Aruco tag code, histogram equalization is used to enhance the image. The steps are as follows:

[0056] First, calculate the grayscale average of the input image :

[0057] .

[0058] in is the image at position The brightness value, M and N are the number of rows and columns of the image.

[0059] Based on the difference between the average brightness and the target brightness, adjust the brightness of each pixel appropriately:

[0060] .

[0061] in is the adjustment factor, is the desired target brightness.

[0062] After the image processing is completed, in order to ensure the effective detection of Aruco code in the environment with changing illumination or complex background, the adaptive threshold technology is used to binarize the image, making the Aruco code boundary in the image more obvious.

[0063] Next, use contour detection to find these possible Aruco code areas. Since Aruco codes are usually square, check the shape ratio and angle of the contour, filter out contours that are too large or too small, and screen candidate areas. For each detected candidate area, verify whether its internal pattern conforms to the standard format of Aruco code. After verification, each Aruco code has a pre-defined ID, and its ID is determined by decoding the binary pattern inside the Aruco code.

[0064] Once the Aruco code is recognized, the PnP algorithm is applied to calculate the rotation matrix and translation vector of the Aruco marker relative to the camera using the known Aruco marker size and detected corner point positions combined with the camera's internal calibration parameters.

[0065] The goal of the PnP problem is to estimate the camera's pose, specifically its rotation and translation states, from a set of known 3D points in the world coordinate system and corresponding 2D points in the image coordinate system. The 3D point coordinates are the positions of the four corner points of the Aruco marker in the world coordinate system, which can be calculated based on the known size of the Aruco marker, and the corresponding 2D point coordinates are the image coordinates of the corresponding four corner points detected in the image.

[0066] The rotation matrix of the Aruco marker relative to the camera is calculated by the PnP algorithm:

[0067] .

[0068] The translation vector is:

[0069] .

[0070] Combining the rotation matrix with the translation vector, we can construct the complete transformation matrix from the Aruco marker coordinate system to the camera coordinate system:

[0071] .

[0072] The matrix T describes the transformation of the Aruco marker relative to the camera.

[0073] In some embodiments, when executing step 103, the specific steps may be as follows:

[0074] According to the label code image collected in real time, the target label code in each frame of the label code image is detected; the target label code is the Aruco label code.

[0075] According to the Aruco label codes in the label code images of adjacent frames, the pose transformation relationship between the label codes is calculated.

[0076] Connect the label code transformation relationships in different frames to create a graph structure.

[0077] The Aruco label codes in the detected image label codes are introduced into the graph structure as nodes, and edges are constructed based on the calculated label code transformation relationship.

[0078] The Levenberg-Marquardt algorithm is used to minimize the reprojection error of each key frame and label code point in the graph structure to obtain the optimized Aruco label code and pose.

[0079] According to the label codes and postures, the global coordinates corresponding to the label codes are integrated into the global point cloud data to construct a point cloud map.

[0080] Obstacle information is filled into the point cloud map to construct a global operation map of the target facility environment.

[0081] Specifically, there are multiple Aruco markers in the environment. When the first frame captured by the camera When there are multiple tags in the image, multiple label codes can be obtained. The transformation relationship relative to the camera , the pose transformation relationship between each two sets of label codes can be obtained by the following formula:

[0082] .

[0083] .

[0084] .

[0085] In the second frame In the above process, we repeat the above process and get a set of images The pose transformation relationship data between each two sets of label codes.

[0086] Through label code detection and pose calculation in multiple frames, the transformation relationship between the same label codes in different frames can be connected to form a complete global pose graph.

[0087] First, initialize the global reference system and select a label code in the first frame as the origin of the global reference system. , and set its pose to the identity matrix .

[0088] Next, add other label codes. For each frame of the image, each label code detected Label code in the global reference frame Calculate the pose transformation relationship. Specifically, in the image In the case of a tag code being detected and , and calculate its position in the global reference system using the following formula:

[0089] .

[0090] .

[0091] Considering that the Aruco label codes recognized in different frames are different, it is necessary to connect the label code poses in different frames:

[0092] .

[0093] in, .

[0094] Create a graph structure, where nodes represent each Aruco coordinate and edges represent the relative pose transformation relationship between them. For each detected Aruco label code, add a node to the graph, and according to the calculated relative pose transformation relationship, add an edge to the graph to record the transformation relationship between each pair of label codes.

[0095] Considering that there is a reprojection error between each frame image, in order to minimize the reprojection error, a global optimization is performed on all key frames and label code points.

[0096] The objective function of global optimization is defined as:

[0097] .

[0098] In the formula, is the total optimization objective function, that is, the sum of the reprojection errors of the marker angles in all frames. The optimization goal is to minimize this error function; is the pose parameter of the label code, representing the pose of each label code in the global reference system; is the pose parameter of the key frame, which represents the pose of each key frame in the global reference system; is the camera intrinsic parameter matrix; is the frame index, indicating the frame currently being processed; For Frame The set of tag codes detected in , including the IDs of all detected tag codes; For Frame The reprojection error of label code i in is defined as:

[0099] .

[0100] In the formula, For Frame 's posture; For label code 's posture; For label code No. The three-dimensional coordinates of the corner points in the label code coordinate system; For the frame In the label code No. The actual observed pixel positions of the corner points; is the projection function, specifically:

[0101] .

[0102] In the formula, is the camera intrinsic parameter matrix; is the coordinate of a 3D point.

[0103] The optimization is performed using the Levenberg-Marquardt algorithm. To provide a good initial estimate, the preliminary position of each tag code is obtained from the initial frame pose estimate, and the relative poses between matching tag code pairs are used to build an initial pose graph.

[0104] In each iteration, the error vector under the current parameters is calculated. For each 3D point of the label code in each observation frame, it is projected onto the image plane, the pixel coordinates of the projected point and the actual observation point are compared, and the reprojection error is calculated. .

[0105] Define the Jacobian matrix Represents the inverse of the error vector to the parameter, that is, the influence of each parameter on the error. In the system mentioned in this embodiment, since only a small part of the label code is observed in each frame in the optimization problem, the Jacobian matrix mentioned in this method is a sparse matrix. The sparse matrix method is used to perform matrix operations to optimize the memory usage of the core controller of this system. Each parameter in the Jacobian matrix Represents the partial derivative of the kth error component with respect to the jth parameter:

[0106] .

[0107] In the formula, is the kth error component; is the jth parameter.

[0108] Update the parameter vector x using the update formula of the LM algorithm:

[0109] .

[0110] In the formula, is the damping factor, used to adjust the update step size; I is the identity matrix; is the parameter update amount.

[0111] By solving the linear equations, we get the new parameters:

[0112] .

[0113] The iteration continues until the error is less than a predetermined threshold level and the iteration stops.

[0114] After optimizing all label codes and keyframe poses by minimizing the reprojection error, the global coordinates of all label codes are integrated into a global point cloud to form the final point cloud map.

[0115] Specifically, in order to obtain complete information about the internal map of the facility, the obstacle information detected by the ultrasonic sensor needs to be added to the already constructed point cloud map. The relative position of the ultrasonic sensor and the camera is as follows: Figure 2 As shown:

[0116] Since the distance information of the obstacle has been measured by the ultrasonic sensor, in order to add the obstacle information to the point cloud map, the measurement results of the ultrasonic sensor need to be converted into the global coordinate system, such as Figure 2 As shown, the relative positions of the three sensors relative to the camera are: , the position of the camera, that is, the working equipment, in the global coordinate system is , then the position and posture of the ultrasonic sensor in the global coordinate system for:

[0117] .

[0118] According to the ultrasonic sensor's measurement distance and the sensor's attitude, including the pitch angle With yaw angle , calculate the coordinates of the obstacle in the sensor coordinate system :

[0119] .

[0120] Convert the obstacle coordinates to the global coordinate system:

[0121] .

[0122] Obstacle points Add it to the global point cloud map. At this point, the construction of the internal map of the facility is completed.

[0123] When executing step 104, the specific steps may be as follows:

[0124] Considering the navigation and control requirements of the equipment in the complex environment inside the facility, gyroscopes, accelerometers and speed sensors are integrated to read the status of the equipment. The selected sensors include high-precision gyroscopes, three-axis accelerometers and wheel speed sensors, all of which support RS485 communication protocol and are connected to the core controller via twisted pair cables. At the same time, all sensors are installed near the center of mass of the equipment to minimize the errors caused by the movement of the equipment. Considering the existence of errors, environmental noise, external interference and other factors in the working environment, sensor data may have abnormal values ​​or contradictory information.

[0125] In order to identify and process these abnormal data in real time and improve the stability of the system and the accuracy of decision-making, this embodiment is based on an abnormality detection algorithm based on statistical analysis, and the algorithm uses a box plot method to identify abnormal values. Specifically, the algorithm process includes: data collection, preprocessing, application of the box plot method and abnormal value processing.

[0126] When preprocessing sensor data, the three sensors have different functions and measurement targets, so the preprocessing methods are also different. The gyroscope measures the angular velocity of the device, and the data is expressed in degrees per second (° / s). When processing the gyroscope data, a low-pass filter is used to remove high-frequency noise and retain useful signals; the accelerometer provides three-axis acceleration data, usually in meters per second squared (m / s²). When processing the accelerometer data, the data is corrected to eliminate the influence of gravity acceleration; the speed sensor measures the speed at the wheel, usually in revolutions per minute (rpm), reflecting the real-time operating status of the device. When processing the speed sensor data, data smoothing is performed to remove possible instantaneous peaks or drops.

[0127] Based on the preprocessing, the box plot method is used to calculate the abnormal data of the sensor. In the box plot method, the first quartile Q1 is the value at the 25% position after sorting the sequence, the third quartile Q3 is the value at the 75% position, and the interquartile range IQR is the difference between the third quartile and the first quartile, that is:

[0128] .

[0129] On this basis, define the abnormal threshold:

[0130] Nether: .

[0131] Upper bound: .

[0132] This method can be used to identify outliers in the data, analyze the marked outliers, and determine whether they are caused by temporary interference or sensor failure. Based on the analysis results, recalibration or temporary elimination of the influence of abnormal data can be performed.

[0133] Under the premise of ensuring that the data collected by the sensor is correct, this embodiment adds a strategy for handling sensor information contradictions.

[0134] In the data conflict processing of gyroscope and accelerometer, the gyroscope is used to estimate the direction of the device, and the accelerometer is used to estimate the acceleration of the device. When the device is turning, the angular velocity of the gyroscope should be consistent with the centripetal acceleration measured by the accelerometer. Based on this principle, the data correlation test is performed, and the centripetal acceleration calculation formula is used to verify whether the data of the gyroscope and accelerometer are consistent:

[0135] .

[0136] in is the centripetal acceleration, It is the linear speed calculated by the speed sensor. It is the radius calculated based on the angular velocity of the gyroscope and the known turning radius.

[0137] In the data conflict processing between the speed sensor and the accelerometer, the speed sensor measures the wheel axle speed, while the accelerometer can provide the linear acceleration information of the vehicle. In the absence of external interference, these two data should confirm each other. Based on this principle, the data correlation test is performed. The speed is obtained by integrating the accelerometer data and compared with the wheel linear speed obtained by converting the speed sensor data:

[0138] .

[0139] .

[0140] In the formula is the acceleration a The speed obtained by integration is is the wheel speed obtained by the speed sensor, is the wheel radius, It is the revolutions per minute measured by the speed sensor.

[0141] Monitor and record all sensor data in real time; regularly check for abnormal values ​​in each sensor output using the box plot method; and check data consistency using the above-mentioned conflict handling strategy.

[0142] After completing the sensor information collection and visual information reading, the improved extended Kalman filter is used to perform multi-source data fusion on the information obtained by the visual system and the gyroscope, accelerometer and speed sensor, and the dynamic system state is predicted and updated.

[0143] Considering the physical particularity of the intelligent unmanned operation system of this embodiment, the improvement of this method is to use a multi-scale time window to process multi-source data. Specifically, considering that in this system, two main types of data will be processed: high-frequency sensor data (gyroscope, speed sensor and accelerometer) and low-frequency visual information data.

[0144] To optimize the fusion of this data, different time window strategies are implemented in the EKF. , attitude angle , acceleration a. A smaller time window is set for these data, which is 10 milliseconds in this embodiment. In each time window, the data is accumulated and its average value is calculated as the representative value of the time window.

[0145] Collect vision system data at a low frequency (e.g. 50 Hz) to provide position and attitude angle A larger time window is set for the visual data, which is 100 milliseconds in this embodiment. Within each time window, the data is processed and integrated to obtain accurate position and attitude estimates.

[0146] In addition, this embodiment designs a time-weighted data fusion strategy to achieve smooth and effective fusion between data sources of different frequencies, and achieves continuous and accurate estimation of the state by weighted accumulation of high-frequency sensor data and visual data information.

[0147] During the visual data update cycle, a weight is assigned to each sensor data point , defined with respect to time Inversely proportional, It is the time interval from the current sensor data point to the next visual update, which is 0.02 seconds in this system. In order to make the weight distribution more reasonable, it is normalized to ensure that the sum of all weights is 1.

[0148] Among them, the normalized weight The calculation formula is:

[0149] .

[0150] Among them, the denominator is the sum of the inverse weights of all sensor data points.

[0151] In this method, the visual data update cycle is 20ms, and the system receives 2 sensor data points within this cycle, with each data point received at an interval of 10ms.

[0152] Define the state vector X as:

[0153] .

[0154] In the formula, p represents the position, which is provided by the vision system. v represents the speed, which is calculated by the vision system through the position differential meter. is the attitude angle, which is calculated from the information provided by the visual system. a is the acceleration, which is calculated by the visual system through the velocity differential meter.

[0155] The observation vector Z includes data directly measured by various sensors:

[0156] .

[0157] in, is the speed measured by the speed sensor, is the angular velocity measured by the gyroscope, is the acceleration measured by the accelerometer.

[0158] Initialize the state vector:

[0159] .

[0160] in, , , , are initial values ​​of position, velocity, angle, and acceleration estimated based on available prior information or first observations.

[0161] Covariance matrix initialization:

[0162] .

[0163] in, , , , is the uncertainty in the initial estimate of position, velocity, angle, and acceleration.

[0164] Entering the prediction stage, the extended Kalman filter uses the state transition model to predict the state of the next time step and updates the covariance matrix. Based on the laws of physical motion, the state transition model is constructed. :

[0165] .

[0166] Where I is the identity matrix, is the time interval.

[0167] Perform status prediction as follows:

[0168] .

[0169] Perform covariance prediction as follows:

[0170] .

[0171] in, is the process noise covariance matrix, describing the uncertainty of the model predictions.

[0172] Perform state prediction and update the state vector every 10ms With the covariance matrix .

[0173] During the update phase, the state is updated simultaneously using the latest averaged sensor data and the averaged visual data.

[0174] calculate ; .

[0175] in, is the observation model matrix, is the observation noise covariance matrix.

[0176] Calculate Kalman gain , as follows:

[0177] .

[0178] Update the state vector using the difference between the actual observation and the predicted observation as follows:

[0179] .

[0180] By formula ,get .

[0181] Update the covariance matrix to reflect the uncertainty after the state update.

[0182] Through this step, the extended Kalman filter can continuously fuse data from multiple sources, including all sensor data updates in each visual data update cycle. At the same time, during the visual update cycle, the accumulated sensor data is used to make accurate state predictions to obtain high-precision positioning information and accurate operating equipment status.

[0183] like Figure 5 As shown, the fused data includes sensor information and visual information; the sensor information and visual information are fused and input into the core controller to construct point cloud maps, grid maps, global positioning, path planning and underlying control.

[0184] In some embodiments, when executing step 105, the specific steps may be as follows:

[0185] According to the operation requirements of the operation equipment at a certain moment, the improved A The algorithm performs path planning to provide efficient guidance and precise control of agricultural machinery, ensuring that the equipment can navigate efficiently and safely in complex greenhouse environments.

[0186] Before path planning, the point cloud map needs to be rasterized with a grid size of 0.05m. 0.05m, traverse each point in the point cloud, and mark the grid cells corresponding to the walls and obstacles as inaccessible areas.

[0187] Among them, traditional A The implementation process of the algorithm is as follows:

[0188] Specifically, an open list and a closed list are created, and the open list is sorted according to the size of the cost estimate f(n), where:

[0189] .

[0190] In the formula represents the cost of each node from the starting point to the node, Represents the estimated cost from the node to the end point, the heuristic function Using Euclidean distance calculation:

[0191] .

[0192] Where n x and n y Respectively represent the horizontal and vertical coordinates of a node in two-dimensional space. x and g y Represents the horizontal and vertical coordinates of the end point in two-dimensional space.

[0193] In the specific implementation stage, firstly, a current node is selected, that is, the node with the lowest f value is taken from the open list as the current node, and then it is checked whether the current node is the end point.

[0194] If it is not the end point, for each neighbor node of the current node, calculate the new g value of the neighbor node through the current node:

[0195] .

[0196] in is the cost from the current node to its neighbors, is the cost from this node to the starting point.

[0197] Calculate the new g After the value is calculated, the current neighborh value, and finally update the cost estimate of each neighbor node f After calculating the f value of each neighbor node, sort them and update the open list and closed list.

[0198] If the neighbor is not in the open list, add it; if the neighbor is already in the open list, but a better one is found g Value path, update the node g , h and f Value. After the operation is completed, move the current node to the closed list.

[0199] The above operation is repeated until the current node is found to be the end point, which means that the path planning is initially completed. Finally, when the end point node is found, the path needs to be reconstructed. The so-called path reconstruction means starting from the end point, using the parent node information stored in each node, tracing back from the end point to the starting point, and constructing a path from the starting point to the end point.

[0200] So far, the traditional A The algorithm has been completed so far. This improved algorithm starts from the operation quality and efficiency of the equipment, combines the sliding average algorithm, smoothes its operation path, and improves the efficiency of the operation equipment.

[0201] The sliding average algorithm used in this embodiment smoothes the path by creating a neighborhood of each point. Specifically, the sliding window n Defines the number of neighboring points used to calculate the average. n After the size of , calculate the average position of the points within the window to determine the new smoothing point :

[0202] .

[0203] In the formula, Indicates half of the window value. Considering the particularity of the starting and ending parts of the path, the window size is defined separately for them. The origin is directly used as the window size to avoid out-of-bounds indexes. i indicates the index of the current path point, that is, the position of the path point P being smoothed. j indicates the position of the current point P. i The offset within the neighborhood of P is usually in the range of 0-5. i The positions of other nearby points. For example, when j=-1, it means P-1, which is the point before P; when j=1, it means P+1, which is the point after P. Therefore, i+j represents the point centered on P and offset j positions before and after it, which is used to calculate the average position of the points within this window.

[0204] After smoothing is completed, a point set is generated ,use Replace the original path points of the above path backtracking to generate a new smooth path. At this point, the path planning for the operating equipment is completed. The specific algorithm flow is as follows Figure 3 As shown:

[0205] In addition, in some embodiments, the underlying control algorithm uses an adaptive PID controller to control the specific implementation of path planning. The system develops a neural network-based adaptive PID control system specifically designed to dynamically adjust the control parameters of mobile working equipment to achieve accurate path tracking. The system directly adjusts the proportional, integral and differential gains of the PID controller by integrating a neural network model, and dynamically optimizes the control strategy based on the real-time feedback data obtained from the fusion of sensors and vision systems.

[0206] The neural network consists of an input layer, a hidden layer, and an output layer. The input layer receives three signals, namely the current error , error rate , Error integral The hidden layer is 1 layer, containing 20 neural units, using the rectified linear unit (ReLU) activation function, allowing the network to learn the complex and nonlinear relationships in the input data. ReLU is defined as:

[0207] .

[0208] The output layer outputs three adjusted PID parameters, corresponding to , , The output of the neural network is used to adjust the gain parameters of the PID controller in real time. Specifically, the network predicts the optimal control parameters based on the error information received, and these parameters are then used to calculate the PID.

[0209] .

[0210] Each gain parameter in the formula , , All are provided in real time by the neural network, ensuring that the controller can adjust itself according to the current system dynamics and changes in the external environment.

[0211] In this embodiment, the control system collects data from sensors and visual systems in real time and calculates the current error , error rate , Error integral , and feed these data into the neural network; the neural network is trained with historical data in the initial stage to learn how to adjust according to different error states PID parameters; during the deployment phase, the network dynamically outputs PID Parameters, no human intervention is required; according to the output of the neural network PID Parameters, the control system dynamically adjusts the drive input of the operating equipment to minimize path deviation and optimize operating efficiency and quality.

[0212] Embodiment 2

[0213] like Figure 4 As shown, a machine vision-based intelligent unmanned operation system for facility environment includes: operation equipment and environmental equipment.

[0214] The operating equipment includes: a core controller, a binocular camera, a pan / tilt control system, a voltage reduction system, a communication system, an ultrasonic sensor, a gyroscope, an accelerometer and a rotation speed sensor.

[0215] The environmental equipment includes a tag code and a temperature and humidity sensor.

[0216] The tag code is arranged inside the target facility environment to provide location information; the temperature and humidity sensor is used to monitor the temperature information and humidity information of the target facility environment, and to correct the sound velocity data of the ultrasonic sensor.

[0217] The core controller is communicatively connected with the step-down system, the rotation speed sensor and the binocular camera; and the system carried on the core controller is a Raspberry Pi system.

[0218] The binocular camera is used to collect and identify images of each label code in the target facility environment.

[0219] The pan / tilt control system is connected to the binocular camera to increase the stability of the binocular camera.

[0220] The step-down system is connected to the input power supply of the operating equipment and is used to output a DC voltage to provide a stable constant voltage source for the system.

[0221] The communication system is embedded in the core controller and is used for communication between the core controller and the operating equipment.

[0222] The ultrasonic sensor is arranged on the front side of the working equipment and is used to detect obstacles in the area in front of the working equipment.

[0223] The gyroscope is used to output the posture information of the operating equipment.

[0224] The accelerometer is used to output acceleration information of the operating equipment.

[0225] The rotation speed sensor is used to collect the wheel rotation speed of the working equipment and determine the speed information of the working equipment according to the wheel circumference and the wheel rotation speed.

[0226] In some embodiments, the tag code used by the system is an Aruco tag code. Aruco tag codes have unique patterns and encoding methods, which can provide high-precision positioning and identification functions. Through these tag codes, the system can quickly and accurately identify and track the position and direction of objects.

[0227] The operating equipment in the system includes: a core controller, a binocular camera, a pan / tilt control system, a voltage reduction system, a communication system, ultrasonic waves, a gyroscope, an accelerometer and a speed sensor, which together constitute the system. The operating equipment mentioned in the system is a general mechanical operating equipment with universal applicability, including but not limited to equipment that can be controlled by communication protocols such as Can-BUS, RS-232 / RS-485, and ISO BUS.

[0228] Specifically, Figure 4 As shown, the speed sensor is located at the wheel of the operating equipment and is used to detect the driving speed of the operating equipment. In this embodiment, the speed sensor is a Hall sensor for detecting the speed of an object and is connected to the core controller via a USB to serial port.

[0229] The gyroscope is located on a horizontal platform inside the operating equipment and is used to determine the posture of the equipment, specifically, the posture includes angle and direction. In this embodiment, the posture data provided by the gyroscope is fused with the visual information captured by the binocular camera. When the line of sight is obstructed or the lighting conditions are poor and the visual system cannot independently provide accurate navigation information, the gyroscope information supplements the data to help the system estimate the current position and motion state more accurately.

[0230] The accelerometer is placed at the center of mass of the device, close to the gyroscope, to ensure that the two obtain similar dynamic conditions. It is used to measure the linear acceleration of the entire device and is integrated with the travel speed and trajectory calculated by the core controller based on visual information to correct each other.

[0231] The communication system is embedded in the core controller and connects the core controller and the operating equipment drive device. In this embodiment, the communication system switches to a communication protocol that matches a specific operating equipment according to the different communication modes of the operating equipment to realize the transmission of signals between the core controller and the operating equipment. Specifically, the communication protocols include Can-BUS, RS-232 / RS-485, and ISO BUS.

[0232] The PTZ control system will ensure that the binocular camera remains stable even on uneven roads inside the facility, thereby ensuring image acquisition quality and optimizing recognition accuracy. The binocular camera PTZ system is used to automatically adjust the posture of the binocular camera to offset the effects of vibration and movement caused by uneven roads inside the facility. The system uses precise motor control to adjust the camera's pitch and yaw angles in real time to maintain the camera's stability and ensure quality and accuracy during image acquisition.

[0233] The gimbal type of this system adopts a three-axis stabilized gimbal, including pitch, yaw and roll control. The specific control method is a closed-loop servo control system, which ensures precise angle adjustment through motor feedback, and at the same time detects and compensates for camera shake caused by device movement or vibration based on the gyroscope and accelerometer sensors integrated in the system.

[0234] Based on this gimbal system, a real-time stabilization control algorithm was developed. The real-time stabilization control algorithm uses data collected from the gyroscope and accelerometer to calculate the necessary adjustment angle and stabilize the camera by controlling the motor on the gimbal. The algorithm can effectively handle sudden vibrations and continuous instability.

[0235] Specifically, the core controller continuously collects the device's motion data from the gyroscope and accelerometer, applies a low-pass filter to process the sensor signal to eliminate high-frequency noise, and uses the PID control algorithm to calculate the pitch, yaw, and roll angles required for gimbal adjustment based on the processed signal. It controls the servo motor on the gimbal and adjusts it according to the calculated angles to stabilize the camera's position.

[0236] The step-down system is the power input part of the system, connected to the power input of the operating equipment, and converts the power voltage to supply the core controller. Specifically, the step-down system can step down the AC 220V, 380V or DC 36-48V voltage to DC 5V, thereby powering the system, including the core controller, binocular camera, communication system and sensor.

[0237] The binocular camera is set on the turret, and the camera is used to collect the label code inside the greenhouse. In this embodiment, the camera is located in the front half of the operating equipment, and its position is relatively fixed consistent with the turret. Specifically, the binocular camera is connected to the core controller via a USB interface.

[0238] The core controller is connected to the speed sensor, communication system, step-down system and camera respectively. The core controller is used to obtain the current position information of the operating equipment according to the distance information, posture information, travel speed and travel trajectory, and make real-time decision adjustment information according to the operation requirements at a certain moment, and send it to the operating equipment drive device through the communication system, so that the operating equipment can realize path planning and autonomous navigation inside the facility, thereby completing the operation requirements in the facility and realizing unmanned and intelligent operation. In this embodiment, the core controller is a Raspberry Pi 5B development board, and the core controller is equipped with a human-computer interaction system. Specifically, the human-computer interaction system is a Raspberry Pi system.

[0239] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0240] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for intelligent unmanned operation of a facility environment based on machine vision, based on an intelligent unmanned operation system for a facility environment based on machine vision, the system comprising: Operating equipment and environmental equipment; The operation equipment includes: a core controller, a binocular camera, a pan / tilt control system, a voltage reduction system, a communication system, an ultrasonic sensor, a gyroscope, an accelerometer and a speed sensor; the environmental equipment includes a tag code and a temperature and humidity sensor; the characteristic is that the facility environment intelligent unmanned operation method includes: Based on ultrasonic sensors, obstacles inside the target facility environment are scanned to determine obstacle information; Based on a binocular camera, an adaptive image enhancement algorithm is used to collect the label code in front of the operating equipment to obtain a label code image; the label code image includes a number of label codes, each of which has a mark; the operating equipment is an intelligent unmanned operating vehicle; Construct a global operation map of the target facility environment based on the label code image and obstacle information; Based on the extended Kalman filter, multi-source data fusion is performed on the label code image and the information obtained by the gyroscope, accelerometer and speed sensor to obtain the positioning information and status of the operating equipment; According to the positioning information and status of the operating equipment, based on the operating requirements of the operating equipment, the improved A algorithm to plan the path of the operating equipment; the improved A The algorithm is based on the traditional A Based on the algorithm, a sliding average algorithm is used to create a neighborhood of each point to achieve path smoothing; Among them, the improved A Algorithms, including: Get the number of neighboring points in the sliding window used to calculate the average; According to each point in the original path , according to the formula Calculate new smooth points within the sliding window :in, represents half of the window value, n is the number of neighboring points, i Indicates the index of the current path point, j Indicates at the current point The offset within the neighborhood of ; Based on the set of generating points, the original path is reconstructed to generate a new smooth path; the set of generating points includes a number of smooth points; the path is reconstructed to build a path from a starting point to an end point; the path uses the parent node information stored in each node to trace back from the end point to the starting point.

2. The method for intelligent unmanned operation of a facility environment based on machine vision according to claim 1, characterized in that: Based on ultrasonic sensors, the obstacles inside the target facility environment are scanned to determine the distance between the obstacles and the sensor, including: The sound velocity is corrected based on the temperature and humidity data detected by the temperature and humidity sensor to obtain a corrected sound velocity V; According to the formula Determine the distance between the obstacle and the sensor; in, is the time difference between the ultrasonic sensor transmitting and receiving signals, , is the temperature, H For humidity.

3. The method for intelligent unmanned operation of facility environment based on machine vision according to claim 2, characterized in that: Based on the binocular camera, an adaptive image enhancement algorithm is used to collect the label code in front of the operating equipment to obtain the label code image, which includes: Get the image taken by the binocular camera; Performing histogram equalization on the image to obtain an image with adjusted brightness; Adopting adaptive threshold technology to perform binarization processing on the brightness-adjusted image to obtain an image with enhanced contour clarity; Based on the contour detection method, label code recognition is performed on the image with enhanced contour clarity to obtain a label code image.

4. The method for intelligent unmanned operation of a facility environment based on machine vision according to claim 3 is characterized in that: Based on the label code image and obstacle information, a global operation map of the target facility environment is constructed, including: According to the label code image collected in real time, the target label code in each frame of the label code image is detected; the target label code is the Aruco label code; According to the Aruco label codes in the label code images of adjacent frames, the pose transformation relationship between the label codes is calculated; Connect the label code transformation relationships in different frames to create a graph structure; The Aruco label codes in the detected image label codes are introduced into the graph structure as nodes, and the edges are constructed based on the calculated label code transformation relationship; The Levenberg-Marquardt algorithm is used to minimize the reprojection error of each key frame and label code point in the graph structure to obtain the optimized Aruco label code and pose; According to the label codes and postures, the global coordinates corresponding to the label codes are integrated into the global point cloud data to construct a point cloud map; Obstacle information is filled into the point cloud map to construct a global operation map of the target facility environment.

5. A machine vision-based intelligent unmanned operation system for facility environment for implementing the unmanned operation method of claim 1, characterized in that: include: Operating equipment and environmental equipment; The operating equipment includes: a core controller, a binocular camera, a pan / tilt control system, a voltage reduction system, a communication system, an ultrasonic sensor, a gyroscope, an accelerometer and a speed sensor; The environmental equipment includes a tag code and a temperature and humidity sensor; The tag code is arranged inside the target facility environment to provide location information; the temperature and humidity sensor is used to monitor the temperature and humidity information of the target facility environment, and to correct the sound velocity data of the ultrasonic sensor; The core controller is connected to the step-down system, the speed sensor and the binocular camera for communication; the system installed on the core controller is a Raspberry Pi system; The binocular camera is used to collect and identify images of each tag code in the target facility environment; The pan / tilt control system is connected to the binocular camera to increase the stability of the binocular camera; The step-down system is connected to the input power supply of the operating equipment and is used to output a DC voltage to provide a stable constant voltage source for the system; The communication system is embedded in the core controller and is used for communication between the core controller and the operating equipment; The ultrasonic sensor is arranged on the front side of the working equipment and is used to detect obstacles in the front area of ​​the working equipment; The gyroscope is used to output the posture information of the operating equipment; The accelerometer is used to output acceleration information of the operating equipment; The rotation speed sensor is used to collect the wheel rotation speed of the working equipment and determine the speed information of the working equipment according to the wheel circumference and the wheel rotation speed.

6. The machine vision-based intelligent unmanned operation system for facility environment according to claim 5, characterized in that: The label code is an Aruco label code.

7. The machine vision-based intelligent unmanned operation system for facility environment according to claim 6, characterized in that: The rotation speed sensor is a Hall sensor for detecting the rotation speed of an object, and the rotation speed sensor is connected to the core controller via a USB to serial port.

8. The machine vision-based intelligent unmanned operation system for facility environment according to claim 7, characterized in that: The gimbal control system adopts a three-axis stabilized gimbal; the three-axis stabilized gimbal includes pitch, yaw and roll control, which is used to detect / compensate for camera shake caused by movement or vibration of the operating equipment based on information obtained by gyroscope and accelerometer sensors.

9. The machine vision-based intelligent unmanned operation system for facility environment according to claim 8, characterized in that: The tag codes in the target facility environment are tag codes that are not posted in accordance with a specific order or rule.

Images