AGV robot picking control system based on visual positioning

Abstract

The application relates to the technical field of industrial robots, in particular to an AGV robot material taking control system based on visual positioning. The system comprises a data acquisition module, which is used for acquiring a real-time image sequence of an AGV robot and AGV robot motion state data from an inertial measurement unit (IMU); a feature processing module, which is used for extracting and matching a feature point sequence from the real-time image sequence, and calculating a time sequence optical flow sequence of each feature point; and determining the motion speed of a camera of the AGV robot based on the motion state data; an evaluation and screening module, which is used for performing abnormal analysis and value evaluation on the feature point sequence, and screening out a target feature point; and a positioning control module, which is used for matching the target feature point with preset map data, completing visual positioning of the AGV robot, and generating a control instruction based on a positioning result, so as to adjust the pose of the AGV robot and control a mechanical arm to complete material taking.

Description

Technical Field

[0001] This application belongs to the field of industrial robot technology, specifically relating to a vision-based AGV robot material handling control system. Background Technology

[0002] The visual positioning AGV (Automated Guided Vehicle) material handling control system is an intelligent logistics equipment that integrates computer vision and automatic guidance technologies. It uses cameras to capture environmental images and AI algorithms to identify the position and posture of materials, guiding the AGV robot to accurately grasp and transport them. Because AGV robots are less reliant on fixed markers such as magnetic strips and QR codes, and can adapt to frequently adjusted production lines, they have significant advantages in deployment and adaptability. Furthermore, visual cameras are less expensive than LiDAR, enabling large-scale application in production workshops. Therefore, they are widely used in the field of intelligent manufacturing. The visual positioning AGV material handling control system is moving from the "proof-of-concept" stage to the "large-scale deployment" stage, becoming a standard feature of intelligent manufacturing and smart logistics.

[0003] Controlling the robotic arm is a crucial step in the material handling process of an AGV robot. This control requires target tracking or visual localization. Visual localization determines the target's spatial location, and then, based on this location, the robotic arm is controlled, and the AGV robot's pose is adjusted. This enables the AGV robot to control the material handling process. The accuracy of target localization directly affects the accuracy of the AGV robot's material handling.

[0004] Current positioning methods are based on feature points, such as ground markings or shelf locations. Target feature points are determined through image matching to locate the target material. However, this approach relies on image analysis, and the matching of target feature points only utilizes image-related information. In actual material handling scenarios, the distribution of materials in the actual environment affects the accuracy of location. Feature points at different locations have varying importance. For example, feature points near shelves may have fixed positions, but if they belong to the material itself, their location may change as the material moves (losing their positioning effectiveness), leading to inaccurate positioning. Summary of the Invention

[0005] To address the aforementioned issues, this application provides a vision-based AGV robot material handling control system, which includes: a data acquisition module, a feature processing module, an evaluation and screening module, and a positioning control module. The data acquisition module is used to acquire real-time image sequences of the AGV robot and motion state data of the AGV robot from the inertial measurement unit (IMU). The feature processing module is used to extract and match feature point sequences from the real-time image sequence, and calculate the temporal optical flow sequence of each feature point; and to determine the motion speed of the AGV robot's camera based on the motion state data. The evaluation and screening module is used to perform anomaly analysis and value assessment on the feature point sequence based on the temporal optical flow sequence and the motion speed, and to screen out target feature points. The positioning control module is used to match the target feature points with preset map data to complete the visual positioning of the AGV robot, and generate control commands based on the positioning results to adjust the posture of the AGV robot and control the robotic arm to complete the material picking.

[0006] In one embodiment, the data acquisition module includes: Visual sensors are used to acquire images of the environment; An inertial measurement unit (IMU) is used to collect the acceleration and angular velocity information of the AGV robot to calculate its motion state. The data synchronization unit is used to timestamp and synchronize the environmental image with the motion state information.

[0007] In one embodiment, the feature processing module includes: The image preprocessing submodule is used to perform distortion removal and scale normalization on the real-time image sequence to generate an image sequence to be analyzed. The feature extraction and matching submodule is used to extract image feature points from the image sequence to be analyzed, and to perform feature point matching between adjacent frames using a sparse optical flow algorithm to generate a feature point sequence and its corresponding temporal optical flow sequence. The motion speed projection submodule is used to project the speed information in the motion state data toward the camera to obtain the motion speed of the camera.

[0008] In one implementation, the evaluation and screening module includes a probability calculation submodule, which calculates, for each matched feature point, the probability value of the feature point representing a fixed position in each image frame based on its temporal optical flow and the motion speed of the camera.

[0009] In one implementation, the evaluation and screening module further includes an anomaly analysis submodule, used for: Anomaly detection is performed on the probability value of each matching feature point to obtain the first set of anomalous feature points in each image frame determined by temporal analysis. Anomaly detection is performed on all feature points within each image frame to obtain the second set of anomalous feature points determined by intra-frame analysis within that frame. Based on the intersection-union ratio of the first set of abnormal feature points and the second set of abnormal feature points, the degree of agreement between the temporal analysis and the intra-frame analysis detection results is calculated. Based on the degree of similarity and the proportion of abnormal points in the second set of abnormal feature points in the current frame, the accuracy of feature point analysis in the current frame is calculated.

[0010] In one implementation, the evaluation and screening module further includes a root cause discrimination submodule, used for: Analyze the consistency of the change trend of the probability values ​​of all feature points in the current frame relative to the previous frame to obtain the abnormal performance value of the AGV robot in the current frame. Based on the accuracy of feature point analysis of the current frame and the abnormal performance value of the AGV robot status in the current frame, the possibility of abnormal operation of the AGV robot in the current frame is obtained. The probability of the AGV robot malfunctioning is compared with a first preset threshold to filter out candidate abnormal moments. The candidate abnormal time is compared with the abnormal time detected based on the IMU data mutation point. If they match, the root cause of the abnormality of the feature point in the current frame is determined to be the abnormal movement of the AGV robot. Otherwise, the root cause of the abnormality of the feature point in the current frame is determined to be the invalidity of the feature point itself.

[0011] In one implementation, the feature points include abnormal feature points and normal feature points, and the root cause discrimination submodule is further configured to: Obtain the first mean value of the changing trend of the probability values ​​representing fixed positions of all abnormal feature points in the current frame; Obtain the second mean of the changing trends of the probability values ​​representing fixed positions of all normal feature points in the current frame; Obtain the first standard deviation of the changing trend of the probability values ​​of all abnormal feature points representing fixed positions in the current frame; Obtain the second standard deviation of the changing trend of the probability values ​​of all normal feature points representing fixed positions in the current frame; Based on the first mean, the second mean, the first standard deviation, and the second standard deviation, the abnormal state performance value of the AGV robot in the current frame is obtained.

[0012] In one implementation, the evaluation and screening module further includes a value adjustment submodule, which is used to adaptively adjust the probability value based on the discrimination result of the anomaly root cause of the feature point, and generate a reference value for each feature point in each frame.

[0013] In one implementation, the evaluation and filtering module further includes a weighted filtering submodule, used to perform the following operations for each matching feature point: Calculate the temporal mean of the reference value of the feature point in each frame; Based on the number of frames identified as having optical flow anomalies and the total number of image frames in the optical flow temporal anomaly detection, the percentage of frequencies with normal optical flow is calculated. Based on the time series mean and the frequency proportion, calculate the comprehensive importance weight of the feature point; The comprehensive importance weight is compared with the second preset threshold to filter out target feature points.

[0014] In one embodiment, the positioning control module includes: The visual positioning submodule is used to match the target feature points with the preset map data to calculate the current pose of the AGV robot. The control decision submodule is used to generate travel adjustment instructions for the AGV robot chassis and grasping control instructions for the robotic arm end effector based on the deviation between the current pose and the target material picking pose.

[0015] The embodiments of this application have the following beneficial effects: This application provides a vision-based AGV robot material handling control system, which includes: a data acquisition module, a feature processing module, an evaluation and screening module, and a positioning control module; the data acquisition module is used to acquire real-time image sequences of the AGV robot and motion state data of the AGV robot from the inertial measurement unit (IMU); the feature processing module is used to extract and match feature point sequences from the real-time image sequences, and calculate the temporal optical flow sequence of each feature point; and determine the movement speed of the AGV robot's camera based on the motion state data; the evaluation and screening module is used to perform anomaly analysis and value assessment on the feature point sequences, and screen out target feature points; the positioning control module is used to match the target feature points with preset map data to complete the visual positioning of the AGV robot, and generate control commands based on the positioning results to adjust the AGV robot's pose and control the robotic arm to complete material handling. This application embodiment performs anomaly analysis and value assessment on feature point sequences based on image data and AGV robot motion state data from the inertial measurement unit (IMU), filters out target feature points, and uses the target feature points to match with preset map data to complete the visual positioning of the AGV robot. Based on the positioning results, control commands are generated to adjust the AGV robot's pose and control the robotic arm to complete material picking. Because anomaly analysis and value assessment are performed on the target feature points, and the final positioning points are obtained by filtering the target feature points based on the assessment results, the positioning accuracy and the accuracy of the robotic arm picking up materials are improved.

[0016] Other features and advantages of this application will be described in detail in the following detailed description section. Attached Figure Description

[0017] To more clearly illustrate the implementation schemes of this application, the accompanying drawings used in the implementation schemes will be briefly introduced below. It should be understood that the accompanying drawings only show some implementation schemes of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from the accompanying drawings without creative effort.

[0018] Figure 1 This is a block diagram illustrating a vision-based positioning AGV robot material handling control system according to an exemplary embodiment; Figure 2 This is a block diagram illustrating a data acquisition module according to an exemplary embodiment; Figure 3 This is a block diagram illustrating a feature processing module according to an exemplary embodiment; Figure 4 This is a block diagram illustrating an evaluation and screening module according to an exemplary embodiment; Figure 5 This is a block diagram illustrating a positioning control module according to an exemplary embodiment. Detailed Implementation

[0019] The specific embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this application.

[0020] It should be understood that the term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description.

[0021] It should be noted that the concepts of "first" and "second" mentioned in this application are only used to distinguish different devices, modules, or units, and are not used to limit the order of functions performed by these devices, modules, or units or their interdependencies. The modifications of "one" and "multiple" mentioned in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless explicitly stated in the context, they should be understood as "one or more". In the description of this application, unless otherwise stated, "multiple" means two or more, and other quantifiers are similar; "at least one item", "one item or multiple items", or similar expressions refer to any combination of these items, including any combination of single or multiple items.

[0022] Although operations or steps are described in a specific order in the accompanying drawings in the embodiments of this application, this should not be construed as requiring these operations or steps to be performed in the specific order or serial order shown, or requiring all of the shown operations or steps to be performed to obtain the desired result. In the embodiments of this application, these operations or steps may be performed serially; they may be performed in parallel; or a portion of these operations or steps may be performed.

[0023] The names of the messages or information exchanged between multiple devices in the embodiments of this application are for illustrative purposes only and are not intended to limit the scope of these messages or information. It is understood that before using the technical solutions of the various embodiments of this application, the type, scope of use, and usage scenarios of the personal information involved in this application should be disclosed to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means.

[0024] Figure 1 This is a block diagram illustrating a vision-based positioning AGV robot material handling control system according to an exemplary embodiment. Figure 1 As shown in the figure, this application provides a vision-based AGV robot material handling control system, which may include the following modules: a data acquisition module, a feature processing module, an evaluation and screening module, and a positioning control module.

[0025] The data acquisition module is used to acquire real-time image sequences of the AGV robot and motion state data of the AGV robot from the inertial measurement unit (IMU). For example, the data acquisition module may include: a vision sensor for acquiring environmental images; an IMU for acquiring acceleration and angular velocity information of the AGV robot to calculate its motion state; and a data synchronization unit for timestamping the environmental images and motion state information.

[0026] The feature processing module is used to extract and match feature point sequences from the real-time image sequence, calculate the temporal optical flow sequence of each feature point, and determine the motion speed of the AGV robot's camera based on the motion state data. For example, the feature processing module may include: an image preprocessing submodule, used to perform distortion correction and scale normalization on the real-time image sequence to generate an image sequence to be analyzed; a feature extraction and matching submodule, used to extract image feature points from the image sequence to be analyzed, and use a sparse optical flow algorithm to match feature points between adjacent frames to generate a feature point sequence and its corresponding temporal optical flow sequence; and a motion speed projection submodule, used to project the speed information in the motion state data towards the camera direction to obtain the camera's motion speed.

[0027] The evaluation and screening module is used to perform anomaly analysis and value assessment on the feature point sequence and screen out target feature points. For example, the evaluation and screening module includes a probability calculation submodule, used to calculate the probability value of each matched feature point representing a fixed position in each image frame based on its temporal optical flow and the camera's motion speed. The anomaly analysis submodule is used to: perform anomaly detection on the probability value of each matched feature point to obtain a first set of anomalous feature points determined by temporal analysis in each image frame; perform anomaly detection on all feature points within each image frame to obtain a second set of anomalous feature points determined by intra-frame analysis within that frame; calculate the consistency between the temporal analysis and intra-frame analysis detection results based on the intersection-union ratio (IUU) of the first and second anomalous feature point sets; and calculate the feature point analysis accuracy of the current frame based on the consistency and the proportion of anomalous points in the second anomalous feature point set in the current frame. The root cause discrimination submodule is used to: analyze the consistency of the change trend of the probability values ​​of all feature points in the current frame relative to the previous frame, and obtain the abnormal performance value of the AGV robot in the current frame; based on the accuracy of feature point analysis in the current frame and the abnormal performance value of the AGV robot in the current frame, obtain the probability of abnormal operation of the AGV robot in the current frame; compare the probability of abnormal operation of the AGV robot with a first preset threshold to filter out candidate abnormal moments; compare the candidate abnormal moments with the abnormal moments detected based on IMU data mutation points. If they match, the root cause of the abnormality of the feature points in the current frame is determined to be the abnormal movement of the AGV robot; otherwise, the root cause of the abnormality of the feature points in the current frame is determined to be the invalidity of the feature points themselves. The root cause discrimination submodule is further configured to: obtain the first mean of the changing trend of the probability values ​​of all abnormal feature points representing fixed positions in the current frame; obtain the second mean of the changing trend of the changing trend of the probability values ​​of all normal feature points representing fixed positions in the current frame; obtain the first standard deviation of the changing trend of the changing trend of the changing trend of the probability values ​​of all abnormal feature points representing fixed positions in the current frame; obtain the second standard deviation of the changing trend of the changing trend of the changing trend of the changing trend of the changing trend of the probability values ​​of all normal feature points representing fixed positions in the current frame; and obtain the abnormal performance value of the AGV robot state in the current frame based on the first mean, second mean, first standard deviation, and second standard deviation. The evaluation and screening module also includes a value adjustment submodule, which is used to adaptively adjust the probability values ​​according to the discrimination results of the abnormal root causes of feature points, and generate the reference value of each feature point in each frame. The evaluation and screening module also includes a weighted screening submodule, which performs the following operations for each matched feature point: calculates the temporal mean of the reference value of the feature point in each frame; calculates the frequency proportion of normal optical flow based on the number of frames identified as optical flow abnormalities in optical flow temporal anomaly detection and the total number of image frames; calculates the comprehensive importance weight of the feature point based on the temporal mean and the frequency proportion; and compares the comprehensive importance weight with a second preset threshold to screen out target feature points.

[0028] The positioning control module is used to match the target feature points with preset map data to complete the visual positioning of the AGV robot, and generate control commands based on the positioning results to adjust the AGV robot's pose and control the robotic arm to complete the material picking. For example, the positioning control module may include: a visual positioning submodule, used to match the target feature points with map data to calculate the current pose of the AGV robot; and a control decision submodule, used to generate travel adjustment commands for the AGV robot chassis and grasping control commands for the robotic arm end effector based on the deviation between the current pose and the target material picking pose.

[0029] The embodiments of this application have the following beneficial effects: Based on image data and AGV robot motion state data from the inertial measurement unit (IMU), the embodiments of this application perform anomaly analysis and value assessment on feature point sequences, screen out target feature points, and use the target feature points to match with preset map data to complete the visual positioning of the AGV robot. Based on the positioning results, control commands are generated to adjust the AGV robot's pose and control the robotic arm to complete material picking. Because anomaly analysis and value assessment are performed on the target feature points, and the final positioning points are obtained by screening the target feature points based on the assessment results, the positioning accuracy and the accuracy of the robotic arm picking up materials are improved.

[0030] Figure 2 This is a block diagram illustrating a data acquisition module according to an exemplary embodiment. For example... Figure 2 As shown, the data acquisition module may include the following sub-modules: A visual sensor is used to acquire images of the environment. For example, external environmental data can be obtained through a robot's camera, and this data is acquired in real time.

[0031] An inertial measurement unit (IMU) is used to collect acceleration and angular velocity information of an AGV robot to calculate its motion state. For example, the robot's current motion state data can be obtained through its motion sensors. This data is collected in real time, such as real-time position, speed, acceleration, and angle, to determine the robot's pose.

[0032] The data synchronization unit is used to timestamp and synchronize environmental images and motion state information. This ensures time alignment of the two types of data, providing synchronized and accurate foundational data for subsequent feature processing and localization, thus guaranteeing the consistency and accuracy of localization control.

[0033] Figure 3 This is a block diagram illustrating a feature processing module according to an exemplary embodiment. For example... Figure 3 As shown, the feature processing module may include the following sub-modules: The image preprocessing submodule is used to perform distortion correction and normalization on the real-time image sequence to generate an image sequence to be analyzed. For example, due to the change in field of view during the AGV robot's movement, the localized feature points undergo a certain degree of rotation and image scale change over time. Therefore, it is necessary to transform the image based on the focal length, principal point, distortion coefficients, etc. of the AGV camera at this time, as well as perform scale transformation of the image pyramid to eliminate image distortion. The distortion-corrected image frames are then used as the image sequence to be analyzed subsequently. .

[0034] The feature extraction and matching submodule is used to extract image feature points from the image sequence to be analyzed, and to perform feature point matching between adjacent frames using a sparse optical flow algorithm, generating a feature point sequence and its corresponding temporal optical flow sequence. For example, to achieve visual localization for robot material handling, this localization is mainly based on fixed objects, such as warehouse shelves, ground markings, and other relatively fixed objects. Therefore, a feature point extraction algorithm, such as the corner point Harris algorithm, is used to obtain feature points in the image. A sparse optical flow algorithm is then used on adjacent frames to obtain the optical flow corresponding to the feature points and the matching results between adjacent frames. This can be represented in the following form. . This represents the feature points matched between adjacent frames; This represents the motion optical flow corresponding to the feature points. The above processing is performed on each frame of the image to obtain the sequence of matched temporal feature points in the matched images. and the corresponding temporal optical flow sequence .

[0035] The motion speed projection submodule is used to project the speed information in the motion state data towards the camera direction to obtain the camera's motion speed. For example, the robot's motion characteristics, such as IMU position data, acceleration data, and angle data, as well as the robot camera's focal length and angle data, reflect the impact of the robot's motion characteristics on the camera. Therefore, projecting the robot's motion speed towards the camera direction is used to identify the camera's motion speed, denoted as... The camera's movement reflects the motion characteristics of the captured images. The above operation is performed for each analysis time point to obtain the corresponding temporal sequence of motion features. .

[0036] The above process first performs distortion correction and scale normalization on the real-time image sequence to obtain the image sequence to be analyzed; then, image feature points are extracted from it, and the sparse optical flow algorithm is used to complete the feature point matching between adjacent frames to generate a feature point sequence and a time-series optical flow sequence; finally, the velocity information in the motion state data is projected onto the camera direction to obtain the camera's motion velocity, providing high-quality feature data and motion reference for feature point evaluation and localization.

[0037] Figure 4 This is a block diagram illustrating an evaluation and screening module according to an exemplary embodiment. For example... Figure 4 As shown, the evaluation and screening module may include the following sub-module: probability calculation sub-module.

[0038] The probability calculation submodule is used to calculate the probability of each matched feature point based on its temporal optical flow velocity. Speed ​​of movement of the camera Calculate the probability value of a feature point representing a fixed position in each image frame. For example, feature points represent the probability values ​​of a fixed location in each image frame. It can be obtained from the following formula: Formula 1 in, This indicates the speed of camera motion projection, expressed in pixels. For normalization function, To find the norm.

[0039] Based on motion state data, camera intrinsic parameter matrix, and depth information of feature points, the AGV's three-dimensional motion speed is calculated. The projection onto the image plane is determined. Its implementation is as follows: Specifically, let the AGV's three-dimensional motion speed in the camera coordinate system be... The camera focal length is The coordinates of the feature points in the image are Its depth is Z. Then... Pixel velocity generated on the image plane The following formula can be used for approximate calculation: ; The depth Z of the feature points can be obtained in real time through triangulation during the synchronous SLAM (Simultaneous Localization and Mapping) process, or by querying from a pre-built map.

[0040] Based on probability value The initial assessment determines whether the feature points are stable reference points required for localization, laying the foundation for subsequent anomaly analysis and value assessment.

[0041] In one possible implementation, the evaluation and screening module further includes an anomaly analysis submodule, used for: Anomaly detection is performed on the probability value of each matching feature point to obtain the first set of anomalous feature points in each image frame as determined by temporal analysis. For example, feature point matching is entirely based on image features. Ideally, matched feature points should represent the same specific location in actual three-dimensional space; for instance, temporal feature points in an image frame all represent the same shelf location. However, this is affected by the AGV robot's motion state and its environment. For example, if the robot is currently in a densely packed area of ​​shelf materials, where there are many materials with the same characteristics, the successfully matched feature points in the image frame may not represent the same location in actual three-dimensional space. Alternatively, the robot may be in an abnormal motion state, resulting in inaccurate optical flow at that moment. Therefore, the possibility of obtaining a fixed location for the matched feature points is limited. For time series sequences, anomaly detection algorithms are used to monitor for anomalies and obtain the probability of fixed locations with obvious anomalies. The set of anomalous matching feature points is obtained, and the corresponding image frame number is returned. The above analysis is performed on each feature point to obtain the set of anomalous matching feature points in each frame (the first set of anomalous feature points). ), This is the image frame sequence number.

[0042] Anomaly detection is performed on all feature points within each image frame to obtain the second set of anomalous feature points determined by intra-frame analysis within that frame. For example, the same frame of an image is analyzed to obtain all feature points corresponding to that image. The same anomaly detection algorithm is applied to these feature points to obtain the anomalies at that moment. Anomaly detection is performed on each frame of the image in the above manner. The detected anomalies are obtained by comparing the feature points of the same frame of the image, and the anomaly detection result for that frame is denoted as the second set of anomaly feature points. Let t be the image at frame t. In extreme cases, if the anomaly detection result of the image is zero, the feature point analysis accuracy of the corresponding frame image is directly set to 0.

[0043] Based on the first set of abnormal feature points With the second set of abnormal feature points The Intersection over Union (IoU) is used to calculate the consistency between the detection results from temporal analysis and intra-frame analysis. .

[0044] Formula 2 Where IoU is the intersection-union ratio operation.

[0045] Based on fit and the proportion of anomalies in the second set of anomaly features in the current frame. Calculate the accuracy of feature point analysis for the current frame. For example, the accuracy of feature point analysis in the current frame. It can be obtained from the following formula: Formula 3 in, The proportion of outliers in the second set of anomalous feature points in the current frame can be determined by the ratio of the number of feature points in the second set of anomalous feature points to the total number of feature points in the current frame. It is an exponential function with the natural number e as its base.

[0046] When anomalies detected by two different methods within the same frame are obtained, a higher degree of overlap (higher similarity) and a smaller proportion of anomalies at that moment indicate that different feature analysis methods yield the same feature point analysis results, thus indicating higher accuracy of feature point analysis at that moment. Conversely, if there is a significant difference between the number of anomalies obtained through image frame analysis and those obtained through temporal matching feature points (significantly lower overlap), it indicates a large discrepancy in the anomaly detection results obtained by the two methods, resulting in lower accuracy of feature point analysis at that moment.

[0047] In one possible implementation, the evaluation and screening module further includes a root cause discrimination submodule, used for: Analyze the consistency of the change trend of the probability values ​​of all feature points in the current frame relative to the previous frame to obtain the abnormal state performance value of the AGV robot in the current frame. .

[0048] Accuracy of feature point analysis based on the current frame And the abnormal performance value of the AGV robot in the current frame To determine the likelihood of an AGV robot malfunctioning in the current frame. For example, the possibility of the AGV robot malfunctioning in the current frame. It can be obtained from the following formula: Formula 4 in, This is the normalization function.

[0049] The possibility of AGV robot malfunction Candidate abnormal moments are filtered out by comparing them with a first preset threshold (e.g., 0.8, an empirical value). For example, the probability of an AGV robot malfunctioning can be considered. The time of the corresponding frame that is greater than the first preset threshold is regarded as the candidate abnormal time of the AGV robot's operation.

[0050] The candidate abnormal moments are compared with the abnormal moments detected based on IMU data mutation points. If they match, the root cause of the current frame feature point anomaly is determined to be abnormal movement of the AGV robot; otherwise, the root cause of the current frame feature point anomaly is determined to be that the feature point itself is invalid.

[0051] Feature points are not actually of reference value; that is, the feature points themselves are invalid. For example, although a feature point may be of high importance at other times, indicating that it may be in a fixed position, such as warehouse materials that have not been taken away, the location of the material is not stable because the material may be taken away at any time, causing the feature point at that location to be missing. The current matching feature point is considered an abnormal feature point, while normal feature points refer to the remaining feature points other than abnormal feature points, because the material taking process has affected the feature point at that location.

[0052] When the AGV robot is in a special working state, such as obvious turning, the camera may change significantly within adjacent time periods. This may also cause the feature points of the frame image to be abnormal. However, the three-dimensional space corresponding to the feature point actually represents the same positioning position. As a result, the feature point that should be normal is regarded as an abnormal point due to the robot's working state. That is, the root cause of the abnormal feature points in the current frame is the abnormal movement of the AGV robot.

[0053] In one possible implementation, the feature points include abnormal feature points and normal feature points. The root cause discrimination submodule is further used for: Obtain the probability values ​​of all abnormal feature points representing fixed locations in the current frame. The first mean of the trend of change δa Wherein, the trend δa represents the adjacent probability values ​​in the probability value sequence. A sequence composed of the differences. Obtain the probability values ​​of all normal feature points representing fixed positions in the current frame. Trend of change Second mean .

[0054] Obtain the probability values ​​of all abnormal feature points representing fixed locations in the current frame. The first standard deviation of the trend δa .

[0055] Obtain the probability values ​​of all normal feature points representing fixed positions in the current frame. Trend of change Second standard deviation .

[0056] Based on the first mean Second mean First standard deviation and the second standard deviation Get the abnormal status value of the AGV robot in the current frame. .

[0057] Formula 5 in, This represents a non-zero constant, used to avoid the case where the denominator of a fraction is zero.

[0058] If the AGV robot is in an abnormal or special working state, all feature points at that moment will have a large error, thus affecting the probability value of the fixed position corresponding to the matching feature point at that moment. It decreases significantly. Therefore, it has a fixed positional probability value at this moment. The uniform trend of change is observed. By calculating the absolute value of the sum of the two means, the overall deviation of outliers from normal points in terms of the trend of change can be measured. The larger this absolute value, the more consistent the trend of change of the probability values ​​of all feature points (whether normal or outlier) (e.g., a simultaneous sharp decrease), which is usually due to drastic changes in the state of the AGV itself. The smaller the sum of the standard deviations, the more consistent this trend of change is within various types of feature points.

[0059] In one possible implementation, the evaluation and screening module further includes a value adjustment submodule, used to adjust the probability value based on the discrimination result of the anomaly root cause of the feature points. Adaptive adjustments are made to generate the reference value of each feature point in each frame. For example, the reference value of each feature point in each frame. It can be obtained from the following formula: If it is determined to be abnormal robot movement, then ; Formula 6 If the feature point is determined to be invalid, then ; Formula 7 If it is not an abnormal time, then ; Formula 8 Where exp is an exponential function with the natural number e as its base.

[0060] Based on the identification results of the anomaly root cause of the feature point, the probability value of the feature point is adaptively adjusted, and a corresponding reference value is generated for each feature point in each frame, so that the value assessment of the feature point is more in line with the actual situation and the interference of the anomaly root cause on the value judgment of the feature point is avoided.

[0061] In one possible implementation, the evaluation and filtering module further includes a weighted filtering submodule, which performs the following operations for each matching feature point: Calculate the temporal mean of the reference value g of the feature point in each frame. .

[0062] Based on the number of frames n identified as having optical flow anomalies in the optical flow temporal anomaly detection (i.e., for this feature point, the number of frames n identified as abnormal by the aforementioned temporal analysis out of all image frames containing it) and the total number of image frames N, the percentage of frequencies with normal optical flow b is calculated. For example, the percentage of frequencies with normal optical flow b can be obtained by the following formula: Formula 9 Where N is not zero.

[0063] Based on time series mean Calculate the overall importance weight w of the feature point based on its frequency proportion b. For example, the overall importance weight w of a feature point can be obtained using the following formula: Formula 10 Where norm is the normalization function.

[0064] The overall importance weight w is compared with a second preset threshold (e.g., 0.8, an empirical value) to filter out target feature points. For example, feature points with an overall importance weight w greater than the second preset threshold can be used as target feature points for final visual positioning, ensuring that the feature points used for positioning have high stability and high reference value, thereby improving positioning accuracy and the accuracy of the robotic arm's material handling.

[0065] The first and second preset thresholds in this embodiment can be calibrated using test datasets collected under typical working scenarios. Specifically, high-quality and low-quality feature points, as well as the times of normal and abnormal AGV movement, are manually labeled in the dataset. Then, by plotting the Receiver Operational Characteristic (ROC) curve, the point achieving the best balance between precision and recall is selected as the final threshold.

[0066] Figure 5 This is a block diagram illustrating a positioning control module according to an exemplary embodiment. For example... Figure 5 As shown, the positioning control module may include the following sub-modules: The visual positioning submodule is used to match target feature points with map data to calculate the current pose of the AGV robot. For example, matching target feature points with map data to calculate the current pose of the AGV robot is an existing technology, such as triangulation or trilateration, and will not be elaborated upon here.

[0067] The control decision submodule is used to generate travel adjustment commands for the AGV robot chassis and grasping control commands for the robotic arm end effector based on the deviation between the current pose and the target material picking pose. For example, generating travel adjustment commands for the AGV robot chassis and grasping control commands for the robotic arm end effector based on the deviation between the current pose and the target material picking pose is existing technology, such as Stanley control or PID control, and will not be elaborated further here.

[0068] The above process enables AGV robots to achieve precise positioning and efficient material handling.

[0069] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application.

Claims

1. A vision-based positioning-based AGV robot material handling control system, characterized in that, The system includes: a data acquisition module, a feature processing module, an evaluation and screening module, and a positioning and control module; The data acquisition module is used to acquire real-time image sequences of the AGV robot and motion state data of the AGV robot from the inertial measurement unit (IMU). The feature processing module is used to extract and match feature point sequences from real-time image sequences and calculate the temporal optical flow sequence of each feature point; and to determine the motion speed of the AGV robot's camera based on motion state data. The evaluation and screening module is used to perform anomaly analysis and value assessment on feature point sequences based on temporal optical flow sequences and motion velocity, and to screen out target feature points. The positioning control module is used to match target feature points with preset map data to complete the visual positioning of the AGV robot, and generate control commands based on the positioning results to adjust the AGV robot's pose and control the robotic arm to complete the material picking. The evaluation and screening module includes a probability calculation submodule, which calculates the probability value of each matching feature point representing a fixed position in each image frame based on its temporal optical flow and the camera's motion speed. The evaluation and screening module also includes an anomaly analysis submodule, which is used to: perform anomaly detection on the probability value of each matching feature point to obtain a first set of anomalous feature points in each image frame determined by temporal analysis; perform anomaly detection on all feature points in each image frame to obtain a second set of anomalous feature points in that frame determined by intra-frame analysis; calculate the consistency between the temporal analysis and intra-frame analysis detection results based on the intersection-union ratio of the first and second anomalous feature point sets; and calculate the feature point analysis accuracy of the current frame based on the consistency and the proportion of anomalous points in the second anomalous feature point set in the current frame. The evaluation and screening module also includes a root cause discrimination submodule, which is used to: analyze the consistency of the change trend of the probability values ​​of all feature points in the current frame relative to the previous frame, and obtain the abnormal performance value of the AGV robot in the current frame; based on the accuracy of feature point analysis in the current frame and the abnormal performance value of the AGV robot in the current frame, obtain the probability of abnormal operation of the AGV robot in the current frame; compare the probability of abnormal operation of the AGV robot with a first preset threshold to filter out candidate abnormal moments; compare the candidate abnormal moments with the abnormal moments detected based on IMU data mutation points. If they match, the root cause of the abnormality of the feature points in the current frame is determined to be the abnormal movement of the AGV robot; otherwise, the root cause of the abnormality of the feature points in the current frame is determined to be the invalidity of the feature points themselves. The evaluation and screening module also includes a value adjustment submodule, which is used to adaptively adjust the probability value based on the judgment result of the anomaly root cause of the feature point, and generate the reference value of each feature point in each frame.

2. The AGV robot material handling control system based on vision positioning according to claim 1, characterized in that, The data acquisition module includes: Visual sensors are used to acquire images of the environment; An inertial measurement unit (IMU) is used to collect the acceleration and angular velocity information of the AGV robot to calculate its motion state. The data synchronization unit is used to timestamp and synchronize the environmental image with the motion state information.

3. The AGV robot material handling control system based on vision positioning according to claim 1, characterized in that, The feature processing module includes: The image preprocessing submodule is used to perform distortion removal and scale normalization on the real-time image sequence to generate an image sequence to be analyzed. The feature extraction and matching submodule is used to extract image feature points from the image sequence to be analyzed, and to perform feature point matching between adjacent frames using a sparse optical flow algorithm to generate a feature point sequence and its corresponding temporal optical flow sequence. The motion speed projection submodule is used to project the speed information in the motion state data toward the camera to obtain the motion speed of the camera.

4. The AGV robot material handling control system based on vision positioning according to claim 1, characterized in that, The feature points include abnormal feature points and normal feature points. The root cause discrimination submodule is further used for: Obtain the first mean value of the changing trend of the probability values ​​representing fixed positions of all abnormal feature points in the current frame; Obtain the second mean of the changing trends of the probability values ​​representing fixed positions of all normal feature points in the current frame; Obtain the first standard deviation of the changing trend of the probability values ​​of all abnormal feature points representing fixed positions in the current frame; Obtain the second standard deviation of the changing trend of the probability values ​​of all normal feature points representing fixed positions in the current frame; Based on the first mean, the second mean, the first standard deviation, and the second standard deviation, the abnormal state performance value of the AGV robot in the current frame is obtained.

5. The AGV robot material handling control system based on vision positioning according to claim 1, characterized in that, The evaluation and filtering module also includes a weighted filtering submodule, which performs the following operations for each matched feature point: Calculate the temporal mean of the reference value of the feature point in each frame; Based on the number of frames identified as having optical flow anomalies and the total number of image frames in the optical flow temporal anomaly detection, the percentage of frequencies with normal optical flow is calculated. Based on the time series mean and the frequency proportion, calculate the comprehensive importance weight of the feature point; The comprehensive importance weight is compared with the second preset threshold to filter out target feature points.

6. The AGV robot material handling control system based on vision positioning according to claim 1, characterized in that, The positioning control module includes: The visual positioning submodule is used to match the target feature points with the preset map data to calculate the current pose of the AGV robot. The control decision submodule is used to generate travel adjustment instructions for the AGV robot chassis and grasping control instructions for the robotic arm end effector based on the deviation between the current pose and the target material picking pose.

Images