A method to improve the performance of robot 3D vision guidance tasks based on large models

Abstract

This invention discloses a method for improving the performance of 3D vision-guided tasks for robots based on large models. The method includes acquiring three-dimensional information from multiple types of sensors, analyzing this information through a large model to generate an attribute map that characterizes the spatial distribution and characteristics of objects; generating a projected image based on the three-dimensional data to display the detection results of multiple sensors; receiving identification data from the three-dimensional detection results to determine the location coordinates or characteristics of the objects; generating a radar projected image to optimize the parameters of the large model; and dynamically adjusting the model parameters through local optimization for different environmental conditions or object types. This method utilizes the trained model to improve the robot's task execution capability and adaptability in complex environments, enabling high-precision object recognition, path planning, and obstacle avoidance. It is suitable for a variety of scenarios, including industrial, logistics, and service robots.

Description

Technical Field

[0001] The present invention relates to the technical field of robot vision guidance, and in particular to a method for improving the performance of a robot's 3D vision guidance task based on a large model. Background Art

[0002] With the rapid development of robotics, robots are increasingly being used in a variety of complex environments. This is particularly true in areas such as autonomous driving, intelligent logistics, industrial automation, and service robotics. These applications place increasing demands on robots' environmental perception and autonomous decision-making capabilities. In these scenarios, robots require high-precision three-dimensional (3D) visual perception capabilities to perform object recognition, path planning, and task execution in dynamic and uncertain environments. However, existing 3D vision-guided robotics technologies still face several technical challenges, particularly in areas such as data fusion, perception accuracy, and task execution performance.

[0003] First, traditional 3D vision technology typically relies on a single type of sensor, such as LiDAR or stereo cameras, each of which has its own advantages and limitations. For example, LiDAR can provide accurate distance measurement, but it is prone to errors when processing complex object shapes and low-reflectivity environments; while cameras can capture rich color and texture information, but perform poorly in low-light or dynamic scenes. Because a single sensor is difficult to meet the changing application requirements, multi-sensor fusion technology has become a research hotspot in the field of robotic perception in recent years. By fusing data from different sensors, the advantages of various sensors can be combined to improve the robustness and accuracy of the perception system. However, in multi-sensor data fusion, how to effectively process and align data from different sensors remains a technical challenge.

[0004] Secondly, with the rise of deep learning and neural network technologies, more and more robot vision guidance systems have begun to rely on large-scale neural network models (large models). These large models learn complex visual features and environmental relationships by processing large amounts of training data, thereby achieving significant progress in tasks such as object recognition and path planning. However, large models require a large amount of computing resources and data during training, which places higher demands on the hardware platform and training process. At the same time, the generalization ability and real-time performance of large models are also one of the bottlenecks facing current robot vision guidance technology. Although current neural network models can achieve good performance in static scenes, how to improve the model's adaptability and real-time responsiveness in dynamic environments, rapidly changing scenes, or complex tasks remains an urgent problem.

[0005] Furthermore, despite extensive research on 3D vision-guided technology, existing techniques largely focus on processing single tasks or single sensor data. There is still a lack of a unified, systematic approach to how robots can perform comprehensive vision-guided tasks in complex environments, effectively coordinating and executing multiple perception tasks such as object recognition, path planning, and obstacle avoidance. Within the framework of multi-task learning, the key to improving a robot's task execution capabilities lies in how to simultaneously handle the demands of multiple perception tasks and effectively combine the characteristics of different perception data.

[0006] Therefore, the main challenge facing existing technologies is how to improve the perception accuracy, robustness and real-time performance of robots in complex three-dimensional environments through multi-sensor data fusion, deep learning and large model optimization, so that robots can successfully perform multiple and complex tasks in dynamic and changing environments. Summary of the Invention

[0007] In order to solve the above problems in the prior art, the present invention proposes a method for improving the performance of robot 3D vision guidance tasks based on a large model, comprising:

[0008] Acquire three-dimensional information from multiple types of sensors;

[0009] Analyzing the three-dimensional information using a large-scale model to generate one or more attribute graphs representing the spatial location distribution and characteristics of objects in the collected three-dimensional information;

[0010] generating one or more projected images based on the three-dimensional data, wherein the projected images represent three-dimensional detection results from a plurality of sensors;

[0011] receiving identification data of one or more three-dimensional detection results generated by the plurality of sensors, the identification data identifying the position coordinates or object characteristics of an object in a projected image corresponding to the three-dimensional detection results;

[0012] generating a radar projection image based on the radar information, wherein the radar projection image represents the detection results of the object detected by the radar detector;

[0013] updating one or more parameters in a large-scale model using the radar projected image and ground information generated based on the identification data;

[0014] Based on different environmental conditions or object types, large models are locally optimized for specific areas and some parameters of the model are dynamically adjusted.

[0015] The trained large-scale model is applied to the task of 3D robotic vision guidance.

[0016] The three-dimensional information includes lidar data, depth camera data or other types of three-dimensional sensor data

[0017] The attribute map represents the spatial distribution of items, the size, shape, surface structure, or other physical characteristics of the items.

[0018] The projected image is an image obtained by projecting the three-dimensional information from a three-dimensional space onto a two-dimensional plane, and the two-dimensional plane is a visual guidance reference plane of a robot control system.

[0019] The identification data includes the spatial coordinates of the object in three-dimensional space, the object type, the surface characteristics of the object, or the relative position relationship of the objects.

[0020] The radar projected image is obtained by scanning the object with a radar detector and converting it into a two-dimensional image.

[0021] The large model is a deep neural network (DNN) or a convolutional neural network (CNN) model.

[0022] The local optimization includes adjusting parameters in the model according to environmental conditions or object characteristics, and the parameters in the adjusted model include a weight matrix, a bias value, and a learning rate.

[0023] The adjustment formulas for the weight matrix, bias value, and learning rate are:

[0024] W new =W old +△W·(1+λ local (x env ))

[0025] b new =b old +△b·(1+λ local (x env ))

[0026] α new =α old ·(1+λ local (x env ))

[0027] in:

[0028] W new : updated weight matrix;

[0029] W old : Current weight matrix value;

[0030] △W: weight update value calculated by the current gradient descent algorithm;

[0031] b new : updated bias value;

[0032] b old : Current bias value;

[0033] Δb: bias update value calculated by the current gradient descent algorithm;

[0034] α new : updated learning rate;

[0035] α old : current learning rate;

[0036] λ local (x env ): Environmental factor adjustment coefficient.

[0037] Beneficial effects:

[0038] This paper provides a method for improving the performance of robots in 3D vision-guided tasks based on large models. By fusing data from multiple sensor types, the large model generates attribute maps and projected images, accurately characterizing the spatial distribution and characteristics of objects. Local optimization is achieved by dynamically adjusting model parameters. This method significantly improves the robot's object recognition accuracy, path planning capabilities, and task execution efficiency in complex environments. It also exhibits high adaptability and robustness to dynamic environmental changes, and is widely applicable to industrial, logistics, and service robotics fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application, but do not constitute an improper limitation of the present invention. In the drawings:

[0040] Figure 1 A flow chart of the method of the present invention is shown. DETAILED DESCRIPTION

[0041] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The exemplary embodiments and descriptions are only used to explain the present invention but are not intended to limit the present invention.

[0042] As attached Figure 1 As shown, a method of improving the performance of a robot's three-dimensional vision guidance task based on a large model in this application includes the following steps:

[0043] Step 1: Obtain information from various sensors

[0044] In this embodiment, the robot uses multiple sensors working together to comprehensively acquire three-dimensional information about the target environment. These sensors include, but are not limited to, radar detectors, LiDAR sensors, depth cameras, and infrared sensors. Each sensor provides different types of information, ensuring accurate perception and modeling of the environment.

[0045] Specifically, radar detectors sense the presence of surrounding objects by emitting electromagnetic waves and receiving reflected wave signals. The reflected intensity and return time of the radar waves can be used to calculate the distance, shape, and relative position of the object. In complex environments, radar detectors are particularly suitable for detecting objects with strong reflective properties, such as metal surfaces or large objects. They are also highly adaptable to the environment and can operate normally in conditions such as fog, haze, rain, snow, and low light.

[0046] LiDAR sensors generate precise 3D point cloud data by emitting laser beams and receiving the reflected signals. LiDAR's advantage lies in its ability to provide highly accurate distance measurements, making it suitable for building high-resolution 3D models. It is particularly important in scenarios requiring detailed depiction of object surfaces and spatial structures. Using LiDAR, robots can obtain the precise position and outline of objects in their environment, providing a solid data foundation for subsequent path planning and object recognition.

[0047] Furthermore, depth cameras further enrich the acquisition of 3D data by capturing the depth information between objects and the camera. Depth cameras can provide real-time 3D image data from a wider range of viewing angles, enabling robots to perceive their surroundings in real time, identify moving objects, and effectively avoid obstacles.

[0048] Infrared sensors can operate in low-light or no-light environments, detecting an object's thermal radiation to determine its outline and location. They are particularly useful at night or in low-light environments, helping robots navigate and identify objects in dimly lit areas.

[0049] By combining these multiple sensors, the robot can acquire environmental information from various dimensions and angles. These sensors provide radar information (such as reflection intensity and distance data) and three-dimensional data (such as laser point cloud data and depth images), which together form complete environmental perception data. This data comprehensively describes the spatial distribution, shape characteristics, and other relevant attributes of objects in the environment, significantly improving data integrity and accuracy.

[0050] By integrating multiple sensors, robots can overcome the limitations of a single sensor, improve data robustness and reliability, and ensure high-precision perception in complex environments. This multi-sensor data fusion process is the foundation for subsequent tasks such as object recognition, path planning, and obstacle avoidance, and provides the system with powerful environmental perception capabilities.

[0051] Step 2: Analyze 3D information using the large model and generate property maps

[0052] In this embodiment, the acquired three-dimensional information is preprocessed and then input into a large-scale model for analysis. Large-scale models typically use deep neural networks (DNNs) or convolutional neural networks (CNNs), which are capable of efficiently processing and understanding complex spatial information. Through multi-level and multi-dimensional analysis of three-dimensional data, large-scale models can extract key features of objects and generate corresponding attribute maps based on these features, further supporting subsequent task decision-making.

[0053] The acquired 3D data first needs to be preprocessed to remove noise and normalize it. For point cloud data, filtering techniques (such as VoxelGrid filters or statistical filters) are often used to reduce data redundancy and noise. For depth image data, dedistortion, normalization, and missing data filling may be required. This data preprocessing step ensures the accuracy and efficiency of subsequent analysis.

[0054] The preprocessed 3D data is fed into a larger model for analysis. In this embodiment, the larger model is based on a convolutional neural network (CNN) architecture, which includes multiple convolutional layers, pooling layers, and fully connected layers. This network structure is highly effective in processing 3D spatial data, particularly capturing both local features and global spatial structure of objects.

[0055] For point cloud data, the model first processes the point cloud using PointNet or its variants to extract the local geometric features of each point. By stacking multiple convolutional layers, the model is able to capture features of objects at different scales in the point cloud. For image data, the convolutional layers extract local features using a sliding window and use pooling to reduce spatial dimensionality while retaining key information.

[0056] During the model analysis phase, the network extracts key features of the object from the 3D data. These features include information such as the object's shape, surface structure, size, and spatial distribution. For example, the model can identify edges, corners, and surface features from point cloud data, or identify the object's outline and surface texture from a depth image.

[0057] Based on the extracted features, the model generates one or more attribute maps. An attribute map graphically displays information about objects in three-dimensional space. Each attribute map contains the object's location in three-dimensional space, its characteristics (such as its size, shape, and surface roughness), and other relevant information. An attribute map can be thought of as a "map" of an object in space, helping the robot better understand its environment.

[0058] The process of generating a property graph includes:

[0059] Spatial distribution map: reflects the position and arrangement of objects in three-dimensional space. For example, by clustering point clouds, the model can distinguish the distance, position, and relative relationship between objects.

[0060] Feature maps: These map the shape, size, surface texture, and other properties of an object. For example, they can generate an object's outline based on surface normals and geometry, or generate a surface feature map based on texture extraction.

[0061] Density map: A description of the density and surface complexity of an object, which helps understand the physical properties of the object, such as surface roughness.

[0062] These attribute graphs will serve as the basis for the robot's environmental understanding and decision-making, supporting tasks such as path planning, object recognition, and obstacle avoidance.

[0063] The generated attribute map is typically output as a high-dimensional feature map, which can be a two-dimensional image, three-dimensional volume data, or graphical point cloud data. In a robotic system, the attribute map can be presented to subsequent modules (such as path planning or task execution modules) in various ways. For example, spatial distribution maps can be used for obstacle detection and path planning; feature maps can help robots identify object types and determine surface characteristics, thereby classifying objects or selecting appropriate operation strategies.

[0064] For robots that need to perform complex tasks in real-world applications, multiple attribute graphs may be used simultaneously and fused. By combining different types of attribute graphs, robots can understand the environment from multiple dimensions, achieving more accurate object recognition and path planning.

[0065] The model used by the robot is a convolutional neural network (CNN), and the input three-dimensional data is point cloud data. After the input data undergoes the convolution operation of the neural network, the generated feature map can be described by the following formula:

[0066] F=CNN(P)

[0067] Among them, P is the input point cloud data, CNN is the convolutional neural network, and F is the feature map extracted after network processing.

[0068] For the generation of the spatial distribution map, assuming that the object coordinates extracted by the network are X and Y, the spatial distribution map can be constructed based on the coordinate relationship of the object. The formula is as follows:

[0069]

[0070] Where X and Y are the coordinates of the objects in space, and D(X, Y) is the distance between the objects.

[0071] The parameters are explained as follows:

[0072] X: represents the set of coordinates of objects along the X-axis in space. Each object's X-coordinate value corresponds to its position on the X-axis. For example, x1 and x2 are the X-coordinate values ​​of the first and second objects, respectively.

[0073] Y: Represents the set of coordinates of objects along the Y axis in space. Each object's Y coordinate value corresponds to its position on the Y axis. For example, y1 and y2 are the Y coordinate values ​​of the first and second objects, respectively.

[0074] x1: The coordinate value of the first object on the X axis.

[0075] x2: The coordinate value of the second object on the X axis.

[0076] y1: The coordinate value of the first object on the Y axis.

[0077] y2: The coordinate value of the second object on the Y axis.

[0078] D(X, Y): represents the Euclidean distance between two objects on a two-dimensional plane (in the X and Y coordinate system).

[0079] By deeply analyzing 3D data and generating an attribute graph, this embodiment helps robots accurately understand the characteristics of objects in complex environments. The attribute graph provides precise object information for subsequent task decisions, thereby optimizing the efficiency and accuracy of tasks such as path planning, obstacle avoidance, and object recognition.

[0080] In addition, by using deep neural networks to extract features from three-dimensional information, the robot's environmental perception and adaptability can be significantly improved, ensuring that it can complete tasks efficiently in different environments.

[0081] Step 3: Generate a projected image based on the 3D information

[0082] Based on the 3D data collected from different sensors, the system further generates one or more projected images. Projected images are the result of projecting an object or scene in 3D space onto a 2D plane. These projected images show the detection results of different sensors for the same object or scene, including detection results from radar sensors and other types of sensors (such as LiDAR).

[0083] In this embodiment, the basic method for generating a projected image is to project point cloud data or a 3D geometric model in a 3D space onto a 2D plane. Specifically, the projection process can be divided into the following steps:

[0084] Radar data projection: The three-dimensional data captured by the radar sensor is usually generated by the reflected radar waves, which contains the distance information of the object, reflection intensity, etc. In order to generate the projected image, the system first maps these three-dimensional data to a two-dimensional plane. This can be done by projecting the three-dimensional coordinate points of the object in the radar coordinate system according to a specific perspective (such as a top-down angle or a front-view angle). For each three-dimensional point (x, y, z coordinates), it can be converted into coordinates on the two-dimensional plane (x ′ ,y ′ ):

[0085]

[0086] Among them, ProjectionMatrix is ​​the projection matrix. The specific matrix value depends on the selected projection method (for example, positive projection, perspective projection, etc.), x, y, z are the coordinates of the radar data in three-dimensional space, and x ′ ,y ′ is the projected two-dimensional coordinate.

[0087] LiDAR Data Projection: LiDAR data is typically provided as a point cloud, where each point contains its spatial location (x, y, z coordinates). This point cloud data needs to be converted into a two-dimensional representation through a similar projection method. The data projection process for LiDAR point clouds may differ from that for radar data, particularly when dealing with data density and distribution, as LiDAR generally provides higher accuracy. Therefore, when generating a projected image, LiDAR data may require interpolation or filtering techniques to optimize the projection results to improve image accuracy and continuity.

[0088] For lidar data, in addition to standard projection, the system may also need to perform Gaussian filtering or other smoothing on the data to remove noise and ensure image quality after projection.

[0089] The generated projected image shows the spatial distribution and relative positions of objects in three-dimensional space. Specifically, the projected image can reflect the following:

[0090] Spatial position of an object: By projecting the object's three-dimensional coordinates onto a two-dimensional plane, the projected image shows the object's relative position from a specific perspective, helping the robot understand the object's distribution in space.

[0091] Object size: The size of the projected image can reflect the object's size. By analyzing the area occupied by the object in the image, the object's actual size can be estimated. This information is particularly important for object recognition and path planning, especially when multiple objects are present in the environment. The robot can determine whether the object's size will affect the robot's path based on its location.

[0092] Relative Position of Objects: Using projected images generated by multiple sensors, the robot can compare the positions of objects from different perspectives, thereby determining the distance and relative relationships between objects. For example, images from radar and lidar sensors can complement each other to provide more comprehensive information about object layout.

[0093] In this embodiment, the projected images generated by data from different sensors (such as radar and lidar) usually have different accuracy and characteristics. To improve the accuracy and comprehensiveness of the projected images, the projected images from multiple sensors can be fused. This process includes the following steps:

[0094] Alignment: First, the data from different sensors are aligned to the same coordinate system through geometric transformation. Common alignment methods include coordinate transformation based on calibration information or automatic alignment of different sensor data through image registration algorithms.

[0095] Fusion: Once the data is aligned, the system can fuse multiple projected images. Fusion methods include weighted averaging and projection intersection. Weighted averaging assigns different weights based on the reliability of sensor data, combining the strengths of two or more sensors. Projection intersection calculates the intersection of images from different sensors to obtain more accurate object position and spatial information.

[0096] The generated projected image serves as the basis for the robot to understand the environment and can be used for subsequent tasks such as path planning, obstacle avoidance, object recognition, etc. Specifically:

[0097] Path Planning: By projecting images, the robot can clearly see the location and shape of obstacles, providing an accurate map of the environment for the path planning algorithm. Based on these images, the robot can choose the best path to avoid collisions with obstacles.

[0098] Object recognition and classification: Projected images reveal the spatial characteristics of objects, such as location, size, and shape. Robots can use these characteristics to identify and classify objects. For example, based on projected images, robots can distinguish between walls, furniture, and other dynamic objects.

[0099] Environmental reconstruction: After collecting 3D data and generating projected images multiple times, the robot can construct a more complete environmental model based on images from different perspectives, providing support for task execution in complex scenarios.

[0100] Through the above technical means, robots can understand complex environments more efficiently and perform tasks such as object recognition, path planning, and obstacle avoidance, thereby improving the intelligence and adaptability of the system.

[0101] Step 4: Receive labeled data and optimize the model

[0102] In this embodiment, the identification data collected by the robot via sensors is used to optimize an existing large-scale model to improve object recognition accuracy and environmental adaptability. This identification data provides the robot with the spatial coordinates, feature information, and relative positional relationships of objects. This identification data corresponds one-to-one with the projected image generated by the sensor and serves as a key input for model updating and optimization.

[0103] Acquisition of identification data:

[0104] Spatial coordinates and feature information: Identification data includes the coordinates of an object in space. Specifically, the spatial coordinates of each object can be accurately located using data collected by sensors. Identification data also includes object feature information, such as size, shape, reflectivity, and surface texture. Identification data is typically acquired through sensors working in conjunction with the robot's internal control system to generate object identification tags in real time.

[0105] Object Position Relationships: In addition to the spatial coordinates of individual objects, identification data also provides the relative positional relationships between multiple objects. This information enables robots to determine the relative placement of objects and further understand their distribution throughout the environment. This information is particularly important when robots need to plan their paths, effectively helping them avoid obstacles or adjust their routes.

[0106] Matching and calibration of identification data and projected images:

[0107] Matching: Each set of identification data must be accurately matched to the object location in the projected image. Specifically, through calibration of the sensor data, each identification point is matched to the corresponding object feature in the generated projected image. For example, corresponding objects in the projected images of radar and lidar should have the same spatial coordinates and feature information in the identification data.

[0108] Calibration: The calibration process is a core step in ensuring that the identification data matches the projected image. During calibration, a calibration plate or object with known features can be used as a reference to ensure the accuracy of the 3D data captured by the sensor in physical space. Calibration methods can use the least squares method or other mathematical models to adjust the data from different sensors so that they can be accurately mapped to the same reference frame.

[0109] Model update and optimization:

[0110] Parameter Update: After acquiring the identification data, the robot compares the identification data with the object features in the projected image and analyzes the differences. These differences are used to adjust the internal parameters of the larger model. Specifically, the robot calculates errors (such as position error, size error, shape error, etc.) and feeds these errors back to the model, which then updates the model's weights and biases to better adapt to the actual environmental data.

[0111] This process can be performed using the backpropagation algorithm. Specifically, the model calculates a loss function (such as mean square error (MSE)) based on the error between the identified data and the generated projected image, and then uses gradient descent or other optimization algorithms to update the model parameters. The loss function is calculated as:

[0112]

[0113] The parameters are explained as follows:

[0114] L(θ): represents the value of the loss function, which is used to quantify the error between the model's predicted value and the true value.

[0115] During the optimization process, the value of the loss function is continuously reduced through backpropagation and gradient descent algorithms to improve the prediction accuracy of the model.

[0116] θ: represents the set of model parameters, including weights and biases. The model parameters are continuously adjusted during the training process to minimize the loss function L(θ).

[0117] N: represents the number of samples, which represents the number of data points used in a loss calculation. The sum is taken over the entire dataset and the average is taken to measure the overall error of the model on all samples.

[0118] i: represents the sample index, indicating the number of the sample currently being processed, with a value range of 1 ≤ i ≤ N. It is used to calculate all samples one by one and accumulate the errors.

[0119] y i : represents the true value of the i-th sample, that is, the target value provided in the identification data. Indicates the correct answer that the model prediction needs to be close to.

[0120] T(x i ,θ): represents the prediction result of the i-th sample obtained by the model. It represents the output of the model, which is the input data x i The result is determined jointly by the current model parameters θ.

[0121] (y i -f(x i ,θ)) 2 : The square error of the i-th sample. Used to measure the predicted value f(x i ,θ) and the true value y i The squared form ensures that the error is always non-negative while amplifying the effect of larger errors.

[0122] Dynamic Adjustment: To improve recognition accuracy, the system dynamically adjusts based on the differences between the new identification data and the projected image. These adjustments primarily involve fine-tuning model parameters, enabling the model to better adapt to changes in environmental conditions. For example, in complex or dynamically changing environments, the shape or position of objects may change, requiring the model to automatically adjust its parameters based on the new data to maintain high-precision object recognition.

[0123] Model training and validation:

[0124] Training: By continuously receiving new identification data and projected images, the robot continuously optimizes its large model. During training, the model learns the characteristics of objects from a large amount of sample data, gradually improving its recognition capabilities. During each training session, the model adjusts its parameters based on the new identification data to minimize the error between the model output and the actual identification.

[0125] Validation: After each parameter update, the robot can test the model using a validation set to ensure that the updated model performs better in real-world environments. The validation set typically consists of identification data and projected images collected in different scenarios, and is used to evaluate model performance metrics such as precision and recall.

[0126] Ongoing process of model optimization:

[0127] Model optimization is an ongoing process. As the robot performs tasks in different environments, the system continuously receives new identification data, which further drives model optimization. Through continuous training and real-time adjustments, the robot can gradually improve its recognition accuracy and adaptability in various environmental conditions.

[0128] The implementation of step 4 receives identification data and pairs it with the projected image, providing the robot with precise object location and feature information. This identification data not only optimizes the parameters of the large-scale model but also helps the robot adjust its recognition capabilities in real time to better adapt to tasks in different environments. This optimization process provides the robot with powerful adaptive capabilities, enabling accurate object recognition and task execution in constantly changing environments.

[0129] Step 5: Radar projection image generation and optimization

[0130] In this embodiment, radar data is used to generate a radar projection image. The goal is to convert information about objects detected by the radar into a two-dimensional image, simplifying the representation of the object information and providing effective data for further processing and analysis. By projecting radar data into two-dimensional space, the robot can more concisely understand and process data from different sensors, thereby better performing its tasks.

[0131] Generating a radar projected image involves mapping the three-dimensional spatial data acquired from the radar sensor onto a two-dimensional plane. Specifically, radar data typically includes information such as the object's distance, azimuth, and elevation. To generate the projected image, this three-dimensional spatial information must be converted into a two-dimensional representation through a projective transformation.

[0132] Radar Data Format: Data collected by radar sensors typically includes reflection intensity, distance information, and relative position to the object. Distance measurement determines the linear distance of an object from the radar, while azimuth and elevation angles determine the object's orientation in three-dimensional space. This data provides a rough estimate of the object's position in space and its characteristics.

[0133] Projection Transformation: To convert radar data into a two-dimensional image, common projection transformation methods such as polar coordinate transformation or perspective projection are used. In a polar coordinate system, the position of each object is expressed as its distance (r) and azimuth (θ) from the radar. By transforming this data onto a two-dimensional plane, the resulting projected image displays the object's position and size in two-dimensional space.

[0134] Calculate the distance r and azimuth θ of each object relative to the radar based on the radar data;

[0135] Convert the radar data into the corresponding image coordinates in the two-dimensional coordinate system:

[0136] x=r·cos(θ)

[0137] y=r·sin(θ)

[0138] Where r is the distance from the radar to the object, θ is the azimuth of the object relative to the radar, and x and y are the image coordinates on the two-dimensional plane.

[0139] The projected image can concisely express the position of an object in space, has obvious spatial structure information, and is compatible with other types of sensor data (such as lidar data and camera data).

[0140] By combining identification data generated by other sensors (such as LiDAR or depth cameras) with the projected image, the robot can further optimize the model parameters. Identification data includes the spatial coordinates of objects, feature information, and the relative positions of objects. Combining this information with the radar projected image can enhance the model's object recognition accuracy through comparison and calibration.

[0141] Optimizing Parameters: The optimization process is based on comparing the error between the object positions in the radar projection image and the object positions in the identification data. This error is used to further adjust the model's weights and biases, thereby optimizing the representation of objects in the radar projection image. This error can be calculated using common loss functions such as mean squared error (MSE) or cross-entropy loss. This optimization process updates the model parameters using gradient descent or other optimization algorithms.

[0142] Adjusting Object Information: After the radar projection image is generated, object information within the image, such as position, size, or shape, can be adjusted based on the feedback from the identification data. This process ensures effective integration of radar data with data from other sensors, further improving the model's object recognition accuracy. For example, when an object's boundaries are unclear, the model can adjust the object's outline to more closely resemble the real object's shape.

[0143] Fusion of radar data with data from other sensors (such as LiDAR and cameras) is a key step in improving recognition accuracy. To effectively fuse data from different sensors, data alignment techniques, such as registration based on relative coordinate systems, are often employed. Specifically, by aligning the coordinate systems of different sensors, the radar projection image and data generated by other sensors can be fused within the same reference frame, thereby improving overall recognition accuracy.

[0144] Sensor fusion technology: Common sensor fusion methods include Kalman filtering and particle filtering. These methods can effectively integrate information from different sensors and reduce the impact of noise on model optimization to a certain extent. Through these methods, radar data, LiDAR data, and other sensor data can be accurately aligned in space, thereby optimizing object detection results.

[0145] By generating radar projection images and feeding back identification data, the robot can continuously refine its object recognition capabilities, particularly in complex and dynamic environments. Optimizing radar data effectively reduces sensor noise and uncertainty, improving the model's speed and accuracy in identifying objects. This process enables the robot to more quickly and accurately perform tasks such as path planning and obstacle avoidance in a variety of environments.

[0146] Enhanced object recognition capabilities: Through repeated optimization, the object information in the radar projection image can more accurately reflect the object's true location and characteristics. This enables robots to more effectively identify and handle different objects in real-world applications.

[0147] In summary, the radar projection image generation and optimization process in step 5 not only simplifies radar data processing but also improves object recognition accuracy by effectively integrating it with other sensor data. By continuously optimizing the radar projection image and model parameters, the robot can achieve more efficient and accurate task execution in dynamic environments.

[0148] Step 6: Local optimization and task execution

[0149] In this example, a local optimization strategy was employed to improve the robot's performance under varying environmental conditions or object characteristics. The core concept of local optimization is to dynamically adjust certain parameters within a large model based on real-time environmental changes and the specific requirements of the task, enabling the robot to more accurately perform tasks in specific areas or conditions. This optimization process not only improves the robot's adaptability in complex environments but also enhances the accuracy and efficiency of task execution.

[0150] Strategies and methods for local optimization:

[0151] Target Area Identification: The robot uses sensor data (such as radar, LiDAR, and depth cameras) to identify target areas in the current environment in real time, particularly those that are complex or difficult to navigate. For example, narrow passages, areas with large lighting variations, and densely populated areas often complicate path planning or affect object recognition accuracy. By scanning the environment in real time, the robot can identify these specific areas and perform local optimization based on their characteristics.

[0152] Feature Recognition and Analysis: The robot analyzes key features of the target area (such as object shape, size, distance, orientation, and surface roughness) to identify factors that may affect task execution. For example, in areas with large lighting variations, the robot may enhance its ability to recognize object edges; in narrow passages, the robot may optimize its path planning algorithm to ensure accurate obstacle avoidance.

[0153] Implementation steps of local optimization:

[0154] Dynamic Parameter Adjustment: The robot optimizes the target area by adjusting the parameters of the corresponding parts of the larger model. For example, the weights and bias parameters related to object recognition in the model can be dynamically adjusted for complex environments to improve recognition accuracy. If an object with complex shapes or ambiguous features is detected in a certain area, the system can dynamically increase the weight of the model related to shape detection, making object recognition more accurate.

[0155] Local Optimization of Path Planning: For narrow passages or complex terrain, the robot can adjust the parameters of the path planning model to make it more flexible and precise within these specific areas. For example, increasing the sensitivity of obstacle detection and reducing errors in path calculation can help avoid collisions in narrow areas. Furthermore, adjusting the optimization objective of path planning to focus on obstacle avoidance rather than optimizing overall motion efficiency can ensure safe completion of the mission.

[0156] Feedback mechanism of local optimization process:

[0157] Real-time environmental perception: While performing its tasks, the robot continuously acquires new sensor data (such as radar scans and camera images) by perceiving its surroundings in real time. This data is used to assess changes in environmental conditions and determine whether to activate local optimization mechanisms. When the robot enters a new environment or the distribution of objects changes, the system fine-tunes the model based on the new data to ensure the continued effectiveness of the optimization.

[0158] Optimization Verification: To verify the effectiveness of local optimization, the robot uses a feedback mechanism for real-time testing. For example, during object recognition, if the robot detects an error in the recognition of a particular object, it will re-evaluate the optimization parameters for that area and make fine-tuning adjustments. This closed-loop feedback mechanism allows the robot to continuously adjust its optimization strategy and improve task execution accuracy.

[0159] Optimization in areas with varying lighting: In environments with high lighting variations, the robot may experience visual errors due to reflections or shadows. To address this, the robot enhances its sensitivity to object edges and surface features through local optimization in this area. For example, by adjusting the weights of the convolutional layers of a deep learning model to focus more on contours and edges in the image, the robot can improve object detection accuracy in low-light conditions.

[0160] Path Planning in Narrow Aisles: In narrow aisles, robots must precisely control their trajectory to avoid collisions. Through local optimization, the path planning model can adjust its decision-making process in real time, making the robot's navigation more precise. For example, the robot can increase the frequency of monitoring for surrounding obstacles and dynamically adjust its movement speed based on the distance between the obstacle and the robot to ensure safe passage through narrow aisles.

[0161] Local optimization involves more than just adjusting model parameters; it's also closely tied to the model's training process. Through techniques like transfer learning, robots can quickly adapt to new tasks based on previous learning from similar environments. For example, when a robot performs a task in a specific type of environment, local optimization can adjust the model based on existing knowledge, shortening training time and improving task execution efficiency. The use of transfer learning enables robots to respond quickly in ever-changing environments, enhancing system robustness.

[0162] During the dynamic parameter adjustment process, focus on the following parameters: weight matrix, bias value, learning rate, and gradient update.

[0163] Among them, the deep neural network model, the prediction result of the output layer is y pred , the true label is y true , the model's weight matrix is ​​W, the bias is b, the learning rate is α, and the gradient descent update formula is:

[0164]

[0165] ΔW: represents the updated value of the weight matrix W, that is, the magnitude by which the weight matrix needs to be adjusted in each iteration. It is the core calculation in the gradient descent process and is used to optimize the weight parameters of the model.

[0166] α: Represents the learning rate, which controls the step size of model parameter adjustment. A larger α will accelerate convergence, but may cause the parameter update to be too large and miss the optimal solution; a smaller α will slow convergence but improve accuracy.

[0167] Represents the gradient of the loss function L with respect to the weight matrix W. This is the key part of the optimization algorithm and reflects the current model prediction value y pred and the true value y true The error between them, and the contribution of the weight matrix W to the error.

[0168] L(y pred ,y true ): loss function, used to quantify the model prediction value y pred and the true value y true The difference between them. Loss functions include mean squared error (MSE), cross entropy, etc.

[0169] In deep neural networks, bias b is one of the parameters of each layer and is usually used to adjust the output of the activation function, similar to the intercept term in a linear equation. During the gradient descent process, the update rule of bias b is similar to the update rule of weight W. The key is to calculate the gradient of the loss function with respect to the bias through the backpropagation algorithm. The calculation formula for the bias gradient is:

[0170]

[0171] Parameter explanation:

[0172] Δb: Update value of bias, used to adjust bias b in the current optimization step.

[0173] α: learning rate, which controls the step size of parameter adjustment.

[0174] The gradient of the loss function L with respect to the bias b indicates the contribution of the bias to the model prediction error.

[0175] In order to achieve local optimization, this application uses the following formula to dynamically adjust parameters:

[0176] W new =W old +△W·(1+λ local (x env ))

[0177] b new =b old +△b·(1+λ local (x env ))

[0178] in:

[0179] W new : The updated weight matrix represents the weight value of the model after the current optimization step.

[0180] W old: Current weight matrix value, model parameters before this optimization.

[0181] △W: The weight update value calculated by the current gradient descent algorithm (refer to the above formula).

[0182] b new : Updated bias value, which indicates the bias value of the model after the current optimization step.

[0183] b old : Current bias value, the model bias parameter before this optimization.

[0184] Δb: Bias update value calculated by the current gradient descent algorithm.

[0185] λ local (x env ): Environmental factor adjustment coefficient, used to dynamically adjust the parameter update amplitude according to the current environmental conditions. Its value is determined by the environmental characteristic parameter x env The calculations show the impact of environmental complexity on model optimization.

[0186] The specific calculation method is as follows:

[0187]

[0188] in:

[0189] λ local (x env ):Indicates the environmental factor adjustment coefficient, which is used to adjust the update amplitude of weights and biases. When the environmental conditions are complex (such as high obstacle density and drastic changes in light), λ local (x env ) will increase, making the adjustment of model parameters more sensitive.

[0190] x env : Characteristic parameters of the current environment, used to describe indicators of environmental complexity. For example: the change in light intensity, obstacle density, channel width, etc.

[0191] γ: Adjustment coefficient, used to control the sensitivity of environmental factors to weight updates. A larger γ will enhance x env Adjustment coefficient λ local (x env ); a smaller γ will weaken this effect.

[0192] Represents an exponential function used to smooth the environmental characteristic parameter x env The influence of makes the adjustment coefficient more stable in practical applications.

[0193] In complex environments, such as areas with large changes in lighting or dense obstacles, the environmental factor λ in the formula local (x env ) will increase, making the model's parameter adjustment more sensitive in this environment, ensuring that the robot can improve its recognition accuracy when performing tasks in these areas.

[0194] On the contrary, in a simpler or more stable environment, the environmental factor λ local (x env ) is close to 0, the parameter update of the model tends to be stable and will not be over-adjusted, thus ensuring the stability of task execution.

[0195] For narrow passages or areas with significant features of specific objects, the robot can calculate λ based on the characteristic parameters of these areas (such as the shape, size, and spacing of the obstacles). local (x env ) and then adjust the relevant parameters in the model. For example, in a narrow passage, environmental factors may increase the model’s sensitivity to obstacle detection, allowing for better obstacle avoidance.

[0196] During dynamic adjustments, the learning rate α is also adjusted based on changes in environmental factors. When the environment is complex, a smaller learning rate may be required to avoid drastically updating the model and preventing overfitting. In simpler environments, the learning rate can be increased moderately to enhance model training efficiency.

[0197] The formula for dynamically adjusting the learning rate is:

[0198] α new =α old ·(1+λ local (x env ))

[0199] Parameter explanation: α new : The updated learning rate is used to control the step size of the next optimization.

[0200] α old : Current learning rate, which indicates the learning step size of the model before this optimization.

[0201] λ local (x env ): Environmental factor adjustment coefficient, used to dynamically adjust the learning rate according to the complexity of the environment.

[0202] When the robot performs a task, it updates the environmental factor x based on real-time feedback (such as sensor data, task execution status, etc.) env , and then adjust the locally optimized parameters. Through this feedback mechanism, the robot can dynamically adapt to changes in the environment and optimize its parameter configuration in real time.

[0203] Optimization in complex environments: When the robot enters an area with large changes in lighting, x env (Light intensity change) value is large, the calculated λ local (x env ) will increase, and the model will adjust its perception of object edges and surface features, thereby improving object recognition accuracy in low-light conditions.

[0204] Path planning optimization for narrow channels: When the robot passes through a narrow channel, x env (channel width, obstacle density) values ​​are small, and the calculated λ local (x env ) increases, the model will optimize the path planning algorithm to ensure that the robot can avoid obstacles more flexibly and thus avoid collisions.

[0205] Through the above formula, the various parameters work together, and the model can dynamically adjust the optimization amplitude and learning efficiency according to the complexity of the real-time environment, ensuring that the model's responsiveness is improved in complex environments, while maintaining parameter stability in simple environments and avoiding over-adjustment.

[0206] Step 7: Apply the trained model to the robot 3D vision guidance task

[0207] In this example, a large, trained and locally optimized model was successfully applied to a robot's 3D vision-guided task. By optimizing the deep learning model, the robot can efficiently perform multiple tasks, such as object recognition, path planning, and obstacle avoidance, in complex and dynamically changing environments. The model's role in this application is not limited to providing static decision-making; it can also dynamically respond to real-time environmental changes, significantly improving task execution accuracy and system robustness.

[0208] Specifically, the trained and optimized model is deployed into the robot's control system, serving as the core of the robot's perception and decision-making. The robot can now use 3D visual data (e.g., environmental information acquired through sensors like LiDAR and depth cameras) to perceive its spatial environment in real time. During mission execution, the model identifies objects based on real-time sensor data and dynamically adjusts its path planning.

[0209] Object Recognition:

[0210] The robot uses deep learning models to identify objects in its surroundings, accurately detecting both static and dynamic objects. The model processes images or 3D data to identify an object's type, location, size, and other key characteristics, and makes decisions based on task requirements. For example, in a complex warehouse environment, the robot can identify shelves, obstacles, or people and adjust its path or perform avoidance maneuvers.

[0211] Path Planning: Based on object recognition, the robot uses an optimized path planning algorithm to calculate the optimal path in real time. In complex environments, the robot can dynamically adjust its path planning strategy using the model. For example, if an obstacle is detected or the environment changes, the robot can quickly replan its path to avoid collisions and maintain task continuity.

[0212] Obstacle Avoidance: The robot uses an optimized model to accurately detect and avoid obstacles. When the robot encounters an obstacle in a complex environment, the model calculates the relative position and size of the obstacle and, through real-time path adjustment strategies, guides the robot to avoid the obstacle and continue its mission. For example, when navigating a narrow passage, the robot accurately measures the width of the passage based on 3D vision data and adjusts its trajectory to ensure safe passage.

[0213] Environmental Adaptability: The continuously optimized model can flexibly respond to real-time environmental changes. In complex environments, such as those experiencing changes in light intensity, obstacle density, or topology, the model automatically adjusts its parameters, enabling the robot to maintain efficient task execution. Through local optimization and dynamic adjustments, the robot can continuously adapt to changing environments and complete tasks under uncertain environmental conditions.

[0214] Improved robustness: By simulating the diversity of training data and the complexity of the environment, the robot can effectively cope with various unknown and unexpected situations during application. The model can detect and correct errors in task execution in real time through sensor feedback, ensuring that the robot maintains efficient and stable task execution capabilities under different circumstances, significantly improving the robustness of the system.

[0215] In summary, the trained large-scale model has been successfully applied to robotic 3D vision-guided tasks, comprehensively improving the robot's ability to identify, plan, avoid, and adapt in complex environments. Through continuous optimization and local adjustments, the robot can efficiently and accurately perform a variety of tasks, ensuring efficient completion and maintaining stability and safety in various complex environments.

[0216] This approach enables the robot to efficiently perform 3D vision-guided tasks based on multi-sensor data, combined with a large model and local optimization strategies. This approach effectively improves the robot's environmental perception and task execution accuracy, maintaining efficient task processing capabilities in complex environments. Furthermore, the local optimization strategy ensures the robot can adapt in real time to varying task scenarios and object characteristics, enhancing the system's robustness and flexibility.

[0217] The above description is only a preferred embodiment of the present invention. Therefore, any equivalent changes or modifications made according to the structure, characteristics and principles described in the scope of the patent application of the present invention are included in the scope of the patent application of the present invention.

Claims

1. A method for improving the performance of robot 3D vision guidance tasks based on large models, characterized in that: include: Acquire three-dimensional information from multiple types of sensors; Analyzing the three-dimensional information using a large-scale model to generate one or more attribute graphs representing the spatial location distribution and characteristics of objects in the collected three-dimensional information; generating one or more projected images based on the three-dimensional information, wherein the projected images represent three-dimensional detection results from a plurality of sensors; receiving identification data of one or more three-dimensional detection results generated by the plurality of sensors, the identification data identifying the position coordinates or object characteristics of an object in a projected image corresponding to the three-dimensional detection results; generating a radar projection image based on the radar information, wherein the radar projection image represents the detection results of the object detected by the radar detector; updating one or more parameters in a large-scale model using the radar projected image and ground information generated based on the identification data; Based on different environmental conditions or object types, a large model is locally optimized for a specific area, and some parameters of the model are dynamically adjusted; the local optimization includes adjusting the parameters in the model according to the environmental conditions or object characteristics, and the parameters in the adjusted model include weight matrix, bias value, and learning rate; Apply the trained large-scale model to the 3D vision guidance task of the robot; The adjustment formulas for the weight matrix, bias value, and learning rate are: IN new =In old +ΔW (1+λ local (x env )) b new =b old +Δb·(1+λ local (x env )) a new =a old ·(1+λ local (x env )) in: W new : updated weight matrix; W old : Current weight matrix value; ΔW: weight update value calculated by the current gradient descent algorithm; b new : updated bias value; b old : Current bias value; Δb: bias update value calculated by the current gradient descent algorithm; α new : updated learning rate; α old : current learning rate; λ local (x env ): Environmental factor adjustment coefficient; λ local (x env ) The specific calculation method is as follows: Among them, λ local (x env ): represents the environmental factor adjustment coefficient, which is used to adjust the update amplitude of weights and biases; when the environmental conditions are complex, λ local (x env ) will increase, making the adjustment of model parameters more sensitive; x env : Characteristic parameters of the current environment, used to describe the indicators of environmental complexity; γ: Adjustment coefficient, used to control the sensitivity of environmental factors to weight updates; a larger γ will enhance x env Adjustment coefficient λ local (x env ); a smaller γ will weaken this effect; Represents an exponential function used to smooth the environmental characteristic parameter x env The influence of makes the adjustment coefficient more stable in practical applications.

2. The method for improving the performance of a robot's 3D vision-guided task based on a large model according to claim 1, characterized in that: The three-dimensional information includes lidar data or depth camera data.

3. The method for improving the performance of a robot's 3D vision-guided task based on a large model according to claim 1, characterized in that: The attribute map represents the spatial distribution of objects, the size, shape or surface structure of objects.

4. The method for improving the performance of a robot's 3D vision-guided task based on a large model according to claim 1, characterized in that: The projected image is an image obtained by projecting the three-dimensional information from a three-dimensional space onto a two-dimensional plane, and the two-dimensional plane is a visual guidance reference plane of a robot control system.

5. The method for improving the performance of a robot's 3D vision-guided task based on a large model according to claim 1, characterized in that: The identification data includes the spatial coordinates of the object in three-dimensional space, the object type, the surface characteristics of the object, or the relative position relationship of the objects.

6. The method for improving the performance of a robot's 3D vision-guided task based on a large model according to claim 1, characterized in that: The radar projected image is obtained by scanning the object with a radar detector and converting it into a two-dimensional image.

7. The method for improving the performance of a robot's 3D vision-guided task based on a large model according to claim 1, characterized in that: The large model is a deep neural network (DNN) or a convolutional neural network (CNN) model.

Images