Method for improving 3D visual guidance task performance of robot based on large model

Abstract

The invention discloses a method for improving the 3D visual guidance task performance of a robot based on a large model, and the method comprises the steps: obtaining the three-dimensional information of various types of sensors, analyzing the three-dimensional information through the large model, generating an attribute graph, and representing the spatial distribution and characteristics of an object; generating a projection image based on the three-dimensional data, and displaying detection results of various sensors; receiving identification data of the three-dimensional detection result, and determining position coordinates or characteristics of the article; generating a radar projection image, and optimizing parameters of the large model; model parameters are dynamically adjusted through local optimization according to different environment conditions or object types. According to the method, the task execution capability and adaptability of the robot in a complex environment are improved by utilizing the trained model, high-precision object recognition, path planning and obstacle avoidance can be realized, and the method is suitable for various scenes such as industry, logistics and service robots.

Description

Technical Field

[0001] The present invention relates to the field of robot vision guidance technology, and in particular to a method for improving the performance of robot 3D vision guidance tasks based on a large model. Background Art

[0002] With the rapid development of robotics technology, robots are increasingly used in a variety of complex environments, especially in the fields of autonomous driving, intelligent logistics, industrial automation and service robots. Robots are required to have more stringent requirements on environmental perception and autonomous decision-making when performing tasks. In these application scenarios, robots need to have high-precision three-dimensional (3D) visual perception capabilities to perform object recognition, path planning and task execution in dynamic and uncertain environments. However, existing robot 3D vision guidance technology still faces some technical challenges, especially in data fusion, perception accuracy and task execution performance.

[0003] First, traditional 3D vision technology usually relies on a single type of sensor, such as LiDAR or stereo cameras, each of which has its own advantages and limitations. For example, LiDAR can provide accurate distance measurement, but it is prone to errors when dealing with complex object shapes and low reflectivity environments; while cameras can capture rich color and texture information, but perform poorly in low light or dynamic scenes. Since a single sensor is difficult to meet the changing application requirements, multi-sensor fusion technology has become a research hotspot in the field of robot perception in recent years. By fusing data from different sensors, the advantages of various sensors can be combined to improve the robustness and accuracy of the perception system. However, in multi-sensor data fusion, how to effectively process and align data from different sensors remains a technical challenge.

[0004] Secondly, with the rise of deep learning and neural network technology, more and more robot vision guidance systems have begun to rely on large-scale neural network models (large models). These large models learn complex visual features and environmental relationships by processing a large amount of training data, thereby making significant progress in tasks such as object recognition and path planning. However, large models require a lot of computing resources and data during the training process, which places higher requirements on the hardware platform and the training process. At the same time, the generalization ability and real-time problems of large models are also one of the bottlenecks facing the current robot vision guidance technology. Although the current neural network model can achieve good performance in static scenes, how to improve the adaptability and real-time response ability of the model in dynamic environments, rapidly changing scenes or complex tasks is still a problem that needs to be solved.

[0005] In addition, although there have been many studies on 3D vision guidance technology, most existing technologies focus on the processing of a single task or a single sensor data. There is still a lack of a unified and systematic method for how robots can perform comprehensive vision guidance tasks in complex environments and efficiently coordinate and execute multiple perception tasks such as object recognition, path planning, and obstacle avoidance. In the framework of multi-task learning, how to handle the needs of multiple perception tasks at the same time and effectively combine the characteristics of different perception data is the key to improving the robot's task execution capabilities.

[0006] Therefore, the main challenge facing existing technologies is how to improve the perception accuracy, robustness and real-time performance of robots in complex three-dimensional environments through multi-sensor data fusion, deep learning and large model optimization, so that robots can successfully perform multiple and complex tasks in dynamic and changing environments. Summary of the invention

[0007] In order to solve the above problems in the prior art, the present invention proposes a method for improving the performance of robot 3D vision guidance tasks based on a large model, comprising:

[0008] Acquire three-dimensional information from multiple types of sensors;

[0009] Analyzing the three-dimensional information using a large model to generate one or more attribute graphs to represent the spatial location distribution and item characteristics of the items in the collected three-dimensional information;

[0010] generating one or more projected images based on the three-dimensional data, wherein the projected images represent three-dimensional detection results from a plurality of sensors;

[0011] Receiving identification data of one or more three-dimensional detection results generated by the plurality of sensors, the identification data identifying the position coordinates or object characteristics of an object in a projected image corresponding to the three-dimensional detection result;

[0012] Generate a radar projection image based on the radar information, wherein the radar projection image represents the detection results of the object detected by the radar detector;

[0013] updating one or more parameters in a large-scale model using the radar projection image and ground information generated based on the identification data;

[0014] Based on different environmental conditions or object types, large models are locally optimized for specific areas and some parameters of the model are dynamically adjusted.

[0015] The trained large-scale model is applied to the robotic 3D vision guidance task.

[0016] The 3D information includes lidar data, depth camera data or other types of 3D sensor data

[0017] The attribute map represents the spatial distribution of objects, the size, shape, surface structure, or other physical characteristics of the objects.

[0018] The projected image is an image obtained by projecting the three-dimensional information from a three-dimensional space onto a two-dimensional plane, and the two-dimensional plane is a visual guidance reference plane of a robot control system.

[0019] The identification data includes the spatial coordinates of the object in three-dimensional space, the object type, the surface characteristics of the object, or the relative position relationship of the objects.

[0020] The radar projected image is obtained by scanning the object through a radar detector and converting it into a two-dimensional image.

[0021] The large model is a deep neural network (DNN) or a convolutional neural network (CNN) model.

[0022] The local optimization includes adjusting parameters in the model according to environmental conditions or object characteristics, and the parameters in the adjusted model include a weight matrix, a bias value, and a learning rate.

[0023] The adjustment formulas for the weight matrix, bias value, and learning rate are:

[0024] W new =W old +△W·(1+λ local (x env ))

[0025] b new =b old +△b·(1+λ local (x env ))

[0026] α new =α old ·(1+λ local (x env ))

[0027] in:

[0028] W new : updated weight matrix;

[0029] W old : Current weight matrix value;

[0030] △W: weight update value calculated by the current gradient descent algorithm;

[0031] b new : Updated bias value;

[0032] b old : Current bias value;

[0033] Δb: bias update value calculated by the current gradient descent algorithm;

[0034] α new : updated learning rate;

[0035] α old : current learning rate;

[0036] λ local (x env ): Environmental factor adjustment coefficient.

[0037] Beneficial effects:

[0038] The present invention provides a method for improving the performance of 3D vision-guided tasks of robots based on large models. By fusing data from multiple types of sensors, large models are used to generate attribute graphs and projected images, accurately characterizing the spatial distribution and characteristics of objects, and dynamically adjusting model parameters to achieve local optimization. This method significantly improves the robot's object recognition accuracy, path planning ability, and task execution efficiency in complex environments. It also has high adaptability and robustness to dynamic changes in the environment and can be widely used in the fields of industry, logistics, and service robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present application, but do not constitute an improper limitation of the present invention. In the drawings:

[0040] Figure 1 A flow chart of the method of the present invention is shown. DETAILED DESCRIPTION

[0041] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments, wherein the illustrative embodiments and descriptions are only used to explain the present invention but are not intended to limit the present invention.

[0042] As attached Figure 1 As shown, a method of improving the performance of a robot's three-dimensional vision guidance task based on a large model in the present application includes the following steps:

[0043] Step 1: Get information from various sensors

[0044] In this embodiment, the robot uses multiple sensors to work together to fully obtain three-dimensional information of the target environment. The sensors used include but are not limited to radar detectors, LiDAR sensors, depth cameras, and infrared sensors. Each sensor can provide different types of information to ensure accurate perception and modeling of the environment.

[0045] Specifically, radar detectors can sense the presence of surrounding objects by emitting electromagnetic waves and receiving reflected wave signals. The reflection intensity and return time of radar waves can be used to calculate the distance, shape and relative position of objects. In complex environments, radar detectors are particularly suitable for detecting objects with strong reflective properties, such as metal surfaces or large objects. They are also highly adaptable to the environment and can work normally in conditions such as haze, rain, snow or low light.

[0046] The LiDAR sensor obtains accurate 3D point cloud data by emitting laser beams and receiving the returned reflected signals. The advantage of LiDAR is that it can provide high-precision distance measurement and is suitable for building high-resolution 3D models. It is especially important in scenarios that require detailed depiction of the surface and spatial structure of objects. Through LiDAR, the robot can obtain the precise position and contour information of each object in the environment, thus providing a solid data foundation for subsequent path planning and object recognition.

[0047] In addition, the depth camera further enriches the acquisition of 3D data by capturing the depth information between the object and the camera. The depth camera can provide real-time 3D image data in a wider range of viewing angles, allowing the robot to perceive its surroundings in real time, identify moving objects, and effectively avoid obstacles.

[0048] Infrared sensors can work in low-light or no-light environments, and can detect the thermal radiation information of objects to obtain the outline and position of objects. Infrared sensors are particularly suitable for nighttime or low-light environments, and can help robots complete navigation and object recognition tasks at night or in dim places.

[0049] Through the collaborative work of the above multiple sensors, the robot can obtain environmental information from different dimensions and angles. The radar information (such as reflection intensity, distance data) and three-dimensional data (such as laser point cloud data, depth images, etc.) provided by these sensors together constitute complete environmental perception data. These data can fully describe the spatial distribution, shape characteristics and other related attributes of objects in the environment, significantly improving the integrity and accuracy of the data.

[0050] Through the fusion of multiple sensors, the robot can make up for the limitations of a single sensor, improve the robustness and reliability of data, and ensure high-precision perception in complex environments. This multi-sensor data fusion process is the basis for the smooth implementation of subsequent tasks such as object recognition, path planning, and obstacle avoidance, and provides the system with powerful environmental perception capabilities.

[0051] Step 2: Analyze 3D information using large models and generate property maps

[0052] In this embodiment, the acquired three-dimensional information is preprocessed and then input into a large model for analysis. Large models usually use deep neural networks (DNN) or convolutional neural networks (CNN), which can efficiently process and understand complex spatial information. Through multi-level and multi-dimensional analysis of three-dimensional data, large models can extract key features of objects and generate corresponding attribute graphs based on these features, further providing support for subsequent task decisions.

[0053] The collected 3D data first needs to be preprocessed to remove noise and normalize it. For point cloud data, filtering techniques (such as VoxelGrid filters or statistical filters) are usually used to reduce data redundancy and noise. For depth image data, dedistortion, normalization, and missing data completion may be required. The data preprocessing step ensures the accuracy and efficiency of subsequent analysis.

[0054] The pre-processed 3D data will be fed into a large model for analysis. In this embodiment, the large model is based on a convolutional neural network (CNN) architecture, which includes multiple convolutional layers, pooling layers, and fully connected layers. This network structure can effectively process 3D spatial data, especially capable of capturing local features and global spatial structures of objects.

[0055] For point cloud data, the model first processes the point cloud through PointNet or its variants to extract the local geometric features of each point. By stacking multiple convolutional layers, the model can capture the features of objects of different scales in the point cloud. For image data, the convolutional layer extracts local features through sliding windows and reduces the spatial dimension through pooling operations while retaining key information.

[0056] In the model analysis phase, the network extracts key features of the object based on the 3D data. These features include information such as the object’s shape, surface structure, size, and spatial distribution. For example, the model can identify the edges, corners, and surface features of an object from point cloud data, or the outline and surface texture of an object from a depth image.

[0057] Based on the extracted features, the model generates one or more attribute graphs. Attribute graphs are the result of displaying object information in three-dimensional space in a graphical way. Each attribute graph contains the location distribution of objects in three-dimensional space, object characteristics (such as the size, shape, surface roughness, etc. of the object), and other related information. Attribute graphs can be regarded as "maps" of objects in space, helping robots better understand their environment.

[0058] The process of generating a property graph includes:

[0059] Spatial distribution map: reflects the position and arrangement of objects in three-dimensional space. For example, by clustering point clouds, the model can distinguish the distance, position, and relative relationship between objects.

[0060] Feature map: Displays the shape, size, surface texture and other properties of an object. For example, based on the surface normal and geometry, a contour map of an object is generated, or based on texture extraction, a surface feature map of an object is generated.

[0061] Density map: A description of the density and surface complexity of an object, which helps understand the physical properties of the object, such as surface roughness.

[0062] These attribute graphs will serve as the basis for the robot's environmental understanding and decision-making, supporting the robot's path planning, object recognition, obstacle avoidance and other tasks.

[0063] The generated attribute map is usually output in the form of a high-dimensional feature map, which can be a two-dimensional image, three-dimensional volume data, or graphical point cloud data. In a robotic system, the attribute map can be presented to subsequent modules (such as path planning or task execution modules) in different ways. For example, the spatial distribution map can be used for obstacle detection and path planning; the feature map can help the robot identify the type of object, judge the surface characteristics of the object, and then classify it or choose the appropriate operation strategy.

[0064] For robots that need to perform complex tasks in practical applications, multiple attribute graphs may be used simultaneously and fused. By combining different types of attribute graphs, the robot can understand the environment from multiple dimensions and achieve more accurate object recognition and path planning.

[0065] The model used by the robot is a convolutional neural network (CNN), and the input three-dimensional data is point cloud data. After the input data is convolved by the neural network, the generated feature map can be described by the following formula:

[0066] F=CNN(P)

[0067] Among them, P is the input point cloud data, CNN is the convolutional neural network, and F is the feature map extracted after network processing.

[0068] For the generation of the spatial distribution map, assuming that the object coordinates extracted by the network are X and Y, the spatial distribution map can be constructed according to the coordinate relationship of the object. The formula is as follows:

[0069]

[0070] Among them, X and Y are the coordinates of the objects in space, and D(X, Y) is the distance between the objects.

[0071] The parameters are explained as follows:

[0072] X: represents the coordinate set of the object along the X axis in space. The X coordinate value of each object corresponds to its position on the X axis, for example, 1 and x 2 They are the X coordinate values ​​of the first object and the second object respectively.

[0073] Y: represents the coordinate set of the object along the Y axis in space. The Y coordinate value of each object corresponds to its position on the Y axis, for example, 1 and 2 They are the Y coordinate values ​​of the first object and the second object respectively.

[0074] x 1 : The coordinate value of the first object on the X-axis.

[0075] x 2 : The coordinate value of the second object on the X-axis.

[0076] y 1 : The coordinate value of the first object on the Y axis.

[0077] y 2 : The coordinate value of the second object on the Y axis.

[0078] D(X, Y): represents the Euclidean distance between two objects on a two-dimensional plane (in the X and Y coordinate system).

[0079] By deeply analyzing the three-dimensional data and generating an attribute graph, this embodiment can help the robot accurately understand the characteristics of objects in complex environments. The attribute graph provides accurate object information for subsequent task decisions, thereby optimizing the efficiency and accuracy of the robot's path planning, obstacle avoidance, object recognition and other tasks.

[0080] In addition, by using deep neural networks to extract features from three-dimensional information, the robot's environmental perception and adaptability can be significantly improved, ensuring that it can efficiently complete tasks in different environments.

[0081] Step 3: Generate a projected image based on the 3D information

[0082] Based on the three-dimensional data collected from different sensors, the system further generates one or more projected images. Projected images are the result of projecting objects or scenes in three-dimensional space onto a two-dimensional plane. These projected images show the detection results of different sensors on the same object or scene, including detection results from radar sensors and detection results from other types of sensors (such as LiDAR).

[0083] In this embodiment, the basic method of generating a projected image is to project point cloud data or a three-dimensional geometric model in a three-dimensional space onto a two-dimensional plane. Specifically, the projection process can be divided into the following steps:

[0084] Radar data projection: The three-dimensional data captured by the radar sensor is usually generated by the reflected radar waves, which contains the distance information of the object, the reflection intensity, etc. In order to generate the projected image, the system first maps these three-dimensional data to a two-dimensional plane. This can be done by projecting the three-dimensional coordinate points of the object in the radar coordinate system according to a specific viewing angle (such as a top-down angle or a front-view angle). For each three-dimensional point (x, y, z coordinates), it can be converted into coordinates on the two-dimensional plane (x ′ ,y ′ ):

[0085]

[0086] Among them, ProjectionMatrix is ​​the projection matrix. The specific matrix value depends on the selected projection method (for example, orthographic projection, perspective projection, etc.), x, y, z are the coordinates of the radar data in three-dimensional space, and x ′ ,y ′ is the projected two-dimensional coordinate.

[0087] LiDAR data projection: LiDAR data is usually provided in the form of a point cloud, where each point contains its spatial position (x, y, z coordinates). These point cloud data need to be converted into a two-dimensional representation through a similar projection method. The data projection process of LiDAR point clouds may be different from that of radar data projection, especially when dealing with the density and distribution of data, LiDAR usually provides higher accuracy. Therefore, when generating a projected image, LiDAR data may need to be optimized through interpolation or filtering techniques to improve the accuracy and continuity of the image.

[0088] For lidar data, in addition to standard projection, the system may also need to perform Gaussian filtering or other smoothing on the data to remove noise and ensure image quality after projection.

[0089] The generated projected image shows the spatial distribution and relative position of objects in three-dimensional space. Specifically, the projected image can reflect the following:

[0090] Spatial position of an object: By projecting the three-dimensional coordinate points of an object onto a two-dimensional plane, the projected image shows the relative position of the object at a specific perspective, helping the robot understand the distribution of the object in space.

[0091] Object size: The size of the projected image can reflect the size of the object. By analyzing the size of the area occupied by the object in the image, the actual size of the object can be estimated. This information is particularly important for object recognition and path planning, especially when there are multiple objects in the environment. The robot can determine whether the size of the object will affect the robot's path based on the size of the object.

[0092] Relative position between objects: Through the projected images generated by multiple sensors, the robot can compare the position of objects from different perspectives, and then determine the distance and relative relationship between objects. For example, images from radar sensors and lidar sensors can complement each other to provide more comprehensive object layout information.

[0093] In this embodiment, the projected images generated by data from different sensors (such as radar and lidar) usually have different precision and characteristics. In order to improve the accuracy and comprehensiveness of the projected images, the projected images from multiple sensors can be fused. This process includes the following steps:

[0094] Alignment: First, the data from different sensors are aligned to the same coordinate system through geometric transformation. Common alignment methods include coordinate transformation based on calibration information or automatic alignment of different sensor data through image registration algorithms.

[0095] Fusion: Once the data is aligned, the system can fuse multiple projected images. Fusion methods include weighted averaging and projection intersection strategies. The weighted averaging method assigns different weights according to the reliability of the data from different sensors, combining the advantages of two or more sensors. Projection intersection can obtain more accurate object position and spatial information by calculating the intersection of images from different sensors.

[0096] The generated projected image serves as the basis for the robot to understand the environment and can be used for subsequent tasks such as path planning, obstacle avoidance, object recognition, etc. Specifically:

[0097] Path planning: By projecting images, the robot can clearly see the location and shape of obstacles, thus providing an accurate map of the environment for the path planning algorithm. Based on these images, the robot can choose the best path to avoid collisions with obstacles.

[0098] Object recognition and classification: Projected images can reveal the spatial characteristics of objects, such as location, size, and shape, and robots can recognize and classify objects based on these characteristics. For example, based on projected images, robots can distinguish between walls, furniture, and other dynamic objects.

[0099] Environmental reconstruction: After collecting 3D data and generating projected images multiple times, the robot can build a more complete environmental model based on images from different perspectives to support task execution in complex scenarios.

[0100] Through the above technical means, robots can understand complex environments more efficiently and perform tasks such as object recognition, path planning, obstacle avoidance, etc., improving the intelligence and adaptability of the system.

[0101] Step 4: Receive identification data and optimize the model

[0102] In this embodiment, the identification data collected by the robot through the sensor is used to optimize the existing large-scale model to improve the accuracy of object recognition and environmental adaptability. The identification data provides the robot with the spatial coordinates, feature information and relative positional relationship between objects. These identification data correspond one-to-one with the projected image generated by the sensor and are the key input for model updating and optimization.

[0103] Acquisition of identification data:

[0104] Spatial coordinates and feature information: The identification data contains the coordinate information of the object in space. Specifically, the spatial coordinates of each object can be accurately located through the data collected by the sensor. The identification data also includes the feature information of the object, such as the size, shape, reflection intensity, surface texture, etc. The acquisition of identification data is usually achieved through the cooperation of sensors and the control system inside the robot to generate the identification label of the object in real time.

[0105] Object position relationship: In addition to the spatial coordinates of a single object, the identification data also provides the relative position relationship between multiple objects. With this information, the robot can determine the relative layout of objects and further understand the distribution of objects in the entire environment. This information is particularly important when the robot needs to plan a path, which can effectively help the robot avoid obstacles or adjust its route.

[0106] Matching and calibration of identification data and projected images:

[0107] Matching: Each set of identification data needs to be accurately matched with the object position in the projected image. Specifically, by calibrating the sensor data, each identification point is matched with the corresponding object features in the generated projected image. For example, the corresponding objects in the projected images of radar and lidar should have the same spatial coordinates and feature information in the identification data.

[0108] Calibration: The calibration process is the core step to ensure that the identification data matches the projected image. During calibration, a calibration plate or an object with known features can be used as a reference to ensure the accuracy of the 3D data captured by the sensor in the physical space. The calibration method can use the least squares method (Least Squares Method) or other mathematical models to adjust the data of different sensors so that it can be accurately mapped to the same reference frame.

[0109] Model update and optimization:

[0110] Parameter update: After acquiring the identification data, the robot will analyze the differences between the identification data and the object features in the projected image. These differences will be used to adjust the internal parameters of the large model. Specifically, the robot calculates the errors (such as position error, size error, shape error, etc.), feeds these errors back to the model, and then updates the model's weights and bias items so that the model can better adapt to the actual environmental data.

[0111] This process can be performed through the backpropagation algorithm. Specifically, the model calculates the loss function (such as mean square error MSE) based on the error between the identification data and the generated projection image, and then uses the gradient descent method or other optimization algorithms to update the model parameters. The calculation formula of the loss function is:

[0112]

[0113] The parameters are explained as follows:

[0114] L(θ): represents the value of the loss function, which is used to quantify the error between the model prediction value and the true value.

[0115] During the optimization process, the value of the loss function is continuously reduced through back-propagation and gradient descent algorithms to improve the prediction accuracy of the model.

[0116] θ: represents the set of model parameters, including weights and biases. The model parameters are continuously adjusted during the training process to minimize the loss function L(θ).

[0117] N: represents the number of samples, which represents the number of data points used in a loss calculation. The sum is taken over the entire dataset and the average is taken to measure the overall error of the model on all samples.

[0118] i: represents the index of the sample, indicating the number of the sample currently being processed, and its value range is 1≤i≤N. It is used to calculate all samples one by one and accumulate the errors.

[0119] y i : represents the true value of the i-th sample, that is, the target value provided in the identification data. Indicates the correct answer that the model prediction needs to be close to.

[0120] T(x i ,θ): represents the prediction result of the i-th sample through the model. It represents the output of the model, which is the input data x i The result is determined jointly by the current model parameters θ.

[0121] (y i -f(x i ,θ)) 2 : The square error of the i-th sample. Used to measure the predicted value f(x i ,θ) and the true value y i The squared form ensures that the error is always non-negative while amplifying the impact of larger errors.

[0122] Dynamic Adjustment: To improve recognition accuracy, the system makes dynamic adjustments based on the difference between the new identification data and the projected image. These adjustments are mainly reflected in the fine-tuning of model parameters, so that the model can better adapt to changes in different environmental conditions. For example, in some complex or dynamically changing environments, the shape or position of objects may change, and the model needs to automatically adjust its parameters based on the new data to maintain high-precision object recognition capabilities.

[0123] Model training and validation:

[0124] Training: By continuously receiving new identification data and projected images, the robot can continuously optimize the large model. During the training process, the model learns the characteristics of objects through a large amount of sample data and gradually improves its recognition ability. During each training process, the model adjusts its parameters based on the new identification data to minimize the error between the model output and the real identification.

[0125] Verification: After each parameter update, the robot can test the model through the validation set to ensure that the updated model has improved performance in the actual environment. The validation set usually consists of identification data and projected images collected in different scenarios, and is used to evaluate the model's performance indicators such as precision and recall.

[0126] Ongoing process of model optimization:

[0127] Model optimization is an ongoing process. As the robot performs tasks in different environments, the system will continue to receive new identification data, which will continue to drive model optimization. Through continuous training and real-time adjustments, the robot can gradually improve its recognition accuracy and adaptability in various environmental conditions.

[0128] The implementation of step 4 provides the robot with accurate object location and feature information by receiving identification data and pairing it with the projected image. This identification data is not only used to optimize the parameters of the large model, but also helps the robot adjust its recognition capabilities in real time to better adapt to task execution in different environments. This optimization process provides the robot with powerful adaptive capabilities, enabling accurate object recognition and task execution in a constantly changing environment.

[0129] Step 5: Radar projection image generation and optimization

[0130] In this embodiment, radar data is used to generate a radar projection image, the purpose of which is to convert the object information detected by the radar into an image in a two-dimensional space, simplify the representation of the object information, and provide effective data for further processing and analysis. By projecting radar data into a two-dimensional space, the robot can more concisely understand and process data from different sensors, thereby better performing tasks.

[0131] The process of generating a radar projection image involves mapping the three-dimensional spatial data obtained from the radar sensor to a two-dimensional plane. Specifically, radar data usually includes information such as the distance, azimuth, and elevation of an object. When generating a projection image, this three-dimensional spatial information needs to be converted into a two-dimensional image representation through a projection transformation.

[0132] Format of radar data: The data collected by the radar sensor usually includes reflection intensity, distance information, and relative position to the object. The distance measurement can determine the straight-line distance from the object to the radar, while the azimuth and elevation angles can determine the direction of the object in three-dimensional space. These data provide the rough position of the object in space and the characteristics of the object.

[0133] Projection transformation: To convert radar data into a two-dimensional image, common projection transformation methods such as polar coordinate transformation or perspective projection are used. In the polar coordinate system, the position of each object is represented by its distance (r) and azimuth (θ) from the radar. By transforming this data onto a two-dimensional plane, the resulting projected image will show the position and size of the object in two-dimensional space.

[0134] Calculate the distance r and azimuth θ of each object relative to the radar for the radar data;

[0135] Convert the radar data into the corresponding image coordinates in the two-dimensional coordinate system:

[0136] x=r·cos(θ)

[0137] y=r·sin(θ)

[0138] Among them, r is the distance from the radar to the object, θ is the azimuth of the object relative to the radar, and x and y are the image coordinates on the two-dimensional plane.

[0139] The projected image can concisely express the position of an object in space, has obvious spatial structure information, and is compatible with other types of sensor data (such as lidar data and camera data).

[0140] By combining the identification data generated by other sensors (such as LiDAR or depth cameras) with the projected image, the robot can further optimize the model parameters. The identification data includes the spatial coordinates of the object, feature information, and the relative position of the objects. Combining this information with the radar projected image can enhance the model's recognition accuracy of the object through comparison and calibration.

[0141] Optimizing parameters: The optimization process is based on comparing the error between the position of the object in the radar projection image and the position of the object in the identification data. By calculating the error, the weights and biases of the model can be further adjusted to optimize the representation of the object in the radar projection image. The error can be calculated using common loss functions such as mean squared error (MSE) or cross entropy loss function. The optimization process updates the model parameters through gradient descent or other optimization algorithms.

[0142] Adjustment of object information: After the radar projection image is generated, the object information in the image, such as position, size or shape, can be adjusted based on the feedback of the identification data. This process ensures the effective fusion between radar data and other sensor data, so that the object recognition accuracy of the model is further improved. For example, when the boundary of an object is not clear, the model can adjust the outline of the object to make it more consistent with the shape of the real object.

[0143] The fusion of radar data and other sensor data (such as LiDAR, camera, etc.) is a key step to improve recognition accuracy. In order to effectively fuse data from different sensors, data alignment technology is usually used, such as registration method based on relative coordinate system. Specifically, by aligning the coordinate systems from different sensors, it is ensured that the radar projection image and the data generated by other sensors can be fused in the same reference frame, thereby improving the overall recognition accuracy.

[0144] Sensor fusion technology: Common sensor fusion methods include Kalman filtering and particle filtering, which can effectively integrate information from different sensors and reduce the impact of noise on model optimization to a certain extent. Through these methods, radar data, LiDAR data and other sensor data can be accurately aligned in space, thereby optimizing object detection results.

[0145] Through the generation of radar projection images and the feedback of identification data, the robot can continuously optimize its object recognition capabilities, especially in complex and dynamic environments. The optimization of radar data can effectively reduce sensor noise and uncertainty, and improve the model's response speed and accuracy to objects. This process enables the robot to perform tasks such as path planning and obstacle avoidance more quickly and accurately in various environments.

[0146] Enhanced object recognition capabilities: Through repeated optimization, the object information in the radar projection image can more accurately reflect the actual location and characteristics of the object. This enables the robot to more effectively identify and handle different objects in practical applications.

[0147] In summary, the radar projection image generation and optimization process in step 5 not only simplifies the processing of radar data, but also improves the accuracy of object recognition by effectively integrating data with other sensors. By continuously optimizing the radar projection image and model parameters, the robot can achieve more efficient and accurate task execution in a dynamic environment.

[0148] Step 6: Local optimization and task execution

[0149] In this embodiment, a local optimization strategy is used to improve the performance of the robot under different environmental conditions or object characteristics. The core idea of ​​local optimization is to dynamically adjust some parameters in the large model according to the real-time changes in the environment and the specific requirements of the task, so that the robot can perform tasks more accurately in specific areas or conditions. This optimization process not only improves the adaptability of the robot in complex environments, but also enhances the accuracy and efficiency of task execution.

[0150] Strategies and methods for local optimization:

[0151] Target area recognition: The robot uses sensor data (such as radar, LiDAR, depth camera, etc.) to identify target areas in the current environment in real time, especially those complex or difficult to handle areas. For example, narrow passages, areas with large changes in lighting, areas with densely distributed objects, etc. These areas usually increase the difficulty of path planning or affect the accuracy of object recognition. By scanning the environment in real time, the robot can determine these specific areas and perform local optimization based on the characteristics of these areas.

[0152] Feature recognition and analysis: The robot analyzes the key features of the target area (such as the shape, size, distance, direction, surface roughness, etc.) to identify factors that may affect the execution of the task. For example, in areas with large changes in lighting, the robot may enhance its ability to recognize the edges of objects; in narrow passages, it may focus on optimizing the path planning algorithm to ensure that the robot can accurately avoid obstacles.

[0153] Implementation steps of local optimization:

[0154] Dynamic parameter adjustment: The robot will optimize by adjusting the parameters of the corresponding parts of the large model according to the characteristics of the target area. For example, the weights and bias parameters related to object recognition in the model can be dynamically adjusted for certain complex environments to improve recognition accuracy. If an object with a complex shape or fuzzy features is detected in a certain area, the system can dynamically increase the model weight related to shape detection to make the object recognition more accurate.

[0155] Local optimization of path planning: For narrow passages or complex terrain, the robot can adjust the parameters of the path planning model to make it more flexible and accurate in these specific areas. For example, increasing the sensitivity of obstacle detection and reducing the error in path calculation can avoid collisions of the robot in narrow areas. At the same time, adjusting the optimization goal of path planning to focus on avoiding obstacles rather than optimizing overall motion efficiency can ensure the safe completion of the task.

[0156] Feedback mechanism of local optimization process:

[0157] Real-time environmental perception: During the execution of tasks, the robot continuously acquires new sensor data (such as radar scan results, images captured by cameras, etc.) by sensing its environment in real time. This data will be used to evaluate changes in current environmental conditions to determine whether local optimization mechanisms need to be activated. When the robot enters a new environment or the distribution of objects changes, the system will fine-tune the model based on the new data to ensure that the optimization effect continues to be effective.

[0158] Optimization effect verification: To verify the effect of local optimization, the robot will perform real-time detection through a feedback mechanism. For example, during object recognition, if the robot recognizes an error in an object, it will re-evaluate the optimization parameters of that area and make fine adjustments. Through this closed-loop feedback mechanism, the robot can continuously adjust its optimization strategy and improve the accuracy of task execution.

[0159] Optimization in areas with varying lighting: In environments with large lighting variations, the robot may face visual errors caused by reflections or shadows. To cope with this, the robot enhances its sensitivity to object edges and surface features through local optimization in this area. For example, the convolutional layer weights of a deep learning model can be adjusted to make it pay more attention to the contours and edge information in the image, thereby improving the accuracy of object detection in low-light conditions.

[0160] Path planning in narrow passages: In narrow passages, the robot must precisely control its trajectory to avoid collisions. Through local optimization, the path planning model can adjust its decision-making process in real time, making the robot's movement in the passage more precise. For example, the robot can increase the frequency of monitoring surrounding obstacles and dynamically adjust the movement speed according to the distance between the obstacle and the robot to ensure safe passage in narrow passages.

[0161] Local optimization is not just about adjusting model parameters, it is also closely related to the model's training process. Through technologies such as transfer learning, robots can quickly adapt to new task requirements based on previous learning experience in similar environments. For example, when a robot performs a task in a specific type of environment, local optimization can adjust the model based on existing knowledge to shorten training time and improve task execution efficiency. The use of transfer learning enables robots to respond quickly in a changing environment and improve the robustness of the system.

[0162] During the dynamic adjustment of parameters, focus on the following parameters: weight matrix, bias value, learning rate, and gradient update.

[0163] Among them, the prediction result of the output layer of the deep neural network model is y pred , the true label is y true , the model's weight matrix is ​​W, the bias is b, the learning rate is α, and the update formula for gradient descent is:

[0164]

[0165] ΔW: represents the updated value of the weight matrix W, that is, the magnitude by which the weight matrix needs to be adjusted in each iteration. It is the core of the calculation in the gradient descent process and is used to optimize the weight parameters of the model.

[0166] α: represents the learning rate, which is used to control the step size of model parameter adjustment. A larger α will accelerate convergence, but may cause the parameter update to be too large and miss the optimal solution; a smaller α will slow convergence but increase accuracy.

[0167] Represents the gradient of the loss function L to the weight matrix W. This is the key part of the optimization algorithm and reflects the current model prediction value y pred and the true value y true The error between them, and the contribution of the weight matrix W to the error.

[0168] L(y pred ,y true ): loss function, used to quantify the model prediction value y pred and the true value y true The difference between them. Loss functions include mean square error (MSE), cross entropy, etc.

[0169] In deep neural networks, bias b is one of the parameters of each layer and is usually used to adjust the output of the activation function, similar to the intercept term in a linear equation. In the gradient descent process, the update rule of bias b is similar to the update rule of weight W. The key is to calculate the gradient of the loss function to the bias through the back-propagation algorithm. The calculation formula for the bias gradient is:

[0170]

[0171] Parameter explanation:

[0172] Δb: Update value of bias, used to adjust bias b in the current optimization step.

[0173] α: learning rate, which controls the step size of parameter adjustment.

[0174] The gradient of the loss function L with respect to the bias b indicates the contribution of the bias to the model prediction error.

[0175] In order to achieve local optimization, this application uses the following formula to dynamically adjust parameters:

[0176] W new =W old +△W·(1+λ local (x env ))

[0177] b new =b old +△b·(1+λ local (x env ))

[0178] in:

[0179] W new : The updated weight matrix represents the weight value of the model after the current optimization step.

[0180] W old: Current weight matrix value, model parameters before this optimization.

[0181] △W: The weight update value calculated by the current gradient descent algorithm (refer to the above formula).

[0182] b new : Updated bias value, which indicates the bias value of the model after the current optimization step.

[0183] b old : Current bias value, the model bias parameter before this optimization.

[0184] Δb: Bias update value calculated by the current gradient descent algorithm.

[0185] λ local (x env ): Environmental factor adjustment coefficient, used to dynamically adjust the parameter update amplitude according to the current environmental conditions. Its value is determined by the environmental characteristic parameter x env The calculations show the impact of environmental complexity on model optimization.

[0186] The specific calculation method is as follows:

[0187]

[0188] in:

[0189] λ local (x env ): represents the environmental factor adjustment coefficient, which is used to adjust the update amplitude of weights and biases. When the environmental conditions are complex (such as high obstacle density and drastic changes in light), λ local (x env ) will increase, making the adjustment of model parameters more sensitive.

[0190] x env :Characteristic parameters of the current environment, used to describe indicators of environmental complexity. For example: the change in light intensity, obstacle density, channel width, etc.

[0191] γ: Adjustment coefficient, used to control the sensitivity of environmental factors to weight updates. A larger γ will enhance x env Adjustment coefficient λ local (x env ); a smaller γ will weaken this effect.

[0192] Represents an exponential function, used to smooth the environmental characteristic parameter x env The influence of makes the adjustment coefficient more stable in practical applications.

[0193] In complex environments, such as areas with large changes in lighting or dense obstacles, the environmental factor λ in the formula local (x env ) will increase, making the model's parameter adjustment more sensitive in this environment, ensuring that the robot can improve its recognition accuracy when performing tasks in these areas.

[0194] On the contrary, in a simpler or more stable environment, the environmental factor λ local (x env ) is close to 0, the parameter update of the model tends to be stable and will not be over-adjusted, thus ensuring the stability of task execution.

[0195] For narrow passages or areas with prominent features of specific objects, the robot can calculate λ based on the characteristic parameters of these areas (such as the shape, size, and spacing of obstacles). local (x env ), and then adjust the relevant parameters in the model. For example, in a narrow passage, environmental factors may increase the model’s sensitivity to obstacle detection so that it can better avoid obstacles.

[0196] During the dynamic adjustment process, the learning rate α will also be adjusted according to changes in environmental factors. When the environmental conditions are complex, a smaller learning rate may be required to avoid drastically updating the model and prevent overfitting; in a simpler environment, the learning rate can be increased appropriately to enhance the training efficiency of the model.

[0197] The formula for dynamically adjusting the learning rate is:

[0198] α new =α old ·(1+λ local (x env ))

[0199] Parameter explanation: α new : The updated learning rate is used to control the step size of the next optimization.

[0200] α old : The current learning rate, which indicates the learning step size of the model before this optimization.

[0201] λ local (x env ): Environmental factor adjustment coefficient, used to dynamically adjust the learning rate according to the complexity of the environment.

[0202] When the robot performs a task, it updates the environmental factor x based on real-time feedback (such as sensor data, task execution status, etc.) env , and then adjust the locally optimized parameters. Through this feedback mechanism, the robot can dynamically adapt to changes in the environment and optimize its parameter configuration in real time.

[0203] Optimization in complex environments: When the robot enters an area with large changes in lighting, x env (Light intensity change) value is large, and the calculated λ local (x env ) will increase, and the model will adjust its perception of object edges and surface features, thereby improving object recognition accuracy in low-light conditions.

[0204] Path planning optimization for narrow channels: When the robot passes through a narrow channel, x env (channel width, obstacle density) values ​​are small, and the calculated λ local (x env ) increases, the model will optimize the path planning algorithm to ensure that the robot can avoid obstacles more flexibly and thus avoid collisions.

[0205] Through the above formula, various parameters work together, and the model can dynamically adjust the optimization amplitude and learning efficiency according to the complexity of the real-time environment, ensuring that the responsiveness of the model is improved in complex environments, while maintaining the stability of parameters in simple environments to avoid excessive adjustments.

[0206] Step 7: Apply the trained model to the robot 3D vision guidance task

[0207] In this embodiment, the large-scale model that has been trained and locally optimized is finally successfully applied to the robot's three-dimensional vision guidance task. By optimizing the deep learning model, the robot can efficiently perform multiple tasks in a complex and dynamically changing environment, such as object recognition, path planning, obstacle avoidance, etc. The role of the model in the application is not limited to providing static decisions, but can also respond dynamically according to real-time changes in the environment, significantly improving the accuracy of task execution and the robustness of the system.

[0208] Specifically, the trained and optimized model will be deployed in the robot control system as the core of the robot's perception and decision-making. At this point, the robot can use three-dimensional visual data (such as environmental information obtained by sensors such as LiDAR and depth cameras) to perceive its spatial environment in real time. During the execution of the task, the model will identify objects based on real-time sensor data and dynamically adjust the path planning plan.

[0209] Object Recognition:

[0210] The robot uses deep learning models to identify objects in the surrounding environment and can accurately detect static and dynamic objects in the environment. The model processes images or three-dimensional data to identify the type, location, size and other key features of the object and make corresponding decisions based on task requirements. For example, in a complex warehouse environment, the robot can identify shelves, obstacles or people and adjust the path or perform avoidance actions.

[0211] Path planning: Based on object recognition, the robot uses an optimized path planning algorithm to calculate the best path in real time. In complex environments, the robot can dynamically adjust the path planning strategy through the model. For example, when an obstacle is detected ahead or the environment changes, the robot can quickly replan the path to avoid collision with the obstacle and maintain the continuity of task execution.

[0212] Obstacle avoidance: The robot can accurately detect and avoid obstacles through the optimized model. When the robot encounters an obstacle in a complex environment, the model will calculate the relative position and size of the obstacle, and guide the robot to avoid the obstacle and continue to perform the task through real-time path adjustment strategies. For example, when passing through a narrow passage, the robot will accurately measure the width of the passage based on 3D visual data and adjust the movement trajectory to ensure safe passage.

[0213] Environmental adaptability: The continuously optimized model can respond flexibly to real-time changes in the environment. In complex environments, such as when light intensity changes, obstacle density changes, or environmental topology changes, the model can automatically adjust its parameters to keep the robot able to perform tasks efficiently. Through local optimization and dynamic adjustment, the robot can continuously adapt to changes in different environments and complete tasks under uncertain environmental conditions.

[0214] Improved robustness: By simulating the diversity of training data and the complexity of the environment, the robot can effectively deal with various unknown and unexpected situations during application. The model can detect and correct errors in task execution in real time through sensor feedback, ensuring that the robot maintains efficient and stable task execution capabilities under different circumstances, significantly improving the robustness of the system.

[0215] In summary, the trained large-scale model has been successfully applied to the robot's 3D vision guidance tasks, which has comprehensively improved the robot's recognition, planning, avoidance and adaptation capabilities in complex environments. Through continuous optimization and local adjustments, the robot can perform a variety of tasks efficiently and accurately, ensuring the efficient completion of tasks and ensuring stability and safety in various complex environments.

[0216] Through the above method, the robot can achieve efficient 3D vision-guided tasks based on multi-sensor data, combined with large models and local optimization strategies. This method can effectively improve the robot's environmental perception ability and task execution accuracy, and maintain efficient task processing capabilities in complex environments. At the same time, the local optimization strategy ensures that the robot can adapt in real time according to different task scenarios and object characteristics, improving the robustness and flexibility of the system.

[0217] The above description is only a preferred embodiment of the present invention, so all equivalent changes or modifications made according to the structure, characteristics and principles described in the scope of the patent application of the present invention are included in the scope of the patent application of the present invention.

Claims

1. A method for improving the performance of robot 3D vision guidance tasks based on a large model, characterized in that: include: Acquire three-dimensional information from multiple types of sensors; Analyzing the three-dimensional information using a large model to generate one or more attribute graphs to represent the spatial location distribution and item characteristics of the items in the collected three-dimensional information; generating one or more projected images based on the three-dimensional data, wherein the projected images represent three-dimensional detection results from a plurality of sensors; Receiving identification data of one or more three-dimensional detection results generated by the plurality of sensors, the identification data identifying the position coordinates or object characteristics of an object in a projected image corresponding to the three-dimensional detection result; Generate a radar projection image based on the radar information, wherein the radar projection image represents the detection results of the object detected by the radar detector; updating one or more parameters in a large-scale model using the radar projection image and ground information generated based on the identification data; According to different environmental conditions or object types, large models are locally optimized for specific areas and some parameters of the model are dynamically adjusted; The trained large-scale model is applied to the robotic 3D vision guidance task.

2. A method for improving the performance of robot 3D vision guidance tasks based on a large model as claimed in claim 1, characterized in that: The three-dimensional information includes lidar data, depth camera data, or other types of three-dimensional sensor data.

3. A method for improving the performance of robot 3D vision guidance tasks based on a large model as claimed in claim 1, characterized in that: The attribute map represents the spatial distribution of objects, the size, shape, surface structure, or other physical characteristics of the objects.

4. A method for improving the performance of robot 3D vision guidance tasks based on a large model as claimed in claim 1, characterized in that: The projected image is an image obtained by projecting the three-dimensional information from a three-dimensional space onto a two-dimensional plane, and the two-dimensional plane is a visual guidance reference plane of a robot control system.

5. The method for improving the performance of robot 3D vision guidance tasks based on a large model as claimed in claim 1, characterized in that: The identification data includes the spatial coordinates of the object in three-dimensional space, the object type, the surface characteristics of the object, or the relative position relationship of the objects.

6. A method for improving the performance of robot 3D vision guidance tasks based on a large model as claimed in claim 1, characterized in that: The radar projected image is obtained by scanning the object through a radar detector and converting it into a two-dimensional image.

7. A method for improving the performance of robot 3D vision guidance tasks based on a large model as claimed in claim 1, characterized in that: The large model is a deep neural network (DNN) or a convolutional neural network (CNN) model.

8. The method for improving the performance of robot 3D vision guidance tasks based on a large model as claimed in claim 1, characterized in that: The local optimization includes adjusting parameters in the model according to environmental conditions or object characteristics, and the parameters in the adjusted model include a weight matrix, a bias value, and a learning rate.

9. A method for improving the performance of robot 3D vision guidance tasks based on a large model as claimed in claim 8, characterized in that: The adjustment formulas for the weight matrix, bias value, and learning rate are: IN new =In old +ΔW (1+λ local (x env )) b new =b old +Δb·(1+λ local (x env )) a new =a old ·(1+λ local (x env )) in: W new : updated weight matrix; W old : Current weight matrix value; ΔW: weight update value calculated by the current gradient descent algorithm; b new : Updated bias value; b old : Current bias value; Δb: bias update value calculated by the current gradient descent algorithm; α new : updated learning rate; α old : current learning rate; λ local (x env ): Environmental factor adjustment coefficient.

Images