An operating robot with lighting recognition

Abstract

The present invention relates to the field of robot control technology and discloses an operating robot with lighting recognition, comprising: an illumination coding projection module, which is used to generate an illumination coding pattern sequence according to the material characteristics of a target object and project the illumination coding pattern sequence onto the surface of the target object; a multi-angle optical perception module, which is used to obtain the optical response characteristics of the target object; a non-rigid deformation analysis module, which analyzes the changes of the illumination coding pattern at different times based on the optical response characteristics to construct a non-rigid deformation field and predict the material state of the target object at the next moment; a mechanical property analysis module, which is used to analyze the relationship between the optical response characteristics and the mechanical properties of the target object using a deep learning model, and preset operating force parameters according to the material state; and a closed-loop control module, which is used to form a closed-loop control mode. The present invention can achieve high-precision perception and precise operational control of non-rigid deformable objects.

Description

Technical Field

[0001] The present invention relates to the technical field of robot control, and more particularly, to an operating robot with lighting recognition. Background Art

[0002] In modern manufacturing, medical surgery, and flexible material processing, robotic manipulation often requires handling objects with non-rigid deformation properties, such as viscoelastic materials and biological tissues. These objects undergo complex deformations when subjected to forces, and their material properties may change over time.

[0003] Traditional robotic systems rely primarily on visual and tactile feedback for control, making them unable to effectively capture and respond to dynamic changes and local deformations of objects during manipulation, particularly when deformation occurs within the object or in areas not directly observable. Furthermore, traditional systems lack the ability to analyze the microscopic features of objects under varying lighting conditions and adjust operational parameters accordingly. Force control of deforming objects often relies on tentative contact, a method that can cause irreversible damage to fragile or sensitive objects, increasing the risk of damage.

[0004] Therefore, it is of great significance to develop a technical solution that can perceive the deformation of non-rigid objects with high precision and precisely control operations. Summary of the Invention

[0005] The present invention provides an operating robot with lighting recognition, which solves the technical problem in the related art that it is unable to effectively capture and respond to changes in the dynamic characteristics of non-rigid deformable objects.

[0006] The present invention provides an operating robot with lighting recognition, comprising:

[0007] An illumination coding projection module is used to generate an illumination coding pattern sequence according to the material characteristics of the target object and project the illumination coding pattern sequence onto the surface of the target object;

[0008] A multi-angle optical perception module is used to capture the optical response data of the target object under the projection conditions of the light-encoded pattern sequence and obtain the optical response characteristics of the target object;

[0009] The non-rigid deformation analysis module analyzes the changes of the illumination coding pattern at different times based on the optical response characteristics to construct a non-rigid deformation field and predict the material state of the target object at the next moment;

[0010] A mechanical properties analysis module, which uses a deep learning model to analyze the relationship between optical response characteristics and the mechanical properties of the target object, and presets the operation force parameters based on the material state;

[0011] The closed-loop control module is used to adjust the robot's operating parameters in real time according to the non-rigid deformation field and material state to form a closed-loop control mode.

[0012] In a preferred embodiment, generating the illumination coding pattern sequence in the illumination coding projection module includes:

[0013] Perform spectral scanning tests to obtain basic material characteristic parameters of the target object;

[0014] Based on the basic material characteristic parameters, the corresponding coding pattern type is selected. The coding pattern types include: a high-frequency phase-shifted sinusoidal grating pattern for highly reflective materials, a mixed pattern of random dot matrix and gradient stripes for translucent materials, or an orthogonal cross multi-directional stripe pattern for fabric materials;

[0015] The optimal encoding parameters are determined by maximizing the deformation response function through a Bayesian optimization framework.

[0016] In a preferred embodiment, the multi-angle optical perception module acquires the optical response characteristics of the target object including:

[0017] Preprocessing, encoding pattern recognition and feature calculation are performed on the image data collected by the multi-angle optical sensor array to form an optical response feature map;

[0018] The optical feature changes between consecutive time frames are analyzed through the inter-frame feature comparison method enhanced by spatiotemporal correlation, and the feature response change is calculated. This method can distinguish the feature changes caused by the deformation of the object itself and the changes in ambient lighting.

[0019] In a preferred embodiment, the non-rigid deformation analysis module constructs the non-rigid deformation field using a deep neural network structure, including an encoder-decoder architecture, wherein:

[0020] The encoder part includes a convolutional layer for extracting multi-scale features from the coding pattern changes;

[0021] The decoder part contains transposed convolutional layers, which are connected to the corresponding layers of the encoder through skip connections, gradually restoring the spatial resolution and outputting a dense deformation field;

[0022] The deep neural network is optimized by a combined loss function, which includes a reconstruction error term and a smoothness constraint term.

[0023] In a preferred embodiment, the non-rigid deformation analysis module predicts the material state of the target object at the next moment using a long short-term memory network structure. The long short-term memory network includes an input layer, two LSTM layers and a fully connected output layer. The network input is an optical response feature sequence within a time window, and the output is a predicted material parameter vector, which includes elastic modulus, viscosity coefficient and hardness.

[0024] In a preferred embodiment, the deep learning model in the mechanical properties analysis module includes:

[0025] A feature extraction network is used to extract material-related feature vectors from optical response features;

[0026] Mechanical mapping network, used to establish the mapping relationship between optical features and mechanical parameters;

[0027] The force prediction network is used to generate an operation force recommendation value based on mechanical parameters, wherein the force recommendation value includes contact force, grasping force and operation speed parameters.

[0028] In a preferred embodiment, the closed-loop control mode in the closed-loop control module includes:

[0029] In the prediction phase, the optimal parameters for the next operation step are predicted based on the currently acquired optical response characteristics and historical operation data;

[0030] In the verification phase, the response state of the target object is monitored in real time during the operation, and the deviation between the actual response and the predicted response is calculated;

[0031] During the adjustment phase, when the deviation exceeds the preset threshold, the parameter recalculation process is triggered, and a smooth transition to the new operating parameters is achieved without interrupting the operation.

[0032] In a preferred embodiment, it also includes an abnormal state detection module:

[0033] Establishing a distribution model of the optical response characteristics of the target object under normal conditions;

[0034] Real-time calculation of the Mahalanobis distance between the current optical response characteristics and the normal distribution model;

[0035] When the Mahalanobis distance exceeds the dynamic threshold, the exception handling process is triggered, including operation suspension, parameter reset, and operation path replanning.

[0036] In a preferred embodiment, the real-time adjustment of the robot's operating parameters in the closed-loop control module is based on a multi-objective optimization algorithm, taking into account:

[0037] Operational accuracy goal, minimizing the deviation between the operation position and the target position;

[0038] Safety objectives: ensuring that the operating force does not exceed the safety threshold of the target object;

[0039] The efficiency goal is to minimize the operation time while meeting the accuracy and safety constraints.

[0040] In a preferred embodiment, a computer-readable storage medium is used to store computer-readable instructions, which can operate an operating robot with lighting recognition when the computer-readable instructions are read by a computer.

[0041] The beneficial effects of the present invention are:

[0042] Achieved sub-millimeter non-rigid deformation tracking accuracy;

[0043] Ability to select appropriate operating force on objects of unknown materials upon first contact, reducing the damage rate caused by improper force;

[0044] Improved ability to perceive internal deformation of objects, enabling tracking of internal structural changes that are not observable using traditional methods;

[0045] The system has real-time response capabilities to the hardening or softening process of viscoelastic materials and can adjust operating parameters within a short period of time when material properties change;

[0046] The closed-loop control mode based on "prediction-verification-adjustment" improves the operation success rate, which is much higher than traditional fixed parameter control. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a module diagram of an operating robot with lighting recognition of the present invention;

[0048] Figure 2 is a detailed flow chart of projecting a light-encoded pattern sequence onto a target object surface according to the present invention;

[0049] Figure 3 is a detailed flow chart of obtaining the optical response characteristics of a target object according to the present invention;

[0050] Figure 4 It is a detailed flow chart of the present invention for predicting the material state of a target object at the next moment;

[0051] Figure 5 is a detailed flow chart of the preset operation force parameters of the present invention;

[0052] Figure 6 It is a detailed flow chart of forming a closed-loop control mode of the present invention. DETAILED DESCRIPTION

[0053] The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that these embodiments are discussed solely to enable those skilled in the art to better understand and implement the subject matter described herein, and that the functions and arrangements of the elements discussed may be varied without departing from the scope of this specification. Various examples may omit, substitute, or add various processes or components as needed. Furthermore, features described in some examples may be combined in other examples.

[0054] At least one embodiment of the present invention discloses an operating robot with lighting recognition, such as Figures 1 to 6 Shown, including:

[0055] An illumination coding projection module is used to generate an illumination coding pattern sequence according to the material characteristics of the target object and project the illumination coding pattern sequence onto the surface of the target object;

[0056] The specific steps include:

[0057] Step 1.1, object material characteristics collection;

[0058] The system uses a preset material database or initial illumination response testing to obtain the target object's basic material characteristic parameters, including optical parameters such as reflectivity, transmittance, and scattering properties. For objects of unknown materials, the system performs a rapid spectral scan test to build an initial material characteristic model.

[0059] In some embodiments, the spectral scanning test can use multi-wavelength rapid scanning technology. The system can complete the spectral scan in the range of 380nm to 1100nm within 0.5 seconds, and quickly determine the basic material type of the object (such as metal, plastic, biological tissue, etc.) by comparing and analyzing the reflectivity curve characteristics at different wavelengths.

[0060] Step 1.2, generating coding pattern sequence;

[0061] Based on the material properties of the object, a deformation-sensitive illumination coding algorithm is applied to generate a specific coding pattern sequence ,in, 、 、 Respectively represent 、 、 Patterns in coordinates The light intensity value at Indicates the number of coding patterns. The principle of constructing coding patterns is to maximize deformation sensitivity, so that even a small deformation of the object surface can produce a certain change in the reflected pattern.

[0062] In this application, the deformation-sensitive illumination coding algorithm adopts a multi-scale structured light pattern that is adaptive based on material properties. The specific implementation of the algorithm is as follows:

[0063] For highly reflective materials, the system generates a sinusoidal grating pattern with a high-frequency phase shift, which can maintain good texture distinction in strong reflections;

[0064] For translucent materials, the system generates a composite pattern consisting of a mixture of random dots and gradient stripes to improve the feature recognizability of light-transmitting materials;

[0065] For fabric materials, the system generates orthogonal cross-directional stripe patterns to reduce the interference of fabric texture on the coding pattern.

[0066] In practical applications, such as when performing surgery on silicone simulated tissue, the algorithm can automatically select the optimal encoding mode based on the reflective properties of the tissue, forming a clear and traceable texture pattern on the tissue surface, and maintaining a high feature recognition rate even under the strong light of the surgical lamp.

[0067] The generation of coding patterns can also utilize transfer learning technology. By migrating the optimal coding parameters of similar materials from a large number of pre-trained material samples and performing fine-tuning and adaptation, the optimization time of coding patterns for unknown materials can be greatly shortened, enabling the system to complete the optimal coding parameter configuration for new materials within 2 to 3 seconds.

[0068] The algorithm adopts a Bayesian optimization framework by maximizing the deformation response function To determine the optimal encoding parameters:

[0069] ;

[0070] in, represents the maximized deformation response function, Indicates the coordinate point The coded pattern light intensity value at Indicates the light intensity value of the coding pattern at that point after a slight deformation occurs. and Represents the simulated small deformation displacement, which is used to evaluate the sensitivity of the coding pattern to deformation. Represents the Euclidean norm, which is used to calculate the difference between the coding pattern before and after deformation. For all sampling points The larger the function value, the more sensitive the coding pattern is to object deformation, which is more conducive to the system accurately capturing small deformations.

[0071] Step 1.3, multi-angle light source array configuration and projection;

[0072] The generated coding pattern sequence is projected onto the surface of the target object through a controllable multi-angle light source array.

[0073] The light source array consists of multiple independently controlled LED or laser projection units, capable of producing precise spatial light intensity distribution. The system achieves precise projection of the coded pattern by controlling the intensity, wavelength and projection angle of each light source.

[0074] The light source parameters are determined by the following mapping relationship:

[0075] ;

[0076] in, Indicates time In coordinates The light intensity at a specific time and a specific spatial position; Represents the amplitude, determines the maximum value of light intensity, and controls the brightness of the light; Indicates the frequency of the light source, which determines how fast the light changes; Indicates the phase of the light source, which determines the initial position of the sine wave and has a value range of , used to control the synchronization or staggered relationship of lighting at different positions; The function is used to generate periodic changes in light intensity, giving the illumination a sinusoidal intensity distribution. By adjusting these parameters, the system can generate complex spatiotemporal illumination patterns on the surface of an object, providing an accurate encoding basis for deformation detection.

[0077] In some embodiments, the light source array may adopt a projection system based on a digital micromirror device (DMD), which can switch different coding patterns at a speed of microseconds and achieve a pattern refresh rate of up to 1000 Hz to meet the real-time monitoring needs of rapidly deforming objects.

[0078] A multi-angle optical perception module is used to capture the optical response data of the target object under the projection conditions of the light-encoded pattern sequence and obtain the optical response characteristics of the target object;

[0079] The specific steps include:

[0080] Step 2.1, multi-angle optical data acquisition;

[0081] Multiple high-speed optical sensors are deployed around the object, synchronously capturing the coded pattern reflected from the object's surface from different angles. The sensor array is configured to cover the maximum field of view, ensuring global deformation information of the object's surface is captured.

[0082] Each sensor at a time point The acquired image is recorded as ,in Indicates the index of the sensor, representing the The number of the sensor.

[0083] The collected image data forms a spatiotemporal data cube ,in and Represents the pixel coordinate position in the image, Indicates the collection time point, The sensor number represents the sensor, forming a four-dimensional data structure that contains complete information about spatial position, time, and multi-angle views. This multi-dimensional data structure enables the system to comprehensively analyze changes in the surface characteristics of an object at different angles and time points, providing a rich data foundation for subsequent deformation analysis.

[0084] In application scenarios involving translucent objects with good light transmittance, the system can additionally deploy backlight sources and transmitted light sensors to simultaneously obtain the object's reflected light and transmitted light information, further improving the ability to perceive deformation of the internal structure.

[0085] Step 2.2, optical response feature extraction;

[0086] Based on the collected multi-angle optical data, a feature extraction algorithm is applied to construct the optical response feature map of the object The feature extraction process includes three steps: image preprocessing, coding pattern recognition and feature calculation.

[0087] In the preprocessing stage, the image is denoised, rectified, and normalized;

[0088] The coding pattern recognition stage uses template matching or feature point detection algorithms to identify the deformation of the projected pattern;

[0089] The feature calculation stage extracts key optical features such as reflection intensity distribution, scattering angle, phase change, etc.

[0090] In some embodiments, the feature extraction algorithm may use multi-scale pyramid feature fusion technology, which can simultaneously capture the macroscopic deformation and microscopic texture changes of the object surface, thereby improving the system's ability to extract surface features of complex materials.

[0091] Step 2.3, analysis of temporal feature changes;

[0092] By analyzing the changes in optical characteristics between consecutive time frames, the characteristic response change is calculated:

[0093] ;

[0094] in, Indicates the current time The characteristic response change is used to quantify the degree of change in the optical properties of the object between adjacent time frames; Indicates the current time The optical response characteristic map contains the optical characteristic information of the object at that moment, such as reflection and scattering; Indicates the previous moment The optical response characteristic diagram is used as a reference benchmark to calculate the change; It represents the feature difference calculation function, which is a mathematical mapping used to quantitatively evaluate the difference in features between two time points.

[0095] This function comprehensively considers changes in the spatial distribution and intensity of features, effectively capturing subtle deformations and material variations in objects. For specific application scenarios, different difference metrics can be used, such as mean squared error (used to evaluate the sum of squared deviations of overall features), structural similarity (used to assess the structural consistency of features), or optical flow (used to accurately describe the displacement vectors of feature points).

[0096] The temporal feature change analysis algorithm adopts an inter-frame feature comparison method with enhanced spatiotemporal correlation, which can distinguish feature changes caused by deformation of the object itself and changes in ambient lighting.

[0097] Specific implementation methods include:

[0098] Establish the corresponding relationship between the feature points of the reference frame and the current frame;

[0099] Screening reliable feature changes through local consistency constraints;

[0100] The overall deformation is calculated by weighted fusion of multi-scale feature changes.

[0101] The algorithm is robust to changes in lighting, maintaining millimeter-level deformation detection accuracy even when ambient light intensity fluctuates by 30%. In practical applications, such as when operating on curing bio-glue materials, the algorithm can accurately distinguish between deformation caused by the material's own hardening and deformation caused by external disturbances, providing accurate timing feedback for subsequent force control.

[0102] The system can also adopt a fast feature change detection method based on an event camera. This method takes advantage of the fact that the event camera only records changes in pixel intensity, and can detect instantaneous deformation of the object surface with a delay of microseconds. It is particularly suitable for scenarios that require an immediate response to rapid deformation, such as monitoring sudden tissue deformation in precision surgery.

[0103] The non-rigid deformation analysis module analyzes the changes of the illumination coding pattern at different times based on the optical response characteristics to construct a non-rigid deformation field and predict the material state of the target object at the next moment;

[0104] The specific steps include:

[0105] Step 3.1, light encoding deformation tracking;

[0106] By analyzing the changes of the coded pattern projected onto the surface of the object at different times, a non-rigid deformation field is constructed. The deformation field describes the three-dimensional displacement vector of each point on the surface of the object, and its calculation method is:

[0107] ;

[0108] in, Represented in spatial coordinates Place, time point The three-dimensional deformation field vector describes the displacement of each point on the surface of the object; A mapping function representing the change of the coding pattern to the deformation field, which converts the optical observation data into the physical deformation; Indicated in coordinates Current time The observed coded pattern contains the optical characteristic information of the object in its current state; Indicated in coordinates At the initial moment The observed coded pattern serves as a deformation reference; Represents the initial reference time point. The mapping function is built based on the optical flow algorithm and deep learning model, and can handle complex nonlinear deformations.

[0109] The non-rigid deformation field construction model adopts a deep neural network structure, which includes an encoder-decoder architecture. The encoder part consists of 5 convolutional layers, each followed by batch normalization and ReLU activation function, which is used to extract multi-scale features from the encoded pattern changes; the decoder part contains 5 transposed convolutional layers, which are connected to the corresponding layers of the encoder through skip connections, gradually restoring the spatial resolution and outputting a dense deformation field.

[0110] The model is trained using a large amount of simulated data and a small amount of real annotated data, optimized using a combined loss function that includes a reconstruction error term and a smoothness constraint term. In minimally invasive surgery scenarios, the model can accurately reconstruct subtle deformations of the tissue surface from structured light images captured by an endoscope, identifying local deformations caused by pressure applied by surgical instruments, and helping surgeons accurately assess tissue stress.

[0111] For certain types of objects, such as layered structural materials, the system can use a layered deformation field model, which treats the object as multiple coupled deformation layers, tracking the deformation characteristics of each layer separately, and more accurately expressing the deformation state of complex internal structures.

[0112] Step 3.2, physical deformation model constraint optimization;

[0113] The acquired surface deformation field is combined with the physical deformation model, and the deformation field estimation result is optimized by minimizing the energy function. The energy function includes a data term and a regularization term:

[0114] ;

[0115] in, represents the total energy function, which is used to evaluate the overall quality of the deformation field estimation; It represents the data fit, which measures how well the estimated deformation field matches the observed data; Represents physical constraints to ensure that the deformation field conforms to physical laws; To balance the parameters, control the relative importance between the data term and the regularization term, the larger The value increases the influence of physical constraints. Physical constraints are based on elastic body mechanics models or finite element analysis, taking into account the material's properties such as continuity, elasticity, and volume conservation.

[0116] In some embodiments, the physical deformation model may use a hybrid model that combines data-driven and physical constraints. On the one hand, the model learns the behavior of materials from a large amount of deformation data, and on the other hand, introduces physical laws as constraints, which not only ensures the physical rationality of the model, but also has the ability to express complex nonlinear deformations.

[0117] Step 3.3, material state prediction;

[0118] Timing changes based on optical response characteristics , predict the material state of the object at the next moment , the calculation method is:

[0119] ;

[0120] in, Indicates the next moment The material state, including the predicted material mechanical properties; Indicates the current time The material state of the ,is used as the basic input for prediction; Indicates the change in optical response characteristics at the current moment, reflecting the change in the optical properties of the material surface; Represents the state transition function, which is constructed based on recurrent neural networks or long short-term memory networks and can simulate the dynamic response characteristics and state transition laws of materials; Represents the time step, which defines the time interval for prediction.

[0121] Material Status This includes mechanical parameters such as elastic modulus, viscosity, hardness, and their spatial distribution and temporal variation trends. These parameters collectively describe the mechanical properties of the material at different locations and times, providing a basis for decision-making in subsequent operations.

[0122] The material state prediction network utilizes a long short-term memory (LSTM) network architecture, consisting of an input layer, two LSTM layers (each with 128 hidden units), and a fully connected output layer. The network input is a sequence of optical response features within a time window, and the output is a vector of predicted material parameters. This network is able to capture the time-dependent and nonlinear variations of material states, making it particularly well-suited for processing the complex dynamic responses of viscoelastic materials.

[0123] When applied to liquid silicone curing operations, the network can predict the internal curing degree based on slight changes in the silicone surface reflection spectrum, detecting the critical point of state transition 15 to 30 seconds earlier than traditional time-based curing models, providing key guidance for the selection of operation timing.

[0124] For materials with complex phase change processes, such as the melting and solidification processes of thermoplastics, the system can use a neural network model guided by physical knowledge. This model directly encodes the physical laws of phase change into the network structure, thereby improving the accuracy of prediction of key state transition points.

[0125] A mechanical properties analysis module, which uses a deep learning model to analyze the relationship between optical response characteristics and the mechanical properties of the target object, and presets the operation force parameters based on the material state;

[0126] The specific steps include:

[0127] Step 4.1, optical-mechanical property correlation modeling;

[0128] Based on the pre-built data set, the deep learning model is trained to establish the mapping relationship between optical characteristics and mechanical parameters. The model adopts a multi-layer convolutional neural network structure, and the input is the optical response characteristics of the object. , the output is the estimated mechanical parameter vector , including elastic modulus, hardness, viscoelastic parameters, etc. The model training adopts supervised learning method, and the loss function is the mean square error between the predicted parameters and the true parameters.

[0129] In some embodiments, the model can adopt a multi-task learning framework to simultaneously predict multiple related mechanical parameters, and use the intrinsic correlation between parameters to improve prediction accuracy; it can also adopt few-sample learning technology to quickly adapt to new materials from a small number of labeled samples through meta-learning methods, reducing dependence on large amounts of labeled data.

[0130] Step 4.2, distributed mechanical property inference;

[0131] Apply the mechanical property inference model to different regions of the object to generate a spatial distribution map of mechanical parameters For objects with complex internal structures, the optical transmission and scattering properties are combined to infer the mechanical properties of different internal layers and construct a three-dimensional mechanical property volume model.

[0132] For composite materials with uneven internal structures, the system can combine computed tomography technology to reconstruct the distribution of internal mechanical properties through multi-angle optical measurement data, thereby improving the ability to perceive the internal state of heterogeneous materials.

[0133] Step 4.3, operation intensity and strategy preset;

[0134] Based on the inferred mechanical properties, the system presets the initial operating force and contact strategy for the robot's end effector. For different types of manipulation tasks (such as grasping, cutting, and separation), the system calculates the most appropriate combination of operating parameters, including contact position, contact angle, applied force, and operating speed, based on the task requirements and the object's mechanical properties.

[0135] The preset strategy is determined by the following optimization goals:

[0136] ;

[0137] in, It represents the optimal force parameter, which refers to the optimal force that the robot end effector should exert; It represents the optimal speed parameter, which refers to the optimal movement speed of the robot to perform the operation; It represents the optimal position parameter, which refers to the optimal contact position and posture of the robot end effector; represents the velocity parameter space, which contains all possible velocity choices; represents the speed parameter space, which contains all possible operating speed choices; represents the position parameter space, which contains all possible contact positions and posture choices; represents the damage risk assessment function; represents the operation error evaluation function; represents the operation time evaluation function; 、 、 Represent the weight coefficients of injury risk, operation error and operation time respectively; It means finding the parameter combination that minimizes the objective function, that is, finding the parameter setting that can achieve the best balance among damage risk, operation error and operation time.

[0138] The operation force preset algorithm adopts a multi-objective optimization method based on optical features. The algorithm takes into account the spatial heterogeneity and multi-dimensional mechanical parameters of the object, and can generate adaptive optimal parameter combinations for different operation tasks.

[0139] In medical applications, the algorithm can accurately infer the location and hardness distribution of cysts inside silicone simulated tissue from its optical properties, and accordingly plan the optimal operation path to avoid vulnerable areas, dynamically adjusting the contact force according to the local mechanical properties of the tissue, thereby maximizing operational efficiency while ensuring operational safety.

[0140] In some embodiments, the operation parameter optimization may adopt an evolutionary algorithm or a reinforcement learning method to gradually optimize the parameter combination by simulating multiple operation processes to find a strategy that can achieve the best balance between multiple objectives.

[0141] Closed-loop control module, used to adjust the robot's operating parameters in real time according to the non-rigid deformation field and material state, forming a closed-loop control mode;

[0142] The specific steps include:

[0143] Step 5.1, real-time status monitoring and evaluation;

[0144] During the operation, the system continuously monitors the illumination code, tracking the object's deformation state and changes in material properties in real time. Key status indicators are evaluated, including deformation degree, material state parameters, and operational response characteristics. When changes in the object's state exceed preset thresholds, an adjustment mechanism for the operational strategy is triggered.

[0145] The system can establish a multi-level status monitoring framework and set different response mechanisms for status changes of different criticalities, from parameter fine-tuning to operation suspension to strategy replanning, to achieve hierarchical response and improve the flexibility and stability of the system.

[0146] Step 5.2, dynamic adjustment of operating parameters;

[0147] Based on the real-time state evaluation results, the robot's operating parameters are adaptively adjusted. The force adjustment adopts the Proportional-Integral-Derivative (PID) control strategy, which dynamically corrects the force value according to the actual response characteristics of the object:

[0148] ;

[0149] in, Indicates time The adjusted force is the actual force value that the robot should apply after PID control; Indicates the preset force, which is the initial force value pre-set based on the previous analysis; Indicates state deviation, that is, the difference between the actual state and the expected state, reflecting the error degree of the current operation; is the proportional control parameter, which determines the response strength of the system to the current error; is the integral control parameter used to eliminate the steady-state error of the system, taking into account the cumulative effect of historical errors; It is a differential control parameter used to predict the future trend of the system and provide damping to reduce overshoot and oscillation of the system; Indicates the time from the start of the operation to the current moment The error integral reflects the historical accumulation of errors; It represents the rate of change of the error and reflects the changing trend of the system state deviation.

[0150] In some embodiments, the control parameters 、 and It can automatically adjust according to the material properties and current state of the object, forming an adaptive PID control, improving the system's adaptability to different materials. For highly nonlinear material responses, Model Predictive Control (MPC) methods can also be used, using a material state prediction model to plan the optimal control sequence for a period of time in the future.

[0151] Step 5.3, re-planning of operation strategy;

[0152] When the state of an object changes or unexpected circumstances arise in the operating environment, the system triggers a replanning process for the operation strategy. The replanning process considers the current state of the object, environmental constraints, and the operation goal to generate a new operation sequence.

[0153] Replanning utilizes a fast iterative optimization algorithm to ensure policy updates are completed within milliseconds, meeting real-time control requirements. In exceptional circumstances, the system will implement safety measures such as reducing the force of the operation, pausing the operation, or executing a predefined safe exit process.

[0154] The operational strategy replanning algorithm utilizes a real-time planning approach based on dynamic risk assessment. This approach transforms changes in an object's state into a risk map and searches for the optimal operational path within that map in real time. A key advantage of this algorithm lies in its ability to handle unexpected changes during operation. For example, if a sudden increase in the hardness of a biological tissue is detected, the system can quickly calculate an alternative path around the high-risk area without interrupting overall operation.

[0155] In collaborative operation scenarios, the system can also integrate an operation intention prediction model to analyze the human operator's action sequence, predict their operation intention, and adjust the robot's collaborative strategy accordingly to achieve intelligent operation control of human-machine collaboration.

[0156] Application examples of this implementation:

[0157] The following describes in detail the practical application effects of the operating robot control method based on multi-dimensional illumination coding provided by the present application in combination with specific application scenarios.

[0158] Tissue manipulation applications in medical surgery scenarios:

[0159] In the field of minimally invasive surgery, especially in liver tumor resection, surgeons often need to accurately distinguish and operate on tissue areas of different hardness. The control method of this application is applied to a surgical robot-assisted system, and the specific implementation process is as follows:

[0160] Before surgery, the system collected material properties from simulated liver tissue (a silicone model containing a simulated tumor area). Using a spectral scanner with a wavelength range of 450-950nm, the system analyzed the simulated tissue's spectral characteristics within three seconds, confirming its translucent elastomer material and capturing its reflectivity curve.

[0161] Based on the collected material data, the system automatically selects a random dot matrix and gradient stripe hybrid pattern specifically designed for translucent materials as the illumination encoding scheme. The encoded pattern sequence contains 23 different structured light patterns, each composed of a 128×128 pixel grayscale image, forming a clearly recognizable light spot pattern on the simulated liver surface. The DMD projection system projects these patterns in a loop at a frequency of 120Hz, ensuring continuous acquisition of tissue deformation information during the procedure.

[0162] During the surgical procedure, as the robot's end-effector approached the simulated liver surface, a multi-angle optical sensor array (consisting of six high-speed cameras with a 500fps acquisition rate) synchronously captured the reflected coded pattern from different angles. An optical response feature extraction algorithm constructed an optical response map of the tissue surface within 10ms. A temporal feature change analysis algorithm detected significant optical property differences at the interface between the simulated tumor and normal tissue, with the spectral reflectance characteristics in this region exhibiting distinct temporal response patterns.

[0163] After receiving the optical signature data, the deformation field construction model generates a high-precision 3D deformation field within 30 milliseconds, accurately displaying minute deformations on the simulated tissue surface (with an accuracy of 0.08mm). Simultaneously, the material state prediction network analyzes the temporal changes in the optical response signature and predicts that the hardness of the simulated tumor region is 2.7 times that of normal tissue, with a hardness gradient within it.

[0164] Based on this information, the mechanical parameter mapping model generates customized operating force parameters for different areas: the recommended contact force for normal tissue is 0.8N, the area surrounding the tumor is 1.2N, and the tumor core is 1.5N. The surgical robot system uses these preset force parameters to ensure that the operation does not cause excessive pressure damage to surrounding healthy tissue.

[0165] During the subsequent simulated resection operation, when the robot's end effector contacted the tissue surface and began applying force, the system monitored tissue deformation in real time and detected unexpected softening in a localized area (a change in optical characteristics exceeding a threshold of 25%). The adaptive operation strategy execution module immediately triggered parameter adjustments, reducing the operating force in that area from 1.2N to 0.7N within 52ms. It also adjusted the operation path to avoid the abnormally softened area and ensure operational safety.

[0166] During the entire operation, the system continuously performed closed-loop control of "prediction-verification-adjustment", identified a total of 5 areas with abnormal tissue properties, and adjusted the operating parameters for each area in real time. It finally completed the precise separation operation of the simulated tumor without causing any visible damage to the surrounding simulated healthy tissue.

[0167] Applications in flexible electronic assembly scenarios:

[0168] In the field of flexible electronics manufacturing, precise manipulation of thin-film circuits and deformable electronic components is a key challenge. This application's control method is applied to a high-precision flexible circuit assembly system, achieving precise manipulation and control of deformable materials:

[0169] Before the assembly process begins, the system scans the polyimide flexible substrate (25μm thick) for its material properties and detects its highly reflective surface. Based on this information, the system selects a high-frequency phase-shifted sinusoidal grating pattern as the illumination encoding scheme. This pattern consists of 15 sets of structured light patterns with different frequencies and phases, creating clear reference marks on the highly reflective surface.

[0170] During component placement, multi-angle optical sensors (four fixed-angle cameras and one dynamic camera that moves with the robotic arm) capture the deformation of the flexible substrate in real time. When the system detects minute fluctuations (maximum amplitude 0.3mm) in the substrate under the influence of airflow, a deformation field model instantly generates a 3D deformation field, accurately mapping the fluctuations in the substrate surface.

[0171] The material state prediction network analyzes the thermodynamic properties of the substrate and components, predicting that during soldering, for every 10°C increase in local substrate temperature, the material's elastic modulus decreases by approximately 5%. Based on this information, the system generates a temperature-material property map to provide a reference for subsequent operations.

[0172] During the placement of micro-components (0201-size resistors, 0.6mm x 0.3mm), the mechanical parameter mapping system, based on real-time optical analysis data, determined the optimal placement force to be 0.05N, significantly lower than the 0.15N used by traditional fixed-parameter systems. Placement accuracy was controlled within ±0.01mm, maintaining stability even with minor substrate vibrations.

[0173] When the ambient temperature unexpectedly rose by 3°C, the system immediately detected the change in the substrate material state (flexibility increased by 7%). The adaptive operation strategy execution module replanned the placement strategy in less than 100ms, adjusting the robot arm's approach speed from 5mm / s to 3mm / s, and reducing the placement force to 0.04N, ensuring precise positioning of components without damaging the substrate.

[0174] During a four-hour assembly test, the system precisely placed 2,500 micro-components, maintaining a positioning accuracy of ±0.015mm, surpassing the ±0.035mm achieved by conventional methods. Comparative testing showed that the system using this application's method achieved a 98.5% component placement success rate when processing deformed substrates, while conventional fixed-parameter systems achieved only 92%.

[0175] Technical effect verification:

[0176] The control method for an operating robot based on multi-dimensional illumination coding provided in this application has demonstrated certain technical effects in practical applications, as shown in Tables 1 and 2, which are specific data verifications of two key effects:

[0177] Table 1, Comparison of non-rigid deformation tracking accuracy:

[0178]

[0179] Table 2, comparison of first contact force control accuracy:

[0180]

[0181] Through the above practical application examples and data verification, it is fully proved that the control method provided in this application has certain technical advantages when processing non-rigid deformable objects, and can achieve sub-millimeter deformation tracking accuracy and high-precision first contact force control, greatly reducing the risk of object damage during operation and improving the success rate and efficiency of operation.

[0182] The above describes an embodiment of the present invention, but this embodiment is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Ordinary technicians in this field can also make more forms of equivalent embodiments based on the inspiration of this embodiment, all of which are protected by this embodiment.

Claims

1. A manipulator robot with lighting recognition, characterized in that: include: An illumination coding projection module is used to generate an illumination coding pattern sequence according to the material characteristics of the target object and project the illumination coding pattern sequence onto the surface of the target object; A multi-angle optical perception module is used to capture the optical response data of the target object under the projection conditions of the light-encoded pattern sequence and obtain the optical response characteristics of the target object; A non-rigid deformation analysis module constructs a non-rigid deformation field based on the optical response characteristics of the illumination coding pattern at different times. It also uses a long short-term memory network structure to predict the material state of the target object at the next moment. The long short-term memory network comprises an input layer, two LSTM layers, and a fully connected output layer. The two LSTM layers each contain 128 hidden units. The network input is the optical response feature sequence within a time window, and the output is the predicted material state, which includes the elastic modulus, viscosity coefficient, and hardness. The mechanical property analysis module is used to use a deep learning model to analyze the relationship between the optical response characteristics and the mechanical properties of the target object and preset the operation force parameters. The specific steps include: based on a pre-constructed data set, training the deep learning model to establish a mapping relationship between optical characteristics and mechanical parameters; the model adopts a multi-layer convolutional neural network structure, with the input being the optical response characteristics of the object and the output being an estimated mechanical parameter vector, including elastic modulus, hardness, and viscoelastic parameters. For different areas of the object, a spatial distribution map of the mechanical parameters is generated; based on the inferred mechanical parameters, the initial operation force and contact strategy of the robot end effector are preset; for different types of operation tasks, the most suitable operation parameters are calculated according to the task requirements and the object mechanics, including contact position, contact angle, applied force, and operation speed; The closed-loop control module is used to adjust the robot's operating parameters in real time according to the non-rigid deformation field and material state to form a closed-loop control mode.

2. The operating robot with lighting recognition according to claim 1, characterized in that: Generating the illumination coding pattern sequence in the illumination coding projection module includes: Perform spectral scanning tests to obtain basic material characteristic parameters of the target object; Based on the basic material characteristic parameters, the corresponding coding pattern type is selected. The coding pattern types include: a high-frequency phase-shifted sinusoidal grating pattern for highly reflective materials, a mixed pattern of random dot matrix and gradient stripes for translucent materials, or an orthogonal cross multi-directional stripe pattern for fabric materials; The optimal encoding parameters are determined by maximizing the deformation response function through a Bayesian optimization framework.

3. The operating robot with lighting recognition according to claim 1, characterized in that: The optical response characteristics of the target object obtained in the multi-angle optical perception module include: Preprocessing, encoding pattern recognition and feature calculation are performed on the image data collected by the multi-angle optical sensor array to form an optical response feature map; The optical feature changes between consecutive time frames are analyzed through the inter-frame feature comparison method enhanced by spatiotemporal correlation, and the feature response change is calculated. This method can distinguish the feature changes caused by the deformation of the object itself and the changes in ambient lighting.

4. The operating robot with lighting recognition according to claim 1, characterized in that: The non-rigid deformation analysis module constructs a non-rigid deformation field using a deep neural network structure, including an encoder-decoder architecture, where: The encoder part includes a convolutional layer for extracting multi-scale features from the coding pattern changes; The decoder part contains transposed convolutional layers, which are connected to the corresponding layers of the encoder through skip connections, gradually restoring the spatial resolution and outputting a dense deformation field; The deep neural network is optimized by a combined loss function, which includes a reconstruction error term and a smoothness constraint term.

5. The operating robot with lighting recognition according to claim 1, characterized in that: The closed-loop control mode in the closed-loop control module includes: In the prediction phase, the optimal parameters for the next operation step are predicted based on the currently acquired optical response characteristics and historical operation data; In the verification phase, the response state of the target object is monitored in real time during the operation, and the deviation between the actual response and the predicted response is calculated; During the adjustment phase, when the deviation exceeds the preset threshold, the parameter recalculation process is triggered, and a smooth transition to the new operating parameters is achieved without interrupting the operation.

6. The operating robot with lighting recognition according to claim 1, characterized in that: It also includes an abnormal state detection module: Establishing a distribution model of the optical response characteristics of the target object under normal conditions; Real-time calculation of the Mahalanobis distance between the current optical response characteristics and the normal distribution model; When the Mahalanobis distance exceeds the dynamic threshold, the exception handling process is triggered, including operation suspension, parameter reset, and operation path replanning.

7. The operating robot with lighting recognition according to claim 1, characterized in that: The closed-loop control module adjusts the robot's operating parameters in real time based on a multi-objective optimization algorithm, taking into account: Operational accuracy goal, minimizing the deviation between the operation position and the target position; Safety objectives: ensuring that the operating force does not exceed the safety threshold of the target object; The efficiency goal is to minimize the operation time while meeting the accuracy and safety constraints.

8. A computer-readable storage medium, characterized in that It is used to store computer-readable instructions, and when the computer-readable instructions are read by a computer, it can run an operating robot with lighting recognition as described in any one of claims 1 to 7.

Images