Lighting control in robotic end effector manipulation

Abstract

A system component includes: a processor circuit system; and a non-transitory computer-readable storage medium including instructions that, when executed by the processor circuit system, cause the processor circuit system to: receive image data of an object captured by a camera; analyze visual features of the object based on the received image data; generate an illumination pattern based on the analyzed visual features; and control an array of light sources integrated into a plurality of fingers of a robotic gripper to project the illumination pattern within a grasping volume defined by the plurality of fingers during object manipulation, thereby enhancing the detection of visual features of the object, wherein each light source in the light source array is individually controllable.

Description

Background Technology

[0001] Collaborative robots (cobots) are increasingly seen as a cost-effective solution for automating high-mix, low-batch processes. However, their application faces significant challenges when handling small, fragile objects common in semiconductor manufacturing and high-precision applications. Existing end effectors lack the ability to perform pressure-sensitive handling of sub-centimeter objects, particularly those made of delicate, reflective, or refractive materials, such as glass and semiconductor components used in optoelectronics.

[0002] Previous methods for addressing these limitations relied on static light sources or light sources mounted on the robot's wrist. However, such configurations have drawbacks. There is occlusion at the tool center point (TCP) and blurring due to hard shadows or vibrations. The robot's range of motion is limited due to increased clearance requirements for the lighting source. Robot joint limitations are reduced to avoid self-collisions. Furthermore, the operation time is prolonged due to the alternation between sensing and recognition for six-dimensional pose estimation and grasping. In practice, these drawbacks create obstacles in industrial and manufacturing scenarios involving cluttered, small equipment, pallets, or machine maintenance rooms where tools, parts, and other auxiliary components must be picked and placed. Attached Figure Description

[0003] Figures 1A-1D The illustration shows a robotic system with a vision-tactile grasping system according to various aspects of this disclosure.

[0004] Figures 2A-2B The illustration shows the electromechanical aspects of a near structured illumination (NSI) unit according to various aspects of this disclosure.

[0005] Figures 3A-3B The diagram illustrates the structure of a near-structured illumination (NSI) unit in the signal domain according to various aspects of this disclosure.

[0006] Figure 4 (spanning four sheets of paper and) Figures 4A-4D The diagram illustrates an architecture for generating adaptive dynamic lighting patterns according to various aspects of this disclosure.

[0007] Figures 5A-5B The illustration shows an artificial intelligence module for generating dynamic lighting patterns from an image, according to various aspects of this disclosure.

[0008] Figures 6A-6D The illustration shows tactile sensing and self-calibration according to various aspects of this disclosure. Detailed Implementation

[0009] To address challenging grasping scenarios involving small objects and non-Lambertian surfaces, aspects of this disclosure provide dynamic lighting from multiple sources pointing towards the object near the grasping point. Both the end effector and the camera remain stationary, manipulating only the light pattern. This approach employs simple software-based inter-frame synchronization between the finger-integrated light and one or more in-hand cameras.

[0010] I. Electromechanical aspects of near-structured lighting (NSI) units

[0011] Figures 1A-1D The illustration shows a visual-tactile grasping system (robotic system) 100 with a visual-tactile grasping system according to various aspects of the present disclosure.

[0012] Figure 1A The illustration shows the hardware setup of a robotic system 100, including a collaborative robot (cobot) equipped with a gripper 110 having fingers. The gripper 110 integrates a tactile differentially-fused near-structured illumination (T-Diffused-NSI) unit 120, characterized by a light source 122, into each of the fingers. In this example, an RGB (W) (red, green, blue, and white) LED (light-emitting diode) light source is capable of achieving 24-bit color depth. Alternatively, the light source could be an LED in the non-visible spectrum coupled to one or more multispectral cameras.

[0013] Figure 1B The illustration shows one of a haptic differential fusion NSI unit 120 (hereinafter referred to as "NSI unit") configured to dynamically generate adaptive lighting patterns. This enhances the haptic feedback generated by... Figure 1A The economical ultra-high resolution USB-RGB camera 130 controls the detection of micron-level discontinuities with high sharpness and signal-to-noise ratio.

[0014] Figure 1C The illustration shows the NSI unit 110 integrated into the corresponding finger of the gripper.

[0015] Figure 1D The illustration depicts a gripper that uses artificial intelligence (AI)-driven illumination to grasp reflective / refractive objects, such as optoelectronic components. The gripper effectively handles sub-millimeter misalignments in manufacturing pallets during inspection, assembly, and metrology processes.

[0016] Figures 2A-2B The illustration shows a near-structured lighting (NSI) unit 200 according to various aspects of this disclosure. Figures 1A-1DIn the electromechanical aspects of 120, among which Figure 2B It is an exploded view.

[0017] NSI unit 200 (120) includes finger unit assembly 210, which includes USB data and power connectors, test probes, and a dielectric elastomer for haptic / force sensing. Mounting frame 220 encapsulates the components and includes a microcontroller unit (MCU) 230 (a component with a processor circuitry) configured for haptic analog sampling, signal smoothing and linearization, control of the dynamic lighting subsystem, management of communication with the robot host unit based on the robot operating system (ROS), etc. Printed circuit board (PCB) 240 mounts both static components (LEDs, MCU, capacitors, and resistors) and dynamic components including replaceable haptic (pressure) transducers / sensors. An array of light sources 250 (LEDs or dynamic lighting sources) contains red, green, blue, and white (RGBW) channels for projecting accurate luminance and chromaticity patterns in high dynamic range modes. The example shown has 18 light sources. The quick-replacement attachment mechanism, socket 260, and transducer 270 enable rapid sensor replacement and automatic calibration. The pressure transducer 280 exhibits characteristic force / voltage behavior as shown in graph 270b. A semi-transparent diffuser with a protective shield 290 acts as a bandpass filter for the illumination space, while protecting the PCB and LEDs from impacts and slippage during gripping operations. The NSI unit 200 (120) integrates both tactile sensing and dynamic lighting capabilities with a compact form factor optimized for robotic gripping applications.

[0018] II. Learning optimal dynamic lighting from simulation to reality

[0019] A. Dynamic lighting Gabor function basis

[0020] Figures 3A-3B The figure illustrates a near-structured illumination (NSI) unit 300A with respect to the signal domain 300B according to various aspects of this disclosure. Figures 1A-1D The structure of (120) is shown in the figures. These figures use Gabor kernel dynamic lighting with five degrees of freedom. Reference numeral 310 shows the robot finger unit, while reference numeral 320 is the configuration of the light source (LED) array.

[0021] The set of RGB (W) pixels, 330, is considered an image. Due to the space occupied by haptic feedback, fasteners, and the MCU, some elements... They do not exist. Formally, these elements are computed as if they exist. In reality, due to the superposition and diffusion of coverage, a few of these elements remain unchanged in the application.

[0022] Reference set 340 for illumination base It consists of a set of discrete Gabor filters, where each value is mapped to a brightness value. or chromaticity value For aperture Phase , directional (because and Smoothness / Sharpness and spatial aspect ratio Gabor It includes five degrees of freedom (350).

[0023] Due to the size and number of elements in g, only a subset of values ​​produces visible patterns beyond constant illumination. Figure 360 ​​illustrates five different illumination patterns ( arrive The phase of the Gabor function is altered to produce a shifted illumination pattern. These functions, combined with the saliency extractor 4110, allow for the training of the AI ​​model as described below.

[0024] Figure 4 (spanning four sheets of paper) Figures 4A-4D The diagram illustrates an architecture 400 for generating adaptive dynamic lighting patterns according to various aspects of this disclosure.

[0025] Architecture 400 includes three processing stages that work together to enable lighting control during robot manipulation.

[0026] B. Offline geometric process

[0027] Figure 4A and Figure 4B Block diagrams of offline geometric process stages 400A and 400B according to various aspects of this disclosure are shown.

[0028] The offline geometry process stages 400A and 400B include five processing stages 410-450, which generate a visible structural region 450 and a pre-grabbing pose 410d optimized for both visibility and grasping functionality.

[0029] The process begins with grasping synthesis using the robot's URDF (Unified Robot Description Format) 410a and associated inverse kinematics function 410b to generate a feasible grasping sequence 410c. The system stores three frames: pre-grasp (before the finger gripper closes), grasp (contact with the target object), and post-grasp (non-contact with a collision-free location based on scene post-conditions). The resulting pre-grasp poses 410d are stored in a database after ensuring that they maintain the visibility of the structural regions of both the tray and the target object.

[0030] The system uses parameter deviations, represented as probability functions, to process the input describing the spatial location of the target object. This enables a bounded variational generation process within physically reasonable parameters. The input CAD files for both pallet 430a and target object 440a are processed into a surface-subdivided mesh with small crease angles, allowing for efficient removal of invisible structural elements such as coplanar edges and short segments along curvature, such as chamfers and other smoothing mesh artifacts. Similarly, the system filters edges by length and aperture while identifying concave coplanar regions (430b, 440b) of effective area.

[0031] This unidirectional process selects and associates visually salient areas that remain detectable even if reflections or refractions are caused by discontinuous boundaries. The final result is stored in a visible structural region 450, which maintains the relationship between the visible structural region and its associated grasping posture, allowing the system to optimize both visual detection and physical manipulation capabilities. The resulting set of pre-grasping postures ensures that the visible structural regions of the tray or target object are not occluded and is stored in a database.

[0032] C. Offline simulation and self-annotated dataset creation

[0033] Figure 4C The diagram illustrates a block diagram of the offline simulation and self-annotated dataset creation process 400C according to various aspects of this disclosure.

[0034] The offline simulation and self-annotation phases of the 400C implement a pipeline for generating training data, enabling the creation and training of encoder-dual-decoder models for adaptive dynamic lighting pattern generation. These patterns dynamically illuminate the scene under inspection to amplify the salience of visual elements used for grasping.

[0035] The process begins by creating a training dataset 4120 by ensuring sufficient variability in the vantage points and poses of the target object in the tray. Parameter sampler 460 is derived from a labeled six-dimensional placement distribution 420. Figure 4AThe configuration obtained (as shown in the image) defines whether the target object is stable within its slot or unstable due to partial contact. Using this sample configuration, which represents the complete spatial layout of objects in the scene, the structure selector 470 identifies edges or regions that serve as key grab points for subsequent rendering.

[0036] The system then selects a lighting pattern from the lighting pattern set 480a. and (480b) and apply them (480c) to the simulated scene 490 ( Figure 4B (As shown in the diagram). This creates a fully composite scene 4100a using sampled spatial layout and lighting, achieving realistic rendering with a pixel-wise visibility index of geometric features (edges and bulges) encoded in a five-channel image format (RGB 4100b, [indexed edges, indexed-concave-region] 4100c) to prevent aliasing. For efficiency, this rendering can be performed only in the region of interest.

[0037] In the saliency calculation phase 4110, the system applies edge detection (Gabor-Jet) and semantic segmentation. The system retains only image pairs that show minimal correlation (cross-union ratio of 5%-10%) between visible edges and stable regions for subsequent processing stages.

[0038] These patterns dynamically illuminate the scene during inspection to amplify the salience of key visual elements used for grasping, thereby creating a self-supervised dataset that enables adaptive lighting control during robot manipulation tasks.

[0039] Figure 4D The diagram illustrates the block diagram of the AI ​​module for image-to-dynamic patterned lighting generation 4130a, which is referenced below. Figure 5A and Figure 5B It is described in detail.

[0040] Figures 5A-5B The illustration shows an artificial intelligence architecture 500 for generating dynamic lighting patterns from images according to various aspects of this disclosure.

[0041] D. AI model training for autoencoders and dual decoders

[0042] Figure 5A The diagram illustrates an AI training and inference architecture 500A for adaptively creating sequences of dynamic lighting pairs in a self-supervised manner.

[0043] The inference architecture 500A employs an autoencoder 510 and dual decoders 520 and 530 to adaptively create dynamic lighting patterns. The system begins with an input image. These input images were captured under low-light conditions without dynamic illumination, which allowed for proper focusing at a given object distance. These images were... Figure 4C It was generated in 4100a.

[0044] Based on the set of lighting patterns (480) The system selects components with low cardinality based on the block diagram in the 4100c. subset of This range is directly calculated using the significance function in 4110. This process utilizes tuples of the following form. :

[0045]

[0046] The structure of an automatic encoder includes: an encoding subnet. It maps the input image into the latent space Z; and has two decoder subnets: one used only during training to shape the latent space. decoder base 520 Decoder extension 530 for generating lighting pattern pairs .

[0047] Once the model is trained, this subnet is no longer needed during inference, significantly reducing workload. On the other hand, the decoder expands by 530 to access the latent space. Input (query image) and targeted prompts ) and related targeted prompts To decode as a pair of lower-dimensional lighting patterns in a single reconstructed tensor .

[0048] E. AI model inference and variational orientation cue for extended decoder

[0049] Figure 5B The illustration shows a runtime inference architecture 500B according to various aspects of this disclosure.

[0050] During runtime inference, the system processes real-world images from camera images without dynamic lighting, applying high-orientation cues. To create code Then, the decoder expands 530 to generate... A lighting pattern These patterns are applied to illuminate the scene, resulting in images with enhanced structural features, similar to those learned by a convolutional neural network (CNN).

[0051] The architecture 500 enables adaptive generation of lighting patterns that optimize the visibility of object features during robot grasping operations, while maintaining a compact and efficient implementation suitable for real-time operation.

[0052] F. Sensitivity and vision-based automatic calibration

[0053] Figures 6A-6D The illustration shows tactile sensing (including sensitivity, linear behavior) and automatic calibration 600 according to various aspects of this disclosure.

[0054] Figure 6A The diagram illustrates signal traces 610 and 620 to demonstrate the sensitivity of the demonstration system. Reference numeral 610 indicates a subtle contact that creates peaks with small ADC discrete biases, filtered within a 32-bit MCU operating at 32MHz. Reference numeral 620 indicates a similar artifact that can be observed during steady-state pressure conditions.

[0055] Figure 6B The diagram illustrates the three stages (630-660) of the vision calibration process 600B. The vision-based calibration process utilizes a compressible rubber ball 670 (with unknown compressibility). (The props) are used as calibration tools.

[0056] The calibration process 600B begins with measuring each grip side. pressure As shown at position 640. The system calculates the average value over one second at a sampling rate of 1 kHz during non-contact conditions. and standard deviation This provides approximately 1,000 samples for reliable calibration.

[0057] As the gripper closes in the progressively closed-loop gripper-encoder assembly, calibration continues. Here, the ball's center of mass and its contour are tracked to identify changes in contact, as shown at 650. In practice, the closure continues until both tactile sensors detect contact. The fingers and wrist then open and move separately to ensure the ball is centered.

[0058] The final calibration phase involves shutting down the grip while observing sensor readings until saturation occurs or the grip motor approaches 50% of its power limit to prevent overload and prop damage. This creates a signal ramp at a constant temperature, characterizing the prop's compressive behavior. ,in Represents the volume of a sphere / sphere, in meters (m). 3 ,and Indicates that it is due to increased pressure The resulting deviation. The calibration process 600B uses an ellipsoidal model with a compressed spherical half-axis. The volume approximation is obtained using simplified biaxial observations (ab). Therefore, the approximation is as follows: Here, camera calibration (distortion compensation, i.e., unwrapping) and the known size of the sphere radius r allow for a pixel-to-millimeter conversion approximation. Furthermore, only the elliptic semi-axis 2 (ab) is observable from the top view (abc), thus the system is... Perform further approximations. Based on URDF and its encoder... The setting of the aperture on the grip restricts approximate physical consistency, but allows for... Establish a deformable visual two-degree-of-freedom approximation. In the absence of known facts, it is through This data was obtained from experimental data or work on pseudo-Pascal. The latter method provides a linear behavior to model the proportion of applied force. This means assuming that the tactile readings follow a Gaussian noise distribution and observing... and As a function of the aperture of the gripper rather than the time domain, the visual approximate volume change of the gripper and the aperture are related to... Related. This simplified model establishes a linear relationship between aperture, ball drag, and applied force / pressure, neglecting the case of perfect contact at the tactile sensor surface and the fact that the ball does not maintain its bulk modulus throughout the entire aperture / compression range of the gripper. It considers temperature, humidity, and compressibility. Full finite element modeling of the distribution is impractical in deployment. Therefore, this approximation allows mapping the 12-bit ADC value and voltage divisor setting to a proportional pressure value in units of . , and unknown compressibility Proportional.

[0059] Figures 6C-6D (Figure references 680 and 690) demonstrate the system's high sensitivity and signal-to-noise ratio capabilities. Figure reference 680 refers to the detection of slight human touch with a high amplitude response, while figure reference 690 refers to the system's ability to detect small vibrations transmitted from the texture of a ball during finger gliding, thus providing sufficient nuance for inference across various use cases or surface characteristics. This allows for consistent pressure threshold determination for grasping objects, regardless of location, orientation, and finger aperture, calibrated up to a scaling factor of compressibility κ.

[0060] III. Application of visual-tactile grasping in the manufacturing of fragile products

[0061] This system can optionally utilize an array of light sources 122 integrated into the fingers of the gripper and a camera mounted on the end effector to generate a detailed multidimensional model of the object. By synchronizing dynamic lighting patterns projected from multiple angles with camera capture, the system enables rapid acquisition of object features under varying lighting conditions. This is particularly advantageous for Neural Radiance Field (NeRF) and neural point-based graphics (splatting) techniques, in which controlled lighting and different viewpoints enhance the learning of implicit multidimensional representations—especially for objects with complex surface properties, such as reflective and refractive materials commonly found in semiconductor components.

[0062] Semiconductor, pharmaceutical, and biotechnology manufacturing processes and technology development often occur in highly complex environments that present challenges to material handling due to the sensitivity, fragility, and cleaning intensity of workpieces, as well as stringent cost constraints. Existing solutions for transporting partially assembled workpieces are typically complex, expensive, and task-specific, limiting their flexibility. Consequently, many highly complex or non-value-added operations, such as metering, frequently rely on manual handling, increasing the risks of contamination, damage, and human error.

[0063] In contrast, the visual-tactile approach for grasping via collaborative robots, as disclosed herein, offers a more cost-effective option for automated material handling while reducing the risks associated with manual operation. By integrating real-time information, this approach dynamically avoids obstacles during pickup and identifies optimal pickup parameters (such as position, orientation, and lighting) for each unique situation. Unlike existing solutions that may require costly redesigns to accommodate a wide range of sample sizes, the aspects disclosed herein provide a single, scalable solution that delivers robust quality and cleanliness comparable to automated equipment, but at a lower cost and overhead.

[0064] Furthermore, this vision-tactile approach can be applied to various other small-batch, highly complex processes involving sensitive workpieces, where effective human-machine collaboration improves overall process quality, output, and throughput. In such environments, it is considered a superior method compared to risky manual handling and expensive fully automated systems.

[0065] Furthermore, aspects of this disclosure particularly overcome the limitations of existing technologies in semiconductor and pharmaceutical manufacturing applications. Traditional robotic grippers with tactile sensors often suffer mechanical stress due to compression and wear, leading to frequent recalibration and preventative replacement. In contrast, the disclosed solution integrates a durable transducer with automatic self-calibration capabilities, enabling rapid and cost-effective replacement without the need for specialized tools, saline fluids, or significant engineering time. This solution significantly reduces downtime and operating costs, providing a seamless approach to visual-tactile material handling without the need for replacement instruments. Moreover, sensors using silicone with bonding properties are unsuitable for the semiconductor and pharmaceutical manufacturing industries.

[0066] Furthermore, various aspects of this disclosure introduce active lighting that overcomes the challenges posed by reflective and refractive surfaces, variable 6D object poses, and non-Lambertian materials. Unlike previous setups that relied on heavy, rigid, and fixed configurations, this system integrates compact, energy-efficient lighting directly into the robot's wrist and links. By eliminating the need for bulky setups and additional footprint, this system ensures reliable visual perception while maintaining flexibility and adaptability for highly complex, low-volume applications such as machine maintenance, inspection, and assembly.

[0067] Furthermore, this solution addresses the limitations of existing active lighting methods. It reduces computational and coordination overhead through single-source or dynamically modulated illumination, minimizing sensitivity to occlusion and dynamic shadows. The streamlined and compact end effector design eliminates bulky components and complex wiring that restrict movement in conventional systems, enabling precise manipulation even in confined environments. This adaptability makes it suitable for high-precision tasks requiring dynamic handling and fine servoing in demanding manufacturing environments.

[0068] By combining enhanced tactile sensing, automatic calibration, and advanced visual perception technologies, the disclosed solution redefines the efficiency, reliability, and versatility of manufacturing automation.

[0069] The technology disclosed herein can also be described in the following examples.

[0070] Example 1. A component of a system includes: a processor circuit system; and a non-transitory computer-readable storage medium including instructions that, when executed by the processor circuit system, cause the processor circuit system to: receive image data of an object captured by a camera; analyze visual features of the object based on the received image data; generate an illumination pattern based on the analyzed visual features; and control an array of light sources integrated into a plurality of fingers of a robotic gripper to project the illumination pattern within a grasping volume defined by the plurality of fingers during object manipulation, thereby enhancing the detection of visual features of the object, wherein each light source in the array of light sources is individually controllable.

[0071] Example 2. According to the components of Example 1, wherein: each of the light sources includes an RGB (W) (red, green, blue, and white) light-emitting diode element (LED element), or an LED in the non-visible spectrum coupled to a multispectral camera, configured to project light of variable intensity or color, and the instructions further cause the processor circuitry to generate an illumination pattern by dynamically varying the intensity or color balance of each of the light sources.

[0072] Example 3. According to any one or more of the components in Examples 1-2, wherein the instructions further cause the processor circuitry to: dynamically control the array of light sources to project an illumination pattern within the grasping volume to create a shifted illumination wavefront for edge detection, wherein the shifted illumination wavefront enhances the detection of horizontal, vertical, or diagonal edges and infers surface characteristics.

[0073] Example 4. Based on any one or more of the components in Examples 1-3, wherein the instructions further cause the processor circuitry to: receive pressure data from a pressure sensor integrated into the finger; and adjust the manipulation of the object based on the received pressure data.

[0074] Example 5. Based on any one or more of the components in Examples 1-4, wherein the instructions further cause the processor circuitry to: receive pressure data from a pressure sensor integrated into the finger; acquire visual feedback of the deformation of a compressible calibration object; and automatically calibrate the pressure sensor based on the pressure data and the deformation of the compressible calibration object.

[0075] Example 6. Based on any one or more of the components in Examples 1-5, wherein the instructions further cause the processor circuitry to: capture a sequence of images using a camera mounted on a robot gripper while controlling an array of light sources to project different lighting patterns within the gripping volume; generate a multidimensional model of the object based on the captured sequence of images and the corresponding lighting patterns; and adjust subsequent lighting patterns based on features detected in the multidimensional model to enhance visual detection of the object geometry during manipulation.

[0076] Example 7. Based on any one or more of the components in Examples 1-6, wherein the instructions further cause the processor circuitry to: receive a model of an object; extract visible features from the model; generate a set of lighting patterns using a dynamic kernel function and a saliency function; apply the generated lighting patterns to a simulated scene including the object; evaluate the saliency of the lighting patterns based on the detection of visible features; select a lighting pattern that achieves a minimum saliency threshold; and train a neural network using the selected lighting patterns to generate a lighting pattern decoder for runtime operation.

[0077] Example 8. Based on the components of Example 7, wherein a set of lighting patterns is generated by a kernel function by varying aperture, phase, orientation, smoothness, or spatial aspect ratio parameters.

[0078] Example 9. According to any one or more of the components in Examples 1-8, wherein the instructions further cause the processor circuitry to: generate a training dataset using an object model and simulated lighting patterns; train a neural network using the training dataset; and generate lighting patterns using the trained neural network during operation.

[0079] Example 10. According to any one or more of the components in Examples 1-9, wherein the instructions further cause the processor circuitry to: encode image data into a latent spatial representation; and decode the latent spatial representation into illumination control parameters for a light source array.

[0080] Example 11. According to the components of Example 10, wherein the instructions further cause the processor circuitry to: use the base decoder to shape the latent space representation during training; and use the extended decoder to generate lighting control parameters during runtime operation.

[0081] Example 12. According to the components of Example 10, wherein the instructions further cause the processor circuitry to: encode a camera image captured without dynamic lighting into a latent spatial representation; and decode the latent spatial representation into multiple pairs of lighting patterns for a finger.

[0082] Example 13. Based on any one or more of the components in Examples 1-12, wherein the instructions further cause the processor circuitry to: extract visible geometric elements from a model of an object; define a visible structural region based on the extracted geometric elements; and determine a lighting pattern based on the visible structural region.

[0083] Example 14. According to the components of Example 13, wherein the instructions further cause the processor circuitry to: receive a parameter placement distribution defining the possible positions and orientations of objects; generate a scene layout based on the parameter placement distribution; and generate training data by simulating lighting of the scene layout.

[0084] Example 15. According to the components of Example 14, wherein the instructions further cause the processor circuitry to: render an image of a simulated scene layout with and without a determined lighting pattern; generate a structural map encoding the geometric features of the rendered image; and evaluate the saliency of the geometric features to select a lighting pattern that enhances feature detection.

[0085] Example 16. According to any one or more of the components in Examples 1-15, wherein the instructions further cause the processor circuitry to generate a training dataset comprising: camera images captured without dynamic lighting; pairs of lighting patterns of fingers; and illuminated images showing saliency selection of enhanced geometric features.

[0086] Example 17. According to the components of Example 16, wherein the instructions further cause the processor circuitry to: select a lighting pattern with minimal correlation between visible edges and stable regions; and use the selected pattern to train a neural network to generate runtime lighting control.

[0087] Example 18. A robotic system comprising: a gripper including fingers that together define a grasping volume; an array of light sources integrated into each finger, wherein each light source is individually controllable; and a controller circuit system configured to: receive image data of an object; analyze visual features of the object based on the image data; generate an illumination pattern based on the analyzed visual features; and dynamically control the array of light sources to project the illumination pattern within the grasping volume during object manipulation to enhance the detection of visual features of the object.

[0088] Example 19. The robotic system of Example 18 further includes: a pressure sensor integrated into the finger, wherein the controller circuitry is further configured to: receive pressure data from the pressure sensor; and adjust the manipulation of the object based on the received pressure data.

[0089] Example 20. A robotic system according to any one or more of Examples 18-19, further comprising: a pressure sensor integrated into a finger, wherein a controller circuitry is configured to: receive pressure data from the pressure sensor; acquire visual feedback of the deformation of a compressible calibration object; and automatically calibrate the pressure sensor based on the pressure data and the deformation of the compressible calibration object.

[0090] Although the foregoing description has been combined with exemplary aspects, it should be understood that the term "exemplary" is intended merely as an example, and not as the best or optimal solution. Therefore, this disclosure is intended to cover alternatives, modifications, and equivalents that may be included within the scope of this disclosure.

[0091] Although specific aspects have been illustrated and described herein, it will be appreciated by those skilled in the art that various alternatives and / or equivalent implementations may be used in place of the specific aspects shown and described without departing from the scope of this application. This application is intended to cover any modifications or variations of the specific aspects discussed herein.

Claims

1. A component of a system, comprising: Processor circuitry; as well as A non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium including instructions, which, when executed by the processor circuitry, cause the processor circuitry to: Receive image data of objects captured by the camera; Analyze the visual features of the object based on the received image data; Lighting patterns are generated based on the analyzed visual features; as well as An array of light sources integrated into multiple fingers of a robotic gripper is controlled to project the illumination pattern within a grasping volume defined by the multiple fingers during object manipulation, thereby enhancing the detection of the visual features of the object, wherein each light source in the array is individually controllable.

2. The component according to claim 1, wherein: Each of the light sources includes an RGB(W) (red, green, blue, and white) light-emitting diode (LED) element, or an LED in the non-visible spectrum coupled to a multispectral camera, configured to project light of variable intensity or color, and The instructions further enable the processor circuitry to generate the lighting pattern by dynamically varying the intensity or color balance of each of the light sources.

3. The component of claim 1, wherein, The instructions further cause the processor circuitry to: The light source array is dynamically controlled to project the illumination pattern within the grasping volume to create a shifted illumination wavefront for edge detection. The shifted illumination wavefront enhances the detection and inference of surface properties at horizontal, vertical, or diagonal edges.

4. The component of claim 1, wherein, The instructions further cause the processor circuitry to: Pressure data is received from a pressure sensor integrated into the finger; and The manipulation of the object is adjusted based on the received pressure data.

5. The component of claim 1, wherein, The instructions further cause the processor circuitry to: Pressure data is received from a pressure sensor integrated into the finger; Obtain visual feedback on the deformation of the compressible calibration object; as well as The pressure sensor is automatically calibrated based on the pressure data and the deformation of the compressible calibration object.

6. The component of claim 1, wherein, The instructions further cause the processor circuitry to: A sequence of images is captured using a camera mounted on the robot gripper, while the light source array is controlled to project different lighting patterns within the grasping volume; A multidimensional model of the object is generated based on the sequence of captured images and the corresponding lighting patterns. as well as Subsequent lighting patterns are adjusted based on features detected in the multidimensional model to enhance visual detection of object geometry during manipulation.

7. The component of claim 1, wherein, The instructions further cause the processor circuitry to: Receive the model of the object; Extract visible features from the model; A set of lighting patterns is generated using dynamic kernel functions and saliency functions; The generated lighting pattern is applied to a simulated scene that includes the object; The saliency of the lighting pattern is evaluated based on the detection of the visible features; Choose the lighting pattern that achieves the minimum saliency threshold; as well as The selected lighting pattern is used to train the neural network to generate a lighting pattern decoder for runtime operation.

8. The component of claim 7, wherein, The set of lighting patterns is generated by the kernel function by varying parameters such as aperture, phase, orientation, smoothness, or spatial aspect ratio.

9. The component of claim 1, wherein, The instructions further cause the processor circuitry to: The training dataset is generated using an object model and simulated lighting patterns; The neural network is trained using the training dataset; and A trained neural network is used to generate lighting patterns during operation.

10. The component according to claim 1, wherein, The instructions further cause the processor circuitry to: Encode the image data into a latent spatial representation; and The latent space representation is decoded into illumination control parameters for the light source array.

11. The component according to claim 10, wherein, The instructions further cause the processor circuitry to: The latent space representation is shaped using a base decoder during training; and The lighting control parameters are generated using an extended decoder during runtime operation.

12. The component according to claim 10, wherein, The instructions further cause the processor circuitry to: Camera images captured without dynamic lighting will be encoded into the latent spatial representation; and The latent space representation is decoded into multiple pairs of lighting patterns for the fingers.

13. The component according to claim 1, wherein, The instructions further cause the processor circuitry to: Extract visible geometric elements from the model of the object; The visible structural region is defined based on the extracted geometric elements; and The lighting pattern is determined based on the visible structural region.

14. The component according to claim 13, wherein, The instructions further cause the processor circuitry to: Receive the parameter placement distribution that defines the possible positions and orientations of the object; The scene layout is generated based on the placement distribution of the parameters; and Training data is generated by simulating the lighting of the scene layout.

15. The component according to claim 14, wherein, The instructions further cause the processor circuitry to: Render an image of the simulated scene layout, with and without the defined lighting pattern; Generate a structural map that encodes the geometric features of the rendered image; as well as The saliency of the geometric features is evaluated to select an illumination pattern that enhances feature detection.

16. The component according to any one of claims 1-15, wherein, The instructions further cause the processor circuitry to generate a training dataset, the training dataset comprising: Camera images captured without dynamic lighting; The pair of lighting patterns for the fingers; and The illuminated image shows the saliency selection of enhanced geometric features.

17. The component according to claim 16, wherein, The instructions further cause the processor circuitry to: Choose a lighting pattern with minimal correlation between the visible edges and the stable region; and The selected pattern is used to train a neural network to generate runtime lighting control.

18. A robot system, comprising: A gripper, the gripper comprising fingers that together define a grasping volume; A light source array, said light source array being integrated into each of said fingers, wherein each of said light sources is individually controllable; A controller circuit system configured to: Receive image data of the object; Analyze the visual features of the object based on the image data; Generating lighting patterns based on the analyzed visual features; and During object manipulation, the light source array is dynamically controlled to project the illumination pattern within the grasping volume to enhance the detection of the object's visual features.

19. The robot system of claim 18, further comprising: A pressure sensor, integrated into the finger. The controller circuit system is further configured to: Receive pressure data from the pressure sensor; and The manipulation of the object is adjusted based on the received pressure data.

20. The robot system of claim 18, further comprising: A pressure sensor, integrated into the finger. The controller circuit system is configured to: Receive pressure data from the pressure sensor; Obtain visual feedback on the deformation of the compressible calibration object; and The pressure sensor is automatically calibrated based on the pressure data and the deformation of the compressible calibration object.

Images