Intelligent Adaptive Multi-Layer Flexible Material Handling System

Abstract

This application discloses an intelligent adaptive multi-layer flexible material handling system. The system includes multiple carrier lifting assemblies, multiple pallet conveying assemblies, multiple pusher mechanism assemblies, and multiple machine head assemblies. The carrier lifting assemblies support a basket assembly; the pallet conveying assemblies are positioned on the first side of their corresponding basket assembly; the pusher mechanism assemblies are positioned on the second side of their corresponding basket assembly and are used to push a target pallet from the basket assembly onto the corresponding pallet conveying assembly; the pallet conveying assembly is configured to adjust the track width according to the size of the target pallet to fit it; the machine head assemblies are used to absorb material from a corresponding flexible vibrating plate and place the material into the target pallet; after the material is placed in the target pallet, the pusher unit in the pallet conveying assembly pushes the target pallet into the basket assembly. This method satisfies the functional requirements for pallet handling of various sizes.

Description

Technical Field

[0001] This application relates to the field of material handling system technology, and in particular to an intelligent adaptive multi-layer flexible material handling system. Background Technology

[0002] Existing material handling systems automatically place trays into baskets, which are then manually removed once the baskets are full. The drawback is that these systems are limited in function and cannot accommodate trays of various sizes. Summary of the Invention

[0003] The intelligent adaptive multi-layer flexible material handling system provided in this application can meet the functional requirements of pallet handling for various sizes.

[0004] In a first aspect, this application provides an intelligent adaptive multi-layer flexible material handling system, comprising: multiple carrier lifting assemblies, each carrier lifting assembly for carrying a basket assembly, wherein the basket assembly is used to accommodate a pallet; multiple pallet conveying assemblies, each pallet conveying assembly disposed on a first side of its corresponding basket assembly; multiple pusher mechanism assemblies, each pusher mechanism assembly disposed on a second side of its corresponding basket assembly, for pushing a target pallet in the basket assembly onto a corresponding pallet conveying assembly; wherein the pallet conveying assembly is used to convey the target pallet to a material placement position, and the pallet conveying assembly is configured to adjust the track width in the pallet conveying assembly according to the size of the target pallet to adapt to the target pallet; and multiple machine head assemblies, each machine head assembly for adsorbing material from a corresponding flexible vibrating plate and moving it to the material placement position to place the material in the target pallet; wherein after the material is placed in the target pallet, the pusher unit in the pallet conveying assembly pushes the target pallet into the basket assembly.

[0005] The intelligent adaptive multi-layer flexible material handling system also includes: a first multispectral image acquisition component for acquiring multispectral images of the target pallet; wherein the multispectral images include visible light images, infrared images and ultraviolet images; and a processing component coupled to the first multispectral image acquisition component for fusing the multispectral images, identifying the target pallet from the fused multispectral images, and controlling the pallet conveying component to adjust the track width in the pallet conveying component according to the identification results.

[0006] The processing component is also used to extract image features from visible light, infrared and ultraviolet images respectively, fuse the image features to obtain multispectral fusion features, and use a low-rank metric model based on orthogonal learning to reduce the dimensionality and denoise the multispectral fusion features to obtain a low-rank feature representation, and identify the target tray based on the low-rank feature representation.

[0007] The processing component is also used to compensate for the system error corresponding to the first multispectral image acquisition component according to a preset period.

[0008] The processing component is also used for multi-axis coordinated control of the motors in the vehicle lifting component, pallet conveying component, pusher mechanism component and head assembly.

[0009] The material strength of the vehicle lifting component is greater than the threshold, and the material weight is less than the threshold. The vehicle lifting component is used to dynamically adjust the operating speed according to the weight of the basket component. Each vehicle lifting component is equipped with an independent control unit.

[0010] Each head assembly is equipped with a flexible material suction nozzle, which has a vacuum chamber structure. Each vacuum chamber is configured to adjust the vacuum level according to the shape and size of the material. Each head assembly is used to acquire multispectral images corresponding to the flexible vibrating disk. The multispectral images include visible light images, infrared images, and ultraviolet images. The multispectral images are fused, and the material posture, surface texture, material, predicted contact position, adsorption strength, and temperature in the flexible vibrating disk are identified from the fused multispectral images. The material is then adsorbed from the flexible vibrating disk according to the material posture, surface texture, material, predicted contact position, adsorption strength, and temperature.

[0011] Among them, multiple vehicle lifting components, multiple pallet conveying components, multiple pusher mechanism components, value-based reinforcement learning and policy-based reinforcement learning are used to optimize the movement path, and multiple machine head components use spatiotemporal graph neural networks to dynamically predict the optimal placement path of materials on the target pallet by integrating time information, spatial information and material information; among them, time information includes the arrival time of materials, and spatial information includes the loading status of pallets.

[0012] Several head components are also used to classify defects in the materials in the flexible vibratory feeder; the material information can be queried in the blockchain traceability system.

[0013] The intelligent adaptive multi-layer flexible material handling system also includes an adaptive wireless energy transmission module, which dynamically adjusts the wireless energy transmission power and frequency according to the equipment position and load of multiple carrier lifting components, multiple pallet conveying components, multiple pusher mechanism components, and multiple machine head components. The multiple carrier lifting components, multiple pallet conveying components, multiple pusher mechanism components, and multiple machine head components use time-sensitive network communication, and the multiple carrier lifting components, multiple pallet conveying components, multiple pusher mechanism components, and multiple machine head components are self-assembling mechanisms.

[0014] The beneficial effects of this application are as follows: Unlike existing technologies, the intelligent adaptive multi-layer flexible material handling system provided in this application includes: multiple carrier lifting assemblies, each carrier lifting assembly for carrying a basket assembly, wherein the basket assembly is used to hold a tray; multiple tray conveying assemblies, each tray conveying assembly disposed on a first side of its corresponding basket assembly; multiple pusher mechanism assemblies, each pusher mechanism assembly disposed on a second side of its corresponding basket assembly, for pushing a target tray from the basket assembly to the corresponding tray conveying assembly; wherein the tray conveying assembly is used to convey the target tray to the material placement position, and the tray conveying assembly is configured to adjust the track width in the tray conveying assembly according to the size of the target tray to adapt to the target tray; multiple machine head assemblies, each machine head assembly for adsorbing material from a corresponding flexible vibrating plate and moving it to the material placement position to place the material in the target tray; wherein after the material is placed in the target tray, the pusher unit in the tray conveying assembly pushes the target tray into the basket assembly. In the above manner, each carrier lifting component, each pallet conveying component, each pusher mechanism component, and each machine head component can constitute an independent material traying device. That is, the intelligent adaptive multi-layer flexible material traying system can set up multiple traying stations that do not affect each other, thereby improving the traying efficiency of the equipment. At the same time, the pallet conveying component is configured to adjust the track width in the pallet conveying component according to the size of the target pallet to adapt to the target pallet, thereby meeting the traying function requirements of various pallet sizes. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0016] Figure 1 This is a schematic diagram of the structure of an embodiment of the intelligent adaptive multi-layer flexible material handling system provided in this application;

[0017] Figure 2 A schematic diagram of another embodiment of the intelligent adaptive multi-layer flexible material handling system provided in this application;

[0018] Figure 3 This is a schematic diagram of another embodiment of the intelligent adaptive multi-layer flexible material handling system provided in this application;

[0019] Figure 4 This is a schematic diagram of the structure of an embodiment of the nose assembly provided in this application;

[0020] Figure 5 yes Figure 4 The corresponding front view;

[0021] Figure 6 This is a schematic diagram of a structure of an embodiment of the vehicle lifting assembly provided in this application;

[0022] Figure 7 yes Figure 6 The corresponding front view;

[0023] Figure 8 yes Figure 6 Corresponding side view;

[0024] Figure 9 This is a schematic diagram of the structure of an embodiment of the pallet conveying assembly provided in this application;

[0025] Figure 10 yes Figure 9 The corresponding left view;

[0026] Figure 11 yes Figure 9 The corresponding front view;

[0027] Figure 12 yes Figure 9 The corresponding rear view. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It is understood that the specific embodiments described herein are only for explaining this application and not for limiting it. Furthermore, it should be noted that, for ease of description, only the parts related to this application are shown in the accompanying drawings, not all structures. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0029] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0030] See Figure 1 , Figure 2 and Figure 3The intelligent adaptive multi-layer flexible material handling system 100 includes: multiple carrier lifting assemblies 10, multiple pallet conveying assemblies 20, multiple pusher mechanism assemblies 30, and multiple machine head assemblies 40. Furthermore, the intelligent adaptive multi-layer flexible material handling system 100 also includes: a housing 70 and an operating device 80.

[0031] Each vehicle lifting assembly 10 is used to carry a basket assembly 50, wherein the basket assembly 50 is used to hold a tray.

[0032] Each pallet conveying assembly 20 is disposed on the first side of its corresponding basket assembly 50.

[0033] Each pusher mechanism assembly 30 is disposed on the second side of its corresponding basket assembly 50 and is used to push the target pallet in the basket assembly 50 onto the corresponding pallet conveying assembly 20; wherein, the pallet conveying assembly 20 is used to convey the target pallet to the material placement position, and the pallet conveying assembly 20 is configured to adjust the track width in the pallet conveying assembly according to the size of the target pallet to adapt to the target pallet.

[0034] Each head assembly 40 is used to pick up material from the corresponding flexible vibrating plate 60 and move it to the material placement position to place the material in the target tray; wherein, after the material is placed in the target tray, the push plate unit in the tray conveying assembly 20 pushes the target tray into the basket assembly 50.

[0035] See Figure 4 and Figure 5 The head assembly 40 consists of a motor 45, a motor 43, a slide rail 47, a synchronous pulley 42, a synchronous belt 44, a vision component 41, and a suction nozzle 46. The top motor 43 controls the Z-axis lifting motion of the head assembly, while the bottom R-axis motor 45 drives two other motors to control the rotation of the suction nozzle 46. A flexible vibrating plate 60 vibrates the material, the vision component 41 (camera) detects the front of the material, and the head suction nozzle 46 picks up the material and places it into the tray.

[0036] See Figure 6 , Figure 7 and Figure 8 The carrier lifting assembly 10 consists of a motor 11, a lead screw 14, a guide rail, a synchronous pulley 12, and a synchronous belt 13. When the basket assembly 50 is manually placed to the Z-axis position of the carrier lifting assembly, the carrier lifting assembly 10 pushes the pallet into the pallet conveying assembly 20 via the pusher mechanism assembly 30, allowing it to flow into the next position.

[0037] See Figure 9 , Figure 10 , Figure 11 and Figure 12The pallet conveying assembly 20 consists of stepper motors 21, 27, and 26, a synchronous pulley, a synchronous belt 25, a lead screw 24, a guide rail, a pusher unit 22, and a Z-axis lifting mechanism 23. The pallet conveying assembly 20 is adjustable according to the pallet width. When a pallet flows into the conveying track formed by the synchronous pulley and synchronous belt 25, the Z-axis lifting mechanism 23 lifts the pallet, and the head assembly 40 picks up the material for pallet placement. Once the pallet is full, the Z-axis lifting mechanism 23 lowers the pallet back into the conveying track. The pusher unit 22 then pushes the pallet into the basket assembly 50.

[0038] For example, the production process is as follows:

[0039] 1. Manually place the basket assembly 50 filled with empty pallets into the corresponding vehicle lifting assembly 10.

[0040] 2. The vehicle lifting assembly 10 docks with the push plate mechanism assembly 30 to push the pallet into the pallet conveying assembly 20.

[0041] 3. The pallet in the pallet conveying assembly 20 is lifted by the Z-axis lifting 23 at the end of the belt track.

[0042] 4. The flexible vibratory feeder detects the orientation of the material using a camera, and the head assembly 40 sequentially picks up eight materials and places them into the tray.

[0043] 5. After the materials in the pallet are filled in sequence, the pallet conveying component 20 will transport the pallet to the basket component 50 on the carrier lifting component 10.

[0044] 6. The push plate unit 22 of the pallet conveying assembly 20 pushes the pallets that have been filled with materials into the basket assembly 50 in sequence. After the basket assembly 50 is full, it is removed manually.

[0045] In some embodiments, the intelligent adaptive multi-layer flexible material handling system 100 described above further includes a first multispectral image acquisition component and a processing component. The first multispectral image acquisition component is used to acquire multispectral images of the target tray; wherein, the multispectral images include visible light images, infrared images, and ultraviolet images. The processing component is coupled to the first multispectral image acquisition component and is used to fuse the multispectral images, identify the target tray from the fused multispectral images, and control the tray conveying component to adjust the track width in the tray conveying component according to the identification result.

[0046] The processing component is also used to extract image features from visible light, infrared and ultraviolet images respectively, fuse the image features to obtain multispectral fusion features, and use a low-rank metric model based on orthogonal learning to reduce the dimensionality and denoise the multispectral fusion features to obtain a low-rank feature representation, and identify the target tray based on the low-rank feature representation.

[0047] The processing component is also used to compensate for the system error corresponding to the first multispectral image acquisition component according to a preset period.

[0048] The processing component is also used for multi-axis coordinated control of the motors in the vehicle lifting component, pallet conveying component, pusher mechanism component and head assembly.

[0049] That is, the intelligent adaptive multi-layer flexible material handling system 100 of this application can achieve adaptive tray handling. Specifically, it is achieved by using a high-precision vision recognition system and a servo motor-driven adaptive adjustment mechanism.

[0050] For example, by employing multispectral visual fusion technology, this application adds infrared and ultraviolet spectra in addition to visible light, enhancing the ability to identify trays of different materials and colors, reducing ambient light interference, and improving the robustness of identification, especially making the identification of reflective and transparent trays more accurate.

[0051] Detailed solutions for multispectral visual fusion technology can employ sensor fusion, feature fusion, adaptive multi-scale feature fusion, low-rank metric based on orthogonal learning, hybrid loss function, and data augmentation techniques.

[0052] Sensor fusion is mainly reflected in the simultaneous acquisition of visible light, infrared and ultraviolet images by a multispectral camera (the first multispectral image acquisition component), and the registration of these images to form a multispectral dataset.

[0053] Feature fusion is primarily involved in the feature extraction stage, where features from visible light, infrared, and ultraviolet images are extracted separately and then fused. Fusion methods can include channel stitching, weighted fusion, or attention mechanisms.

[0054] Channel splicing can be manifested in directly splicing feature maps of different spectra together in the channel dimension as a multispectral fusion feature.

[0055] Weighted fusion can be achieved by weighting the feature maps according to the contribution of different spectra. For example, infrared spectra may have a higher weight at night or under low light conditions.

[0056] Among them, the attention mechanism can be reflected in dynamically adjusting the weights of different spectral features to highlight features that are beneficial to recognition.

[0057] The adaptive multi-scale feature fusion mainly focuses on the fused multispectral features. The adaptive multi-scale feature fusion module extracts shallow features at different scales. This module can solve the problem of recognizing pallets of different sizes and types, especially pallets with complex shapes or structures.

[0058] The low-rank metric based on orthogonal learning mainly involves using an orthogonal learning-based low-rank metric model to reduce the dimensionality and denoise the fused multispectral features, resulting in a low-rank feature representation. This step can improve the robustness of the model and reduce the impact of noise on the recognition results.

[0059] The hybrid loss function is mainly reflected in the use of the adaptive hybrid loss module, which optimizes classification results and addresses the problem of uneven distribution of pallet categories. Furthermore, it can balance the learning capabilities of different categories and improve the classification accuracy of smaller categories through adaptive weights and multi-classification heads.

[0060] Data augmentation is mainly reflected in the fact that infrared and ultraviolet image data are relatively scarce, so data augmentation techniques such as rotation, scaling, translation, and noise addition can be used to expand the training dataset.

[0061] Data augmentation methods should be tailored to the characteristics of different spectra.

[0062] The algorithm flow is as follows:

[0063] Visible light, infrared, and ultraviolet images are acquired. Multispectral image registration is then performed to create a multispectral dataset. An adaptive multi-scale feature fusion module is used to extract multispectral fusion features. A low-rank metric model based on orthogonal learning is then used to reduce the dimensionality and denoise the features. Finally, a hybrid loss function is used to optimize the classification model. The output is the tray recognition result.

[0064] The technical advantages of the above method are: improved recognition robustness, more accurate tray recognition, and stronger anti-interference ability.

[0065] Improving recognition robustness is mainly reflected in the fact that multispectral visual fusion can effectively reduce interference from ambient light, thereby enhancing the robustness of the recognition system under different lighting conditions. The addition of infrared and ultraviolet spectra can enhance the system's ability to recognize trays of different materials and colors.

[0066] More accurate pallet identification is primarily due to the fact that infrared spectroscopy can distinguish pallets with different thermal properties, while ultraviolet spectroscopy can identify pallets with special surface coatings, thus improving the accuracy of identifying complex pallets. In particular, infrared and ultraviolet spectroscopy can help solve the identification problems of reflective and transparent pallets.

[0067] The enhanced anti-interference capability is mainly reflected in the fact that data augmentation, low-rank metrics of orthogonal learning, and hybrid loss functions can further improve the system's resistance to noise and interference.

[0068] Furthermore, the processing components can utilize self-calibration and compensation algorithms. For example, a deep learning-based self-calibration algorithm can be developed to periodically calibrate the vision system and automatically compensate for system errors caused by factors such as temperature changes and vibrations, ensuring long-term stability and high accuracy.

[0069] Furthermore, the processing component can upgrade the linear adjustment driven by the servo motor to multi-axis collaborative adjustment, enabling fine adjustment of the pallet in the horizontal and vertical directions, enhancing its adaptability to complex pallet shapes, and making the adjustment process smoother and more stable.

[0070] In some embodiments, the material strength of the vehicle lifting assembly 10 is greater than a threshold and the material weight is less than a threshold, wherein the vehicle lifting assembly 10 is used to dynamically adjust the operating speed according to the weight of the basket assembly; and each vehicle lifting assembly 10 is equipped with an independent control unit.

[0071] In some embodiments, the intelligent adaptive multi-layer flexible material handling system 100 of this application can achieve efficient multi-layer placement. Specifically, it is reflected in the use of a multi-Z-axis lifting mechanism (carrier lifting assembly 10), a multi-head assembly 40, and a synchronous control algorithm.

[0072] For example, a lightweight modular Z-axis mechanism is adopted. This is mainly reflected in the use of high-strength, lightweight materials and the optimization of the Z-axis structural design to reduce inertia, improve motion speed and response, and reduce energy consumption.

[0073] The adoption of a distributed multi-core control system mainly involves upgrading the synchronous control algorithm to a distributed multi-core control system. Each Z-axis mechanism has an independent control unit that can process data in parallel, reducing the computational burden on the main control unit, achieving more refined collaborative control, and improving the real-time performance and reliability of the overall operation.

[0074] The adoption of a dynamic load balancing algorithm mainly involves developing a dynamic load balancing algorithm that dynamically adjusts the running speed and acceleration of the Z-axis based on the weight and quantity of materials on each pallet, thereby reducing equipment vibration, improving overall stability, and reducing equipment wear.

[0075] In some embodiments, each head assembly 40 is provided with a flexible material suction nozzle, which has a vacuum chamber structure. Each vacuum chamber is configured to adjust the vacuum level according to the shape and size of the material. Each head assembly is used to acquire multispectral images corresponding to the flexible vibrating disk 60. The multispectral images include visible light images, infrared images, and ultraviolet images. The multispectral images are fused, and the material posture, surface texture, material, predicted contact position, adsorption force, and temperature in the flexible vibrating disk 60 are identified from the fused multispectral images. The material is then adsorbed from the flexible vibrating disk according to the material posture, surface texture, material, predicted contact position, adsorption force, and temperature.

[0076] In some embodiments, the intelligent adaptive multi-layer flexible material handling system 100 of this application can achieve flexible material gripping. Specifically, this is achieved by using a flexible material suction nozzle, pressure feedback control, deep learning-based material posture recognition, and force feedback sensors.

[0077] For example, the use of an adaptive multi-cavity vacuum nozzle mainly involves designing the flexible material nozzle as multiple independently controlled vacuum cavities. Each vacuum cavity can independently adjust the vacuum level according to the shape and size of the material, improving the flexibility and stability of gripping, especially for irregular and fragile materials.

[0078] The use of multimodal tactile feedback mainly involves adding various tactile sensors such as temperature, vibration, and capacitance in addition to force feedback, to build a more comprehensive tactile feedback system. This enables the perception of material properties, temperature, surface texture, and other characteristics of materials, further improving the success rate of grasping. It also enables the identification and safe grasping of special materials.

[0079] Detailed solutions for multimodal haptic feedback can employ technologies such as multispectral vision, multimodal haptic sensing, deep learning models, data processing and fusion, multimodal data fusion, and system integration.

[0080] Multispectral vision mainly involves using multispectral cameras to capture visible light, infrared, and ultraviolet images to identify the material, color, and shape of materials.

[0081] Multimodal tactile sensing mainly includes force sensors, temperature sensors, vibration sensors, and capacitance sensors.

[0082] Among them, the force sensor is mainly characterized by the use of a microstructure-enhanced flexible force sensor, which is made of transparent PDMS and black graphite / PDMS composite material to improve sensitivity and spatial resolution.

[0083] Temperature sensors are mainly integrated flexible thermistors or thermocouples, used to measure the surface temperature of objects.

[0084] Vibration sensors are mainly integrated piezoelectric materials or microelectromechanical systems (MEMS) accelerometers, used to sense the surface texture and vibration characteristics of objects.

[0085] Among them, capacitive sensors are mainly integrated capacitive sensors used to measure the capacitance characteristics of objects, thereby distinguishing different materials.

[0086] In some embodiments, force, temperature, vibration, and capacitance sensors can be integrated onto the same flexible substrate to form a compact tactile sensing array.

[0087] Signal processing mainly involves developing corresponding circuits and algorithms to process, filter, and calibrate multimodal tactile signals.

[0088] Deep learning models are mainly reflected in technologies such as multimodal feature extraction, feature fusion, multi-task learning, model training, data processing and fusion, and system integration.

[0089] Among them, multimodal feature extraction is mainly reflected in the use of independent CNN models to extract features from multispectral images and data from various tactile sensors.

[0090] Feature fusion mainly involves using attention mechanisms or gating networks to adaptively fuse features from different modalities, thereby obtaining a more comprehensive feature representation.

[0091] Multi-task learning is mainly reflected in the use of multi-task learning methods to simultaneously predict an object's pose, position, forces, temperature, surface texture, and material type.

[0092] Model training mainly involves using a large amount of experimental data to train deep learning models so that they can accurately predict the properties and states of objects.

[0093] Data processing and fusion mainly involve data registration, multimodal data fusion, and data enhancement.

[0094] Data registration mainly involves registering data from multispectral images and multimodal tactile sensors to achieve temporal and spatial synchronization.

[0095] Multimodal data fusion is mainly reflected in early fusion, late fusion, and dynamic fusion.

[0096] Early fusion mainly involves fusing multimodal data before feature extraction, such as channel splicing or weighted summation.

[0097] In particular, the later-stage fusion mainly involves extracting multimodal features separately and then using attention mechanisms, gating networks, or decision fusion methods for fusion.

[0098] Dynamic fusion is mainly reflected in dynamically adjusting the weights of different modal information according to different tasks and scenarios.

[0099] Data augmentation mainly involves using data augmentation methods to increase the diversity of training data and improve the model's generalization ability.

[0100] System integration mainly manifests in hardware integration and software integration.

[0101] Hardware integration is mainly reflected in the integration of a multispectral camera and a multimodal tactile sensor array into a compact sensing unit.

[0102] Software integration mainly involves developing a unified software framework to achieve the acquisition, processing, analysis, fusion, and visualization of multimodal data.

[0103] The technical benefits of the above methods mainly include enhanced object recognition capabilities, safer grasping operations, higher grasping success rates, wider application scenarios, and the ability to identify and securely grasp special materials.

[0104] Enhancing object recognition capabilities is mainly reflected in improving the accuracy of recognizing object materials, temperatures, and surface textures by fusing multimodal information. For example, it can distinguish between metals and plastics, high-temperature objects and normal-temperature objects, as well as surfaces with different textures.

[0105] Safer gripping operations are mainly achieved by using multimodal tactile feedback to safely grip fragile objects, high-temperature objects, or objects with special surface textures.

[0106] The higher success rate of data capture is mainly reflected in the ability to improve the success rate of data capture in complex scenarios through more comprehensive perception and control capabilities.

[0107] The wider range of applications is mainly reflected in the fact that the system can be applied to a wider range of fields, such as intelligent manufacturing, medical robots, home service robots, and wearable electronic devices.

[0108] The identification and safe grasping of special materials are mainly reflected in the combination of multimodal tactile feedback, which can realize the identification of special materials, such as high-temperature objects, fragile objects, or objects with special surface textures, to ensure safe grasping.

[0109] The specific algorithm process includes: data acquisition, preprocessing, feature extraction, data fusion, multi-task prediction, and output.

[0110] Data acquisition primarily involves using multispectral cameras to capture visible light, infrared, and ultraviolet images of the materials. A multimodal tactile sensor array is used to collect data on contact force, temperature, vibration, and capacitance.

[0111] Preprocessing mainly involves preprocessing multispectral images and multimodal tactile data, such as denoising, enhancement, filtering, and calibration.

[0112] Feature extraction is mainly reflected in the use of independent CNN models to extract features from multispectral images and data from various tactile sensors.

[0113] Data fusion is mainly reflected in the use of attention mechanisms or gating networks to adaptively fuse features from different modalities.

[0114] Multi-task prediction is mainly reflected in the use of deep learning models to identify material posture, while simultaneously predicting contact location, force magnitude, temperature, surface texture, and material type.

[0115] The output mainly includes the posture information, force feedback information, temperature information, surface texture information, and material category of the output material.

[0116] In some embodiments, pose recognition based on 3D point clouds can be employed. Specifically, pose recognition based on 2D images can be upgraded to pose recognition based on 3D point clouds, which can more accurately acquire the three-dimensional information of the material, reduce recognition errors caused by factors such as changes in lighting and occlusion, and make the grasping posture more precise.

[0117] 3D point cloud pose recognition mainly includes 3D Gaussian sputtering initialization, adaptive density control, pose estimation, etc.

[0118] The 3D Gaussian sputtering initialization mainly involves adopting a 3D Gaussian sputtering framework based on point map priors and K-nearest neighbor algorithm for adaptive density control. It uses a grouping strategy based on view similarity and a powerful point map prior to initialize the 3D Gaussian ellipsoid to obtain more accurate point clouds and camera poses.

[0119] Among them, adaptive density control is mainly reflected in the use of the K-nearest neighbor algorithm to adaptively segment Gaussian ellipsoids according to the shape differences of adjacent Gaussian ellipsoids, thereby realizing the dynamic adjustment and optimization of point clouds.

[0120] Pose estimation primarily involves using deep learning models to extract features from 3D point clouds and predict the 3D pose of objects. For example, layered approaches can be used to learn local features of point clouds, or the self-attention mechanism of Transformers can be applied to 3D point cloud data and other network structures.

[0121] Multimodal tactile sensing mainly includes force sensors, temperature sensors, vibration sensors, and capacitance sensors.

[0122] Among them, the force sensor is mainly characterized by the use of a microstructure-enhanced flexible force sensor, which is made of transparent PDMS and black graphite / PDMS composite material to improve sensitivity and spatial resolution.

[0123] Temperature sensors are mainly integrated flexible thermistors or thermocouples, used to measure the surface temperature of objects.

[0124] Vibration sensors are mainly integrated piezoelectric materials or microelectromechanical systems (MEMS) accelerometers, used to sense the surface texture and vibration characteristics of objects.

[0125] Among them, capacitive sensors are mainly integrated capacitive sensors used to measure the capacitance characteristics of objects, thereby distinguishing different materials.

[0126] In some embodiments, force, temperature, vibration, and capacitance sensors can be integrated onto the same flexible substrate to form a compact tactile sensing array.

[0127] Signal processing mainly involves developing corresponding circuits and algorithms to process, filter, and calibrate multimodal tactile signals.

[0128] Deep learning models are mainly reflected in technologies such as multimodal feature extraction, feature fusion, multi-task learning, model training, data processing and fusion, and system integration.

[0129] Among them, multimodal feature extraction is mainly reflected in the use of independent CNN models to extract features from multispectral images, 3D point clouds and data from various tactile sensors.

[0130] Feature fusion mainly involves using attention mechanisms or gating networks to adaptively fuse features from different modalities, thereby obtaining a more comprehensive feature representation.

[0131] Multi-task learning is mainly reflected in the use of multi-task learning methods to simultaneously predict an object's pose, position, forces, temperature, surface texture, and material type.

[0132] Model training mainly involves using a large amount of experimental data to train deep learning models so that they can accurately predict the properties and states of objects.

[0133] Data processing and fusion mainly involve data registration, multimodal data fusion, and data enhancement.

[0134] Data registration mainly involves registering data from multispectral images, 3D point clouds, and multimodal tactile sensors to achieve temporal and spatial synchronization.

[0135] Multimodal data fusion is mainly reflected in early fusion, late fusion, and dynamic fusion.

[0136] Early fusion mainly involves fusing multimodal data before feature extraction, such as channel splicing or weighted summation.

[0137] In particular, the later-stage fusion mainly involves extracting multimodal features separately and then using attention mechanisms, gating networks, or decision fusion methods for fusion.

[0138] Dynamic fusion is mainly reflected in dynamically adjusting the weights of different modal information according to different tasks and scenarios.

[0139] Data augmentation mainly involves using data augmentation methods to increase the diversity of training data and improve the model's generalization ability.

[0140] System integration mainly manifests in hardware integration and software integration.

[0141] Hardware integration is mainly reflected in the integration of a multispectral camera, a 3D point cloud sensor, and a multimodal tactile sensor array into a compact sensing unit.

[0142] Software integration mainly involves developing a unified software framework to achieve the acquisition, processing, analysis, fusion, and visualization of multimodal data.

[0143] The technical benefits of the above methods mainly include more accurate posture recognition, enhanced object recognition capabilities, safer grasping operations, higher grasping success rates, wider application scenarios, and the ability to identify and securely grasp special materials.

[0144] More accurate pose recognition is mainly reflected in the fact that pose recognition based on 3D point clouds can more accurately obtain the three-dimensional information of objects, reduce recognition errors caused by factors such as changes in lighting and occlusion, and make the grasping pose more accurate.

[0145] Enhancing object recognition capabilities is mainly reflected in improving the accuracy of recognizing object materials, temperatures, and surface textures by fusing multimodal information. For example, it can distinguish between metals and plastics, high-temperature objects and normal-temperature objects, as well as surfaces with different textures.

[0146] Safer gripping operations are mainly achieved by using multimodal tactile feedback to safely grip fragile objects, high-temperature objects, or objects with special surface textures.

[0147] The higher success rate of data capture is mainly reflected in the ability to improve the success rate of data capture in complex scenarios through more comprehensive perception and control capabilities.

[0148] The wider range of applications is mainly reflected in the fact that the system can be applied to a wider range of fields, such as intelligent manufacturing, medical robots, home service robots, and wearable electronic devices.

[0149] The identification and safe grasping of special materials are mainly reflected in the combination of multimodal tactile feedback, which can realize the identification of special materials, such as high-temperature objects, fragile objects, or objects with special surface textures, to ensure safe grasping.

[0150] The specific algorithm process includes: data acquisition, preprocessing, feature extraction, data fusion, multi-task prediction, and output.

[0151] Data acquisition primarily involves using multispectral cameras to capture visible light, infrared, and ultraviolet images of the material. 3D point cloud sensors are used to acquire three-dimensional point cloud data of the material. Multimodal tactile sensor arrays are used to acquire data on contact force, temperature, vibration, and capacitance.

[0152] Preprocessing mainly involves preprocessing multispectral images, 3D point clouds, and multimodal haptic data, such as denoising, enhancement, filtering, and calibration.

[0153] Feature extraction is mainly reflected in the use of independent CNN models to extract features from multispectral images, 3D point clouds and data from various tactile sensors.

[0154] Data fusion is mainly reflected in the use of attention mechanisms or gating networks to adaptively fuse features from different modalities.

[0155] Multi-task prediction is mainly reflected in the use of deep learning models to identify material posture, while simultaneously predicting contact location, force magnitude, temperature, surface texture, and material type.

[0156] The output mainly includes the posture information, force feedback information, temperature information, surface texture information, and material category of the output material.

[0157] In some embodiments, multiple vehicle lifting assemblies 10, multiple pallet conveying assemblies 20, multiple pusher mechanism assemblies 30 employ value-based reinforcement learning and policy-based reinforcement learning to optimize the movement path, and multiple machine head assemblies 40 employ spatiotemporal graph neural networks to dynamically predict the optimal placement path of materials on the target pallet by integrating time information, spatial information, and material information; wherein, the time information includes the arrival time of the materials, and the spatial information includes the loading status of the pallet.

[0158] In some embodiments, the intelligent adaptive multi-layer flexible material handling system 100 of this application can achieve intelligent path optimization. Specifically, this is reflected in the use of reinforcement learning algorithms and genetic algorithms.

[0159] For example, hybrid reinforcement learning is adopted, which mainly combines traditional value-based reinforcement learning with policy-based reinforcement learning to achieve faster and more stable path optimization, enabling it to better adapt to dynamic environments.

[0160] For example, the use of spatiotemporal graph neural networks mainly involves introducing spatiotemporal graph neural networks to comprehensively consider time, space and material information, dynamically predict the optimal placement path, reduce mechanical movement and waiting time, improve the overall efficiency of the system, and cope with complex and ever-changing placement requirements.

[0161] For example, predictive dynamic programming can be used to pre-plan the optimal placement path based on the arrival time of materials and the loading status of pallets, thereby achieving early optimization, avoiding congestion, and maximizing efficiency.

[0162] Among them, multiple head components 40 are also used to classify defects in the materials in the flexible vibratory feeder 60; the material information can be queried in the blockchain traceability system.

[0163] In some embodiments, the intelligent adaptive multi-layer flexible material handling system 100 of this application can achieve integrated quality control. Specifically, this is reflected in the use of a high-precision three-dimensional vision inspection system, automatic sorting RFID technology, and QR code technology.

[0164] For example, AI-driven defect classification mainly involves deep learning-based defect classification algorithms, which can identify and classify different types of material defects, such as cracks, scratches, deformation, and surface contamination, providing more comprehensive information for quality control.

[0165] For example, the adoption of a blockchain traceability system mainly involves using blockchain technology to build a secure and reliable production traceability system, ensuring the immutability of data, facilitating users to query material production information, improving the transparency and reliability of the traceability system, and ensuring product quality.

[0166] For example, edge computing primarily involves offloading some computational tasks to the device itself, reducing data transmission latency, improving real-time performance, and reducing reliance on cloud platforms. Edge computing can also rapidly process and provide feedback on detection results, improving detection and sorting efficiency.

[0167] In some embodiments, the intelligent adaptive multi-layer flexible material handling system 100 further includes an adaptive wireless power transmission module, used to dynamically adjust the wireless power transmission power and frequency according to the equipment position and load of multiple carrier lifting components 10, multiple pallet conveying components 20, multiple pusher mechanism components 30, and multiple machine head components 40; wherein the multiple carrier lifting components 10, multiple pallet conveying components 20, multiple pusher mechanism components 30, and multiple machine head components 40 use time-sensitive network communication, and the multiple carrier lifting components 10, multiple pallet conveying components 20, multiple pusher mechanism components 30, and multiple machine head components 40 are self-assembling mechanisms.

[0168] In some embodiments, the intelligent adaptive multi-layer flexible material handling system 100 of this application can be wirelessly and flexibly deployed. Specifically, this is reflected in the adoption of wireless power transmission, 5G wireless communication, modular structural design, and a quick-connect mechanism.

[0169] For example, the use of adaptive wireless power transfer mainly involves using adaptive wireless power transfer technology to dynamically adjust the wireless power transfer power and frequency according to the device location and load conditions, ensuring efficient and reliable wireless power supply, and realizing collaborative energy management among multiple devices.

[0170] For example, the adoption of Time-Sensitive Networking (TSN) mainly involves upgrading 5G wireless communication to a time-sensitive network to ensure the real-time performance and reliability of data transmission, especially in scenarios requiring precise synchronization control.

[0171] For example, adopting a modular self-assembly mechanism mainly involves upgrading the modular design to a self-assembly mechanism. The equipment can be automatically assembled and disassembled according to needs, simplifying the deployment process, shortening the deployment time, and improving the ease of use of the equipment.

[0172] The intelligent adaptive multi-layer flexible material handling system of this application can have the following technical effects:

[0173] 1. Significantly improves equipment compatibility: Automatically adapts to pallets of different sizes, expanding the range of equipment applications and improving equipment utilization.

[0174] 2. Significantly improves placement efficiency: Multi-layer synchronous placement and intelligent path optimization greatly shorten the production cycle.

[0175] 3. Improved material adaptability: The flexible suction nozzle and force feedback gripping significantly improve the gripping ability and placement accuracy of irregular and fragile materials.

[0176] 4. Achieve intelligent control: AI path optimization, automatic sorting, and material traceability reduce manual intervention and improve the level of intelligent production.

[0177] 5. Enhanced system flexibility: Wireless deployment and modular design enable rapid deployment and flexible adjustment of equipment to adapt to different production needs.

[0178] 6. Improve production quality and management level: Precise material detection, sorting, and pallet traceability functions ensure product quality and improve production management level.

[0179] 7. Achieve energy saving and remote monitoring: By adopting energy recovery, wireless power supply, and remote monitoring and maintenance, the equipment achieves energy saving and high reliability operation.

[0180] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0181] If the integrated units in the other embodiments described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0182] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An intelligent adaptive multi-layer flexible material handling system, characterized in that, The intelligent adaptive multi-layer flexible material handling system includes: Multiple vehicle lifting assemblies, each of which is used to carry a basket assembly, wherein the basket assembly is used to hold a tray; Multiple pallet conveying assemblies, each of the pallet conveying assemblies being disposed on the first side of its corresponding basket assembly; Multiple pusher mechanism components, each of which is disposed on the second side of its corresponding basket assembly, are used to push the target pallet in the basket assembly onto the corresponding pallet conveying assembly; wherein the pallet conveying assembly is used to convey the target pallet to the material placement position, and the pallet conveying assembly is configured to adjust the track width in the pallet conveying assembly according to the size of the target pallet to adapt to the target pallet; Multiple head assemblies, each of which is used to pick up material from a corresponding flexible vibrating plate and move it to the material placement position to place the material in the target tray; After the materials are placed in the target pallet, the pusher unit in the pallet conveying assembly pushes the target pallet into the basket assembly; The intelligent adaptive multi-layer flexible material handling system also includes: A first multispectral image acquisition component is used to acquire multispectral images of the target tray; wherein the multispectral images include visible light images, infrared images, and ultraviolet images; A processing component, coupled to the first multispectral image acquisition component, is used to fuse the multispectral images, identify the target pallet from the fused multispectral images, and control the pallet conveying component to adjust the track width in the pallet conveying component according to the identification result; The processing component is also used to extract image features from the visible light image, the infrared image, and the ultraviolet image respectively, fuse the image features to obtain multispectral fusion features, and use a low-rank metric model based on orthogonal learning to reduce the dimensionality and denoise the multispectral fusion features to obtain a low-rank feature representation, and identify the target tray based on the low-rank feature representation; Each of the aforementioned head assembly is provided with a flexible material suction nozzle, which has a vacuum chamber structure. Each vacuum chamber is configured to adjust the vacuum level according to the shape and size of the material. Each of the aforementioned head assembly is used to acquire multispectral images corresponding to the flexible vibrating disk. The multispectral images include visible light images, infrared images, and ultraviolet images. The multispectral images are fused, and the material posture, surface texture, material, predicted contact position, adsorption force, and temperature in the flexible vibrating disk are identified from the fused multispectral images. The material is then adsorbed from the flexible vibrating disk according to the material posture, surface texture, material, predicted contact position, adsorption force, and temperature.

2. The intelligent adaptive multi-layer flexible material handling system according to claim 1, characterized in that, The processing component is also used to compensate for the system error corresponding to the first multispectral image acquisition component according to a preset period.

3. The intelligent adaptive multi-layer flexible material handling system according to any one of claims 1 or 2, characterized in that, The processing component is also used for multi-axis coordinated control of the motors in the vehicle lifting component, the pallet conveying component, the pusher mechanism component, and the machine head component.

4. The intelligent adaptive multi-layer flexible material handling system according to claim 1, characterized in that, The material strength of the vehicle lifting assembly is greater than a threshold, and the material weight is less than a threshold. The vehicle lifting assembly is used to dynamically adjust the operating speed according to the weight of the basket assembly. Each vehicle lifting assembly is equipped with an independent control unit.

5. The intelligent adaptive multi-layer flexible material handling system according to claim 1, characterized in that, Multiple vehicle lifting components, multiple pallet conveying components, multiple pusher mechanism components, value-based reinforcement learning and policy-based reinforcement learning are used to optimize the movement path, and multiple machine head components use spatiotemporal graph neural networks to dynamically predict the optimal placement path of materials on the target pallet by integrating time information, spatial information and material information; among which, time information includes the arrival time of materials, and spatial information includes the loading status of pallets.

6. The intelligent adaptive multi-layer flexible material handling system according to claim 1, characterized in that, Multiple head components are also used for defect classification of materials in the flexible vibratory feeder; the material information can be queried in the blockchain traceability system.

7. The intelligent adaptive multi-layer flexible material handling system according to claim 1, characterized in that, The intelligent adaptive multi-layer flexible material handling system also includes an adaptive wireless power transmission module, which dynamically adjusts the wireless power transmission power and frequency according to the equipment position and load of multiple carrier lifting components, multiple pallet conveying components, multiple pusher mechanism components, and multiple machine head components; wherein, multiple carrier lifting components, multiple pallet conveying components, multiple pusher mechanism components, and multiple machine head components use time-sensitive network communication, and multiple carrier lifting components, multiple pallet conveying components, multiple pusher mechanism components, and multiple machine head components are self-assembling mechanisms.

Images