A waste cloth cross-modal virtual-real mapping method based on conveying displacement

Abstract

The application discloses a kind of waste cloth cross-modal virtual-real mapping methods based on conveying displacement, belong to intelligent manufacturing and digital twin field.It includes: S1, high-spectral detection station and RGB detection station are sequentially laid out;S2, identify cloth target i in upstream high-spectral detection station;S3, identify cloth target j in downstream RGB detection station;S4, calculate the predicted coordinate of i at RGB detection time;S5, calculate the Euclidean distance of i and j and the size difference of cloth width;S6, construct the association cost function of i and j;S7, the material of i and the color of j are bound to the same entity cloth;S8, generate a unique corresponding virtual cloth instance for each group of matched i and j;S9, dynamically update the virtual cloth instance coordinates according to the real-time pulse increment of encoder.The method of the application realizes automatic binding of material and color, low-latency virtual-real synchronization, and adapts to high-speed cloth sorting production line.

Description

Technical Field

[0001] This invention relates to the fields of intelligent manufacturing and digital twin technology, and in particular to a method for cross-modal virtual-real mapping of waste fabric based on conveying displacement. Background Technology

[0002] The core of waste textile recycling is automated sorting, where fabric material and color jointly determine the recycling and grading path. Currently, industrial sorting lines typically use two sets of vision equipment to collect information: a hyperspectral camera accurately identifies fabric materials such as cotton, polyester, and blends, while an RGB camera quickly captures fabric colors such as white, red, and blue.

[0003] Due to the physical layout limitations of the conveyor belt, the two types of cameras are set up as independent workstations along the conveying direction. The fabric passes through the two detection positions one after the other, resulting in time and spatial displacement differences in the acquisition time. The existing processing method only relies on a simple temporal sequence for coarse matching, without displacement compensation under a unified coordinate system and multi-dimensional matching verification. Under high-speed conveying conditions, mismatch of adjacent fabrics is very likely to occur, and material and color information cannot be automatically bound to the same piece of fabric.

[0004] Meanwhile, existing workshop digital twin systems only visualize the status of equipment such as conveyor belts and sorting machines, without creating independent virtual objects for individual pieces of fabric moving discretely on the conveyor belt. They lack a closed loop for real-time synchronous mapping of multi-source visual perception data to virtual materials, making it impossible to achieve full-process digital tracking of fabric and difficult to provide the sorting system with complete and real-time fabric attribute and location data. Summary of the Invention

[0005] The purpose of this invention is to provide a method for cross-modal virtual-real mapping of waste fabric based on conveying displacement, thereby solving the above-mentioned technical problems.

[0006] To achieve the above objectives, this invention provides a method for cross-modal virtual-real mapping of waste fabric based on conveying displacement, comprising the following steps: S1. Set up hyperspectral detection station and RGB detection station in sequence along the conveyor belt conveying direction, and calibrate the fixed distance between the two stations along the conveying direction. S2. Identify the fabric at the upstream hyperspectral detection station, collect the fabric material and the initial physical coordinates of the conveyor belt, synchronously record the encoder pulses, and define the fabric as fabric target i. S3. Identify the fabric at the downstream RGB detection station, collect the fabric color and the physical coordinates of the conveyor belt, synchronously record the encoder pulse, and define the fabric as the fabric target j. S4. Based on the encoder pulse difference and unit pulse transport displacement corresponding to fabric target i and fabric target j, perform transport displacement compensation on fabric target i to obtain the predicted coordinates of fabric target i at the RGB detection time. S5. Unify the conveyor belt coordinate system and calculate the Euclidean distance between the predicted coordinates of fabric target i and the observed coordinates of fabric target j, as well as the difference in fabric width between the two. S6. Construct the association cost function between cloth target i and cloth target j based on Euclidean distance and size difference; S7. Using a single cloth target i as the matching subject, iterate through all cloth targets j that have been collected but not yet matched at the current time to calculate the association cost, select the cloth target j with the smallest cost and not exceeding the preset threshold, and bind the material of cloth target i and the color of cloth target j to the same physical cloth. S8. In the digital twin virtual scene, generate a unique virtual cloth instance for each successfully matched cloth target i and cloth target j, and synchronously map the material, color and real-time coordinates. S9. As the conveyor belt runs, the coordinates of the virtual fabric instance are dynamically updated based on the real-time pulse increment of the encoder, so as to achieve real-time position synchronization between the physical fabric and the virtual fabric.

[0007] Preferably, the initial physical coordinates of fabric target i in S2 are obtained through a line-scan hyperspectral image stitching algorithm. The foreground region is extracted by threshold segmentation of the hyperspectral line-scan image, and the overlapping regions in the same row are merged to obtain the two-dimensional region of the fabric. The centroid pixel coordinates of the region are calculated, and the initial coordinates are obtained by pixel-to-physical conversion. The formula is: ; In the formula, Let x be the horizontal coordinate of the fabric target i in the vertical conveying direction. Let i be the initial ordinate of the fabric target along the conveying direction; Synchronous recording of encoder pulses Extract the width of the target i of the fabric. .

[0008] Preferably, the observed physical coordinates of the cloth target j in S3 are obtained through the YOLOv8n target detection model, and the observed coordinates are obtained by perspective projection transformation of the center pixel of the detection bounding box. The formula is: ; In the formula, Let j be the x-coordinate of the fabric target j in the vertical conveying direction. The longitudinal coordinate of the fabric target j along the conveying direction is observed; Synchronous recording of encoder pulses Extract the target width j of the fabric. .

[0009] Preferably, the pulse difference in S4 The formula is: ; Unit pulse delivery displacement The formula is: ; In the formula, Unit pulse displacement; Compensated predicted coordinates The formula is: .

[0010] Preferably, the Euclidean distance in S5 The formula is: ; In the formula, To compensate for the horizontal coordinate of the target i of the fabric, The ordinate of the fabric target i after compensation; Size differences The formula is: .

[0011] Preferably, the association cost function in S6 The formula is: ; In the formula, This is the distance weighting coefficient. Let be the size difference weighting coefficient, and satisfy . .

[0012] Preferably, the preset threshold in S7 is If the minimum association cost between cloth target i and all cloth targets j exceeds If the target fabric i is not found, the fabric will be temporarily stored and the data will be rematched in the next round of RGB fabric data.

[0013] Preferably, after the material of fabric target i in S7 is bound to the color of fabric target j, complete fabric attribute data is generated for fabric identification and control in the sorting system.

[0014] Preferably, the virtual cloth instance in S8 is rendered in the Unity3D virtual scene in the form of a 3D mesh or a 2D sprite. The texture of the virtual model is determined by the material of cloth target i, and the main color of the model is determined by the color of cloth target j.

[0015] Preferably, the encoder in S9 is an incremental rotary encoder with a unit pulse displacement. The formula is: ; In the formula, D is the diameter of the drive wheel, and N is the number of pulses per revolution of the encoder. It is a constant; The coordinates of the virtual cloth instance are updated asynchronously using a coroutine, without blocking the main rendering thread of the virtual scene. The formula for updating the coordinates of the virtual cloth is: ; ; In the formula, The real-time vertical coordinate of the virtual fabric. The initial y-coordinate created for the virtual instance. For encoder real-time pulses, The time pulse created for the virtual instance The real-time horizontal axis of the virtual cloth. The initial x-coordinate created for the virtual instance.

[0016] Therefore, the present invention employs the above-mentioned method for cross-modal virtual-real mapping of waste fabric based on conveying displacement, which has the following beneficial effects: 1. Achieve automatic target-level association of cross-modal information: This invention uses the encoder's transmission displacement as the time reference to dynamically advance the target position of hyperspectral detection to the RGB detection time. Under a unified transmission coordinate system, it constructs an association cost function by combining position distance and size difference, which can automatically complete the target-level binding of material information and color information without manual intervention. The association accuracy is high, which effectively solves the problem of difficulty in matching multi-source information caused by workstation separation and time asynchrony.

[0017] 2. Achieve real-time virtual-physical synchronization for each target: In the digital twin model, this invention generates an independent virtual instance for each physical fabric and uses real-time encoder pulses to directly drive the position update of the virtual instance, so that the movement of the fabric in the virtual scene is strictly synchronized with the physical conveyor belt. The end-to-end synchronization delay can be controlled within 200ms, which meets the real-time monitoring requirements of industrial sorting production lines.

[0018] 3. Adaptable to high-speed dynamic sorting scenarios: This invention can effectively distinguish adjacent fabrics with close spacing by using the size difference term in the cost function, reducing the probability of incorrect matching; at the same time, it adopts a coroutine asynchronous update mechanism to ensure that the high frame rate rendering of the virtual scene is not blocked by data communication, and can be adapted to high-speed sorting conveyor belts.

[0019] 4. Provide a complete data foundation for closed-loop control of sorting: The integrated multi-dimensional complete fabric data can be directly supplied to the sorting control system for fabric grading and automatic control, improving the accuracy and operating efficiency of waste fabric sorting, and supporting the digital upgrade of waste textile recycling production lines.

[0020] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0021] Figure 1This is an overall flowchart of a cross-modal virtual-real mapping method for waste fabric based on conveying displacement according to the present invention; Figure 2 This is a schematic diagram of the production line layout for the hyperspectral detection station and the RGB detection station of the present invention; Figure 3 The figure shows the experimental results of the virtual-real synchronization delay of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages disclosed in the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the embodiments of the present invention and are not intended to limit the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of this application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout.

[0023] It should be noted that the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, such as a process, method, system, product, or server that includes a series of steps or units, not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or device.

[0024] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0025] like Figures 1-3 As shown, this invention provides a cross-modal virtual-real mapping method for waste fabric based on conveying displacement. It is built upon a waste fabric industrial sorting test platform, with the following basic production line parameters set: The conveyor belt is 0.8m wide and operates at a constant speed of 1m / s. The distance between the hyperspectral and RGB stations along the conveying direction is L=1.5m. The encoder is an incremental rotary encoder, mounted on the conveyor belt drive wheel axle. The conveyor belt drive wheel diameter is D=0.1m, and the total number of pulses per revolution of the encoder is N=2000. The displacement corresponding to each pulse is... =0.000157m, the hyperspectral camera is a 900-1700nm line scan camera, the RGB camera is an industrial global shutter area array camera. In this embodiment, Hikvision MV-CS023-10UC is used. Both devices are connected to the encoder synchronous trigger signal. The current pulse value is latched synchronously for each image acquisition. The digital twin platform is built using Unity3D.

[0026] S1. Along the conveyor belt's forward direction, first fix a hyperspectral line scan camera to form an upstream hyperspectral detection station, then set up RGB area array cameras at 1.5m intervals to form a downstream RGB detection station. Measure and record the fixed distance L between the two stations along the conveying direction, and establish a unified conveyor belt plane coordinate system: the x-axis is perpendicular to the conveying direction, and the y-axis is along the fabric conveying forward direction.

[0027] like Figure 2 The production line layout diagram illustrates the overall layout of the production line. Along the conveyor belt's transport direction, the feeding and conveying unit, material detection unit (including a hyperspectral camera bracket, halogen lamp, and hyperspectral camera), and color detection unit (including a light shield and Hikvision industrial camera) are arranged sequentially. The conveyor belt is driven by a motor and supported by a bracket. The sorting execution mechanism includes an active arm, a driven arm, and an end effector. The rejection mechanism includes a suction pipe mechanism (including a suction pipe fixing base and array architecture), used to perform rejection or sorting actions based on the bound complete attribute data. This production line layout corresponds to the workstation settings, conveyor belt movement, and execution mechanism in the method of this invention, providing a physical reference for cross-modal mapping and virtual synchronization.

[0028] S2. A hyperspectral camera continuously acquires line scan images at a line frequency of 500Hz. A dynamic threshold segmentation algorithm is used to separate the fabric foreground from the conveyor belt background. Overlap is determined for each line of foreground intervals in a single frame; if adjacent line intervals overlap, they are determined to be the same piece of fabric. All associated scan lines are stitched together to form a two-dimensional fabric region. The centroid pixel coordinates of the two-dimensional region are calculated (…). , The pixel-to-physical conversion coefficients are obtained through pre-camera calibration. With image center Substituting into the formula, we get: ; ; In the formula, Let x be the horizontal coordinate of the fabric target i in the vertical conveying direction. Let be the initial ordinate of the fabric target i along the conveying direction; thus, the initial physical coordinates of the fabric target i are obtained. : ; The target width i of the fabric is obtained by converting the horizontal span of the two-dimensional region. A 1D-CNN (1D Convolutional Neural Network) is used to classify the average spectrum of the fabric and output the material type (such as cotton, polyester, blended fabric, etc.), while simultaneously latching the encoder pulses at the acquisition time. .

[0029] S3. Image acquisition is triggered when the fabric is conveyed to the center of the RGB camera's field of view. The YOLOv8n lightweight detection model is used to infer the image and output the fabric bounding box. and color categories (such as white, red, green, blue, etc.), among which These are the x and y coordinates of the top left corner of the bounding box. These are the horizontal and vertical coordinates of the bottom right corner of the bounding box, conforming to the standard YOLO detection definition.

[0030] Take the center pixel coordinates of the bounding box, and use the homography matrix obtained from the checkerboard calibration to perform perspective projection transformation, thus obtaining the observed physical coordinates of the cloth target j. : ; In the formula, Let j be the x-coordinate of the fabric target j in the vertical conveying direction. The vertical coordinate of fabric target j along the conveying direction is observed; the width of fabric target j is calculated from the horizontal pixels of the boundary. The model synchronously outputs the fabric color category and latches the current encoder pulse. .

[0031] S4. For each fabric target i, iterate through all collected fabric targets j and calculate the pulse difference. : ; Because the hyperspectral station is located upstream of the conveyor belt and its acquisition time is earlier than that of the RGB station, the pulse count continues to increase. The value is always positive, and the predicted displacement of the fabric target i along the y-direction is... for: ; Formula for compensated predicted coordinates for: ; The fabric exhibits no lateral conveying offset, and the horizontal axis remains unchanged. The displacement remains unchanged, only the longitudinal displacement is superimposed to complete the spatiotemporal alignment of the fabric target i to the RGB acquisition time.

[0032] S5. Calculate the two matching parameters in the same conveyor belt coordinate system, as shown below: The Euclidean distance between the predicted coordinates of cloth target i and the observed coordinates of cloth target j: ; In the formula, To compensate for the horizontal coordinate of the target i of the fabric, The ordinate of the fabric target i after compensation; fabric width dimension difference coefficient: .

[0033] S6. Construct the association cost function for pairing cloth target i with cloth target j. : ; In the formula, This is the distance weighting coefficient. Let be the size difference weighting coefficient, and satisfy . In this embodiment, a fixed value is taken. =0.7, =0.3, setting The reasons for the higher accuracy are as follows: In high-speed conveyor belt scenarios, the predicted position accuracy of hyperspectral targets after encoder compensation is relatively high (measured error < 0.05 meters). However, the fabric may wrinkle, rotate, or undergo local deformation during movement, resulting in relatively large fluctuations in the width measurement value. Therefore, the distance term... Size comparison item More reliable, and given higher weight. Under other operating conditions, α and β can be adjusted according to the actual measurement accuracy to meet the requirements. That's all.

[0034] S7. Taking a single cloth target i as the matching subject, iterate through all cloth targets j that have been collected but not yet matched at the current time and calculate the association cost. Filter for minimum values ​​and set a preset threshold for matching. =0.2m; if the minimum cost is ≤0.2m, then the current cloth target i and cloth target j are determined to be the same solid cloth, and the material attribute of cloth target i and the color attribute of cloth target j are bound to generate complete cloth attribute data; if the minimum cost is greater than the preset threshold This indicates no matching fabric. The current fabric target i is temporarily stored, waiting for the next round of RGB data collection for fabric target j to be matched again. The complete fabric attribute data after binding is output to the sorting system for fabric identification and automatic control.

[0035] S8. In the Unity3D digital twin virtual scene, pre-create a general cloth prefab, including two rendering modes: 2D sprite and 3D mesh. For each successfully matched cloth target i and cloth target j, instantiate an independent virtual cloth object: load the corresponding texture map according to the material of cloth target i (e.g., cotton texture for cotton, smooth texture for polyester), adjust the main color of the model according to the color of cloth target j (adjusted through material color parameters), and synchronize its material, color and position; assign the matched physical coordinates to the initial position of the virtual instance, and establish a unique mapping relationship between the virtual object and the physical cloth.

[0036] S9. Enable an independent asynchronous coroutine to periodically poll the encoder's real-time pulses. In this embodiment, the period is 10ms; the pulse at the instant the virtual instance is created is recorded. Initial vertical coordinates Real-time vertical coordinate update formula: ; The virtual instance has no horizontal offset, as shown below: ; In the formula, The real-time horizontal axis of the virtual cloth. The initial horizontal coordinate created for the virtual instance. The coordinate is updated asynchronously by the coroutine and will not occupy rendering resources of the main thread of the virtual scene; when the vertical coordinate of the virtual cloth exceeds the threshold at the end of the conveyor belt, the virtual instance is automatically destroyed to release rendering resources.

[0037] Experimental verification: Fifty sets of repeated tests were conducted on the above-mentioned implementation platform, with two types of colored waste fabrics, pure cotton and polyester, randomly selected. The test results are shown below: The material recognition accuracy rate is 98.0%, the color recognition accuracy rate is 96.0%, and the cross-modal fabric target i-fabric target j association matching success rate is 94.0%, meaning that the proportion of materials and colors correctly associated with the same fabric is 94.0%. Figure 3 As shown, the total end-to-end latency of virtual-real synchronization consists of four parts: material detection latency, color detection latency, OPC UA (Open Platform Communications Unified Architecture) communication latency, and twin system response latency. The end-to-end latency range of virtual-real synchronization is 168.8ms~181.1ms, with an average latency of approximately 175.7ms, which fully meets the requirements of industrial real-time visualization and sorting control. The overall sorting success rate (correct identification and successful synchronization) is 90.0%.

[0038] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for cross-modal virtual-real mapping of waste fabric based on conveying displacement, characterized in that, Includes the following steps: S1. Set up hyperspectral detection station and RGB detection station in sequence along the conveyor belt conveying direction, and calibrate the fixed distance between the two stations along the conveying direction. S2. Identify the fabric at the upstream hyperspectral detection station, collect the fabric material and the initial physical coordinates of the conveyor belt, and synchronously record encoder pulses. Define this fabric as fabric target i. The initial physical coordinates of fabric target i are obtained through a line-scan hyperspectral image stitching algorithm. The foreground region is extracted by threshold segmentation of the hyperspectral line-scan image, and overlapping regions in the same line are merged to obtain the two-dimensional region of the fabric. The centroid pixel coordinates of the region are calculated, and the initial coordinates are obtained through pixel-to-physical conversion. The formula is: ； In the formula, Let x be the horizontal coordinate of the fabric target i in the vertical conveying direction. Let i be the initial ordinate of the fabric target along the conveying direction; Synchronous recording of encoder pulses Extract the width of the target i of the fabric. ; S3. Identify the fabric at the downstream RGB detection station, collect the fabric color and the observed physical coordinates of the conveyor belt, and synchronously record encoder pulses. Define this fabric as fabric target j. The observed physical coordinates of fabric target j are obtained through the YOLOv8n target detection model, which is obtained by perspective projection transformation of the center pixel of the detection bounding box. The formula is: ； In the formula, Let j be the x-coordinate of the fabric target j in the vertical conveying direction. The longitudinal coordinate of the fabric target j along the conveying direction is observed; Synchronous recording of encoder pulses Extract the target width j of the fabric. ; S4. Based on the encoder pulse difference and unit pulse transport displacement corresponding to fabric target i and fabric target j, perform transport displacement compensation on fabric target i to obtain the predicted coordinates of fabric target i at the RGB detection time. S5. Unify the conveyor belt coordinate system and calculate the Euclidean distance between the predicted coordinates of fabric target i and the observed coordinates of fabric target j, as well as the difference in fabric width between the two. S6. Construct the association cost function between cloth target i and cloth target j based on Euclidean distance and size difference; S7. Using a single cloth target i as the matching subject, iterate through all cloth targets j that have been collected but not yet matched at the current time to calculate the association cost, select the cloth target j with the smallest cost and not exceeding the preset threshold, and bind the material of cloth target i and the color of cloth target j to the same physical cloth. S8. In the digital twin virtual scene, generate a unique virtual cloth instance for each successfully matched cloth target i and cloth target j, and synchronously map the material, color and real-time coordinates. S9. As the conveyor belt runs, the coordinates of the virtual cloth instance are dynamically updated based on the real-time pulse increment of the encoder, achieving real-time position synchronization between the physical cloth and the virtual cloth; the encoder is an incremental rotary encoder with a unit pulse displacement. The formula is: ； In the formula, D is the diameter of the drive wheel, and N is the number of pulses per revolution of the encoder. It is a constant; The coordinates of the virtual cloth instance are updated asynchronously using a coroutine, without blocking the main rendering thread of the virtual scene. The formula for updating the coordinates of the virtual cloth is: ； ； In the formula, The real-time vertical coordinate of the virtual fabric. The initial y-coordinate created for the virtual instance. For encoder real-time pulses, The time pulse created for the virtual instance The real-time horizontal axis of the virtual cloth. The initial x-coordinate created for the virtual instance.

2. The method for cross-modal virtual-real mapping of waste fabric based on conveying displacement according to claim 1, characterized in that, Pulse difference in S4 The formula is: ； Unit pulse delivery displacement The formula is: ； In the formula, Unit pulse displacement; Compensated predicted coordinates The formula is: 。 3. The method for cross-modal virtual-real mapping of waste fabric based on conveying displacement according to claim 2, characterized in that: Euclidean distance in S5 The formula is: ； In the formula, To compensate for the horizontal coordinate of the target i of the fabric, The ordinate of the fabric target i after compensation; Size differences The formula is: 。 4. The method for cross-modal virtual-real mapping of waste fabric based on conveying displacement according to claim 3, characterized in that: Association cost function in S6 The formula is: ； In the formula, This is the distance weighting coefficient. Let be the size difference weighting coefficient, and satisfy . .

5. The method for cross-modal virtual-real mapping of waste fabric based on conveying displacement according to claim 1, characterized in that: The preset threshold in S7 is If the minimum association cost between cloth target i and all cloth targets j exceeds If the target fabric i is not found, the fabric will be temporarily stored and the data will be rematched in the next round of RGB fabric data.

6. The method for cross-modal virtual-real mapping of waste fabric based on conveying displacement according to claim 1, characterized in that: After the material of fabric target i in S7 is bound to the color of fabric target j, complete fabric attribute data is generated for fabric identification and control in the sorting system.

7. The method for cross-modal virtual-real mapping of waste fabric based on conveying displacement according to claim 1, characterized in that: In S8, virtual cloth instances are rendered in a Unity3D virtual scene as 3D meshes or 2D sprites. The texture of the virtual model is determined by the material of cloth target i, and the main color of the model is determined by the color of cloth target j.

Images