Electronic tag pasting detection method and system based on image processing

Abstract

The invention discloses an electronic tag pasting detection method and system based on image processing, and belongs to the technical field of data processing.The electronic tag pasting detection method comprises the steps that through a full-stack intelligent system including perception (S1), deep fusion (S2), intelligent decision making (S3), precise execution (S4) and closed-loop evolution (S5), high-dimensional fusion features generated through deep learning are used for replacing traditional manual features, and the detection accuracy is improved; and millimeter-level and even sub-pixel-level detection precision is realized. A dynamic inverse decoupling control equation of coupling of multiple physical quantities (light, heat, geometry and radio frequency) is established, and collaborative correction of complex deviation is achieved. A closed-loop evolution mechanism driven by reinforcement learning is innovatively introduced, and the system is endowed with continuous learning and self-optimization capabilities. The system can realize high-precision electronic tag pasting in a complex environment, and ensures that the readable rate of the RFID is effectively improved at the same time.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an electronic tag pasting detection method and system based on image processing. Background Technology

[0002] The core objectives of electronic tag pasting detection are: existence detection: whether the tag is pasted in the designated position; integrity detection: whether the information on the tag surface (barcode / text) is clear and readable; and positional accuracy: whether the pasting offset is within the tolerance range (e.g., ±2mm).

[0003] However, electronic tags are currently susceptible to interference from factors such as metal surfaces, liquids, and temperature changes, posing a dual challenge of poor environmental adaptability and insufficient reliability for electronic tag adhesion detection. For example, strong reflections from metal surfaces can cause the tag outline to be submerged by light spots (reflectivity > 80%), resulting in a false negative rate of up to 25%. Tags inside liquid containers cannot be positioned due to light refraction / scattering (transmission loss > 50%), leading to the failure of tag presence detection. In addition, temperature deformation (PET tags expand by 0.3 mm at ΔT = 50℃) causes barcode distortion, and curved surface adhesion causes QR code stretching and deformation (deformation rate of cylindrical surfaces > 15%), resulting in distortion of electronic tag adhesion integrity detection. Furthermore, metal / liquid interference can cause visual positioning deviations > 1 mm, and temperature drift can cause coordinate offsets of the robotic arm (thermal expansion of aluminum alloy 23 ppm / ℃), leading to loss of control over the accuracy of electronic tag adhesion position.

[0004] Therefore, there is an urgent need to develop an electronic tag attachment detection method that is environmentally adaptable and highly reliable. Summary of the Invention

[0005] To address the aforementioned technical problems in the prior art, this invention provides an electronic tag pasting detection method based on image processing, comprising the following steps: S1. The production line triggers an electronic tag acquisition command. The camera array acquires polarization images, near-infrared images, temperature data and / or 3D point cloud data of the electronic tags according to the acquisition command, generates a multimodal dataset of electronic tags, and transmits it to the processing industrial control computer. S2. The industrial control computer preprocesses and fuses the features of the electronic tag multimodal dataset to generate an electronic tag feature fusion dataset. S3. The industrial control computer performs multi-scale deep feature extraction and dynamic matching on the electronic tag feature fusion dataset to generate an electronic tag feature matching dataset. S4. The industrial control computer generates deviation correction instructions based on the electronic tag feature matching dataset, sends the deviation correction instructions to the PLC for deviation correction, generates electronic tag quality data and uploads it to the MES system. The S5 and MES systems analyze and learn from the quality data of electronic tags, and feed it back to the industrial control computer for dynamic optimization of production line parameters and adjustment of control strategies.

[0006] Furthermore, before or after the production line triggers the electronic tag collection command in step S1, real-time environmental status detection is also included. Through data processing at the sensor or camera front end, the current environmental status is analyzed in real time. Based on the collected real-time environmental status characteristics, it is determined in real time which cameras in the camera array need to be activated.

[0007] Furthermore, the current environmental state includes metal surfaces, liquid containers, temperature changes, and curved surfaces.

[0008] Furthermore, based on the collected real-time environmental characteristics, it is determined in real time which cameras in the camera array need to be activated, including: The polarization camera is activated when there are metal surfaces in the environment or when metal items are detected on the production line. When a liquid container or liquid environment is detected, the near-infrared camera is triggered to acquire images, penetrate the container, and read tag information; The thermal imaging camera is activated when a change in real-time temperature data or temperature data detected by a sensor is detected. When a curved surface feature is detected at the label location, or when it is identified that the label is attached to a complex three-dimensional surface, a 3D structured light camera is activated to detect the three-dimensional fit of the label.

[0009] Furthermore, the preprocessing in step S2 includes: Metal surface processing is performed on polarized images: Images with different polarization angles are obtained by a polarized high-definition camera, and the polarization difference method is used to process the images with different polarization angles to remove reflection noise from the metal surface; Perform liquid environment enhancement processing on near-infrared images: perform image enhancement processing such as contrast stretching, histogram equalization, or descattering on near-infrared images; Perform temperature deformation compensation on temperature data: Based on the temperature data collected by the thermal imaging camera and the known thermal expansion coefficient of the electronic tag material, calculate the thermal deformation of the tag's geometric dimensions or position caused by temperature changes. For the thermal deformation of different materials, apply image scaling or affine transformation to spatially compensate the affected image or 3D point cloud data. Perform surface reconstruction on 3D point cloud data: Combine point cloud density and curvature analysis to perform multi-level deep data analysis and detect whether the labels accurately fit the surface.

[0010] Furthermore, the feature fusion includes: Step 1, Dynamic Data Alignment and Calibration: Align all modal data acquired by the camera enabled in Step S1 to a common coordinate system to generate dynamic data alignment and calibration multimodal data; Step 2: Based on the dynamically aligned and calibrated multimodal data, perform feature extraction and representation to generate a series of feature maps or feature vectors with different dimensions but spatial alignment; Step 3: Perform dynamic and intelligent multimodal feature fusion on the series of feature maps or feature vectors with different dimensions but spatial alignment to generate an electronic tag feature fusion dataset.

[0011] Furthermore, the fusion strategy for the dynamic and intelligent multimodal feature fusion includes: Step 31: Early feature complementarity fusion to capture low-level general correlations: The series of feature maps or feature vectors with different dimensions but spatial alignment in Step 2 are concatenated or operated on element-wise according to the channel dimension to capture the complementary information of different modalities at the low level and form a joint core fusion feature map; For the modal data obtained by the camera that was not enabled in Step S1, their respective features are generated by independent, pre-trained and deepened neural network encoders to generate other independently extracted high-level modal features. Step 32, Intermediate Layer Adaptive Fusion, Learning Advanced Dynamic Associations: The core fusion feature map and other unique features are combined... The high-level modal features extracted are input into the attention mechanism encoder model to learn the complex correlation between different modal features, and the fusion weights are dynamically adjusted according to the modalities actually enabled in step S1 to generate a global fusion feature tensor. Step 33: Decision layer adaptation and task-specific branch, task-customized output: The global fusion feature tensor is connected as input to independent head networks for different detection and analysis tasks, and the results are output independently for each task branch.

[0012] Furthermore, the multi-scale depth feature extraction in step S3 includes the following steps: S31. Use the “global fusion feature tensor” output in step S2 as the starting point for multi-scale deep feature extraction; S32. Introduce a deep learning method based on EfficientNetB7 backbone network and feature pyramid to extract features at multiple scales, capturing label information from different scales including P2, P3, P4 and P5 layers. S33. Based on the multi-scale feature pyramid, feature decoupling and enhancement are performed on the label information captured at different scales to generate real-time feature tensors.

[0013] Furthermore, the dynamic matching compares the real-time feature tensor generated in the multi-scale deep feature extraction stage with the ideal label features pre-stored in the multi-modal template library. Using the conditional attention mechanism, weights are intelligently assigned according to the availability and importance of different modalities to calculate the similarity or difference between the real-time label features and the template features. Then, deformation robustness enhancement is used to compensate for thermal deformation.

[0014] This invention also provides an electronic tag affixing detection system based on image processing, which applies the aforementioned electronic tag affixing detection method based on image processing. The system includes: The image acquisition module uses a camera array to acquire image data from electronic tags at high speed and generate a multimodal dataset of electronic tags. The image preprocessing module preprocesses and fuses features in the electronic tag multimodal dataset to generate an electronic tag feature fusion dataset. The feature extraction and matching module uses a deep learning method based on the EfficientNetB7 backbone network and feature pyramid to extract, decouple, and enhance the electronic tag features of the electronic tag feature fusion dataset at multiple scales, generate a real-time feature tensor, and dynamically match the real-time feature tensor with a predefined multimodal standard template library. The deviation correction module uses image analysis algorithms to calculate the deviation amount for detected deviations in the position, angle, and shape of electronic tags, and then corrects or issues an alarm. The real-time feedback and control module connects the image processing results of the deviation correction module with the production line control system to automatically adjust the electronic tag pasting parameters during the production process, thereby achieving closed-loop control.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention forms a full-stack intelligent system from "perception (S1) - deep fusion (S2) - intelligent decision-making (S3) - precise execution (S4) - and closed-loop evolution (S5)". High-dimensional fusion features generated by deep learning replace traditional manual features, achieving millimeter-level or even sub-pixel-level detection accuracy. A dynamic inverse decoupling control equation for the coupling of multiple physical quantities (light, heat, geometry, radio frequency) is established to achieve collaborative correction of complex deviations. An innovative reinforcement learning-driven closed-loop evolution mechanism is introduced, endowing the system with the ability to continuously learn and self-optimize. The system can achieve high-precision electronic tag affixing in complex environments while ensuring that the RFID readability is effectively improved (e.g., exceeding 99.5%).

[0016] This invention overcomes the physical limitations of metals, liquids, temperature, and curved surfaces through multimodal sensing in step S1, improving environmental adaptability. Through sensor feedback and synchronous acquisition of camera data, the system can automatically adjust the state of the camera array, ensuring complete multimodal data acquisition under various environmental influences. Providing original light-thermal-geometric multidimensional features guarantees data completeness. High frame rate and low latency meet production line cycle time, improving system real-time performance, which is a prerequisite for high-precision detection of electronic tags in complex industrial environments.

[0017] This invention employs a fusion strategy that integrates early feature layer fusion with later decision layer fusion, combining the advantages of multimodal approaches and encompassing visual details, internal structure, 3D geometry, and temperature characteristics. Through dynamic data alignment and conditional attention mechanisms, it can effectively integrate with the dynamically activated camera in step S1, maintaining high performance even when some modalities are missing. By employing high-precision quantization and a "feature-deviation-control" mapping model, deviations are transformed into coordinated control commands for multiple actuators, enabling precise correction of the spatial position, orientation, and deformation of electronic tags. This process fully leverages panoramic information from multimodal matching and high-precision deviation quantization, combined with physical models and real-time signal feedback, ensuring that the detection and correction of electronic tags in industrial environments achieves a high level of reliability and precision for industrial applications.

[0018] By establishing a data-driven closed-loop feedback system, the MES system (including high-quality data analysis and learning) is tightly integrated with step S4 (deviation correction based on high-precision vision and multimodal matching). Through continuous data analysis, optimization, and feedback, performance is continuously improved. This system fully utilizes the panoramic information from multimodal matching and high-precision deviation quantification, combined with physical models and real-time signal feedback, to ensure that the detection and correction of electronic tags in industrial environments achieves a high level of reliability and precision for industrial applications, ultimately resulting in precise positioning, high efficiency, automation, and high reliability in labeling. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1This is a flowchart of an electronic tag pasting detection method based on image processing according to the present invention; Figure 2 This is a structural block diagram of an electronic tag pasting detection system based on image processing according to the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0023] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0024] Furthermore, the use of terms such as "first" and "second" in this invention is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" and "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. When the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed by this invention.

[0025] Example 1 See Figure 1 As shown, the present invention provides an electronic tag pasting detection method based on image processing, comprising the following steps: S1. The production line triggers an electronic tag acquisition command; the camera array acquires polarization images, near-infrared images, temperature data and / or 3D point cloud data of the electronic tags according to the acquisition command, generates a multimodal dataset of electronic tags, and transmits it to the processing industrial control computer; The camera array includes, for example, a polarized high-definition camera, a near-infrared camera, a thermal imaging camera, and / or a 3D structured light camera. The polarized high-definition camera, with its 20MP (20 million pixels) pixel resolution and global shutter (simultaneous exposure of all pixels) and 60fps (60 frames per second) frame rate, can effectively suppress overexposure or information loss caused by specular reflection by analyzing the intensity differences in light along different polarization directions, thus enabling the acquisition of polarized images. The near-infrared camera, with a wavelength of 850nm and a resolution of 5MP (5 million pixels), can effectively penetrate the walls of opaque or semi-transparent liquid containers, revealing internal label information. The system includes: a near-infrared image acquisition camera; a thermal imaging camera with an accuracy of ±0.5℃ (temperature measurement error range) and a resolution of 640×480, which can monitor temperature deformation and acquire temperature data; and a 3D structured light camera with a Z-axis accuracy of ±0.01mm (depth measurement error (using the phase shift method)) and a point cloud density of 500pt / mm² (number of three-dimensional points per unit area), and using a blue light wavelength of 450nm (the structured light projection wavelength, which is more resistant to ambient light interference than red light), which can perform surface pasting detection and acquire 3D point cloud data.

[0026] In this embodiment, a polarized high-definition camera suppresses metal reflection and a near-infrared camera penetrates the liquid to collect internal feature data of the electronic tag; a thermal imaging camera performs temperature compensation and a 3D structured light camera performs surface reconstruction to collect three-dimensional fit data of the electronic tag.

[0027] In this embodiment, before or after the production line triggers the electronic tag acquisition command, real-time environmental status detection is also included. This is achieved through data processing at the sensor or camera front end, analyzing the current environmental status (e.g., metal surfaces, liquid containers, temperature changes, curved surfaces, etc.) in real time. Based on the collected real-time environmental status characteristics, the detection system determines in real time which cameras in the camera array need to be activated. For example: When a metal surface is present in the environment, or when a metal object is detected on the production line, the polarization camera will automatically activate to suppress metal reflection. When a liquid container or liquid environment is detected, the near-infrared camera is triggered to acquire images so that it can penetrate the container and clearly read the label information; When a large change in real-time temperature data or temperature data detected by a sensor is detected, the thermal imaging camera is activated to compensate for the deformation caused by the temperature change. When a label with curved surface features is detected, or when a label is identified as being attached to a complex three-dimensional surface, the detection system activates a 3D structured light camera to ensure the three-dimensional fit of the label is detected.

[0028] The specific process is as follows: When an automated production line in the workshop is ready to be labeled, the detection system issues an "acquisition command" from a PLC or industrial control computer. Through data processing at the sensor or camera front end, it analyzes the current environmental status in real time. Based on the current environmental status, it activates a camera array (including polarization camera, near-infrared camera, thermal imaging camera, and 3D structured light camera) to capture the multimodal dataset of electronic tags under different environmental conditions. Then, it transmits the captured multimodal dataset of electronic tags (including polarization image, infrared image, temperature data, and / or 3D point cloud data) to the processing industrial control computer.

[0029] Example: On a production line, an electronic tag is about to be affixed to a metal surface, and the tag is contained within a liquid container. In this situation, the system will activate a polarization camera (to suppress metal reflection) and a near-infrared camera (to penetrate the liquid container), while a thermal imaging camera will also be activated to monitor the effects of temperature, ensuring that all interfering factors are effectively handled.

[0030] This invention overcomes the physical limitations of metals, liquids, temperature, and curved surfaces through multimodal sensing in step S1, improving environmental adaptability. Through sensor feedback and synchronous acquisition of camera data, the system can automatically adjust the state of the camera array, ensuring complete multimodal data acquisition under various environmental influences. Providing original light-thermal-geometric multidimensional features guarantees data completeness. High frame rate and low latency meet production line cycle time, improving system real-time performance, which is a prerequisite for high-precision detection of electronic tags in complex industrial environments.

[0031] S2. The industrial control computer preprocesses and fuses the features of the electronic tag multimodal dataset to generate an electronic tag feature fusion dataset. The preprocessing includes: If the polarization-enhanced high-definition camera is activated and polarization images are acquired in step S1, then metal surface processing is performed: Images at different polarization angles acquired by the polarization-enhanced high-definition camera are processed using a polarization difference method to remove reflection noise from the metal surface, thereby improving the realism of the polarization images, highlighting label details, and avoiding the influence of noise caused by surface reflection. To improve data realism, especially when reading labels around metal objects, multi-angle polarization image acquisition can minimize interference from reflections. If the near-infrared camera was activated and acquired a near-infrared image in step S1, liquid environment enhancement is performed: Taking advantage of the low absorption characteristics of the near-infrared band (850nm) in liquids, image enhancement (e.g., contrast stretching, histogram equalization, or specific descattering algorithms) is applied to the near-infrared image, ensuring that the label's internal information (such as printed content, chip location, etc.) can still be clearly read even in a liquid container. During liquid environment enhancement, the light source intensity and exposure time of the near-infrared camera are dynamically adjusted to avoid excessive penetration or reflection, thus optimizing image quality.

[0032] If the thermal imaging camera is activated and acquires temperature data in step S1, temperature deformation compensation is performed: based on the temperature data acquired by the thermal imaging camera and combined with the known thermal expansion coefficient of the electronic tag material, the minute deformations in the tag's geometric dimensions (such as length and width) or position that may be caused by temperature changes are calculated. For thermal deformation of different materials, image scaling, affine transformation, or other geometric correction algorithms are applied to spatially compensate the affected images (such as polarized images, near-infrared images) or 3D point cloud data to eliminate the negative impact of temperature changes on labeling accuracy and 3D fit analysis. During the temperature compensation process, the time delay of temperature changes is considered, and data is updated in a timely manner for dynamic compensation. Temperature deformation compensation ensures that the geometric dimensions and positional information of the tags remain consistent in the dataset under different temperature environments, improving measurement accuracy.

[0033] If the 3D structured light camera is activated and acquires 3D point cloud data in step S1, surface reconstruction is performed: High-precision depth data analysis is used to detect whether the label accurately fits the surface. By analyzing changes in the point cloud at different depths, labeling errors caused by the surface shape are compensated. To improve the accuracy of surface reconstruction, the system combines point cloud density and curvature analysis to perform multi-level depth data analysis, ensuring the accuracy and reliability of surface detection.

[0034] Step S2 also includes performing feature fusion on one or more modal data that have been preprocessed by the industrial control computer (determined according to the camera actually activated in step S1) to generate an electronic tag feature fusion dataset.

[0035] The feature fusion includes: Step 1, Dynamic Data Alignment and Calibration: Regardless of which cameras are enabled in step S1, the data from all modalities must be aligned to a common coordinate system. This embodiment uses 3D point cloud data (if acquired) as the reference, or a polarization image as the visual reference. Specifically: If the 3D structured light camera is enabled: polarization images, near-infrared images, thermal imaging temperature data, etc., are accurately registered to the coordinate system of the 3D point cloud through camera calibration parameters and geometric transformations, so that the visual information, internal information and temperature information of the tag can be directly correlated with the precise three-dimensional geometry and spatial position of the tag.

[0036] If the 3D structured light camera is not enabled: Select a polarized image or a near-infrared image as the visual reference benchmark. Data from other modalities (such as temperature data) will be aligned to the visual reference benchmark in two dimensions or approximately three dimensions. The alignment method can be any method commonly used in the art, and will not be elaborated here.

[0037] Through the above dynamic data alignment and calibration, it is ensured that the pixel / voxel positions of the same physical point in different modal data correspond to each other, providing a unified spatial basis for subsequent feature fusion.

[0038] Step 2: Based on the dynamically aligned and calibrated multimodal data, perform feature extraction and representation to generate a series of feature maps or feature vectors with different dimensions but spatial alignment. Specifically: For polarized images: Extract visual detail features such as surface texture, text, patterns, and edges of the labels.

[0039] For near-infrared images: Extract the internal structure and coding of the tag (such as the location and shape of the RFID chip), and the concealed features after liquid penetration.

[0040] Temperature data can be used as additional pixel-level feature channels to represent the temperature distribution characteristics of different regions of the label. After temperature compensation, this data can also be used to verify the thermal uniformity of the material.

[0041] For 3D point cloud data: extract the three-dimensional geometric features of the labels (such as shape, curvature, flatness, height, volume, etc.) and the fit features between the labels and the carrier (such as gap, degree of warping, etc.).

[0042] Step 3: Dynamically and intelligently fuse the aforementioned series of spatially aligned feature maps or feature vectors with different dimensions. This effectively combines the preprocessed and aligned modal features to generate a more expressive "electronic tag feature fusion dataset." The specific fusion strategy is as follows: Step 31: Early Feature Complementary Fusion to Capture Low-Level General Correlations: The series of feature maps or feature vectors from Step 2, with different dimensions but spatial alignment, are concatenated along the channel dimension or subjected to simple element-wise operations to capture complementary information from different modalities at a low level, forming a joint core fusion feature map. This includes: The series of spatially aligned feature maps or feature vectors with different dimensions from step 2 (such as polarization visual features, near-infrared penetration features, temperature feature maps, 3D geometry and fit features, etc.) are used as input. The inputs are concatenated along the channel dimension to form a joint core fusion feature map, including: for the modalities enabled in step S1, their respective feature maps or feature vectors (after alignment in step 1 of step S2, it is assumed that they have been unified to a compatible feature map size) are concatenated along the channel dimension to form a joint multi-channel feature map; the concatenated joint multi-channel feature map is input into a lightweight shared encoder network (e.g., several layers of convolutional network or several Transformer encoder layers) for feature learning, extracting high-level semantic information across modalities, and generating a preliminary, richer core fusion feature map.

[0043] Example: Given the fundamental and complementary nature of polarization images (visual details) and 3D point clouds (3D geometry) in electronic tag detection tasks, if both the polarization high-resolution camera and the 3D structured light camera are enabled in step S1, their respective feature maps or feature vectors (assuming they have been unified to a compatible feature map size after alignment in step 1 of step S2) are stitched together along the channel dimension. After stitching, the resulting image is input into the lightweight shared encoder network for feature learning, aiming to learn the underlying correlation between polarization visual information and 3D geometric information (e.g., how to correct visual distortion through depth information, how to enhance the details of geometric reconstruction through texture information, etc.), generating a preliminary, richer core fusion feature map.

[0044] Furthermore, this embodiment also includes other modalities (such as near-infrared, thermal imaging, etc.) that may be enabled or disabled in step S1. Their respective features continue to be processed by independent, but possibly pre-trained and deepened, neural network encoders to obtain higher-level modality-specific features, generating other independently extracted high-level modality features. Thus, even if these modalities are not enabled (data is missing), the core fusion process will not be interrupted; when they are enabled, their features can be independently extracted and represented, awaiting subsequent higher-level fusion.

[0045] Step 32, Intermediate Layer Adaptive Fusion, Learning Advanced Dynamic Associations: The core fusion feature map and other unique features are combined... The system extracts high-level modal features and modal presence indicators, and uses input attention mechanisms (such as Conditional Attention Networks) or cross-modal encoder models to intelligently learn the complex relationships between different modal features (including the fusion results from step 31 and independent modal features). It also dynamically adjusts the fusion weights based on the modalities actually enabled in step S1, generating a unified, high-level "global fusion feature tensor," which is the initial "electronic tag feature fusion dataset." This "global fusion feature tensor" contains the most discriminative and expressive information from all available modalities after deep interaction.

[0046] The modal presence indicator is a binary vector indicating which cameras are currently enabled in step S1 (e.g., [1, 1, 0, 1] indicates that polarization, near-infrared, and 3D cameras are enabled, while thermal imaging is disabled). This modal presence indicator is incorporated into the conditional mechanism of the fusion network. The attention mechanism (such as a Conditional Attention Network) or the cross-modal encoder model is obtained or constructed using fusion mechanisms commonly used in the art, which will not be elaborated upon here.

[0047] Step 33: Decision Layer Adaptation and Task-Specific Branches, Task-Customized Output: The global fusion feature tensor is connected as input to independent head networks (Task-specific Heads) for different detection and analysis tasks. The results are output independently for each task branch, allowing for more refined, task-driven adjustments to the decision layer. Each head network is a dedicated deep learning sub-network used to extract the final decision information most relevant to the task from the fusion features. Examples include: The defect detection branch can be a segmentation network (such as a U-Net decoder) or an object detection network (such as a Region Proposal Network + Bounding Box / Mask Head) used to identify and locate various defects on the label (scratches, bubbles, printing errors, damage, etc.), and its output is a defect mask or bounding box.

[0048] The fit analysis branch—which can be a regression network—outputs a precise distance map between the label surface and the carrier, a deformation vector, or scalar values ​​indicating the degree of warping or bubbling. This branch relies heavily on depth and temperature deformation information contributed by 3D point cloud data and temperature data.

[0049] Quality assessment branch: A classification or regression network can be used to output the overall quality level of the electronic tag or a continuous quality score.

[0050] As a further preferred embodiment, in order to increase credibility or interpretability, the model training is automatically optimized through multi-task learning: the final decision level performs additional weighting or logical judgment on the outputs of different task branches (for example, if both defect detection and fit analysis indicate serious problems, it is ultimately judged as "unqualified").

[0051] The fusion strategy, which integrates early feature layer fusion with later decision layer fusion, combines the advantages of multimodal data, encompassing visual details, internal structure, 3D geometry, and temperature characteristics. Through dynamic data alignment and conditional attention mechanisms, it can effectively combine with the dynamically activated camera in step S1, maintaining high performance even with partial modal loss. The deep learning model automatically learns complex nonlinear relationships between modalities, eliminating the need for manual feature engineering. Flexible task branching ensures that the fused data efficiently serves specific defect detection and fit analysis objectives. Therefore, the processing industrial control computer can transform the dynamic multimodal data collected in step S1 into a highly refined and information-rich "electronic tag feature fusion dataset," laying a solid foundation for subsequent intelligent analysis and decision-making.

[0052] S3. The industrial control computer performs multi-scale deep feature extraction and dynamic matching on the electronic tag feature fusion dataset to generate an electronic tag feature matching dataset. The purpose of multi-scale deep feature extraction and dynamic matching is to achieve more advanced and accurate cross-modal collaborative analysis, thereby accurately determining the position, orientation and deformation state of the electronic tag on the carrier.

[0053] The multi-scale deep feature extraction specifically includes the following steps: S31. The “global fusion feature tensor” output in step S2 is used as the starting point for multi-scale deep feature extraction. This means that subsequent feature extraction operations are no longer independent processing of the original image, but operations are performed on the fused, more information-rich, and denoised high-dimensional feature space, which greatly improves the robustness and discriminative power of the features.

[0054] S32. Introducing a deep learning method based on the EfficientNetB7 backbone network and feature pyramids (such as UNet++) for multi-scale feature extraction means that the system no longer simply extracts a few predefined features, but can capture label information from different scales (P2, P3, P4, P5 layers) like the human eye. For example, the P2 layer may focus on fine edge and texture details, while the P5 layer captures macroscopic structural information.

[0055] S33. Based on the multi-scale feature pyramid, feature decoupling and enhancement are performed for different detection tasks to generate a "real-time feature tensor" that is richer in information, more diverse in dimensions, and more specific. Specifically, this includes: Geometric Feature Layer (P2 Layer): In the P2 layer, sub-pixel convolution is performed on channels related to label edges, corners, and contours in the global fusion feature tensor, as well as channels reflecting the label surface and internal texture (especially those incorporating polarization and near-infrared information). This extracts the precise geometric contours and key points of the label, and robust texture features of the label surface (such as printed characters, barcodes, and brand logos) and interior (such as RFID chip layout and wire traces). This resists illumination changes and background texture interference, better distinguishes label texture from cluttered background textures, and improves the positioning accuracy of label entity edges. It can achieve edge and curvature extraction accuracy up to ±0.05px. This means that the positioning accuracy of label edges reaches an unprecedented level, providing an ultimate benchmark for subsequent position and shape correction.

[0056] Radio Frequency Feature Layer (P4 Layer): The RSSI intensity heatmap acquired by the RFID reader is embedded into the P4 layer features and stitched together with the near-infrared feature channel. This allows the model to simultaneously consider the intensity distribution of the RFID signal during visual and geometric analysis, thereby better understanding the physical state and readability of the tag, especially in areas affected by the signal (such as metal shielding).

[0057] Thermal stability layer (P3 layer): The P3 layer channels inject 3D geometry, curvature-related information, and temperature-compensated parameters (as prior knowledge) from the global fusion feature tensor into the P3 layer channels. This allows the model to "know" the deformation of the label in 3D space (such as warping and bulging), the fit with the carrier, and the minor geometric deviations caused by temperature changes during matching analysis. Based on this, more accurate matching and deviation quantification can be performed, further improving deformation robustness.

[0058] P5 layer: Contains other macroscopic structural information, etc.

[0059] Through the above multi-scale deep feature extraction, high-quality, multi-scale features are formed, ensuring that these features not only have the abstraction capabilities of deep learning, but are also specially decoupled and enhanced to cope with specific detection tasks (geometry, RF, thermal stability), providing "all the information needed" for subsequent dynamic matching.

[0060] Dynamic matching compares the real-time feature tensor generated in the "multi-scale deep feature extraction" stage with the ideal label features pre-stored in the multi-modal template library. Utilizing a conditional attention mechanism, it intelligently assigns weights based on the availability and importance of different modalities, calculating a highly complex "similarity" or "difference" between the real-time label features and the template features. Deformation robustness enhancement (differential homeomorphism and GAN) further improves the accuracy and reliability of the matching, enabling the matching algorithm to understand and quantify the non-rigid deformation of the label (e.g., caused by temperature), rather than just rigid transformations. In short, dynamic matching determines "what the currently detected label looks like" and "how much it deviates from the ideal state" based on the extracted features. Specifically, it includes: Multimodal template library construction: First, a pre-stored multimodal standard template library is built. This library contains ideal feature fusion tensors for electronic tags in various complex environments (such as metal surfaces, liquid containers, large curved surfaces, and their combinations). This makes the matching process more targeted and better adaptable to diverse production environments.

[0061] Conditional Attention Matching Algorithm: An advanced Cross Modality Attention (CMA) algorithm is introduced for matching. Matching scores are dynamically calculated based on real-time feature tensors (features extracted from multi-scale deep feature extraction in step S3), template features, and "modality presence indicators" (which cameras were actually activated in step S1). The Conditional Attention matching algorithm intelligently allocates weights to different modal features during the matching process based on modality availability. When a modality is missing, it automatically reduces the influence of that modality and strengthens the contributions of other available modalities. It can also learn the complex, non-linear correspondence between real-time features and template features, surpassing traditional keypoint- or geometric transformation-based matching.

[0062] Enhanced Deformation Robustness: Diffeomorphic Registration is employed to compensate for thermal deformation and other non-rigid deformations. Diffeomorphic Registration finds a smooth, reversible deformation field that accurately maps the current tag features to template features, thus quantifying deformation more precisely. Simultaneously, a Generative Adversarial Network (GAN) is used to synthesize samples under extreme environmental or deformation conditions, which are then injected into the training set to improve the generalization ability of the electronic tag detection system. This allows the detection system to maintain high accuracy even when facing complex situations that are uncommon but likely to occur in actual production.

[0063] After multi-scale deep feature extraction and dynamic matching, an electronic tag feature matching dataset is generated. This dataset includes the external geometry, surface texture, internal structure, thermal deformation state, and deviation and confidence levels of the electronic tags compared to their ideal state. Specifically, it includes: successful matching indications (such as matching confidence scores), the tag's precise three-dimensional position and orientation, deviation vectors from the standard template (such as positional deviation, angular deviation, and surface fit deviation), internal structure and radio frequency characteristic evaluation results, and potential local deformation parameters and modal contribution weights / influence factors. This electronic tag feature matching dataset forms a high-dimensional, multi-attribute, and precisely quantified data set, providing a very solid data foundation for subsequent automated correction and quality control.

[0064] S4. The industrial control computer generates deviation correction instructions based on the electronic tag feature matching dataset, sends the deviation correction instructions to the PLC for deviation correction, generates electronic tag quality data and uploads it to the MES system. First, the deviation information in the electronic tag feature matching dataset is transformed into quantitative indicators that can be used for control. This deviation information includes external geometric deviations of the electronic tags (such as position and orientation), surface texture and internal structural deviations, thermal deformation states (such as warping and bulging), RFID signal characteristic deviations (RSSI strength, signal distribution entropy), and may also include local deformation parameters and modal contribution weights. Specifically, Positional deviation: Through sub-pixel edge detection and 3D point cloud reference coordinates, a high-precision deviation value of ±0.1mm is achieved through quantization.

[0065] Attitude deviation: Principal component analysis (PCA) is used to extract the main axis of the label and compare it with the template angle to quantify the angle deviation index of ±0.5°.

[0066] Curved surface fit: The Hausdorff distance of the point cloud is used to measure the fit between the label and the carrier, and the deviation is quantified as <0.05mm.

[0067] RFID signal quality: Analyze the entropy value of RSSI strength, avoid areas affected by signal interference, and measure signal stability.

[0068] Then, based on the quantitative indicators, a quantitative mapping model of "feature-deviation-control" is established. The model consists of: deviation: derived from multi-scale feature matching results (position, attitude, deformation, signal deviation); control model: using a dynamic inverse decoupling control model to map the deviation to the specific control quantity of the mechanical actuator; stiffness matrix K: describing the linear relationship between the deviation and the specific control quantity, used to adjust the stiffness of the control response; RFID signal feedback coefficient B: introducing the RFID signal change rate (d / dtRSSI entropy) as feedback to consider the impact of real-time signal changes on position adjustment.

[0069] Finally, control commands are generated and sent to the PLC for deviation correction control, including: multi-actuator collaborative correction: correcting the tag position through coordinate compensation of the robotic arm to achieve precise spatial position and angle adjustment; correcting the angular deflection of the tag by adjusting the angle of the rotating platform; and vacuum suction nozzle: adjusting pressure and gripping angle to ensure a stable and secure fit of the tag. The deviation correction results generate electronic tag quality data and are uploaded to the MES system. This achieves overall optimal correction of the electronic tag's position, orientation, and deformation state, ensuring accurate tag fit in complex environments and meeting industrial needs such as automatic identification, detection, and correction. The robustness and accuracy of the correction are improved through the combined action of multiple physical quantities (light, heat, geometry, and radio frequency).

[0070] In step S4, through high-precision quantization and a "feature-deviation-control" mapping model, the deviation is converted into coordinated control commands for multiple actuators to achieve precise correction of the electronic tag's spatial position, attitude, and deformation. This process fully utilizes panoramic information from multimodal matching and high-precision deviation quantization, combined with physical models and real-time signal feedback, to ensure that the detection and correction of electronic tags in industrial environments achieves a high level of reliability and precision for industrial applications.

[0071] The S5 and MES systems analyze and learn from the quality data of electronic tags, and feed this data back to the industrial control computer for dynamic optimization of production line parameters and adjustment of control strategies. This module, based on an adaptive PID controller, provides real-time feedback on system status and adjusts control parameters according to deviations to ensure accurate tag application and efficient production. This includes: Adaptive PID controller: This controller enhances response when deviations are large and reduces oscillations when deviations are small, thus optimizing control performance. For example, it triggers a PID parameter adjustment strategy based on the magnitude of changes in position accuracy and attitude deviation (e.g., exceeding a threshold). Larger positional deviations lead to increased Kp: This accelerates response speed and enables rapid correction.

[0072] With small positional deviation, KD adjustment: Reduce overshoot to avoid oscillations caused by excessive correction. An adaptive PID controller can set different Kp, Ki, and Kd values ​​for different labels and environments to achieve the best labeling effect.

[0073] Control strategy adjustment: Optimize the "feature-deviation-control" mapping model of S4 based on data analysis results, including: correcting the stiffness matrix K in the "feature-deviation-control" model, accurately adjusting the control quantity of the robotic arm to better match the electronic tag and carrier; using historical data (quality indicators, environmental information, control parameters) to train a machine learning model to predict the optimal control parameters or control strategy; alarming or automatically correcting faults and other anomalies, such as automatically adjusting control parameters within a certain range to attempt to correct deviations, and automatically pausing the labeling process and issuing an alarm to notify the operator to check if the deviation is too large.

[0074] RFID signal feedback coefficient B optimization: Based on signal quality and variation patterns, the RSSI signal feedback coefficient B is adjusted to increase the contribution of the RFID signal to position correction. This reduces RFID interference and improves signal reception stability while ensuring bonding quality. Real-time feedback mechanism: The above analysis and control strategy adjustments are fed back to the processing industrial control computer to monitor labeling quality parameters, such as positional accuracy, angular deviation, and RSSI intensity. Simultaneously, parameter optimization, control strategy updates, and abnormal alarm and fault information processing are performed. It also includes iterative loops by the processing industrial control computer based on information obtained from the production line.

[0075] Step S5 establishes a data-driven closed-loop feedback system, tightly integrating the MES system (including high-quality data analysis and learning) with Step S4 (deviation correction based on high-precision vision and multimodal matching). Through continuous data analysis, optimization, and feedback, performance is continuously improved. This system fully utilizes the panoramic information from multimodal matching and high-precision deviation quantification, combined with physical models and real-time signal feedback, to ensure that the detection and correction of electronic tags in industrial environments achieves a high level of reliability and precision for industrial applications, ultimately resulting in precise positioning, high efficiency, automation, and high reliability in labeling.

[0076] Through the above technical solutions, this invention forms a full-stack intelligent system from "perception (S1) - deep fusion (S2) - intelligent decision-making (S3) - precise execution (S4) - and closed-loop evolution (S5)". High-dimensional fusion features generated by deep learning replace traditional manual features, achieving millimeter-level or even sub-pixel-level detection accuracy. A dynamic inverse decoupling control equation for the coupling of multiple physical quantities (light, heat, geometry, radio frequency) is established to achieve collaborative correction of complex deviations. An innovative reinforcement learning-driven closed-loop evolution mechanism is introduced, endowing the system with the ability to continuously learn and self-optimize. The system can achieve high-precision electronic tag affixing in complex environments while ensuring that the RFID readability is effectively improved (e.g., exceeding 99.5%).

[0077] Example 2 This invention also provides an image processing-based electronic tag pasting detection system, which applies the image processing-based electronic tag pasting detection method described in Embodiment 1. The system specifically includes: The image acquisition module uses a camera array to acquire image data from electronic tags at high speed and generate a multimodal dataset of electronic tags. The image preprocessing module preprocesses and fuses features in the electronic tag multimodal dataset to generate an electronic tag feature fusion dataset, thereby improving image quality and removing background interference.

[0078] The feature extraction and matching module uses a deep learning method based on the EfficientNetB7 backbone network and feature pyramids (such as UNet++) to perform multi-scale feature extraction, decoupling and enhancement of electronic tag features in the electronic tag feature fusion dataset, generate a real-time feature tensor, and dynamically match the real-time feature tensor with a predefined multimodal standard template library.

[0079] The deviation correction module uses image analysis algorithms to calculate the deviation amount for detected deviations in the position, angle, and shape of electronic tags, and then corrects or issues an alarm.

[0080] The real-time feedback and control module connects the image processing results from the deviation correction module to the production line's control system, automatically adjusting the electronic tag pasting parameters during production to achieve closed-loop control.

[0081] This system uses multimodal sensors to acquire polarization optical images, near-infrared images, temperature information, and 3D point cloud information of the tags, enabling high-precision detection and positioning. Subsequently, through preprocessing, feature fusion, and dynamic matching, combined with a closed-loop control strategy, deviation correction is performed to ensure the adhesion quality of the electronic tags and the readability of RFID.

[0082] The specific implementation methods of the functions of the above modules are the same as those of the image processing-based electronic tag pasting detection method in Embodiment 1, and will not be repeated here.

[0083] In the description of this specification, the references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0084] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for detecting the attachment of electronic tags based on image processing, characterized in that, Includes the following steps: S1. The production line triggers an electronic tag acquisition command. The camera array acquires polarization images, near-infrared images, temperature data and / or 3D point cloud data of the electronic tags according to the acquisition command, generates a multimodal dataset of electronic tags, and transmits it to the processing industrial control computer. S2. The industrial control computer preprocesses and fuses the features of the electronic tag multimodal dataset to generate an electronic tag feature fusion dataset. S3. The industrial control computer performs multi-scale deep feature extraction and dynamic matching on the electronic tag feature fusion dataset to generate an electronic tag feature matching dataset. S4. The industrial control computer generates deviation correction instructions based on the electronic tag feature matching dataset, sends the deviation correction instructions to the PLC for deviation correction, generates electronic tag quality data and uploads it to the MES system. The S5 and MES systems analyze and learn from the quality data of electronic tags, and feed it back to the industrial control computer for dynamic optimization of production line parameters and adjustment of control strategies.

2. The electronic tag pasting detection method based on image processing according to claim 1, characterized in that, Before or after the production line triggers the electronic tag collection command in step S1, real-time environmental status detection is also included. Through data processing of sensors or camera front end, the current environmental status is analyzed in real time; based on the collected real-time environmental status characteristics, it is determined in real time which cameras in the camera array need to be activated.

3. The electronic tag pasting detection method based on image processing according to claim 2, characterized in that, The current environmental conditions include metal surfaces, liquid containers, temperature changes, and curved surfaces.

4. The electronic tag pasting detection method based on image processing according to claim 3, characterized in that, Based on the collected real-time environmental characteristics, it is determined in real time which cameras in the camera array need to be activated, including: The polarization camera is activated when there are metal surfaces in the environment or when metal items are detected on the production line. When a liquid container or liquid environment is detected, the near-infrared camera is triggered to acquire images, penetrate the container, and read tag information; The thermal imaging camera is activated when a change in real-time temperature data or temperature data detected by a sensor is detected. When a curved surface feature is detected at the label location, or when it is identified that the label is attached to a complex three-dimensional surface, a 3D structured light camera is activated to detect the three-dimensional fit of the label.

5. The electronic tag pasting detection method based on image processing according to claim 1, characterized in that, The preprocessing in step S2 includes: Metal surface processing is performed on polarized images: Images with different polarization angles are obtained by a polarized high-definition camera, and the polarization difference method is used to process the images with different polarization angles to remove reflection noise from the metal surface; Perform liquid environment enhancement processing on near-infrared images: perform image enhancement processing such as contrast stretching, histogram equalization, or descattering on near-infrared images; Perform temperature deformation compensation on temperature data: Based on the temperature data collected by the thermal imaging camera and the known thermal expansion coefficient of the electronic tag material, calculate the thermal deformation of the tag's geometric dimensions or position caused by temperature changes. For the thermal deformation of different materials, apply image scaling or affine transformation to spatially compensate the affected image or 3D point cloud data. Perform surface reconstruction on 3D point cloud data: Combine point cloud density and curvature analysis to perform multi-level deep data analysis and detect whether the labels accurately fit the surface.

6. The electronic tag pasting detection method based on image processing according to claim 5, characterized in that, The feature fusion includes: Step 1, Dynamic Data Alignment and Calibration: Align all modal data acquired by the camera enabled in Step S1 to a common coordinate system to generate dynamic data alignment and calibration multimodal data; Step 2: Based on the dynamically aligned and calibrated multimodal data, perform feature extraction and representation to generate a series of feature maps or feature vectors with different dimensions but spatial alignment; Step 3: Perform dynamic and intelligent multimodal feature fusion on the series of feature maps or feature vectors with different dimensions but spatial alignment to generate an electronic tag feature fusion dataset.

7. The electronic tag pasting detection method based on image processing according to claim 6, characterized in that, The fusion strategy for dynamic and intelligent multimodal feature fusion includes: Step 31: Early feature complementarity fusion to capture low-level general correlations: The series of feature maps or feature vectors with different dimensions but spatial alignment in Step 2 are concatenated or operated on element-wise according to the channel dimension to capture the complementary information of different modalities at the low level and form a joint core fusion feature map; For the modal data obtained by the camera that was not enabled in Step S1, their respective features are generated by independent, pre-trained and deepened neural network encoders to generate other independently extracted high-level modal features. Step 32, Intermediate Layer Adaptive Fusion, Learning Advanced Dynamic Associations: The core fusion feature map and other unique features are combined... The high-level modal features extracted are input into the attention mechanism encoder model to learn the complex correlation between different modal features, and the fusion weights are dynamically adjusted according to the modalities actually enabled in step S1 to generate a global fusion feature tensor. Step 33: Decision layer adaptation and task-specific branch, task-customized output: The global fusion feature tensor is connected as input to independent head networks for different detection and analysis tasks, and the results are output independently for each task branch.

8. The electronic tag pasting detection method based on image processing according to claim 7, characterized in that, The multi-scale depth feature extraction in step S3 includes the following steps: S31. Use the "global fusion feature tensor" output in step S2 as the starting point for multi-scale deep feature extraction; S32. Introduce a deep learning method based on EfficientNetB7 backbone network and feature pyramid to extract features at multiple scales, capturing label information from different scales including P2, P3, P4 and P5 layers. S33. Based on the multi-scale feature pyramid, feature decoupling and enhancement are performed on the label information captured at different scales to generate real-time feature tensors.

9. The electronic tag pasting detection method based on image processing according to claim 8, characterized in that, The dynamic matching compares the real-time feature tensor generated in the multi-scale deep feature extraction stage with the ideal label features pre-stored in the multi-modal template library. It uses a conditional attention mechanism to intelligently allocate weights according to the availability and importance of different modalities, calculates the similarity or difference between the real-time label features and the template features, and then uses deformation robustness enhancement to compensate for thermal deformation.

10. An electronic tag affixing detection system based on image processing, characterized in that, The system, employing the image processing-based electronic tag affixing detection method as described in any one of claims 1 to 9, comprises: The image acquisition module uses a camera array to acquire image data from electronic tags at high speed and generate a multimodal dataset of electronic tags. The image preprocessing module preprocesses and fuses features in the electronic tag multimodal dataset to generate an electronic tag feature fusion dataset. The feature extraction and matching module uses a deep learning method based on the EfficientNetB7 backbone network and feature pyramid to extract, decouple, and enhance the electronic tag features of the electronic tag feature fusion dataset at multiple scales, generate a real-time feature tensor, and dynamically match the real-time feature tensor with a predefined multimodal standard template library. The deviation correction module uses image analysis algorithms to calculate the deviation amount for detected deviations in the position, angle, and shape of electronic tags, and then corrects or issues an alarm. The real-time feedback and control module connects the image processing results of the deviation correction module with the production line control system to automatically adjust the electronic tag pasting parameters during the production process, thereby achieving closed-loop control.

Images