Intelligent label printing optimization system, method and device based on image recognition

Abstract

The invention relates to the technical field of intelligent printing and industrial automation control, and discloses an intelligent label printing optimization system, method and device based on image recognition. The method comprises the following steps: constructing an object surface feature matrix according to surface three-dimensional point cloud data and texture information of a to-be-labeled object; performing regional adaptability evaluation on the object surface feature matrix to obtain a thermodynamic map so as to determine a target labeling region; obtaining semantic information corresponding to preset label contents to generate typesetting constraint conditions corresponding to the label contents; generating an optimized typesetting scheme corresponding to the label content according to the typesetting constraint condition, and controlling preset printing equipment to print according to the optimized typesetting scheme to form a target label; and according to the first pose information corresponding to the object to be labeled, the preset second pose information of the printing equipment and the area range corresponding to the target area, a preset movement track corresponding to the mechanical arm is generated, and the mechanical arm is controlled to paste the target label to the target area according to the movement track.

Description

Technical Field

[0001] The present application relates to the field of intelligent printing and industrial automation control technology, and in particular to an intelligent label printing optimization system, method and device based on image recognition. Background Art

[0002] With the rapid development of industrial automation technology, intelligent label printing systems have been widely used in intelligent manufacturing, logistics management and other fields. Traditional label printing technology mainly relies on manual experience to select labeling locations, and has the following technical bottlenecks:

[0003] Poor adaptability to complex surfaces: Conventional monocular vision systems cannot accurately capture changes in surface curvature and material reflectivity, resulting in problems such as label warping and falling off on concave and convex surfaces or reflective materials;

[0004] Static solidification of typesetting rules: Existing label generation systems mostly use fixed template libraries, without considering the importance differences of semantic information and the dynamic matching of physical constraints of the labelable area, resulting in the obstruction of key information or imbalance of character proportions;

[0005] Insufficient mechanical control accuracy: Traditional robot arm path planning algorithms have difficulty maintaining the normal fit of the end effector on uneven surfaces, which can easily cause bubbles or wrinkles.

[0006] Therefore, there is an urgent need for an intelligent label printing optimization system, method and device based on image recognition to solve at least one of the above problems. Summary of the invention

[0007] The present application provides an intelligent label printing optimization system, method and device based on image recognition, aiming to solve the problems that traditional label printing technology mainly relies on manual experience to select labeling positions, has poor adaptability to complex surfaces, static solidification of typesetting rules and insufficient mechanical control accuracy.

[0008] In a first aspect, the present application provides a smart label printing optimization method based on image recognition, comprising:

[0009] The surface three-dimensional point cloud data and texture information of the object to be labeled are obtained according to the preset multispectral image acquisition device, so as to construct the surface feature matrix of the object including the spatial curvature features and material properties;

[0010] Performing a regional adaptability evaluation on the surface feature matrix of the object to obtain a thermal map including a probability distribution of labelable regions, so as to determine a target labeling region according to a maximum response value corresponding to the thermal map;

[0011] Acquire semantic information corresponding to preset label content, calculate semantic element weights corresponding to the semantic information, and generate typesetting constraints corresponding to the label content according to the area range corresponding to the target labeling area, the semantic element weights and the semantic information; the typesetting constraints at least include a character scaling ratio threshold, information level visualization rules, label image size parameters and material attachment stability parameters;

[0012] Generate an optimized typesetting scheme corresponding to the label content according to the typesetting constraint conditions, and control a preset printing device to print according to the optimized typesetting scheme to form a target label;

[0013] Based on the non-uniform rational B-spline curve fitting algorithm, a motion trajectory corresponding to a preset robotic arm is generated according to the first pose information corresponding to the object to be labeled, the second pose information of the preset printing device, and the area range corresponding to the target area, so as to control the robotic arm to paste the target label to the target area according to the motion trajectory.

[0014] In some embodiments, the surface three-dimensional point cloud data and texture information of the object to be labeled are obtained according to a preset multispectral image acquisition device, and are used to construct an object surface feature matrix including spatial curvature features and material properties, including: inputting the texture information into a preset parallel convolutional neural network to extract texture feature maps of multiple spectral channels; inputting the surface three-dimensional point cloud data into a preset graph convolutional network to perform spatial curvature modeling on the three-dimensional point cloud data, and outputting spatial curvature features including spectral reflectance and material properties; splicing the texture feature map with the spatial curvature features into feature tensors, performing dimensionality reduction processing on the spliced ​​feature tensors through a differentiable pooling layer, and generating an object surface feature matrix with local geometry preservation.

[0015] In some embodiments, the regional adaptability evaluation of the object surface feature matrix is ​​performed to obtain a heat map including a probability distribution of labelable areas, including: constructing a deep regional segmentation network; the deep regional segmentation network adopts a codec structure of a U-Net architecture, and introduces a channel attention mechanism in the encoding stage to weight the importance of the object surface feature matrix; embeds a deformable convolution module in the decoding stage to dynamically adjust the receptive field to adapt to the morphological characteristics of different curvature areas; inputs the object surface feature matrix into the deep regional segmentation network, and outputs a probability distribution map including the labeling suitability of each pixel; and eliminates discrete noise points by processing the probability distribution map according to a morphological closing operation to generate a heat map with regional connectivity.

[0016] In some embodiments, determining the target labeling area according to the maximum response value corresponding to the thermal map includes: constructing an objective function according to the maximum area of ​​the target labeling area, the minimum principal curvature variance and the maximum edge sharpness index; optimizing the thermal map according to a preset objective function and a multi-objective optimization algorithm to obtain multiple candidate labeling areas; performing non-dominated sorting on the multiple candidate labeling areas according to the NSGA-II algorithm, and calculating the weight coefficient corresponding to each sorted candidate labeling area according to the entropy weight method; among the multiple candidate labeling areas, obtaining a continuous area whose weight coefficient is greater than a preset threshold as the target labeling area.

[0017] In some embodiments, the calculating of the semantic element weight corresponding to the semantic information to generate the layout constraint conditions corresponding to the label content according to the area range, semantic element weight and semantic information corresponding to the target labeling area includes: extracting the semantic dependency tree corresponding to the semantic information according to a preset BERT model; calculating the importance score of each semantic node in the semantic dependency tree as the semantic element weight based on a preset graph attention network; converting the area range parameters into two-dimensional packing constraints and mapping the semantic element weights into weighting coefficients of information level visualization rules; the area range parameters include one or more of geometric size parameters, curvature feature parameters, edge feature parameters, material attribute parameters, spatial position and direction parameters, topological parameters and stability parameters; obtaining the layout constraint conditions corresponding to the label content according to the two-dimensional packing constraints and weighting coefficients based on the Lagrange multiplier method.

[0018] In some embodiments, generating an optimized typesetting scheme corresponding to the label content according to the typesetting constraints includes: constructing an intelligent typesetting architecture based on a generative adversarial network; the generator of the generative adversarial network adopts a multi-scale feature fusion network with a self-attention mechanism, and the discriminator of the generative adversarial network introduces a differentiable rendering module to simulate the actual printing effect; optimizing the generator parameters of the generative adversarial network based on reinforcement learning, and generating a reward function corresponding to the reinforcement learning according to information entropy, aesthetic evaluation points and material matching; generating a style transfer vector according to preset visual specifications and typesetting constraints to embed the style transfer vector into the latent space corresponding to the intelligent typesetting architecture; inputting the label content into the intelligent typesetting architecture to generate the optimized typesetting scheme.

[0019] In some embodiments, based on the non-uniform rational B-spline curve fitting algorithm, a motion trajectory corresponding to a preset robotic arm is generated according to the first pose information corresponding to the object to be labeled, the second pose information of a preset printing device, and the area range corresponding to the target area, including: constructing a kinematic model based on the Lie group space for converting the first pose information and the second pose information into a rigid body transformation matrix in a preset space; performing path planning according to the kinematic model and the rigid body transformation matrix to generate an initial motion trajectory; wherein, a curvature continuity constraint of the non-uniform rational B-spline curve is introduced into the configuration space corresponding to the initial motion trajectory; simulating and calculating the dynamic obstacle avoidance information of the initial motion trajectory according to a preset physical engine to optimize the initial motion trajectory according to the dynamic obstacle avoidance information to generate the motion trajectory; wherein, the dynamic obstacle avoidance information is calculated according to the collision probability, the kinetic energy of the obstacle, and the path energy consumption corresponding to the initial motion trajectory.

[0020] In some embodiments, the step of controlling the robotic arm to paste the target label onto the target area according to the motion trajectory includes: obtaining the torque feedback information of the end effector corresponding to the robotic arm, and adjusting the dynamic parameters of the robotic arm according to the torque feedback information; monitoring the labeling process of the robotic arm in real time according to a preset online vision feedback device, and when it is detected that the object to be labeled has a pose offset and / or surface abnormal deformation, performing real-time optimization on the motion trajectory, so as to control the robotic arm to complete the labeling process according to the dynamically optimized dynamic parameters and motion trajectory.

[0021] In a second aspect, the present application provides an intelligent label printing optimization system based on image recognition, including:

[0022] A multi-spectral image acquisition device for obtaining the surface three-dimensional point cloud data and texture information of the object to be labeled;

[0023] A printing device for printing the target label;

[0024] A robotic arm for pasting the target label on the object to be labeled;

[0025] A control device is used to construct an object surface feature matrix including spatial curvature features and material attributes according to the surface three-dimensional point cloud data and texture information; perform regional adaptability evaluation on the object surface feature matrix, obtain a thermal map including a probability distribution of a labelable area, and determine a target labeling area according to a maximum response value corresponding to the thermal map; obtain semantic information corresponding to a preset label content, calculate a semantic element weight corresponding to the semantic information, and generate a typesetting constraint condition corresponding to the label content according to an area range, a semantic element weight, and the semantic information corresponding to the target labeling area; the typesetting constraint condition at least includes a character scaling ratio threshold, an information level visualization rule, a label image size parameter, and a material attachment stability parameter; generate an optimized typesetting scheme corresponding to the label content according to the typesetting constraint condition, and control a preset printing device to print according to the optimized typesetting scheme to form a target label; based on a non-uniform rational B-spline curve fitting algorithm, generate a motion trajectory corresponding to a preset mechanical arm according to the first pose information corresponding to the object to be labeled, the second pose information of the preset printing device, and the area range corresponding to the target area, and control the mechanical arm to paste the target label to the target area according to the motion trajectory.

[0026] In a third aspect, the present application provides a smart label printing optimization device based on image recognition, comprising:

[0027] A matrix construction module is used to obtain the surface three-dimensional point cloud data and texture information of the object to be labeled according to a preset multispectral image acquisition device, and to construct an object surface feature matrix including spatial curvature features and material properties;

[0028] A region determination module is used to perform a regional adaptability evaluation on the surface feature matrix of the object, obtain a thermal map including a probability distribution of labelable regions, and determine a target labeling region according to a maximum response value corresponding to the thermal map;

[0029] An information acquisition module is used to acquire semantic information corresponding to preset label content, calculate semantic element weights corresponding to the semantic information, and generate typesetting constraints corresponding to the label content according to the area range corresponding to the target labeling area, the semantic element weights and the semantic information; the typesetting constraints at least include a character scaling ratio threshold, information level visualization rules, label image size parameters and material attachment stability parameters;

[0030] A scheme generating module, used for generating an optimized typesetting scheme corresponding to the label content according to the typesetting constraint conditions, so as to control a preset printing device to print according to the optimized typesetting scheme to form a target label;

[0031] The labeling completion module is used to generate a preset motion trajectory corresponding to the robotic arm based on the non-uniform rational B-spline curve fitting algorithm according to the first posture information corresponding to the object to be labeled, the second posture information of the preset printing device and the area range corresponding to the target area, so as to control the robotic arm to stick the target label to the target area according to the motion trajectory.

[0032] The present application provides an intelligent label printing optimization system, method and device based on image recognition. The three-dimensional point cloud data and texture information of the object to be labeled are obtained through a multispectral image acquisition device, and a surface feature matrix containing spatial curvature features and material properties is constructed. Based on the matrix, regional adaptability evaluation is performed, a probability distribution heat map of the labelable area is generated, and the optimal target labeling area is dynamically determined by the maximum response value. The fusion of multi-dimensional data (curvature, material) improves the recognition accuracy of complex surfaces (such as curved surfaces and multi-material splicing surfaces) and avoids subjective errors of manual experience.

[0033] By analyzing the semantic information of the label content, calculating the weight of semantic elements (such as key words and graphic priorities), and combining the size, shape and material stability parameters (such as friction coefficient and adhesion) of the target area, a layout solution including character scaling, information hierarchy and size constraints is generated. Dynamic adaptation of label content and labeling area is achieved to ensure information readability and label attachment reliability.

[0034] The non-uniform rational B-spline (NURBS) curve fitting algorithm is used to generate a smooth motion trajectory of the robot arm by combining the object posture, printing device posture and target area range. This solves the positioning deviation problem caused by path mutation in traditional mechanical control, and is especially suitable for high-precision fitting of curved or special-shaped surfaces.

[0035] Through three-dimensional feature modeling and thermal map analysis, the optimal labeling position of irregular surfaces (such as concave and convex surfaces, multi-material mixed surfaces) can be automatically identified, overcoming the poor adaptability of traditional methods due to reliance on manual experience. By integrating semantic weights and physical constraints (such as material attachment stability), adaptive scaling and layout optimization of label content can be achieved to avoid missing or deformed label information due to inconsistent area size or shape under static rules. The entire process from data collection, area selection, layout generation to robotic arm labeling is automated to reduce manual intervention; through NURBS trajectory planning and dual-pose collaborative control, the labeling position error is controlled at the millimeter level to meet the needs of precision manufacturing scenarios. It can be applied to scenes that are difficult to handle with traditional methods, such as curved packaging, electronic component labeling, and medical device identification. It is especially suitable for complex working conditions with flexible materials, micro devices, or multi-label collaborative layout.

[0036] Compared with traditional label printing technology, this method solves the three core problems of strong dependence on manual experience, poor generalization of static rules, and insufficient mechanical positioning accuracy through full-link optimization of "3D feature modeling-semantic dynamic typesetting-high-precision motion control", and significantly improves the efficiency, quality and scene coverage of label printing.

[0037] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 It is a schematic block diagram of the structure of a smart label printing optimization system provided by an embodiment of the present application;

[0040] Figure 2 is a schematic flow chart of the steps of a smart label printing optimization method based on image recognition provided by an embodiment of the present application;

[0041] Figure 3 It is a schematic block diagram of the structure of a control device provided in one embodiment of the present application.

[0042] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0044] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0045] It should be understood that, in order to facilitate the clear description of the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, the words "first", "second", etc. are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art can understand that the words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not necessarily limit the difference.

[0046] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0047] It should also be understood that the term “and / or” used in the specification and appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0048] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0049] With the rapid development of industrial automation technology, intelligent label printing systems have been widely used in intelligent manufacturing, logistics management and other fields. Traditional label printing technology mainly relies on manual experience to select labeling locations, and has the following technical bottlenecks:

[0050] Poor adaptability to complex surfaces: Conventional monocular vision systems cannot accurately capture changes in surface curvature and material reflectivity, resulting in problems such as label warping and falling off on concave and convex surfaces or reflective materials;

[0051] Static layout rules: Existing label generation systems mostly use fixed template libraries, without considering the importance differences of semantic information and the dynamic matching of physical constraints of the labelable area, resulting in the obstruction of key information or imbalance of character proportions;

[0052] Insufficient mechanical control accuracy: Traditional robot arm path planning algorithms have difficulty maintaining the normal fit of the end effector on uneven surfaces, which can easily cause bubbles or wrinkles.

[0053] To solve the above problems, please refer to Figure 1 , the present application provides an intelligent label printing optimization system based on image recognition, comprising: a multispectral image acquisition device for acquiring surface three-dimensional point cloud data and texture information of an object to be labeled;

[0054] A printing device, used for printing a target label;

[0055] A robotic arm, used for attaching the target label to the object to be labeled;

[0056] A control device is used to construct an object surface feature matrix including spatial curvature features and material attributes according to the surface three-dimensional point cloud data and texture information; perform regional adaptability evaluation on the object surface feature matrix, obtain a thermal map including a probability distribution of a labelable area, and determine a target labeling area according to a maximum response value corresponding to the thermal map; obtain semantic information corresponding to a preset label content, calculate a semantic element weight corresponding to the semantic information, and generate a typesetting constraint condition corresponding to the label content according to an area range, a semantic element weight, and the semantic information corresponding to the target labeling area; the typesetting constraint condition at least includes a character scaling ratio threshold, an information level visualization rule, a label image size parameter, and a material attachment stability parameter; generate an optimized typesetting scheme corresponding to the label content according to the typesetting constraint condition, and control a preset printing device to print according to the optimized typesetting scheme to form a target label; based on a non-uniform rational B-spline curve fitting algorithm, generate a motion trajectory corresponding to a preset mechanical arm according to the first pose information corresponding to the object to be labeled, the second pose information of the preset printing device, and the area range corresponding to the target area, and control the mechanical arm to paste the target label to the target area according to the motion trajectory.

[0057] Specifically, multispectral image acquisition equipment uses multispectral imaging technology to obtain three-dimensional point cloud data (including curvature, concave-convex features) and texture information (such as material reflectivity and roughness) on the surface of an object, providing high-precision input for subsequent analysis.

[0058] The control device serves as the computing center, completing the entire process from data analysis to motion planning.

[0059] The printing device prints highly adaptable labels (such as anti-warping materials) according to the optimized typesetting scheme.

[0060] The robotic arm is equipped with a flexible end effector to achieve precise labeling on complex surfaces.

[0061] The surface feature matrix of the object is constructed through point cloud data, and the spatial curvature (such as Gaussian curvature) and material properties (such as friction coefficient and reflectivity) are integrated to quantitatively evaluate the feasibility of labeling in different areas. The convolutional neural network (CNN) is used to evaluate the regional adaptability of the feature matrix and output a probabilistic heat map. The area with the maximum response value is the optimal labeling position (such as avoiding high curvature or reflective areas). The semantic information of the label content (such as product name, barcode) is parsed, and the weights of key elements (such as barcode weight > description text) are extracted through natural language processing (NLP) technology. Combined with the physical constraints (area, curvature) of the target area, the layout rules are generated: character scaling threshold: ensure the minimum readable size; information hierarchy visualization rules: high-weight content is preferentially enlarged in the center; material attachment parameters: adjust the label adhesive type according to the surface roughness. Based on the NURBS (non-uniform rational B-spline) algorithm to fit the motion trajectory of the robot arm, the normal angle and pressure of the end effector are dynamically adjusted to ensure that the label fits on the non-flat surface without bubbles.

[0062] For example, by scanning the surface of an object through a multispectral device, point cloud data with an accuracy of 0.1mm is generated, and RGB and near-infrared textures are collected simultaneously. For example, when scanning a metal tank, the near-infrared band can penetrate the oil on the surface and accurately identify the actual material. The system has achieved a leap from "empirical labeling" to "adaptive labeling" through the collaboration of multimodal data fusion and intelligent algorithms.

[0063] See also Figure 2 , Figure 2 The figure is a schematic flow chart of a method for optimizing smart label printing based on image recognition provided by an embodiment of the present application. The execution device of the method is a control device of a smart label printing optimization system provided by any embodiment of the present application.

[0064] like Figure 2 As shown, the provided method includes steps S101 to S105. The control device may be a handheld terminal, a notebook computer, a wearable device or a robot, etc., for implementing steps S101 to S105 and their corresponding embodiments.

[0065] Step S101. Acquire the surface three-dimensional point cloud data and texture information of the object to be labeled according to the preset multispectral image acquisition device, so as to construct the object surface feature matrix including the spatial curvature features and material properties.

[0066] Specifically, this step uses a multispectral image acquisition device to obtain the three-dimensional point cloud data and texture information of the object to be labeled, and combines the spatial curvature and material properties to build the object surface feature matrix. The three-dimensional point cloud data reflects the geometric shape of the object surface (such as concave and convex, curvature), and the texture information includes color, reflective properties, etc. Multispectral data enhances the recognition ability of different materials (such as metal, plastic). The feature matrix integrates the physical space and material properties, providing a data basis for subsequent regional assessment.

[0067] For example, a laser radar or structured light scanner is used to obtain three-dimensional point cloud data, and a multi-spectral camera is used to collect multi-band texture information such as RGB and infrared. The curvature features in the point cloud data (calculated by local surface fitting) are fused with texture features (such as texture statistics extracted from the gray-level co-occurrence matrix) to form a multidimensional matrix. For example, each point cloud coordinate corresponds to parameters such as curvature value and material reflectivity. The feature matrix is ​​stored in a tensor data structure, and the dimensions include spatial coordinates (X, Y, Z), curvature (C), material type (M), etc.

[0068] Combining geometric and material information can avoid misjudgment of labeling areas caused by single data (such as excluding areas with high curvature but suitable materials for labeling). Multispectral data overcomes the problem of texture distortion under a single light source. For example, overexposure of reflective materials under visible light can be compensated by infrared data.

[0069] Step S102: Perform regional adaptability evaluation on the surface feature matrix of the object to obtain a thermal map including a probability distribution of labelable regions, so as to determine a target labeling region according to a maximum response value corresponding to the thermal map.

[0070] Specifically, the regional adaptability evaluation is performed based on the feature matrix, and the probability distribution of the labeling area is quantified by generating a thermal map, and the optimal labeling position is determined by the maximum response value. The adaptability evaluation includes indicators such as surface flatness, material adhesion, and visibility. The high-probability area in the thermal map meets the preset physical and visual conditions.

[0071] For example, a graph-based superpixel segmentation algorithm (such as the SLIC algorithm) is used to divide the feature matrix into candidate regions. A convolutional neural network (CNN) classifier is trained, and the feature matrix sub-blocks of the candidate regions are input and the labeling suitability score (0-1 probability value) is output. The training data annotation includes manually annotated "suitable / unsuitable" region samples. The scores of each region are mapped to color intensity, a heat map is generated, and the maximum response region is selected as the target region through non-maximum suppression (NMS).

[0072] Through machine learning, the evaluation criteria are dynamically optimized to adapt to different object types (such as curved containers and flat packaging). The thermal map intuitively locates the optimal area, reducing the time cost of traditional manual trial and error.

[0073] Step S103. Obtain semantic information corresponding to preset label content, calculate semantic element weights corresponding to the semantic information, and generate typesetting constraints corresponding to the label content according to the area range, semantic element weights and semantic information corresponding to the target labeling area; the typesetting constraints include at least character scaling ratio thresholds, information level visualization rules, label image size parameters and material attachment stability parameters.

[0074] Specifically, the semantic information of the label content (such as text and icons) is parsed, the weight of the semantic elements is calculated, and the layout rules are generated in combination with the geometric constraints of the target area. For example, important information (such as the shelf life) needs to be displayed in a magnified manner, and icons need to avoid deformation caused by crossing high curvature areas.

[0075] For example, NLP technology is used to extract text keywords (such as "dangerous" and "important"), and icons are identified by pre-trained target detection models (such as YOLO). The text importance weight is calculated based on the TF-IDF algorithm, and the icon weight is based on preset rules (such as safety sign weight = 0.9).

[0076] The scaling factor is calculated based on the minimum bounding rectangle size and weight of the area. The adhesion stability is adjusted by material properties (such as the affinity coefficient between the adhesive and the material) to adjust the upper limit of the label size.

[0077] The weight drives the highlighting of key information to avoid information loss due to scaling or deformation of label content. Combined with material stability parameters (such as stickiness threshold) to prevent labels from falling off.

[0078] Step S104: Generate an optimized layout scheme corresponding to the label content according to the layout constraint conditions, and control a preset printing device to print according to the optimized layout scheme to form a target label.

[0079] Specifically, an optimization scheme is generated according to the layout constraints to control the printing device to output labels. The optimization objectives include information completeness, aesthetics and physical feasibility, and a multi-objective optimization algorithm is used to generate a Pareto optimal solution.

[0080] For example, using genetic algorithms or reinforcement learning models, with constraints as boundaries, and information density, alignment, and aesthetic scores (through pre-trained aesthetic evaluation models) as optimization targets to generate layout plans. Convert the layout plan into printing instructions (such as PDF / vector graphics), drive the output of inkjet or laser printers, and adjust printing parameters (such as ink concentration and heating temperature) according to material properties.

[0081] Automatic typesetting replaces manual design and is particularly suitable for large-scale differentiated labels (such as different language versions). Printing accuracy is adjusted based on material parameters to avoid printing blur or penetration problems.

[0082] Step S105. Based on the non-uniform rational B-spline curve fitting algorithm, a motion trajectory corresponding to the preset robotic arm is generated according to the first pose information corresponding to the object to be labeled, the second pose information of the preset printing device, and the area range corresponding to the target area, so as to control the robotic arm to paste the target label to the target area according to the motion trajectory.

[0083] Specifically, the robot arm motion trajectory is generated based on NURBS curve fitting to ensure that the label fits the complex surface accurately. Trajectory planning requires coordinating the relative posture of the robot arm and the object to avoid collision and optimize the path length.

[0084] For example, the object pose (first pose) and the printing device export pose (second pose) are acquired through visual sensors (such as binocular cameras), and the coordinate system transformation matrix is ​​established. The boundary points of the target area are used as control points, and the NURBS algorithm is used to generate a smooth trajectory to meet the acceleration constraints of the robot joints. The trajectory is discretized into a time-position-speed sequence, and the robot is driven by a PID controller, and the path is adjusted with real-time feedback.

[0085] NURBS's high degree of freedom curves adapt to irregular surfaces and avoid label warping caused by traditional straight line paths. Trajectory optimization reduces the robot's idle movement and increases labeling speed (such as reducing cycle time by 30%).

[0086] This method realizes the automation of the entire process from data collection to label attachment through multispectral data fusion, dynamic area assessment, semantic-driven typesetting and high-precision motion control, solving the problems of inaccurate position selection, easy label shedding and difficulty in labeling on curved surfaces in traditional methods. It is particularly suitable for high-demand labeling tasks on complex materials and curved surfaces in industrial scenarios.

[0087] In some embodiments, the surface three-dimensional point cloud data and texture information of the object to be labeled are obtained according to a preset multispectral image acquisition device, and are used to construct an object surface feature matrix including spatial curvature features and material properties, including: inputting the texture information into a preset parallel convolutional neural network to extract texture feature maps of multiple spectral channels; inputting the surface three-dimensional point cloud data into a preset graph convolutional network to perform spatial curvature modeling on the three-dimensional point cloud data, and outputting spatial curvature features including spectral reflectance and material properties; splicing the texture feature map with the spatial curvature features into feature tensors, performing dimensionality reduction processing on the spliced ​​feature tensors through a differentiable pooling layer, and generating an object surface feature matrix with local geometry preservation.

[0088] Multi-band texture images such as RGB, near infrared (NIR), and short wave infrared (SWIR) are collected by a multispectral camera. An independent convolution branch is designed for each spectral channel (such as RGB, NIR). The network structure is a ResNet-18 variant. Each branch contains 5 residual blocks and the number of output channels is 256. The feature maps output by each branch are weighted fused through the channel attention mechanism (SE module) to generate a multispectral texture feature map (size H×W×256).

[0089] For example, for metal surfaces, the NIR channel can effectively capture the texture of the oxide layer, while the RGB channel may fail due to reflection interference. Channel attention automatically enhances the NIR feature weight. 3D point cloud data (N points, each point contains XYZ coordinates).

[0090] The graph convolutional network (GCN) constructs the point cloud as a K-nearest neighbor graph (K=20), with each point as a graph node and edge weights calculated by Euclidean distance. Through the hierarchical aggregation of GCN, the principal curvature (maximum curvature C_max, minimum curvature C_min) and average curvature (C_avg) of the local surface are extracted. A fully connected layer is added at the end of the GCN to classify the material type (such as metal, plastic, glass) according to the reflectivity of the point cloud (mapped by multispectral data). The spatial curvature feature vector (C_max, C_min, C_avg) and material category code (one-hot vector) of each point are output. The texture feature map (H×W×256) and the spatial curvature feature (N×6, including curvature value and material code) are tensor-aligned, including using the external parameter matrix of the multispectral camera and 3D scanner to map the texture image pixels and point cloud coordinates to the same coordinate system. For each 3D point, its corresponding texture feature (256 dimensions) and curvature / material feature (6 dimensions) are associated to generate a 262-dimensional local feature vector. Use the maximum-average hybrid pooling layer to reduce the dimension of the local feature vector to 64 dimensions, output the reduced dimension object surface feature matrix (N×64), and retain the key geometry and material information.

[0091] The parallel CNN branch extracts complementary texture information according to different spectral characteristics (such as NIR suppressing reflections and SWIR penetrating surface stains). Combined with the curvature modeling of GCN, it can still stably generate feature matrices on the surfaces of complex materials (such as frosted metal and transparent plastic).

[0092] For example, for transparent bottles, the RGB channel cannot effectively capture the texture due to interference from transmitted light, but the SWIR channel can penetrate the surface of the bottle and accurately divide the labelable area based on the curvature data (high curvature area at the bottle mouth).

[0093] GCN aggregates the curvature of neighborhood points through graph structures, avoiding local fitting errors in traditional ICP algorithms, and reducing the curvature calculation error to ±0.02mm⁻¹ (experimental data). Differentiable pooling dynamically balances the maximum response (highlighting key features) and the average response (retaining global information), reducing feature reconstruction errors by 18% compared to traditional PCA dimensionality reduction.

[0094] In some embodiments, the regional adaptability evaluation of the object surface feature matrix is ​​performed to obtain a heat map including a probability distribution of labelable areas, including: constructing a deep regional segmentation network; the deep regional segmentation network adopts a codec structure of a U-Net architecture, and introduces a channel attention mechanism in the encoding stage to weight the importance of the object surface feature matrix; embeds a deformable convolution module in the decoding stage to dynamically adjust the receptive field to adapt to the morphological characteristics of different curvature areas; inputs the object surface feature matrix into the deep regional segmentation network, and outputs a probability distribution map including the labeling suitability of each pixel; and eliminates discrete noise points by processing the probability distribution map according to a morphological closing operation to generate a heat map with regional connectivity.

[0095] This embodiment proposes an improved deep region segmentation network for the regional adaptability evaluation in step S102, enhances the adaptability to complex surface features through the channel attention mechanism and the deformable convolution module, and optimizes the connectivity of the heat map in combination with morphological post-processing, thereby solving the problems of fixed receptive field and noise sensitivity of traditional U-Net in surface object segmentation.

[0096] The encoder is based on the encoding structure of U-Net, which includes 4 downsampling stages, each of which consists of 2 convolutional layers + channel attention module (SE Block), and the number of output channels is 64, 128, 256, and 512 respectively. The decoder corresponds to 4 upsampling stages, and each stage uses a deformable convolution module (Deformable Convolution v2) instead of a conventional convolution. The convolution kernel size is 3×3, and the offset is predicted by an additional convolution layer. The encoder and decoder feature maps are weighted fused by channel attention. The loss function is jointly optimized by Dice loss + Focal Loss to balance positive and negative samples (the proportion of labelable areas is usually less than 10%). A random affine transformation (rotation ±15°, scaling 0.8-1.2 times) and Gaussian noise (σ=0.05) are applied to the feature matrix.

[0097] Morphological post-processing includes: Binarization: Binarize the probability distribution map (0-1) with a threshold of 0.5 to generate an initial mask. Closing operation: Use a 3×3 elliptical structure kernel, first expand and then erode, fill holes and smooth edges. Connected domain screening: Retain connected areas with an area greater than 50 pixels to generate the final thermal map.

[0098] By adaptively adjusting the convolution kernel sampling position, the convolution kernel in the decoder stage is made to fit the geometric shape of high curvature areas (such as the side of a cylinder), and the segmentation intersection over union (IoU) is improved by 12% (experimental data: from 0.78 to 0.87).

[0099] For example, for automotive curved parts, the traditional U-Net mistakenly divides the continuous curved surface into multiple discrete regions due to the fixed convolution kernel. This embodiment maintains the regional continuity through deformable convolution.

[0100] In some embodiments, determining the target labeling area according to the maximum response value corresponding to the thermal map includes: constructing an objective function according to the maximum area of ​​the target labeling area, the minimum principal curvature variance and the maximum edge sharpness index; optimizing the thermal map according to a preset objective function and a multi-objective optimization algorithm to obtain multiple candidate labeling areas; performing non-dominated sorting on the multiple candidate labeling areas according to the NSGA-II algorithm, and calculating the weight coefficient corresponding to each sorted candidate labeling area according to the entropy weight method; among the multiple candidate labeling areas, obtaining a continuous area whose weight coefficient is greater than a preset threshold as the target labeling area.

[0101] This embodiment proposes a candidate area screening method based on multi-objective optimization and NSGA-II algorithm for the target labeling area selection process in step S102, which solves the local optimal problem caused by the traditional single threshold method by balancing indicators such as area, curvature uniformity, and edge clarity.

[0102] Ensure that the label size is sufficient by maximizing the area of ​​the candidate region. Minimize the variance of the principal curvature in the region to ensure a smooth label surface. Maximize the mean of the edge gradient to ensure clear label boundaries. The candidate region must be a connected domain in the heat map and its area must be ≥ the preset minimum label size.

[0103] The NSGA-II optimization process includes: Population initialization: 100 connected domains are randomly selected from the heat map as the initial population. Non-dominated sorting: The population is stratified by Pareto frontier according to the objective function value. Crowding calculation: The crowding distance of individuals in the same frontier layer is calculated to retain diversity. Selection, crossover, and mutation: The tournament selects parent individuals. Simulate binary crossover (SBX) to generate offspring. Polynomial mutation (mutation rate 0.1) increases diversity. Iterative optimization: Run 50 generations and output the Pareto optimal solution set (about 20 candidate regions).

[0104] Weight calculation includes: normalizing the target value, performing min-max normalization on the area, principal curvature variance and edge gradient mean of each candidate area, calculating information entropy and assigning weights, such as the weights of area, principal curvature variance and edge gradient are 0.4, 0.35 and 0.25 respectively.

[0105] The area, curvature, and edge clarity are considered simultaneously to avoid the deviation of a single indicator (e.g., the maximum area may select a high curvature area).

[0106] In some embodiments, the calculating of the semantic element weight corresponding to the semantic information to generate the layout constraint conditions corresponding to the label content according to the area range, semantic element weight and semantic information corresponding to the target labeling area includes: extracting the semantic dependency tree corresponding to the semantic information according to a preset BERT model; calculating the importance score of each semantic node in the semantic dependency tree as the semantic element weight based on a preset graph attention network; converting the area range parameters into two-dimensional packing constraints and mapping the semantic element weights into weighting coefficients of information level visualization rules; the area range parameters include one or more of geometric size parameters, curvature feature parameters, edge feature parameters, material attribute parameters, spatial position and direction parameters, topological parameters and stability parameters; obtaining the layout constraint conditions corresponding to the label content according to the two-dimensional packing constraints and weighting coefficients based on the Lagrange multiplier method.

[0107] This embodiment proposes an automatic weight allocation method based on semantic dependency analysis and graph attention network (GAT) for the calculation of semantic element weights and the generation of layout constraints in step S103. It combines the Lagrange multiplier method to integrate multi-dimensional constraints into a solvable optimization problem, thereby solving the problem that traditional layout rules rely on manual experience and are difficult to quantify semantic importance.

[0108] Input data such as label text (such as "Dangerous goods: flammable liquids, storage temperature ≤30℃"). Use the pre-trained BERT-base model, fine-tune on the label text dataset (100,000 industrial labels), and output a semantic dependency tree. By parsing the sentence trunk (such as "Dangerous goods" as the root node, "flammable liquids" as the attribute child node, and "temperature ≤30℃" as the conditional child node). Example output: ROOT("Dangerous goods")→ ATTRIBUTE("flammable liquids")→CONDITION("temperature ≤30℃") .

[0109] The graph attention network (GAT) uses dependency tree nodes as graph nodes and dependency relationships as edges. A two-layer GAT is designed with 4 heads in each layer, and the weight of each node is output (such as the root node weight 0.6, the attribute node 0.3, and the conditional node 0.1).

[0110] The area range parameters mainly include the following key parameters, which are used to quantify the physical characteristics and spatial limitations of the target labeling area to guide label layout design:

[0111] 1. Geometric dimension parameters: Area: The surface area of the target region, which determines the maximum expandable size of the label. Boundary dimensions: The length and width of the minimum bounding rectangle (or ellipse) of the region, or the major axis length of an irregular shape, used for two-dimensional bin packing constraints. Shape factor: Such as aspect ratio, compactness, which affects the layout adaptability of the label.

[0112] 2. Curvature feature parameters: Principal curvature: The maximum and minimum curvature values of the region surface, used to evaluate the flatness of label attachment. Curvature variance: The degree of change in surface curvature (as described in the above embodiments), and areas with high variance may need to be avoided. Mean curvature: The overall degree of bending, which affects the visual readability of the label.

[0113] 3. Edge feature parameters: Edge sharpness index: The clarity of the edge, insufficient sharpness may cause label positioning deviation. Edge continuity: Whether the edge is closed or has breaks, which affects the paste integrity of the label.

[0114] 4. Material property parameters: Surface roughness: Affects the adhesion of the printing material and the durability of the label. Adhesion coefficient: The affinity of the material for glue, which determines whether an additional adhesive layer is required.

[0115] 5. Spatial position and orientation parameters: Region center coordinates: The three-dimensional position on the object surface, used for robotic arm path planning. Normal direction: The orientation of the region surface, which determines the rotation angle of the label (such as parallel / perpendicular to the viewing angle).

[0116] 6. Topological parameters: Connectivity: Whether it is a single connected region, or contains multiple discrete sub-regions. Hole detection: Whether there are non-labelable holes or depressions within the region.

[0117] 7. Stability parameters: Material attachment stability: A predicted parameter that combines surface properties and environmental factors (such as humidity, temperature) to ensure label persistence. In industrial labeling scenarios, converting the region range parameters into two-dimensional bin packing constraints requires combining multi-dimensional requirements such as geometric adaptation, physical properties, and visual readability. The specific steps are as follows:

[0118] 1: Geometric parameter mapping. Define the basic layout boundary: Determine the maximum placeable range of the label through the geometric parameters of the target region: Minimum bounding rectangle (MBR): Input: The boundary dimensions of the target region (length L, width W). Constraint: Label size l ≤ L, w ≤ W; Optimization goal: Maximize the label area utilization rate l×w. Shape factor adaptation: Input: Aspect ratio R = L / W; Constraint: The label aspect ratio r = l / w needs to satisfy |r - R| ≤ ΔR (ΔR is the tolerance threshold). Adjust the label ratio through affine transformation, or divide the content into a multi-column layout.

[0119] 2: Curvature compensation - dynamically adjust label deformation. For curved surface areas, introduce curvature constraints to solve the label fitting deformation problem: Principal curvature compensation: Input: maximum principal curvature kmax and curvature direction. Constraint: character scaling threshold s≤1 / (1+α×kmax ) (α is the material elastic coefficient). Use NURBS curve to fit the label edge and dynamically adjust the character spacing. Curvature variance constraint: Input: curvature variance σ k2 . Constraint: Labels need to avoid high variance areas (σ k2 > threshold). Algorithm: Generate a mask based on the curvature field to restrict label placement to low variance areas.

[0120] 3. Edge alignment and sharpness enhancement: Combine edge sharpness parameters to optimize label alignment accuracy: Edge sharpness constraint input: Edge gradient amplitude: G. Constraint: The offset between the label edge and the region edge d≤β / G (β is the alignment tolerance coefficient). Align the label contour using the Canny edge detection + ICP registration algorithm. Connectivity verification: Input: Region connectivity mark (whether it is a single connected domain). Constraint: If the region is not connected, the label needs to be split into multiple sub-labels, and the sub-label spacing >γ (anti-overlap). Generate sub-label layout points based on region skeleton extraction (Medial Axis).

[0121] 4: Material stability fusion: Convert material properties into physical constraints to ensure the reliability of label attachment: Roughness compensation: Input: Surface roughness Ra. Constraint: Minimum label contact area A min = λ × Ra (λ is the adhesion safety factor). Add label edge anchor points through Delaunay triangulation. Adhesion enhancement: Input: Adhesion coefficient μ. Constraint: Blank width b ≥ μ reserved at the label edge -1 ×Safety margin. Avoid edge lift by generating a circular buffer around the label.

[0122] 5. Multi-objective optimization modeling: Based on the above constraints, a two-dimensional packing optimization model is constructed: the objective function is: max(k1×l×w / (L×W)+k2×sustainability score-k3×deformation penalty); where k1×l×w / (L×W) is the area utilization; weights k1, k2, and k3 are dynamically assigned by semantic element weights (for example, safety warning labels need to increase k2, while decorative labels may need to increase k1), k1 reflects the user's emphasis on space utilization, k3 reflects the tolerance of the control label due to deformation caused by surface fitting, and l and w are the actual length and width of the label. The readability score quantifies the visual clarity of the label content in the target area, and is weighted by font contrast (the color difference between the label and the background (such as calculated by the CIELAB color difference formula)), information hierarchy (the typesetting priority of key information (such as barcodes, warning symbols) (defined by semantic element weights)), and character legibility (the ratio of character height to viewing distance (such as the minimum viewing angle requirement in the ISO standard)). The deformation penalty item is used to penalize the deformation of the label caused by the surface fitting. If the label is attached to a high curvature area (such as the side of a cylinder), the characters may be distorted, and the penalty item needs to be used to limit the label size or force line breaks. If the material has poor elasticity (such as hard plastic), a slight deformation will trigger a high penalty.

[0123] The solution process uses mixed integer programming (MIP) to handle discrete constraints (such as whether characters wrap) and simulated annealing (SA) to optimize continuous variables (such as label rotation angle).

[0124] The target area (length L × width W) can also be converted into a rectangular layout space, and the variables xi and yi are defined as the upper left corner coordinates of the i-th semantic element, and xj and yj are the upper left corner coordinates of the j-th semantic element, with the constraints: xi+wi≤L,yi+hi≤W( ), where wi and hi are element sizes, mapped by weights (e.g., an element with weight 0.6 occupies 60% of the area).

[0125] Construct the objective function to maximize information density and aesthetics: L = ∑wi × log (Ai) + λ∑ (xi-xj) 2 st two-dimensional packing constraint; where Ai is the element area, λ is the aesthetic penalty coefficient, and the optimal layout is solved by KKT conditions. xj represents the x-axis coordinate component of the jth element.

[0126] GAT dynamically assigns weights based on grammatical structures (e.g., "dangerous goods" as a core noun has the highest weight). Compared with statistical methods such as TF-IDF, the accuracy of important element recognition is increased by 25% (F1 value from 0.72 to 0.90). For example, in drug labels, the "taboo" node of "taboo: prohibited for pregnant women" is given a higher weight through GAT, ensuring that the font is enlarged and prominent.

[0127] In some embodiments, generating an optimized typesetting scheme corresponding to the label content according to the typesetting constraints includes: constructing an intelligent typesetting architecture based on a generative adversarial network; the generator of the generative adversarial network adopts a multi-scale feature fusion network with a self-attention mechanism, and the discriminator of the generative adversarial network introduces a differentiable rendering module to simulate the actual printing effect; optimizing the generator parameters of the generative adversarial network based on reinforcement learning, and generating a reward function corresponding to the reinforcement learning according to information entropy, aesthetic evaluation points and material matching; generating a style transfer vector according to preset visual specifications and typesetting constraints to embed the style transfer vector into the latent space corresponding to the intelligent typesetting architecture; inputting the label content into the intelligent typesetting architecture to generate the optimized typesetting scheme.

[0128] This embodiment proposes an intelligent typesetting architecture based on generative adversarial network (GAN) and reinforcement learning (RL) for the generation of optimized typesetting scheme in step S104. It simulates the actual printing effect through differentiable rendering and adapts to diverse visual specifications in combination with style transfer to solve the problems of poor flexibility and strong subjectivity of aesthetic evaluation in traditional template typesetting.

[0129] The generator's multi-scale self-attention network contains 4 resolution branches (256×256 to 32×32), each of which integrates a Transformer self-attention module to generate layout parameters (element position, size, font) through cross-scale feature fusion. The input is the semantic vector of the label text (the weight output by Example 4) plus the style transfer vector (from the visual specification library).

[0130] The differentiable rendering module of the discriminator is used to render the layout parameters output by the generator into a bitmap, simulating the effects of halftone jitter, ink diffusion, etc. of the printer. The discriminator distinguishes between real label images (sampled from industrial libraries) and generated images, and the loss function is the Wasserstein GAN (WGAN) loss.

[0131] By maximizing the information entropy between elements, it encourages balanced content layout. Based on the pre-trained ResNet-50 aesthetic scoring model (fine-tuned on the AVA dataset), it outputs a score of 0-1. Calculate the adhesion stability of the generated layout (e.g., small fonts have low scores on rough materials). Use the PPO (Proximal Policy Optimization) algorithm to update the generator parameters, extract the typographic style (font family, color scheme, icon style) from the corporate visual specification library (e.g., ISO standards, brand VI manuals), and encode it into a 128-dimensional latent vector through VAE. Adjust the generator output according to different specification requirements (e.g., "medical serious style" and "consumer fashion style").

[0132] The discriminator simulates printing effects (such as the texture of toner particles in laser printing) to make the layout output by the generator adapt to the actual printing device. Tests show that the PSNR value between the printed draft and the design draft reaches 38dB (the traditional method is only 32dB).

[0133] In some embodiments, based on a non-uniform rational B-spline curve fitting algorithm, a preset motion trajectory corresponding to a robotic arm is generated according to the first pose information corresponding to the object to be labeled, the second pose information of a preset printing device, and the area range corresponding to the target area, including: constructing a kinematic model based on a Lie group space, for converting the first pose information and the second pose information into a rigid body transformation matrix in a preset space; performing path planning according to the kinematic model and the rigid body transformation matrix to generate an initial motion trajectory; wherein, a curvature continuity constraint of a non-uniform rational B-spline curve is introduced into the configuration space corresponding to the initial motion trajectory; and calculating the dynamic obstacle avoidance information of the initial motion trajectory according to a preset physical engine simulation, so as to optimize the initial motion trajectory according to the dynamic obstacle avoidance information and generate the motion trajectory; wherein, the dynamic obstacle avoidance information is calculated according to the collision probability, obstacle kinetic energy, and path energy consumption corresponding to the initial motion trajectory.

[0134] In this embodiment, a path planning method based on Lie group space kinematics modeling and non-uniform rational B-spline (NURBS) curve optimization is proposed to solve the problem of generating the robot arm motion trajectory in step S105. The method solves the problem of non-smooth trajectory and high collision risk of traditional planning algorithms under complex posture constraints based on improved RRT* algorithm combined with physical engine simulation dynamic obstacle avoidance.

[0135] The positions of the robot base (first position) and the printing device (second position) are represented as T1, T2∈SE(3) Lie group elements, namely:

[0136] ;

[0137] Where R is the rotation matrix and t is the translation vector. The rigid body transformation matrix from the base to the printing device is obtained through Lie group operation. , used for subsequent path planning.

[0138] Improvements to the RRT* extension include the introduction of curvature continuity constraints for NURBS curves when sampling nodes in the configuration space (C-space) to ensure that the paths between nodes meet the robot joint acceleration limits. The curvature constraints include the maximum curvature of the path segment ≤ the maximum joint acceleration / the square of the command velocity.

[0139] Perform NURBS interpolation on the discrete path points generated by RRT* to generate a parameterized trajectory, input the trajectory into the PyBullet physics engine, calculate the collision probability Pc, obstacle kinetic energy Eobs (related to the speed of the moving obstacle) and path energy consumption Epath, and construct the loss function λ1Pc+λ2Eobs+λ3Epath; λ1, λ2 and λ3 are the corresponding weights. Optimize the NURBS control points through gradient descent to minimize the loss function value.

[0140] In some embodiments, controlling the robotic arm to stick the target label to the target area according to the motion trajectory includes: obtaining torque feedback information of the end effector corresponding to the robotic arm, and adjusting the dynamic parameters of the robotic arm according to the torque feedback information; monitoring the labeling process of the robotic arm in real time according to a preset online visual feedback device, and when it is detected that the object to be labeled sends a posture offset and / or abnormal surface deformation, optimizing the motion trajectory in real time, so as to control the robotic arm to complete the labeling process according to the real-time optimized dynamic parameters and motion trajectory.

[0141] This embodiment proposes a closed-loop control strategy based on torque feedback and online visual correction for the dynamic control of the robot arm labeling process in step S105. By adjusting the dynamic parameters and trajectory parameters in real time, the labeling deviation caused by the object posture offset and surface deformation is solved. According to the six-dimensional force / torque sensor data of the end effector, the joint impedance parameters (mass-damping-stiffness matrix M, D, K) are adjusted in real time.

[0142] A binocular RGB-D camera (frame rate 30Hz) and a laser profiler are used to reconstruct the 3D point cloud of the object surface in real time. The point cloud and the CAD model are matched by the ICP algorithm to detect the pose offset ΔT∈SE(3) and the deformation ΔS. If ΔS>Sth (deformation threshold), the deformation trend of the labeling area is predicted according to the finite element analysis (FEA) model, and the end contact force is adjusted. If ΔT causes a collision risk with the original trajectory, the local NURBS trajectory optimization is triggered, and the response delay is <50ms.

[0143] Torque feedback controls the end contact force fluctuation within ±1N (traditional PID control is ±5N), and the label bubble rate is reduced by 90%. In the labeling of flexible packaging bags, surface depressions cause the label to warp in traditional methods. This embodiment achieves complete fitting through force control adaptation.

[0144] The embodiments of the present application also provide a smart label printing optimization device. The smart label printing optimization device is used to execute the steps of the smart label printing optimization method based on image recognition shown in the above embodiments. The smart label printing optimization device can be a single server or a server cluster, or the smart label printing optimization device can be a terminal, which can be a handheld terminal, a laptop computer, a wearable device or a robot, etc.

[0145] The intelligent label printing optimization device includes:

[0146] A matrix construction unit, used to obtain the surface three-dimensional point cloud data and texture information of the object to be labeled according to a preset multi-spectral image acquisition device, and to construct an object surface feature matrix including spatial curvature features and material properties;

[0147] a target determination unit, configured to perform a regional adaptability evaluation on the surface feature matrix of the object, obtain a thermal map including a probability distribution of a labelable region, and determine a target labeling region according to a maximum response value corresponding to the thermal map;

[0148] An information acquisition unit is used to acquire semantic information corresponding to preset label content, calculate semantic element weights corresponding to the semantic information, and generate typesetting constraints corresponding to the label content according to the area range corresponding to the target labeling area, the semantic element weights and the semantic information; the typesetting constraints at least include a character scaling ratio threshold, information level visualization rules, label image size parameters and material attachment stability parameters;

[0149] A scheme generating unit, used for generating an optimized typesetting scheme corresponding to the label content according to the typesetting constraint condition, so as to control a preset printing device to print according to the optimized typesetting scheme to form a target label;

[0150] The labeling completion unit is used to generate a preset motion trajectory corresponding to the robotic arm based on the non-uniform rational B-spline curve fitting algorithm according to the first posture information corresponding to the object to be labeled, the second posture information of the preset printing device and the area range corresponding to the target area, so as to control the robotic arm to stick the target label to the target area according to the motion trajectory.

[0151] It should be noted that technical personnel in the relevant field can clearly understand that, for the convenience and conciseness of description, the specific working process of the smart label printing optimization device and each unit described above can refer to the corresponding process in the embodiment of the smart label printing optimization method based on image recognition described in the above embodiments, and will not be repeated here.

[0152] The above-mentioned smart label printing optimization method is implemented in the form of a computer program, which can be run on the above-mentioned device.

[0153] See also Figure 3 , Figure 3 : is a schematic block diagram of the structure of a control device provided in an embodiment of the present application. The control device includes a processor, a memory and a network interface connected via a device bus, wherein the memory may include a storage medium and an internal memory.

[0154] The storage medium can store an operating device and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any embodiment of the intelligent label printing optimization method based on image recognition.

[0155] The processor is used to provide computing and control capabilities to support the operation of the entire control device.

[0156] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any one of the basic smart label printing optimization system methods.

[0157] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the terminal to which the scheme of the present application is applied. The specific control device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0158] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0159] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0160] The surface three-dimensional point cloud data and texture information of the object to be labeled are obtained according to the preset multispectral image acquisition device, so as to construct the surface feature matrix of the object including the spatial curvature features and material properties;

[0161] Performing a regional adaptability evaluation on the surface feature matrix of the object to obtain a thermal map including a probability distribution of labelable regions, so as to determine a target labeling region according to a maximum response value corresponding to the thermal map;

[0162] Acquire semantic information corresponding to preset label content, calculate semantic element weights corresponding to the semantic information, and generate typesetting constraints corresponding to the label content according to the area range corresponding to the target labeling area, the semantic element weights and the semantic information; the typesetting constraints at least include a character scaling ratio threshold, information level visualization rules, label image size parameters and material attachment stability parameters;

[0163] Generate an optimized typesetting scheme corresponding to the label content according to the typesetting constraint conditions, and control a preset printing device to print according to the optimized typesetting scheme to form a target label;

[0164] Based on the non-uniform rational B-spline curve fitting algorithm, a motion trajectory corresponding to a preset robotic arm is generated according to the first pose information corresponding to the object to be labeled, the second pose information of the preset printing device, and the area range corresponding to the target area, so as to control the robotic arm to paste the target label to the target area according to the motion trajectory.

[0165] It should be noted that technicians in the relevant field can clearly understand that for the convenience and brevity of description, the specific working process of the processor described above can refer to the corresponding process in the method embodiments described in the above embodiments, and will not be repeated here.

[0166] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement the steps of the image recognition-based smart label printing optimization method provided in the above-mentioned embodiments of the present application.

[0167] The computer-readable storage medium may be an internal storage unit of the control device described in the foregoing embodiment, such as a hard disk or memory of the control device. The computer-readable storage medium may also be an external storage device of the control device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the control device.

[0168] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A smart label printing optimization method based on image recognition, characterized in that: include: The surface three-dimensional point cloud data and texture information of the object to be labeled are obtained according to the preset multispectral image acquisition device, so as to construct the surface feature matrix of the object including the spatial curvature features and material properties; Performing a regional adaptability evaluation on the surface feature matrix of the object to obtain a thermal map including a probability distribution of labelable regions, so as to determine a target labeling region according to a maximum response value corresponding to the thermal map; Acquire semantic information corresponding to preset label content, calculate semantic element weights corresponding to the semantic information, and generate typesetting constraints corresponding to the label content according to the area range corresponding to the target labeling area, the semantic element weights and the semantic information; the typesetting constraints at least include a character scaling ratio threshold, information level visualization rules, label image size parameters and material attachment stability parameters; Generate an optimized typesetting scheme corresponding to the label content according to the typesetting constraint conditions, and control a preset printing device to print according to the optimized typesetting scheme to form a target label; Based on the non-uniform rational B-spline curve fitting algorithm, a motion trajectory corresponding to a preset robotic arm is generated according to the first pose information corresponding to the object to be labeled, the second pose information of the preset printing device, and the area range corresponding to the target area, so as to control the robotic arm to paste the target label to the target area according to the motion trajectory.

2. The method according to claim 1, characterized in that The method of obtaining the surface three-dimensional point cloud data and texture information of the object to be labeled according to the preset multispectral image acquisition device, and constructing the object surface feature matrix including the spatial curvature features and material properties, includes: Inputting the texture information into a preset parallel convolutional neural network to extract texture feature maps of multiple spectral channels; Inputting the surface three-dimensional point cloud data into a preset graph convolutional network to perform spatial curvature modeling on the three-dimensional point cloud data, and outputting spatial curvature features including spectral reflectance and material properties; The texture feature map and the spatial curvature feature are concatenated into feature tensors, and the concatenated feature tensors are subjected to dimensionality reduction processing through a differentiable pooling layer to generate an object surface feature matrix with local geometry preservation.

3. The method according to claim 1, characterized in that The performing of regional adaptability evaluation on the surface feature matrix of the object to obtain a thermal map including a probability distribution of labelable regions includes: Constructing a deep region segmentation network; the deep region segmentation network adopts a U-Net architecture encoding and decoding structure, introduces a channel attention mechanism in the encoding stage to weight the importance of the object surface feature matrix; embeds a deformable convolution module in the decoding stage, and dynamically adjusts the receptive field to adapt to the morphological characteristics of different curvature regions; Input the object surface feature matrix into a deep region segmentation network, and output a probability distribution map containing the suitability of labeling each pixel point; The probability distribution graph is processed according to a morphological closing operation to eliminate discrete noise points and generate a heat map with regional connectivity.

4. The method according to claim 1, characterized in that: The step of determining the target labeling area according to the maximum response value corresponding to the thermal map includes: The objective function is constructed according to the maximum area, minimum principal curvature variance and maximum edge sharpness index corresponding to the target labeling area; Optimizing the thermal map according to a preset objective function and a multi-objective optimization algorithm to obtain a plurality of candidate labeling areas; According to the NSGA-II algorithm, multiple candidate labeling regions are non-dominatedly sorted, and the weight coefficient corresponding to each sorted candidate labeling region is calculated according to the entropy weight method; Among multiple candidate labeling regions, a continuous region whose weight coefficient is greater than a preset threshold is obtained as the target labeling region.

5. The method according to claim 1, characterized in that The calculating the semantic element weight corresponding to the semantic information to generate the typesetting constraint condition corresponding to the label content according to the area range corresponding to the target labeling area, the semantic element weight and the semantic information includes: Extract the semantic dependency tree corresponding to the semantic information according to the preset BERT model; Calculating the importance score of each semantic node in the semantic dependency tree based on a preset graph attention network as the semantic element weight; The area range parameters corresponding to the area range are converted into two-dimensional packing constraints, and the semantic element weights are mapped into weighted coefficients of information level visualization rules; the area range parameters include one or more of geometric size parameters, curvature feature parameters, edge feature parameters, material attribute parameters, spatial position and direction parameters, topological parameters and stability parameters; Based on the Lagrange multiplier method, the typesetting constraint condition corresponding to the label content is obtained according to the two-dimensional packing constraint and the weighting coefficient.

6. The method according to claim 1, characterized in that Generating the optimized layout scheme corresponding to the tag content according to the layout constraint condition includes: Construct an intelligent typesetting architecture based on a generative adversarial network; the generator of the generative adversarial network adopts a multi-scale feature fusion network with a self-attention mechanism, and the discriminator of the generative adversarial network introduces a differentiable rendering module to simulate the actual printing effect; optimize the generator parameters of the generative adversarial network based on reinforcement learning, and generate the reward function corresponding to the reinforcement learning according to information entropy, aesthetic evaluation score and material matching degree; Generate a style transfer vector according to preset visual specifications and typesetting constraints to embed the style transfer vector into a latent space corresponding to the intelligent typesetting architecture; The tag content is input into the intelligent typesetting framework to generate the optimized typesetting solution.

7. The method according to claim 1, characterized in that Based on the non-uniform rational B-spline curve fitting algorithm, the preset motion trajectory corresponding to the robotic arm is generated according to the first posture information corresponding to the object to be labeled, the second posture information of the preset printing device and the area range corresponding to the target area, including: Construct a kinematic model based on Lie group space to convert the first pose information and the second pose information into a rigid body transformation matrix in a preset space; Performing path planning according to the kinematic model and the rigid body transformation matrix to generate an initial motion trajectory; wherein a curvature continuity constraint of a non-uniform rational B-spline curve is introduced into the configuration space corresponding to the initial motion trajectory; The dynamic obstacle avoidance information of the initial motion trajectory is calculated according to a preset physical engine simulation, so as to optimize the initial motion trajectory according to the dynamic obstacle avoidance information and generate the motion trajectory; wherein the dynamic obstacle avoidance information is calculated according to the collision probability, obstacle kinetic energy and path energy consumption corresponding to the initial motion trajectory.

8. The method according to claim 1, characterized in that The step of controlling the robot arm to attach the target label to the target area according to the motion trajectory includes: Obtaining torque feedback information of the end effector corresponding to the robotic arm, and adjusting the dynamic parameters of the robotic arm according to the torque feedback information; The labeling process of the robotic arm is monitored in real time according to a preset online visual feedback device. When it is detected that the object to be labeled sends a posture deviation and / or an abnormal surface deformation, the motion trajectory is optimized in real time to control the robotic arm to complete the labeling process according to the real-time optimized dynamic parameters and motion trajectory.

9. An intelligent label printing optimization device based on image recognition, characterized in that: include: A matrix construction module is used to obtain the surface three-dimensional point cloud data and texture information of the object to be labeled according to a preset multispectral image acquisition device, and to construct an object surface feature matrix including spatial curvature features and material properties; A region determination module is used to perform a regional adaptability evaluation on the surface feature matrix of the object, obtain a thermal map including a probability distribution of labelable regions, and determine a target labeling region according to a maximum response value corresponding to the thermal map; An information acquisition module is used to acquire semantic information corresponding to preset label content, calculate semantic element weights corresponding to the semantic information, and generate typesetting constraints corresponding to the label content according to the area range corresponding to the target labeling area, the semantic element weights and the semantic information; the typesetting constraints at least include a character scaling ratio threshold, information level visualization rules, label image size parameters and material attachment stability parameters; A scheme generating module, used for generating an optimized typesetting scheme corresponding to the label content according to the typesetting constraint conditions, so as to control a preset printing device to print according to the optimized typesetting scheme to form a target label; The labeling completion module is used to generate a preset motion trajectory corresponding to the robotic arm based on the non-uniform rational B-spline curve fitting algorithm according to the first posture information corresponding to the object to be labeled, the second posture information of the preset printing device and the area range corresponding to the target area, so as to control the robotic arm to stick the target label to the target area according to the motion trajectory.

10. An intelligent label printing optimization system based on image recognition, characterized in that: include: Multispectral image acquisition equipment, used to obtain the surface three-dimensional point cloud data and texture information of the object to be labeled; A printing device, used for printing a target label; A robotic arm, used for attaching the target label to the object to be labeled; A control device, used to construct an object surface feature matrix including spatial curvature features and material properties according to the surface three-dimensional point cloud data and texture information; Perform regional adaptability evaluation on the surface feature matrix of the object, obtain a thermal map including the probability distribution of the labelable area, and determine the target labeling area according to the maximum response value corresponding to the thermal map; obtain semantic information corresponding to the preset label content, calculate the semantic element weight corresponding to the semantic information, and generate typesetting constraints corresponding to the label content according to the area range, semantic element weight and semantic information corresponding to the target labeling area; the typesetting constraints at least include character scaling ratio threshold, information level visualization rules, label image size parameters and material attachment stability parameters; generate an optimized typesetting scheme corresponding to the label content according to the typesetting constraints, and control a preset printing device to print according to the optimized typesetting scheme to form a target label; Based on the non-uniform rational B-spline curve fitting algorithm, a motion trajectory corresponding to a preset robotic arm is generated according to the first pose information corresponding to the object to be labeled, the second pose information of the preset printing device, and the area range corresponding to the target area, so as to control the robotic arm to paste the target label to the target area according to the motion trajectory.

Images