Household appliance assembling and warehousing method based on visual guidance and energy label identification

Abstract

The application discloses a kind of based on visual guidance and energy consumption label identification household electrical appliances assembly and warehousing method, belong to industrial automation control technical field.Pass through target detection and the multi-target tracking algorithm of extended state vector, combined with mask matching strategy, real-time acquisition is assembled household electrical appliances parts pose information and its corresponding tracking trajectory;According to predetermined assembly process order, control mechanical arm completes dynamic capture and assembly;Position product energy efficiency label, further extract the energy efficiency grade area containing grade color bar and indicating arrow therein;Using the identification algorithm based on dynamic color mapping to process the image of energy efficiency grade area, determine the energy efficiency grade of product;According to energy efficiency grade, guide product warehousing.The application realizes stable tracking and accurate assembly to moving parts, and high reliability identification and automatic classification to finished product energy efficiency information by improved tracking, analysis control and robust identification algorithm deep fusion.

Description

Technical Field

[0001] This invention belongs to the field of industrial automation control technology, and in particular relates to a method for assembling and warehousing home appliances based on visual guidance and energy consumption label recognition. Background Technology

[0002] In modern manufacturing, especially in discrete assembly industries such as home appliances, automated material handling, assembly, and finished product sorting on assembly lines are crucial for improving production efficiency and ensuring consistent quality. Traditional automation solutions typically rely on pre-programmed robotic arms and fixed tooling fixtures, suitable for mass production of standardized products. However, as the market shifts towards flexible manufacturing models with diverse product types and smaller batches, production lines need to be able to quickly adapt to the automated assembly of parts of different specifications and in disordered order, and to automatically sort and store finished products based on multiple attributes (such as energy efficiency ratings). This requires systems with a high degree of environmental awareness, real-time decision-making capabilities, and precise control capabilities to interact with the physical world.

[0003] To address these needs, the industry has developed solutions combining machine vision with industrial robots. A typical technical approach involves capturing images of the production line using industrial cameras, employing deep learning-based target detection algorithms (such as the YOLO series) to identify workpiece positions in real time, and then converting the image coordinates into poses in the robot's coordinate system through hand-eye calibration, ultimately driving the robot to perform the grasping action. For scenarios requiring continuous tracking of moving targets, multi-target tracking algorithms are introduced. Regarding finished product information recognition, for products with energy efficiency labels, traditional methods often use color segmentation based on fixed thresholds combined with optical character recognition (OCR) technology to read the rating information.

[0004] However, such existing technologies face significant bottlenecks in achieving high-precision, high-speed flexible production:

[0005] 1. Under complex working conditions, changes in lighting, reflections, and occlusions can lead to missed detections, false detections, or fluctuations in pose estimation; while high-frequency grasping requires the control layer to provide millisecond-level, highly deterministic motion commands.

[0006] 2. Standard multi-target tracking algorithms are insufficient in modeling the continuous changes in target size caused by perspective in highly structured motion scenarios such as conveyor belts. They are also prone to ID confusion when targets are parallel or intersecting, leading to broken tracking trajectories or incorrect associations. As a result, they cannot provide stable and accurate motion prediction for dynamic grasping.

[0007] 3. For energy efficiency label recognition, traditional color recognition methods based on fixed thresholds are extremely sensitive to changes in lighting. Label reflections or changes in shooting angle can easily lead to misjudgments of color. Meanwhile, simple OCR technology has limited recognition rates for distorted text and complex backgrounds, making it difficult to meet industrial-grade reliability requirements. Summary of the Invention

[0008] To address the issues of unstable tracking affecting dynamic grasping accuracy and energy efficiency identification being greatly affected by environmental interference in the electrical wiring operations of home appliances, this invention provides a method for assembling and warehousing home appliances based on visual guidance and energy consumption label recognition.

[0009] The technical solution adopted in this invention is as follows:

[0010] In a first aspect, this invention proposes a method for assembling and warehousing home appliances based on visual guidance and energy consumption label recognition, comprising the following steps:

[0011] S1. Process the image stream of the assembly station, and obtain the pose information of the home appliance parts to be assembled and their corresponding tracking trajectories in real time through target detection algorithm and multi-target tracking algorithm.

[0012] The multi-target tracking algorithm uses a Kalman filter based on extended state vectors to model the motion of the parts and employs a band-directed mask matching strategy for data association. The band-directed mask matching strategy uses offline calibrated spatial mapping relationships to constrain the search area for data association.

[0013] The extended state vector incorporates the target motion state and the target size change state.

[0014] S2. According to the predetermined assembly process sequence, based on the pose information of the corresponding parts and their corresponding tracking trajectories obtained in S1, control the robotic arm to grasp and execute assembly operations until the product assembly is completed.

[0015] S3. For images of assembled home appliances with energy efficiency labels, locate the entire area of ​​the energy efficiency label and further extract the energy efficiency level area containing the level color bar and indicator arrow; use a recognition algorithm based on dynamic color mapping to process the image of the energy efficiency level area to determine the energy efficiency level of the product.

[0016] S4. Based on the energy efficiency level identified in S3, guide or transport the home appliances to the designated storage area corresponding to that energy efficiency level.

[0017] Furthermore, the target motion state includes the motion velocity of the center coordinates within the image plane, and the target size change state includes the aspect ratio change rate and the height change rate.

[0018] Furthermore, S1 also includes an adaptive scheduling step:

[0019] During multi-target tracking, a comprehensive quality score is maintained for the tracking trajectory of each target, and the calling frequency of the target detection algorithm in subsequent frames is dynamically adjusted based on the score. When the comprehensive quality score is higher than the first threshold, the calling frequency of target detection is reduced; when the comprehensive quality score is lower than the second threshold, the calling frequency of target detection is increased until frame-by-frame detection is performed; when the comprehensive quality score is between the second threshold and the first threshold, the normal calling frequency is maintained.

[0020] Furthermore, the comprehensive quality score is calculated based on multi-dimensional indicators of the tracking trajectory of a single target. These indicators include trajectory length, ID identifier switching frequency, consistency of historical detection confidence, Mahalanobis distance between predicted and actual observed positions, and similarity between current appearance features and historical appearance features of the trajectory.

[0021] Furthermore, the band-direction mask matching strategy includes:

[0022] Using the homography matrix obtained from offline calibration, the predicted values ​​of the target position by the Kalman filter in the multi-target tracking algorithm are mapped from the image coordinate system to the world coordinate system of the conveyor belt plane;

[0023] In the world coordinate system, a strip-shaped effective area of ​​preset width is set along the direction of conveyor belt movement;

[0024] When performing data association matching, only target trajectories whose predicted locations are within the effective area of ​​the strip are listed as candidate matching objects.

[0025] Furthermore, S2 includes:

[0026] S21. For the part to be assembled, calculate its estimated pose at the moment of grasping based on the pose information and tracking trajectory obtained in S1.

[0027] S22. Based on the estimated pose, the target angles of each joint of the robotic arm are obtained by inverse kinematics solution.

[0028] S23. Based on the target angle and the motion state of the part reflected by the tracking trajectory, perform trajectory planning to generate joint motion control commands that enable the end effector of the robotic arm to move synchronously with the part and realize the grasping action.

[0029] S24. Repeat the above process to complete the gripping and assembly of all parts in sequence.

[0030] Furthermore, in S3, the recognition algorithm based on dynamic color mapping includes:

[0031] S31. Perform color space conversion and background color filtering on the image of the energy efficiency level area to initially filter out interfering pixels;

[0032] S32. Perform K-means clustering on the filtered pixels to extract the main color clusters;

[0033] S33. Match the colors of the main color cluster with the pre-stored standard energy efficiency color library, verify the spatial distribution, and filter out the effective color clusters that simultaneously meet the color matching and spatial distribution rules.

[0034] S34. Sort each effective color cluster according to the number of pixels it contains from least to most, to obtain the corresponding reconstruction sequence from high energy efficiency level to low energy efficiency level.

[0035] S35. Identify the level indicator arrows in the energy efficiency level area, and determine the final energy efficiency level of the product by matching the arrow colors and their pointing positions in the reconstruction sequence.

[0036] Furthermore, S3 also includes an energy efficiency parameter extraction step: extracting the energy consumption parameter values ​​recorded on the energy efficiency label within the entire area of ​​the energy efficiency label using text recognition technology.

[0037] Secondly, the present invention proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for assembling and warehousing home appliances based on visual guidance and energy consumption label recognition.

[0038] Thirdly, a computer electronic device, including a memory and a processor;

[0039] The memory is used to store computer programs;

[0040] The processor is used to implement the above-described method for assembling and warehousing home appliances based on visual guidance and energy consumption label recognition when executing the computer program.

[0041] The beneficial effects of this invention are:

[0042] This invention introduces an extended state vector representing the target's size changes during part tracking, enabling the Kalman filter to more accurately model the target's deformation during motion. Meanwhile, a band-directed mask matching strategy utilizing spatial mapping effectively leverages prior scene knowledge, significantly reducing identity confusion errors when targets are parallel or intersecting. In the finished product energy efficiency rating identification stage, a dynamic color mapping-based identification algorithm replaces the traditional fixed threshold method with clustering and matching mechanisms, giving it excellent adaptability to lighting changes and background interference, ensuring high robustness and accuracy in energy efficiency rating determination. From real-time part tracking and assembly to automatic energy efficiency identification of assembled products, and then to automatic classification and warehousing based on the identification results, the entire process requires no manual intervention, achieving full-process intelligence and automation. This greatly improves production cycle time and sorting efficiency, ensuring the system's long-term operational reliability in industrial environments. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating a method for assembling and warehousing home appliances based on visual guidance and energy consumption label recognition.

[0044] Figure 2 This is a schematic diagram illustrating the process of controlling a robotic arm to grasp parts and perform assembly operations.

[0045] Figure 3 This is a flowchart illustrating a recognition algorithm based on dynamic color mapping.

[0046] Figure 4 This is a schematic diagram of a home appliance assembly and warehousing system based on visual guidance and energy consumption label recognition;

[0047] Figure 5 This is a schematic diagram of a computer electronic device. Detailed Implementation

[0048] The present invention will be further described and illustrated below with reference to specific embodiments. The embodiments described are merely examples of the content of this disclosure and do not limit the scope of the invention. The technical features of each embodiment in the present invention can be combined accordingly, provided that there is no mutual conflict.

[0049] The accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0050] The flowchart shown in the attached diagram is merely an illustrative example and does not necessarily include all steps. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0051] like Figure 1 As shown, the home appliance assembly and warehousing method based on visual guidance and energy consumption label recognition proposed in this invention adopts an extended state vector tracking algorithm, geometric analytical inverse kinematics, and dynamic color mapping recognition algorithm to achieve high-precision, high-robustness, and highly automated intelligent assembly and warehousing of home appliances.

[0052] In this invention, one implementation process is as follows:

[0053] S1. Process the image stream of the assembly station, and obtain the pose information of the home appliance parts to be assembled and their corresponding tracking trajectories in real time through target detection algorithm and multi-target tracking algorithm.

[0054] From the dynamic assembly station image stream, the spatial pose and continuous motion trajectory of each part to be assembled are extracted stably and accurately. This step uses an improved target detection algorithm and target tracking algorithm.

[0055] In this embodiment, the image stream at the assembly station is acquired by an RGBD depth camera or a USB industrial camera, supporting video stream acquisition from 640×480@30fps to 1920×1080@30fps. The camera deployed above the conveyor belt can simultaneously capture high-resolution RGB color images and precise depth information. The acquired raw data stream is transmitted to the host computer in real time via a high-speed USB interface. A three-factor online scoring framework based on brightness, contrast, and blur is introduced. Video frames with a weighted score ≥0.7 are entered into subsequent processing. The default weighting coefficients for brightness, contrast, and blur are 0.3, 0.4, and 0.3, respectively.

[0056] S11, Target Detection Algorithm

[0057] In this embodiment, the YOLOv8-nano model is used as the object detection algorithm, and its network structure is improved. The improved network adopts a CSPDarknet53-PANet-BiFPN structure. Specifically, the CSPDarknet53 structure is used in the backbone network to enhance the extraction capability of fine-grained features such as edges, contours, and textures of home appliance parts. In the feature fusion part, PANet and BiFPN are combined, and a bidirectional weighted fusion mechanism is introduced on the basis of top-down and bottom-up feature transfer. This allows features of different scales to interact more fully, reducing information loss during the fusion process for small targets, occluded targets, and targets with varying scales. Through the above improvements, the model maintains the lightweight and high-speed inference characteristics of YOLOv8-nano while improving the detection accuracy, recall, and localization stability of parts of different sizes in complex assembly scenarios, reducing the false negative and false positive rates, and making it more suitable for real-time processing needs in industrial settings. For each frame of image, the object detection algorithm outputs a list containing multiple bounding boxes, each containing the part category, confidence score, and its pixel coordinates in the image. The classification loss of this algorithm introduces online hard example mining on top of category awareness, making the model pay more attention to samples that are difficult to classify during the algorithm training phase. The improved category awareness loss is defined as follows:

[0058]

[0059] in, As a category balance factor, Focusing parameters (default) ), This represents the classification probability. This loss function automatically reduces the weight of easily classified samples, causing the model to focus more on difficult samples.

[0060] In addition, the bounding box regression loss is based on Smooth-L1 with an IoU-aware penalty term added in parallel, and the DFL (Distribution Focal Loss) branch is retained to improve the localization accuracy of the bounding boxes. The overall bounding box loss is:

[0061]

[0062] in, It is category perception loss. It is the Smooth-L1 loss. It is an IoU-aware penalty item. It is the penalty coefficient.

[0063] S12, Multi-target tracking algorithm

[0064] Based on the DeepSORT model, the following key improvements are made:

[0065] (1) Eight-dimensional state vector extension: the traditional four-dimensional state vector of DeepSORT is extended. Expanded to an eight-dimensional state vector:

[0066]

[0067] Where x is the x-coordinate of the bounding box center (in pixels), y is the y-coordinate of the bounding box center (in pixels), a is the aspect ratio of the bounding box (dimensionless), and h is the height of the bounding box (in pixels). It represents the speed of the center's x and y coordinates within the image plane (in pixels per frame), corresponding to the target's motion state; These represent the aspect ratio change rate (in 1 / frame) and height change rate (in pixels / frame), corresponding to the target size change state. In a conveyor belt scene, as the part moves with the conveyor belt, its relative distance and viewing angle to the camera change, causing not only its position to move in the image, but also continuous changes in the size (h) and shape (a) of its bounding box. The expanded state vector can more accurately model the target's motion state on the conveyor belt, significantly improving tracking stability and prediction accuracy when the target undergoes scale scaling or slight deformation. The state transition matrix is ​​correspondingly expanded to an 8×8 matrix, and the acceleration term of the process noise covariance matrix can be corrected online by the real-time estimated conveyor belt speed.

[0068] (2) Band-directed mask matching strategy

[0069] This strategy is a dedicated optimization for linear motion scenarios on conveyor belts, designed to resolve potential ID confusion issues when associating data by matching the detection bounding box of the current frame with existing tracking trajectories.

[0070] In this embodiment, the implementation principle of the band-directed mask matching strategy is as follows: A 3×3 homography transformation matrix H is pre-calculated from the image pixel coordinate system to the world coordinate system of the conveyor belt plane using methods such as Zhang Zhengyou's calibration. Before data association in each frame, the algorithm uses a Kalman filter to predict the target position of each tracking trajectory in the current frame (obtained from an eight-dimensional state vector). Then, the homography matrix H is used to map these predicted image coordinates to the world coordinates of the conveyor belt plane. In the world coordinate system of the conveyor belt, a narrow strip-shaped region with the direction of conveyor belt movement as its central axis is defined. This region is considered to be the reasonable spatial range for the target to appear. For example, the region is defined as within ±10cm along the central axis of the conveyor belt. During data association matching using algorithms such as the Hungarian algorithm, the trajectory is only allowed to be matched with the detection box of the current frame if the predicted position of the target falls within the aforementioned strip-shaped region. For trajectories whose predicted positions drift outside the strip-shaped region due to errors, occlusion, or other interference, their predicted boxes are directly masked in the association step of this frame and do not participate in the matching. This strategy leverages the spatially structured priors of the conveyor belt scenario to effectively avoid unreasonable cross-track ID matching caused by short-term target aggregation, intersection, or detection noise.

[0071] S13. Acquisition of pose information and tracking trajectory

[0072] Through the improved target detection and multi-target tracking algorithm described above, for each successfully established and maintained tracking target part, the algorithm assigns it a unique ID and continuously outputs its state in each frame, forming a continuous state sequence over time, i.e., the tracking trajectory. This trajectory contains complete historical and real-time motion information of the target. Similarly, pose data can be calculated based on the algorithm's own principles, which will not be elaborated here.

[0073] S14, Adaptive Scheduling

[0074] During multi-target tracking, a comprehensive quality score is maintained for the tracking trajectory of each target. Based on the score, the calling frequency of the target detection algorithm in subsequent frames is dynamically adjusted. When the comprehensive quality score is higher than a first threshold, the calling frequency of target detection is reduced; when the comprehensive quality score is lower than a second threshold, the calling frequency of target detection is increased until frame-by-frame detection is performed; when the comprehensive quality score is between the second and first thresholds, the normal calling frequency is maintained. In this embodiment, the first threshold is 0.8, the second threshold is 0.5, and the normal calling frequency is one detection triggered every 30 frames.

[0075] The comprehensive quality score is calculated based on a multi-dimensional index of the tracking trajectory of a single target, and the index includes:

[0076] Track length: The number of consecutive frames in which the target has been successfully tracked since it was initialized with a new track. A longer track generally indicates more stable tracking.

[0077] ID switching frequency: This refers to the historical data of the currently evaluated single trajectory within a recent time window. It counts the number of times the target's assigned ID was confused or swapped with other target IDs during tracking. A lower frequency indicates better identity consistency during tracking.

[0078] Consistency of historical detection confidence: For the currently evaluated single trajectory, extract the confidence score sequence of the target detector output for the target bounding box from a series of consecutive frames throughout its history. Calculate the moving coefficient of variation of this confidence sequence. The smaller the coefficient of variation, the smaller the fluctuation in the detection confidence for the target, the higher the consistency, and the more reliable the tracking quality.

[0079] Mahalanobis distance between predicted and observed positions: This refers to the state of the currently evaluated single trajectory in the latest frame. In each frame, a Kalman filter is used to predict its position for the next frame, while simultaneously obtaining the actual detected position. The Mahalanobis distance is the statistical distance between this predicted and observed positions, taking into account the covariance matrix estimated by the filter. The smaller the distance, the more accurate the target motion model prediction and the more reasonable the tracking.

[0080] Similarity between current appearance features and historical trajectory appearance features: For a single trajectory currently being evaluated, the system maintains a historical moving average vector of appearance features for that trajectory. In the current frame, the real-time appearance features of the target are extracted, and the cosine similarity between the current real-time features and the historical average features is calculated. The higher the similarity, the higher the match between the currently detected target and the historical appearance of the trajectory, and the more reliable the tracking.

[0081] All metrics are calculated based on the state history (position, appearance, confidence) and instantaneous observations of each individual tracking trajectory to evaluate the tracking quality of each trajectory and make detection frequency scheduling decisions for that target accordingly.

[0082] S2. Control the robotic arm to grasp parts and perform assembly operations.

[0083] This step, based on the predetermined assembly process sequence, sequentially uses the pose information of the corresponding parts obtained by S1 and their corresponding tracking trajectories to control the robotic arm to grasp and perform assembly operations until the product assembly is completed.

[0084] like Figure 2 As shown, taking a four-degree-of-freedom robotic arm as an example, the geometric analytical method is used to solve for the angles that each joint should rotate when the end effector of the robotic arm reaches its grasping pose. The specific process is as follows:

[0085] S21. Define the robot arm coordinate system using the standard DH (Denavit-Hartenberg) parametric method.

[0086] (1) World coordinate system: The origin is the projection of the center of the robotic arm base onto the ground, the X-axis points in the direction of the conveyor belt movement, and the Z-axis is vertically upward;

[0087] (2) Base coordinate system: fixed on the base of the robotic arm, coincident with the world coordinate system or subject to known fixed transformations;

[0088] (3) End coordinate system: fixed at the end of the gripper, with the origin located at the center of the gripper;

[0089] (4) Joint coordinate system: The local coordinate system of each joint, defined according to the DH rule.

[0090] The geometric parameters of the robotic arm linkage are shown in Table 1.

[0091] Table 1 DH Parameter Table for Robotic Arm

[0092]

[0093] in: These are standard DH parameters, where Indicates the link offset. Indicates the length of the link. This represents the linkage torsion angle. Subsequent formulas primarily use geometric analytical methods to directly calculate the joint angles; therefore, they are not directly written as... Instead of the traditional form, it is written as corresponding structural dimension parameters L1, L2, L3, and L4. For example, in this embodiment, d1 corresponds to L1. These correspond to L2, L3, and L4 respectively. Parameters: L1=104mm (base height), L2=105mm (outer arm length), L3=100mm (outer arm length), L4=70mm (end length).

[0094] S22, Analytical solution of inverse kinematics

[0095] Given the end target location and end pitch angle Solve for the angles of each joint. This invention uses geometric projection and trigonometric analytical methods for solving the problem.

[0096] (1) Base rotation angle Calculation:

[0097] The base joint rotates around the Z-axis, and its angle is determined by the projection direction of the end point onto the XY plane:

[0098]

[0099] in, This is the arctangent function for the four quadrants, which can automatically determine the correct quadrant angle based on the signs of x and y. Its value range is... .

[0100] (2) Geometric projection analysis:

[0101] Set the end point The projection on the ground (XY plane) is Then the projected distance r (horizontal arm span) is:

[0102]

[0103] Considering the end pitch angle , end link Decomposed into horizontal and vertical components :

[0104]

[0105]

[0106] (3) Establishment of key geometric relationships:

[0107] Starting from the shoulder joint (the origin of joint 2), establish the geometric relationship to the wrist joint (the origin of joint 4). Let the position of the wrist joint relative to the base be:

[0108]

[0109]

[0110] in, This represents the horizontal distance from the wrist joint to the Z-axis. The height of the wrist joint relative to the shoulder joint.

[0111] (4) Solving for triangles:

[0112] Applying the law of cosines, in the triangle △SEW formed by the shoulder joint S, elbow joint E, and wrist joint W, the lengths of the three sides are: SE = L2 (upper arm), EW = L3 (forearm), and SW = d (the straight-line distance from the shoulder joint to the wrist joint). According to the Pythagorean theorem:

[0113]

[0114] (5) Joint angles Calculation (elbow joint):

[0115] In triangle △SEW, the law of cosines is used to find the interior angle of the elbow joint:

[0116]

[0117] Summarized as follows:

[0118]

[0119] Or equivalently:

[0120]

[0121] (6) Joint angle 2 Calculation (shoulder joint)

[0122] First, calculate the angle between vector SW and the horizontal plane. :

[0123]

[0124] Then, in triangle △SEW, apply the Law of Cosines to find the angles. (recorded as) ):

[0125]

[0126] Considering the sign relationship in the vertical direction, we finally obtain:

[0127]

[0128] (7) Joint angle 4 The calculation (wrist joint) is based on the end-effector pitch angle constraint, and the wrist joint angle is directly derived from geometric relationships:

[0129]

[0130] S23. Kinematic Constraints and Accessibility Analysis

[0131] The necessary conditions for the existence of inverse kinematic solutions include:

[0132] Workspace constraints—height constraints: The end height z must satisfy:

[0133]

[0134] Triangle inequality constraint: The three sides forming a triangle must satisfy the following conditions:

[0135]

[0136] Effective range constraint for cosine values: The results of cosine theorem calculations must satisfy the following:

[0137]

[0138]

[0139] Workspace Boundaries: The reachable workspace of the robotic arm is a ring-shaped area that satisfies:

[0140]

[0141] S24. Multiple Solution Selection and Security Verification

[0142] Due to the periodicity of trigonometric functions and the symmetry of the robotic arm configuration, inverse kinematics typically has multiple solutions (elbow-on / elbow-off configurations). This invention employs the following multi-solution selection strategy:

[0143] (1) Traversal of pitch angle range: within the allowed pitch angle range The algorithm iterates through the system with a fixed step size and calculates all valid solutions.

[0144] (2) Energy minimization criterion: Define the joint motion energy function,

[0145]

[0146] in, For the current joint angle, w i These are the weighting coefficients for each joint. The solution that minimizes the energy function E(θ) is selected, i.e., the solution that minimizes the amount of joint motion.

[0147] (3) Joint limit verification: Check whether each candidate solution meets the physical limit of the joint according to the following formula, and exclude solutions that exceed the physical range.

[0148]

[0149] (4) Singular configuration avoidance: When the robotic arm approaches a singular configuration (such as fully extended or fully folded), the determinant of the Jacobian matrix approaches zero. At this time, the penalty term is increased to avoid choosing this solution.

[0150] (5) Nearest neighbor projection strategy: If the target point is unreachable (outside the workspace), the nearest neighbor projection strategy is triggered to pull the target point radially back to the reachable boundary sphere, ensuring the continuity of the solution space.

[0151] S25, S-curve trajectory planning and motion smoothing

[0152] To ensure the smoothness and time-optimal nature of the robotic arm's motion, this invention employs an S-curve velocity planning algorithm. Let the joint spatial trajectory of all joint angles of the robotic arm changing with time over a continuous time interval be... It must satisfy the maximum speed constraint, maximum acceleration constraint, and maximum jerk constraint.

[0153] S-curve programming divides the motion process into seven stages: acceleration, uniform acceleration, deceleration, constant velocity, acceleration / deceleration, uniform deceleration, and deceleration / deceleration. The calculation order is as follows:

[0154] (1) From the maximum jerk and maximum acceleration Calculate the duration of the acceleration phase: ;

[0155] (2) From the maximum speed Given the acceleration curve, determine the duration of the uniform velocity segment. ;

[0156] (3) Due to symmetry, the duration of the deceleration segment is equal to the duration of the acceleration segment, and the total motion time is equal to the total motion time. ,in This refers to the duration of the complete acceleration segment.

[0157] (4) Multi-axis synchronization strategy: Using the slowest axis as the time reference, adjust other axes to achieve synchronous arrival.

[0158] S26. Generate control commands

[0159] After motion planning is completed, the discretized joint space trajectory, i.e., a sequence of target joint angles corresponding to a series of timestamps, is converted into control instructions that can be executed by the underlying drive unit of the robotic arm. According to the control characteristics of the servo motors, the target angle of each joint can be obtained by a linear mapping formula to obtain the corresponding PWM signal pulse width value. The encapsulated PWM pulse width instructions, servo motor ID, motion time and other parameters are framed according to a lightweight binary communication protocol and sent to the servo motor drive control module of the robotic arm in real time through a serial communication link, thereby driving the servo motors of each joint to execute the planned motion trajectory.

[0160] S3. Use a recognition algorithm based on dynamic color mapping to determine the product's energy efficiency level.

[0161] This step involves locating the entire area of ​​the energy efficiency label on an image of an assembled home appliance product with an energy efficiency label, and further extracting the energy efficiency level area containing the level color bar and indicator arrow; then using a recognition algorithm based on dynamic color mapping to process the image of the energy efficiency level area to determine the product's energy efficiency level.

[0162] In one specific embodiment of the present invention, such as Figure 3 As shown, the implementation process is as follows:

[0163] S31, Overall Area Positioning

[0164] Using object detection or semantic segmentation models, the overall area of ​​the energy efficiency label is located from the entire product image. This area is usually a regular color block (such as a blue background) containing the title "China Energy Efficiency Label", the energy efficiency level color bar, the level indicator arrow, and the bottom parameter text.

[0165] S32, Energy Efficiency Rating Area Extraction

[0166] Within the identified energy efficiency label area, a text detection network (such as a deep learning-based scene text detector) is used to locate regions containing descriptive text indicating low, medium, or high energy consumption levels. These text regions, along with their adjacent regions containing color bars and arrows, are collectively defined as energy efficiency level regions.

[0167] S33, Dynamic Color Mapping and Sequence Reconstruction

[0168] A dynamic color mapping recognition algorithm is employed. This algorithm does not rely on a fixed color threshold or absolute template position, but rather adapts to changes in lighting, angle, and other factors by dynamically analyzing image features. The main process is as follows:

[0169] (1) Image preprocessing and background filtering

[0170] Input an image of the energy efficiency level region and convert the image to the HSV (hue, saturation, brightness) color space. The HSV space is relatively insensitive to changes in lighting and is more suitable for color analysis.

[0171] Next, based on prior knowledge, the background color of the energy efficiency label (such as dark blue background, white text, and black border) is not the grade color. A filtering rule is set to exclude interfering pixels that are close to white, black, gray, or low-saturation backgrounds, retaining pixels that may belong to the grade color bars. In this embodiment, only pixels with saturation S>50 and brightness V>50 are retained. After HSV filtering, only pixels with clear color information are retained for subsequent color analysis.

[0172] (2) K-means clustering to extract dominant color tone

[0173] This invention employs a clustering method to extract representative colors. All filtered pixels are organized into an N×3 matrix X, and a K-means clustering algorithm is applied, where N is the number of pixels and 3 represents the RGB three-channel values. The number of clusters k can be dynamically set, for example, 5, corresponding to 5 energy efficiency levels. After iteration, k clusters are obtained, each representing a primary color cluster, while ensuring that each cluster has at least 10 pixels.

[0174] (3) Dynamic color mapping and sequence reconstruction

[0175] In the process of screening effective color clusters, this invention first calculates the Euclidean distance between the chromaticity center of each primary color cluster and a pre-stored standard energy-efficient color library to perform preliminary color similarity matching. However, in actual industrial scenarios, relying solely on color matching may misidentify randomly similar interference color blocks in the background as targets. Therefore, this invention further verifies the spatial distribution rules of pixels within a color cluster, such as evaluating whether it exhibits a continuous, concentrated, and approximately rectangular distribution pattern, which corresponds to the physical characteristics that a true energy-efficient color bar should have. Finally, color clusters that simultaneously satisfy the condition that the color distance is less than a set threshold and conforms to the expected spatial distribution rules are determined to be truly effective color clusters, effectively eliminating scattered noise and local interference.

[0176] In one specific embodiment of the present invention, the standard energy efficiency color library generally adopts a five-level classification, and the standard RGB color values ​​corresponding to each level are as follows:

[0177] Table 2 Color Library for Level 5 Energy Efficiency Standards

[0178]

[0179] Considering that actual energy efficiency standards are not unique and may include different systems such as Level 3, Level 4, and Level 5, the same color (e.g., green) may represent different levels in different systems. Furthermore, complex lighting and background interference may cause unclear colors on the label in certain areas. Therefore, this invention does not directly use the comparison results between the chromaticity centers of each primary color cluster and the standard energy efficiency color library as the level identification result, but only uses it to screen valid color clusters and exclude obviously invalid clusters.

[0180] This invention sorts all effective color clusters according to the number of pixels in each cluster from fewest to most, resulting in a reconstructed sequence from high energy efficiency level to low energy efficiency level. For example, for a standard-oriented energy efficiency label, the effective color clusters are sorted according to their vertical positions in the image pixels, thereby reconstructing a complete energy efficiency color sequence from level 1 to level 5.

[0181] This step does not rely on a fixed template location, but rather adaptively determines the sequence through clustering and sorting.

[0182] (4) Matching of indicator arrows and determination of level

[0183] Based on the energy efficiency level area image and the reconstructed energy efficiency color sequence, the level indicator arrows are located on the opposite side of the identified level description text (e.g., "low energy consumption") (or on the right side if the text is on the left), using color features or shape detection methods. The color of the arrow is extracted and matched with the colors of each level in the reconstructed energy efficiency sequence. Simultaneously, it is determined which row in the sequence the arrow points to vertically. Combining the results of the arrow color matching and positional pointing, the specific energy efficiency level indicated by the arrow is determined.

[0184] Specifically, within the energy efficiency rating area, the rating indicator arrows are located, their color features are extracted, and the similarity between the arrows and the colors corresponding to each rating in the reconstructed sequence is calculated to obtain a first candidate rating based on color matching. The direction of the arrows in the image is analyzed to determine their position within the reconstructed sequence, resulting in a second candidate rating based on their position. The first and second candidate ratings are compared; if they match, the rating is determined as the product's final energy efficiency rating. For example, if the rating indicator arrow points to the first row of the sequence and matches the color cluster in the first row, then Level 1 energy efficiency is taken as the final identification result.

[0185] To address the challenges of inconsistent standards and complex environmental interference in industrial settings, this invention does not directly use the comparison results between the chromaticity centers of each primary color cluster and the standard energy efficiency color library as the grade identification result. Instead, it dynamically reconstructs the energy efficiency grade sequence based on the pixel quantity relationship of these effective clusters, thereby establishing the relative order between colors. This allows it to adapt to different grade quantity systems. However, the reconstructed sequence only indicates the relative relationship between grades and cannot determine which grade the current product specifically corresponds to. Therefore, information from grade indicator arrows is introduced. By comprehensively matching the color of the arrows and their precise pointing position in the reconstructed sequence, the final energy efficiency grade of the product is uniquely and accurately determined. This method fundamentally overcomes the uncertainties caused by standard differences and on-site interference, achieving highly robust identification of energy consumption labels.

[0186] (5) Extraction of auxiliary information

[0187] Within the defined energy efficiency label area, OCR technology is used to identify specific numerical text such as "power consumption," "water consumption," and "cleaning ratio." The extracted energy consumption data and other information are then output along with the energy efficiency rating for detailed data entry and classification.

[0188] S4. Based on the energy efficiency level identified in S3, guide or transport the home appliances to the designated storage area corresponding to that energy efficiency level.

[0189] This step involves automated classification and data entry based on the energy efficiency rating identified by S3. One possible implementation process is as follows:

[0190] First, based on preset mapping rules, the energy efficiency level is converted into a specific physical storage target. Then, the robotic arm is controlled to directly grab the product and place it in the corresponding area, or the diversion device of the conveyor belt is controlled to guide the product to different storage channels.

[0191] The entire process is coordinated by a pipeline control system deployed in the actual scenario, ensuring the timing synchronization of the actions of equipment such as robotic arms and conveyor belts. Simultaneously, product tracking information is linked to the system, recording information such as product grade, warehousing location, and time into a database.

[0192] It should also be noted that the method for assembling and warehousing home appliances based on visual guidance and energy consumption label recognition in the above embodiments can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a system for assembling and warehousing home appliances based on visual guidance and energy consumption label recognition, such as... Figure 4 As shown, it includes:

[0193] The target detection and multi-target tracking module is used to process the image stream of the assembly station. Through target detection and multi-target tracking algorithms, it can obtain the pose information of the home appliance parts to be assembled and their corresponding tracking trajectories in real time.

[0194] The multi-target tracking algorithm uses a Kalman filter based on extended state vectors to model the motion of the parts and employs a band-directed mask matching strategy for data association. The band-directed mask matching strategy uses offline calibrated spatial mapping relationships to constrain the search area for data association.

[0195] The extended state vector incorporates the target motion state and the target size change state.

[0196] The robotic arm control module is used to control the robotic arm to grasp and perform assembly operations according to the predetermined assembly process sequence, based on the acquired pose information of the corresponding parts and their corresponding tracking trajectories, until the product assembly is completed.

[0197] The energy efficiency rating recognition module is used to locate the entire area of ​​the energy efficiency label in an image of an assembled home appliance product with an energy efficiency label, and further extract the energy efficiency rating area containing the rating color bar and indicator arrow; the recognition algorithm based on dynamic color mapping is used to process the image of the energy efficiency rating area to determine the energy efficiency rating of the product.

[0198] The warehousing guidance module is used to guide products into the warehouse based on the identified energy efficiency level.

[0199] Similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the method provided in the above embodiments, such as... Figure 5 As shown, it includes a memory and a processor;

[0200] The memory is used to store computer programs;

[0201] The processor is configured to implement the visual guidance and energy consumption label recognition method for home appliance assembly and warehousing in the above embodiments when executing the computer program.

[0202] When the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium.

[0203] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the method provided in the above embodiments. The storage medium stores a computer program, which, when executed by a processor, can realize the method for assembling and warehousing home appliances with visual guidance and energy consumption label recognition in the above embodiments.

[0204] It is understood that the computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0205] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A method for assembling and warehousing home appliances based on visual guidance and energy consumption label recognition, characterized in that, Includes the following steps: S1. Process the image stream of the assembly station, and obtain the pose information of the home appliance parts to be assembled and their corresponding tracking trajectories in real time through target detection algorithm and multi-target tracking algorithm. The multi-target tracking algorithm uses a Kalman filter based on extended state vectors to model the motion of the parts and employs a band-directed mask matching strategy for data association. The directed mask matching strategy uses the spatial mapping relationship of offline calibration to constrain the search area of ​​data association, including: using the homography matrix obtained by offline calibration, mapping the predicted value of the target position by the Kalman filter in the multi-target tracking algorithm from the image coordinate system to the world coordinate system of the conveyor belt plane; in the world coordinate system, setting a strip-shaped effective area of ​​a preset width along the direction of conveyor belt movement; when performing data association matching, only the target trajectory whose predicted position is located within the strip-shaped effective area is listed as a candidate matching object; The extended state vector incorporates the target motion state and the target size change state. S2. According to the predetermined assembly process sequence, based on the pose information of the corresponding parts and their corresponding tracking trajectories obtained in S1, control the robotic arm to grasp and execute assembly operations until the product assembly is completed. S3. For images of assembled home appliances with energy efficiency labels, locate the entire area of ​​the energy efficiency label and further extract the energy efficiency level area containing the level color bar and indicator arrow; use a recognition algorithm based on dynamic color mapping to process the image of the energy efficiency level area to determine the energy efficiency level of the product. The recognition algorithm based on dynamic color mapping includes: S31. Perform color space conversion and background color filtering on the image of the energy efficiency level area to initially filter out interfering pixels; S32. Perform K-means clustering on the filtered pixels to extract the main color clusters; S33. Match the colors of the main color cluster with the pre-stored standard energy efficiency color library, verify the spatial distribution, and filter out the effective color clusters that simultaneously meet the color matching and spatial distribution rules. S34. Sort each effective color cluster according to the number of pixels it contains from least to most, to obtain the corresponding reconstruction sequence from high energy efficiency level to low energy efficiency level. S35. Identify the level indicator arrows in the energy efficiency level area, and determine the final energy efficiency level of the product by matching the arrow colors and their pointing positions in the reconstruction sequence; S4. Guide products into the warehouse based on the identified energy efficiency level.

2. The method for assembling and warehousing home appliances based on visual guidance and energy consumption label recognition according to claim 1, characterized in that, The target motion state includes the motion velocity of the center coordinates within the image plane, and the target size change state includes the aspect ratio change rate and the height change rate.

3. The method for assembling and warehousing home appliances based on visual guidance and energy consumption label recognition according to claim 1, characterized in that, S1 also includes an adaptive scheduling step: During multi-target tracking, a comprehensive quality score is maintained for the tracking trajectory of each target, and the calling frequency of the target detection algorithm in subsequent frames is dynamically adjusted based on the score. When the comprehensive quality score is higher than the first threshold, the calling frequency of target detection is reduced; when the comprehensive quality score is lower than the second threshold, the calling frequency of target detection is increased until frame-by-frame detection is performed; when the comprehensive quality score is between the second threshold and the first threshold, the normal calling frequency is maintained.

4. The method for assembling and warehousing home appliances based on visual guidance and energy consumption label recognition according to claim 3, characterized in that, The comprehensive quality score is calculated based on multi-dimensional indicators of the tracking trajectory of a single target. These indicators include trajectory length, ID switching frequency, consistency of historical detection confidence, Mahalanobis distance between predicted and actual observed positions, and similarity between current appearance features and historical appearance features of the trajectory.

5. The method for assembling and warehousing home appliances based on visual guidance and energy consumption label recognition according to claim 1, characterized in that, S2 include: S21. For the part to be assembled, calculate its estimated pose at the moment of grasping based on the pose information and tracking trajectory obtained in S1. S22. Based on the estimated pose, the target angles of each joint of the robotic arm are obtained by inverse kinematics solution. S23. Based on the target angle and the motion state of the part reflected by the tracking trajectory, perform trajectory planning to generate joint motion control commands that enable the end effector of the robotic arm to move synchronously with the part and realize the grasping action. S24. Repeat the above process to complete the gripping and assembly of all parts in sequence.

6. The method for assembling and warehousing home appliances based on visual guidance and energy consumption label recognition according to claim 1, characterized in that, S3 also includes an energy efficiency parameter extraction step: Within the entire area of ​​the energy efficiency label, the energy consumption parameter values ​​recorded on the energy efficiency label are extracted using text recognition technology.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method for assembling and warehousing home appliances based on visual guidance and energy consumption label recognition as described in any one of claims 1 to 6.

8. A computer electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the method for assembling and warehousing home appliances based on visual guidance and energy consumption label recognition as described in any one of claims 1 to 6.

Images