Control system and method of full-automatic intelligent boxing equipment

Abstract

The invention relates to the field of intelligent boxing, and particularly discloses a control system and method for full-automatic intelligent boxing equipment, and the method comprises the steps: firstly obtaining a small-package product boxing image collected by a camera and boxing task information collected by a database, and then carrying out the feature extraction and correlation analysis of the small-package product boxing image and the boxing task information through a deep learning technology, and finally, the motion trail of the mechanical arm is generated through a generator, so that changes and new task requirements in the boxing process are met, the instant motion trail and action of the mechanical arm are provided, the boxing speed is increased, the boxing precision is improved, and more flexible and intelligent boxing operation is achieved.

Description

Technical Field

[0001] The present application relates to the field of intelligent packing, and more specifically, to a control system and method for fully automatic intelligent packing equipment. Background Art

[0002] As society changes and consumer habits evolve, food and daily necessities packaging in the home is moving towards smaller and more convenient directions. However, this trend brings a new challenge: how to effectively pack a large number of small-packaged products into large packaging boxes for easy transportation and storage by merchants.

[0003] Currently, this packing process mostly relies on manual or basic mechanical gripping systems. These systems can usually only move in the horizontal and vertical directions, which is inadequate when dealing with densely stacked packing situations. For example, when the carton is already full of cartons and the space is very limited, it is difficult to accurately place the new carton in the specified position by horizontal and vertical movement control alone. In this case, the manual will use the flexibility of the hand to adjust the angle of the carton so that it can be more easily inserted into the densely arranged space. However, the current mechanical packing technology cannot simulate this complex manual action, and manual operation is inefficient and has high labor costs.

[0004] Therefore, a control system and method for fully automatic intelligent packing equipment are desired. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a control system and method for fully automatic intelligent packing equipment, which first obtains the packing image of small package products collected by the camera and the packing task information collected by the database, and then uses deep learning technology to perform feature extraction and correlation analysis on the two, and finally generates a robot arm motion trajectory through a generator, so as to cope with changes and new task requirements in the packing process, provide instant robot arm motion trajectory and action, improve the speed and accuracy of packing, and thus achieve more flexible and intelligent packing operations.

[0006] According to one aspect of the present application, a control system for a fully automatic intelligent packing device is provided, which includes: A packing control data acquisition module is used to acquire the small package product packing images collected by the camera and the packing task information collected by the database; A packing control data extraction module, used to extract a significant feature vector of a large packing box's free edge and a feature vector of semantic understanding associated with the packing task information from the small package product packing image collected by the camera and the packing task information collected by the database; The robot arm motion trajectory generation module is used to generate the robot arm motion trajectory based on the significant feature vector of the free edge of the large packaging box and the semantic understanding feature vector associated with the packing task information.

[0007] According to another aspect of the present application, a control method for a fully automatic intelligent packing device is provided, comprising: Acquire the small package product packing images collected by the camera and the packing task information collected by the database; Extracting a significant feature vector of a large packaging box's free edge and a feature vector of semantic understanding associated with the packaging task information from the small packaging product packaging image captured by the camera and the packaging task information captured by the database; Based on the significant feature vector of the free edge of the large packaging box and the semantic understanding feature vector associated with the packing task information, a motion trajectory of the robot arm is generated.

[0008] Compared with the prior art, the present application provides a control system and method for fully automatic intelligent packing equipment, which first obtains the small package product packing image collected by the camera and the packing task information collected by the database, and then uses deep learning technology to perform feature extraction and correlation analysis on the two, and finally generates a robot arm motion trajectory through a generator to cope with changes in the packing process and new task requirements, provide instant robot arm motion trajectory and action, improve the speed and accuracy of packing, and thus achieve more flexible and intelligent packing operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0010] Figure 1 It is a block diagram of a control system of a fully automatic intelligent packing device according to an embodiment of the present application.

[0011] Figure 2 It is a block diagram of a packing control data extraction module in a control system of a fully automatic intelligent packing device according to an embodiment of the present application.

[0012] Figure 3 It is a block diagram of a product packing feature extraction unit in a control system of a fully automatic intelligent packing device according to an embodiment of the present application.

[0013] Figure 4 It is a block diagram of a packing task information feature extraction unit in a control system of a fully automatic intelligent packing device according to an embodiment of the present application.

[0014] Figure 5 It is a block diagram of a robot arm motion trajectory generation module in a control system of a fully automatic intelligent packing device according to an embodiment of the present application.

[0015] Figure 6 The present invention is a flowchart of a control method for fully automatic intelligent packing equipment according to an embodiment of the present application.

[0016] Figure 7 is a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0018] Figure 1 : is a block diagram of the control system of the fully automatic intelligent packing equipment of the embodiment of the present application. Figure 1 As shown, the control system 100 of the fully automatic intelligent packing equipment according to the embodiment of the present application includes: a packing control data acquisition module 110, which is used to obtain the small package product packing image collected by the camera and the packing task information collected by the database; a packing control data extraction module 120, which is used to extract the significant feature vector of the free edge of the large package box and the semantic understanding feature vector associated with the packing task information from the small package product packing image collected by the camera and the packing task information collected by the database; a robot arm motion trajectory generation module 130, which is used to generate the robot arm motion trajectory based on the significant feature vector of the free edge of the large package box and the semantic understanding feature vector associated with the packing task information.

[0019] In the control system 100 of the fully automatic intelligent packing equipment, the packing control data acquisition module 110 is used to obtain the packing images of small packaged products collected by the camera and the packing task information collected by the database. It should be understood that with the changes in social and consumption habits, the packaging of household products is moving towards a smaller and more convenient direction. However, this trend brings a new challenge: how to effectively pack a large number of small packaged products for the convenience of transportation and storage by merchants. The current packing process mainly relies on manual or basic mechanical grasping systems. However, these systems face challenges in dealing with densely stacked packing situations because they can usually only move horizontally and vertically. For example, when the carton is already full of small boxes and the space is extremely limited, it is difficult to accurately place the new box in the specified position by relying solely on horizontal and vertical movement. In this case, manual operation can use the flexibility and adaptability of the hand to adjust the angle of the box so that it can be more easily inserted into a highly dense space. However, the current mechanical packing technology cannot imitate this complex manual action, resulting in low efficiency and high labor costs for manual operation. Therefore, in the technical solution of the present application, by acquiring the small package product packing images collected by the camera and the packing task information collected by the database, and combining with deep learning technology, the motion trajectory of the robot arm is generated to respond to changes in the packing process and new task requirements, and provide real-time robot arm motion guidance, thereby improving the packing speed and accuracy, and realizing more flexible and intelligent packing operations.

[0020] Specifically, the images of small packaged products collected by the camera provide real-time visual information. These images reflect the specific products that need to be packed, including important features such as shape, size, and position. By processing and analyzing these images, the system can accurately identify the packing status and position of each small packaged product, thereby providing the necessary data basis for the subsequent packing process. The packing task information stored in the database contains detailed guidance on packing sequence, packing requirements, space utilization, etc. This information is generated by production plans, customer orders, etc., and is highly instructive and real-time. Based on this information, the system can optimize the packing sequence, reasonably arrange the placement of products, and ensure that the packing process meets the predetermined requirements and standards. Combining these two sources of information, the control system of the fully automatic intelligent packing equipment can achieve efficient and accurate packing operations.

[0021] In the control system 100 of the above-mentioned fully automatic intelligent packing equipment, the packing control data extraction module 120 is used to extract the significant feature vector of the spare edge of the large packaging box and the semantic understanding feature vector associated with the packing task information from the small package product packing image collected by the camera and the packing task information collected by the database. It should be understood that by comprehensively utilizing the real-time image collected by the camera and the task information in the database, the system can achieve a high degree of intelligence and automation in the packing process. Through the significant feature vector of the spare edge, the system can effectively plan and optimize the layout of the product, maximize the use of space, and improve the packing efficiency and the quality of the finished product. At the same time, the semantic understanding feature vector associated with the packing task information ensures the system's comprehensive understanding and execution of the packing target, including accurate response to priorities and special requirements, thereby ensuring the efficient operation and smooth completion of the production process. By rationally utilizing and extracting the key feature vectors in the camera image and database information, the fully automatic intelligent packing equipment control system can realize the intelligent management and optimization of the packing process, and provide reliable technical support and solutions for modern industrial production. This technical application not only improves production efficiency and product quality, but also promotes the further improvement and development of the level of industrial automation.

[0022] Figure 2 FIG. 1 is a block diagram of a packing control data extraction module in a control system of a fully automatic intelligent packing device according to an embodiment of the present application. Figure 2 As shown, in a specific embodiment of the present application, the packing control data extraction module 120 includes: a product packing feature extraction unit 121, which is used to perform feature extraction on the small package product packing image collected by the camera to obtain a significant feature vector of the free edge of the large package box; a packing task information feature extraction unit 122, which is used to perform feature extraction on the packing task information collected by the database to obtain a feature vector associated with the packing task information.

[0023] It should be understood that the images of small packaged products collected by the camera are an important source of real-time information about the packaging environment. These images reflect the actual layout inside the large packaging box, including the position and shape of the products that have been packed, as well as the free edges that have not been occupied. By performing feature extraction on these images, the system can extract representative and descriptive features from complex visual data, such as the position, size, and shape of the free edges, to form a significant feature vector of the free edges. The feature extraction process involves advanced image processing and computer vision technology. For example, by applying image processing algorithms, operations such as noise reduction, edge detection, and target recognition can be performed on the image to accurately identify and describe the free space inside the large packaging box.

[0024] Furthermore, feature extraction of the packing task information collected from the database is a key step in achieving efficient operation of the intelligent packing system. The packing task information usually contains a variety of key data, such as product type, quantity, size, packing order, priority requirements, etc. The comprehensive understanding of this information is crucial for the system to correctly perform the packing task. Through feature extraction, the system can extract semantic understanding feature vectors from these data and transform abstract task instructions into specific instructions that can be operated by the machine, thereby ensuring that the packing process proceeds smoothly as expected. The feature extraction process is not limited to simple data extraction, but also includes in-depth analysis of the background and context of the task information. For example, the system may need to identify key words, important instructions, related constraints, etc. from the task information extracted from the database. This information plays a bridging role in semantic understanding and helps the system understand the overall logic and goals of the packing task.

[0025] Figure 3 FIG. 1 is a block diagram of a product packing feature extraction unit in a control system of a fully automatic intelligent packing device according to an embodiment of the present application. Figure 3 As shown, in a specific embodiment of the present application, the product packing feature extraction unit 121 includes: a product packing image denoising subunit 1211, which is used to pass the small package product packing image collected by the camera through a small package product packing denoising model to obtain a small package product packing denoising enhanced image; a product packing denoising feature encoding subunit 1212, which is used to feature encode the small package product packing denoising enhanced image to obtain a large package box spare edge significant feature map; a spare edge significant feature pooling subunit 1213, which is used to pool the large package box spare edge significant feature map to obtain the large package box spare edge significant feature vector.

[0026] It should be understood that the packing images collected by the camera may be affected by factors such as ambient light, camera resolution, and lens distortion, resulting in noise and interference in the image. These noises not only affect the system's accurate identification of the product, but may also affect the accuracy and reliability of subsequent image processing and data analysis. By applying the small package product packing denoising model, the system can remove noise from the image in a targeted manner, improve image quality and clarity, and enable subsequent data analysis and machine vision algorithms to operate based on cleaner and more accurate data. The small package product packing denoising model is usually based on advanced image processing techniques and algorithms, such as digital filtering, wavelet transform, and deep learning. These technologies can effectively identify and separate the noise components in the image, and compensate or remove the noise, while retaining the useful information and features in the image as much as possible. Through model training and optimization, the system can achieve robust processing of packing images under different environmental conditions, ensuring the accuracy of visual perception and real-time data feedback during the packing process. The small package product packing denoising enhanced image not only helps to improve the visual quality of the image, but also optimizes the automated control and decision-making capabilities of the packing equipment. Clear and accurate packing images can provide the system with more accurate product positioning and shape recognition, thereby guiding the precise operation of the robot arm and the effective placement of the product. This refined image processing and enhancement technology makes the packing process more intelligent and efficient, and maximizes the packing speed and the quality stability of product packing. Specifically, the small package product packing image captured by the camera is input into the encoder of the small package product packing denoising model, wherein the encoder uses a convolutional layer to perform explicit spatial encoding on the small package product packing image captured by the camera to obtain image features; and the image features are input into the decoder of the small package product packing denoising model, wherein the decoder uses a deconvolution layer to perform deconvolution processing on the image features to obtain the small package product packing denoising enhanced image.

[0027] Furthermore, feature encoding is performed on the denoised and enhanced images of small packaged products in order to optimize the space utilization and improve the packing efficiency in the intelligent packing system. The denoised and enhanced images of small packaged products can be converted into a more abstract and processable data form through feature encoding. Considering the high dimensionality and complexity of image data itself, direct use may lead to high computational complexity and low processing efficiency. Through feature encoding, the system can extract representative and informative features from the image, such as the position, size, shape, etc. of the free edge, and encode these features into vectors or feature maps to facilitate the optimization of packing strategies. The feature encoding process includes advanced computer vision technology and machine learning algorithms. For example, convolutional neural networks (CNNs) can be used to extract spatial features in images, or local feature descriptors (such as SIFT, SURF, etc.) can be applied to identify and encode key information in images. These technologies can effectively capture the significant features of the free edges of large packaging boxes, provide the system with accurate spatial layout data, and help achieve the optimal utilization of packing space and maximize packing efficiency.

[0028] Furthermore, the significant feature map of the free edge of the large packaging box is pooled in order to further optimize the data representation and reduce the data dimension, while retaining key information for the system's rapid processing and decision-making. This process involves commonly used techniques in the field of deep learning and data processing, which are specifically reflected in the following key aspects. It should be understood that the significant feature map of the free edge of the large packaging box is usually obtained through feature extraction and encoding steps. These feature maps may have high dimensions and complex data structures, and direct use may increase the computational burden and processing time of the system. Through the pooling operation, the spatial size of the feature map can be reduced while retaining the important information of each area in the feature map. For example, maximum pooling can extract the most significant feature value from each feature map area, and average pooling can calculate the average feature value of each area. These operations effectively reduce the complexity of the data and make subsequent data processing more efficient.

[0029] In a specific embodiment of the present application, the product packing denoising feature encoding subunit 1211 includes: passing the small package product packing denoising enhanced image through a large package box free area edge target detection network to obtain multiple large package box free areas of interest; passing the multiple large package box free areas of interest through a large package box free edge convolutional neural network model containing a significant target detector to obtain the large package box free edge significant feature map.

[0030] It should be understood that after preprocessing and feature extraction, the denoised and enhanced image of the small packaged product packaging still requires further analysis and processing to determine the free area in the large packaging box suitable for placing the product. The large packaging box free area edge target detection network can effectively identify the edges and free areas of the large packaging box in the image, and generate bounding boxes or masks of multiple regions of interest. These regions of interest represent the space that can be used by the system, providing important spatial information for subsequent product layout and packing strategies. The target detection network is usually based on deep learning models, such as target detection algorithms based on convolutional neural networks (CNN) (such as YOLO, Faster R-CNN, etc.). These models can accurately locate and identify the edges and free areas of large packaging boxes in the image, and have high accuracy and robustness. Specifically, the large packaging box free area edge target detection network is used to process the small packaged product packaging denoised and enhanced image using the following target detection formula to obtain the multiple large packaging box free interest regions; Among them, the target detection formula is:

[0031] in, De-noise and enhance images for small packaged products. is the anchor frame, Spare areas of interest for multiple large boxes, Indicates classification, Indicates regression.

[0032] Furthermore, the large packaging box free regions of interest are passed through a large packaging box free edge convolutional neural network model containing a salient object detector, aiming to further refine the extraction of spatial information and the optimization of the packing strategy. This process combines the object detection and feature extraction techniques in deep learning, specifically involving the following key aspects. Multiple large packaging box free regions of interest are usually generated by an object detection network, which represent the space that the system can use in the large packaging box. However, simple region detection cannot provide detailed information about the internal features of each region, such as edge clarity, shape accuracy, etc. By introducing a salient object detector, significant features such as clearly available edges, corners or spatial structures can be further detected and identified inside these regions of interest. Large packaging box free edge convolutional neural network models are usually built under advanced deep learning frameworks, such as CNN-based network structures. These models can learn feature representations specific to packaging scenarios, such as visual features of edges and free regions through training. The model uses convolutional layers and pooling layers to extract spatial features, and combines the output of the object detector to further optimize and refine the feature representation inside the free region of each large packaging box. The salient object detector can help the system distinguish and identify different types of large packaging box free edge features, such as free space of different shapes, sizes and positions. This detailed feature extraction not only helps the packing system to accurately place products, but also optimizes the packing strategy to reduce space waste and unnecessary rearrangement during packing. Specifically, each layer of the large packaging box free edge convolutional neural network model containing the salient target detector performs the following operations on the input data in the forward pass of the layer: convolution processing is performed on the input data using a first convolution kernel to obtain a first convolutional feature map; convolution processing is performed on the first convolutional feature map using a second convolutional kernel to obtain a second convolutional feature map, wherein the size of the first convolution kernel is larger than the size of the second convolution kernel; pooling processing is performed on the second convolutional feature map to obtain a pooled feature map; and activation processing is performed on the pooled feature map to obtain an activation feature map; wherein the output of the last layer of the large packaging box free edge convolutional neural network model containing the salient target detector is the large packaging box free edge salient feature map, and the input of the first layer of the large packaging box free edge convolutional neural network model containing the salient target detector is the multiple large packaging box free regions of interest.

[0033] Figure 4 FIG. 1 is a block diagram of a packing task information feature extraction unit in a control system of a fully automatic intelligent packing device according to an embodiment of the present application. Figure 4As shown, in a specific embodiment of the present application, the packing task information feature extraction unit 122 includes: a packing task information data preprocessing subunit 1221, used to perform data preprocessing on the packing task information collected by the database to obtain a two-dimensional feature matrix of the packing task information; a packing task information feature encoding subunit 1222, used to pass the two-dimensional feature matrix of the packing task information through a packing task information text convolutional neural network to obtain a feature vector associated with the packing task information.

[0034] It should be understood that packing task information usually includes multiple data types and formats, such as product size, quantity, priority, destination, etc. This information may come from multiple sources, such as order systems, inventory management systems, or logistics platforms. In practical applications, these data may have missing values, erroneous values, or inconsistent formats. Therefore, the first step in data preprocessing is data cleaning, which ensures the integrity and consistency of the data by processing missing values, outliers, and duplicate data. The original packing task information collected from the database may require feature extraction and transformation to facilitate the effective application of machine learning models. Feature extraction involves extracting meaningful features from the raw data, such as product size features (length, width, height), quantity features, importance features (priority), packing restriction features (maximum weight, maximum volume), etc.

[0035] Furthermore, the two-dimensional feature matrix of packing task information is passed through the packing task information text convolutional neural network, aiming to further deepen the semantic understanding and contextual analysis of packing task information, so as to support decision-making and optimization processes in the intelligent packing system. The two-dimensional feature matrix of packing task information usually contains various key features and constraints of packing tasks, such as product size, quantity, priority, etc. However, this information still exists in a structured numerical form, lacking a deep understanding of semantics and context. Through the packing task information text convolutional neural network, these structured features can be converted into more semantic representations, so that the system can understand and analyze the background and purpose of the packing task. Text convolutional neural networks (CNNs) have been widely used in natural language processing, especially in tasks such as text classification, semantic understanding, and sentiment analysis. This network structure can effectively capture local features and grammatical structures in text, and perform multi-level and multi-scale semantic analysis on the description of packing task information through convolutional layers and pooling layers. For example, the convolution operation can identify and extract keyword combinations or phrases of a specific length, and the pooling operation can reduce the dimension of the feature and retain the most important semantic information. Specifically, each layer of the packing task information text convolutional neural network is used to perform convolution processing, mean pooling processing based on the local feature matrix and nonlinear activation processing on the input data in the forward pass of the layer, so that the packing task information associated semantic understanding feature vector is output by the last layer of the packing task information text convolutional neural network, wherein the input of the packing task information text convolutional neural network is the packing task information two-dimensional feature matrix.

[0036] In a specific embodiment of the present application, the packing task information data preprocessing subunit 1221 includes: passing the packing task information collected from the database through a packing task information semantic encoding model to obtain a plurality of packing task information semantic feature vectors; arranging the plurality of packing task information semantic feature vectors according to sample dimensions to obtain the packing task information two-dimensional feature matrix.

[0037] It should be understood that packing task information usually contains various complex data and constraints, such as product size, quantity, importance, etc. Although this information can be stored in the database in a structured way, the simple numerical representation lacks sufficient semantic information and is difficult to support the system to perform deeper analysis and decision-making when processing and understanding packing tasks. By introducing the packing task information semantic encoding model, these structured data can be converted into higher-level semantic feature vectors, enabling the system to better understand and process the semantic meaning of packing tasks. Packing task information semantic encoding models are usually based on deep learning architectures, such as recurrent neural networks (RNNs), long short-term memory networks (LSTMs), or transformers. These models can handle the sequential nature of text data, thereby capturing the context and contextual information in the packing task description. By learning the semantic representation of the packing task information, the model can effectively convert the text description into a vector representation with semantic richness, reflecting the important features and constraints of the packing task. Specifically, the packing task information collected from the database is segmented to obtain a packing task word sequence; the embedding layer of the packing task information semantic encoding model is used to map each packing task word in the packing task word sequence into a packing task word embedding vector to obtain a sequence of packing task word embedding vectors; the sequence of the packing task word embedding vectors is semantically encoded based on global context using the converter-based Bert model of the packing task information semantic encoding model to obtain multiple packing task information semantic feature vectors.

[0038] Furthermore, the multiple packing task information semantic feature vectors are arranged according to the sample dimension in order to effectively convert the complex semantic information of each packing task into a data format that can be processed by the machine learning model, thereby supporting the application of the system in packing decision and optimization. Among them, the multiple packing task information semantic feature vectors reflect the key semantic features of each packing task, such as the size, quantity, importance, etc. of the product. These feature vectors are usually generated by the packing task information semantic encoding model. Each vector contains the specific description and semantic information of the packing task, but they are separate vectors and are not convenient to be directly input into the machine learning model for training and analysis. By arranging these vectors according to the sample dimension, each packing task can be used as a sample, and its semantic features can be combined into a complete data structure, namely, the packing task information two-dimensional feature matrix. The packing task information two-dimensional feature matrix arranges the semantic feature vectors of each packing task in rows, and each row represents the semantic description of a packing task. This structured data format is very suitable for the training and application of machine learning models, such as classification, regression or cluster analysis. Each column corresponds to a different semantic feature, such as product size, quantity, priority, etc. This structure enables the system to more effectively understand and process the complex characteristics of the packing task. Through such a matrix structure, the system can easily compare, analyze and predict the packing tasks, such as selecting the best packing solution or making optimization decisions based on similarity. This method not only improves the decision-making efficiency of the system, but also reduces errors and waste in the packing process.

[0039] In the control system 100 of the fully automatic intelligent packing equipment, the robot arm motion trajectory generation module 130 is used to generate the robot arm motion trajectory based on the significant feature vector of the free edge of the large packaging box and the semantic understanding feature vector associated with the packing task information. It should be understood that the significant feature vector of the free edge of the large packaging box is an important feature extracted from the real-time image of the large packaging box by computer vision technology. These feature vectors usually include information such as the position, size, shape, etc. of the free space in the box, and can accurately describe the available space inside each large packaging box. This information is crucial for the intelligent packing system because it directly affects how to effectively arrange and place goods to maximize the use of space. The semantic understanding feature vector associated with the packing task information provides a detailed description and semantic information about each packing task, such as product size, quantity, importance, etc. These vectors are extracted from the text description by natural language processing technology, and can effectively express the requirements and constraints of each packing task. Combining these two types of feature vectors, the system can generate the motion trajectory of the robot arm in real-time or predictive mode to achieve automated packing operations. Specifically, by analyzing the significant feature vectors of the free edges of large packaging boxes, the system can determine the optimal location and posture of the goods to maximize the use of space and ensure the safety and stability of packing. At the same time, combined with the semantic understanding feature vector associated with the packing task information, the system can adjust the motion trajectory of the robot arm to meet the specific needs of each packing task, such as placing high-priority goods, avoiding product damage, or ensuring the correctness of the packing order. This method of generating the motion trajectory of the robot arm based on feature vectors not only improves the degree of automation of the packing process, but also significantly reduces human intervention and errors, thereby reducing the cost and time of packing operations. In addition, through real-time monitoring and feedback mechanisms, the system can also adjust the motion trajectory according to actual conditions to further optimize packing efficiency and overall logistics management.

[0040] Figure 5 FIG. 1 is a block diagram of a robot arm motion trajectory generation module in a control system of a fully automatic intelligent packing device according to an embodiment of the present application. Figure 5 As shown, in a specific embodiment of the present application, the robot arm motion trajectory generation module 130 includes: a robot arm packing motion feature fusion unit 131, which is used to fuse the significant feature vector of the free edge of the large packaging box and the semantic understanding feature vector associated with the packing task information to obtain a robot arm packing motion feature vector; a robot arm packing motion feature optimization unit 132, which is used to perform a back-propagation expression correction based on metric learning on the robot arm packing motion feature vector to obtain an optimized robot arm packing motion feature vector; a motion trajectory generation unit 133, which is used to pass the optimized robot arm packing motion feature vector through a generator to generate a robot arm motion trajectory.

[0041] It should be understood that the fusion of the significant feature vector of the free edge of the large packaging box and the feature vector of the semantic understanding of the packaging task information can generate the feature vector of the robot arm packaging motion. These feature vectors integrate the physical space characteristics of the current packaging environment and the semantic requirements of each packaging task. Specifically, they may include information such as suitable placement, posture adjustment, and motion path planning. Through these feature vectors, the system can accurately determine the best placement of each cargo in the large packaging box and the optimal motion trajectory of the robot arm, thereby achieving efficient automatic packaging operations. This method not only improves the automation and efficiency of the packaging process, but also reduces human intervention and errors, and reduces operating costs and time. In addition, the method of fusing feature vectors can also be dynamically adjusted through real-time monitoring and feedback mechanisms to cope with complex packaging scenarios and changing needs, and further optimize the accuracy and reliability of packaging operations.

[0042] In particular, in the technical solution of the present application, considering that the image captured by the camera may contain noise, such as uneven lighting, blurred images, etc., these noises may still exist after being processed by the denoising model, affecting the accuracy of subsequent feature extraction. The edge target detection network of the free area of ​​a large packaging box may not be able to accurately identify all free areas with 100%, especially when the internal layout of the packaging box is complex or there is occlusion. The salient target detector may not be able to fully capture all the salient features related to the packing task, resulting in an incomplete feature map. When the pooling operation is performed on the salient feature map, some detail information useful for the packing task may be lost, resulting in an abnormal distribution of the feature vector. If there is a local abnormal distribution of the feature vector of the robot arm packing motion, the generator may not be able to correctly learn the feature distribution of the normal packing motion during the training process, resulting in a bias towards outliers when generating the motion trajectory, that is, category outlier offset. Category outlier offset will affect the prediction accuracy and generalization ability of the generator, so that the generated robot arm motion trajectory may not meet the actual packing requirements, affecting the packing efficiency and accuracy. In order to solve this problem, in the technical solution of the present application, the robot arm packing motion feature vector is modified by back-propagation expression based on metric learning to obtain an optimized robot arm packing motion feature vector.

[0043] Among them, the robot arm packing motion feature vector is modified by back-propagation expression based on metric learning to obtain an optimized robot arm packing motion feature vector, including: determining the decoding weight matrix of the generator; performing matrix decoupling on the decoding weight matrix in units of row vectors to obtain a set of decoding weight vectors; calculating the metric learning loss information item between each decoding weight vector in the set of decoding weight vectors and the robot arm packing motion feature vector to obtain a metric learning compensation representation vector; inputting the metric learning compensation representation vector into An activation function is used to obtain a probabilistic metric learning compensation weight vector; the positional point multiplication of the probabilistic metric learning compensation weight vector and the robot arm packing motion feature vector is calculated to obtain a point product feature vector, and a weighted sum of the point product feature vector and the robot arm packing motion feature vector is calculated to obtain the optimized robot arm packing motion feature vector.

[0044] The metric learning loss information item between each decoding weight vector in the set of decoding weight vectors and the robot arm packing motion feature vector is calculated using the following compensation formula to obtain a metric learning compensation representation vector; Wherein, the compensation formula is:

[0045] in, represents the first in the set of decoding weight vectors A decoding weight vector; represents the first in the set of decoding weight vectors The decoding weight vector The eigenvalues ​​at the positions, represents the robot arm packing motion feature vector, The first character vector representing the robot arm packing motion The eigenvalues ​​at the positions, represents the length of the feature vector, represents the metric learning compensation representation vector The feature value at each position.

[0046] That is, the robot arm packing motion feature vector is modified by back propagation expression based on metric learning, which first determines the decoding weight matrix of the generator to understand the generator distinguishing different categories of features. Subsequently, the decoding weight matrix is ​​decoupled by row vector to obtain a set of decoding weight vectors to allow the model to independently analyze the weight of each category and provide more refined control for feature compensation. Next, the metric learning loss information item is calculated. This process involves quantitative analysis of the interaction between each category weight vector and the robot arm packing motion feature vector. Through the calculation of a specific loss function, the inconsistency between the feature vector and the category weight is captured, providing a quantitative indicator for compensation. Next, the metric learning compensation representation vector is input into the activation function, nonlinearly transformed, and converted into a probabilistic form. The application of the activation function introduces nonlinearity, enhances the model's ability to interpret the compensation weight, and makes the compensation weight more consistent with the actual distribution characteristics. Then, the probabilistic metric learning compensation weight vector is calculated using the metric learning compensation representation vector processed by the activation function. This step provides probabilistic guidance for the adjustment of the feature vector in a probabilistic form, making the optimization process more consistent with statistical characteristics, thereby enhancing the accuracy of the model's feature adjustment. Finally, the optimized robotic arm packing motion feature vector is obtained by calculating the dot product of the probabilistic metric learning compensation weight vector and the robotic arm packing motion feature vector, and then calculating the weighted sum with the original robotic arm packing motion feature vector. This step combines the original feature information and the compensation information to generate an adjusted feature vector, reducing the impact of abnormal distribution and improving the accuracy of generation.

[0047] Furthermore, the optimized robot arm packing motion feature vector is obtained by fusing the significant feature vector of the free edge of the large packaging box and the feature vector of semantic understanding associated with the packing task information. These feature vectors integrate the physical spatial characteristics of the current packing environment and the semantic requirements of each packing task, including information such as suitable placement, posture adjustment and motion path planning. Each feature vector represents the action guidance of the robot arm when performing a specific packing task. Through the generator, the optimized robot arm packing motion feature vector is converted into an actual robot arm motion trajectory. The generator may use a deep learning model, such as a generative adversarial network (GAN) or a variational autoencoder (VAE), and a traditional path planning algorithm to predict and generate the motion trajectory of the robot arm in three-dimensional space. These trajectories not only take into account the specific requirements of the packing task, but also integrate the physical limitations and safety considerations of the packing environment. The method of generating robot arm motion trajectories through the generator has brought significant technological progress and application potential to modern logistics and production management. This technology is not only suitable for automated packing systems, but can also be widely used in other industrial automation fields that require precise control and efficient motion, providing enterprises with more reliable and intelligent solutions.

[0048] In summary, the embodiment of the present application first obtains the small package product packing image collected by the camera and the packing task information collected by the database, and then uses deep learning technology to perform feature extraction and correlation analysis on the two, and finally generates the robot arm motion trajectory through the generator, so as to cope with changes in the packing process and new task requirements, provide real-time robot arm motion trajectory and action, improve the speed and accuracy of packing, and thus achieve more flexible and intelligent packing operations.

[0049] As described above, the control system 100 of the fully automatic intelligent packing equipment according to the embodiment of the present application can be implemented in various terminal devices. In one example, the control system 100 of the fully automatic intelligent packing equipment can be integrated into the terminal device as a software module and / or a hardware module. For example, the control system 100 of the fully automatic intelligent packing equipment can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the control system 100 of the fully automatic intelligent packing equipment can also be one of the many hardware modules of the terminal device.

[0050] Alternatively, in another example, the control system 100 of the fully automatic intelligent packing equipment and the terminal device may be separate devices, and the control system 100 of the fully automatic intelligent packing equipment may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0051] Figure 6 Flow chart of the control method of the fully automatic intelligent packing equipment according to the embodiment of the present application. Figure 6 As shown, according to the control method of the fully automatic intelligent packing equipment of the embodiment of the present application, it includes: S110, acquiring the small package product packing image collected by the camera and the packing task information collected by the database; S120, extracting the significant feature vector of the free edge of the large package box and the semantic understanding feature vector associated with the packing task information from the small package product packing image collected by the camera and the packing task information collected by the database; S130, generating the motion trajectory of the robot arm based on the significant feature vector of the free edge of the large package box and the semantic understanding feature vector associated with the packing task information.

[0052] Here, those skilled in the art can understand that the specific operations of each step in the control method of the above-mentioned fully automatic intelligent packing equipment have been referred to above. Figures 1 to 5 The control system of the fully automatic intelligent packing equipment has been introduced in detail, and therefore, its repeated description will be omitted.

[0053] Below, reference Figure 7 To describe an electronic device according to an embodiment of the present application.

[0054] Figure 7 A block diagram of an electronic device according to an embodiment of the present application is illustrated.

[0055] like Figure 7 As shown, the electronic device 10 includes an input device 11, an input interface 12, a central processing unit 13, a memory 14, an output interface 15, an output device 16, and a bus 17. The input interface 12, the central processing unit 13, the memory 14, and the output interface 15 are connected to each other through the bus 17, and the input device 11 and the output device 16 are connected to the bus 17 through the input interface 14 and the output interface 15, respectively, and then connected to other components of the electronic device 10.

[0056] Specifically, the input device 11 receives input information from the outside and transmits the input information to the central processing unit 13 through the input interface 12; the central processing unit 13 processes the input information based on the computer executable instructions stored in the memory 14 to generate output information, temporarily or permanently stores the output information in the memory 14, and then transmits the output information to the output device 16 through the output interface 15; the output device 16 outputs the output information to the outside of the electronic device 10 for user use.

[0057] In one embodiment, Figure 7 The electronic device 10 shown can be implemented as a network device, which may include: a memory configured to store programs; a processor configured to run the programs stored in the memory to execute any one of the control methods for fully automatic intelligent packing equipment described in the above embodiments.

[0058] According to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network, and / or installed from a removable storage medium.

[0059] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0060] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present application, but the present application is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of the present application, and these modifications and improvements are also considered to be within the scope of protection of the present application.

Claims

1. A control system for fully automatic intelligent packing equipment, characterized in that: include: A packing control data acquisition module is used to acquire the small package product packing images collected by the camera and the packing task information collected by the database; A packing control data extraction module, used to extract a significant feature vector of a large packing box's free edge and a feature vector of semantic understanding associated with the packing task information from the small package product packing image collected by the camera and the packing task information collected by the database; The robot arm motion trajectory generation module is used to generate the robot arm motion trajectory based on the significant feature vector of the free edge of the large packaging box and the semantic understanding feature vector associated with the packing task information.

2. The control system of the fully automatic intelligent packing equipment according to claim 1 is characterized in that: The packing control data extraction module comprises: A product packing feature extraction unit, used for extracting features from the small package product packing image captured by the camera to obtain a significant feature vector of the free edge of the large packaging box; The packing task information feature extraction unit is used to extract features from the packing task information collected from the database to obtain a semantic understanding feature vector associated with the packing task information.

3. The control system of the fully automatic intelligent packing equipment according to claim 2 is characterized in that: The product packing feature extraction unit comprises: A product packing image denoising subunit, configured to pass the small package product packing image acquired by the camera through a small package product packing denoising model to obtain a small package product packing denoising enhanced image; A product packaging noise reduction feature encoding subunit is used to perform feature encoding on the small package product packaging noise reduction enhanced image to obtain a large package box free edge significant feature map; The spare edge significant feature pooling subunit is used to pool the spare edge significant feature map of the large packaging box to obtain the spare edge significant feature vector of the large packaging box.

4. The control system of the fully automatic intelligent packing equipment according to claim 3 is characterized in that: The product packing noise reduction feature encoding subunit includes: The denoised and enhanced image of the small packaged products is passed through a large package box free area edge target detection network to obtain a plurality of large package box free areas of interest; The plurality of large packaging box free regions of interest are passed through a large packaging box free edge convolutional neural network model including a salient target detector to obtain a salient feature map of the large packaging box free edge.

5. The control system of the fully automatic intelligent packing equipment according to claim 4 is characterized in that: The packing task information feature extraction unit includes: A packing task information data preprocessing subunit, used for performing data preprocessing on the packing task information collected from the database to obtain a two-dimensional feature matrix of the packing task information; The packing task information feature encoding subunit is used to pass the packing task information two-dimensional feature matrix through the packing task information text convolutional neural network to obtain the packing task information associated semantic understanding feature vector.

6. The control system of the fully automatic intelligent packing equipment according to claim 5 is characterized in that: The packing task information data preprocessing subunit includes: The packing task information collected from the database is passed through a packing task information semantic encoding model to obtain a plurality of packing task information semantic feature vectors; The multiple packing task information semantic feature vectors are arranged according to the sample dimension to obtain the packing task information two-dimensional feature matrix.

7. The control system of the fully automatic intelligent packing equipment according to claim 6 is characterized in that: The robot arm motion trajectory generation module includes: A robot arm packing motion feature fusion unit, used to fuse the large packaging box free edge salient feature vector and the packing task information associated semantic understanding feature vector to obtain a robot arm packing motion feature vector; A robot arm packing motion feature optimization unit, used for performing a back propagation expression correction based on metric learning on the robot arm packing motion feature vector to obtain an optimized robot arm packing motion feature vector; The motion trajectory generating unit is used to pass the optimized robot arm packing motion feature vector through a generator to generate a robot arm motion trajectory.

8. The control system of the fully automatic intelligent packing equipment according to claim 7 is characterized in that: The robot arm packing motion feature optimization unit comprises: Determining a decoding weight matrix of the generator; Decoupling the decoding weight matrix in units of row vectors to obtain a set of decoding weight vectors; Calculating a metric learning loss information item between each decoding weight vector in the set of decoding weight vectors and the robot arm packing motion feature vector to obtain a metric learning compensation representation vector; The metric learning compensation representation vector is input Activation function to obtain a probabilistic metric learning compensation weight vector; Calculate the positional point multiplication of the probabilistic metric learning compensation weight vector and the robot arm packing motion feature vector to obtain a point product feature vector, and calculate the weighted sum of the point product feature vector and the robot arm packing motion feature vector to obtain the optimized robot arm packing motion feature vector.

9. A control method for fully automatic intelligent packing equipment, characterized in that: include: Acquire the small package product packing images collected by the camera and the packing task information collected by the database; Extracting a significant feature vector of a large packaging box's free edge and a feature vector of semantic understanding associated with the packaging task information from the small packaging product packaging image captured by the camera and the packaging task information captured by the database; Based on the significant feature vector of the free edge of the large packaging box and the semantic understanding feature vector associated with the packing task information, a motion trajectory of the robot arm is generated.

10. The control method of the fully automatic intelligent packing equipment according to claim 9, characterized in that: Extracting a significant feature vector of the free edge of a large packaging box and a feature vector of semantic understanding associated with the packaging task information from the packaging image of the small packaging product collected by the camera and the packaging task information collected by the database, including: Performing feature extraction on the small package product packing image captured by the camera to obtain a significant feature vector of the free edge of the large package box; Feature extraction is performed on the packing task information collected from the database to obtain a semantic understanding feature vector associated with the packing task information.

Images