Mechanical hand commodity grabbing system and method based on visual recognition

Abstract

The application provides a mechanical hand commodity grabbing system and method based on visual recognition, and relates to the technical field of equipment control. The system comprises the following steps: collecting image information of commodities in a working area based on a multi-modal visual collection device, analyzing the commodity image information, and obtaining depth information of the commodity image; an intelligent detection module is used for intelligently analyzing the depth information of the commodity image, extracting image features of the commodity image, and determining attribute information of the commodity according to the image features; an optimal grabbing frame parameter acquisition module is used for analyzing the attribute information of the commodity to generate optimal grabbing frame parameters; a spatial pose acquisition module is used for determining spatial pose data of the mechanical hand in grabbing the commodity according to the optimal grabbing frame parameters; and a grabbing control module is used for generating grabbing control instructions of the mechanical hand according to the spatial pose data, and controlling the mechanical hand to grab the commodity according to the grabbing control instructions. The grabbing success rate and work efficiency in a complex scene are significantly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of device control, in particular to a mechanical hand commodity grabbing system and method based on visual recognition. BACKGROUND

[0002] At present, in the fields of modern logistics, warehousing and industrial production, the demand for automatic grabbing and carrying of commodities is increasing;

[0003] Traditional mechanical hand grabbing often relies on pre-set fixed positions or simple sensor detection, which has poor environmental adaptability and low grabbing efficiency, especially in complex lighting or dynamic scenes, recognition errors are prone to occur, and when facing commodities of different shapes, sizes and placement positions, grabbing is inaccurate and inefficient;

[0004] Therefore, in order to overcome the above defects, the present application provides a mechanical hand commodity grabbing system and method based on visual recognition. SUMMARY

[0005] The present application provides a mechanical hand commodity grabbing system and method based on visual recognition, which can automatically adapt to the shape characteristics and spatial posture of different types of commodities by combining intelligent visual perception with deep learning technology, significantly improving the grabbing success rate and work efficiency in complex scenes; through high-precision three-dimensional positioning and intelligent grabbing planning, precise closed-loop operation from visual recognition to mechanical control is realized, greatly reducing the dependence on fixed tooling and manual intervention, providing an efficient and reliable automated solution for modern logistics and intelligent warehousing.

[0006] The present application provides a mechanical hand commodity grabbing system based on visual recognition, comprising:

[0007] A multi-modal visual acquisition and analysis module is used to acquire image information of commodities in a working area based on a multi-modal visual acquisition device, and analyze the commodity image information to obtain depth information of the commodity image;

[0008] An intelligent detection module is used to intelligently analyze the depth information of the commodity image, extract image features of the commodity image, and determine attribute information of the commodity according to the image features;

[0009] An optimal grabbing frame parameter acquisition module is used to analyze the attribute information of the commodity to generate optimal grabbing frame parameters;

[0010] A spatial pose acquisition module is used to determine spatial pose data of the mechanical hand in grabbing the commodity according to the optimal grabbing frame parameters;

[0011] The grabbing control module is configured to generate grabbing control instructions of the robot arm according to the spatial pose data, and control the robot arm to grab the commodity according to the grabbing control instructions.

[0012] The application provides a robot arm commodity grabbing method based on visual recognition.

[0013] The multi-modal visual acquisition device is used to acquire image information of the commodity in the working area, and the image information of the commodity is analyzed to obtain depth information of the image of the commodity.

[0014] The depth information of the image of the commodity is intelligently analyzed to extract image features of the image of the commodity, and the attribute information of the commodity is determined according to the image features.

[0015] The attribute information of the commodity is analyzed to generate optimal grabbing frame parameters.

[0016] The spatial pose data of the robot arm in grabbing the commodity is determined according to the optimal grabbing frame parameters.

[0017] The grabbing control instructions of the robot arm are generated according to the spatial pose data, and the robot arm is controlled to grab the commodity according to the grabbing control instructions.

[0018] Preferably, the robot arm commodity grabbing method based on visual recognition is based on a multi-modal visual acquisition device to acquire image information of a commodity in a working area, and the image information of the commodity is analyzed to obtain depth information of the image of the commodity, comprising:

[0019] The multi-modal visual acquisition device is controlled to simultaneously acquire multi-modal image information based on a preset synchronization signal instruction, and depth data of the corresponding modal image information acquired by each modal visual acquisition device is calculated, and a depth map corresponding to each modal is generated according to the depth data.

[0020] The depth maps corresponding to each modal are mapped into the same three-dimensional coordinate system for data alignment, and the depth maps corresponding to each modal after data alignment are input into a pre-trained depth fusion network for processing to obtain a fused multi-source depth map.

[0021] The fused multi-source depth map is optimized to obtain the final depth information of the image of the commodity.

[0022] Preferably, the robot arm commodity grabbing method based on visual recognition optimizes the fused multi-source depth map, comprising:

[0023] The contour features of the commodity are acquired, and the contour features of the commodity are positioned in the multi-source depth map to determine the contour boundary of the commodity in the multi-source depth map.

[0024] According to the contour boundary of the commodity in the multi-source depth map, the multi-source depth map is divided into a key region and a non-key region, and the area proportions of the key region and the non-key region are determined, and a preset area proportion threshold is obtained;

[0025] The area proportions of the key region and the non-key region are compared with the preset area proportion threshold;

[0026] When the area proportions of the key region and the non-key region are less than or equal to the preset proportion threshold, the non-key region in the multi-source depth map is deleted according to the contour boundary to obtain a target key region;

[0027] When the area proportions of the key region and the non-key region are greater than the preset proportion threshold, the current multi-source depth map is taken as the target key region;

[0028] The contour boundary of the target key region is smoothed, and the fused multi-source depth map is optimized according to the smoothing result.

[0029] Preferably, a mechanical hand commodity grabbing method based on visual recognition is provided, which intelligently analyzes the depth information of a commodity image, extracts image features of the commodity image, and determines attribute information of the commodity according to the image features, including:

[0030] A cross-modal attribute intelligent mapping network is constructed, wherein the cross-modal attribute intelligent mapping network is used to establish a mapping relationship between commodity visual features and physical attributes;

[0031] According to the visual features and the physical features, historical grabbing feedback data is divided to obtain visual-physical sample pairs, and the visual-physical sample pairs are input into the cross-modal attribute intelligent mapping network for training to obtain a target cross-modal attribute intelligent mapping network;

[0032] The depth information of the collected commodity image is input into the target cross-modal attribute intelligent mapping network for intelligent analysis, and an appearance feature vector and a three-dimensional attribute description vector of the current commodity are output, and the physical attribute corresponding to the current commodity is determined according to the appearance feature vector and the three-dimensional attribute description vector of the current commodity;

[0033] The appearance feature vector, the three-dimensional attribute description vector, and the physical attribute are used to determine the attribute information of the commodity.

[0034] Preferably, a mechanical hand commodity grabbing method based on visual recognition is provided, which intelligently analyzes the depth information of a commodity image, extracts image features of the commodity image, and determines attribute information of the commodity according to the image features, including:

[0035] The appearance feature attribute library of the commodity is called, and the appearance feature vector of the current commodity and the three-dimensional attribute description vector are input into the appearance feature attribute library of the commodity for analysis and matching, and the appearance attribute features of the current commodity are output, and the visual capture weight is adjusted based on the multi-head attention mechanism according to the appearance attribute features, and the first commodity grabbing strategy is determined according to the weight adjustment result;

[0036] A historical crawling library is constructed, and a static label of the current commodity is generated based on the physical attribute corresponding to the current commodity, and at the same time, the running track of the current commodity is collected in real time, and the running track trend of the current commodity is determined according to the running track of the current commodity, and the spatial position change feature of the current commodity in the preset time interval is predicted according to the running track trend, and a dynamic attribute label is generated according to the spatial position change feature;

[0037] A static-dynamic label pair is generated based on the static label and the dynamic label of the current target commodity, and the static-dynamic label pair is input into the preset expert library for matching to determine the attribute label of the current commodity, and at the same time, the second commodity grabbing strategy for grabbing the commodity is determined in the preset expert library based on the attribute label of the current commodity;

[0038] The constraint condition of the grabbing frame is determined according to the second commodity grabbing strategy, and the initial grabbing frame is generated according to the preset condition of the grabbing frame, and the best grabbing frame parameter is generated by optimizing the initial grabbing frame according to the first commodity grabbing strategy.

[0039] Preferably, a mechanical hand commodity grabbing method based on visual recognition, the spatial pose data of the mechanical hand in grabbing the commodity is determined according to the best grabbing frame parameter, comprising:

[0040] The first coordinate system of the multi-modal visual acquisition device corresponding to the best grabbing frame parameter is obtained, and the second coordinate system of the mechanical hand is obtained, and the calibration rules of the first coordinate system and the second coordinate system are determined respectively;

[0041] The conversion rule of the first coordinate system and the second coordinate system is determined based on the calibration rule, and the coordinate transformation matrix is constructed based on the conversion rule;

[0042] The grabbing points and the first coordinates corresponding to each grabbing point are determined based on the best grabbing frame parameter, and the pose matrix of the grabbing points is constructed based on the first coordinates corresponding to each grabbing point;

[0043] The coordinates of each grabbing point in the pose matrix are converted based on the transformation matrix, and the target coordinates of each grabbing point in the second coordinate system are obtained based on the coordinate conversion result;

[0044] The opening amplitude and the orientation of the mechanical hand are determined based on the target coordinates, and the spatial pose data of the mechanical hand in grabbing the commodity is determined based on the opening amplitude and the orientation of the mechanical hand.

[0045] Preferably, a mechanical hand commodity grabbing method based on visual recognition determines the spatial pose data of the mechanical hand in grabbing the commodity based on the opening amplitude and orientation of the mechanical hand, comprising:

[0046] The obtained spatial pose data is obtained, and the appearance structure data of the commodity is obtained, and a virtual commodity is constructed in the computer based on the appearance structure data of the commodity;

[0047] Meanwhile, a virtual mechanical hand is constructed, and the virtual mechanical hand is simulated based on the spatial pose data, and the virtual commodity is virtually grabbed based on the simulation result;

[0048] The running state of the mechanical hand and the commodity in the virtual grabbing process is monitored in real time, and the running stability parameter of the mechanical hand and the stability parameter of the commodity being grabbed are determined based on the real-time monitoring result;

[0049] When the running stability parameter and the stability parameter of the commodity being grabbed meet the preset limited indicators respectively, it is determined that the spatial pose data of the mechanical hand in grabbing the commodity is qualified, and the spatial pose data is added with authorization permission;

[0050] Otherwise, the spatial pose data of the mechanical hand in grabbing the commodity is determined again until the preset limited indicators are met.

[0051] Preferably, a mechanical hand commodity grabbing method based on visual recognition generates a grabbing control instruction of the mechanical hand according to the spatial pose data, comprising:

[0052] The obtained spatial pose data is obtained, and the appearance structure data of the commodity is obtained, and a virtual commodity is constructed in the computer based on the appearance structure data of the commodity;

[0053] The execution action of each component in the mechanical hand is quantified based on the sub-pose parameter, the execution action feature of each component is obtained, and the original instruction code of each component is adaptively corrected based on the execution action feature;

[0054] The adaptive correction results of each component are logically associated based on the working process of the mechanical hand, and the grabbing control instruction of the mechanical hand is obtained based on the logical association result.

[0055] Preferably, a mechanical hand commodity grabbing method based on visual recognition controls the mechanical hand to grab the commodity according to the grabbing control instruction, comprising:

[0056] The grabbing control instruction of each component is determined based on the composition of each component in the mechanical hand, and the obtained grabbing control instruction is sent to the corresponding component based on the sending route;

[0057] Based on the result of the delivery, the components are cooperatively controlled, and based on the cooperative control result, the robot is controlled to grab the goods.

[0058] Compared with the prior art, the application has the following advantages:

[0059] By combining intelligent visual perception with deep learning technology, the system can automatically adapt to the shape characteristics and spatial posture of different types of goods, significantly improving the success rate of grabbing and work efficiency in complex scenarios. Through high-precision three-dimensional positioning and intelligent grabbing planning, the system realizes precise closed-loop operation from visual recognition to mechanical control, greatly reducing the dependence on fixed tooling and manual intervention, and provides an efficient and reliable automated solution for modern logistics and intelligent warehousing.

[0060] Other features and advantages of the present application will be described in the following description, and some will become apparent from the description, or will be learned by practice of the present application. The purpose and other advantages of the present application can be achieved and obtained by the structure specifically pointed out in this application document.

[0061] The technical solutions of the present application will be further described in detail below with the help of the accompanying drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0062] The accompanying drawings are used to provide a further understanding of the present application, and constitute a part of the specification, together with the embodiments of the present application, used to explain the present application, and do not constitute a limitation on the present application. In the drawings:

[0063] Figure 1 is a structural diagram of a robot goods grabbing system based on visual recognition in an embodiment of the present application;

[0064] Figure 2 is a flowchart of a robot goods grabbing method based on visual recognition in an embodiment of the present application;

[0065] Figure 3 is a flowchart of step 5 in a robot goods grabbing method based on visual recognition in an embodiment of the present application. DETAILED DESCRIPTION

[0066] The preferred embodiments of the present application will be described below in conjunction with the accompanying drawings, and it should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and do not limit the present application.

[0067] Embodiment 1: The present embodiment provides a robot goods grabbing system based on visual recognition, as shown in Figure 1 , comprising:

[0068] a multi-modal vision acquisition analysis module, configured to acquire image information of a commodity in a working area based on a multi-modal vision acquisition device, and analyze the image information of the commodity to obtain depth information of the image of the commodity;

[0069] an intelligent detection module, configured to intelligently analyze the depth information of the image of the commodity, extract image features of the image of the commodity, and determine attribute information of the commodity according to the image features;

[0070] an optimal grasping frame parameter acquisition module, configured to analyze the attribute information of the commodity to generate optimal grasping frame parameters;

[0071] a spatial pose acquisition module, configured to determine spatial pose data of a robot in grasping the commodity according to the optimal grasping frame parameters;

[0072] a grasping control module, configured to generate grasping control instructions of the robot according to the spatial pose data, and control the robot to grasp the commodity according to the grasping control instructions.

[0073] In this embodiment, the attribute information of the commodity includes: a category of the commodity, positions and sizes of parts of the commodity.

[0074] In this embodiment, the optimal grasping frame parameters are parameters in image coordinates, including pixel coordinates of a grasping point center, a rotation angle (relative to an image horizontal axis) of a grasping frame, and a pixel size of a gripper opening width.

[0075] In this embodiment, the multi-modal is based on the forms of data collected by an RGB camera, a binocular camera, a structured light sensor, a TOF camera, and an infrared camera.

[0076] The above technical solution has the following beneficial effects: through the combination of intelligent visual perception and deep learning technology, the shape characteristics and spatial poses of different categories of commodities can be automatically adapted, and the success rate of grasping and the work efficiency in a complex scene are significantly improved; through high-precision three-dimensional positioning and intelligent grasping planning, precise closed-loop operations from visual recognition to mechanical control are realized, the dependence on fixed tooling and manual intervention is greatly reduced, and a high-efficiency and reliable automatic solution is provided for modern logistics and intelligent warehousing.

[0077] Embodiment 2: Based on embodiment 1, this embodiment provides a robot commodity grasping method based on visual recognition, as shown in Figure 2 The robot commodity grasping method based on visual recognition is applied to the robot commodity grasping system based on visual recognition as described in embodiment 1, and includes the following steps:

[0078] Step 1: Acquire image information of a commodity in a working area based on a multi-modal vision acquisition device, and analyze the image information of the commodity to obtain depth information of the image of the commodity;

[0079] Step 2: Intelligent analysis of the depth information of the product image, extraction of the image features of the product image, and determination of the attribute information of the product according to the image features;

[0080] Step 3: Analysis of the attribute information of the product to generate the best capture frame parameters;

[0081] Step 4: Determining the spatial pose data of the robot hand in capturing the product according to the best capture frame parameters;

[0082] Step 5: Generating the capture control command of the robot hand according to the spatial pose data, and controlling the robot hand to capture the product according to the capture control command.

[0083] The beneficial effects of the above technical solutions are: through the combination of intelligent visual perception and deep learning technology, the shape characteristics and spatial attitude of different categories of products can be automatically adapted, the success rate of capture and work efficiency in complex scenes are significantly improved; through high-precision three-dimensional positioning and intelligent capture planning, precise closed-loop operation from visual recognition to mechanical control is realized, the dependence on fixed tooling and manual intervention is greatly reduced, and a high-efficiency and reliable automatic solution is provided for modern logistics and intelligent warehousing.

[0084] In the embodiment 2, a mechanical hand product capture method based on visual recognition is provided, image information of products in a working area is collected based on a multi-modal visual collection device, and the product image information is analyzed to obtain depth information of the product image, including:

[0085] The multi-modal visual collection device is controlled based on a preset synchronization signal instruction to simultaneously collect multi-modal image information, and depth data of the corresponding modal image information collected by each modal visual collection device is calculated, and a depth map corresponding to each modal is generated according to the depth data;

[0086] The depth maps corresponding to each modal are mapped into the same three-dimensional coordinate system for data alignment, and the depth maps corresponding to each modal after data alignment are input into a pre-trained depth fusion network for processing to obtain a fused multi-source depth map;

[0087] The fused multi-source depth map is optimized to obtain the final depth information of the product image.

[0088] In this embodiment, the preset synchronization signal instruction refers to a pre-set, unified trigger signal. This signal is sent to all visual collection devices of different modalities at the same time, instructing them to collect images at the same time to ensure that all image data is acquired at the same instant.

[0089] In this embodiment, the multi-modal visual acquisition device refers to an acquisition system composed of multiple different types (modalities) of visual sensors, such as a collection of different sensors including a camera capable of acquiring RGB color images and a depth camera capable of acquiring depth information (such as a structured light, binocular, ToF camera), etc.

[0090] In this embodiment, the multi-modal image information refers to various types of image data acquired by different sensors in the multi-modal visual acquisition device, such as two-dimensional color images acquired by an RGB camera and depth maps containing object surface distance information acquired by a depth camera.

[0091] In this embodiment, the depth fusion network refers to a specially designed artificial neural network model trained on a large amount of data, whose core function is to receive multiple depth maps from different modalities that have been spatially aligned as input, and through the learning and computing power of the neural network, fuse the advantages of these inputs, output a depth map with higher quality and more completeness. The network architecture of the depth fusion network is a convolutional neural network based on an encoder-decoder structure, the encoder part is composed of multiple convolutional layers and max-pooling layers, used to extract features of each modality depth map layer by layer and reduce dimension; the decoder part is composed of multiple deconvolutional layers and up-sampling layers, used to restore the spatial dimension step by step and generate the fused depth map. In order to preserve multi-scale features, skip connections are introduced between the corresponding levels of the encoder and the decoder, the high-resolution feature maps of the encoder are concatenated with the up-sampled feature maps of the decoder in the channel, thereby fusing low-level details and high-level semantic information. The training data for training the depth fusion network is trained on a self-built logistics commodity dataset, the training samples include: pairs of depth maps acquired by different modal sensors and spatially aligned as input, and high-quality depth maps obtained by high-precision laser scanners or manually refined as supervision labels; the loss function is a combination of mean square error loss function and structural similarity loss function as the optimization objective in the training process. Mean square error loss: calculates the pixel-by-pixel difference between the output fused depth map and the supervision label. Structural similarity loss: measures the similarity of output and label in brightness, contrast and structure. The training process is as follows: input the preprocessed training data into the network, and use the Adam optimizer for iterative optimization. Learning rate decay and early stopping strategies are used in the training process to prevent overfitting. Finally, when the loss on the validation set no longer decreases significantly, the training is stopped, the model parameters are saved, and the pre-trained depth fusion network is obtained.

[0092] In this embodiment, the multi-source depth map refers to the final depth map generated after processing by the depth fusion network, which fuses multiple sources of information. The multi-source emphasizes that the data comes from the aforementioned multiple different modal sensors.

[0093] The beneficial effects of the above technical solution are: by presetting the synchronization signal instruction to control the multi-modal visual acquisition device to collect simultaneously, the consistency of multi-source image data in time is ensured, the misalignment problem caused by object movement or environmental change is effectively avoided, an accurate data foundation is laid for subsequent depth fusion, at the same time, by mapping each modal depth map to the same three-dimensional coordinate system for alignment, the spatial difference between different sensors is eliminated, high-precision registration of data is realized, the pre-trained depth fusion network is used to process the aligned data, which can intelligently complement the advantages of each modal (such as supplementing missing areas and correcting error points), generate a multi-source depth map with more complete details, less noise and higher precision, and finally obtain more reliable and accurate commodity image depth information through optimization processing, which greatly improves the positioning accuracy and success rate of subsequent mechanical hand grabbing, sorting and other operations.

[0094] In the embodiment 3, based on the embodiment 3, the embodiment 4 provides a mechanical hand commodity grabbing method based on visual recognition, which optimizes the fused multi-source depth map, comprising:

[0095] Collecting commodity contour features and positioning in the multi-source depth map according to the commodity contour features to determine the contour boundary of the commodity in the multi-source depth map;

[0096] According to the contour boundary of the commodity in the multi-source depth map, the multi-source depth map is divided into a key area and a non-key area, and the area proportion of the key area and the non-key area is determined, and a preset area proportion threshold is obtained;

[0097] Comparing the area proportion of the key area and the non-key area with the preset area proportion threshold;

[0098] When the area proportion of the key area and the non-key area is less than or equal to the preset proportion threshold, the non-key area in the multi-source depth map is deleted according to the contour boundary to obtain a target key area;

[0099] When the area proportion of the key area and the non-key area is greater than the preset proportion threshold, the current multi-source depth map is taken as a target key area;

[0100] The contour boundary of the target key area is smoothed, and the optimization of the fused multi-source depth map is completed according to the smoothing result.

[0101] In this embodiment, the commodity contour features refer to feature information that can describe the external shape and boundary of the commodity extracted from the commodity image based on edge detection algorithm.

[0102] In this embodiment, the multi-source depth map refers to a single depth image containing scene depth information generated after the data collected by multiple visual sensors is fused.

[0103] In this embodiment, the key region refers to an image region in the multi-source depth map that is related to the target commodity and contains its effective depth information.

[0104] In this embodiment, the non-key region refers to a background or other irrelevant depth information region in the multi-source depth map other than the key region.

[0105] In this embodiment, the preset region proportion threshold refers to a pre-set critical value for judging the area proportion of the key region and the non-key region, which is used to decide whether to retain the complete depth map or only retain the key region of the commodity. The setting basis and specific implementation are as follows:

[0106] The main purpose of the preset region proportion threshold is to balance between computational efficiency and information integrity. If the proportion of the key region is too small, deleting a large amount of redundant non-key region (background) can significantly reduce the data volume for subsequent processing and improve system response speed. The determination of this threshold is based on statistical analysis of a large number of typical warehouse commodity images, and takes into account the minimum requirement of the subsequent processing module (such as feature extraction) for contextual information. The preset region proportion threshold is a fixed value, which is set to 0.6 (i.e. 60%) through experimental verification on historical data sets. That is, when the proportion of the area of the key region of the commodity to the entire depth map is ≤ 60%, the system deletes the non-key region; when the proportion is > 60%, the entire depth is retained. In actual application, it can also be dynamically adjusted based on the preliminary type of the commodity, for example: when the system preliminarily identifies the commodity as "small items", the threshold can be set to 0.4; when it is identified as "standard box", the threshold is set to 0.65. By comparing the area proportion of the key region and the non-key region with this threshold, the system intelligently decides whether to crop the background, thereby optimizing the computational resources while ensuring the integrity of the information.

[0107] In this embodiment, the target key region refers to the core region finally determined to be subjected to smoothing and other optimization operations after the proportion decision.

[0108] The beneficial effects of the above technical solution are: through accurate positioning of the commodity contour and division of the key and non-key regions, the intelligent focusing of the depth map data is realized. Secondly, by adaptively comparing the region proportion with the threshold, it can intelligently decide whether to retain the full image or delete the redundant non-key region, thereby significantly reducing the data volume for subsequent processing, improving the computational efficiency, and at the same time, smoothing the contour of the target key region, effectively eliminating the contour sawtooth and defects caused by depth map noise or inaccurate segmentation, and finally obtaining an optimized depth map with clearer boundaries and purer data, which provides a reliable guarantee for subsequent high-precision pose calculation.

[0109] In this embodiment, based on the embodiment 2, a mechanical hand commodity grabbing method based on visual recognition is provided, the depth information of the commodity image is intelligently analyzed, the image features of the commodity image are extracted, and the attribute information of the commodity is determined according to the image features, including:

[0110] A cross-modal attribute intelligent mapping network is constructed, wherein the cross-modal attribute intelligent mapping network is used to establish a mapping relationship between the visual features of the commodity and the physical attributes;

[0111] The historical grabbing feedback data is divided according to the visual features and the physical features, the visual-physical sample pairs are obtained, and the visual-physical sample pairs are input into the cross-modal attribute intelligent mapping network for training to obtain a target cross-modal attribute intelligent mapping network;

[0112] The depth information of the collected commodity image is input into the target cross-modal attribute intelligent mapping network for intelligent analysis, and the appearance feature vector and the three-dimensional attribute description vector of the current commodity are output, and the physical attribute corresponding to the current commodity is determined according to the appearance feature vector and the three-dimensional attribute description vector of the current commodity;

[0113] The determination of the attribute information of the commodity is completed according to the appearance feature vector, the three-dimensional attribute description vector and the physical attribute.

[0114] In this embodiment, multi-scene local data is collected, and the multi-scene local data is divided into visual-physical sample pairs, wherein each visual-physical sample pair includes an RGB image, a depth map, a mechanical hand grabbing force-deformation curve, and an artificially labeled physical attribute reference value.

[0115] In this embodiment, the cross-modal attribute intelligent mapping network refers to an artificial intelligence neural network model, and the design purpose is to establish and learn the complex corresponding relationship between the visual modal data (such as image, depth information) of the commodity and the physical attribute modal data (such as mass, friction coefficient) thereof.

[0116] In this embodiment, the visual-physical sample pair refers to a data structure unit for training the network, which is composed of the visual feature data (such as the depth map) of a commodity and the corresponding verified real physical attribute data.

[0117] In this embodiment, the appearance feature vector is a numerical vector, which is an output result obtained after the network analyzes the depth information of the commodity, and is used to digitally represent the visual appearance characteristics such as the form and texture of the surface of the commodity.

[0118] In this embodiment, the three-dimensional attribute description vector is a numerical vector, which is another output result obtained after the network analyzes the depth information of the commodity, and is used to digitally represent the volume, shape, size and other attributes of the commodity in the three-dimensional space.

[0119] In this embodiment, the cross-modal attribute intelligent mapping network is used to establish the mapping relationship from the visual features of the commodity to its physical attributes. The implementation process is as follows: the input of the cross-modal attribute intelligent mapping network is the depth information of the product image, specifically the optimized multi-source depth map, and the output of the network includes two parts: one is the appearance feature vector (a high-dimensional numerical vector used to represent the surface characteristics of the commodity such as color, texture, etc.), and the other is the three-dimensional attribute description vector (a high-dimensional numerical vector used to represent the three-dimensional spatial attributes of the commodity such as size, volume, approximate geometric shape (such as whether it is a cube, cylinder), etc.). The network implicitly establishes the association between the two output vectors and the physical attributes of the commodity (such as mass, center of mass, surface friction coefficient, etc.). The design of the cross-modal attribute intelligent mapping network includes: composed of a shared feature extraction backbone network and parallel branch networks, wherein the feature extraction backbone includes: using a pre-trained ResNet or VGG convolutional neural network, removing its last fully connected layer, and used to extract rich shared visual features from the input depth map. The appearance feature branch is composed of several fully connected layers, which map the global features extracted by the backbone network into appearance feature vectors; the three-dimensional attribute branch is also composed of several fully connected layers, which map the same shared features into three-dimensional attribute description vectors.

[0120] The training method of the cross-modal attribute intelligent mapping network adopts a supervised learning method for training, specifically including: step one: constructing a sample pair, that is, constructing a visual-physical sample pair (that is, composed of the visual feature data (such as depth map) of a commodity and its corresponding verified true physical attribute data) according to the historical crawling feedback data; step two: training according to the sample pair, the training process is to input the depth map into the network to obtain the predicted appearance feature vector and three-dimensional attribute description vector; through an additional regressor (which can be regarded as part of the network), these vectors are mapped to the final physical attribute prediction value; the training target is to minimize the mean square error between the physical attribute prediction value and the true benchmark value. Through end-to-end training, the network learns to extract the most effective features from visual data for physical attribute prediction.

[0121] The beneficial effects of the above technical solution are: by constructing and training the cross-modal attribute intelligent mapping network, the precise and automated mapping relationship from the visual features of the commodity to its physical attributes is successfully established, at the same time, through the network, the knowledge learned from the historical crawling data can be used for intelligent analysis of the depth information of new commodities, and the quantitative vectors of their appearance and three-dimensional attributes can be automatically output, and the physical attributes can be accurately inferred, which significantly reduces the dependence on manual experience labeling, improves the automation degree and accuracy of unknown commodity attribute identification, and provides reliable and quantifiable physical information support for subsequent crawling planning.

[0122] Embodiment 6: Based on Embodiment 2 or 5, this embodiment provides a mechanical hand commodity grabbing method based on visual recognition, which analyzes the attribute information of the commodity to generate the best grabbing box parameters, including:

[0123] The appearance feature attribute library of the commodity is called, and the appearance feature vector and the three-dimensional attribute description vector of the current commodity are input into the appearance feature attribute library of the commodity for analysis and matching, and the appearance attribute features of the current commodity are output, and the visual capture weight is adjusted based on the multi-head attention mechanism, and the first commodity grabbing strategy is determined according to the weight adjustment result;

[0124] The historical grabbing library is constructed, and the static label of the current commodity is generated based on the physical attribute corresponding to the current commodity, and at the same time, the running track of the current commodity is collected in real time, and the running track trend of the current commodity is determined according to the running track of the current commodity, and the spatial position change feature of the current commodity in the preset time interval is predicted according to the running track trend, and the dynamic attribute label is generated according to the spatial position change feature;

[0125] The static-dynamic label pair is generated based on the static label and the dynamic label of the current target commodity, and the static-dynamic label pair is input into the preset expert library for matching to determine the attribute label of the current commodity, and at the same time, the second commodity grabbing strategy for grabbing the commodity is determined in the preset expert library based on the attribute label of the current commodity;

[0126] The constraint condition of the grabbing box is determined according to the second commodity grabbing strategy, and the initial grabbing box is generated according to the preset condition of the grabbing box, and the initial grabbing box is optimized according to the first commodity grabbing strategy to generate the best grabbing box parameters.

[0127] In this embodiment, the specific process of visual capture weight adjustment based on multi-head attention mechanism is as follows: the appearance feature map of the commodity is input into a multi-head attention module. This module calculates the importance weight of each region of the feature map for the grabbing task from different dimensions such as geometric shape, texture structure and spatial context through multiple parallel attention heads. Then, the outputs of all attention heads are fused to generate a comprehensive attention weight heat map, and this heat map is used to weight the original feature map, so as to enhance the feature response of the regions strongly related to the grabbing point (such as handle, edge, and center of gravity nearby), and suppress the interference of irrelevant regions, thereby providing a basis for generating accurate first commodity grabbing strategy.

[0128] In this embodiment, the appearance feature attribute library of the commodity is a pre-constructed database, which stores the standard appearance feature vectors and three-dimensional attribute description vectors of various known commodities, and is used for matching and comparison with newly input commodity data. The commodity appearance feature attribute library is a structured database, and its core data table contains the following fields:

[0129] Product ID (unique identifier), standard appearance feature vector (i.e. the feature vector extracted from the standard product sample by the cross-modal attribute intelligent mapping network), standard three-dimensional attribute description vector (i.e. the three-dimensional attribute vector extracted from the standard product sample by the cross-modal attribute intelligent mapping network), product category (such as carton, bottled water, and soft package), typical physical parameters (such as mass range, recommended clamping force, etc.).

[0130] The appearance feature attribute library of the product supports dynamic updating. The initial construction is completed by batch entering the data of known products for initialization. During the system running process, when a new product is successfully captured, the system can add its confirmed effective feature vector and attribute information as a new entry to the library. At the same time, version management and data cleaning mechanism can be set to optimize the library content regularly.

[0131] In this embodiment, the first product grabbing strategy refers to the preliminary grabbing scheme obtained by matching and attention mechanism calculation according to the appearance visual features (such as shape, texture) of the product, mainly aiming at the static and inherent attributes of the product.

[0132] In this embodiment, the historical grabbing library refers to a database that stores the data related to the past grabbing tasks (such as success / failure records, grabbing parameters, object attributes, etc.), which is used to provide historical experience and data support for the current decision.

[0133] In this embodiment, the static label refers to an identifier used to describe the physical attributes of the product that are stable and unchanging (such as mass, material, inherent shape).

[0134] In this embodiment, the dynamic attribute label refers to an identifier used to describe the attributes (such as instantaneous speed, motion trend, predicted pose) exhibited by the product during the current motion process, which change over time.

[0135] In this embodiment, the static-dynamic label pair refers to a data pair composed of a static label and a dynamic attribute label, which is used to comprehensively represent the complete state of an object at a specific moment.

[0136] In this embodiment, the first product grabbing strategy is based on static appearance features, the second product grabbing strategy is based on dynamic attributes, and the optimization is mainly constrained by the second product grabbing strategy, with the first product grabbing strategy as fine-tuning.

[0137] In this embodiment, the preset expert library refers to a knowledge base or rule engine system encapsulating expert knowledge and rules in the field of grabbing (such as which grabbing method should be used for an object in a certain material or a certain motion state). The preset expert library is a rule-based expert system, and its rules mainly have two sources: first, domain knowledge, which encapsulates expert experience from the fields of robotics, materials science, etc., for example: for a cylindrical object with a smooth surface, the clamping force should be increased to prevent slipping, and for a product labeled as “fragile”, the clamping force should be reduced and envelope grabbing should be preferred during grabbing. The second is historical data mining, which automatically summarizes effective grabbing strategy rules by machine learning (such as decision tree, association rule mining) on successful and failed cases stored in the historical grabbing library. The matching logic is: input: static-dynamic label pair, inference engine: the core component of the expert system; it matches the input label with the rule premise in the library. The matching algorithm uses forward chain reasoning, that is, according to the input facts, all rules that meet the conditions are activated, and their conclusions are integrated; output: generate one or more recommended second commodity grabbing strategies, which include suggestions for adjusting the grabbing parameters (such as grabbing speed, clamping force, obstacle avoidance path, etc.) for the current commodity dynamic state.

[0138] In this embodiment, the second commodity grabbing strategy refers to the grabbing strategy matched by the preset expert library according to the static-dynamic label pair. This strategy takes into account the dynamic environment and static physical properties of the object and is a higher-level decision result.

[0139] The beneficial effects of the above technical solution are: by fusing the static physical properties of the commodity and the dynamic running trajectory, multi-dimensional intelligent decision of the grabbing strategy is realized. First, the appearance feature attribute library and the multi-head attention mechanism are used to generate a first grabbing strategy that adapts to the appearance of the commodity based on the visual features of the commodity, ensuring the static adaptability of the strategy. Then, by constructing a historical grabbing library and analyzing the commodity running trajectory in real time, the future pose change of the commodity is predicted, and a label reflecting its dynamic state is generated. The dynamic-static label pair is input into the expert library for matching to determine the optimal second grabbing strategy that takes into account all current states. Finally, the second strategy based on the dynamic environment is used as a constraint to generate an initial grabbing frame, and the first strategy based on the appearance is used for optimization, so as to obtain the best grabbing parameters that have both environmental adaptability and object specificity, significantly improving the accuracy, adaptability and success rate of grabbing planning in dynamic environment.

[0140] Embodiment 7: Based on embodiment 2, this embodiment provides a mechanical hand commodity grabbing method based on visual recognition, which determines the spatial pose data of the mechanical hand in grabbing the commodity according to the best grabbing frame parameters, comprising:

[0141] The first coordinate system of the multi-modal visual acquisition device corresponding to the optimal grasping frame parameter is obtained, and the second coordinate system of the manipulator is obtained, and the calibration rules of the first coordinate system and the second coordinate system are determined respectively;

[0142] The conversion rule of the first coordinate system and the second coordinate system is determined based on the calibration rule, and the coordinate transformation matrix is constructed based on the conversion rule;

[0143] The grasping points and the first coordinates corresponding to each grasping point are determined based on the optimal grasping frame parameter, and the pose matrix of the grasping points is constructed based on the first coordinates corresponding to each grasping point;

[0144] The coordinates of each grasping point in the pose matrix are converted based on the transformation matrix, and the target coordinates of each grasping point in the second coordinate system are obtained based on the coordinate conversion result;

[0145] The opening amplitude and the orientation of the manipulator are determined based on the target coordinates, and the spatial pose data of the manipulator in grasping the goods is determined based on the opening amplitude and the orientation of the manipulator.

[0146] In this embodiment, the first coordinate system refers to the coordinate system of the multi-modal visual acquisition device itself, and all image data (such as grasping frame parameters) collected by the device are initially defined in this coordinate system.

[0147] In this embodiment, the second coordinate system refers to the base coordinate system of the manipulator, and all motion control and pose description of the manipulator are ultimately based on this coordinate system.

[0148] In this embodiment, the calibration rule refers to the specific operation process and calculation method (such as the steps of hand-eye calibration using a calibration board) required to determine the spatial transformation relationship between the first coordinate system and the second coordinate system.

[0149] In this embodiment, the conversion rule refers to the mathematical rule and formula followed to transform the point or pose data in one coordinate system to another coordinate system under the premise of knowing the spatial relationship of the two coordinate systems (i.e. completing calibration).

[0150] In this embodiment, the coordinate transformation matrix refers to a 4x4 homogeneous transformation matrix, which mathematically represents the rotation and translation relationship from the first coordinate system to the second coordinate system, and is used for accurate coordinate conversion calculation. The purpose of the coordinate transformation matrix is to accurately convert the best grasp box parameters (defined in the camera coordinate system, i.e., the first coordinate system) recognized by the vision system into executable spatial pose data in the robot base coordinate system (i.e., the second coordinate system). The implementation is as follows: in the system deployment stage, the spatial transformation relationship between the camera coordinate system (the first coordinate system) and the robot base coordinate system (the second coordinate system) is accurately obtained through hand-eye calibration technology (for example, using Zhang Zhengyou calibration method to take multiple pictures of the checkerboard). This relationship is encapsulated in a 4x4 homogeneous coordinate transformation matrix, which contains rotation and translation information. Based on the transformation matrix, the coordinates of each grasp point in the pose matrix are converted, specifically by wherein, represents the coordinates of the grasp point in the camera coordinate system, represents the horizontal coordinate value of the grasp point; represents the vertical coordinate value of the grasp point; represents the vertical coordinate value of the grasp point; represents the transpose; represents the base coordinate system; represents the camera grasp point; represents a 4x4 homogeneous coordinate transformation matrix; represents the target coordinates in the robot base coordinate system after conversion; based on the target coordinates of all grasp points, the accurate spatial pose data of the robot end effector when grasping the goods is finally determined through inverse kinematics calculation.

[0151] In this embodiment, the pose matrix refers to a 4x4 homogeneous transformation matrix, which is used to describe the three-dimensional position (X, Y, Z) and three-dimensional orientation (rotation angle) of the grasp point in the first coordinate system. In robotics, pose contains two core elements of the position and orientation of an object in three-dimensional space. Pose matrix is a standard mathematical tool (usually in the form of 4x4 homogeneous transformation matrix), which can accurately describe both types of information at the same time to facilitate coordinate transformation and robot motion control. Specifically, a 4x4 homogeneous transformation matrix is constructed for each grasp point as its pose matrix. The 3x3 submatrix in the upper left corner of the matrix is a rotation matrix , which represents the three-dimensional orientation of the grasp point and is based on the rotation angle in the best grasp box parameters and the surface normal vector of the goods calculated from the depth information; the 3x1 subvector on the far right side of the pose matrix is a translation vector , ​The content is the three-dimensional coordinates of the grabbing point in the first coordinate system (camera coordinate system), which is calculated by back-projection of the pixel coordinates and depth value of the grabbing frame through the intrinsic matrix of the camera. Based on the above description, the complete form of the pose matrix is: .

[0152] In this embodiment, the target coordinates refer to the corresponding three-dimensional position and orientation of the pose data of the grabbing point in the second coordinate system (manipulator base coordinate system) after coordinate transformation.

[0153] The beneficial effects of the above technical solution are: by establishing accurate coordinate system calibration and conversion rules, seamless connection of visual perception and mechanical execution is achieved, the best grabbing point pose detected by the visual device is accurately converted to the manipulator coordinate system using the constructed coordinate transformation matrix, the fundamental problem of cross-system data unification is solved, the target coordinates, opening amplitude and orientation of the manipulator end are automatically calculated, and finally accurate spatial pose data for driving the manipulator motion is generated, which significantly improves the precision, automation and reliability of the grabbing system.

[0154] Embodiment 8: Based on embodiment 7, the embodiment provides a manipulator commodity grabbing method based on visual recognition, determines the spatial pose data of the manipulator in grabbing the commodity based on the opening amplitude and orientation of the manipulator, comprising:

[0155] The obtained spatial pose data, at the same time, the appearance structure data of the commodity is obtained, and a virtual commodity is constructed in the computer based on the appearance structure data of the commodity;

[0156] At the same time, a virtual manipulator is constructed, and the virtual manipulator is simulated based on the spatial pose data, and the virtual commodity is virtually grabbed based on the simulation result;

[0157] The running state of the manipulator and the commodity in the virtual grabbing process is monitored in real time, and the running stability parameter of the manipulator and the stable parameter of the commodity being grabbed are determined based on the real-time monitoring result;

[0158] When the running stability parameter and the stable parameter of the commodity being grabbed meet the preset limited indicators respectively, it is determined that the spatial pose data of the manipulator in grabbing the commodity is qualified, and the spatial pose data is added with authorization permission;

[0159] Otherwise, the spatial pose data of the manipulator in grabbing the commodity is re-determined until the preset limited indicators are met.

[0160] In this embodiment, the virtual commodity refers to a digital and interactive three-dimensional model created in the computer simulation environment according to the actual appearance structure data (such as three-dimensional size, shape) of the commodity.

[0161] In this embodiment, the virtual mechanical finger is a digital model created in a computer simulation environment with the same kinematics and dynamics as the real mechanical hand.

[0162] In this embodiment, the running smoothness parameter refers to a set of indicators for quantitatively evaluating the motion state of the virtual mechanical hand during simulation motion, which may include joint jitter amplitude, speed / acceleration mutation rate, whether severe jitter or collision occurs, etc., and the core indicator is the variance of the angular velocity of each joint of the mechanical hand, i.e., the smaller the angular velocity variance, the smoother the motion, and the qualified threshold is that the angular velocity variance of all joints is less than 0.1 (rad / s)²; this threshold ensures smooth motion of the mechanical arm without unacceptable jitter; in simulation, the angular velocity sequence of each joint is calculated in real time by the physics engine, and the variance is compared with the threshold.

[0163] In this embodiment, the stable parameter refers to a set of indicators for quantitatively evaluating the state of the virtual goods being grabbed during the virtual grabbing process, including whether the clamping force is within a safe range, whether the goods are sliding or overturning, whether the grabbing center is stable, etc., and the core indicator is the clamping force safety factor, i.e., the ratio of the maximum static friction force provided by the mechanical hand to the gravity of the goods (clamping force safety factor=(clamping force x static friction coefficient) / (goods mass x gravitational acceleration)), wherein to ensure reliable grabbing, the clamping force safety factor must be greater than 1; the clamping force is determined by the grabbing control command and is the normal force exerted by the mechanical hand on the goods; the static friction coefficient can be obtained from the appearance feature attribute library; the goods mass is one of the physical attributes of the goods inferred by the cross-modal attribute intelligent mapping network; the gravitational acceleration is a constant, usually taken as g=9.8 m / s². The qualified threshold is greater than 1.5, which ensures that the grabbing action has sufficient anti-sliding ability.

[0164] In this embodiment, the pre-set limit indicator refers to the qualified threshold or range pre-set for the running smoothness parameter and the stable parameter, which is used to objectively judge whether the simulation result meets the standard.

[0165] In this embodiment, the authorized permission refers to a digital identifier or instruction attached to the spatial pose data that has passed the verification, indicating that the data has been verified by simulation and approved to be sent to the real mechanical hand for execution.

[0166] In this embodiment, when the spatial pose data of the mechanical hand during grabbing the goods is unqualified, the spatial pose data of the mechanical hand during grabbing the goods is re-determined, and the re-determined manner is to adjust the spatial pose data through a gradient descent algorithm, re-perform virtual simulation until the pre-set limit indicator is met.

[0167] The beneficial effects of the above technical solutions are: by constructing a high-fidelity virtual environment and a physical model in the computer, the pre-simulation and verification of the mechanical hand grabbing strategy are realized, and secondly, through real-time monitoring and quantitative analysis of the whole virtual grabbing process, the motion stability and grabbing stability caused by the pose data can be accurately evaluated, so as to effectively identify and eliminate the suboptimal scheme with the risk of collision, falling or grabbing failure before being put into practical application, significantly reducing the cost and risk of physical testing, and ensuring the safety and reliability of the spatial pose data finally issued to the real mechanical hand.

[0168] In the embodiment 2, the embodiment 9 provides a mechanical hand commodity grabbing method based on visual recognition, as shown in the following table: Figure 3 As shown in the table, the grabbing control instruction of the mechanical hand is generated according to the spatial pose data, which includes:

[0169] Step 501: obtaining the spatial pose data, and based on the components of the mechanical hand, the spatial pose data is object decomposed to obtain the sub-pose parameters corresponding to each component;

[0170] Step 502: quantifying the execution action of each component in the mechanical hand based on the sub-pose parameters, obtaining the execution action features of each component, and adaptively correcting the original instruction code of each component based on the execution action features;

[0171] Step 503: logically associating the adaptive correction results of each component based on the working process of the mechanical hand, and obtaining the grabbing control instruction of the mechanical hand based on the logical association results.

[0172] In this embodiment, the object decomposition refers to the process of decomposing the composite data describing the overall pose of the mechanical hand end into the rotation or translation parameters required by each joint (component) of the mechanical hand to be independently completed.

[0173] In this embodiment, the sub-pose parameter refers to the parameter assigned to each independent joint (or motion component) of the mechanical hand after object decomposition, which is used to describe the specific motion target (such as rotation angle, displacement).

[0174] In this embodiment, the execution action feature refers to the detailed index obtained after further analyzing the sub-pose parameter, which is used to quantitatively describe how the component executes the action, such as the speed curve of the motion, the acceleration limit, the expected torque, etc.

[0175] In this embodiment, the original instruction code refers to the basic control command or signal generated by the upper control system and sent to the driver of each component of the mechanical hand, which is usually not fine-tuned for this specific task and the current state.

[0176] In this embodiment, the adaptive correction refers to the process of automatically adjusting and optimizing the original instruction code sent to each component according to the requirements proposed by the execution action characteristics, aiming to make the final execution action smoother, more accurate and more stable.

[0177] In this embodiment, logical association refers to the process of ensuring that all components after adaptive correction maintain consistency and coordination in time sequence and logical relationship when generating the final control instruction, avoiding action conflicts or time sequence disorders.

[0178] The beneficial effects of the above technical solutions are: through fine decomposition and component-level quantization of the manipulator spatial pose data, high-precision conversion from macro pose to micro execution instruction is realized, secondly, the overall pose is disassembled into sub-parameters of each joint and optimized independently, effectively eliminating compound motion error and improving the coordination and precision of overall motion, then, according to the adaptive instruction correction of execution action characteristics, individual optimization can be performed according to different component characteristics, effectively suppressing jitter and overshoot, finally, through logical association to integrate the correction results of each component, coordinated and unified, accurate and reliable grasping control instruction is generated, significantly improving the stability, accuracy and success rate of manipulator grasping action.

[0179] Embodiment 10: Based on embodiment 2, this embodiment provides a manipulator commodity grasping method based on visual recognition, which controls the manipulator to grasp the commodity according to the grasping control instruction, comprising:

[0180] Based on the grasping control instruction of each component, the grasping control instruction is sent to the corresponding component based on the delivery route;

[0181] Based on the delivery result, the components are controlled in coordination, and based on the coordination control result, the manipulator is controlled to grasp the commodity.

[0182] In this embodiment, the delivery route refers to the transmission path or addressing rule preset for accurately sending different control instructions to specific components (such as joint motors and gripper controllers) according to the hardware structure and control system layout of the manipulator.

[0183] In this embodiment, the coordination control refers to the process of system overall management of each component after the instruction delivery, ensuring that they cooperate with each other in time sequence and action when executing, so as to jointly complete a smooth and stable grasping action control process.

[0184] The beneficial effects of the above technical solutions are: through intelligent instruction routing distribution and coordination control mechanism, it ensures that the grasping control instruction is accurately and efficiently delivered to each execution component, and realizes the synchronous and coordinated operation of multiple components, significantly improving the accuracy of manipulator grasping operation and the overall system response efficiency.

[0185] Obviously, many modifications and variations of the present application are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A vision recognition based robot product picking system, characterized in that, Comprise: A multi-modal visual acquisition analysis module for acquiring image information of a commodity in a working area based on a multi-modal visual acquisition device and analyzing the commodity image information to obtain depth information of the commodity image; An intelligent detection module for intelligently analyzing the depth information of the commodity image, extracting image features of the commodity image, and determining attribute information of the commodity according to the image features; An optimal grasping frame parameter acquisition module for analyzing the attribute information of the commodity to generate optimal grasping frame parameters; A spatial pose acquisition module for determining spatial pose data of a robot hand in grasping the commodity according to the optimal grasping frame parameters; A grasping control module for generating grasping control instructions of the robot hand according to the spatial pose data and controlling the robot hand to grasp the commodity according to the grasping control instructions; The multi-modal visual acquisition analysis module comprises: Acquiring image information of a commodity in a working area based on a multi-modal visual acquisition device and analyzing the commodity image information to obtain depth information of the commodity image, comprising: Controlling the multi-modal visual acquisition device to simultaneously acquire multi-modal image information based on a preset synchronization signal instruction, and calculating depth data of the image information acquired by each modal visual acquisition device, and generating a depth map corresponding to each modal according to the depth data; Mapping the depth maps corresponding to each modal to the same three-dimensional coordinate system for data alignment, and inputting the depth maps corresponding to each modal after data alignment into a pre-trained depth fusion network for processing to obtain a fused multi-source depth map; Optimizing the fused multi-source depth map to obtain the final depth information of the commodity image; Optimizing the fused multi-source depth map, comprising: Collecting the contour features of the commodity and positioning the commodity in the multi-source depth map according to the contour features to determine the contour boundary of the commodity in the multi-source depth map; Dividing the multi-source depth map into a key area and a non-key area according to the contour boundary of the commodity in the multi-source depth map, and determining the area ratio of the key area and the non-key area and obtaining a preset area ratio threshold; Comparing the area ratio of the key area and the non-key area with the preset area ratio threshold; When the area ratio of the key area and the non-key area is less than or equal to the preset ratio threshold, deleting the non-key area in the multi-source depth map according to the contour boundary to obtain a target key area; When the area ratio of the key area and the non-key area is greater than the preset ratio threshold, the current multi-source depth map is taken as the target key area; Smoothing the contour boundary of the target key area and completing the optimization of the fused multi-source depth map according to the smoothing result.

2. A visual recognition-based robot product picking method, characterized by, Applied to a robot hand commodity grasping system based on visual recognition as claimed in claim 1, comprising: Acquiring image information of a commodity in a working area based on a multi-modal visual acquisition device and analyzing the commodity image information to obtain depth information of the commodity image; Intelligently analyzing the depth information of the commodity image, extracting image features of the commodity image, and determining attribute information of the commodity according to the image features; Analyzing the attribute information of the commodity to generate optimal grasping frame parameters; Determining spatial pose data of a robot hand in grasping the commodity according to the optimal grasping frame parameters; The spatial pose data is used to generate a grabbing control instruction of the robot arm, and the robot arm is controlled to grab the commodity according to the grabbing control instruction; Image information of the commodity in the working area is collected based on a multi-modal vision acquisition device, and the commodity image information is analyzed to obtain depth information of the commodity image, including: The multi-modal vision acquisition device is controlled to collect multi-modal image information based on a preset synchronization signal instruction, and depth data of the corresponding modal image information collected by each modal vision acquisition device is calculated, and a depth map corresponding to each modal is generated according to the depth data; The depth maps corresponding to each modal are mapped into the same three-dimensional coordinate system for data alignment, and the depth maps corresponding to each modal after data alignment are input into a pre-trained depth fusion network for processing to obtain a fused multi-source depth map; The fused multi-source depth map is optimized to obtain the final depth information of the commodity image; The fused multi-source depth map is optimized, including: The commodity contour feature is collected, and the commodity contour feature is positioned in the multi-source depth map to determine the contour boundary of the commodity in the multi-source depth map; The multi-source depth map is divided into a key area and a non-key area according to the contour boundary of the commodity in the multi-source depth map, and the area proportion of the key area and the non-key area is determined, and a preset area proportion threshold is obtained; The area proportion of the key area and the non-key area is compared with the preset area proportion threshold; When the area proportion of the key area and the non-key area is less than or equal to the preset proportion threshold, the non-key area in the multi-source depth map is deleted according to the contour boundary to obtain a target key area; When the area proportion of the key area and the non-key area is greater than the preset proportion threshold, the current multi-source depth map is taken as the target key area; The contour boundary of the target key area is smoothed, and the fused multi-source depth map is optimized according to the smoothing result.

3. The visual recognition-based robot product picking method according to claim 2, wherein, The depth information of the commodity image is intelligently analyzed, the image features of the commodity image are extracted, and the attribute information of the commodity is determined according to the image features, including: A cross-modal attribute intelligent mapping network is constructed, wherein the cross-modal attribute intelligent mapping network is used to establish a mapping relationship between the visual features and the physical attributes of the commodity; The historical grabbing feedback data is divided according to the visual features and the physical features to obtain a visual-physical sample pair, and the visual-physical sample pair is input into the cross-modal attribute intelligent mapping network for training to obtain a target cross-modal attribute intelligent mapping network; The collected depth information of the commodity image is input into the target cross-modal attribute intelligent mapping network for intelligent analysis, and the appearance feature vector and the three-dimensional attribute description vector of the current commodity are output, and the physical attribute corresponding to the current commodity is determined according to the appearance feature vector and the three-dimensional attribute description vector of the current commodity; The attribute information of the commodity is determined according to the appearance feature vector, the three-dimensional attribute description vector and the physical attribute.

4. The robot product picking method based on visual recognition according to claim 3, characterized in that, The attribute information of the commodity is analyzed to generate the best grabbing frame parameter, including: The appearance feature attribute library of the commodity is called, and the appearance feature vector of the current commodity and the three-dimensional attribute description vector are input into the appearance feature attribute library of the commodity for analysis and matching, and the appearance attribute features of the current commodity are output, and the visual capture weight is adjusted based on the multi-head attention mechanism according to the appearance attribute features, and the first commodity grabbing strategy is determined according to the weight adjustment result; A historical grabbing library is constructed, and a static label of the current commodity is generated based on the physical attribute corresponding to the current commodity, and at the same time, the running track of the current commodity is collected in real time, and the running track trend of the current commodity is determined according to the running track of the current commodity, and the spatial position change feature of the current commodity in the preset time interval is predicted according to the running track trend, and a dynamic attribute label is generated according to the spatial position change feature; The static-dynamic label pair is generated based on the static label and the dynamic label of the current target commodity, and the static-dynamic label pair is input into the preset expert library for matching to determine the attribute label of the current commodity, and at the same time, the second commodity grabbing strategy for grabbing the commodity is determined in the preset expert library based on the attribute label of the current commodity; The constraint condition of the grabbing frame is determined according to the second commodity grabbing strategy, and the initial grabbing frame is generated according to the preset condition of the grabbing frame, and the best grabbing frame parameter is generated by optimizing the initial grabbing frame according to the first commodity grabbing strategy.

5. The robot product picking method based on visual recognition according to claim 2, characterized in that, The spatial pose data of the mechanical hand in grabbing the commodity is determined according to the best grabbing frame parameter, including: The first coordinate system of the multi-modal visual acquisition device corresponding to the best grabbing frame parameter is obtained, and the second coordinate system of the mechanical hand is obtained, and the calibration rules of the first coordinate system and the second coordinate system are determined respectively; The conversion rule of the first coordinate system and the second coordinate system is determined based on the calibration rule, and the coordinate transformation matrix is constructed based on the conversion rule; The first coordinates corresponding to each grabbing point are determined based on the best grabbing frame parameter, and the pose matrix of the grabbing point is constructed based on the first coordinates corresponding to each grabbing point; The coordinates of each grabbing point in the second coordinate system are obtained by coordinate conversion of each grabbing point in the pose matrix based on the transformation matrix; The opening amplitude and direction of the mechanical hand are determined based on the target coordinates, and the spatial pose data of the mechanical hand in grabbing the commodity is determined based on the opening amplitude and direction of the mechanical hand.

6. The robot product picking method based on visual recognition according to claim 5, characterized in that, The spatial pose data of the mechanical hand in grabbing the commodity is determined based on the opening amplitude and direction of the mechanical hand, including: The obtained spatial pose data is obtained, and the appearance structure data of the commodity is obtained, and a virtual commodity is constructed in the computer based on the appearance structure data of the commodity; At the same time, a virtual mechanical hand is constructed, and the virtual mechanical hand is simulated based on the spatial pose data, and the virtual commodity is virtually grabbed based on the simulation result; The running state of the mechanical hand and the commodity in the virtual grabbing process is monitored in real time, and the running stable parameter of the mechanical hand and the stable parameter of the commodity grabbed are determined based on the real-time monitoring result; When the running stable parameter and the stable parameter of the commodity grabbed meet the preset limited indicators respectively, it is determined that the spatial pose data of the mechanical hand in grabbing the commodity is qualified, and the spatial pose data is added with authorization permission; Otherwise, the spatial pose data of the manipulator when grabbing the commodity is re-determined until the preset limited index is met.

7. The robot product picking method based on visual recognition according to claim 2, characterized in that, The spatial pose data is obtained, and the spatial pose data is object-split based on the components of the manipulator to obtain sub-pose parameters corresponding to each component. The execution actions of each component in the manipulator are quantified based on the sub-pose parameters to obtain execution action features of each component, and the original instruction code of each component is adaptively corrected based on the execution action features. The adaptive correction results of each component are logically associated based on the working process of the manipulator, and the grabbing control instruction of the manipulator is obtained based on the logical association results. The spatial pose data is obtained, and the spatial pose data is object-split based on the components of the manipulator to obtain sub-pose parameters corresponding to each component.

8. The robot product picking method based on visual recognition according to claim 2, characterized in that, The execution actions of each component in the manipulator are quantified based on the sub-pose parameters to obtain execution action features of each component, and the original instruction code of each component is adaptively corrected based on the execution action features. The adaptive correction results of each component are logically associated based on the working process of the manipulator, and the grabbing control instruction of the manipulator is obtained based on the logical association results. The spatial pose data is obtained, and the spatial pose data is object-split based on the components of the manipulator to obtain sub-pose parameters corresponding to each component. The execution actions of each component in the manipulator are quantified based on the sub-pose parameters to obtain execution action features of each component, and the original instruction code of each component is adaptively corrected based on the execution action features. The adaptive correction results of each component are logically associated based on the working process of the manipulator, and the grabbing control instruction of the manipulator is obtained based on the logical association results.

Images