A logistics warehousing and handling robot

Abstract

The present application relates to the field of intelligent robots, and specifically to a logistics warehousing and handling robot. The handling robot obtains cargo information by scanning a QR code attached to the cargo, and uses its camera to collect cargo images, and then uses artificial intelligence technology based on deep learning to perform semantic analysis on the cargo information and cargo images contained in the QR code, and confirms whether the handling object is accurate based on the semantic consistency between the two, so as to avoid incorrect handling due to incorrect QR code affixed. In this way, the accuracy and reliability of the logistics warehousing and handling robot can be effectively improved, ensuring the correct execution of the handling task and avoiding unnecessary losses due to handling errors.

Description

Technical Field

[0001] The present application relates to the field of intelligent robots, and specifically to a logistics warehousing and handling robot. Background Art

[0002] In the modern logistics industry, storage and handling efficiency and accuracy are key factors in improving overall operational efficiency and reducing costs. Traditional storage management methods mainly rely on manual identification, handling and recording, which is not only labor-intensive and inefficient, but also prone to errors. Especially when handling large quantities of goods, human-induced errors such as misplacement, omission, and mishandling of goods are common, which seriously affects the accuracy and timeliness of logistics.

[0003] With the advancement of science and technology, especially the rapid development of technologies such as the Internet of Things, artificial intelligence, and machine vision, intelligent logistics robots have gradually become an important tool for improving the efficiency and accuracy of warehouse management. The invention patent with publication number CN110498370B discloses a handling robot for warehouse logistics, which can realize the handling of goods on shelves by performing lifting operations through a lifting drive component.

[0004] In the actual application of handling robots, accurate acquisition and verification of cargo information is an important prerequisite for ensuring the correct execution of handling tasks. At present, many warehousing systems use QR code technology to identify cargo information. By scanning the QR code, the specific information of the cargo, such as name, quantity, storage location, etc., is obtained to drive the handling robot to perform corresponding operations. However, this method has certain limitations. For example, the QR code may cause incorrect information due to incorrect pasting, which in turn causes incorrect handling.

[0005] Therefore, we expect an optimized logistics warehousing and handling robot that can accurately obtain and verify cargo information, thereby ensuring the correct execution of the handling task. Summary of the invention

[0006] This application is made in consideration of the above problems. One object of this application is to provide a logistics storage and handling robot.

[0007] The embodiment of the present application provides a logistics warehousing handling robot, which includes a movable chassis, a lifting platform, a lifting drive component, a tray, a rotation drive component and an intelligent camera; wherein the movable chassis includes a chassis base, which can be controllably moved relative to the ground; the lifting platform is installed on the movable chassis, and it can perform lifting and lowering movements relative to the movable chassis; the lifting drive component is used to drive the lifting platform to perform lifting and lowering movements relative to the movable chassis; the tray is installed on the lifting platform, and it can rotate relative to the lifting platform; the rotation drive component is used to drive the tray to rotate relative to the lifting platform; the intelligent camera is installed on the lifting platform, and is used to scan a QR code attached to the goods to confirm and extract the goods, wherein the intelligent camera includes:

[0008] A QR code scanning module, used to scan the QR code attached to the goods through the smart camera to obtain goods information;

[0009] A cargo information semantic understanding module, used for semantically encoding the cargo information to obtain a cargo information semantic encoding feature vector;

[0010] A cargo image acquisition module, used to acquire the product image of the cargo through the smart camera;

[0011] A cargo image multi-level feature extraction module, used for performing multi-level feature extraction on the product image to obtain a cargo local feature map and a cargo global feature map;

[0012] A cargo image semantic understanding module, used for performing cargo image semantic understanding based on the attention joint features of the cargo local feature map and the cargo global feature map to obtain a cargo image semantic understanding feature vector;

[0013] The transport object confirmation module is used to confirm whether the transport object is correct based on the feature difference between the semantic coding feature vector of the cargo information and the semantic understanding feature vector of the cargo image.

[0014] For example, according to the logistics warehousing and handling robot of the embodiment of the present application, the cargo information semantic understanding module is used to:

[0015] The cargo information is input into a cargo information semantic encoder based on the Bert model for semantic encoding to obtain a semantic encoding feature vector of the cargo information.

[0016] For example, according to the logistics warehousing and handling robot of the embodiment of the present application, the cargo image multi-level feature extraction module includes:

[0017] A product local feature extraction unit, used for inputting the product image into a product local feature extractor based on a hole convolutional neural network model to obtain the product local feature map;

[0018] The product global feature extraction unit is used to input the local feature map of the goods into a product global feature extractor based on a converter structure to obtain the global feature map of the goods.

[0019] For example, according to the logistics warehousing handling robot of the embodiment of the present application, the cargo image semantic understanding module includes:

[0020] A joint perception unit, used for inputting the cargo local feature map and the cargo global feature map into an attention network for joint feature saliency perception to obtain a cargo local-global joint restricted perception feature map;

[0021] The cargo image semantic decoding unit is used to input the cargo local-global joint restricted perception feature map into a decoder-based cargo semantic understander to obtain the cargo image semantic understanding feature vector.

[0022] For example, according to the logistics warehousing and handling robot of the embodiment of the present application, the joint perception unit includes:

[0023] A first characteristic shape reshaping subunit is used to perform characteristic shape reshaping on the local characteristic map of the goods and the global characteristic map of the goods to obtain a local characteristic shape reshaping matrix of the goods and a global characteristic shape reshaping matrix of the goods;

[0024] A bidirectional dependency calculation subunit, used for inputting the cargo local feature shape reshaping matrix and the cargo global feature shape reshaping matrix into a feature channel-by-channel interactive perception module to obtain a local-global dependency matrix and a global-local dependency matrix;

[0025] a random dropout subunit, configured to input the local-global dependency matrix and the global-local dependency matrix into a random dropout module to obtain a pruned local-global dependency matrix and a pruned global-local dependency matrix;

[0026] A dependency optimization subunit is used to optimize the cargo global feature shape reshaping matrix and the cargo local feature shape reshaping matrix based on the pruned local-global dependency matrix and the pruned global-local dependency matrix to obtain a dependency-optimized cargo global feature matrix and a dependency-optimized cargo local feature matrix;

[0027] A second feature shape reshaping subunit is used to perform feature shape reshaping on the dependency-optimized goods local feature matrix and the dependency-optimized goods global feature matrix to obtain an optimized goods local feature map and an optimized goods global feature map;

[0028] The multi-scale joint subunit is used to calculate the weighted sum of the optimized cargo local feature map and the optimized cargo global feature map to obtain the cargo local-global joint restricted perception feature map.

[0029] For example, according to the logistics warehousing and handling robot of the embodiment of the present application, the bidirectional dependency relationship calculation subunit is used to:

[0030] Calculate the transposed matrix of the cargo local feature shape reshaping matrix multiplied by the cargo global feature shape reshaping matrix to obtain a cargo local-global association representation matrix;

[0031] The cargo local-global association representation matrix is ​​divided by the scale of the cargo global feature shape reshaping matrix and then input into the softmax function to obtain the local-global dependency matrix;

[0032] Calculate the transposed matrix of the cargo local feature shape reshaping matrix and multiply it by the cargo global feature shape reshaping matrix to obtain a cargo global-local association representation matrix;

[0033] The cargo local-global association representation matrix is ​​divided by the scale of the cargo local feature shape reshaping matrix and then input into the softmax function to obtain the global-local dependency matrix.

[0034] For example, according to the logistics warehousing and handling robot of the embodiment of the present application, the dependency optimization subunit is used to:

[0035] Performing matrix multiplication of the pruned local-global dependency matrix and the cargo global feature shape reshaping matrix to obtain the dependency-optimized cargo global feature matrix;

[0036] The pruned global-local dependency matrix is ​​matrix-multiplied with the cargo local feature shape reshaping matrix to obtain the dependency-optimized cargo local feature matrix.

[0037] For example, according to the logistics warehousing handling robot of the embodiment of the present application, the handling object confirmation module includes:

[0038] An information consistency coefficient calculation unit, used to calculate the information consistency coefficient between the semantic encoding feature vector of the cargo information and the semantic understanding feature vector of the cargo image;

[0039] The comparison unit is used to confirm whether the transported object is correct based on the comparison between the information consistency coefficient and a preset threshold value.

[0040] According to the logistics warehousing handling robot of the embodiment of the present application, the handling robot obtains the cargo information by scanning the QR code attached to the cargo, and uses its camera to collect cargo images, and then uses artificial intelligence technology based on deep learning to perform semantic analysis on the cargo information and cargo images contained in the QR code, and confirms whether the handling object is accurate based on the semantic consistency between the two, so as to avoid incorrect handling due to incorrect QR code affixed. In this way, the accuracy and reliability of the logistics warehousing handling robot can be effectively improved, ensuring the correct execution of the handling task and avoiding unnecessary losses due to handling errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings of the embodiments of the present application are briefly introduced below. Obviously, the drawings described below only relate to some embodiments of the present application, and are not intended to limit the present application.

[0042] Figure 1 A schematic diagram of the structure of a logistics warehousing and handling robot in an embodiment of the present application is shown;

[0043] Figure 2 A schematic diagram showing the structure of the intelligent camera of the logistics warehousing and handling robot in the embodiment of the present application is shown; and

[0044] Figure 3 A flow chart of a logistics warehousing and handling method in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work also fall within the scope of protection of the present application.

[0046] The terms used in this specification are those common terms currently widely used in the art in consideration of the functions of the present application, but these terms may vary according to the intentions of those skilled in the art, precedents, or new technologies in the art. In addition, specific terms may be selected, and in this case, their detailed meanings will be described in the detailed description of the present application. Therefore, the terms used in the specification should not be understood as simple names, but rather as a general description based on the meaning of the terms and the present application.

[0047] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0048] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. At the same time, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0049] In addition, the application architecture diagram in the embodiment of the present application is intended to more clearly illustrate the technical solution in the embodiment of the present application, and does not constitute a limitation on the technical solution provided in the embodiment of the present application. Of course, for other application architectures and business applications, the technical solution provided in the embodiment of the present application is also applicable to similar problems.

[0050] The following is a non-restrictive description of the logistics warehousing and handling robot provided according to at least one embodiment of the present application through several examples or embodiments. As described below, different features in these specific examples or embodiments can be combined with each other without conflicting with each other to obtain new examples or embodiments, and these new examples or embodiments also fall within the scope of protection of the present application.

[0051] In response to the above technical problems, the technical concept of this application is as follows: the handling robot obtains cargo information by scanning the QR code attached to the cargo, and uses its camera to collect cargo images, and then uses artificial intelligence technology based on deep learning to perform semantic analysis on the cargo information and cargo images contained in the QR code, and confirms whether the handling object is accurate based on the semantic consistency between the two, so as to avoid incorrect handling due to incorrect QR code affixed. In this way, the accuracy and reliability of logistics warehousing handling robots can be effectively improved, ensuring the correct execution of handling tasks and avoiding unnecessary losses due to handling errors.

[0052] Based on this, this application proposes a logistics warehousing and handling robot. Figure 1The schematic diagram of the structure of the logistics warehousing and handling robot in the embodiment of the present application is shown. The logistics warehousing and handling robot comprises: a movable chassis 10, a lifting platform 20, a lifting drive assembly 30, a tray 40, a rotation drive assembly 50 and an intelligent camera 60; wherein the movable chassis 10 comprises a chassis base 11, which can be controllably moved relative to the ground; the lifting platform 20 is mounted on the movable chassis 10, and it can perform lifting and lowering movements relative to the movable chassis 10; the lifting drive assembly 30 is used to drive the lifting platform 20 to perform lifting and lowering movements relative to the movable chassis 10; the tray 40 is mounted on the lifting platform 20, and it can rotate relative to the lifting platform 20; the rotation drive assembly 50 is used to drive the tray 40 to rotate relative to the lifting platform 20; the intelligent camera 60 is mounted on the lifting platform 20, and is used to scan the QR code attached to the goods to confirm and extract the goods.

[0053] Among them, Figure 2 As shown, the smart camera 60 includes: a QR code scanning module 61, which is used to scan the QR code attached to the goods through the smart camera to obtain the goods information; a goods information semantic understanding module 62, which is used to semantically encode the goods information to obtain the goods information semantic encoding feature vector; a goods image acquisition module 63, which is used to acquire the product image of the goods through the smart camera; a goods image multi-level feature extraction module 64, which is used to perform multi-level feature extraction on the product image to obtain a goods local feature map and a goods global feature map; a goods image semantic understanding module 65, which is used to perform goods image semantic understanding based on the attention joint feature of the goods local feature map and the goods global feature map to obtain a goods image semantic understanding feature vector; a transport object confirmation module 66, which is used to confirm whether the transport object is accurate based on the feature difference between the goods information semantic encoding feature vector and the goods image semantic understanding feature vector.

[0054] Specifically, first, the QR code attached to the goods is scanned by the smart camera to obtain the goods information. It should be understood that the goods information stored in the QR code includes but is not limited to the name, color, size, quantity and other attribute information of the goods, which is an important basis for identifying and verifying the goods. In order to fully understand the semantic meaning of the goods information, this application uses a goods information semantic encoder based on the Bert model to perform semantic analysis on the goods information, so as to utilize the bidirectional Transformer structure of the Bert model to fully capture the semantic meaning of each word unit in the goods information and the semantic association between words, dig out the contextual meaning of the goods information, and form an accurate description of the goods information in the semantic feature space to obtain the semantic encoding feature vector of the goods information.

[0055] Correspondingly, the cargo information semantic understanding module 62 is used to: input the cargo information into a cargo information semantic encoder based on the Bert model for semantic encoding to obtain the cargo information semantic encoding feature vector.

[0056] Considering that although the QR code can quickly and accurately provide basic information about the goods, there is a risk of being posted incorrectly, which may lead to incorrect information. Therefore, in order to further improve the accuracy of the handling task, the application also uses the smart camera to collect product images of the goods and obtain visual appearance information of the goods, so as to verify it with the QR code information, thereby improving the accuracy and reliability of goods identification.

[0057] Next, the product image is further processed using a dilated convolutional neural network model with excellent performance in image feature extraction as a product local feature extractor to mine the local detail information of the product image, capture the product's edge, corner, texture and other detail features, and generate a local feature map of the goods.

[0058] Since convolution is a local receptive field operation, it cannot fully capture the global structural information of the product in the image. In order to overcome this limitation, the present application further uses a product global feature extractor based on a converter structure to process the local feature map of the goods, by dividing the local feature map of the goods into multiple sub-regions, and using the self-attention mechanism of the converter structure to perform global information context association encoding on each sub-region, capturing the long-distance dependency between each sub-region, so as to better understand the overall structure and spatial layout of the goods and generate a global feature map of the goods.

[0059] Correspondingly, the cargo image multi-level feature extraction module 64 includes: a product local feature extraction unit, used to input the product image into a product local feature extractor based on a hole convolutional neural network model to obtain the cargo local feature map; a product global feature extraction unit, used to input the cargo local feature map into a product global feature extractor based on a converter structure to obtain the cargo global feature map.

[0060] The local feature map of the goods and the global feature map of the goods each have their own emphasis on the visual description of the goods. In order to comprehensively utilize the advantages of both and obtain a more comprehensive description of the visual features of the goods, in the technical solution of the present application, the local feature map of the goods and the global feature map of the goods are further input into the attention network of the joint feature saliency perception for information fusion. It is worth mentioning that the attention network is based on the dual attention interaction mechanism, and by modeling the bidirectional dependency between the local feature map of the goods and the global feature map of the goods, it can achieve deep fusion of multi-level features. Specifically, the network first reshapes the shape of the local feature map of the goods and the global feature map of the goods into a feature matrix so that its dimension and shape are adapted to the input requirements of the attention mechanism. Then, by calculating the bidirectional dependency between the two sets of feature matrices, the interaction between the features is quantified, and the local-global and global-local dependency matrices are generated, which are used as a guide for the optimization fusion of feature attention. At the same time, in order to reduce the sensitivity of the model to noise and prevent overfitting, the two sets of dependency matrices are further subjected to random inactivation processing, and the generalization ability of the model is enhanced by selectively ignoring or suppressing part of the feature interaction. Furthermore, based on the two sets of dependency matrices after random inactivation processing, the two sets of feature matrices after shape reshaping are weighted optimized, and then restored to the original feature map form through shape reshaping and weighted fusion to generate the local-global joint restricted perception feature map of the goods, thereby achieving a comprehensive description of the appearance of the goods.

[0061] Furthermore, since images and texts belong to different data modalities, there are significant differences in the feature representations of the product images and the cargo information in the semantic space. Therefore, in order to achieve semantic alignment between the two and make the cargo local-global joint restricted perception feature map and the cargo information semantic encoding feature vector comparable, the present application further uses a decoder-based cargo semantic understander to perform semantic analysis on the cargo local-global joint restricted perception feature map, and uses the autoregressive characteristics of the decoder to gradually convert the visual feature representation into a textual semantic feature representation, generating a cargo image semantic understanding feature vector, thereby achieving deep semantic understanding of the cargo image and semantic alignment of cross-modal information.

[0062] Accordingly, the cargo image semantic understanding module 65 includes: a joint perception unit, used to input the cargo local feature map and the cargo global feature map into an attention network of joint feature saliency perception to obtain a cargo local-global joint restricted perception feature map; and a cargo image semantic decoding unit, used to input the cargo local-global joint restricted perception feature map into a decoder-based cargo semantic understander to obtain the cargo image semantic understanding feature vector.

[0063] The joint perception unit includes: a first feature shape reshaping subunit, used for performing feature shape reshaping on the local feature map of the goods and the global feature map of the goods to obtain a local feature shape reshaping matrix of the goods and a global feature shape reshaping matrix of the goods; a bidirectional dependency calculation subunit, used for inputting the local feature shape reshaping matrix of the goods and the global feature shape reshaping matrix of the goods into a feature channel-by-channel interactive perception module to obtain a local-global dependency matrix and a global-local dependency matrix; a random deactivation subunit, used for inputting the local-global dependency matrix and the global-local dependency matrix into a random deactivation module to obtain a pruned local-global dependency matrix and a pruned global-local dependency matrix; a dependency relationship calculation ... The first is an optimization subunit, used for optimizing the cargo global feature shape reshaping matrix and the cargo local feature shape reshaping matrix based on the pruned local-global dependency matrix and the pruned global-local dependency matrix to obtain a dependency-optimized cargo global feature matrix and a dependency-optimized cargo local feature matrix; the second feature shape reshaping subunit is used for performing feature shape reshaping on the dependency-optimized cargo local feature matrix and the dependency-optimized cargo global feature matrix to obtain an optimized cargo local feature map and an optimized cargo global feature map; the multi-scale joint subunit is used for calculating the weighted sum of the optimized cargo local feature map and the optimized cargo global feature map to obtain the cargo local-global joint restricted perception feature map.

[0064] Among them, the bidirectional dependency calculation subunit is used to: calculate the transpose matrix of the local feature shape reshaping matrix of the goods multiplied by the transposed matrix of the global feature shape reshaping matrix of the goods to obtain the local-global association representation matrix of the goods; divide the local-global association representation matrix of the goods by the scale of the global feature shape reshaping matrix of the goods and input it into the softmax function to obtain the local-global dependency matrix; calculate the transpose matrix of the local feature shape reshaping matrix of the goods multiplied by the global feature shape reshaping matrix of the goods to obtain the global-local association representation matrix of the goods; divide the local-global association representation matrix of the goods by the scale of the local feature shape reshaping matrix of the goods and input it into the softmax function to obtain the global-local dependency matrix.

[0065] Among them, the dependency optimization subunit is used to: matrix multiply the pruned local-global dependency matrix with the cargo global feature shape reshaping matrix to obtain the dependency optimized cargo global feature matrix; matrix multiply the pruned global-local dependency matrix with the cargo local feature shape reshaping matrix to obtain the dependency optimized cargo local feature matrix.

[0066] In one example, the joint perception unit is used to process the cargo local feature map and the cargo global feature map using the following joint perception formula to obtain the cargo local-global joint restricted perception feature map, wherein the joint perception formula is:

[0067] M 1 =Reshape(F 1 )

[0068] M 2 =Reshape(F 2 )

[0069]

[0070]

[0071] W 21 = Dropout(M 21 )

[0072] W 12 = Dropout(M 12 )

[0073]

[0074]

[0075] F 1 ′=Reshape(M 1 ′)

[0076] F 2 ′=Reshape(M 2 ′)

[0077] F c =αF 1 ′+βF 2 '

[0078] Among them, F 1 represents the local feature map of the cargo, F 2 represents the global feature map of the goods, reshape(·) represents the feature shape reshaping, M 1 Represents the local feature shape reshaping matrix of the cargo, M 2 Represents the cargo global feature shape reshaping matrix, represents matrix multiplication, (·) Trepresents the transpose of the matrix, S represents the scale of the local feature shape reshaping matrix of the goods, that is, the width of the local feature shape reshaping matrix of the goods multiplied by the height, and the scale of the local feature shape reshaping matrix of the goods is the same as that of the global feature shape reshaping matrix of the goods, Softmax is a normalized exponential function, M 21 represents the local-global dependency matrix, M 12 represents the global-local dependency matrix, Dropout(·) represents random dropout processing, W 21 represents the pruned local-global dependency matrix, W 12 represents the pruned global-local dependency matrix, M 1 ′ Represents the dependency relationship to optimize the local feature matrix of goods, M 2 ′ Represents the dependency relationship optimization cargo global feature matrix, F 1 ′ represents the optimized local feature map of goods, F 2 ′ represents the optimized global feature map of goods, α and γ represent different weight coefficients, F c Represents the local-global joint restricted perception feature map of the goods.

[0079] Then, in order to quantitatively describe the semantic consistency between the product image and the goods information, the present application further measures the similarity between the goods information semantic encoding feature vector and the goods image semantic understanding feature vector by calculating the information consistency coefficient between the two, thereby revealing the degree of matching between the goods information stored in the QR code and the actual goods. In the embodiment of the present application, the Mahalanobis distance is used as the calculation method of the information consistency coefficient.

[0080] Then, based on the comparison between the information consistency coefficient and the preset threshold, it is confirmed whether the transported object is correct. If the information consistency coefficient is higher than the preset threshold, it is considered that the transported object is correct and the robot can perform the transport task; otherwise, it is considered that there may be problems with the transported object and further manual confirmation or re-identification is required.

[0081] Correspondingly, the transport object confirmation module includes: an information consistency coefficient calculation unit, which is used to calculate the information consistency coefficient between the cargo information semantic encoding feature vector and the cargo image semantic understanding feature vector; and a comparison unit, which is used to confirm whether the transport object is accurate based on the comparison between the information consistency coefficient and a preset threshold.

[0082] Preferably, the information consistency coefficient calculation unit is used to: cascade the cargo information semantic coding feature vector and the cargo image semantic understanding feature vector to obtain a cargo information semantic coding-cargo image semantic understanding cascade representation vector; subtract the 0 norm of the cargo information semantic coding-cargo image semantic understanding cascade representation vector from the length of the cargo information semantic coding-cargo image semantic understanding cascade representation vector to obtain a cargo information semantic coding-cargo image semantic understanding cascade isolated representation value; calculate a power function with each eigenvalue of the cargo information semantic coding-cargo image semantic understanding cascade representation vector as the base, and the difference of the cargo information semantic coding-cargo image semantic understanding cascade isolated representation value minus one as the exponent, and multiply it by the logarithm of the product of the cargo information semantic coding-cargo image semantic understanding cascade isolated representation value plus one and the cargo information semantic coding-cargo image semantic understanding cascade isolated representation value with the base 2 to obtain a cargo information semantic coding-cargo image semantic understanding cascade leading value; After each eigenvalue of the semantic understanding cascade representation vector is multiplied by the difference of the cargo information semantic coding-cargo image semantic understanding cascade isolated representation value minus one and divided by the cargo information semantic coding-cargo image semantic understanding cascade isolated representation value, an exponential function with a natural constant as the base is calculated to obtain a cargo information semantic coding-cargo image semantic understanding cascade bias value; the cargo information semantic coding-cargo image semantic understanding cascade leading value is added to the cargo information semantic coding-cargo image semantic understanding cascade bias value to obtain each eigenvalue of the optimized cargo information semantic coding-cargo image semantic understanding cascade representation vector; the optimized cargo information semantic coding-cargo image semantic understanding cascade representation vector is split into an optimized cargo information semantic coding feature vector and an optimized cargo image semantic understanding feature vector according to the cascade of the cargo information semantic coding feature vector and the cargo image semantic understanding feature vector; and, the information consistency coefficient between the optimized cargo information semantic coding feature vector and the optimized cargo image semantic understanding feature vector is calculated.

[0083] Specifically, the cargo information semantic encoding-cargo image semantic understanding cascade representation vector, for example, each eigenvalue v of V is denoted as i , where v i The optimization process of ∈V is expressed as follows:

[0084]

[0085] n=L-‖V‖ 0

[0086] Among them, v irepresents the ith eigenvalue of the cascade representation vector of the goods information semantic encoding-goods image semantic understanding, L represents the length of the cascade representation vector of the goods information semantic encoding-goods image semantic understanding, ‖·‖ 0 represents the zero norm of the vector, n represents the isolated representation value of the cargo information semantic encoding-cargo image semantic understanding cascade, log represents the logarithm operation with base 2, exp represents the exponential operation with base e, (·) ⊙(n-1) It represents the power function of calculating the eigenvalues ​​of each position in the eigenvector with the difference of the cascade isolated representation value of the cargo information semantic encoding-cargo image semantic understanding minus one as the exponent, v i ′ Represents the i-th eigenvalue of the optimized cargo information semantic encoding-cargo image semantic understanding cascade representation vector.

[0087] Here, the cargo information semantic encoding feature vector and the cargo image semantic understanding feature vector respectively express the text decoding cross-domain mapping features of the text semantic encoding features of the cargo information and the image semantic features of the cargo product image, so that when calculating the information consistency coefficient therebetween to map to a common semantic space, it is expected to further improve the comprehensibility of the common mapping regression, thereby improving the accuracy of the calculated information consistency coefficient.

[0088] Thus, for the high-dimensional feature manifold of the cargo information semantic coding-cargo image semantic understanding cascade representation vector, which is the set representation of the cargo information semantic coding feature vector and the cargo image semantic understanding feature vector, a vector field representation is obtained with the feature value of the feature set as the aggregation dimension. In addition, the value of the derivative of the vector field of the cargo information semantic coding-cargo image semantic understanding cascade representation vector at the isolated zero position is used as the order information to fix the local position of the feature value of the feature set. Then, a bias for the reversibility of the feature regression distribution of the cargo information semantic encoding-cargo image semantic understanding cascade representation vector is added as a reward to achieve mapping target tracking of the regression distribution of the cargo information semantic encoding-cargo image semantic understanding cascade representation vector for the feature value position, so that the feature set of the cargo information semantic encoding-cargo image semantic understanding cascade representation vector perceives the mapping migration to the aggregate distribution, thereby promoting the common mapping regression comprehensibility of the cargo information semantic encoding feature vector and the cargo image semantic understanding feature vector to the common semantic space by improving the mapping comprehensibility of the cargo information semantic encoding-cargo image semantic understanding cascade representation vector, and improving the calculation accuracy of the information consistency coefficient between the cargo information semantic encoding feature vector and the cargo image semantic understanding feature vector.

[0089] Based on the above embodiments, see Figure 3As shown, it is a flow chart of a logistics warehousing and handling method in an embodiment of the present application. Figure 3 As shown, the logistics warehousing and handling method according to the embodiment of the present application includes the following steps: S510, scanning a QR code attached to a cargo through a smart camera to obtain cargo information; S520, semantically encoding the cargo information to obtain a cargo information semantic encoding feature vector; S530, collecting a product image of the cargo through the smart camera; S540, performing multi-level feature extraction on the product image to obtain a cargo local feature map and a cargo global feature map; S550, performing cargo image semantic understanding based on the attention joint features of the cargo local feature map and the cargo global feature map to obtain a cargo image semantic understanding feature vector; S560, confirming whether the handling object is accurate based on the feature difference between the cargo information semantic encoding feature vector and the cargo image semantic understanding feature vector.

[0090] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned logistics warehousing and handling method have been referred to above. Figure 1 to Figure 2 The description of the logistics warehousing and handling robot has been introduced in detail, and therefore, its repeated description will be omitted.

[0091] Based on the above embodiments, another exemplary embodiment of an electronic device is also provided in the embodiments of the present application. In some possible implementations, the electronic device in the embodiments of the present application may include a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can implement the steps of the logistics warehousing and handling method in the above embodiments when executing the program.

[0092] The embodiment of the present application also provides a computer-readable storage medium, on which computer-executable instructions are stored. When the computer-executable instructions are executed by the processor, the logistics warehousing and handling method according to the embodiment of the present application described with reference to the above figures can be executed. The computer-readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory may, for example, include a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may, for example, include a read-only memory (ROM), a hard disk, a flash memory, etc.

[0093] The embodiment of the present application also provides a computer program product or computer program, which includes computer executable instructions, and the computer executable instructions are stored in a computer readable storage medium. The processor of the computer device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions, so that the computer device executes the logistics warehousing and handling method according to the embodiment of the present application.

[0094] Those skilled in the art will appreciate that the contents disclosed in this application may be subject to various variations and improvements. For example, the various devices or components described above may be implemented by hardware, or by software, firmware, or a combination of some or all of the three.

[0095] In addition, although the present application makes various references to certain units in the system according to embodiments of the present application, any number of different units can be used and run on the client and / or server. The units are only illustrative, and different aspects of the system and method can use different units.

[0096] Those skilled in the art will appreciate that all or part of the steps in the above method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk. Optionally, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or in the form of software function modules. The present application is not limited to any particular form of combination of hardware and software.

[0097] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an idealized or extremely formal sense, unless explicitly defined as such herein.

[0098] The above is an explanation of the present application and should not be considered as limiting thereof. Although several exemplary embodiments of the present application have been described, those skilled in the art will readily appreciate that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present application.

Claims

1. A logistics storage and handling robot, comprising a movable chassis, a lifting platform, a lifting drive component, a tray, a rotating drive component and an intelligent camera; wherein: The movable chassis includes a chassis base, which can be controllably moved relative to the ground; the lifting platform is installed on the movable chassis, and can perform lifting movement relative to the movable chassis; the lifting drive assembly is used to drive the lifting platform to perform lifting movement relative to the movable chassis; The tray is mounted on the lifting platform and can rotate relative to the lifting platform; the rotation drive assembly is used to drive the tray to rotate relative to the lifting platform; the smart camera is mounted on the lifting platform and is used to scan the QR code attached to the goods to confirm and extract the goods, characterized in that the smart camera includes: A QR code scanning module, used to scan the QR code attached to the goods through the smart camera to obtain goods information; A cargo information semantic understanding module, used for semantically encoding the cargo information to obtain a cargo information semantic encoding feature vector; A cargo image acquisition module, used to acquire the product image of the cargo through the smart camera; A cargo image multi-level feature extraction module, used for performing multi-level feature extraction on the product image to obtain a cargo local feature map and a cargo global feature map; A cargo image semantic understanding module, used for performing cargo image semantic understanding based on the attention joint features of the cargo local feature map and the cargo global feature map to obtain a cargo image semantic understanding feature vector; A transport object confirmation module, used to confirm whether the transport object is correct based on the feature difference between the semantic coding feature vector of the cargo information and the semantic understanding feature vector of the cargo image; The cargo information semantic understanding module is used to: Inputting the cargo information into a cargo information semantic encoder based on a Bert model for semantic encoding to obtain a semantic encoding feature vector of the cargo information; The cargo image multi-level feature extraction module includes: A product local feature extraction unit, used for inputting the product image into a product local feature extractor based on a hole convolutional neural network model to obtain the product local feature map; A product global feature extraction unit, used for inputting the goods local feature map into a product global feature extractor based on a converter structure to obtain the goods global feature map; The cargo image semantic understanding module includes: A joint perception unit, used for inputting the cargo local feature map and the cargo global feature map into an attention network for joint feature saliency perception to obtain a cargo local-global joint restricted perception feature map; A cargo image semantic decoding unit, used for inputting the cargo local-global joint restricted perception feature map into a decoder-based cargo semantic understander to obtain a cargo image semantic understanding feature vector; The joint sensing unit comprises: A first characteristic shape reshaping subunit is used to perform characteristic shape reshaping on the local characteristic map of the goods and the global characteristic map of the goods to obtain a local characteristic shape reshaping matrix of the goods and a global characteristic shape reshaping matrix of the goods; A bidirectional dependency calculation subunit, used for inputting the cargo local feature shape reshaping matrix and the cargo global feature shape reshaping matrix into a feature channel-by-channel interactive perception module to obtain a local-global dependency matrix and a global-local dependency matrix; a random dropout subunit, configured to input the local-global dependency matrix and the global-local dependency matrix into a random dropout module to obtain a pruned local-global dependency matrix and a pruned global-local dependency matrix; A dependency optimization subunit is used to optimize the cargo global feature shape reshaping matrix and the cargo local feature shape reshaping matrix based on the pruned local-global dependency matrix and the pruned global-local dependency matrix to obtain a dependency-optimized cargo global feature matrix and a dependency-optimized cargo local feature matrix; A second feature shape reshaping subunit is used to perform feature shape reshaping on the dependency-optimized goods local feature matrix and the dependency-optimized goods global feature matrix to obtain an optimized goods local feature map and an optimized goods global feature map; The multi-scale joint subunit is used to calculate the weighted sum of the optimized cargo local feature map and the optimized cargo global feature map to obtain the cargo local-global joint restricted perception feature map.

2. The logistics storage and handling robot according to claim 1, characterized in that: The bidirectional dependency calculation subunit is used to: Calculate the transposed matrix of the cargo local feature shape reshaping matrix multiplied by the cargo global feature shape reshaping matrix to obtain a cargo local-global association representation matrix; The cargo local-global association representation matrix is ​​divided by the scale of the cargo global feature shape reshaping matrix and then input into the softmax function to obtain the local-global dependency matrix; Calculate the transposed matrix of the cargo local feature shape reshaping matrix and multiply it by the cargo global feature shape reshaping matrix to obtain a cargo global-local association representation matrix; The cargo local-global association representation matrix is ​​divided by the scale of the cargo local feature shape reshaping matrix and then input into the softmax function to obtain the global-local dependency matrix.

3. The logistics storage and handling robot according to claim 2, characterized in that: The dependency optimization subunit is used to: Performing matrix multiplication of the pruned local-global dependency matrix and the cargo global feature shape reshaping matrix to obtain the dependency-optimized cargo global feature matrix; The pruned global-local dependency matrix is ​​matrix-multiplied with the cargo local feature shape reshaping matrix to obtain the dependency-optimized cargo local feature matrix.

4. The logistics storage and handling robot according to claim 3, characterized in that: The transport object confirmation module includes: An information consistency coefficient calculation unit, used to calculate the information consistency coefficient between the semantic encoding feature vector of the cargo information and the semantic understanding feature vector of the cargo image; The comparison unit is used to confirm whether the transported object is correct based on the comparison between the information consistency coefficient and a preset threshold value.

Images