Object stacking method, system, device and medium for mobile robot

Abstract

The present invention discloses a method, system, device and medium for stacking objects of a mobile robot, wherein the method obtains a multi-view image set and a calibration pattern of objects in a working area, and pre-processes the multi-view image set to obtain a multi-view data set; the multi-view data set is input into an object detection model to obtain an object orientation classification result and an object shape classification result, wherein the object orientation classification result is used to characterize the spatial orientation of the object in the working area, and the object shape classification result is used to characterize the shape posture of the object in the working area; based on the calibration pattern, the object orientation classification result and the object shape classification result are used to locate the object position, and the object position positioning result is obtained; according to the object position positioning result, the objects are stacked. The method can effectively reduce the computing resources required for object stacking and the degree of dependence on the environment, improve the accuracy of object stacking, and reduce costs. The present invention relates to the field of target detection technology.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to an object stacking method, system, equipment and medium for a mobile robot. Background Art

[0002] With the development of science and technology, the application of vision-guided robot stacking technology in industrial production lines in related fields can effectively improve the production efficiency of industrial products and reduce the workload of related personnel, and has become one of the technical contents that people are paying more and more attention to.

[0003] At present, traditional vision-guided robot stacking technology usually obtains the image and three-dimensional information of objects based on visual sensors, thereby controlling the robot arm to grasp and place objects in stacking operations. This stacking method is relatively complex, requires more computing resources, is highly dependent on the environment, has limited accuracy, and is costly.

[0004] Therefore, the problems existing in the prior art still need to be solved and optimized. Summary of the invention

[0005] The purpose of the present invention is to solve one of the technical problems existing in the related art to at least a certain extent.

[0006] To this end, an object of an embodiment of the present invention is to provide an object stacking method for a mobile robot, which can effectively reduce the computing resources required for object stacking and the degree of dependence on the environment, improve the accuracy of object stacking, and reduce costs.

[0007] Another object of an embodiment of the present application is to provide an object stacking system for a mobile robot.

[0008] In order to achieve the above technical objectives, the technical solutions adopted by the embodiments of the present application include:

[0009] In a first aspect, an embodiment of the present application provides an object stacking method of a mobile robot, comprising:

[0010] Acquire a multi-view image set and a calibration pattern of an object in a working area, and pre-process the multi-view image set to obtain a multi-view data set;

[0011] Inputting the multi-view data set into an object detection model to obtain an object orientation classification result and an object shape classification result, wherein the object orientation classification result is used to characterize the spatial orientation of the object in the working area, and the object shape classification result is used to characterize the shape posture of the object in the working area;

[0012] Based on the calibration pattern, performing object position positioning on the object orientation classification result and the object shape classification result to obtain an object position positioning result;

[0013] The objects are stacked according to the object position positioning results.

[0014] In addition, the object stacking method of the mobile robot according to the above embodiment of the present application may also have the following additional technical features:

[0015] Furthermore, in one embodiment of the present application, preprocessing the multi-view image set to obtain a multi-view data set includes:

[0016] Performing spatial data enhancement on the multi-view image set to obtain a multi-view enhanced set;

[0017] Noise removal is performed on the multi-view enhancement set to obtain the multi-view data set.

[0018] Further, in one embodiment of the present application, the multi-view dataset includes a first-view dataset and a second-view dataset, and the object detection model is used to perform the following steps:

[0019] Performing first-view feature extraction on the first-view data set to obtain a first-view feature set;

[0020] Performing second-view feature extraction on the second-view data set to obtain a second-view feature set;

[0021] Performing feature fusion on the first viewing angle feature set and the second viewing angle feature set to obtain a viewing angle feature fusion set;

[0022] Identifying the view feature fusion set to obtain the object orientation classification result and the object shape classification result;

[0023] Among them, the first viewing angle feature set is used to characterize multiple object features of the object in the first viewing angle and have an information interaction relationship with the second viewing angle feature set; the second viewing angle feature set is used to characterize multiple object features of the object in the second viewing angle and have an information interaction relationship with the first viewing angle feature set.

[0024] Further, in one embodiment of the present application, the object orientation classification result and the object shape classification result are performed object position positioning based on the calibration pattern to obtain the object position positioning result, including:

[0025] Acquiring pattern three-dimensional information corresponding to the calibration pattern;

[0026] Based on the three-dimensional information of the pattern, performing pixel conversion on the calibration pattern to obtain a pixel conversion value;

[0027] According to the pixel conversion value, performing orientation pixel conversion on the object orientation classification result to obtain an object orientation positioning result, and, according to the pixel conversion value, performing shape pixel conversion on the object shape classification result to obtain an object shape positioning result;

[0028] The object orientation positioning result and the object shape positioning result are integrated to obtain the object position positioning result.

[0029] Further, in one embodiment of the present application, stacking the objects according to the object position positioning result includes:

[0030] Obtain environmental data and the current robotic arm data of the mobile robot;

[0031] According to the environmental data and the current robotic arm data, path planning is performed on the object position positioning result to obtain path planning information, wherein the path planning information includes the mobile planning path of the mobile robot, the robotic arm grasping shape, the robotic arm grasping size, the robotic arm grasping direction and the robotic arm operation trajectory;

[0032] According to the path planning information, the mobile robot is controlled to stack the objects.

[0033] Further, in an embodiment of the present application, controlling the mobile robot to stack the objects according to the path planning information includes:

[0034] Obtaining the current stacking operation status of the mobile robot;

[0035] If the current stacking operation state is in stacking operation, the robot arm of the mobile robot is controlled to stack the objects according to the robot arm operation trajectory, the robot arm grasping shape, the robot arm grasping size and the robot arm grasping direction.

[0036] Further, in one embodiment of the present application, controlling the mobile robot to stack the objects according to the path planning information includes:

[0037] Obtaining the current mobile operation status of the mobile robot;

[0038] If the current mobile operation status is in progress, the mobile robot is controlled to move to the target planned position according to the mobile planning path, and the current mobile operation status is updated; or, if the current mobile operation status is completed, return to execute the step of obtaining the current stacking operation status of the mobile robot.

[0039] In a second aspect, an embodiment of the present application provides an object stacking system for a mobile robot, comprising:

[0040] An acquisition module is used to acquire a multi-view image set and a calibration pattern of an object in a working area, and pre-process the multi-view image set to obtain a multi-view data set;

[0041] A detection module, used to input the multi-view data set into an object detection model to obtain an object orientation classification result and an object shape classification result, wherein the object orientation classification result is used to characterize the spatial orientation of the object in the working area, and the object shape classification result is used to characterize the shape posture of the object in the working area;

[0042] A positioning module, used to perform object position positioning on the object orientation classification result and the object shape classification result based on the calibration pattern to obtain an object position positioning result;

[0043] The stacking module is used to stack the objects according to the positioning results of the object positions.

[0044] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0045] at least one processor;

[0046] at least one memory for storing at least one program;

[0047] When the at least one program is executed by the at least one processor, the at least one processor implements the method of the first aspect described above.

[0048] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a program executable by a processor, and the program executable by the processor is used to implement the method of the first aspect when executed by the processor.

[0049] The advantages and benefits of the present application will be partially given in the following description, and partially become apparent from the following description, or be understood through the practice of the present application:

[0050] The present application discloses an object stacking method, system, device and medium for a mobile robot, wherein the stacking method obtains a multi-view image set and a calibration pattern of an object in a working area, and pre-processes the multi-view image set to obtain a multi-view data set; the multi-view data set is input into an object detection model to obtain an object orientation classification result and an object shape classification result, wherein the object orientation classification result is used to characterize the spatial orientation of the object in the working area, and the object shape classification result is used to characterize the shape posture of the object in the working area; based on the calibration pattern, the object orientation classification result and the object shape classification result are used to locate the object position, and the object position positioning result is obtained; according to the object position positioning result, the objects are stacked. The stacking method predicts the orientation and shape of the object based on the object detection model, which can effectively reduce the computational complexity of object stacking in the target detection stage, requires less computing power resources, can be predicted quickly and with high accuracy; in addition, the stacking method locates the object position based on the calibration pattern by combining the object orientation classification and the object shape classification results, which can effectively reduce the hardware requirements of the visual sensor and the degree of dependence on the environment, and reduce costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the embodiments of the present application or the drawings of the related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly expressing some embodiments of the technical solutions of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 A schematic diagram of a flow chart of an object stacking method of a mobile robot provided in an embodiment of the present application;

[0053] Figure 2 A schematic diagram of the structure of an object stacking system of a mobile robot provided in an embodiment of the present application;

[0054] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limitations on the present application. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0057] At present, traditional vision-guided robot stacking technology usually uses visual sensors to obtain images and three-dimensional information of objects, identifies the position of objects through image processing algorithms, and then controls the robot arm to perform object grasping and placement stacking operations.

[0058] Specifically, the visual sensors used in this method are usually composed of high-precision cameras and depth sensors (such as lidar and structured light sensors), among which the high-precision cameras are used to capture two-dimensional images in the working area, and the depth sensors are used to provide three-dimensional depth information of objects to help determine the spatial position of objects. The hardware cost required for this method is relatively high, and the technology is usually relatively complex. The subsequent object target recognition requires more computing resources, which makes the computing resources required for object stacking by mobile robots more and the overall cost higher. In addition, this method is greatly affected by environmental conditions such as complex environment, dynamic environmental changes, unstable or unsuitable environmental lighting conditions, and has a high dependence on the environment. When the environmental conditions are not good, the accuracy and success rate of object stacking by mobile robots are unsatisfactory.

[0059] In view of this, an embodiment of the present invention provides an object stacking method for a mobile robot. The stacking method predicts the orientation and shape of an object based on an object detection model, which can effectively reduce the accuracy requirements of a multi-view image set, thereby reducing costs, and reducing the computational complexity in the target detection stage. The required computing power resources are relatively small, and predictions can be made quickly with high accuracy. In addition, the stacking method locates the object based on a calibration pattern, combines the object orientation classification and the object shape classification results, and determines the object position based on the comparison idea, which can effectively reduce the hardware requirements of the visual sensor and the degree of dependence on the environment, thereby reducing costs.

[0060] Reference Figure 1 In an embodiment of the present application, a method for stacking objects using a mobile robot includes:

[0061] Step 110: acquiring a multi-view image set and a calibration pattern of an object in a working area, and preprocessing the multi-view image set to obtain a multi-view data set;

[0062] In some embodiments, the step 110 of preprocessing the multi-view image set to obtain a multi-view data set includes:

[0063] Step 111, performing spatial data enhancement on the multi-view image set to obtain a multi-view enhanced set;

[0064] Step 112: performing noise elimination on the multi-view enhancement set to obtain the multi-view data set.

[0065] In the embodiment of the present application, the multi-view image set and the calibration pattern can be acquired in real time. The specific acquisition method can be based on the binocular camera on the mobile robot, or the binocular camera on the mechanical arm of the mobile robot, which can capture images of two different viewing angles of the same working area. Specifically, the set of two different viewing angle images of objects in the working area can be used as a multi-view image set; the calibration pattern can be a pattern with the function of converting the calibration pixel to the real aspect ratio. There are many specific pattern types. This application takes the specific pattern type of the calibration pattern as a calibration QR code as an example.

[0066] It can be understood that the preprocessing in step 110 is used to improve the quality of the view images in the multi-view dataset. Specifically, the spatial data enhancement in step 111 can be at least one of image alignment, image enhancement, grayscale change and other operations, and the noise elimination in step 112 is used to filter out the noise of the enhanced view images, so as to obtain the multi-view dataset.

[0067] Step 120: Input the multi-view data set into an object detection model to obtain an object orientation classification result and an object shape classification result, wherein the object orientation classification result is used to characterize the spatial orientation of the object in the working area, and the object shape classification result is used to characterize the shape posture of the object in the working area;

[0068] In some embodiments, the multi-view dataset includes a first-view dataset and a second-view dataset, and the object detection model in step 120 is used to perform the following steps:

[0069] Step 121: extract first viewing angle features from the first viewing angle data set to obtain a first viewing angle feature set;

[0070] Step 122: extracting second perspective features from the second perspective data set to obtain a second perspective feature set;

[0071] Step 123: performing feature fusion on the first viewing angle feature set and the second viewing angle feature set to obtain a viewing angle feature fusion set;

[0072] Step 124, identifying the view feature fusion set to obtain the object orientation classification result and the object shape classification result;

[0073] Among them, the first viewing angle feature set is used to characterize multiple object features of the object in the first viewing angle and have an information interaction relationship with the second viewing angle feature set; the second viewing angle feature set is used to characterize multiple object features of the object in the second viewing angle and have an information interaction relationship with the first viewing angle feature set.

[0074] In an embodiment of the present application, an object detection model is used to predict the spatial orientation and shape posture of an object in a working area, wherein the spatial orientation is an orientation in a three-dimensional space with a preliminary prediction of the size of the object; a first-perspective data set may be obtained by shooting from a left-eye perspective of a binocular camera, and a second-perspective data set may be obtained by shooting from a right-eye perspective of a binocular camera.

[0075] It can be understood that, for a first-perspective image in a first-perspective data set, first-perspective feature extraction can be based on algorithms such as edge detection, template matching and feature point extraction to extract the key features of the object at the first perspective from the first-perspective image; then, the key features of the object at the first perspective are interacted with the key features of the object at the second perspective, and the key features of the object at the second perspective are obtained based on the second-perspective image in the second-perspective data set, so as to obtain the first-perspective features. The same applies to the remaining first-perspective features, and the first-perspective feature set can be obtained by integrating all first-perspective features.

[0076] It should be noted that, in the first-perspective feature extraction stage, the key features of the object in the first perspective and the key features of the object in the second perspective are exchanged with each other. This can be based on the interactive learning mechanism to set the backbone network layer (backbone) in the object detection model, and the key features of the object in the first perspective and the key features of the object in the second perspective are complemented with each other. The specific complementation method can be to directly splice the key features of the object in the first perspective with the key features of the object in the second perspective; or, based on the hierarchical design in the backbone network layer of the object detection model, the key features of the object in the first perspective are set as high-level features, and the key features of the object in the second perspective are set as low-level features, and the high-level features are upsampled to the same dimension as the low-level features for fusion; or, the key features of the object in the first perspective can also be set as low-level features, and the key features of the object in the second perspective can be set as high-level features, and the low-level features can be downsampled to the same dimension as the high-level features for fusion. The examples in this application are for illustration only and do not impose any limitation on this application.

[0077] It is worth mentioning that the first-perspective feature extraction and the second-perspective feature extraction in the embodiment of the present application can adopt a lightweight high-precision backbone network (ConvNext Lite), and realize information complementarity between the first-perspective feature and the second-perspective feature by performing an interactive learning mechanism in the middle layer of the backbone network; step 123 can perform feature fusion on the first-perspective feature set and the second-perspective feature set based on the Concat channel superposition method; step 124 can be based on a multi-scale FPN layer, using two parallel task heads (head) to respectively identify the perspective feature fusion set, so as to obtain the object orientation classification result and the object shape classification result.

[0078] Step 130: Based on the calibration pattern, perform object position positioning on the object orientation classification result and the object shape classification result to obtain an object position positioning result;

[0079] In some embodiments, the step 130 of performing object position positioning on the object orientation classification result and the object shape classification result based on the calibration pattern to obtain the object position positioning result includes:

[0080] Step 131, obtaining pattern three-dimensional information corresponding to the calibration pattern;

[0081] Step 132: Based on the three-dimensional information of the pattern, pixel conversion is performed on the calibration pattern to obtain a pixel conversion value;

[0082] Step 133: performing orientation pixel conversion on the object orientation classification result according to the pixel conversion value to obtain an object orientation positioning result, and performing shape pixel conversion on the object shape classification result according to the pixel conversion value to obtain an object shape positioning result;

[0083] Step 134: Integrate the object orientation positioning result and the object shape positioning result to obtain the object position positioning result.

[0084] In the embodiment of the present application, the calibration pattern is a calibration QR code with a front view and a side view obtained by a binocular camera, and the corresponding pattern three-dimensional information includes the QR code shape, size, and the distance between the calibration front and side views. Specifically, step 132 may be to perform pixel conversion on the rectangular frame of the calibration pattern by comparison based on the corresponding three-dimensional information of the pattern. Specifically, the conversion ratio (i.e., pixel conversion value) between each pixel in the calibration pattern and the true aspect ratio may be obtained by the ratio between the size of the rectangular frame of the calibration pattern and the three-dimensional information of the pattern. For the object orientation classification result, step 133 may be to perform a product operation on the pixel size of the object detection frame of the object orientation classification result based on the pixel conversion value obtained by comparison, so as to obtain the length and width of the front of the object, the length and width of the side of the object, and the depth distance of the object, and determine the precise position and preliminary posture data of the object in the three-dimensional space (i.e., the object orientation positioning result) according to the length and width of the front of the object, the length and width of the side of the object, the depth distance of the object, and the original object orientation classification result. The shape pixel conversion in step 133 is similar to the aforementioned orientation pixel conversion, and can be simply deduced by analogy, so as to determine the preliminary position and precise posture data of the object in the three-dimensional space (i.e., the object shape positioning result).

[0085] It can be understood that after obtaining the object orientation positioning results and the object shape positioning results, the object orientation positioning results and the object shape positioning results can be simply integrated, and the precise spatial coordinates and posture data of the object in three-dimensional space (i.e., the object position positioning results) can be generated based on the integration results.

[0086] Step 140: stack the objects according to the object location positioning result.

[0087] In some embodiments, the step 140 of stacking the objects according to the object location positioning result includes:

[0088] Step 141, obtaining environmental data and current mechanical arm data of the mobile robot;

[0089] Step 142: perform path planning on the object position positioning result according to the environmental data and the current robot arm data to obtain path planning information, wherein the path planning information includes the mobile planning path of the mobile robot, the robot arm grasping shape, the robot arm grasping size, the robot arm grasping direction and the robot arm operation trajectory;

[0090] In the embodiment of the present application, after obtaining the object position positioning result about the object in the working area, the mechanical arm of the mobile robot can be controlled to stack the object based on the object position positioning result. Specifically, the environmental data in step 141 is used to characterize at least one of the fixed obstacles, mobile obstacles and environmental maps of the target planning position in the environment, and the current mechanical arm data can be the current grasping shape, grasping size, grasping direction and mechanical arm position of the mobile robot.

[0091] It can be understood that path planning is used to determine the optimal path from the current robotic arm position to the target planning position, and the optimal path includes the moving path of the mobile robot and / or the stacking operation trajectory of the mobile robot robotic arm; and the stacking operation trajectory also includes information about the gripping shape, gripping size and gripping direction changes of the robotic arm during the movement; the path planning in the embodiment of the present application can be based on any one of the implementations of the A-star path optimization (A-star) algorithm, the Dijkstra algorithm, etc.

[0092] Step 143: Control the mobile robot to stack the objects according to the path planning information.

[0093] Furthermore, the step 143, controlling the mobile robot to stack the objects according to the path planning information, includes:

[0094] A1. Obtaining the current stacking operation status of the mobile robot;

[0095] A2. If the current stacking operation state is in stacking operation, the robotic arm of the mobile robot is controlled to stack the objects according to the robotic arm operation trajectory, the robotic arm grasping shape, the robotic arm grasping size and the robotic arm grasping direction.

[0096] In the embodiment of the present application, the gripping shape of the robot arm corresponds to the shape posture of the object in the object shape classification result, the gripping direction of the robot arm corresponds to the spatial orientation in the object orientation classification result, and the gripping size of the robot arm corresponds to the size information in the object position positioning result. Specifically, when the current stacking operation state of the mobile robot is in stacking operation, step A2 can be to control the robot arm of the mobile robot to grip the object according to the gripping shape of the robot arm, the gripping size of the robot arm and the gripping direction of the robot arm, and stack the object to the target stacking position according to the operation trajectory of the robot arm; or, when the current stacking operation state of the mobile robot is not in stacking operation, the mobile robot can exit the object stacking state, or return to execute step 110, so as to perform subsequent stacking operations.

[0097] Furthermore, the step 143, controlling the mobile robot to stack the objects according to the path planning information, includes:

[0098] B1. Obtaining the current mobile operation status of the mobile robot;

[0099] B2. If the current mobile operation status is in progress, the mobile robot is controlled to move to the target planned position according to the mobile planning path, and the current mobile operation status is updated; or, if the current mobile operation status is completed, return to execute the step of obtaining the current stacking operation status of the mobile robot.

[0100] In an embodiment of the present application, when the current mobile operation state of the mobile robot is in mobile operation, the mobile robot can be controlled to move to the target planned position according to the mobile planning path, and then the current mobile operation state is updated to movement completed, and then the process returns to execute step B1; or, when the current mobile operation state of the mobile robot is movement completed, the mobile robot's robotic arm can be controlled to stack objects and execute step A1.

[0101] The object stacking system of a mobile robot proposed according to an embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0102] Reference Figure 2 , an object stacking system of a mobile robot proposed in an embodiment of the present application includes:

[0103] An acquisition module 101 is used to acquire a multi-view image set and a calibration pattern of an object in a working area, and pre-process the multi-view image set to obtain a multi-view data set;

[0104] A detection module 102, configured to input the multi-view data set into an object detection model to obtain an object orientation classification result and an object shape classification result, wherein the object orientation classification result is used to characterize the spatial orientation of the object in the working area, and the object shape classification result is used to characterize the shape posture of the object in the working area;

[0105] A positioning module 103 is used to perform object position positioning on the object orientation classification result and the object shape classification result based on the calibration pattern to obtain an object position positioning result;

[0106] The stacking module 104 is used to stack the objects according to the positioning results of the objects.

[0107] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0108] Reference Figure 3 , the embodiment of the present application further provides an electronic device, including:

[0109] at least one processor 201;

[0110] At least one memory 202, used to store at least one program;

[0111] When the at least one program is executed by the at least one processor 201, the at least one processor 201 implements the above method embodiment.

[0112] Similarly, it can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0113] The embodiment of the present application further provides a computer-readable storage medium, in which a program executable by the processor 201 is stored. The program executable by the processor 201 is used to implement the above-mentioned method embodiment when executed by the processor 201.

[0114] Similarly, the contents of the above method embodiments are all applicable to the computer-readable storage medium embodiments. The functions specifically implemented by the computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0115] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the application is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part described as a larger operation is performed independently.

[0116] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present application. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional techniques of the engineer. Therefore, those skilled in the art can implement the present application set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the attached claims and their equivalents.

[0117] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method of the embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0118] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0119] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0120] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0121] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0122] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0123] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the embodiments. Technical personnel familiar with the field can make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A method for stacking objects using a mobile robot, characterized in that: include: Acquire a multi-view image set and a calibration pattern of an object in a working area, and pre-process the multi-view image set to obtain a multi-view data set; Inputting the multi-view data set into an object detection model to obtain an object orientation classification result and an object shape classification result, wherein the object orientation classification result is used to characterize the spatial orientation of the object in the working area, and the object shape classification result is used to characterize the shape posture of the object in the working area; Based on the calibration pattern, performing object position positioning on the object orientation classification result and the object shape classification result to obtain an object position positioning result; stacking the objects according to the object position positioning results; The performing object position positioning on the object orientation classification result and the object shape classification result based on the calibration pattern to obtain the object position positioning result includes: Acquiring pattern three-dimensional information corresponding to the calibration pattern; Based on the three-dimensional information of the pattern, performing pixel conversion on the calibration pattern to obtain a pixel conversion value; According to the pixel conversion value, performing orientation pixel conversion on the object orientation classification result to obtain an object orientation positioning result, and, according to the pixel conversion value, performing shape pixel conversion on the object shape classification result to obtain an object shape positioning result; The object orientation positioning result and the object shape positioning result are integrated to obtain the object position positioning result.

2. The object stacking method according to claim 1, characterized in that: The preprocessing of the multi-view image set to obtain a multi-view data set includes: Performing spatial data enhancement on the multi-view image set to obtain a multi-view enhanced set; Noise removal is performed on the multi-view enhancement set to obtain the multi-view data set.

3. The object stacking method according to claim 1, characterized in that: The multi-view dataset includes a first-view dataset and a second-view dataset, and the object detection model is used to perform the following steps: Performing first-view feature extraction on the first-view data set to obtain a first-view feature set; Performing second-view feature extraction on the second-view data set to obtain a second-view feature set; Performing feature fusion on the first viewing angle feature set and the second viewing angle feature set to obtain a viewing angle feature fusion set; Identifying the view feature fusion set to obtain the object orientation classification result and the object shape classification result; The first viewing angle feature set is used to characterize a plurality of object features of the object at the first viewing angle and having an information interaction relationship with the second viewing angle feature set; The second viewing angle feature set is used to characterize a plurality of object features of the object at the second viewing angle and having an information interaction relationship with the first viewing angle feature set.

4. The object stacking method according to claim 1, characterized in that: The stacking of the objects according to the object position positioning result includes: Obtain environmental data and the current robotic arm data of the mobile robot; According to the environmental data and the current robotic arm data, path planning is performed on the object position positioning result to obtain path planning information, wherein the path planning information includes the mobile planning path of the mobile robot, the robotic arm grasping shape, the robotic arm grasping size, the robotic arm grasping direction and the robotic arm operation trajectory; According to the path planning information, the mobile robot is controlled to stack the objects.

5. The object stacking method according to claim 4, characterized in that: The controlling the mobile robot to stack the objects according to the path planning information includes: Obtaining the current stacking operation status of the mobile robot; If the current stacking operation state is in stacking operation, the robot arm of the mobile robot is controlled to stack the objects according to the robot arm operation trajectory, the robot arm grasping shape, the robot arm grasping size and the robot arm grasping direction.

6. The object stacking method according to claim 5, characterized in that: The controlling the mobile robot to stack the objects according to the path planning information includes: Obtaining the current mobile operation status of the mobile robot; If the current mobile operation status is in progress, the mobile robot is controlled to move to the target planned position according to the mobile planning path, and the current mobile operation status is updated; or, if the current mobile operation status is completed, return to execute the step of obtaining the current stacking operation status of the mobile robot.

7. An object stacking system for a mobile robot, characterized in that: include: An acquisition module, used to acquire a multi-view image set and a calibration pattern of an object in a working area, and pre-process the multi-view image set to obtain a multi-view data set; A detection module, used to input the multi-view data set into an object detection model to obtain an object orientation classification result and an object shape classification result, wherein the object orientation classification result is used to characterize the spatial orientation of the object in the working area, and the object shape classification result is used to characterize the shape posture of the object in the working area; A positioning module, used to perform object position positioning on the object orientation classification result and the object shape classification result based on the calibration pattern to obtain an object position positioning result; A stacking module, used for stacking the objects according to the positioning results of the objects; The performing object position positioning on the object orientation classification result and the object shape classification result based on the calibration pattern to obtain the object position positioning result includes: Acquiring pattern three-dimensional information corresponding to the calibration pattern; Based on the three-dimensional information of the pattern, performing pixel conversion on the calibration pattern to obtain a pixel conversion value; According to the pixel conversion value, performing orientation pixel conversion on the object orientation classification result to obtain an object orientation positioning result, and, according to the pixel conversion value, performing shape pixel conversion on the object shape classification result to obtain an object shape positioning result; The object orientation positioning result and the object shape positioning result are integrated to obtain the object position positioning result.

8. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the method according to any one of claims 1 to 6 when executed by the processor.

Images