Method, apparatus, device, system and storage medium for inserting a hanger into a garment

Abstract

The application provides a method, device, equipment, system and storage medium for inserting a clothes hanger into clothes, that is, a first position is obtained based on a first depth image and a first neural network, to control the end of a first mechanical arm to move to the first position, and a first operation of grabbing clothes is performed by a gripper at the end; a second position is obtained based on the first depth image, the first position and a second neural network, to control the first end point of a clothes hanger fixed by the end of a second mechanical arm to move to the second position, and a second operation of pressing down the clothes and inserting the first end point into a collar of the clothes is performed by the clothes hanger; a third position and a fourth position are obtained based on a second depth image and a third neural network, to control the gripper to move to the third position to perform a third operation of grabbing the clothes, and to control the second mechanical arm to perform a fourth operation of rotating the clothes hanger, moving a second end point of the clothes hanger to the fourth position and inserting the second end point into the collar of the clothes. Therefore, the clothes are helped to be dried, and the housework pressure is relieved.

Description

Technical Field

[0001] This application relates to the field of machinery and equipment technology, and in particular to a method, apparatus, device, system and storage medium for inserting a clothes hanger into clothing. Background Technology

[0002] With the increasing aging population, machines and devices to help people with housework have emerged. Drying clothes is an essential part of life, and considering that inserting hangers into clothes is a crucial step in the drying process, but currently there is no feasible technology for this, there is an urgent need for a technical solution to help people dry clothes and alleviate the burden of housework. Summary of the Invention

[0003] The purpose of this application is to provide a method, apparatus, device, system, and storage medium for inserting a clothes hanger into clothing, so as to help people dry clothes and relieve housework pressure by means of machinery and equipment.

[0004] The technical solution of this application embodiment is implemented as follows: In a first aspect, embodiments of this application provide a method for inserting a clothes hanger into clothing, the method comprising: Based on the acquired first depth image, a first position is obtained through a first neural network to control the end of the first robotic arm to move to the first position, and the first operation of grasping clothing is performed by the gripper at the end of the first robotic arm. Based on the first depth image and the first position, a second position is obtained through a second neural network to control the first end of the hanger fixed at the end of the second robotic arm to move to the second position. The second robotic arm controls the hanger to perform a second operation of pressing down the clothes and inserting the first end into the collar of the clothes. Based on the acquired second depth image, a third position and a fourth position are obtained through a third neural network. The gripper at the end of the first robotic arm is then moved to the third position to perform a third operation of grasping clothing. The second robotic arm controls the hanger to perform a fourth operation of rotating and moving the second end of the hanger to the fourth position and inserting it into the collar of the clothing.

[0005] Secondly, embodiments of this application provide a device for inserting a clothes hanger into clothing, the device comprising: The processing module is used to obtain the first position based on the acquired first depth image through a first neural network; The control module is used to control the end of the first robotic arm to move to the first position, and to perform the first operation of grasping clothing through the gripper at the end of the first robotic arm; The processing module is further configured to obtain a second position based on the first depth image and the first position using a second neural network; The control module is also used to control the first end of the hanger fixed at the end of the second robotic arm to move to the second position, and to control the hanger to perform a second operation of pressing down the clothes and inserting the first end into the collar of the clothes through the second robotic arm; The processing module is also used to obtain the third and fourth positions based on the acquired second depth image through a third neural network; The control module is also used to control the gripper at the end of the first robotic arm to move to the third position to perform the third operation of grabbing clothing, and to control the hanger to perform the fourth operation of rotating and moving the second end of the hanger to the fourth position and inserting it into the collar of the clothing through the second robotic arm.

[0006] Thirdly, embodiments of this application provide an electronic device, which includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes the program stored in the memory, it implements the steps of the method described in the first aspect.

[0007] Fourthly, this application provides a dual-arm platform system, the system comprising a first robotic arm, a second robotic arm, and a depth camera; The depth camera is deployed above the operating table and is used to take pictures of the operating table and the clothing placed on the operating table from below. The first robotic arm and the second robotic arm are deployed on the operating table and located on both sides of the depth camera. The end of the first robotic arm is provided with a gripper, and the end of the second robotic arm is provided with a clothes hanger. The first robotic arm works in conjunction with the second robotic arm to insert the two ends of the hanger into the clothing placed on the operating table one by one.

[0008] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0009] In a sixth aspect, embodiments of this application provide a computer program including computer-readable code, wherein when the computer-readable code is run in a computer device, a processor in the computer device performs steps for implementing the method described in the first aspect.

[0010] In a seventh aspect, embodiments of this application provide a computer program product, the computer program product including a non-transitory computer-readable storage medium storing a computer program, wherein when the computer program is read and executed by a computer, it implements the steps of the method described in the first aspect.

[0011] In this embodiment, based on the acquired first depth image, a first position is obtained through a first neural network to control the end of the first robotic arm to move to the first position, and the gripper at the end of the first robotic arm performs a first operation of grasping clothing; based on the first depth image and the first position, a second position is obtained through a second neural network to control the first end of the hanger fixed at the end of the second robotic arm to move to the second position, and the second robotic arm controls the hanger to perform a second operation of pressing down on the clothing and inserting the first end into the collar of the clothing; based on the acquired second depth image, a third position and a fourth position are obtained through a third neural network to control the gripper at the end of the first robotic arm to move to the third position to perform a third operation of grasping clothing, and the second robotic arm controls the hanger to perform a fourth operation of rotating and moving the second end of the hanger to the fourth position and inserting it into the collar of the clothing.

[0012] This allows the first and second robotic arms to collaborate and progressively insert the hanger into the collar of the garment, based on the first to fourth positions output by the neural network. In practice, the acquired depth image, or a depth image combined with the previous position, serves as input to the neural network to obtain a more precise output position. This guides the movement of the gripper at the end of the first robotic arm and the hanger at the end of the second robotic arm, enabling them to work together to complete the entire operation. This fills a technological gap in existing methods for inserting hangers into clothing, helping people to dry clothes and alleviate household chores.

[0013] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the structure of a dual-arm fixed platform system provided in an embodiment of this application; Figure 2 A flowchart illustrating a method for inserting a clothes hanger into clothing, as provided in this application embodiment; Figure 3 This application provides schematic diagrams of the structures of a first and a second neural network according to embodiments of the present application. Figure 4 This is a schematic diagram of the structure of a third neural network provided in an embodiment of this application; Figure 5 A schematic diagram of a device for inserting a clothes hanger into clothing, provided as an embodiment of this application; Figure 6This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] In the following description, various implementations are mentioned, which describe a subset of all possible embodiments. However, it is understood that these possible implementations may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. The terms "first / second / third" are used only to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0017] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to be limiting of this application.

[0018] This application provides a dual-arm fixed-stage system, such as... Figure 1 As shown, the dual-arm platform system 10 includes a first robotic arm 11, a second robotic arm 12, and a depth camera 13. The depth camera 13 is positioned above the operating platform 14 to capture images of the platform and the clothing 15 placed on it. The first robotic arm 11 and the second robotic arm 12 are positioned on the operating platform 14, located on either side of the depth camera 13. The first robotic arm 11 has a gripper 1101 at its end, and the second robotic arm 12 has a clothes hanger 1201 at its end. The first robotic arm 11 and the second robotic arm 12 work together to insert the two ends of the clothes hanger 1201 into the clothing placed on the operating platform 14 one by one. This helps people dry clothes and alleviates household chores.

[0019] In the actual deployment of the dual-arm platform system, for example, a detachable bracket 16 can be deployed on the operating table to fix the depth camera on the bracket, so as to realize the imaging of the corresponding area of ​​the operating table. The bracket can be fixed to the operating table, or a slide rail can be set on the operating table so that the bracket can slide on the operating table with the help of the slide rail, thereby adjusting the relative positional relationship between the depth camera and the operating table. The deployment of the first and second robotic arms on the operating table is similar to that of the bracket. Taking the detachable fixed to the operating table as an example, the two robotic arms are usually set on opposite sides of the operating table, with the bracket where the depth camera is located as the dividing line. The robotic arms on each side can be set in the center, or they can be set at opposite corners of the operating table, that is, near the corners of each side.

[0020] It should be noted that, Figure 1 This diagram illustrates only one possible structure of a dual-arm platform system and is not intended to limit the system in actual applications. For example, the deployment positions and methods of the two robotic arms and the depth camera are not limited, as long as the depth camera can acquire clear depth images for determining the first and second positions, and the two robotic arms can be used to manipulate clothing placed on the platform. Furthermore, the aforementioned grippers and hangers are typically detachably fixed to the ends of their respective robotic arms. To improve the success rate of inserting the hanger into the clothing, the grippers can be parallel grippers, and the hanger can be a 3D-printed hanger. This ensures the hanger is securely mounted at the end of the second robotic arm and guarantees the quality of the hanger and the smoothness of its rotation and translation operations. Since the hanger is 3D-printed, its specifications are known, which also means the positions of the first and second endpoints of the hanger relative to the second robotic arm are known.

[0021] To ensure the garment hangers can be easily inserted into the clothing, in this application, the garments are laid flat on the worktable by default, especially around the neckline, which must be facing upwards and at a roughly upright angle. The aforementioned garments typically refer to those without zippers or buttons, or those with zippers zipped up and buttons fastened. Of course, in actual operation, the placement and angle of the garments can be random within a certain range, as long as the depth camera can clearly capture them and they are within the control range of the two robotic arms.

[0022] Regarding the control panel, soft materials are typically used to protect the grippers and reduce damage. Considering the known height of the control panel relative to the ground, when establishing a coordinate system using the control panel as the xy-plane, the known height of the control panel can be used as the origin of the z-axis (vertical direction) of the world coordinate system.

[0023] This application provides a method for inserting a clothes hanger into clothing, such as... Figure 2As shown, the method includes steps S101 to S103.

[0024] Step S101: Based on the acquired first depth image, a first position is obtained through a first neural network to control the end of the first robotic arm to move to the first position, and the first operation of grasping clothing is performed by the gripper at the end of the first robotic arm.

[0025] Compared to the operating table, the first depth image is captured from top to bottom, allowing for the capture of clothing placed on the operating table, with the neckline of the clothing clearly visible. This first position can be represented using point coordinates. For example, the plane containing the operating table can be considered the xy-plane, and the position projected onto the xy-plane can be set as the origin. The direction in which the clothing is placed approximately upright in the image captured by the depth camera, and the upward direction in the first depth image, can be defined as the positive y-axis. Correspondingly, the direction perpendicular to the y-axis and pointing to the right in the image can be defined as the positive x-axis, and the positive z-axis can be the direction perpendicular to the xy-plane upwards away from the ground. To better confirm the relative positional relationship between the robotic arm, the depth camera, and the operating table, the x-axis and y-axis are typically parallel to one set of corresponding edges in the depth image captured by the depth camera, and perpendicular to another set of corresponding edges. For example, using a positively presented depth image as a reference, the axes parallel to the top and bottom of the image are defined as the x-axis, with their positive direction from left to right. The axes parallel to the left and right sides of the image are defined as the y-axis, with their positive direction from bottom to top. In a depth image, the arrangement of clothing tends to be positive, meaning the neckline is higher, with a slight tilt to the left or right. Other methods can be used to determine the origin and the positive directions of each axis during the construction of the coordinate system; these are not limited here.

[0026] After obtaining the accurate first position through the first neural network, the end of the first robotic arm with grippers can be controlled to move to the first position, which corresponds to the area near the neckline on one side of the garment, such as the position below the left side of the neckline in the first depth image. The grippers are then moved downwards to grasp the garment, which is the first operation.

[0027] Step S102: Based on the first depth image and the first position, the second position is obtained through the second neural network to control the first end of the hanger fixed at the end of the second robotic arm to move to the second position. The second robotic arm controls the hanger to perform the second operation of pressing down the clothes and inserting the first end into the collar of the clothes.

[0028] After determining the first position, the second neural network can obtain the second position based on the first depth image and the first position. This allows control to move the first endpoint of the hanger fixed to the end of the second robotic arm to the second position. The first endpoint refers to the endpoint corresponding to the neckline of the garment being grasped. For example, if the garment is grasped near the left side of the neckline, the first endpoint is the left end of the hanger; similarly, if it's grasped near the right side of the neckline, the first endpoint is the right end of the hanger. Taking the grasping of the left side of the neckline as an example, after the first endpoint of the hanger moves to the second position, the second robotic arm can be controlled to press the hanger downwards to insert the first endpoint into the neckline—this is the second operation.

[0029] It should be noted that, considering that clothing usually has two layers, top and bottom, pressing down on clothing refers to pressing down on the bottom layer of clothing with the hanger. This creates a sufficient gap between the two layers of clothing to insert the first end of the hanger into the neckline of the clothing, thus completing the second operation.

[0030] Step S103: Based on the acquired second depth image, the third position and the fourth position are obtained through the third neural network, so as to control the gripper at the end of the first robotic arm to move to the third position to perform the third operation of grabbing the clothes, and control the hanger to perform the fourth operation of rotating and moving the second end of the hanger to the fourth position and inserting it into the collar of the clothes through the second robotic arm.

[0031] After completing steps S101 and S102, the depth camera will acquire a second depth image. In this second depth image, one end of the hanger is typically successfully inserted into the neckline of the garment. Using this second depth image as input to the third neural network, a third and fourth position can be obtained. The third position refers to the position where the gripper at the end of the first robotic arm moves to, thus grasping the garment near the other side of the neckline. For example, if the first grasping action was of the area near the left side of the neckline, then the second grasping action, i.e., this current action, grasps the area near the right side of the neckline. This means controlling the gripper at the end of the first robotic arm to move to the third position to perform the third operation of grasping the garment. The fourth position refers to the position where the hanger at the end of the second robotic arm rotates and the second end of the hanger (the end opposite to the first end) is moved to, inserting the second end of the hanger into the neckline of the garment. This means controlling the hanger to rotate and move the second end of the hanger to the fourth position and insert it into the neckline via the second robotic arm.

[0032] After completing steps S101 to S103, the clothes hanger set on the second robotic arm is successfully inserted into the collar of the clothes, thereby helping people to dry their clothes and relieving the pressure of housework.

[0033] Considering that the first position affects not only the execution of the first operation but also the determination of the second position and the execution of the second operation, in one possible implementation, the first neural network often obtains a probability map. This probability map can reflect the coordinate information of the first position in the constructed coordinate system. This coordinate information can also be used to generate a one-heatmap to more intuitively represent the coordinate information of the first position and aid in determining the second position. A one-heatmap refers to an image where only one position is assigned a value of 1, while the remaining positions are assigned a value of 0. Therefore, step S101, obtaining the first position based on the acquired first depth image through the first neural network, can be implemented as step S201.

[0034] Step S201: Take the first depth image as input, use the first neural network to obtain the first probability map, and obtain the first unique heat map to represent the first position coordinate information based on the first probability map.

[0035] like Figure 3 The diagram shows the structural schematics of the first and second neural networks. Both the first and second neural networks can use the UNet network structure, but their parameters differ. The first depth image serves as the input to the first neural network, resulting in a first value map. After operations such as maximizing the value (e.g., using the argmax function), a first one-hot map is obtained. Finally, in practical applications, the first position can be represented as... Figure 3 The diagram of the first operation in the first stage shows the location of the center pixel within the rectangular area.

[0036] Similarly, in one possible implementation, the step S102, which obtains the second position based on the first depth image and the first position through the second neural network, can be implemented as step S202.

[0037] Step S202: Using the first depth image and the first unique heatmap as input, a second probability map is obtained using the second neural network, and a second unique heatmap is obtained based on the second probability map to represent the second position coordinate information.

[0038] The first probability map and the second probability map are different, but both are used to reflect the success rate of the first-stage operation corresponding to each pixel. The first-stage operation includes the first operation and the second operation.

[0039] like Figure 3 As shown, the first depth image and the first one-hot map are used as inputs to the second neural network to obtain the second probability map. After processing operations such as solving for the maximum value of the independent variable, the second one-hot map can be obtained. Finally, when mapped to the actual application scenario, the second position can be represented as... Figure 3The diagram of the second operation in the first stage shows the location of the center pixel within the rectangular area.

[0040] The first stage mentioned above refers to the process of inserting one end of the hanger into the collar of the garment.

[0041] As can be seen, obtaining the second position depends on obtaining the first position, which means that changes in the first position will directly affect the second position. It is for this reason that, during training, the first and second neural networks are usually trained together, as detailed in the following description of the training process.

[0042] Furthermore, the reason for using the first depth image and the first one-heatmap as inputs to the second neural network is that the first depth image may contain images of either the first or second robotic arm. However, the presence of the robotic arm would inevitably increase the complexity of visual processing. Therefore, the first one-heatmap was introduced to reduce the necessary visual complexity caused by the robotic arm itself in the first depth image. Figure 1 And used as input.

[0043] Considering the correlation between the third and fourth positions in practical applications, the third position used to guide the grasping area and the fourth position used to guide the rotation of the second end of the hanger will be obtained together by the third neural network. In one possible implementation, the step S103, which obtains the third and fourth positions based on the acquired second depth image through the third neural network, can be implemented as step S301.

[0044] Step S301: Using the second depth image as input, the third probability map is obtained using the third neural network, and the third position coordinate information and the fourth position coordinate information are obtained based on the position of the voxel point with the maximum probability value in the third probability map.

[0045] The third probability map is used to reflect the success rate of the second-stage operation corresponding to each voxel point. The second-stage operation includes the third and fourth operations.

[0046] like Figure 4The diagram shows the structure of the third neural network. The network structure of the third neural network can be the same as or different from that of the first and second neural networks. For example, the third neural network can adopt the UNet network structure, but its parameters will differ from those of the first and second neural networks. The second depth image serves as the input to the third neural network, resulting in a third probability map. After processing such as taking the maximum value, the position of the voxel can be determined. The third position coordinate information can be obtained based on the row and column of the voxel in the third probability map. Considering that the relative positional relationship between the fourth position coordinate information and the third position coordinate information in different layers of the third probability map is predefined, after obtaining the third position coordinate information, the fourth position coordinate information can be obtained based on the layer in the third probability map and the relative positional relationship between the third and fourth position coordinate information. Finally, in practical applications, the third position can be represented as... Figure 4 The third operation in the second stage corresponds to the location of the center pixel within the rectangular area displayed in the image. The fourth position can be represented as... Figure 4 The fourth operation in the second stage corresponds to the location of the center pixel within the rectangular area displayed in the image. To more intuitively demonstrate the relative positional relationship between the third and fourth positions, a third and fourth unique heatmap can be introduced. However, considering that the third and fourth unique heatmaps are not used in subsequent reasoning processes compared to the first unique heatmap, they are not required to be displayed. The definitions of the third and fourth unique heatmaps are similar to those of the first and second unique heatmaps, respectively representing the coordinate information of the third and fourth positions.

[0047] It should be noted that the second stage mentioned above refers to the process of inserting the other end of the hanger into the collar of the garment. The third position coordinate information mentioned above is a coordinate representation obtained after determining the row and column of the voxel points and transforming them using the intrinsic and extrinsic parameters of the depth camera.

[0048] In actual operation, after the gripper at the end of the first robotic arm grabs the clothing at the third position, it will drag the clothing and move it a preset fixed distance to the upper right corner of the xy plane. At this time, the hanger fixed at the end of the second robotic arm can be rotated and translated so that the second end of the hanger can be moved to the fourth position to realize the operation of inserting the second end of the hanger into the collar of the clothing.

[0049] Considering that inserting a hanger into the collar of clothing may cause the clothing to bunch up irregularly within a certain area, or that the clothing inserted into the hanger may not adhere securely to the hanger, the clothing itself can be tidied up and / or the state of the hanger can be adjusted after inserting the hanger into the collar to ensure proper hanging of the clothing. In one possible implementation, the above method may further include at least one of steps S401 and S402.

[0050] Step S401: In response to the execution of the fourth operation, the fifth operation of sorting clothes is performed by the gripper at the end of the first robotic arm.

[0051] After performing the fourth operation, the garment can be pulled down using the gripper at the end of the first robotic arm, which is the fifth operation, to organize the garment. The movement distance, angle, and speed of the gripper's pull-down operation can all be preset. The purpose is to spread out the clumps of the garment and make the collar and shoulder areas fit the hanger as closely as possible for easy hanging.

[0052] Step S402: In response to the insertion of the first and second ends of the hanger into the neckline of the garment, the second robotic arm controls the hanger to adjust from a horizontal state to a vertical state so that the garment inserted into the hanger is hung up.

[0053] After inserting both ends of the hanger into the collar of the garment, it can be considered as inserting the hanger into the garment. At this point, the second robotic arm can be used to control the rotation of the hanger to complete the state switch, that is, to adjust the hanger from a horizontal state parallel to the operating table to a vertical state.

[0054] It should be noted that the vertical position can be considered a posture different from the horizontal position. A better hanging effect is achieved when the plane of the hanger in the vertical position is perpendicular to the plane of the hanger in the horizontal position. Of course, the requirement for verticality does not need to be so strict; as long as the vertical hanger forms a certain angle with the operating table, sufficient to hang the clothes, it is acceptable. The hanger's position can be adjusted by adjusting the posture of the second robotic arm's end effector or by adjusting the overall posture of the second robotic arm, with the aim of placing the hanger in a natural hanging position for the clothes.

[0055] In one possible implementation, the training of the first and second neural networks described above can be carried out in the following manner, which may include steps S501 and S502.

[0056] Step S501: Obtain the first depth sample image and the first sample parameters corresponding to the first depth image sample.

[0057] The first sample parameters include a first training position, a second training position, and a first reward function. The first sample parameters include parameters obtained based on the simulator, and the first reward function is used to indicate whether clothing is inserted into the first end of the hanger.

[0058] Step S502: Based on the first depth sample image and the first sample parameters, train two initial neural networks simultaneously to obtain the first neural network and the second neural network.

[0059] To train the first and second neural networks, trajectories can be collected in advance using a simulator, and then the neural networks can be trained based on the collected trajectories. This provides abundant sample data while reducing the consumption of manpower and resources.

[0060] Based on a trajectory, a first depth sample image matching it and first sample parameters can be obtained. These parameters include the position where the first robotic arm's end gripper performs the first operation of grasping the clothing (i.e., the first training position), the position where the first end of the hanger fixed to the second robotic arm's end needs to move (i.e., the second training position), and a first reward function to indicate whether the first stage was successful. Typically, when the first stage operation is successful (i.e., the first end of the hanger successfully inserts into the collar of the clothing), the first reward function is assigned a value of 1; when the first stage operation fails, the first reward function is assigned a value of 0. The specific assignment strategy can be adjusted according to actual needs.

[0061] For the second neural network corresponding to the second operation in the first stage (i.e., the press network) can utilize the first reward function. Supervision is performed, similar to supervised learning, and the loss function can be expressed as the pressing loss. The details are as follows: (Formula 1) in, Used to represent binary cross-entropy loss The coordinates of the first training position are represented as follows: , This represents the second one-hot encoded sample image. Indicates input The second sample probability plot on the first The value of each pixel. This represents the first depth sample image. During backpropagation, the gradient of the press loss only propagates to the parameters in the second neural network. Since the second one-hot encoded sample image is a discrete integer, the gradient does not propagate to the first neural network. (i.e., the capture network). Furthermore, the first reward function is defined. ,in, This represents the solid angle of the garment covering the first endpoint of the hanger (taking the left endpoint as an example, but the right endpoint could also be used). Covering means that the ray emanating from the first endpoint of the hanger in the direction corresponding to this solid angle intersects with the garment itself. In practice, this solid angle can be obtained by simulating sampling in 1024 different directions and testing whether the emanating rays intersect with the garment.

[0062] However, for the first neural network, it is difficult to directly define a reward function for supervision. Therefore, in this application, the maximum probability value corresponding to the second operation in the first stage is used for supervision, where the maximum probability value... Represented as: (Formula 2) in, The output after the Sigmoid function corresponds to the loss function of the first neural network, which can be expressed as the grasping loss. The details are as follows: (Formula 3) in, The coordinates of the second training position are represented as follows: .

[0063] In one possible implementation, the training of the third neural network described above can be carried out in the following manner, which may include steps S601 and S602.

[0064] Step S601: Obtain the second depth sample image and the second sample parameters corresponding to the second depth image sample.

[0065] The second sample parameters include the third training position, the fourth training position, and the second reward function. The second sample parameters include parameters obtained based on the simulator, and the second reward function is used to indicate whether clothing is inserted into the second end of the hanger.

[0066] Step S602: Based on the second depth sample image and the second sample parameters, train an initial neural network to obtain a third neural network.

[0067] The acquisition of the second depth sample image and the second sample parameters can be referenced from the acquisition of the first depth sample image and the first sample parameters, both of which rely on the simulator and will not be elaborated here.

[0068] In this application, the first depth sample image is a two-dimensional image, and similarly, the second depth sample image is also a two-dimensional image. However, considering that it supports the outputs of the third and fourth training positions, the third probability map output by the third neural network is defined as a three-dimensional image, rather than the two-dimensional first and second probability maps output by the first and second networks. Then, if the first and second probability maps are W... If the specification of H is given, then the corresponding 3D image specification is defined as C. W H.

[0069] It should be noted that the third training position must be ideal to ensure the success of the second stage. For example, the third training position needs to be on the surface of the clothing, close to the collar, on the side away from the first training position relative to the collar, and should not collide with the hanger. The coordinates of the fourth training position... It can be considered as the coordinates based on the third training position. Fine-tuning is possible, and selection can be made within a small range around the third training location. The third neural network... Able to predict C W The third probability diagram of H. Here, C can be considered as... , The side length of the discretized square region surrounding the third training position, for example, Corresponding to the discretization of 6cm around the third training position A 6cm square area.

[0070] Similar to the first stage, a reward function can also be defined in the second stage, namely the second reward function. ,in, This represents the solid angle of the garment covering the second end point of the hanger (taking the right end point as an example, but the left end point could also be used). "Covering" refers to the intersection of a ray emanating from the second end point of the hanger in the direction corresponding to this solid angle and the garment itself. The method for determining this solid angle can be found in [reference needed]. The loss function of the third neural network can be expressed as the second loss. The details are as follows: (Formula 4) in, Used to represent binary cross-entropy loss The coordinates of the third training position are represented as follows: , The coordinates of the fourth training position are represented as follows: , This represents the second depth sample image.

[0071] In one possible implementation, to ensure the diversity of training data, more new trajectories can be obtained by running one or more of the first to third neural networks during the actual training process. These new trajectories can then be mixed into the original dataset to fine-tune the parameters involved in the first to third neural networks.

[0072] In this application, the first to third neural networks trained as described above can output accurate first to fourth positions, and, in conjunction with some predefined fixed parameters, complete the operation of inserting a hanger into clothing. For the insertion operation, based on pre-configuration, the hanger can be selectively inserted first into the left collar of the clothing and then into the right collar, or vice versa. In actual operation, based on the first implementation strategy, the gripper at the end of the first robotic arm can grasp the clothing and release it after inserting the left end of the hanger into the collar. Through rotation of the hanger and a second grasping operation by the gripper, the hanger fixed to the end of the second robotic arm rotates and inserts into the right collar of the clothing, and the gripper is then controlled to release the grasped clothing again.

[0073] For the first robotic arm, the starting point for the secondary clothing gripping operation of the gripper can be either the release position after the previous clothing gripping or the starting position of the previous clothing gripping operation. Of course, to further reduce the interference of the robotic arm in the second depth image, the latter implementation method can be chosen, that is, after each release of clothing, the gripper is placed at the preset starting position. For the gripping operation of the first stage, the gripper usually needs to be placed above the double-layer clothing, then moved down and gripped to grab both layers of clothing. The operation of pressing down and inserting the hanger requires pressing down on the area of ​​one layer of clothing before insertion. The operation of the second stage is similar to the first stage. The gripper grabs and drags two layers of clothing, while the insertion position of the right end of the hanger after rotation is usually one layer of clothing. This ensures that clothing can be smoothly inserted into both ends of the hanger. For the second robotic arm, the clothes hanger is usually parallel to the operating table. After inserting clothes into both ends of the clothes hanger, in order to ensure smooth hanging, the posture of the clothes hanger needs to be adjusted so that the clothes hanger is not parallel to the operating table, and the hook of the clothes hanger is higher than the rest of the clothes hanger body, so as to achieve the hanging of clothes.

[0074] It should be noted that the first, second, third, and fourth positions mentioned in the technical solution can all be obtained through reasoning during the actual application of the first to third neural networks. Based on this, the first and second robotic arms can be controlled to cooperate in completing the various stages of inserting the clothes hanger into the clothes. After completing the aforementioned operations, the clothes can be sorted and hung as needed.

[0075] This application provides a device for inserting a clothes hanger into clothing, such as... Figure 5 As shown, the device 20 for inserting a hanger into clothing includes: The processing module 21 is used to obtain the first position based on the acquired first depth image through the first neural network.

[0076] The control module 22 is used to control the end of the first robotic arm to move to the first position and perform the first operation of grabbing clothing through the gripper at the end of the first robotic arm.

[0077] The processing module 21 is also used to obtain the second position based on the first depth image and the first position through the second neural network.

[0078] The control module 22 is also used to control the first end of the hanger fixed at the end of the second robotic arm to move to the second position, and to control the hanger to perform the second operation of pressing down the clothes and inserting the first end into the collar of the clothes through the second robotic arm.

[0079] The processing module 21 is also used to obtain the third and fourth positions based on the acquired second depth image through a third neural network.

[0080] The control module 22 is also used to control the gripper at the end of the first robotic arm to move to the third position to perform the third operation of grabbing the clothing, and to control the hanger to perform the fourth operation of rotating and moving the second end of the hanger to the fourth position and inserting it into the collar of the clothing through the second robotic arm.

[0081] In one possible implementation, the processing module 21 is further configured to take the first depth image as input, use the first neural network to obtain a first probability map, and obtain a first unique heatmap representing the first position coordinate information based on the first probability map.

[0082] In one possible implementation, the processing module 21 is further configured to take the first depth image and the first unique heatmap as input, use the second neural network to obtain a second probability map, and obtain a second unique heatmap to represent the second position coordinate information based on the second probability map.

[0083] The first probability map and the second probability map are different, but both are used to reflect the success rate of the first-stage operation corresponding to each pixel. The first-stage operation includes the first operation and the second operation.

[0084] In one possible implementation, the processing module 21 is further configured to take the second depth image as input, use the third neural network to obtain a third probability map, and obtain third position coordinate information and fourth position coordinate information based on the location of the voxel point with the maximum probability value in the third probability map.

[0085] The third probability map is used to reflect the success rate of the second-stage operation corresponding to each voxel point. The second-stage operation includes the third and fourth operations.

[0086] In one possible implementation, the control module 22 is also configured to perform a fifth operation of sorting clothes via the gripper at the end of the first robotic arm in response to the execution of the fourth operation.

[0087] In one possible implementation, the control module 22 is also configured to, in response to the insertion of the first and second ends of the hanger into the neckline of the garment, control the hanger to adjust from a horizontal state to a vertical state via the second robotic arm, so that the garment inserted into the hanger is hung up.

[0088] In one possible implementation, device 20 further includes: The acquisition module 23 is used to acquire a first depth sample image and a first sample parameter corresponding to the first depth image sample. The first sample parameter includes a first training position, a second training position, and a first reward function. The first sample parameter includes parameters obtained based on the simulator. The first reward function is used to indicate whether clothing is inserted into the first end of the hanger.

[0089] Training module 24 is used to simultaneously train two initial neural networks based on the first depth sample image and the first sample parameters to obtain the first neural network and the second neural network.

[0090] In one possible implementation, the acquisition module 23 is further configured to acquire a second depth sample image and second sample parameters corresponding to the second depth image sample. The second sample parameters include a third training position, a fourth training position, and a second reward function. The second sample parameters include parameters obtained based on the simulator. The second reward function is used to indicate whether clothing is inserted into the second end of the hanger.

[0091] Training module 24 is also used to train an initial neural network based on the second depth sample image and the second sample parameters to obtain a third neural network.

[0092] In this embodiment, based on the acquired first depth image, a first position is obtained through a first neural network to control the end of the first robotic arm to move to the first position, and the gripper at the end of the first robotic arm performs a first operation of grasping clothing; based on the first depth image and the first position, a second position is obtained through a second neural network to control the first end of the hanger fixed at the end of the second robotic arm to move to the second position, and the second robotic arm controls the hanger to perform a second operation of pressing down on the clothing and inserting the first end into the collar of the clothing; based on the acquired second depth image, a third position and a fourth position are obtained through a third neural network to control the gripper at the end of the first robotic arm to move to the third position to perform a third operation of grasping clothing, and the second robotic arm controls the hanger to perform a fourth operation of rotating and moving the second end of the hanger to the fourth position and inserting it into the collar of the clothing.

[0093] This allows the first and second robotic arms to collaborate and progressively insert the hanger into the collar of the garment, based on the first to fourth positions output by the neural network. In practice, the acquired depth image, or a depth image combined with the previous position, serves as input to the neural network to obtain a more precise output position. This guides the movement of the gripper at the end of the first robotic arm and the hanger at the end of the second robotic arm, enabling them to work together to complete the entire operation. This fills a technological gap in existing methods for inserting hangers into clothing, helping people to dry clothes and alleviate household chores.

[0094] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0095] It should be noted that, in the embodiments of this application, if the above-described model quantization method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Of course, the embodiments of this application are not limited to any specific hardware and software combination.

[0096] This application provides an electronic device, such as... Figure 6 As shown, the electronic device 30 includes a processor 31, a communication interface 32, a memory 33, and a communication bus 34, wherein the processor 31, the communication interface 32, and the memory 33 communicate with each other through the communication bus 34. Memory 33 is used to store computer programs; The processor 31 typically controls the overall operation of the electronic device 30 and, when executing the program stored in the memory 33, implements the above-mentioned method steps.

[0097] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0098] The communication interface is used for communication between the aforementioned electronic device and other devices. For example, it enables the electronic device to communicate with other terminals or servers via a network.

[0099] The memory is configured to store processor-executable instructions and applications, and may also cache data to be processed or already processed by various modules in the processor and electronic device, such as image data, audio data, voice communication data, and video communication data. It may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0100] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0101] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the above-described method. The computer-readable storage medium can be transient or non-transient.

[0102] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied as a computer storage medium; in another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0103] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0104] It should be noted that the descriptions of the above-described storage media, computer program products, and device embodiments are similar to those of the above-described method embodiments, and have similar beneficial effects. For technical details not disclosed in the embodiments of the storage media, computer program products, and devices of this application, please refer to the descriptions of the method embodiments of this application for understanding. That is, all embodiments in this specification are described in a related manner, and the same or similar parts between the embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions of the method embodiments.

[0105] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for inserting a clothes hanger into clothing, characterized in that, The method includes: Based on the acquired first depth image, a first position is obtained through a first neural network to control the end of the first robotic arm to move to the first position, and the first operation of grasping clothing is performed by the gripper at the end of the first robotic arm. Based on the first depth image and the first position, a second position is obtained through a second neural network to control the first end of the hanger fixed at the end of the second robotic arm to move to the second position. The second robotic arm controls the hanger to perform a second operation of pressing down the clothes and inserting the first end into the collar of the clothes. Based on the acquired second depth image, a third position and a fourth position are obtained through a third neural network. The gripper at the end of the first robotic arm is then moved to the third position to perform a third operation of grasping clothing. The second robotic arm controls the hanger to perform a fourth operation of rotating and moving the second end of the hanger to the fourth position and inserting it into the collar of the clothing.

2. The method according to claim 1, characterized in that, The process of obtaining the first position based on the acquired first depth image through a first neural network includes: The first depth image is used as input, the first neural network is used to obtain a first probability map, and a first unique heat map is obtained based on the first probability map to represent the first position coordinate information. And / or, The step of obtaining the second position based on the first depth image and the first position through a second neural network includes: The first depth image and the first unique heatmap are used as inputs, and the second probability map is obtained using the second neural network. Based on the second probability map, a second unique heatmap is obtained to represent the second position coordinate information. The first probability map and the second probability map are different, and both are used to reflect the success rate of the first stage operation corresponding to each pixel. The first stage operation includes the first operation and the second operation.

3. The method according to claim 1, characterized in that, The process of obtaining the third and fourth positions based on the acquired second depth image using a third neural network includes: Using the second depth image as input, the third neural network is used to obtain a third probability map, and based on the location of the voxel point with the maximum probability value in the third probability map, the third position coordinate information and the fourth position coordinate information are obtained. The third probability map is used to reflect the success rate of the second-stage operation corresponding to each voxel point, and the second-stage operation includes the third operation and the fourth operation.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes at least one of the following: In response to the execution of the fourth operation, a fifth operation of sorting clothes is performed by the gripper at the end of the first robotic arm; In response to the insertion of the first and second ends of the hanger into the neckline of the garment, the second robotic arm controls the hanger to adjust from a horizontal state to a vertical state so that the garment inserted into the hanger is hung up.

5. The method according to any one of claims 1 to 3, characterized in that, The method further includes: A first depth sample image is obtained, and a first sample parameter corresponding to the first depth image sample is obtained. The first sample parameter includes a first training position, a second training position, and a first reward function. The first sample parameter includes parameters obtained based on the simulator. The first reward function is used to indicate whether clothing is inserted into the first end of the hanger. Based on the first depth sample image and the first sample parameters, two initial neural networks are trained simultaneously to obtain the first neural network and the second neural network.

6. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Acquire a second depth sample image and a second sample parameter corresponding to the second depth image sample. The second sample parameter includes a third training position, a fourth training position, and a second reward function. The second sample parameter includes parameters obtained based on the simulator. The second reward function is used to indicate whether clothing is inserted into the second end of the hanger. Based on the second depth sample image and the second sample parameters, an initial neural network is trained to obtain the third neural network.

7. A device for inserting a hanger into clothing, characterized in that, The device includes: The processing module is used to obtain the first position based on the acquired first depth image through a first neural network; The control module is used to control the end of the first robotic arm to move to the first position, and to perform the first operation of grasping clothing through the gripper at the end of the first robotic arm; The processing module is further configured to obtain a second position based on the first depth image and the first position using a second neural network; The control module is also used to control the first end of the hanger fixed at the end of the second robotic arm to move to the second position, and to control the hanger to perform a second operation of pressing down the clothes and inserting the first end into the collar of the clothes through the second robotic arm; The processing module is also used to obtain the third and fourth positions based on the acquired second depth image through a third neural network; The control module is also used to control the gripper at the end of the first robotic arm to move to the third position to perform the third operation of grabbing clothing, and to control the hanger to perform the fourth operation of rotating and moving the second end of the hanger to the fourth position and inserting it into the collar of the clothing through the second robotic arm.

8. An electronic device, characterized in that, The electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes the program stored in the memory, it implements the steps of the method described in any one of claims 1 to 6.

9. A dual-arm fixed-stage system, characterized in that, The system includes a first robotic arm, a second robotic arm, and a depth camera; The depth camera is deployed above the operating table and is used to take pictures of the operating table and the clothing placed on the operating table from below. The first robotic arm and the second robotic arm are deployed on the operating table and located on both sides of the depth camera. The end of the first robotic arm is provided with a gripper, and the end of the second robotic arm is provided with a clothes hanger. The first robotic arm cooperates with the second robotic arm to insert the two ends of the hanger into the clothing placed on the operating table one by one, according to the method steps of any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1 to 6.

Images