Control method, device and system of carrying equipment and computer readable storage medium

Abstract

The invention discloses a control method, device and system of carrying equipment and a computer readable storage medium, and relates to the field of intelligent logistics. The control method of the carrying equipment comprises the steps of obtaining an image of an object collected by the carrying equipment, wherein the object is provided with a specified identifier; identifying a specified identifier in the image; determining deviation information of the object relative to the position of the carrying equipment according to the position of the specified identifier in the image; and indicating the carrying equipment to adjust the pose according to the deviation information. According to the embodiment of the invention, whether the object deviates or not is determined according to the position of the specified identifier in the image acquired by the carrying equipment, so that the carrying equipment can adjust the self pose based on the deviation. Therefore, the carrying equipment with the pose adjusted can more accurately obtain the object or unload the object, the risk that the object is overturned in the object carrying process and when the object is placed can be further reduced, and the carrying control accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of logistics technology, and in particular to a control method, device, system and computer-readable storage medium for handling equipment. Background Art

[0002] In automated warehouses, automated guided vehicles (AGVs) can be used to transport objects such as pallets. In some scenarios, AGVs need to be connected to other automated equipment such as forklifts. For example, a forklift can place a pallet on a rack, and an AGV can take the pallet from the rack and transport it to another location. Summary of the invention

[0003] After analysis, the inventor found that during the docking process, there are certain requirements for the position of the pallet, because the forks of the forklift and stacker need to be inserted into the pallet to pick up and place the pallet. If the position is inaccurate, there will be a risk of cargo tipping over during subsequent automated equipment handling, or even the equipment will report an error and fail to pick up the pallet. In addition, when the AGV transports the pallet, within a certain tolerance range, the pallet also needs to be in the center of the AGV as much as possible, otherwise there will be a risk of cargo tipping over.

[0004] The above risks of pallets come from errors in the handling process. The error may be small each time, but the accumulated errors after frequent handling will have adverse consequences.

[0005] A technical problem to be solved by the embodiments of the present invention is: how to reduce the error generated during the object handling process.

[0006] According to a first aspect of some embodiments of the present invention, there is provided a method for controlling a handling equipment, comprising: acquiring an image of an object captured by the handling equipment, the object being provided with a designated mark; identifying the designated mark in the image; determining deviation information of the object relative to a position of the handling equipment based on a position of the designated mark in the image; and instructing the handling equipment to adjust its posture based on the deviation information.

[0007] In some embodiments, identifying a designated marker in an image includes: extracting one or more contours from the image; and identifying the designated marker in the image based on attribute information of each contour and attribute information of a preset contour of the designated marker.

[0008] In some embodiments, the attribute information includes at least one of aspect ratio, area, and number of bumps.

[0009] In some embodiments, the attribute information is attribute information of a circumscribed rectangle of the outline.

[0010] In some embodiments, determining the deviation information of the object relative to the position of the handling equipment based on the position of the designated marker in the image includes: determining the displacement deviation of the object relative to the position of the handling equipment based on the displacement deviation between the center point of the circumscribed rectangle of the outline of the designated marker in the image and the center point of the circumscribed rectangle of the outline of the designated marker in the reference image; determining the angular deviation of the object relative to the direction of the handling equipment based on the angular deviation between the sides of the circumscribed rectangle of the outline of the designated marker in the image and the corresponding sides of the circumscribed rectangle of the outline of the designated marker in the reference image.

[0011] In some embodiments, the reference image is an image of the object acquired by the transport device when there is no relative position deviation between the transport device and the object.

[0012] In some embodiments, extracting one or more contours from an image includes: determining a binary image of the image; sequentially performing dilation and erosion processing on the binary image; and extracting one or more contours from the image obtained after the erosion processing.

[0013] In some embodiments, determining a binary image of an image includes: performing grayscale processing on the image; performing mean filtering on the image after the grayscale processing; and performing a binarization operation on the image after the mean filtering.

[0014] In some embodiments, the designated identifier is a structural feature of the object body.

[0015] In some embodiments, the designated mark is located at the bottom of the object, the object is placed on a carrying device, and the contact surface between the carrying device and the bottom of the object has an avoidance structure to reveal the designated mark.

[0016] In some embodiments, the object is a container for carrying cargo.

[0017] In some embodiments, the handling device determines its own position based on a ground identification code.

[0018] In some embodiments, the object is an object to be transported, and the image is a first image of the object captured by a transport device located at a first target position.

[0019] In some embodiments, the control method further includes: instructing the transport device to transport the object to a second target location.

[0020] In some embodiments, instructing the transport device to transport the object to the second target position includes: sending an object acquisition instruction to the transport device; sending a return instruction to the transport device, wherein the return instruction is used to instruct the transport device to return to the first target position; sending a move instruction to the transport device, wherein the move instruction is used to instruct the transport device to move to the second target position.

[0021] In some embodiments, the object is an object to be unloaded, and the image is a second image of the object captured by a handling device located at a second target location.

[0022] In some embodiments, the control method further includes: instructing the transport device to place the object at a second target location.

[0023] In some embodiments, determining the deviation information of the object relative to the position of the handling device according to the position of the designated mark on the object in the image includes: determining the deviation information according to the position of the designated mark on the object in the image and the deviation between the designated position in the image.

[0024] In some embodiments, determining deviation information based on the position of a designated mark on the object in the image and the deviation between the designated position in the image includes: determining a position deviation based on the position of a designated mark on the object in the image and the deviation between the designated position; determining a direction deviation based on the direction of a designated mark on the object in the image and the deviation between one or more designated directions in the image; determining deviation information based on the position deviation and the direction deviation.

[0025] In some embodiments, determining the deviation information based on the position of a designated mark on the object in the image and the deviation between the designated position in the image includes: determining the deviation information based on the position of the designated mark on the object in the image and the deviation between the position of the designated mark in a reference image, wherein the reference image is an image captured by the transport equipment when the relative position deviation between the transport equipment and the object is within a specified range.

[0026] According to a second aspect of some embodiments of the present invention, there is provided a control device for a handling equipment, comprising: an acquisition module, configured to acquire an image of an object captured by the handling equipment; an identification module, configured to identify a designated mark located on the object in the image; a determination module, configured to determine deviation information of the object relative to a position of the handling equipment based on a position of the designated mark in the image; and a posture adjustment indication module, configured to adjust the posture of the handling equipment based on the deviation information.

[0027] According to a third aspect of some embodiments of the present invention, there is provided a control device for a handling equipment, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute any of the aforementioned control methods for the handling equipment based on instructions stored in the memory.

[0028] According to a fourth aspect of some embodiments of the present invention, there is provided a control system for a transporting device, comprising: a control device for any one of the aforementioned transporting devices; and a transporting device.

[0029] In some embodiments, the transport equipment includes a vehicle body and an upper structure that is liftably disposed on the top of the vehicle body, and the upper structure is used to hold up objects.

[0030] In some embodiments, the upper structure is rotatably disposed on the vehicle body in a horizontal plane.

[0031] In some embodiments, the control system further includes: a carrying device for carrying the object, and a contact surface between the carrying device and the bottom of the object has an avoidance structure to reveal a designated mark located at the bottom of the object.

[0032] In some embodiments, the support device includes a bracket.

[0033] In some embodiments, the carrying device includes two brackets arranged at an interval, each bracket has an overlapping surface for overlapping objects, and a passage for the transport equipment to pass through is formed between the two brackets.

[0034] According to a fifth aspect of some embodiments of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the aforementioned control methods for transport equipment.

[0035] Some embodiments of the above invention have the following advantages or beneficial effects: the embodiments of the present invention determine whether an object is offset by the position of a designated mark in an image captured by a handling device, so that the handling device can adjust its posture based on the deviation. Thus, the handling device after adjusting the posture can more accurately obtain or unload an object, and can further reduce the risk of tipping over during the handling process and when placing the object, thereby improving the accuracy of handling control.

[0036] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0038] Figure 1 A schematic flow chart of a control method for handling equipment according to some embodiments of the present invention is shown.

[0039] Figure 2A A schematic flow chart of a control method for handling equipment according to other embodiments of the present invention is shown.

[0040] Figure 2B A schematic flow chart of a transport control method according to some further embodiments of the present invention is shown.

[0041] Figure 3A and 3B A schematic diagram of a transport control scenario is shown.

[0042] Figure 4 A schematic top view of a carrying device implemented with brackets arranged at intervals is shown.

[0043] Figures 5A to 5D The schematic diagram exemplarily shows the process of a latent AGV acquiring a pallet.

[0044] Figures 5E to 5H A schematic diagram of the process of a latent AGV unloading a pallet is shown as an example.

[0045] Figure 6 A schematic flow chart of a designated identification recognition method according to some embodiments of the present invention is shown.

[0046] Figures 7A to 7D The schematic diagram of identifying a designated logo by using image processing technology is exemplarily described.

[0047] Figure 8 A schematic structural diagram of a control device for a handling device according to some embodiments of the present invention is shown.

[0048] Fig. 9 A schematic structural diagram of a control system of a handling device according to some embodiments of the present invention is shown.

[0049] Fig.10 A schematic structural diagram of a control device for a handling device according to other embodiments of the present invention is shown.

[0050] Fig.11 A schematic structural diagram of a control device for handling equipment according to some further embodiments of the present invention is shown. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] The relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.

[0053] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0054] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0055] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0056] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0057] After analysis, the inventor found that if mechanical structures, positioning mechanisms and other pick-and-place mechanisms are used to achieve pallet correction, the system cost will be increased, and the installation of such equipment will reduce the flexibility of the system, because when laying out the equipment in the warehouse, it is necessary to consider the docking method and docking position with other equipment. At the same time, such equipment will also reduce the efficiency of the entire system, because mechanical equipment is required to position the pallet every time, but in fact, not every handling has errors that need to be corrected.

[0058] Therefore, the present invention proposes a control method, device, system and computer-readable storage medium for a handling equipment, which determines whether an object deviates from a target position through a designated identifier inherent in the object, and adjusts the posture of the handling equipment when a deviation exists, so that the handling equipment can accurately pick up and place the object.

[0059] Reference below Figure 1 An embodiment of a method for controlling a transport device according to the present invention will be described.

[0060] Figure 1 FIG. 1 is a flow chart showing a control method for a handling device according to some embodiments of the present invention. Figure 1 As shown, the control method of the transport equipment in this embodiment includes steps S102 to S108.

[0061] In step S102, an image of an object captured by a handling device is acquired, and a designated mark is set on the object.

[0062] The handling equipment is, for example, an AGV, and may also be other types of equipment for handling goods as needed. An image acquisition device, such as a camera, a video camera, etc., is provided on the handling equipment. The handling equipment may acquire images according to instructions, for example, receiving an image acquisition instruction after reaching the target location; or, the handling equipment may acquire images periodically; or, the handling equipment may acquire images after performing a specified operation, such as stopping, starting, accelerating, decelerating, controlling the handling structure of the handling equipment, etc. Those skilled in the art may also acquire images in combination with a variety of methods.

[0063] In an embodiment of the present invention, the type of the object can be determined according to the needs of the scene. In some embodiments, the object to be transported is a container for loading goods, such as a pallet.

[0064] The object to be transported has a designated mark, and the relative position of the designated mark and the object is fixed. In some embodiments, the designated mark is a structural feature of the object body, such as the edge of a cover at the bottom of a tray, or a raised or recessed structure with a clear corner, etc. The local structural features of the object have at least two characteristics. First, since the structural features are features generated by the batch and template production of the object, the position of the structural features in the object is the same for different objects of the same type and model; second, the structural features of the object body are difficult to be affected by external forces during the transportation process and change. Therefore, by using the structural features of the object body as the designated mark, and using whether the designated mark has deviations in the image to determine whether the object has deviations relative to a specific position, the accuracy of error calibration can be improved.

[0065] In step S104, a designated mark in the image is identified. The method of identifying the designated mark in the image will be further described below.

[0066] In step S106, the deviation information of the object relative to the position of the handling device is determined according to the position of the designated mark in the image.

[0067] The location of the handling equipment is the target location, which refers to the standard placement location of the object to be handled. The information of the location can be pre-stored for the handling equipment to read and navigate, such as stored in a warehouse management system, a database, a mobile terminal, or any handling equipment, and can also be stored in other devices as needed. However, due to errors in the object to be handled during the last handling or unloading process, or errors in the object to be handled due to collisions with other objects, or offsets of the object to be unloaded during transportation, there may be some deviations between the object to be handled and the target location. For example, the object to be handled should be located in the center of the bracket, but it is offset; for another example, the object to be unloaded should be located in the center of the handling mechanism of the handling equipment, but it is offset.

[0068] If there is a deviation between the position of the designated mark in the image and the standard position of the designated mark in the image, it means that there is a deviation between the object and the handling equipment, and further, there is a deviation with the position of the handling equipment. Therefore, in some embodiments, the deviation information is determined based on the position of the designated mark on the object in the image and the deviation between the designated positions. The designated position is the position of the designated mark of the object in the image, which is determined by pre-calculation or actual measurement, when there is no deviation or very small deviation in the placement of the object. For example, when it is expected that the object can be placed in the middle of the handling mechanism of the handling equipment, and the center of the camera and the handling mechanism are located in the same straight line perpendicular to the horizontal plane, the designated position is the center of the image.

[0069] In addition, when there is no deviation or a very small deviation in the placement of the object, the image of the object can be captured by the handling equipment located at the target position, and the position of the designated mark in the image can be determined as the designated position. In some embodiments, the deviation information is determined based on the deviation between the position of the designated mark on the object in the image and the position of the designated mark in the reference image, wherein the reference image is an image captured by the handling equipment when the relative position deviation between the handling equipment and the object is within a specified range.

[0070] In some embodiments, a designated direction can also be preset, and the deviation between the handling equipment and the designated direction can be determined, so that the angle deviation can be eliminated when the handling equipment adjusts its posture. The designated direction can also be obtained by pre-calculation or actual measurement. For example, when there is no deviation in the placement of the object or there is a very small deviation, an image of the object can be captured by a handling equipment located at the target position and in a direction that enables it to obtain the object, and the direction of the designated mark in the image is determined as the designated direction. There can be one or more designated directions. In some embodiments, the position deviation is determined based on the deviation between the position of the designated mark on the object in the image and the designated position; the direction deviation is determined based on the deviation between the direction of the designated mark on the object in the image and the one or more designated directions in the image; the deviation information is determined based on the position deviation and the direction deviation. By combining the position and direction, the current deviation information can be determined more accurately, thereby improving the accuracy of the handling control.

[0071] In some embodiments, the handling device determines its own position based on the ground identification code, and the deviation information is the deviation information of the object relative to the ground identification code (also referred to as "ground code") at the target position. For example, multiple ground codes can be deployed on the floor of the warehouse for positioning and navigation of the handling structure. The handling device can collect images through a camera with a shooting direction facing the ground, and identify the ground code in the collected image, thereby obtaining the current position information.

[0072] In some embodiments, the transport device is instructed to adjust its position according to the deviation information so that the relative position deviation between the transport device and the object is within a specified range. The adjustment includes at least one of displacement adjustment and direction adjustment.

[0073] Considering that the distance between two objects in the image is smaller than the distance between two objects in the real world, the mapping relationship between the image coordinate system and the world coordinate system can be predetermined. In the case where the deviation information includes the position deviation, the position where the handling device should be adjusted is determined according to the mapping relationship, for example, moving 10 cm to the east. At this time, the adjustment can be made by rotating and displacing the handling device.

[0074] In the case where the deviation information includes direction information, the deviation information can be adjusted by, for example, rotating the transport device. For example, in order to make the direction of the designated mark in the image meet the conditions, the front of the transport device needs to be placed facing west.

[0075] It should be noted here that although both the adjustment according to the displacement deviation and the adjustment according to the direction deviation may involve the adjustment of the direction, the purpose of the former is to drive the transport device to the specified position, while the latter is to make the transport device face the correct direction. For example, when adjusting according to the displacement deviation, the transport device moves 10 cm to the east, and the transport device may move with the front of the vehicle facing east; then, when adjusting according to the direction deviation, the front of the vehicle is rotated to face west. Of course, the adjustment of position and direction can also be performed at the same time, which will not be elaborated in the present invention.

[0076] In step S108, the transport device is instructed to adjust its posture according to the deviation information.

[0077] The above embodiment determines whether the object is offset by the position of the designated mark in the image captured by the handling device, so that the handling device can adjust its posture based on the deviation. Therefore, the handling device after adjusting the posture can more accurately obtain or unload the object, and can further reduce the risk of the object tipping over during the handling process and when placing the object, thereby improving the accuracy of the handling control.

[0078] After the posture adjustment of the transport device is performed based on the above embodiment, the transport device can be controlled to perform a transport operation. For example, the current target position acquires the object to be transported and moves to the next target position, or the object being carried by the transport device is unloaded at the target position. The following will exemplarily describe these two situations respectively.

[0079] Figure 2A FIG. 2 is a flow chart showing a control method for a handling device according to some other embodiments of the present invention. Figure 2A As shown, the control method of the transport equipment in this embodiment includes steps S202 to S210.

[0080] In step S202, a first image of an object to be transported, which is captured by a transport device located at a first target position, is acquired.

[0081] The first target position is a standard placement position of the object to be transported.

[0082] In step S204, a designated mark in the image is identified.

[0083] In step S206, first deviation information of the object relative to the first target position is determined according to the position of the designated mark on the object in the first image.

[0084] In step S208, the transport device is instructed to adjust its posture according to the first deviation information.

[0085] In step S210 , the transport device is instructed to transport the object to a second target location.

[0086] In some embodiments, an object acquisition instruction is sent to the transport device; a return instruction is sent to the transport device, wherein the return instruction is used to instruct the transport device to return to the first target position; and a movement instruction is sent to the transport device, wherein the movement instruction is used to instruct the transport device to move to the second target position. When returning to the right position, an adjustment opposite to the posture adjustment in step S206 can be made. Thus, the transport device can be made to move along the preset route as much as possible, reducing additional risks such as collisions caused by changes in the route.

[0087] Of course, as needed, the transport device may also be instructed to move directly to the second target position after acquiring the object.

[0088] Through the above embodiment, the deviation of the object to be transported relative to the first target position can be determined with the help of the first image, and the posture of the transport device can be adjusted based on the deviation. Therefore, the transport device after adjusting the posture can more accurately obtain the object, and can further reduce the risk of the object tipping over during the object transport process and when the object is placed, thereby improving the accuracy of the transport control.

[0089] Even if the Figure 2A Although the embodiment accurately obtains the object, during the handling process, the object may still move slightly on the handling equipment due to factors such as acceleration, deceleration, steering, and external force collision. In order to minimize the error in each link, some embodiments of the present invention also provide an embodiment for controlling the handling equipment during the unloading process to reduce the error.

[0090] Figure 2B FIG. 2 shows a flow chart of a transport control method according to some other embodiments of the present invention. Figure 2B As shown, the transport control method of this embodiment includes steps S212 to S220.

[0091] In step S212, a second image of the object to be unloaded, which is captured by the handling equipment located at the second target position, is acquired, wherein the object is located on the handling equipment.

[0092] In step S214, a designated logo in the second image is identified.

[0093] In step S216, second deviation information of the object relative to the second target position is determined according to the position of the designated mark on the object in the second image.

[0094] In step S218, the transport device is instructed to adjust its posture according to the second deviation information.

[0095] In step S220, the transport device is instructed to place the object at the second target location.

[0096] For example, the transport mechanism of the transport device is instructed to place the object at the second target position.

[0097] In some embodiments, after the transport device unloads the object, a return instruction is sent to the transport device, wherein the return instruction is used to instruct the transport device to return to the second target position. When returning to the right position, an adjustment opposite to the posture adjustment in step S216 can be made. Thus, the transport device can be returned to a predetermined position to advance to the next target position according to the positioning and navigation information. As needed, the transport device can also be instructed to move directly to the next target position after unloading the object.

[0098] Through the above embodiment, the deviation of the object to be unloaded relative to the second target position can be determined with the help of the second image, and the posture of the handling device can be adjusted based on the deviation. Therefore, the handling device after adjusting the posture can place the object more accurately, and can further reduce the risk of the object being handled again later, thereby improving the accuracy of handling control.

[0099] And, combined with Figure 2A and Figure 2B The embodiment can reduce the handling error in each link, thereby improving the accuracy of handling control as a whole.

[0100] A transport control scenario is described below by way of example.

[0101] In some embodiments, the designated mark is located at the bottom of the object, the object is placed on a carrying device, and the contact surface between the carrying device and the bottom of the object has an avoidance structure to reveal the designated mark.

[0102] This situation is applicable to handling equipment that can lurk at the bottom of the carrying device, such as a lurk-type AGV. At this time, the handling equipment has a camera with an upward shooting direction, so that the designated mark at the bottom of the object can be photographed.

[0103] In some embodiments, the support device includes a bracket.

[0104] Figure 3A and 3B The schematic diagram of the transport control scenario is shown in FIG. Figure 3A As shown, the tray 31 is placed on the bracket 32; Figure 3B In the embodiment, the latent AGV 33 moves under the bracket 33 to obtain the object.

[0105] In some embodiments, the transport equipment includes a vehicle body and an upper structure that is liftably disposed on the top of the vehicle body, and the upper structure is used to hold up objects.

[0106] For example, the bearing device is a bracket with a hollowed-out bearing surface, and the hollowed-out portion can reveal a designated logo.

[0107] For another example, the carrying device includes two brackets arranged at an interval, each bracket has an overlapping surface for overlapping objects, and a passage for the transport equipment to pass through is formed between the two brackets. Figure 4 The top view of the carrying device implemented by the brackets arranged at intervals is shown. The brackets 41 and 42 are arranged at intervals, and the handling device 43 can freely enter and exit the space between the brackets 41 and 42.

[0108] This deployment method is convenient and flexible, and for a handling device that lifts an object by raising the upper structure, it is easy for the handling device to leave the current position. Of course, for other types of carrying devices, the free entry and exit of the handling device can also be achieved by setting a walking channel for the handling device and a avoidance structure of the upper structure.

[0109] Figures 5A to 5D The schematic diagram of the process of the latent AGV acquiring a pallet is exemplarily shown. For the sake of convenience, it is assumed that the upper structure of the AGV is located at the center of the handling equipment.

[0110] like Figure 5A As shown, the pallet 51 is placed on the load-bearing surfaces of the brackets 52 and 53. Due to the error caused by the previous handling process, the center line of the ground code 54 (indicated by a dotted line) and the center line of the pallet 51 carrying the goods (indicated by a dotted line) do not overlap, that is, there is a deviation between the pallet 51 and the first target position represented by the ground code 54. Based on the navigation and positioning results, AGV55 enters between the brackets 52 and 53 and searches for the ground code 54. After moving to the position of the ground code 54, AGV55 uses an upward camera to capture a first image, and determines the deviation between the pallet 51 and the ground code 54 based on the designated mark in the first image. The deviation also represents the deviation between the pallet 51 and the AGV55. Figure 5B In , AGV55 adjusts its own position according to the determined deviation information to minimize the deviation between AGV55 and pallet 51, and the center lines of the two coincide. Figure 5C In the process, AGV55 raises its upper structure and lifts up the tray 51. Figure 5D In the process, AGV55 carrying pallet 51 returns to ground code 54 and proceeds to the next target location from there.

[0111] Figures 5E to 5H A schematic diagram of the process of a latent AGV unloading a pallet is shown as an example.

[0112] like Figure 5EAs shown, AGV55 carrying pallet 51 moves between brackets 56 and 57 and moves toward ground code 58. After moving to the ground code 58, AGV55 uses an upward camera to capture a second image and determines the deviation between pallet 51 and ground code 58 based on the specified mark in the second image. This deviation also represents the deviation between pallet 51 and AGV55. Fig. 5F In the process, AGV55 adjusts its own position according to the determined deviation information to minimize the deviation between pallet 51 and ground code 58, and the center lines of the two coincide with each other. Figure 5G In the process, AGV 55 lowers its upper structure to place the pallet 51 on the brackets 56 and 57, completing the unloading of the pallet 51. Figure 5H In the process, AGV55 carrying pallet 51 returns to ground code 58 and proceeds to the next target location from there.

[0113] Reference below Figure 6 An embodiment of identifying a specified logo from an image is described.

[0114] Figure 6 FIG. 2 is a flow chart showing a method for identifying a designated identifier according to some embodiments of the present invention. Figure 6 As shown, the designated identification identification method of this embodiment includes steps S602 to S604.

[0115] In step S602, one or more contours are extracted from the image.

[0116] In some embodiments, a binary image of an image is determined; the binary image is dilated and eroded in sequence; and one or more contours are extracted from the image obtained after the erosion process. The dilation operation can handle defects such as disconnected image edges.

[0117] In some embodiments, the following method can be used to determine the binary image: grayscale processing the image; mean filtering the grayscale processed image; and binarization of the mean filtered image. When mean filtering is performed, for example, the current pixel and the pixels above, below, left, and right thereof (a total of 5 pixels) are mean filtered.

[0118] In step S604, the designated marker in the image is identified according to the attribute information of each contour and the attribute information of the preset contour of the designated marker.

[0119] In some embodiments, the attribute information includes at least one of aspect ratio, area, and number of convex points. Thus, according to the attribute information, the designated mark in the image can be quickly and accurately identified.

[0120] The attribute information is the attribute information of the circumscribed rectangle of the contour. The circumscribed rectangle is close to the designated mark on the one hand, and is easy to identify and calculate on the other hand, so the efficiency of recognition can be improved. For example, by calculating the aspect ratio, area, number of convex points and other information of the circumscribed rectangle, the designated mark can be quickly and accurately identified.

[0121] For example, after obtaining the outer contour of the image, save all continuous contour points on the boundary; calculate the height and width of the bounding rectangle of each contour; confirm the candidate bounding rectangle according to the ratio of width to height; from the candidate bounding rectangles, confirm the bounding rectangle with the target area and the rectangle with contour convex points greater than or equal to 4 as the target bounding rectangle; further, the center point of the target bounding rectangle can be obtained, and the target contour can be drawn and confirmed. The center point can be used to assist in determining the deviation information of the specified mark in the image, such as the position deviation.

[0122] In some embodiments, the displacement deviation of the object relative to the position of the handling device is determined based on the displacement deviation between the center point of the circumscribed rectangle of the outline of the designated mark in the image and the center point of the circumscribed rectangle of the outline of the designated mark in the reference image; the angular deviation of the object relative to the direction of the handling device is determined based on the angular deviation between the side of the circumscribed rectangle of the outline of the designated mark in the image and the corresponding side of the circumscribed rectangle of the outline of the designated mark in the reference image. The corresponding side refers to the same type of side, for example, the longer side or the shorter side of the two lengths of sides in the rectangle.

[0123] In some embodiments, the reference image is an image of the object obtained by the handling device when there is no relative position deviation between the handling device and the object. Of course, the reference image can also be obtained by other means, for example, by three-dimensional modeling of a virtual scene and obtaining an image from the virtual scene.

[0124] Through the above embodiment, the designated mark in the image can be accurately identified. Therefore, when the deviation information is determined according to the identified designated mark, a more accurate result can be obtained, thereby improving the accuracy of the control of the handling equipment.

[0125] Figures 7A to 7D The schematic diagram of identifying the designated mark by image processing technology is described exemplarily. In this embodiment, the designated mark is the NFC cover at the bottom of the tray, and the cover is a rounded rectangle. Fig. 7A The result of binarization of the image is shown. Figure 7B Shows the Fig. 7A The result of the expansion process. Figure 7C Shows the Figure 7B The result of the corrosion treatment. Fig.7D Shows the Figure 7CBy calculating the attribute information of the circumscribed rectangle and comparing it with the reference information, it can be confirmed that the outline of the current mark is the outline of the cover body, that is, the outline of the designated mark.

[0126] Reference below Figure 8 An embodiment of a control device for handling equipment of the present invention will be described.

[0127] Figure 8 FIG. 2 is a schematic diagram showing a structure of a control device for a handling device according to some embodiments of the present invention. Figure 8 As shown, the control device 800 of the handling equipment of this embodiment includes: an acquisition module 8100, configured to acquire an image of an object captured by the handling equipment; an identification module 8200, configured to identify a designated mark located on the object in the image; a determination module 8300, configured to determine the deviation information of the object relative to the position of the handling equipment according to the position of the designated mark in the image; and a posture adjustment indication module 8400, configured to adjust the posture of the handling equipment according to the deviation information.

[0128] In some embodiments, the recognition module 8200 is further configured to extract one or more contours from the image; and recognize a designated identifier in the image based on the attribute information of each contour and the attribute information of the preset contour of the designated identifier.

[0129] In some embodiments, the attribute information includes at least one of aspect ratio, area, and number of bumps.

[0130] In some embodiments, the attribute information is attribute information of a circumscribed rectangle of the outline.

[0131] In some embodiments, the determination module 8300 is further configured to determine the displacement deviation of the object relative to the position of the handling equipment based on the displacement deviation between the center point of the circumscribed rectangle of the outline of the specified mark in the image and the center point of the circumscribed rectangle of the outline of the specified mark in the reference image; and determine the angular deviation of the object relative to the direction of the handling equipment based on the angular deviation between the sides of the circumscribed rectangle of the outline of the specified mark in the image and the corresponding sides of the circumscribed rectangle of the outline of the specified mark in the reference image.

[0132] In some embodiments, the reference image is an image of the object acquired by the transport device when there is no relative position deviation between the transport device and the object.

[0133] In some embodiments, the recognition module 8200 is further configured to determine a binary image of the image; perform dilation and erosion processing on the binary image in sequence; and extract one or more contours from the image obtained after the erosion processing.

[0134] In some embodiments, the recognition module 8200 is further configured to perform grayscale processing on the image; perform mean filtering on the image after grayscale processing; and perform binarization operation on the image after mean filtering.

[0135] In some embodiments, the designated identifier is a structural feature of the object body.

[0136] In some embodiments, the designated mark is located at the bottom of the object, the object is placed on a carrying device, and the contact surface between the carrying device and the bottom of the object has an avoidance structure to reveal the designated mark.

[0137] In some embodiments, the object is a container for carrying cargo.

[0138] In some embodiments, the handling device determines its own position based on a ground identification code.

[0139] In some embodiments, the object is an object to be transported, and the image is a first image of the object captured by a transport device located at a first target position.

[0140] In some embodiments, the control device 800 further includes: a first instruction module 8500 configured to instruct the transport device to transport the object to a second target location.

[0141] In some embodiments, the first indication module 8500 is further configured to send an object acquisition instruction to the transport device; send a return instruction to the transport device, wherein the return instruction is used to instruct the transport device to return to the first target position; send a move instruction to the transport device, wherein the move instruction is used to instruct the transport device to move to the second target position.

[0142] In some embodiments, the object is an object to be unloaded, and the image is a second image of the object captured by a handling device located at a second target location.

[0143] In some embodiments, the control device 800 further includes: a second instruction module 8600 configured to instruct the transport device to place the object at a second target location.

[0144] In some embodiments, the second indication module 8600 is further configured to determine deviation information according to a deviation between a position of a designated mark on the object in the image and a designated position in the image.

[0145] In some embodiments, the determination module 8300 is further configured to determine the position deviation based on the position of a designated mark on the object in the image and the deviation between the designated positions; determine the direction deviation based on the direction of the designated mark on the object in the image and the deviation between one or more designated directions in the image; and determine the deviation information based on the position deviation and the direction deviation.

[0146] In some embodiments, the determination module 8300 is further configured to determine deviation information based on a deviation between a position of a designated mark on the object in the image and a position of the designated mark in a reference image, wherein the reference image is an image captured by the transport device when a relative position deviation between the transport device and the object is within a specified range.

[0147] Reference below Fig. 9 An embodiment of a control system for a handling device of the present invention will be described.

[0148] Fig. 9 FIG. 1 is a schematic diagram showing a control system of a handling device according to some embodiments of the present invention. Fig. 9 As shown, the control system 90 of this embodiment includes: a control device 910 for the transport equipment; and the transport equipment 920.

[0149] The control device 910 of the transporting device and the transporting device 920 are electrically connected or communicatively connected.

[0150] The control device 910 of the transport equipment may be deployed in the transport equipment 920, or may be remotely deployed, for example, deployed in a server.

[0151] The handling equipment is, for example, an AGV. The AGV can adjust its posture based on a differential model, that is, it can control the rotation speed and direction of the drive wheels respectively. In the process of adjusting the posture, the AGV can, for example, change the center position of the AGV by rotating. Of course, the AGV can also adjust its position only by walking in a straight line, that is, adjust the direction of the front of the vehicle to walk for a while, and then adjust the front of the vehicle to walk for another while to adjust its position.

[0152] In some embodiments, the transport equipment includes a vehicle body and an upper structure that is liftably disposed on the top of the vehicle body, and the upper structure is used to hold up objects.

[0153] In some embodiments, the upper structure is rotatably disposed on the vehicle body in a horizontal plane.

[0154] For example, when the vehicle body rotates, the upper structure can move in the opposite direction, so that from an outside perspective, only the trolley is rotating, and the object held by the upper structure does not rotate. In some embodiments, when adjusting the posture, the vehicle body first rotates the trolley to adjust the position, and after the trolley posture is correct, the upper structure is rotated alone to return the positive direction of the object to the positive direction. In this way, the movement of the object on the upper structure can be avoided as much as possible, further reducing the error.

[0155] However, it should be clear to those skilled in the art that it is also feasible to adjust the vehicle body and the upper structure simultaneously; or, it is also feasible to first rotate the upper structure to a desired angle and then adjust the position of the vehicle body.

[0156] In some embodiments, the transport control system further includes: a carrying device for carrying the object, and a contact surface between the carrying device and the bottom of the object has an avoidance structure to reveal a designated mark located at the bottom of the object.

[0157] In some embodiments, the support device includes a bracket.

[0158] In some embodiments, the carrying device includes two brackets arranged at an interval, each bracket has an overlapping surface for overlapping objects, and a passage for the transport equipment to pass through is formed between the two brackets.

[0159] Fig.10 Schematic diagrams of the structure of the control device of the handling equipment according to other embodiments of the present invention are shown. Fig.10 As shown, the control device 100 of the handling equipment of this embodiment includes: a memory 1010 and a processor 1020 coupled to the memory 1010, and the processor 1020 is configured to execute the control method of the handling equipment in any of the aforementioned embodiments based on the instructions stored in the memory 1010.

[0160] The memory 1010 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs.

[0161] Fig.11 FIG. 2 shows a schematic diagram of a control device for a handling device according to some other embodiments of the present invention. Fig.11 As shown, the control device 110 of the handling equipment of this embodiment includes: a memory 1110 and a processor 1120, and may also include an input / output interface 1130, a network interface 1140, a storage interface 1150, etc. These interfaces 1130, 1140, 1150 and the memory 1110 and the processor 1120 may be connected, for example, via a bus 1160. Among them, the input / output interface 1130 provides a connection interface for input / output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 1140 provides a connection interface for various networked devices. The storage interface 1150 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0162] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements any of the aforementioned control methods for the transport equipment.

[0163] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0164] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0165] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0166] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0167] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for controlling a handling device, comprising: Acquire an image of an object captured by a handling device, wherein a designated mark is provided on the object; identifying the designated identifier in the image; Determining deviation information of the object relative to the position of the handling device according to the position of the designated mark in the image; According to the deviation information, the transport device is instructed to adjust its posture.

2. The control method according to claim 1, wherein: The identifying the designated mark in the image comprises: From the image, extracting one or more contours; The designated mark in the image is identified according to the attribute information of each contour and the preset attribute information of the contour of the designated mark.

3. The control method according to claim 2, wherein: The attribute information includes at least one of aspect ratio, area, and number of convex points.

4. The control method according to claim 2, wherein: The attribute information is the attribute information of the circumscribed rectangle of the outline.

5. The control method according to claim 4, wherein: Determining the deviation information of the object relative to the position of the handling device according to the position of the designated mark in the image includes: Determine the displacement deviation of the object relative to the position of the handling device according to the displacement deviation between the center point of the circumscribed rectangle of the outline of the designated mark in the image and the center point of the circumscribed rectangle of the outline of the designated mark in the reference image; The angular deviation of the object relative to the direction of the transport device is determined based on the angular deviation between the sides of the circumscribed rectangle of the outline of the designated mark in the image and the corresponding sides of the circumscribed rectangle of the outline of the designated mark in the reference image.

6. The control method according to claim 5, wherein: The reference image is an image of the object acquired by the transport device when there is no relative positional deviation between the transport device and the object.

7. The control method according to claim 2, wherein: Extracting one or more contours from the image comprises: Determining a binary image of the image; Sequentially performing dilation and erosion processing on the binary image; Extract one or more contours from the image obtained after the erosion process.

8. The control method according to claim 7, wherein: Determining a binary image of the image comprises: Performing grayscale processing on the image; Perform mean filtering on the grayscale processed image; Binarize the image after mean filtering.

9. The control method according to claim 1, wherein: The designated mark is a structural feature of the object body.

10. The conveyance control method according to any one of claims 1 to 9, wherein: The designated mark is located at the bottom of the object, the object is placed on a carrying device, and the contact surface between the carrying device and the bottom of the object has an avoidance structure to reveal the designated mark.

11. The conveyance control method according to any one of claims 1 to 9, wherein: The object is a container for loading goods.

12. The control method according to claim 1, wherein: The handling equipment determines its own position and posture according to the ground identification code.

13. The control method according to any one of claims 1 to 9, wherein: The object is an object to be transported, and the image is a first image of the object captured by a transport device located at a first target position.

14. The control method according to claim 13, further comprising: The transport device is instructed to transport the object to a second target location.

15. The control method according to claim 14, wherein: The instructing the transport device to transport the object to the second target location comprises: sending an object acquisition instruction to the handling device; Sending a return instruction to the transport device, wherein the return instruction is used to instruct the transport device to return to the first target position; A movement instruction is sent to the transport device, wherein the movement instruction is used to instruct the transport device to move to the second target position.

16. The control method according to any one of claims 1 to 9, wherein: The object is an object to be unloaded, and the image is a second image of the object captured by a handling device located at a second target position.

17. The control method according to claim 16, further comprising: The handling device is instructed to place the object at a second target location.

18. The control method according to any one of claims 1 to 9, wherein: Determining the deviation information of the object relative to the position of the handling device according to the position of the designated mark on the object in the image includes: The deviation information is determined according to a deviation between a position of a designated mark on the object in the image and a designated position in the image.

19. The control method according to claim 18, wherein: Determining the deviation information according to the deviation between the position of the designated mark on the object in the image and the designated position in the image includes: Determine a position deviation according to a deviation between a position of a designated mark on the object in the image and the designated position; Determining a direction deviation according to a deviation between a direction of a designated mark on the object in the image and one or more designated directions in the image; The deviation information is determined according to the position deviation and the direction deviation.

20. The control method according to claim 18, wherein: Determining the deviation information according to the deviation between the position of the designated mark on the object in the image and the designated position in the image includes: The deviation information is determined based on a deviation between a position of a designated mark on the object in the image and a position of the designated mark in a reference image, wherein the reference image is an image captured by the transport device when a relative position deviation between the transport device and the object is within a specified range.

21. A control device for a handling device, comprising: an acquisition module, configured to acquire an image of the object captured by the handling device; a recognition module configured to recognize a designated mark located on the object in the image; a determination module configured to determine deviation information of the object relative to the position of the handling device according to the position of the designated mark in the image; The posture adjustment instruction module is configured to adjust the posture of the handling device according to the deviation information.

22. A control device for a handling device, comprising: Memory; as well as A processor coupled to the memory, wherein the processor is configured to execute the control method of the transport equipment according to any one of claims 1 to 20 based on instructions stored in the memory.

23. A control system for a handling device, comprising: A control device for a handling device according to claim 21 or 22; as well as Handling equipment.

24. The control system of claim 23, wherein: The transport equipment comprises a vehicle body and an upper structure which is liftably arranged on the top of the vehicle body, and the upper structure is used for holding up the object.

25. The control system of claim 24, wherein: The upper structure is rotatably arranged on the vehicle body in a horizontal plane.

26. The control system of claim 24, further comprising: The carrying device is used for carrying an object, and the contact surface between the carrying device and the bottom of the object has an avoidance structure to reveal a designated mark located at the bottom of the object.

27. The control system of claim 26, wherein: The carrying device comprises a bracket.

28. The control system of claim 26, wherein: The carrying device comprises two brackets arranged at an interval, each bracket having an overlapping surface for overlapping the object, and a passage for the transport equipment to pass through is formed between the two brackets.

29. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the control method of the transport equipment according to any one of claims 1 to 20 is implemented.

Images