Cross-device Linkage Method and Visual-Assisted Linkage System in Intelligent Warehouse

Abstract

The present application provides a cross-device linkage method and a vision-assisted linkage system in an intelligent warehouse. In the present application, a camera installed at the top of the warehouse captures real-time images of the loading and unloading equipment in the warehouse, a camera installed in front of the loading and unloading equipment collects real-time images of the goods in front of the equipment, and a target detection module calculates and identifies the real-time coordinate positions and transportation states of the loading and unloading equipment and the goods, so as to correspondingly trigger the four-way vehicle, the conveyor line and the corresponding personnel and equipment in the warehouse to adjust their working states. Through the vision-assisted technology, the present application can trigger the scheduling of the four-way vehicle to a transfer area matching the preset position when it is recognized that the loading and unloading equipment reaches the preset position, trigger the conveyor line to transport the goods when it is recognized that the goods reach the transportation position of the conveyor line, and trigger the avoidance of the equipment within the associated area range in real time according to the real-time coordinate position of the loading and unloading equipment, so as to improve the efficiency of warehouse work, improve the automation and intelligence level of the warehouse, and reduce the labor cost at the same time.

Description

Technical Field

[0001] This application relates to the field of intelligent warehouses, and more particularly to a cross-device linkage method and a vision-assisted linkage system in an intelligent warehouse. Background Art

[0002] An intelligent warehousing system is an intelligent system composed of three-dimensional shelves, rail-guided stacker cranes, inbound and outbound conveying systems, information recognition systems, automatic control systems, computer monitoring systems, computer management systems, and other auxiliary equipment. Intelligent warehousing is the product of warehouse automation. Similar to smart homes, intelligent warehousing can be achieved through a variety of automation and interconnection technologies. These technologies work together to improve the productivity and efficiency of the warehouse, minimize the number of manual workers, and reduce errors at the same time. In a manual warehouse, we usually see workers carrying a list, picking products, loading them into a shopping cart, and then transporting them to the shipping dock; but in an intelligent warehouse, orders are received automatically, and then the system confirms whether the products are in stock. Then the picking list is sent to the robotic cart, the ordered products are placed in a container, and then they are handed over to the workers for the next step. And intelligent warehousing completely solves the problem of dependence on manual labor. With the help of an intelligent warehousing system (such as C-WMS), goods are automatically received, identified, sorted, organized, and retrieved. The best intelligent warehousing solutions can almost automatically complete the entire operation from the supplier to the customer with the fewest errors.

[0003] Intelligent warehouses can save labor and land; on the same area of land, the storage capacity of an intelligent warehouse is several times or even more than a dozen times higher than that of a conventional warehouse. The inbound and outbound operations of an intelligent warehouse are rapid, accurate, and effective in shortening the operation time. At the same time, the operation accuracy rate is improved, and the warehouse can be coordinated organically with the suppliers and users, which is conducive to shortening the goods circulation time. Intelligent warehouses are conducive to the storage of goods. In an automated warehouse, there are many types of goods that can be stored, with a large quantity and diverse varieties. Through barcode technology and others, the flow of goods can be accurately tracked, and the traceability of goods can be achieved.

[0004] In existing intelligent warehouses, AGVs are usually used to replace manual handling, freeing people from handling operations. The AGV carts transport goods to the designated loading or unloading machines, realizing the intelligence and unmanned operation of the logistics transportation from material warehousing to the production line and then to finished product warehousing and shipping. In large intelligent warehouses, a large number of AGV forklifts are used to complete the handling of large goods. During the daily work of AGV forklifts, they need to interact and cooperate with other automated and intelligent devices in the intelligent warehouse. For example, the cooperation with the conveyor line. When the AGV forklift places the goods on the conveyor line, the conveyor line starts to transport. The cooperation with the four-way vehicle. After the goods are transported, the four-way vehicle performs the operation of warehousing or shipping the goods. In the process of these interactions and cooperations, the situation of insufficiently tight linkage connection may occur. For example, in the traditional situation, only after the goods are transported on the conveyor line, the four-way vehicle will start to transport the goods. At this time, often due to the large size of the warehouse and the large number of storage locations, the four-way vehicle may not be idle, resulting in a relatively long time spent in the connection process between the conveyor line and the four-way vehicle. This greatly reduces the work efficiency. In addition, in the traditional mode, after the AGV forklift places the goods on the conveyor line, manual operation is often required to trigger the start of the conveyor line to complete the transportation of the goods. This greatly reduces the automation and intelligence level of the warehouse and increases the labor cost at the same time. Summary of the Invention

[0005] In view of the deficiencies of the prior art, this application provides a cross-device linkage method and a vision-assisted linkage system in an intelligent warehouse. This application realizes cross-device linkage in the intelligent warehouse through a computer vision service system to improve the work efficiency of the warehouse, enhance the automation and intelligence level of the warehouse, and further reduce the labor cost. The specific technical solutions adopted in this application are as follows.

[0006] First, to achieve the above object, a cross-device linkage method in an intelligent warehouse is proposed. The steps include: receiving the real-time images of the handling equipment and goods captured by the vision hardware module; calculating and identifying the real-time coordinate positions and transportation states of the handling equipment and goods; according to the real-time coordinate positions and transportation states of the handling equipment and goods, triggering the real-time communication feedback module to send instructions to the four-way vehicle, conveyor line, and corresponding personnel and equipment in the warehouse to adjust their working states, including: triggering the scheduling of the four-way vehicle to the transfer area matching the preset position when it is recognized that the handling equipment reaches the preset position, triggering the conveyor line to transport the goods when it is recognized that the goods reach the transportation position of the conveyor line, and triggering the setting of the corresponding handling equipment, four-way vehicle, conveyor line, and / or the equipment set in the warehouse to be prohibited from operating in the area range associated with the real-time coordinate position according to the real-time coordinate position of the handling equipment obtained by the recognition.

[0007] Optionally, in the cross-device linkage method in the intelligent warehouse described above, the specific steps of calculating and identifying the real-time coordinate position of the loading and unloading equipment include: receiving the real-time image of the loading and unloading equipment collected by the camera installed above the warehouse; annotating the real-time image of the loading and unloading equipment to form a data set, constructing a deep neural network model, then identifying the loading and unloading equipment in the data set image through the deep neural network model, training the deep neural network model to determine the parameters in the deep neural network model, and then using the trained deep neural network model to identify the real-time image to obtain the pixel coordinates of the position box of the loading and unloading equipment, and calculating the real-time coordinate position corresponding to the loading and unloading equipment according to the pixel coordinates and the camera shooting parameters.

[0008] Optionally, in the cross-device linkage method in the intelligent warehouse described above, the specific steps of calculating and identifying the goods transportation status include: receiving the real-time image of the goods pallet in front of the loading and unloading equipment collected by the binocular or monocular camera installed in front of the loading and unloading equipment; annotating the real-time image of the goods pallet collected by the binocular or monocular camera in front of the loading and unloading equipment to form a data set, constructing a deep neural network model, then identifying the goods pallet in the data set image through the deep neural network model, training the deep neural network model to determine the parameters in the deep neural network model, and then using the trained deep neural network model to identify the real-time image to obtain the pixel coordinates of the position box on the front of the goods pallet, and calculating the front distance between the goods pallet and the front of the loading and unloading equipment according to the pixel coordinates to determine whether the goods pallet reaches the transportation position of the conveyor line to start transporting goods.

[0009] Optionally, in the cross-device linkage method in the intelligent warehouse described above, after calculating and identifying the real-time coordinate positions and transportation status of the loading and unloading equipment and the goods, the following steps are also executed: performing error analysis, mathematical modeling, and error reduction on the real-time coordinate positions and transportation status of the loading and unloading equipment and the goods obtained by the calculation and identification.

[0010] Optionally, in the cross-device linkage method in the intelligent warehouse described above, the specific steps of performing error analysis, mathematical modeling, and error reduction on the real-time coordinate positions and transportation status of the loading and unloading equipment and the goods obtained by the calculation and identification include: setting the error between the real-time coordinate positions of the loading and unloading equipment and the goods and the actual positions as E, calculating the distance between the shooting position of the vision hardware module and the actual position as L, calculating the camera distortion coefficient of the vision hardware module as T, constructing an error function, fitting the error function according to the collected error data to determine the error function parameters, and calibrating and reducing the error of the real-time coordinate positions of the loading and unloading equipment and the goods obtained by the calculation and identification according to the determined error function.

[0011] Meanwhile, to achieve the above object, the present application further provides a visual - assisted linkage system in an intelligent warehouse, which includes: a visual hardware module, which is arranged on the top of the warehouse and the handling equipment operating in the warehouse, and is used for taking real - time images of the handling equipment and goods; a target detection module, which receives the real - time images of the handling equipment and goods, calculates and identifies the real - time coordinate positions and transportation states of the handling equipment and goods, and triggers the four - way vehicle, conveyor line and corresponding personnel and equipment in the warehouse to adjust their working states according to the real - time coordinate positions and transportation states of the handling equipment and goods; a real - time communication feedback module, which is communicatively connected to the visual hardware module, target detection module, handling equipment, four - way vehicle, conveyor line and equipment set in the warehouse, transmits the real - time images of the handling equipment and goods between the visual hardware module and the target detection module, and transmits instructions to the corresponding handling equipment, four - way vehicle, conveyor line and equipment set in the warehouse according to the trigger signal of the target detection module to adjust their working states.

[0012] Optionally, for the visual - assisted linkage system in an intelligent warehouse described in any of the above, where the target detection module triggers the four - way vehicle, conveyor line and corresponding personnel and equipment in the warehouse to adjust their working states according to the real - time coordinate positions and transportation states of the handling equipment and goods, including: triggering the scheduling of the four - way vehicle to a transfer area matching the preset position when it is recognized that the handling equipment reaches the preset position; triggering the conveyor line to transport the goods when it is recognized that the goods reach the transportation position of the conveyor line; and triggering the corresponding handling equipment, four - way vehicle, conveyor line and / or equipment set in the warehouse to be set as prohibited from operating in the area range associated with the real - time coordinate position according to the real - time coordinate position of the handling equipment obtained by recognition.

[0013] Optionally, for the visual - assisted linkage system in an intelligent warehouse described in any of the above, which further includes an error analysis module, which performs error analysis, mathematical modeling and error reduction on the real - time coordinate positions and transportation states of the handling equipment and goods obtained by the target detection module.

[0014] Optionally, for the visual - assisted linkage system in an intelligent warehouse described in any of the above, where the visual hardware module includes: a binocular or monocular camera arranged in front of the AGV forklift, which is used for collecting real - time images of the goods pallet in front of the AGV forklift; the target detection module includes: a small server arranged on the AGV forklift, which is used for annotating the real - time images of the goods pallet collected by the binocular or monocular camera in front of the AGV forklift to form a data set, constructing a deep neural network model, then identifying the goods pallet through the deep neural network model, obtaining the pixel coordinates of the position box on the front of the goods pallet, calculating the front - face distance between the goods pallet and the AGV forklift according to the pixel coordinates, and triggering the real - time communication feedback module to issue an instruction to the conveyor line to transport the goods when the goods pallet reaches the transportation position of the conveyor line.

[0015] Optionally, in the visual - assisted linkage system in the intelligent warehouse described in any of the above, the visual hardware module further includes: a camera installed above the warehouse for collecting real - time images of the AGV forklift; a server connected to the camera above the warehouse for annotating the real - time images of the AGV forklift to form a data set, constructing a deep - neural - network model, then identifying the AGV forklift through the deep - neural - network model to obtain the pixel coordinates of the position box of the AGV forklift, calculating the real - time coordinate position corresponding to the AGV forklift based on the pixel coordinates and the camera shooting parameters, triggering the real - time communication feedback module to issue an instruction to the four - way vehicle to dispatch the four - way vehicle to the transfer area matching the preset position when the AGV forklift reaches the preset position, or triggering the real - time communication feedback module to issue an instruction to the four - way vehicle, the conveyor line, and / or other devices in the warehouse to prohibit them from running to the area range associated with the real - time coordinate position according to the real - time coordinate position of the AGV forklift.

[0016] Beneficial effects

[0017] In this application, a camera installed on the top of the warehouse is used to capture real - time images of the loading and unloading equipment in the warehouse, a binocular or monocular camera installed in front of the loading and unloading equipment is used to collect real - time images of the goods in front of the equipment, and the target - detection module calculates and identifies the real - time coordinate positions and transportation states of the loading and unloading equipment and the goods, so as to trigger the four - way vehicle, the conveyor line, and the corresponding personnel and equipment in the warehouse to adjust their working states accordingly. This application automatically realizes the cross - equipment linkage in the intelligent warehouse through visual - assistance technology, so as to trigger the dispatch of the four - way vehicle to the transfer area matching the preset position when it is recognized that the loading and unloading equipment reaches the preset position, trigger the conveyor line to transport the goods when it is recognized that the goods reach the transportation position of the conveyor line, and trigger the equipment within the associated area range to avoid in real - time according to the real - time coordinate position of the loading and unloading equipment. This application can realize the cross - equipment linkage in the intelligent warehouse through the computer - vision service system, improve the efficiency of warehouse work, enhance the automation and intelligence level of the warehouse, and further reduce the labor cost.

[0018] This application is also designed with an error - analysis module, which can analyze various factors affecting the results of the two - dimensional coordinates of the warehouse plane and the front - face distance of the pallet, such as the distortion coefficient of camera imaging, the height distance from the warehouse camera to the AGV forklift, the actual front - face distance from the camera in front of the AGV forklift to the goods pallet, etc. Then, taking the influencing factors as independent variables and the errors of the two - dimensional coordinates of the AGV forklift in the warehouse plane and the front - face distance error of the goods pallet as dependent variables, a mathematical model is established. Based on a large amount of data related to the influencing factors collected during the daily operation and maintenance of the warehouse, as well as a large amount of data on the two - dimensional coordinate errors of the warehouse plane and the front - face distance errors of the goods pallet, data fitting is performed on the mathematical model to find a more suitable relationship between the result error and the influencing factors, and then it is used to reduce the errors generated when the target - detection module identifies and calculates the loading and unloading equipment and the goods.

[0019] In addition, the real-time communication module of the present application uses the TCP or IP communication protocol to send the linkage control requirements calculated by the target detection module to the corresponding device units in real time in the form of instructions. When the AGV forklift reaches a specific position, the four-way vehicle is scheduled in advance to wait at the warehouse entrance. In this way, when the goods arrive, the goods can be directly obtained without waiting for the four-way vehicle to be dispatched from a distance. At the same time, during the warehousing process, the present application can also send the position distance between the goods pallet and the conveyor line to the control system of the conveyor line in real time. When the distance between the goods pallet reaches a certain value, it is determined that the AGV forklift has completely placed the goods on the conveyor line. At this time, the conveyor line is automatically triggered to move, reducing the steps of manually operating the conveyor line during the warehousing process, reducing labor costs, and improving the automation and intelligence level of the warehouse. In addition, the present application can also determine the working areas of each AGV forklift by obtaining the accurate position of the AGV forklift in the warehouse in real time, and then prohibit the entry of warehouse staff and other mobile intelligent devices into this area to improve the safety of equipment and personnel in the warehouse. During the outbound process, according to the real-time position of the AGV forklift, the AGV forklift closest to the conveyor line can be selected to arrive at the conveyor line position in advance to quickly receive and obtain the goods when the goods arrive. Through the comprehensive scheduling between devices, the present application can effectively improve the warehousing and outbound efficiency of the warehouse and improve the safety of the warehouse operation environment.

[0020] Other features and advantages of the present application will be described in the following specification, and in part, will be obvious from the specification, or will be understood by implementing the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings are used to provide a further understanding of the present application, and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation to the present application. In the drawings:

[0022] Figure 1 is a schematic diagram of the visual assistance linkage system in the intelligent warehouse of the present application;

[0023] Figure 2 is a schematic diagram of the binocular camera imaging model adopted in the system of the present application;

[0024] Figure 3 is a schematic diagram of the principle of the real-time image of the loading and unloading equipment collected by the camera above the warehouse of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0025] To make the objectives and technical solutions of the embodiments of this application clearer, the following will clearly and completely describe the technical solutions of the embodiments of this application with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, rather than all of them. Based on the described embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of this application.

[0026] Those skilled in the art of this technology can understand that, unless otherwise defined, all terms used herein (including technical terms and scientific terms) have the same meaning as the general understanding of those of ordinary skill in the field to which this application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have a meaning consistent with the meaning in the context of the prior art, and will not be interpreted with an idealized or overly formal meaning unless defined as such here.

[0027] The meaning of "and / or" as described in this application refers to the situation where each exists alone or both exist simultaneously.

[0028] The meaning of "connection" as described in this application can be a direct connection between components or an indirect connection between components through other components.

[0029] The meaning of "up and down" as described in this application refers to when the user is facing the forward direction of the conveyor line, the direction from the ground to the goods on the conveyor line is up, and vice versa, rather than a specific limitation on the device mechanism of this application.

[0030] Figure 1 A vision-assisted linkage system in an intelligent warehouse according to this application includes:

[0031] A vision hardware module that deploys vision hardware for the functions to be completed. For example, cameras, edge computing servers, etc. are correspondingly arranged on the top of the warehouse and loading and unloading equipment such as AGV forklifts operating in the warehouse, and are used to capture real-time images of the operating conditions of loading and unloading equipment such as AGV forklifts and the goods they carry.

[0032] A target detection module that receives real-time images of the loading and unloading equipment and goods, calculates and identifies the real-time coordinate positions and transportation states of the loading and unloading equipment and goods on the basis of the vision hardware, and automatically triggers the four-way vehicle, conveyor line, and corresponding personnel and equipment in the warehouse to adjust their working states according to the real-time coordinate positions and transportation states of the loading and unloading equipment and goods and the equipment linkage requirements.

[0033] A real-time communication feedback module, which is communicatively connected to a vision hardware module, a target detection module, handling equipment, an omnidirectional vehicle, a conveyor line, and equipment installed in a warehouse, is used to transmit real-time images of the handling equipment and goods between the vision hardware module and the target detection module, and send the trigger signals calculated and output in real time by the target detection module to the corresponding handling equipment, omnidirectional vehicle, conveyor line, and equipment installed in the warehouse through various types of communication protocols such as the TCP protocol and the IP protocol, so that each device can adjust its working state according to the received linkage instructions, reach a specified position to receive goods, or avoid entering a specific area to ensure equipment safety, or start adjusting the operating state to achieve goods transfer.

[0034] Considering that in practical applications, machine vision is often affected by factors such as camera distortion and distance error, which affect the accurate judgment of the real-time positions of handling equipment and goods. Therefore, this application can further set an error analysis module in the above system to perform error analysis, mathematical modeling, and error reduction on the real-time coordinate positions and transportation states of the handling equipment and goods obtained by the target detection module. The specific working process can be set and referenced as follows:

[0035] First, analyze the factors affecting the results of the two-dimensional coordinates of the warehouse plane and the front distance of the pallet, such as the distortion coefficient of camera imaging, the height distance from the warehouse camera to the AGV forklift, and the actual front distance from the front camera of the AGV forklift to the goods pallet. Then, establish two linear or quadratic constant coefficient equations with the influencing factors as independent variables and the two-dimensional coordinate error of the AGV forklift in the warehouse plane and the front distance error of the goods pallet as dependent variables. Collect a large amount of data related to the influencing factors, as well as a large amount of two-dimensional coordinate error data of the warehouse plane and front distance error data of the goods pallet, perform data fitting, determine the coefficients of the two equations established above, find a more suitable relationship between the result error and the influencing factors, and then correct the detection error through the above error equations to reduce the result error. For example, taking the error E between the real-time coordinate distance and the actual distance between the AGV forklift and the goods pallet as an example, the main influencing factors of this error E are the distance L between the shooting position and the actual position of the vision hardware module (it is found during testing that the larger L is, the larger the error is), and the distortion coefficient T of the front camera of the AGV. Respectively construct the error functions E = aL + bT + c and E = aL 2+bT+c, where a, b, c are constants. Then, the above error function is fitted according to the collected error data, a function with better effect is determined to reduce the error, and the specific parameter values in the error function are clarified. The construction of the above error function is not limited to these two forms, and the function form can be adjusted according to the actual situation. The method of reducing the error of the AGV forklift warehouse plane two-dimensional coordinate is similar. The error of the real-time coordinate position of the loading and unloading equipment and goods obtained by the calculation and identification is calibrated and reduced according to the determined error function. The focus of the above error reduction process is to continuously analyze and test the factors that affect the error size.

[0036] For general smart warehouses, during the warehousing process, the goods are usually placed on cargo pallets, and the AGV forklift is first called to receive the goods in the warehousing area. After the AGV forklift transports the goods to the corresponding position of the warehouse conveyor line, the conveyor line is triggered to transport the goods to the entrance of the three-dimensional shelf, and then the four-way vehicle running in the three-dimensional shelf is called accordingly to receive the goods on the conveyor line and transport them to the corresponding cargo position on the three-dimensional shelf. The outbound process corresponds to this, and generally the four-way vehicle is required to first take out the goods on the corresponding cargo position in the three-dimensional shelf and transport them to the conveyor line output port, start the conveyor line, transport the goods received on the conveyor line to the outer delivery position, and then take out the goods by the AGV forklift to realize outbound delivery.

[0037] In this process, the coordination and linkage between the four-way vehicle, transmission line, AGV forklift and other equipment is required. In order to improve the warehouse throughput and minimize the waiting time during the operation of the four-way vehicle, transmission line and AGV forklift, this application can set the target detection module to perform linkage control in the following manner according to the real-time image collected by the visual hardware module:

[0038] Receive real-time images of loading and unloading equipment and cargo captured by the visual hardware module;

[0039] Calculate and identify the real-time coordinate location and transportation status of loading and unloading equipment and cargo;

[0040] According to the real-time coordinate position and transportation status of the loading and unloading equipment and goods, the real-time communication feedback module is triggered to send instructions to the four-way vehicles, conveyor lines and corresponding personnel and equipment in the warehouse to adjust their working status, including: when it is identified that the loading and unloading equipment has arrived at a preset position, the four-way vehicle is triggered to be dispatched to the transfer area matching the preset position; when it is identified that the goods have arrived at the conveyor line transportation position, the conveyor line is triggered to transport the goods; according to the real-time coordinate position of the loading and unloading equipment obtained by identification, the corresponding loading and unloading equipment, four-way vehicles, conveyor lines and / or equipment in the warehouse are triggered to be set to prohibit running to the area range associated with the real-time coordinate position.

[0041] Therefore, during the process of goods warehousing, the present application can send the real-time warehouse position of the AGV forklift to the four-way vehicle scheduling system. When the AGV forklift reaches a certain specific position, the four-way vehicle scheduling system can mobilize the four-way vehicle in advance to wait at the warehousing entrance. In this way, when the goods arrive, the four-way vehicle can directly obtain the goods without waiting. If in the traditional way, after the goods arrive, the four-way vehicle scheduling system then mobilizes the four-way vehicle to pick up the goods. Due to the large number of warehouse storage locations, this process often takes a considerable amount of time, and the warehousing efficiency is greatly improved under the visual assistance system. In addition, during the warehousing process, the real-time distance of the goods pallet can also be sent to the conveyor line control system. Only when the distance of the goods pallet reaches a certain value can it be considered that the AGV forklift has completely placed the goods on the conveyor line. At this time, the conveyor line movement can be automatically triggered through the real-time communication feedback module, greatly improving the warehousing safety, reducing the manpower required in the conveyor line control link during the warehousing process, further reducing the labor cost, and improving the automation and intelligence level of the warehouse. In addition, the above process can also determine the working area of the AGV forklift through its real-time warehouse position, and then prohibit the entry of warehouse staff and other moving intelligent devices into this area to improve the safety of equipment and personnel work in the warehouse. During the outbound process, the AGV closest to the conveyor line can be selected in advance to reach the conveyor line to obtain the goods according to the real-time warehouse position of the AGV forklift. Compared with the traditional way, this linkage method does not require waiting for the AGV forklift to be dispatched to obtain the goods after the goods arrive at the conveyor line, and can avoid a large amount of time wasted in the process of the AGV forklift. The present invention can greatly improve the outbound and inbound efficiency with the help of the visual assistance system.

[0042] To reduce the detection blind area and improve the accuracy of identifying the position of goods, the present application can specifically set the visual hardware module as follows:

[0043] A camera installed above the warehouse according to the area where the connection and linkage operation needs to be performed in the inbound and outbound processes. The top camera can deploy a server at the same time for image processing, and provide real-time feedback on the warehouse dynamics to the target detection module to assist multi-device linkage;

[0044] And a binocular or monocular camera installed in front of the AGV forklift. This camera can perform image processing through a small server for edge computing deployed on the AGV forklift. By real-time processing of the video stream of the camera installed in the front, it can complete the analysis and feedback of the real-time situation in front of the AGV forklift, and also provide real-time feedback on the warehouse dynamics to the target detection module to assist multi-device linkage.

[0045] Among them, the server on the AGV forklift is mainly used for identifying the cargo pallets on the conveyor chain and calculating the distance from the AGV forklift to the pallets. The camera above the warehouse is mainly used for identifying large targets such as AGV forklifts. When it comes to identifying cargo pallets, since the warehouse camera is relatively high and the cargo pallet targets are relatively small, the effect is not good and the calculation error is relatively large in practice. Therefore, the calculation of the real-time position of the goods is mainly completed through the images captured by the camera in front of the AGV forklift. The identification of the AGV forklift target and the cargo target can be achieved through two groups of independent cameras and the server. The images of the camera above the warehouse can be processed by a separate server.

[0046] When the target detection module identifies the real-time coordinate positions and transportation states of the loading and unloading equipment and the goods for the images collected by the vision hardware module, the following method can be specifically adopted.

[0047] First, use the camera above the warehouse to collect pictures of the AGV forklift, and then label the positions of the AGV forklift in these pictures to form a data set. Use the YOLO series of target detection algorithms to construct a deep neural network model, and then use the server deployed in advance on the warehouse top to complete the training of the deep neural network model, determine the parameters in the neural network model, and then deploy the trained model to the server. By obtaining the video stream of the camera above the warehouse in real time, the AGV forklift working in the warehouse can be identified, and the real-time position box of the AGV forklift in the pixel coordinate system can be obtained. The real-time position box of the AGV forklift is determined by four two-dimensional points and their connections in the pixel coordinate system. The center point coordinates of this box in the pixel coordinate system can be obtained by taking the average value. According to the camera imaging principle and using the internal parameters of the camera such as focal length and resolution and the coordinates of this center point in the pixel coordinate system, the two-dimensional coordinates of this center point in the world coordinate system (i.e., the real-time position of the AGV forklift on the warehouse plane) can be calculated. The calculation process is as follows:

[0048] Reference Figure 3 As shown, the world coordinate system is the coordinate system of the three-dimensional world defined by the user and is introduced to describe the position of the target object in the real three-dimensional space. The spatial position (Xw, Yw, Zw) of the AGV forklift on the object point plane in the world coordinate system can correspond to a spatial position (Xc, Yc, Zc) with the camera as the coordinate origin in the camera coordinate system. After the camera pinhole imaging, the spatial position (Xc, Yc, Zc) on its camera coordinate system will be corresponding to a set of pixel coordinates (x, y) on the image plane. The unit is m.

[0049] Among them, the camera coordinate system is established with the camera as the coordinate origin, describing the spatial position of an object from the perspective of the camera. The image coordinate system: takes the center of the pixel image captured by the camera as the coordinate origin, and its X and Y axes are parallel to the two sides of the image. The coordinate value of an object can be represented by (x, y), and the unit is m. The AGV forklift in the image coordinate system corresponds to the pixel coordinates (u, v) in the pixel coordinate system of this image. In the pixel coordinate system, a coordinate system with the upper left corner of the image as the origin and the X and Y axes parallel to the two sides of the image can be selected. The unit in the pixel coordinate system is the number of pixels.

[0050] Based on the above mapping relationship, first perform the conversion from the world coordinate system to the camera coordinate system: The motion of a three-dimensional rigid body in space consists of a rotation matrix R and a translation matrix t, and the two matrices form a transformation matrix.

[0051] The relationship between the world coordinate system and the camera coordinate system is shown in the following formula. Among them, is the spatial coordinate in the world coordinate system. is the coordinate in the camera coordinate system. Here, homogeneous coordinates are used. The main purpose of introducing homogeneous coordinates is to combine multiplication and addition in matrix operations, and homogeneous coordinates can represent points at infinity.

[0052] Then perform the conversion from the camera coordinate system to the image coordinate system:

[0053] According to Figure 3 the pinhole imaging principle of the camera shown: The pinhole plane (camera coordinate system) is between the image plane (image coordinate system) and the object point plane (chessboard plane), and the image formed is an inverted real image.

[0054] According to the principle of similar triangles, obtain the mapping relationship from in the camera coordinate system to the image plane as: Among them, f represents the focal length of the camera, which can be determined through camera parameters during shooting. Convert it into the form of matrix multiplication to obtain:

[0055]

[0056] Then perform the conversion from the image coordinate system to the pixel coordinate system:

[0057] Since the origin of the defined pixel coordinate system does not coincide with the origin of the image coordinate system, assuming that the coordinates of the origin of the image coordinate system in the pixel coordinate system are (u0, v0), the sizes of each pixel in the x-axis and y-axis directions of the image coordinate system are: dx, dy, and the coordinates of the image point in the actual image coordinate system are (x, y). Thus, the coordinates of the image point in the pixel coordinate system can be obtained as follows:

[0058]

[0059]

[0060] Converting it into the form of matrix multiplication is:

[0061]

[0062] Based on the above conversion process, the mapping relationship from the world coordinate system to the pixel coordinate system should be as follows:

[0063]

[0064] Among them, the camera internal parameter matrix M is: In the actual calculation process, the camera internal parameter M is known, and the height of the warehouse is fixed, that is, Zc is known. When the AGV forklift is recognized, we can obtain the pixel coordinates of the AGV forklift. Using the relationship between the world coordinate system and the pixel coordinate system, we can calculate the position of the AGV forklift in the world coordinate system.

[0065] For other devices associated with the real-time communication module, such as AGV forklifts, four-way vehicles, and conveyor lines, they can use the cameras installed in front of this device to collect pictures of the cargo trays in front of the device (mainly the cargo trays in front of the AGV forklift that need to be transported to the conveyor line), and then label the positions of these cargo trays to form a dataset. Use the YOLO series of object detection algorithms to build a deep neural network model, and then use the pre-deployed server to complete the training of the deep neural network model, determine the parameters in the neural network model, and then deploy the trained model to the edge computing servers installed on devices such as AGV forklifts. By obtaining the video stream of the cameras in front of devices such as AGV forklifts in real time, the recognition of the cargo trays is completed, and the position box of the front of the cargo tray is also obtained.

[0066] When the camera installed in front of equipment such as an AGV forklift is a depth camera, the present application can directly obtain the front distance from the AGV forklift to the conveyor line tray according to the coordinates of the cargo tray position box in pixel coordinates. If the camera installed in front of equipment such as an AGV forklift is a simple binocular camera, then, according to the binocular imaging principle of the camera, by calculating the binocular disparity, the front distance from the AGV forklift to the conveyor line tray can be calculated using the coordinates of the cargo tray position box in the pixel coordinates of the left and right cameras.

[0067] The specific process of performing binocular disparity method operations on the images collected by the binocular camera can be set as follows:

[0068] According to Figure 2 the binocular depth camera imaging model, P is a point in space, P1 and P2 are the imaging points of point P on the left and right image planes, f is the focal length, and OR and OT are the optical centers of the left and right cameras. It can be seen from the following figure that the optical axes of the left and right cameras are parallel. XR and XT are the distances of the two imaging points from the left edge of the image on the left and right image planes. If the two cameras have been calibrated to achieve parallel epipolar lines and the directions of the two optical axes are also parallel. Then the relationship between the disparity and the object depth is: It can be derived that: The proof process is as follows:

[0069] Known Figure 2 in: The widths of the left and right images are both L. According to the principle of similar triangles, it can be obtained that: where b1 can be expressed in terms of b, XR, and XT as

[0070] By transforming this derivation, the depth distance, that is, the distance from the AGV to the tray, can be obtained using the binocular disparity of the binocular camera.

[0071] For the images collected by the depth camera, since the camera contains laser ranging hardware, therefore, it can directly obtain the depth of a specified pixel point. Thus, the distance from the AGV to the tray can be directly obtained by recognizing the tray through the image data.

[0072] Based on the above imaging data, the present application can accordingly determine the real-time position of the AGV cart or the goods to trigger other devices to perform linkage accordingly, prepare to receive the corresponding goods, achieve efficient connection, and improve the efficiency of goods inbound and outbound. The present application installs cameras above the warehouse and in front of the AGV forklift, and uses the server to recognize the real-time position of the AGV forklift and the real-time position of the cargo tray corresponding to the images of each camera. Thus, through comprehensive scheduling based on the real-time position of the AGV forklift and the real-time position of the cargo tray, the following can be achieved:

[0073] Trigger the four-way vehicle to be mobilized to the warehousing entrance in advance according to the real-time position of the AGV forklift for waiting;

[0074] During the warehousing process, according to the real-time position of the goods pallet, automatically trigger the movement of the conveyor line when the AGV forklift has completely placed the goods on the conveyor line;

[0075] Set a prohibited area according to the real-time position of the AGV forklift to ensure the safety of equipment and personnel;

[0076] During the outbound process, select the AGV forklift closest in distance according to the real-time position of the AGV forklift to obtain the goods.

[0077] The above is only the implementation mode of this application, and its description is relatively specific and detailed, but it should not be understood as a limitation on the patent scope of this application. It should be noted that for those of ordinary skill in the art, without departing from the concept of this application, several deformations and improvements can still be made, and these all belong to the protection scope of this application.

Claims

1. A cross-device linkage method in an intelligent warehouse, characterized in that the steps Including: Receiving real-time images of the handling equipment and goods captured by the vision hardware module; Calculating and identifying the real-time coordinate positions and transportation status of the handling equipment and goods; According to the real-time coordinate positions and transportation status of the handling equipment and goods, triggering the real-time communication feedback module to send instructions to the four-way vehicle, conveyor line, corresponding personnel, and other equipment in the warehouse to adjust their working states, including: Triggering the scheduling of the four-way vehicle to the transfer area matching the preset position when it is recognized that the handling equipment reaches the preset position, triggering the conveyor line to transport the goods when it is recognized that the goods reach the transportation position of the conveyor line, and triggering the setting of the corresponding handling equipment, four-way vehicle, conveyor line, and / or other equipment set in the warehouse to be prohibited from running to the area range associated with the real-time coordinate position according to the real-time coordinate position of the handling equipment obtained by recognition.

2. The cross-device linkage method in the intelligent warehouse according to claim 1, wherein, The specific steps for calculating and identifying the real-time coordinate position of the handling equipment include: Receiving real-time images of the handling equipment captured by the camera installed above the warehouse; Annotating the real-time images of the handling equipment to form a data set, constructing a deep neural network model, then identifying the handling equipment in the data set images through the deep neural network model, training the deep neural network model to determine the parameters in the deep neural network model, and then using the trained deep neural network model to identify the real-time images to obtain the pixel coordinates of the position box of the handling equipment, and calculating the real-time coordinate position corresponding to the handling equipment according to the pixel coordinates and the camera shooting parameters.

3. The cross-device linkage method in the intelligent warehouse according to claim 1, characterized in that, The specific steps for calculating and identifying the transportation status of the goods include: Receiving real-time images of the goods pallet in front of the handling equipment captured by the binocular or monocular camera installed in front of the handling equipment; Annotating the real-time images of the goods pallet captured by the binocular or monocular camera in front of the handling equipment to form a data set, constructing a deep neural network model, then identifying the goods pallet in the data set images through the deep neural network model, training the deep neural network model to determine the parameters in the deep neural network model, and then using the trained deep neural network model to identify the real-time images to obtain the pixel coordinates of the position box on the front of the goods pallet, and calculating the distance between the goods pallet and the front of the handling equipment according to the pixel coordinates to determine whether the goods pallet reaches the transportation position of the conveyor line to start transporting the goods.

4. The cross-device linkage method in the intelligent warehouse according to claim 1, characterized in that, After calculating and identifying the real-time coordinate positions and transportation status of the handling equipment and goods, the following steps are also executed: Performing error analysis, mathematical modeling, and error reduction on the real-time coordinate positions and transportation status of the handling equipment and goods obtained by calculation and identification.

5. The cross-device linkage method in the intelligent warehouse according to claim 4, wherein, The specific steps for performing error analysis, mathematical modeling, and error reduction on the real-time coordinate positions and transportation status of the handling equipment and goods obtained by calculation and identification include: Let the error between the real-time coordinate position and the actual position of the loading and unloading equipment and the goods be E, let the distance between the shooting position of the vision hardware module and the actual position be L, and let the camera distortion coefficient of the vision hardware module be T. Based on constructing a first-order or second-order constant coefficient equation as the error function, and , where a, b, and c are constants. Then, according to the error data collected, fit the above error function to determine the specific parameter values in the error function, and calibrate and reduce the error of the real-time coordinate position of the loading and unloading equipment and the goods obtained by calculation and recognition according to the determined error function.

6. A visual assistance linkage system in an intelligent warehouse, characterized in that, Including: A vision hardware module, which is set on the top of the warehouse and the handling equipment running in the warehouse, and is used to capture real-time images of the handling equipment and goods; A target detection module that receives real-time images of the loading and unloading equipment and goods, calculates and identifies the real-time coordinate positions and transportation states of the loading and unloading equipment and goods, and triggers the adjustment of the working states of the four-way vehicle, conveyor line, corresponding personnel, and other equipment in the warehouse according to the real-time coordinate positions and transportation states of the loading and unloading equipment and goods; A real-time communication feedback module that is communicatively connected to the vision hardware module, target detection module, loading and unloading equipment, four-way vehicle, conveyor line, and other equipment installed in the warehouse, transmits real-time images of the loading and unloading equipment and goods between the vision hardware module and the target detection module, and transmits instructions to the corresponding loading and unloading equipment, four-way vehicle, conveyor line, and other equipment installed in the warehouse according to the trigger signal of the target detection module to adjust their working states; Among them, The target detection module triggers the adjustment of the working states of the four-way vehicle, conveyor line, corresponding personnel, and other equipment in the warehouse according to the real-time coordinate positions and transportation states of the loading and unloading equipment and goods, including: When it is recognized that the loading and unloading equipment reaches the preset position, trigger the scheduling of the four-way vehicle to the transfer area matching the preset position; When it is recognized that the goods reach the transportation position of the conveyor line, trigger the conveyor line to transport the goods; According to the real-time coordinate position of the loading and unloading equipment obtained by recognition, trigger the setting of the corresponding loading and unloading equipment, four-way vehicle, conveyor line, and / or other equipment installed in the warehouse to be prohibited from running to the area range associated with the real-time coordinate position.

7. The visual assistance linkage system in the intelligent warehouse according to claim 6, characterized in that, It further includes an error analysis module that performs error analysis, mathematical modeling, and error reduction on the real-time coordinate positions and transportation states of the loading and unloading equipment and goods obtained by the target detection module.

8. The visual-aided linkage system in the intelligent warehouse according to claim 6, wherein The loading and unloading equipment is an AGV forklift; the vision hardware module includes: a binocular or monocular camera installed in front of the AGV forklift for collecting real-time images of the goods pallet in front of the AGV forklift; The target detection module includes: a small server installed on the AGV forklift for annotating the real-time images of the goods pallet collected by the binocular or monocular camera in front of the AGV forklift, forming a data set, constructing a deep neural network model, then identifying the goods pallet through the deep neural network model, obtaining the pixel coordinates of the position box on the front of the goods pallet, calculating the front distance between the goods pallet and the AGV forklift according to the pixel coordinates, and triggering the real-time communication feedback module to send an instruction to the conveyor line to transport the goods when the goods pallet reaches the transportation position of the conveyor line.

9. The visual-aided linkage system in the intelligent warehouse according to claim 8, characterized in that, The vision hardware module further includes: A camera installed above the warehouse for collecting real-time images of the AGV forklift; A server connected to a camera above the warehouse is used to label the real-time images of the AGV forklift, form a dataset, construct a deep neural network model, then identify the AGV forklift through the deep neural network model to obtain the pixel coordinates of the position box of the AGV forklift, calculate the real-time coordinate position corresponding to the AGV forklift according to the pixel coordinates and the camera shooting parameters, trigger the real-time communication feedback module to issue an instruction to the four-way vehicle when the AGV forklift reaches the preset position to dispatch the four-way vehicle to the transfer area matching the preset position, or trigger the real-time communication feedback module to issue an instruction to the four-way vehicle, the conveyor line and / or other equipment in the warehouse according to the real-time coordinate position of the AGV forklift to prohibit them from running to the area range associated with the real-time coordinate position.

Images