Control method, system, and storage medium of a warehouse system

Abstract

The application discloses a control method of a warehouse system, a warehouse system and a storage medium. The warehouse system receives a live image collected by a visual sensor based on a warehouse robot, and a live state parameter collected by a state sensor arranged on the warehouse robot and / or a warehouse site. Then, based on the live image, the live state parameter and the identification mark of the robot, a virtual live picture is generated and sent to a user end. The user operation of the user end based on the virtual live picture is received, and based on the corresponding instruction conversion rule of the identification mark and the user operation, a robot control instruction is generated. Then, the control instruction is sent to the warehouse robot to control the warehouse robot to perform corresponding actions. Thus, the problem of high dependence on rules in related technologies is solved, and the adaptation between the solution and the problem is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing of a warehousing system, and in particular to a control method of a warehousing system, a warehousing system and a storage medium. Background Art

[0002] In modern warehousing scenarios, while artificial intelligence and robotics have significantly improved efficiency, this has also led to intelligent warehousing systems being highly dependent on preset rules, making it difficult to respond to sudden anomalies without historical data. When faced with some non-standardized problems, the need for human resources to make emergency responses based on comprehensive considerations still exists. For example, when a warehouse fire occurs, the cause and severity of the fire are unpredictable, and the same situation will not usually recur. When the system has no historical data to rely on, it is difficult to make effective emergency responses based on the current situation. Therefore, when faced with non-standardized problems, the decision-making solutions provided by the warehousing systems provided by related technologies are poorly adapted to the actual problems. Summary of the Invention

[0003] The main purpose of this application is to provide a control method for a warehousing system, a warehousing system and a storage medium, aiming to solve the technical problem in related technologies that the warehousing system has a high dependence on rules.

[0004] To achieve the above objectives, an embodiment of the present application provides a control method for a warehousing system, which is applied to a control terminal, one side of which is communicatively connected to a warehousing robot, and the other side of which is communicatively connected to a user terminal. The control method for the warehousing system includes:

[0005] Receive the on-site image collected by the storage robot based on the visual sensor, and the on-site status parameters collected by the status sensors arranged on the storage robot and / or the storage site;

[0006] generating a virtual scene image based on the scene image, the scene state parameters, and the identification mark of the robot, and sending the virtual scene image to the user terminal;

[0007] receiving a user operation of the user terminal in response to the virtual scene image, and generating a robot control instruction based on an instruction conversion rule corresponding to the identification identifier and the user operation;

[0008] The control instruction is sent to the storage robot to control the storage robot to perform corresponding actions.

[0009] In an embodiment of the present application, the step of receiving a user operation of the user terminal based on the virtual scene image response and generating a robot control instruction based on the instruction conversion rule corresponding to the identification identifier and the user operation includes:

[0010] determining a task target according to the user operation;

[0011] determining an executable action of the warehouse robot based on the identified identifier, and obtaining the instruction conversion rule;

[0012] decomposing the task target according to the task target and the executable action, and generating at least one subtask;

[0013] in the instruction conversion rule, searching for the robot control instruction corresponding to the subtask.

[0014] In the embodiment of the application, the step of determining a task target according to the user operation comprises:

[0015] determining a state change of at least one of a virtual cargo, a virtual device, and a virtual equipment in the virtual scene picture according to the user operation and the virtual scene picture;

[0016] determining the task target according to the state change.

[0017] In the embodiment of the application, the step of determining the task target according to the state change comprises:

[0018] if the virtual cargo position state in the virtual scene picture is updated, determining that the task is to move the cargo;

[0019] determining an initial position and an end position of the virtual cargo;

[0020] setting the task target as moving the cargo in the warehouse from the initial position to the end position.

[0021] In the embodiment of the application, the step of determining the task target according to the state change comprises:

[0022] if the trigger state of the virtual device is updated, setting the task target as updating the trigger state of the alarm device corresponding to the virtual device.

[0023] In the embodiment of the application, the step of determining the task target according to the state change comprises:

[0024] if the virtual equipment state is changed, obtaining a disassembly step and / or an assembly step of the virtual device;

[0025] setting the task target as a maintenance task, and generating an operation step corresponding to the maintenance task based on the disassembly step and / or the assembly step.

[0026] In an embodiment of the present application, the step of determining a state change of at least one of virtual goods, virtual devices, and virtual equipment in the virtual scene according to the user operation and the virtual scene includes:

[0027] Determine an operation object corresponding to the task target based on the current mode, wherein the operation object includes at least one of cargo, an alarm device, and equipment to be repaired;

[0028] The step of determining a state change of at least one of virtual goods, virtual devices, and virtual equipment in the virtual scene according to the user operation and the virtual scene is performed based on the operation object.

[0029] In the embodiment of the present application, before the step of determining the operation object corresponding to the task goal based on the current mode, the method further includes:

[0030] Sending a mode selection interface to the user terminal, wherein the mode selection interface includes at least one selectable mode among a shipping mode, an alarm mode, and a maintenance mode;

[0031] Receive a mode selection result of the user terminal based on the mode selection interface response, and determine the current mode according to the mode selection result.

[0032] The present application also provides a storage system, which includes:

[0033] A receiving module, configured to receive on-site images collected by the storage robot based on a visual sensor, and on-site status parameters collected by status sensors arranged on the storage robot and / or the storage site;

[0034] a data processing module, configured to generate a virtual scene image based on the scene image, the scene state parameters, and the identification identifier of the robot, and transmit the virtual scene image to the user terminal; and receive user operations of the user terminal in response to the virtual scene, and generate robot control instructions based on the instruction conversion rules corresponding to the identification identifier and the user operations;

[0035] The sending module is used to send the control instruction to the storage robot to control the storage robot to perform corresponding actions.

[0036] An embodiment of the present application further provides a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium, and when the computer program is executed by a processor, the steps of the control method of the warehousing system as described above are implemented.

[0037] The embodiment of the application discloses a control method of a warehouse system. The control method of the warehouse system is configured to control an end to receive a field image and a state parameter first, generate a virtual field picture based on the field image, the field state parameter and a recognition mark of a robot, and send the virtual field picture to the user end, and then generate a robot control instruction according to a user operation collected by the user end, a corresponding instruction conversion rule of the recognition mark, so as to realize the purpose of controlling the warehouse robot to realize the artificial control intention through the robot control instruction.

[0038] In the above method, the intelligent terminal can generate the robot control instruction based on the instruction conversion rule and the operation performed by the user on the virtual field picture, so that the method can realize the control of any warehouse robot as long as the corresponding instruction conversion rule is updated when the warehouse robot is newly added or the warehouse robot is updated. And there is no need to rearrange and design a new remote control system. At the same time, since the virtual field picture is generated according to the field image and the field state parameter, the remote controller can learn more accurate field conditions based on it, and then make more accurate remote control actions. In addition, the scheme realizes the purpose of remotely controlling the warehouse robot in a specific situation, so that the warehouse system can make artificial decisions and realize equipment control when facing non-standardized problems. In this way, the adaptation degree of the decision scheme to the actual problem is improved to a certain extent. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is an architecture schematic diagram of the warehouse system related to the embodiment scheme of the application;

[0040] Figure 2 is a flow schematic diagram of an embodiment of the control method of the warehouse system related to the embodiment scheme of the application;

[0041] Figure 3 is a structure schematic diagram of the control device of the warehouse system of the application;

[0042] Figure 4 is a modular structure schematic diagram of the warehouse system of the application.

[0043] The implementation, functional features and advantages of the application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0044] It should be understood that the specific embodiments described herein are only used to explain the application, and are not used to limit the application.

[0045] In modern warehousing scenarios, while artificial intelligence and robotics have significantly improved efficiency, this has also led to intelligent warehousing systems being highly dependent on preset rules, making it difficult to respond to sudden anomalies without historical data. When faced with some non-standardized problems, the need for human resources to make emergency responses based on comprehensive considerations still exists. For example, when a warehouse fire occurs, the cause and severity of the fire are unpredictable, and the same situation will not usually recur. When the system has no historical data to rely on, it is difficult to make effective emergency responses based on the current situation. Therefore, when faced with non-standardized problems, the decision-making solutions provided by the warehousing systems provided by related technologies are poorly adapted to the actual problems.

[0046] In related technologies, there are also some solutions that can remotely control storage equipment or storage robots to perform corresponding work tasks. For example, a communication module and a camera are installed on an intelligent forklift, and the camera captures on-site images and displays them on a computer. At the same time, a control interface is arranged on the computer, allowing the user to remotely send corresponding control instructions to the intelligent forklift through the control interface. Alternatively, for some storage robots, their travel videos can be customized on the remote control terminal so that they can travel according to manually set routes. However, the above solutions still have defects.

[0047] First, existing remote control solutions require the development of a specific control system and command synchronization system for each specific device. Therefore, whenever any device in the smart warehousing system is updated or a new device is added, a corresponding remote control system must be developed for that device. This approach, on the one hand, leads to increasing complexity during system updates, which in turn increases the probability of failure and the difficulty of troubleshooting when it does occur. On the other hand, if a new system needs to be developed for each device update, the development cost is high.

[0048] Second, the remote control solutions in related technologies are all based on surveillance video, rather than the first-person perspective of the device. Furthermore, video-only solutions often overlook many on-site details, leading to incorrect control or decision-making by the remote operator.

[0049] In order to solve the above-mentioned defects existing in the relevant technologies, an embodiment of the present application provides a control method for a warehousing system implemented based on a new warehousing system. The warehousing system is provided with a warehousing robot, a control terminal and a user terminal connected in sequence. Among them, the user terminal is configured to display on-site information and obtain user control operations, the control terminal is used to convert user operations into instructions executable by the warehousing robot, and the warehousing robot realizes the user's control intention based on the instructions. The control method of the warehousing system is configured such that the control terminal first receives the on-site image and status parameters, and then generates a virtual on-site picture based on the on-site image, on-site status parameters and the robot's identification mark, and sends the virtual on-site picture to the user terminal, and then generates a robot control instruction according to the user operation collected by the user terminal and the instruction conversion rules corresponding to the above identification mark, thereby realizing the purpose of controlling the warehousing robot to realize the artificial control intention through the robot control instruction.

[0050] In the above method, since the intelligent terminal can generate robot control instructions based on instruction conversion rules and user operations performed on the virtual scene image, this method can achieve control of any warehouse robot by simply updating the corresponding instruction conversion rules when a new warehouse robot is added or updated. This does not require the rearrangement and design of a new remote control system. At the same time, since the virtual scene image is generated based on the on-site image and on-site state parameters, the remote controller can obtain a more accurate understanding of the on-site situation based on it, and thus make more accurate remote control actions. In addition, this solution achieves the purpose of remotely controlling warehouse robots in specific circumstances, thereby enabling the warehouse system to make manual decisions and implement equipment control when faced with non-standardized problems. This improves the adaptability of the decision-making solution to actual problems to a certain extent.

[0051] For ease of understanding, the technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0052] Please refer to Figure 1, an embodiment of the present application provides a warehousing system, which includes a control end, a warehousing robot end and a user end. Among them, the user end includes virtual reality (VR), so that when the user end receives the virtual scene picture sent by the control end, it can realize it through the virtual reality device. The user end is also provided with a user operation acquisition device. Among them, the user operation acquisition device can be a handle, or a user behavior detection device. Among them, the user behavior detection device can determine the user's intention by detecting the user's body movements. Then, the user can control the control object in the virtual scene picture through body movements and perform corresponding actions in the virtual scene picture. Alternatively, the user operation acquisition device can also be other devices that can collect the user's operation intentions, so that the user can interact with the virtual objects in the virtual scene picture. This embodiment does not limit this.

[0053] Optionally, the control end includes a warehouse robot side communication module, a data processing module and a user-end communication module. Among them, the warehouse robot communication module is deployed with one or more communication methods. This allows the control terminal to connect with different robots based on user selection. It is understandable that the warehouse robot side communication module can establish communication with one or more machines at the same time. Moreover, the warehouse robots that establish communication with it at the same time can be robots of the same type or robots of different types. The data processing module is used to generate a virtual scene image based on the data received by the warehouse robot side communication module, and send it to the user end through the user-end communication module. It is also configured to generate control instructions based on user operations received by the user-end communication module, and send them to the warehouse robot through the warehouse robot side communication module.

[0054] It should be noted that in a remote control scenario, a warehouse robot entity can establish communication with a user-end entity, forming a warehouse robot-controller-user communication link. The control end can also connect to multiple warehouse robots via the warehouse robot-side communication module, and to multiple users via the user-end communication module. This allows for multiple warehouse robot-controller-user communication links to exist simultaneously. That is, in a remote control scenario, there is a one-to-one correspondence between warehouse robots and users. However, communication between the control end and warehouse robots, or between the control end and users, can be one-to-many.

[0055] Alternatively, the warehouse robot may be a warehouse device that can execute remote control instructions, such as a robotic arm, an autonomous mobile robot, an inventory drone, a cross-belt sorter, a shuttle, or an intelligent forklift.

[0056] Please refer to Figure 1In an optional embodiment, the control method of the warehouse system comprises the following steps:

[0057] The control method of the warehouse system provided by the embodiment can be executed on a control terminal of the warehouse system, which can be a background server or a distributed data processing node. In an embodiment, the control method of the warehouse system comprises steps S10 to S40.

[0058] S10: receiving on-site images collected by a visual sensor of the warehouse robot and on-site state parameters collected by a state sensor arranged on the warehouse robot and / or the warehouse site.

[0059] The warehouse system is provided with an automatic mode and a passive mode. The automatic mode and the passive mode can be triggered based on conditions or triggered by a user. When the passive mode is triggered, the control node first sends a data acquisition request to the warehouse robot corresponding to the triggered passive mode, i.e., the controlled object in the subsequent process, to acquire on-site images collected by a visual sensor of the warehouse robot, and sends a data acquisition request to a communication module corresponding to the state sensor to acquire on-site state parameters collected by a state sensor arranged on the warehouse robot and / or the warehouse site.

[0060] It should be noted that in some variant embodiments, the on-site images can also be collected based on other image sensors in the space where the controlled object is located, and the image collection sensor is not limited to being arranged on the warehouse robot. The scheme provided by the embodiment is described by taking the control of the autonomous mobile robot as an example, and therefore, in order to provide a better first visual experience, the on-site images collected by the warehouse robot based on the visual sensor are preferred, but this does not mean that the on-site images collected by the visual sensor arranged at other positions cannot be used. For example, for a cross sorting machine, only the channel of the goods flow needs to be determined in the control process, and therefore, the image collected by the camera arranged at a high position can have a better visual experience, and the data stream of the camera arranged outside the body can be set as needed. The main idea of the scheme given by the embodiment is to determine which data stream collected by the equipment is used as the on-site image collected by the warehouse robot based on the visual sensor through a pre-set image data stream correspondence, rather than limiting the specific deployment position of the visual sensor.

[0061] S20: generating a virtual on-site picture based on the on-site images, the on-site state parameters and the identification of the robot, and sending the virtual on-site picture to the user terminal;

[0062] After receiving the live image, the data can be pre-processed first, that is, for example, based on the camera intrinsic matrix, the barrel distortion of the wide-angle lens is eliminated. Then the depth map is converted into a three-dimensional point cloud (XYZ coordinates + RGB color), and is down-sampled to n cm accuracy using voxel filtering. Wherein n can be set to 3CM-15CM based on system computing power. Further, time synchronization of multiple sensors is realized through hardware timestamp (PTP protocol).

[0063] After completing the pre-processing of the data, modeling can be performed based on the environmental semantics, and dynamic objects can be tracked, so that subsequent spatial mapping and rendering can be performed according to the modeling and tracking results.

[0064] Exemplarily, after obtaining the live image and the live state parameter, the data collected by the multiple sensors can be synchronized through data preprocessing to realize time and space synchronization of the data. Then, based on the synchronized data stream, a live scene model is constructed. Then, in order to have a better visual experience, the pre-established model of the robot can also be obtained based on the identification of the robot, and the mapping of virtual and real spaces (robot is virtual and live scene is real) is performed to render a virtual live scene.

[0065] After generating the virtual live scene, the virtual live scene is updated based on the real-time collected live image and state parameter in the form of stream data, and is sent to the client in the form of data stream, so that the client based on the VR device can see the picture changing with the real change of the live scene in real time.

[0066] It should be noted that the live state parameter mentioned in the embodiment can be any other sensor parameter applied to enrich the virtual live scene. Herein, no specific limitation is made. For example, it can be a robot body state parameter: such as position, attitude, speed, mechanical arm joint angle, etc., which directly affects the motion and behavior of the virtual robot model. Environmental physical parameters: such as light, temperature and humidity, sound, which affect the visual and auditory performance of the virtual environment. Dynamic object parameters: such as the positions and speeds of other mobile devices (AGV, forklift), as well as the state of the goods (whether it is being transported or placed), which ensure that the dynamic elements in the virtual environment are synchronized with the real. Sensor data: such as obstacle information of distance sensor, contact feedback of force / torque sensor, which improves the realism and safety of interaction. Time synchronization parameter: to ensure that the timestamps of all data are consistent, to avoid delay or asynchronization of the virtual picture.

[0067] S30: receiving the user operation of the user end based on the virtual live response, and generating a robot control instruction based on the corresponding instruction conversion rule of the identification and the user operation;

[0068] After sending the virtual scene picture to the user terminal, the user terminal can detect the user's operation based on the user behavior detection device. And send the collected user operation to the control terminal. So that the control terminal can determine the task target according to the user operation, and then determine the executable action of the warehouse robot based on the identification mark, and obtain the instruction conversion rule, and decompose the task target according to the task target and the executable action, and generate at least one subtask, and finally in the instruction conversion rule, retrieve the robot control instruction corresponding to the subtask.

[0069] It should be noted that the above task target refers to the user inputting a control action based on the virtual scene picture through the user behavior detection device, so as to control the action executed by the robot in the virtual scene picture. Because in the generated virtual scene picture, based on the generated virtual scene picture, the user can control the interactive behavior, therefore after receiving the user operation, the control terminal can control the virtual model (i.e. the virtual model of the warehouse robot) to execute what action according to the user operation. And this action is not currently mapped to the actual scene, but only occurs in the virtual world. Therefore, the control terminal also needs to convert it into a robot control instruction executable by the warehouse robot, and send it to the robot for execution, so as to complete the action in the actual space.

[0070] Without realizing the behavior mapping of virtual and real space, the embodiment provides two optional implementation schemes. One is real-time control based on user operation behavior, and the other is segmented control based on user operation behavior. The two schemes can be selectively set in the implementation process based on different warehouse robots and application scenarios.

[0071] Firstly, in the present control scheme, real-time control data stream is generated according to the real-time control behavior of the user to control the warehouse robot and synchronize the execution of the corresponding action. For example, when the user controls the robot in the virtual scene picture to also lift the right arm, the control terminal detects the moving track and speed of the right arm. And based on the moving track and speed, the corresponding real-time control instruction is generated through the instruction conversion rule. Then the real-time control instruction is sent to the warehouse robot to control it to lift the right arm synchronously. Therefore, the task target here is to lift the right arm, and the executable action is the amplitude and angle of rotation of each joint. The generated robot control instruction is to control the corresponding motor of each joint to respond. Such a scheme has better real-time performance, but requires higher computing power of the system.

[0072] Secondly, in the segmented control scheme, the warehouse robot and the virtual robot in the virtual scene picture do not run synchronously. That is, according to the user operation and the virtual scene picture, the state change of at least one of the virtual goods, virtual devices and virtual equipment in the virtual scene picture is determined, and the task target is determined according to the state change.

[0073] Taking the equipment maintenance scenario as an example, if the state of the virtual device changes, the disassembly steps and / or assembly steps of the virtual device can be obtained first, and then the task target can be set as a maintenance task, and the operation steps corresponding to the maintenance task can be generated based on the disassembly steps and / or the assembly steps. Among them, the equipment to be repaired exists in the virtual scene in the form of a virtual model. It allows users to repair the virtual equipment to be repaired based on interactive operations. After the entire maintenance process or part of the maintenance process is completed. The control end records the steps in the maintenance process. Then the maintenance task is decomposed according to the steps. A series of robot control instructions are generated and sent to the execution object in the form of an instruction stream. The execution object can control itself to complete the maintenance process by executing the instructions in the instruction stream in sequence.

[0074] It should be noted that the virtual scene in a maintenance scenario can include at least two virtual models. One is a virtual model corresponding to the equipment being repaired. The on-site image used for this virtual scene can be collected in real time or in advance. This reduces the real-time nature of the operation, thereby increasing available computing time. This allows the system to reduce computing power requirements in scenarios where high real-time performance is not required.

[0075] Optionally, in an alarm scenario, a virtual device corresponding to a real alarm device can be set based on the virtual scene screen. This allows the user to trigger the virtual device in the virtual scene screen by interacting with the virtual scene screen. When the control terminal receives the action of triggering the virtual device, it sets the task target to update the trigger state of the alarm device corresponding to the virtual device. Then, the trigger instruction can be sent to the corresponding real alarm device to trigger an alarm in the alarm system through the alarm device. Of course, in some optional solutions, if the control terminal can directly communicate with the alarm system to trigger an alarm, then after the virtual device is triggered, the alarm can also be directly triggered in the alarm system. This embodiment does not limit this.

[0076] Optionally, in the cargo handling scenario, the user can first control the movement of virtual cargo in the virtual scene screen. The system can then trigger synchronization based on the completion of an action, or it can be set to trigger synchronization based on a preset synchronization duration. The preset synchronization time can be customized to 5s to 60s according to needs. For example, it can be set to 10s, 20s, 30s or 45s, etc. The control section can determine the starting position and ending position of the virtual cargo in the virtual scene screen when the preset synchronization duration is reached. Based on the starting position and ending position, the movement task of the transfer robot in the future time period is generated. After the instruction set is generated based on the movement task, it is sent to the transfer robot to control it to move from the starting position to the end position in the real scene.

[0077] Optionally, in this embodiment, before generating the virtual scene picture, the current mode of the user terminal can also be determined. Because in different modes, the necessary virtual objects in the virtual scene picture are different. The current mode can be used to filter out unnecessary virtual models in the virtual scene picture. That is, the operation object corresponding to the task target is determined based on the current mode, wherein the operation object includes at least one of goods, alarm devices and equipment to be repaired. Then, during the rendering process of the virtual scene picture, only the operation object is rendered as a virtual object. In this way, it is possible to perform the state change of at least one of the virtual goods, virtual devices and virtual equipment in the virtual scene picture based on the user operation and the virtual scene picture based on the operation object. At the same time, this can also effectively reduce the amount of calculation in the process of generating the virtual scene picture.

[0078] Optionally, the mode of the user terminal can be customized. For example, a mode selection interface can be first sent to the user terminal, wherein the mode selection interface includes at least one selectable mode among a shipping mode, an alarm mode, and a maintenance mode. Then, a mode selection result of the user terminal based on the mode selection interface is received, and the current mode is determined based on the mode selection result.

[0079] In other optional implementations, the user's permissions can be determined based on the user's login information, and the user's mode can be automatically selected based on the user's permissions. Alternatively, the user's mode can be automatically selected based on some other mode selection criteria. For example, image content recognition can be performed on the scene image. Then, the user's mode can be automatically selected based on the match between the image content recognition result and the mode. Of course, other solutions are also possible and will not be detailed here.

[0080] In this embodiment, the control method of the warehousing system is configured such that the control end first receives the on-site image and status parameters, and then generates a virtual on-site picture based on the on-site image, on-site status parameters and the identification identifier of the robot, and sends the virtual on-site picture to the user end, and then generates robot control instructions based on the user operations collected by the user end and the instruction conversion rules corresponding to the above identification identifier, thereby achieving the purpose of controlling the warehousing robot to realize the artificial control intention through the robot control instructions.

[0081] In the above method, since the intelligent terminal can generate robot control instructions based on instruction conversion rules and user operations performed on the virtual scene image, this method can achieve control of any warehouse robot by simply updating the corresponding instruction conversion rules when a new warehouse robot is added or updated. This does not require the rearrangement and design of a new remote control system. At the same time, since the virtual scene image is generated based on the on-site image and on-site state parameters, the remote controller can obtain a more accurate understanding of the on-site situation based on it, and thus make more accurate remote control actions. In addition, this solution achieves the purpose of remotely controlling warehouse robots in specific circumstances, thereby enabling the warehouse system to make manual decisions and implement equipment control when faced with non-standardized problems. This improves the adaptability of the decision-making solution to actual problems to a certain extent.

[0082] The present application provides a control device for a warehousing system, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the control method for the warehousing system in the above-mentioned embodiment one.

[0083] Reference below Figure 3 , which shows a schematic diagram of the structure of a control device for a warehousing system suitable for implementing an embodiment of the present application. The control device for the warehousing system in the embodiment of the present application may include, but is not limited to, a server and a computer. Figure 3 The control device of the warehousing system shown is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present application.

[0084] like Figure 3As shown, the control device of the warehouse system can include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. Various programs and data required for the operation of the control device of the warehouse system are also stored in the random access memory 1004. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the input / output interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the control device of the warehouse system to communicate with other devices wirelessly or by wire to exchange data. Although the control device of the warehouse system with various systems is shown in the figure, it should be understood that all the systems shown are not required to be implemented or possessed. More or fewer systems can be alternatively implemented or possessed.

[0085] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are performed.

[0086] The control device of the warehouse system provided in the present application adopts the warehouse system control method in the above-mentioned embodiments, and solves the technical problem that the solutions provided by the warehouse system do not fit the actual problems. Compared with the related art, the control device of the warehouse system provided in the present application has the same beneficial effects as the warehouse system control method provided in the above-mentioned embodiments, and other technical features in the control device of the warehouse system are the same as the features disclosed in the above-mentioned method, which will not be described here.

[0087] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0088] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

[0089] Please refer to Figure 4 The present application provides a warehousing system, wherein the warehousing system 100 includes:

[0090] A receiving module 110 receives a scene image collected by the storage robot based on a visual sensor, and a scene status parameter collected by a status sensor arranged on the storage robot and / or the storage site;

[0091] The data processing module 120 generates a virtual scene image based on the scene image, the scene state parameters, and the identification identifier of the robot, and sends the virtual scene image to the user terminal; receives user operations in response to the virtual scene image from the user terminal, and generates robot control instructions based on the instruction conversion rules corresponding to the identification identifier and the user operations;

[0092] The sending module 130 sends the control instruction to the storage robot to control the storage robot to perform corresponding actions.

[0093] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, and the computer-readable program instructions are used to execute the control method of the warehousing system in the above-mentioned embodiment.

[0094] The computer-readable storage medium provided herein may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including, but not limited to, wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0095] The computer-readable storage medium may be included in the control device of the warehousing system; or it may exist independently without being assembled into the control device of the warehousing system.

[0096] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the control device of the warehousing system, the control device of the warehousing system can improve the warehousing efficiency of the warehouse based on the above method.

[0097] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0098] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0099] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0100] The computer-readable storage medium provided in this application stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned warehousing system control method. This computer-readable storage medium can address the technical issue of mismatch between the warehousing system's proposed solution and the actual problem. Compared to related technologies, the beneficial effects of the computer-readable storage medium provided in this application are similar to those of the warehousing system control method provided in the aforementioned embodiments, and are not further elaborated here.

[0101] An embodiment of the present application provides a computer program product, including a computer program, which implements the steps of the control method of the warehousing system as described above when the computer program is executed by a processor.

[0102] The computer program product provided in this application can solve the technical problem that the solution provided by the warehousing system is not compatible with the actual problem. Compared with the related art, the beneficial effects of the computer program product provided in the embodiment of this application are the same as the beneficial effects of the control method of the warehousing system provided in the above embodiment, and will not be repeated here.

[0103] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

[0104] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system comprising a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or system comprising the element. In the intervals given in this application, all include boundary values ​​unless explicitly defined.

[0105] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0106] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A control method for a warehousing system, characterized in that: Applied to a control terminal, one side of the control terminal is connected to the storage robot for communication, and the other side of the control terminal is connected to the user terminal for communication. The control method of the storage system includes: Receive the scene image collected by the storage robot based on the visual sensor, and the scene status parameters collected by the status sensors arranged on the storage robot and the storage site; After completing the spatiotemporal synchronization of the data, generating a virtual scene image based on the scene image, the scene state parameters and the identification mark of the robot, and sending the virtual scene image to the user terminal; receiving a user operation of the user terminal based on the virtual scene image response; Determining a task objective based on the user operation; Determining executable actions of the storage robot based on the identification identifier, and obtaining instruction conversion rules; Decomposing the task objective according to the task objective and the executable actions to generate at least one subtask; Retrieving the robot control instruction corresponding to the subtask in the instruction conversion rule; Sending the control instruction to the storage robot to control the storage robot to perform a corresponding action; wherein determining the task target according to the user operation includes: Determining a state change of a virtual commodity in the virtual scene according to the user operation and the virtual scene; If the virtual cargo position status is updated in the virtual scene image, it is determined that the task is to move the cargo; Determining the initial location and the final location of the virtual goods; The task objective is set to move the goods in the warehouse from the initial position to the end position.

2. The control method of the warehousing system according to claim 1, characterized in that: The step of determining the task target according to the user operation further includes: determining, based on the user operation and the virtual scene image, a state change of at least one of a virtual device and a virtual equipment in the virtual scene image; The task objective is determined according to the state change.

3. The control method of the storage system according to claim 2, characterized in that: The step of determining the task target according to the state change includes: If the triggering state of the virtual device is updated, the task objective is set to update the triggering state of the alarm device corresponding to the virtual device.

4. The control method of the storage system according to claim 2, characterized in that: The step of determining the task target according to the state change includes: If the state of the virtual device changes, obtaining disassembly steps and / or assembly steps of the virtual device; The task target is set as a maintenance task, and operation steps corresponding to the maintenance task are generated based on the disassembly step and / or the assembly step.

5. The control method of the storage system according to claim 2, characterized in that: The step of determining a state change of at least one of virtual goods, virtual devices, and virtual equipment in the virtual scene according to the user operation and the virtual scene includes: Determine an operation object corresponding to the task goal based on the current mode, wherein the operation object includes at least one of cargo, an alarm device, and equipment to be repaired; Based on the operation object, executing the step of determining a state change of the virtual goods in the virtual scene according to the user operation and the virtual scene; Alternatively, the step of determining a state change of at least one of a virtual device and a virtual equipment in the virtual scene according to the user operation and the virtual scene is performed based on the operation object.

6. The control method of the storage system according to claim 5, characterized in that: Before the step of determining the operation object corresponding to the task goal based on the current mode, the method further includes: Send a mode selection interface to the user terminal, wherein the mode selection interface includes at least one optional mode among a shipping mode, an alarm mode and a maintenance mode; receive a mode selection result of the user terminal based on the response of the mode selection interface, and determine the current mode according to the mode selection result.

7. A storage system, characterized in that: The warehousing system is used to implement the control method of the warehousing system according to any one of claims 1 to 6, and the warehousing system includes: A receiving module is used to receive on-site images collected by the storage robot based on the visual sensor, and on-site status parameters collected by status sensors arranged on the storage robot and / or the storage site; A data processing module is configured to generate a virtual scene image based on the scene image, the scene state parameters, and the identification identifier of the robot, and transmit the virtual scene image to a user terminal; receive user operations in response to the virtual scene image from the user terminal; determine a task objective based on the user operation; determine executable actions of the storage robot based on the identification identifier, and obtain the instruction conversion rule; decompose the task objective based on the task objective and the executable actions to generate at least one subtask; and retrieve the robot control instruction corresponding to the subtask from the instruction conversion rule; A sending module is used to send the control instruction to the storage robot to control the storage robot to perform the corresponding action; wherein, determining the task target according to the user operation includes: Based on the user operation and the virtual scene screen, a state change of the virtual goods in the virtual scene screen is determined; if the position state of the virtual goods in the virtual scene screen is updated, the task is determined to be moving the goods; the initial position and the end position of the virtual goods are determined; and the task goal is set to move the goods in the warehouse from the initial position to the end position.

8. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the control method of the warehousing system according to any one of claims 1 to 6 are implemented.

Images