Workpiece carrying method and device based on robot, electronic equipment and medium

Abstract

The invention relates to the technical field of industrial robots, in particular to a robot-based workpiece carrying method and device, electronic equipment and a medium, and is used for solving the problem that a robot does not have autonomy in a traditional scene. Obtaining the workpiece remaining amount of the reference workpiece of each station in the assembly line in the target place; determining a to-be-carried target workpiece based on the workpiece surplus of the reference workpiece of each station; obtaining a first position of a station where the target workpiece is located and a second position of a warehouse where the target workpiece is located from the workpiece attribute data of the target workpiece; path planning is conducted according to the current equipment position, the first position and the second position of the robot, and a carrying path is obtained; executing a carrying task for the target workpiece according to the carrying path; therefore, autonomous workpiece carrying of the robot is achieved, and the intelligent degree of the robot is improved.

Description

Technical Field

[0001] The present application relates to the technical field of industrial robots, and in particular to a robot-based workpiece handling method, device, electronic equipment and medium. Background Art

[0002] In traditional industrial scenarios, robots are usually used to replace humans to perform some work, such as using handling robots to replace humans to carry workpieces.

[0003] However, currently robots usually work according to predetermined tasks, such as carrying workpieces from a warehouse to a designated workstation according to a predetermined handling task, and the handling task is usually formulated by humans observing the use of workpieces at the designated workstation. It can be seen that currently robots do not have autonomy and have a low level of intelligence. Summary of the invention

[0004] The embodiments of the present application provide a robot-based workpiece handling method, device, electronic equipment and medium to improve the robot's autonomy and the robot's intelligence level.

[0005] The specific technical solutions provided by the embodiments of this application are as follows:

[0006] In a first aspect, an embodiment of the present application provides a robot-based workpiece handling method, comprising:

[0007] Based on the attribute data of each workpiece in the semantic map or knowledge graph of the target place, the remaining amount of the reference workpiece in each workstation in the assembly line in the target place is obtained, wherein the semantic map or the knowledge graph identifies the attributes of the workpieces in the target place and is constructed based on the attributes of each identified workpiece, including multiple workpiece attribute data, each workpiece attribute data including the name of the workpiece and the remaining amount of the workpiece;

[0008] Determine a target workpiece to be transported based on the remaining amount of the reference workpieces at each workstation, and obtain a first position of the workstation where the target workpiece is located and a second position of the warehouse where the target workpiece is located from workpiece attribute data of the target workpiece;

[0009] Path planning is performed according to the current device position of the robot, the first position and the second position to obtain a transport path; and a transport task for the target workpiece is performed according to the transport path.

[0010] In a possible implementation, the workpiece attribute data of any reference workpiece further includes a workpiece consumption rate, and the determining of the target workpiece to be transported based on the remaining amount of the reference workpieces at each workstation includes:

[0011] respectively comparing the workpiece remaining quantity of the reference workpiece at each workstation with a remaining quantity threshold;

[0012] Determine whether the number of reference workpieces whose remaining quantity is less than the remaining quantity threshold is greater than a preset value;

[0013] If yes, then estimating the remaining usage time of each reference workpiece based on the remaining amount of each reference workpiece and the workpiece consumption rate;

[0014] According to the remaining usage time of each reference workpiece, a preset number of reference workpieces are selected from the reference workpieces as the target workpieces.

[0015] In a possible implementation, performing path planning based on the current device position of the robot, the first position, and the second position to obtain a transport path includes:

[0016] Determine a constraint duration based on the preset value and the remaining usage duration of the target workpiece;

[0017] Performing path planning based on the constraint duration, the current device position, the preset number of second positions of the warehouse where the target workpieces are located, and the preset number of first positions of the workstations where the target workpieces are located;

[0018] When a path is planned, the planned path is determined as the transport path;

[0019] When a path is not planned, the following operations are repeatedly performed until a path is planned: selecting an alternative workpiece from each of the target workpieces of the currently planned path, and performing path planning based on the constraint duration, the current device position, and the first position and the second position of each of the target workpieces except the alternative workpiece, wherein the alternative workpiece is a target workpiece whose remaining usage duration meets a preset condition among the target workpieces.

[0020] In a possible implementation, the method further includes:

[0021] If it is determined that the number of each of the target workpieces in the currently planned path is less than or equal to 2, then outputting workpiece handling assistance information to indicate that there is currently a workpiece handling task that requires assistance;

[0022] Based on the workpiece handling assistance information, an assisting robot is determined from other robots in the target location, and the workpiece handling is completed in collaboration with the assisting robot.

[0023] In a possible implementation, the workpiece attribute data of any reference workpiece further includes update time, and the method further includes:

[0024] Based on the update time, the current time, and the workpiece consumption rate, updating the remaining amount of the workpiece in the workpiece attribute data of any one of the reference workpieces;

[0025] The method further comprises:

[0026] If it is determined that the remaining amount of the reference workpiece at each workstation is greater than or equal to the remaining amount threshold, acquiring a first image of the workpiece at each workstation in the assembly line area in the target location;

[0027] Respectively identifying the attributes of the artifacts in each first image, and updating the artifact attribute data of the corresponding artifact in the semantic map or knowledge graph based on the identification results;

[0028] The method further comprises:

[0029] During the execution of the transport task, a second image of the workpiece at each workstation and / or each shelf on the transport path is acquired;

[0030] The attributes of the artifacts in each second image are identified respectively, and the artifact attribute data of the corresponding artifacts in the semantic map or the knowledge graph are updated based on the identification results.

[0031] In a possible implementation, the semantic map of the target place is constructed in the following manner:

[0032] Acquire a workpiece scene image and a first position of each workstation in the assembly line in the assembly line area in the target location, and based on a first prompt instruction in the assembly line area, respectively identify the attributes of the workpiece in each acquired workpiece scene image through a preset language model to obtain the name, remaining quantity and consumption rate of each workpiece in the assembly line;

[0033] Acquire a workpiece site image and a second position of each workpiece on a shelf in a warehouse area in the target location, and based on a second prompt instruction in the warehouse area, identify the attributes of each workpiece in the acquired workpiece site image by using the language model to obtain a name of each workpiece on the shelf and a remaining amount of the workpiece;

[0034] The first position, name, remaining quantity and consumption rate of each workpiece in the assembly line, and the second position, name and remaining quantity of each workpiece on the shelf are respectively associated with the map data of the planar map to construct a semantic map of the target location.

[0035] In a possible implementation, the method further includes:

[0036] Constructing entity types and entity relationship types of the knowledge graph of the target location, wherein the entity types include workpieces, containers, warehouses, shelves, and assembly lines, and the relationship types include assembly line workpiece-container, container-location, shelf-assembly line, warehouse workpiece-location, and workpiece-workpiece attribute data;

[0037] According to the entity type and the entity relationship type, based on the identification, name, remaining quantity, workpiece consumption rate and first position of each workpiece in the assembly line area, and the name, remaining quantity and second position of each workpiece in the warehouse area, a knowledge graph of the target place is constructed.

[0038] In a second aspect, an embodiment of the present application provides a robot-based workpiece handling device, comprising:

[0039] An acquisition module is used to obtain the remaining quantity of the reference workpieces at each station in the assembly line in the target place based on the attribute data of each workpiece in the semantic map or knowledge graph of the target place, wherein the semantic map or the knowledge graph identifies the attributes of the workpieces in the target place and is constructed based on the attributes of each identified workpiece, including multiple workpiece attribute data, each workpiece attribute data including the name of the workpiece and the remaining quantity of the workpiece;

[0040] A decision module, configured to determine a target workpiece to be transported based on the remaining amount of the reference workpieces at each workstation, and obtain a first position of the workstation where the target workpiece is located and a second position of the warehouse where the target workpiece is located from workpiece attribute data of the target workpiece;

[0041] A path planning and execution module is used to perform path planning according to the current device position of the robot, the first position and the second position to obtain a transport path; and execute a transport task for the target workpiece according to the transport path.

[0042] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0043] Memory, used to store computer programs or instructions;

[0044] A processor is used to execute the computer program or instructions in the memory so that any method in the first aspect described above is executed.

[0045] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium. When instructions in the storage medium are executed by a processor, the processor is enabled to execute any one of the methods described in the first aspect above.

[0046] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes: a computer program code, when the computer program code is run on a computer, enables the computer to execute the method as described in any one of the first aspects.

[0047] Compared with the prior art, the beneficial effects of the present application are at least as follows:

[0048] By identifying the attributes of the workpieces in the target place and constructing a semantic map or knowledge graph of the target place based on the attributes of the identified workpieces, the remaining amount of the reference workpieces in each workstation can be obtained based on the attribute data of each workpiece in the semantic map or knowledge graph of the target place, that is, the real-time workpiece usage of each workstation can be obtained. Then, the target workpiece to be transported can be determined based on the remaining amount of workpieces, and based on the first position of the workstation where the target workpiece is located and the second position of the warehouse where the target workpiece is located, a multi-objective path planning algorithm is called for path planning, and a transportation path for transporting the target workpiece can be obtained, and the transportation task for the target workpiece is performed according to the transportation path to transport the target workpiece from the second position to the first position, thereby realizing autonomous workpiece transportation based on the workpiece attribute data in the semantic map or knowledge graph of the target place, improving the degree of intelligence, and thus improving the operation efficiency of the factory. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a schematic diagram of an application scenario of an optional robot-based workpiece handling method in an embodiment of the present application;

[0050] Figure 2 A schematic diagram of a flow chart of a method for constructing a semantic map of a target location in an embodiment of the present application;

[0051] Figure 3 This is a schematic diagram of a shelf structure in a warehouse in an embodiment of the present application;

[0052] Figure 4 A schematic diagram of an inspection route of a warehouse in an embodiment of the present application;

[0053] Figure 5 This is a schematic diagram of workpiece attribute data of a shelf in an embodiment of the present application;

[0054] Figure 6 A schematic diagram of the architecture of a knowledge graph ontology model of a target location in an embodiment of the present application;

[0055] Figure 7 A schematic diagram of a knowledge graph of a target location in an embodiment of the present application;

[0056] Figure 8A schematic diagram of a process flow of a robot-based workpiece handling method in an embodiment of the present application;

[0057] Fig. 9 A schematic diagram of a path planning process in an embodiment of the present application;

[0058] Fig.10 A schematic diagram of the logical architecture of a robot-based workpiece handling device in an embodiment of the present application;

[0059] Fig.11 It is a schematic diagram of the physical structure of the electronic device in the embodiment of the present application. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0061] It should be noted that the terms "first", "second", "third", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0062] Below, some terms in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0063] (1) The term "robot" in the embodiments of the present application refers to the use of sensor technology to obtain information about the surrounding environment, such as various parameters such as the position of objects, light, sound, temperature, pressure, etc., and then to process and analyze the acquired information to control the actuator to move and realize the autonomous movement of the robot. In a specific implementation, after the robot obtains a path, the path can be sent to the robot after planning by the background server, or it can be independently formulated by the robot. The robot's control system generates corresponding control instructions according to the path, and sends the control instructions to the actuator. The actuator drives the motor based on the control instructions, thereby realizing the movement and action of the robot under the action of the motor.

[0064] (2) The term "multi-objective path planning algorithm" in the embodiments of the present application is a mathematical method for solving multi-objective decision-making problems. It occupies an important position in operations research, especially in solving multi-objective optimization problems. The specific application is to effectively solve the multi-objective path planning problem through a heuristic search process and a local replanning process in a dynamic uncertain environment. In the present application, when performing multi-objective path planning, the minimum remaining use time of each target workpiece to be planned can be used as the constraint time. Based on the first position (workstation) and the second position (warehouse) of each target workpiece, a path is first generated, and then a path is generated. The time consumption of the two paths is compared, and the path with shorter time consumption is selected to keep, and then a path is generated. The same method as above is used for processing, and so on. All the generated paths are traversed, and a path with the shortest time consumption is selected from them, that is, the transportation path is obtained.

[0065] The following is a brief introduction to the design concept of the embodiments of the present application.

[0066] The present application relates to the field of industrial robot technology, and mainly to a robot-based workpiece handling method, device, equipment and medium. The above schemes of some embodiments of the present application can be applied to various industrial scenarios, in which a certain number of industrial equipment, industrial data, industrial processes (assembly lines), etc. are involved. For example, chemical industry scenarios, various automated factory scenarios, etc.

[0067] In traditional industrial scenarios, robots are usually used to replace human workers. At present, robots usually work according to predetermined tasks, such as carrying workpieces from a warehouse to a designated workstation according to a predetermined handling task. The predetermined task needs to be formulated by humans after observing the use of workpieces at the designated workstation, and then the execution instructions of the predetermined task are issued to the robot manually, so that the robot can complete the predetermined task. It can be seen that in traditional industrial scenarios, robots are not autonomous and have a low level of intelligence.

[0068] In view of this, in order to solve the problem that robots do not have autonomy in traditional scenarios, an embodiment of the present application provides a robot-based workpiece handling method. In the embodiment of the present application, based on the attribute data of each workpiece in the semantic map or knowledge graph of the target place, the remaining amount of the reference workpiece at each workstation in the assembly line in the target place is obtained, wherein the semantic map or knowledge graph identifies the attributes of the workpieces in the target place and is constructed based on the attributes of each identified workpiece, including multiple workpiece attribute data, each workpiece attribute data including the name of the workpiece and the remaining amount of the workpiece; based on the remaining amount of the reference workpiece at each workstation, the target workpiece to be handled is determined; and from the workpiece attribute data of the target workpiece, the first position of the workstation where the target workpiece is located and the target workstation's the second position of the warehouse where the target workpiece is located; path planning is performed according to the current equipment position, the first position and the second position of the robot to obtain a transport path; the transport task for the target workpiece is executed according to the transport path; in this way, the remaining amount of the reference workpiece at each workstation can be obtained based on the attribute data of each workpiece in the semantic map or knowledge graph of the target place, so as to obtain the real-time workpiece usage of each workstation, so as to determine the target workpiece to be transported based on the remaining amount of the workpiece, perform path planning to obtain the transport path, and execute the transport task for the target workpiece according to the transport path, so as to realize the autonomous workpiece transport of the robot, improve the intelligence level of the robot, so that the robot can perform appropriate operations at appropriate times, thereby improving the operation efficiency of the factory.

[0069] A robot-based workpiece handling method provided in some embodiments of the present application is applicable to electronic devices.

[0070] The electronic device may be a background server of the robot. The server obtains the remaining amount of the reference workpieces at each station in the assembly line in the target location based on the attribute data of each workpiece in the semantic map or knowledge graph of the target location, and determines the target workpiece to be transported based on the remaining amount of the reference workpieces at each station; obtains the first position of the station where the target workpiece is located and the second position of the warehouse where the target workpiece is located from the workpiece attribute data of the target workpiece; thereby realizing path planning according to the current device position, the first position and the second position of the robot, obtaining the transport path, and sending the transport path to the robot, controlling the robot to move according to the transport path, and transporting the target workpiece from the second position to the first position.

[0071] For example, the server may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. In some embodiments, the server and the robot may be connected by wire or wirelessly.

[0072] In some other embodiments of the present application, the electronic device can also be applied to the robot, which can be a mobile robot, a humanoid robot, etc. Specifically, the robot obtains the remaining amount of the reference workpieces at each station in the assembly line in the target place based on the attribute data of each workpiece in the semantic map or knowledge graph of the target place, and determines the target workpiece to be transported based on the remaining amount of the reference workpieces at each station; then, from the workpiece attribute data of the target workpiece, the first position of the station where the target workpiece is located and the second position of the warehouse where the target workpiece is located are obtained; thereby, path planning is performed according to the current device position, the first position and the second position of the robot, and the transport path is obtained and moved according to the transport path to transport the target workpiece from the second position to the first position.

[0073] The following is a schematic diagram of an application scenario of an optional robot-based workpiece handling method in an embodiment of the present application, in conjunction with the accompanying drawings. Figure 1 .like Figure 1 As shown, the application scenario includes a robot 10 and a background for controlling the robot 10, such as a server 20. The server 20 utilizes a robot-based workpiece handling method provided in an embodiment of the present application to realize the ability to perform path planning based on the real-time workpiece usage of the workstations on the assembly line in the target location. When it is determined that there is a target workpiece to be transported, the robot is formulated and controlled to execute a specific handling plan, so that the robot can perform appropriate operations at appropriate times, thereby improving the robot's autonomy, improving the robot's intelligence, and thereby improving the factory's operating efficiency.

[0074] Of course, the method provided in the embodiment of the present application is not limited to Figure 1 The application scenario shown can also be used in other possible application scenarios, which are not limited in the embodiments of the present application.

[0075] After introducing the application scenarios of the embodiments of the present application, the implementation methods of the present application are further described in detail below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments of the present application and the features in the embodiments may be combined with each other if there is no conflict.

[0076] First, a brief overview of constructing a semantic map of a target place is given.

[0077] In some embodiments of the present application, a stereoscopic map (such as 3D) of the target location is imported into the robot or the robot's backend server, and then the imported stereoscopic map is projected into a flat map. For example, the height in the 3D map is directly set to 0, thereby obtaining a flat map of the imported stereoscopic map. Then, a preset language model is used to perform area recognition on the flat map to identify the assembly line area and warehouse area in the target location, such as Figure 1 As shown, the assembly line area includes assembly lines 1 to 5, and the warehouse area includes warehouse 1 and warehouse 2.

[0078] In some embodiments of the present application, the language model may be a vision-language model (VLM), such as a VLM large model. Generally, large models are more versatile, but have slower reasoning speeds, and may not perform specific tasks accurately due to insufficient training data in specific environments. In order to allow the VLM large model to quickly and accurately perform tasks such as identifying the remaining amount of workpieces in the container of the assembly line and identifying the warehouse and assembly line in the map, this can be achieved by fine-tuning the pre-trained large model.

[0079] Specifically, a training data set is collected, wherein the data set contains a variety of scene data, a variety of categories of workpiece images, and each workpiece image is annotated with the remaining percentage of the workpiece, for example, from 100% to 0, divided into 10 groups. Images of maps of target locations (such as factories) are collected, and the assembly lines and warehouses in the images, as well as the shelves in the warehouses, are annotated. After the above training data is collected, the parameters of the VLM large model are fine-tuned using the above images, such as Lora fine-tuning, so that the fine-tuned large model can identify the assembly line area and warehouse area in the map, such as identifying which part of the map is the assembly line, which part is the warehouse and the shelves in the warehouse, etc., as well as monitoring the usage of workpieces in the assembly line and warehouse, and updating the data in the remaining quantity label of the workpiece in the semantic map.

[0080] Following the above content, after using the language model to identify the assembly line area and warehouse area in the target place, refer to Figure 2 As shown, the semantic map of the target place can be constructed through the following process:

[0081] Step 200: Acquire the workpiece site image and the first position of each workstation in the assembly line within the assembly line area in the target location, and based on the first prompt instruction of the assembly line area, identify the attributes of the workpiece in each acquired workpiece site image through a preset language model to obtain the name of each workpiece in the assembly line, the remaining amount of the workpiece and the workpiece consumption rate.

[0082] Usually, each workstation on the assembly line has a corresponding container for holding workpieces. Therefore, in some implementations of the present application, when executing step 200, the following operations may be performed specifically for the assembly line:

[0083] Operation 1: The location of the container containing the workpieces at each workstation in the assembly line is used as the destination of the robot, and a multi-objective path planning algorithm is called to determine the inspection path of the robot for inspecting the assembly line.

[0084] In some embodiments of the present application, the inspection path includes the inspection order of the robot inspecting each workstation in the assembly line, so that the robot can move along the inspection path to smoothly and reasonably obtain the workpiece types (such as names) and usage (such as the remaining amount of workpieces) of different workstations on the assembly line, preparing for the subsequent establishment and maintenance of the semantic map.

[0085] Specifically, when performing path planning, you can first call the path planning algorithm to plan a path from the starting point (such as the robot's starting point, current position, etc.) to each target (such as the location of the container holding the workpiece at each workstation in the assembly line), and find the shortest point from the starting point to one of the paths in the planned path as the target node, set the target node as the current node, and then plan the paths of the remaining targets from the current node, that is, based on the shortest distance from the current node to another target that has not been planned by the path, and so on, find the next target node, and so on, to determine the robot's action route for inspecting the assembly line, such as the aforementioned inspection path.

[0086] In a specific implementation, when determining the inspection path, the robot's inspection path can be planned according to the order of the processes in the assembly line, that is, the location of the first process in the assembly line is used as the first destination in the inspection path, and so on, the location of the last process is used as the last destination. In the case where the same process contains multiple workstations, the shortest route can be used as the inspection path of the process according to the specific location of the workstation.

[0087] Operation 2: Control the robot to move along the inspection path, obtain the on-site image of the container containing the workpiece at each station through the robot, and determine the position of the container containing the workpiece at each station in the assembly line, that is, the first position.

[0088] In some embodiments of the present application, first, the robot is guided to approach the assembly line, and then the robot is controlled to move along the inspection path. During the movement of the robot along the inspection path, the image acquisition device of the robot captures the workpiece scene image of the container containing the workpiece at each station. After obtaining the workpiece scene image, based on the first prompt instruction of the assembly line area, the language model, such as the aforementioned VLM large model, is guided to recognize the attributes of the workpiece in each scene image, thereby identifying the name of each workpiece in the assembly line and the remaining amount of the workpiece.

[0089] The first prompt instruction may be pre-built for the VLM large model. Optionally, the prompt word corresponding to the first prompt instruction may be "You are a production line manager. Please tell me what workpieces are in the workpiece box (container)? How many workpieces are left in the box? Please return the name of the workpiece and the remaining quantity of the workpiece, and do not return the rest of the information." In this way, the name of the workpiece contained in the container of each station in the production line and the remaining quantity of the workpiece can be obtained.

[0090] The workpiece consumption rate is the speed at which the workpieces in the container on the assembly line are used. This parameter is a very important parameter. In some embodiments of the present application, the workpieces that consume 10% of the container can be regarded as a node, and the speed at which the workpieces are used in the container of each station is calculated, and the time taken to consume 10% of the workpieces is recorded to obtain the workpiece consumption rate of each station. In some embodiments of the present application, after the robot collects the on-site image of the workpiece in the container of a certain station, it can wait for a period of time, and then collect images of the container at the station again to identify the remaining amount of workpieces, thereby calculating the workpiece consumption rate of the workpieces in the container. The workpiece consumption rate can be used to estimate the time when the remaining workpieces can be used.

[0091] When determining the position of the container containing the workpieces at each workstation in the assembly line, a Simultaneous Localization and Mapping (SLAM) algorithm can be used to estimate the plane position of the photographed container containing the workpieces, thereby obtaining the plane coordinates of the center position of the container, denoted as (x, y), i.e., the aforementioned first position. In this way, the positioning of the container containing the workpieces at each workstation in the assembly line is achieved.

[0092] In some embodiments, each of the above data may also include an update time to record the data acquisition time of the remaining quantity of the workpiece of the workpiece, and clarify the timeliness information of the data; the update time can also be used to subsequently update the remaining quantity of the workpiece in real time based on the update time, current time and workpiece consumption rate.

[0093] Step 210: Obtain the workpiece on-site image and the second position of each workpiece on the shelf in the warehouse area of ​​the target location, and based on the second prompt instruction of the warehouse area, identify the attributes of the workpiece in each acquired workpiece on-site image through the language model to obtain the name of each workpiece on the shelf and the remaining quantity of the workpiece.

[0094] Shelves are usually used to place workpieces in the warehouse area, and each shelf can be divided into multiple layers. Figure 3 As shown, taking warehouse 1 as an example, warehouse 1 contains two shelves, which are respectively recorded as shelf 1 and shelf 2; each shelf can be divided into two layers, and multiple workpieces are placed on each layer, such as Figure 3 The bottom layer of the shelf 1 shown has 8 workpieces placed thereon, and the upper layer also has 8 workpieces placed thereon. However, it should be noted that the 8 workpieces mentioned here may be workpieces of the same kind / type or workpieces of different kinds / types.

[0095] In some implementations of the present application, when executing step 210, the following operations may be performed specifically for the shelf:

[0096] Operation 1: Set the corner points of the shelves in the warehouse (i.e., each corner of the shelves) as the destinations of the robot, call the multi-objective path planning algorithm, and determine the inspection path for the robot to inspect the shelves.

[0097] In some embodiments of the present application, Figure 4 As shown in the figure, assuming that the warehouse contains two shelves, which are respectively denoted as shelf 1 and shelf 2. Shelf 1 is located in the lower half of the warehouse, and shelf 2 is located in the upper half of the warehouse. When formulating the inspection path, the lower left corner of shelf 1 can be used as the first destination of the robot, the lower right corner of shelf 1 can be used as the second destination of the robot, the upper right corner of shelf 1 can be used as the third destination of the robot, and the upper left corner of shelf 1 can be used as the fourth destination of the robot. Thus, the inspection path of the robot inspecting shelf 1 can be determined as: from the lower left corner - the lower right corner - the upper right corner - the upper left corner. In this way, it can be ensured that the inspection process of shelf 1 can go around shelf 1, thereby completely generating the semantic map of shelf 1.

[0098] Similarly, see Figure 4 As shown, when inspecting shelf 2, the same method as inspecting shelf 1 can also be used, and the inspection path of the robot inspecting shelf 2 is determined to be lower left corner-lower right corner-upper right corner-upper left corner.

[0099] See also Figure 4 As shown, for inspecting shelf 1 and shelf 2, the following can be formulated based on the locations of shelf 1 and shelf 2 and the entrance location of the warehouse: Figure 4In other embodiments, if there are multiple shelves or multiple warehouses, then when the inspection path is specifically executed, a multi-objective path planning algorithm can be called based on the entrance location of each warehouse and the location of each shelf to determine the multi-objective path planning algorithm, which aims to obtain an optimal path and a shortest path that can pass through the above-mentioned warehouses and shelves.

[0100] Based on the principle of the multi-objective path planning algorithm, first, based on the location of each warehouse and each shelf, a path is generated, recorded as path A, and then a second path is generated, recorded as path B. The time consumption of path A and path B is compared, and the path with shorter time consumption is retained. Then a third path is generated and compared with the time consumption of the remaining path, and then the path with shorter time consumption is retained, and so on. All the generated paths are traversed, and the path with the shortest time consumption is recorded as the final inspection path, among which the total number of generated paths can be set in advance.

[0101] Operation 2: Control the robot to move along the inspection path, obtain the workpiece on-site image of each workpiece placed on the shelf through the robot, and determine the position of each workpiece on the shelf, that is, the second position.

[0102] In some embodiments of the present application, first, the robot is guided to enter the warehouse, and then the robot is controlled to move along the inspection path. During the movement of the robot along the inspection path, the robot can be controlled to take pictures of the shelves starting from the bottom layer of the shelves in sequence in combination with the 3D map of the target location, so as to obtain the workpiece site image of each workpiece on the entire shelf. After obtaining the workpiece site image, based on the second prompt instruction of the warehouse area, the language model, such as the aforementioned VLM large model, is guided to identify the attributes of the workpiece in each site image, so as to identify the name of each workpiece on the shelf and the remaining amount of the workpiece.

[0103] The second prompt instruction may be pre-built with the VLM large model. Optionally, the prompt word corresponding to the second prompt instruction may be "You are a warehouse manager. Please combine the label on the shelf and the workpiece placed on the shelf to find out what the remaining quantity of the workpiece is. Please return the name of the workpiece and the remaining quantity of the workpiece. Do not return the rest." In this way, the name and remaining quantity of each workpiece placed on the shelf can be obtained.

[0104] When determining the location of each workpiece on the shelf, the SLAM algorithm may be used to estimate the plane position of the photographed workpiece, thereby obtaining the plane coordinates (x', y') of the workpiece in the workpiece site image.

[0105] In some embodiments of the present application, the coordinates of the workpiece on any shelf in the semantic map can be expressed by (x', y', z'), where (x', y') represents the coordinates of the workpiece in the 2D plane map, and the z' value represents the layer of the current shelf where the workpiece is located. In this way, the positioning of the workpiece is achieved.

[0106] In some embodiments of the present application, each of the above data may further include an update time to record the data acquisition time of the remaining quantity of the workpiece in the data and clarify the timeliness information of the data.

[0107] Step 220: Associate the first position, name, remaining quantity and consumption rate of each workpiece in the assembly line, and the second position, name and remaining quantity of each workpiece on the shelf with the map data of the target location to construct a semantic map of the target location.

[0108] In some embodiments of the present application, when executing step 220, the first position, name, remaining quantity and consumption rate of each workpiece in the assembly line can be mapped to the map data of the target place; similarly, the second position, name and remaining quantity of each workpiece on the shelf can be mapped to the map data of the target place, and the positional relationship between the workpieces at each workstation in the assembly line and the workpieces on the shelves of the warehouse is established, thereby constructing a semantic map of the target place.

[0109] As a specific embodiment, when constructing a semantic map of a target location, a semantic map may be constructed first for the identified warehouse area, and then a semantic map may be constructed for the assembly line area, and the positional relationship between the workpieces at each workstation in the assembly line and the workpieces on the warehouse shelves may be established.

[0110] When constructing a semantic map of the warehouse area, the robot can be controlled to enter each warehouse in the warehouse area. For any warehouse, the robot is controlled to move along the inspection path of the warehouse to inspect each shelf in the warehouse. When inspecting any shelf, the workpiece site image of the workpiece on it is obtained, and the attributes of the workpiece in each workpiece site image are identified through the VLM large model to obtain the name, second position and remaining quantity of each workpiece. After each shelf is inspected, the name, second position and remaining quantity of each workpiece on the shelf are mapped to the map data of the target location, and then the data cache of the large model is cleared to eliminate the impact of the shelf data just inspected on the next shelf data, and determine whether all shelves in the warehouse area have been inspected. If not, the next shelf is inspected until all shelves in the warehouse area have been inspected.

[0111] When constructing a semantic map of the assembly line area, the robot can be controlled to enter the assembly line area. For any assembly line in the assembly line area, the robot can be controlled to move along the inspection path of the assembly line to inspect the container of each station in the assembly line: the workpiece site image of the container of each station is obtained, and the attributes of the workpiece in each workpiece site image are identified through the VLM large model to obtain the name, first position, remaining amount and consumption rate of each workpiece. After each assembly line is inspected, the name, first position, remaining amount and consumption rate of the workpiece in each container in the assembly line are mapped to the map data of the target location. Then, the data cache of the large model is cleaned up once to eliminate the influence of the just inspected assembly line data on the next assembly line data, and determine whether all assembly lines in the assembly line area have been inspected. If not, the next assembly line is inspected until all assembly lines in the assembly line area have been inspected.

[0112] After inspecting the warehouse area and assembly line area in the target place, based on the name of each workpiece in the assembly line, the storage location of the corresponding workpiece is searched from the semantic map corresponding to the warehouse area, and the found storage location (i.e., the second location) is stored as the workpiece attribute data of the workpiece together with the name of the workpiece, the first location, the remaining amount of the workpiece and the workpiece consumption rate. In this way, when there are insufficient workpieces on any assembly line, the second location of the warehouse where the workpiece is located can be quickly located, so that the robot or the robot's background server can complete the path planning operation.

[0113] In some embodiments of the present application, when executing step 220, for any workpiece on the shelf, the second map position of the shelf where the workpiece is located in the map data of the target location is replaced by the second position of the workpiece, and the second position, name and remaining quantity of the workpiece are associated to obtain the workpiece attribute data of the workpiece in the semantic map; similarly, for any workpiece in the assembly line, based on the name of the workpiece, the second position of the workpiece in the warehouse is searched, and then, the first map position of the workstation where the workpiece is located in the map data of the target location is replaced by the first position of the workpiece, and the first position, second position, name, remaining quantity of the workpiece and workpiece consumption rate are associated to obtain the workpiece attribute data for the workpiece in the semantic map.

[0114] Wherein, each of the above-mentioned workpiece attribute data may also include a corresponding update time.

[0115] In some embodiments of the present application, for a workpiece in a warehouse, the workpiece attribute data may include not only the corresponding update time, but also a workpiece on-site image of the workpiece, such as Figure 5 The workpiece attribute data for any workpiece on any shelf shown includes a workpiece on-site image of the workpiece, the name of the workpiece, the remaining amount of the workpiece, the update time, and the second position, such as Figure 5The position of the first workpiece located on the second layer of the shelf is shown (1, 0, 2), wherein 1 represents the horizontal coordinate value of the first workpiece placed on the shelf in the 2D plane map, corresponding to the aforementioned x', 0 represents the vertical coordinate value of the first workpiece placed on the shelf in the 2D plane map, corresponding to the aforementioned y', 2 represents that the first workpiece is placed on the second layer of the shelf, corresponding to the aforementioned z'), etc.

[0116] In this way, in the present application, the attributes of the workpieces in the acquired on-site images of each workpiece in the assembly line area and the warehouse area in the target place can be identified through a language model, such as a visual language model (Vision-Language Model, VLM), to obtain the attribute data of the workpieces in each workstation in each assembly line in the assembly line area, as well as the attribute data of the workpieces on each shelf in the warehouse area, and then based on the above attribute data, a semantic map of the target place is constructed, so that the semantic map includes the attribute data of each workpiece, so that during implementation, the robot can be guided to carry the workpiece based on the semantic map, thereby realizing the autonomy of workpiece handling, and thus improving the operation efficiency of the factory.

[0117] In other embodiments of the present application, the first position, name, remaining quantity and consumption rate of each workpiece in the assembly line obtained by the above steps 200 and 210, as well as the second position, name and remaining quantity of each workpiece on the shelf can also be stored in the form of a knowledge graph.

[0118] The following is a brief description of the process of constructing the knowledge graph for the target location.

[0119] First, construct the entity types and entity relationship types of the knowledge graph of the target location, where the entity types include artifacts, containers, warehouses, shelves, and assembly lines, and the relationship types include assembly line artifacts-containers, containers-locations, shelves-assembly lines, warehouse artifacts-locations, and artifacts-artifact attribute data.

[0120] Specifically, first, define the ontology model of the knowledge graph. In some embodiments of the present application, the OWL / RDF model language can be used. Figure 6 As shown in the figure, the entity types defined are workpiece, container, warehouse, shelf and assembly line; the relationship types defined are containedIn (workpiece->container), locatedAt (container->first position), supplies (shelf->assembly line), locatedAt (workpiece in warehouse->second position), hasProperty (workpiece->workpiece attribute data), etc., among which,

[0121] (1) containedIn(artifact->container) is used to indicate that an artifact is contained in a container;

[0122] (2) locatedAt(container->location) is used to indicate that a container is placed at a specific location or position. For example, if a box containing workpieces is placed at a certain location on a shelf or assembly line, then there is a locatedAt relationship between the box instance and the location instance.

[0123] (3) supplies (shelf -> assembly line) is used to indicate that the workpieces stored on the shelf are used to supply the production line. For example, a shelf with a parts box will have the parts taken out and transported to the assembly line. Therefore, there is a supplies relationship between the shelf instance and the assembly line instance.

[0124] (4) Workpiece attribute data includes the name of the workpiece, the remaining quantity of the workpiece, the workpiece consumption rate, the first position and the second position, the update time, etc.

[0125] Then, according to the entity type and entity relationship type, a knowledge graph of the target location is constructed based on the name, remaining quantity, consumption rate and first position of each workpiece in the assembly line area, and the name, remaining quantity and second position of each workpiece in the warehouse area.

[0126] Specifically, according to the constructed entity type and entity relationship type, the name, remaining quantity, consumption rate and first position of each workpiece in the assembly line area, as well as the name, remaining quantity and second position of each workpiece in the warehouse area are summarized and fused (such as Anzo, Apache Atlas and other tools), so as to construct a knowledge graph of the target place, such as Figure 7 shown.

[0127] In this way, in the present application, the entity types and entity relationship types of the knowledge graph can be constructed, and the above-mentioned attribute data obtained by identifying the attributes of the workpieces in the on-site images of the assembly line area and the warehouse area in the target place through the aforementioned language model can be stored and integrated using the knowledge graph of the target place, so that the knowledge graph includes the attribute data of each workpiece, so that during implementation, the robot can be guided to carry out workpiece transportation based on the knowledge graph, thereby realizing the autonomy of workpiece transportation and improving the operation efficiency of the factory.

[0128] After describing the process of building a semantic map or knowledge graph of a target place, see Figure 8 As shown, the embodiment of the present application provides a robot-based workpiece handling method, which is applied to a robot's backend server or robot. The method may include but is not limited to the following specific processes:

[0129] Step 800: Based on the attribute data of each workpiece in the semantic map or knowledge graph of the target place, obtain the remaining quantity of the reference workpiece in each workstation in the assembly line in the target place, wherein the semantic map or knowledge graph identifies the name and remaining quantity of the workpiece in the target place and is constructed based on the attributes of each identified workpiece, including multiple workpiece attribute data, each workpiece attribute data includes the name of the workpiece and the remaining quantity of the workpiece.

[0130] Based on the foregoing introduction, the constructed semantic map or knowledge graph includes multiple workpiece attribute data, each workpiece attribute data includes the name of the corresponding workpiece, the remaining quantity of the workpiece, etc. Then, when executing step 800, the remaining quantity of the reference workpiece at each workstation in the assembly line in the target place can be obtained based on each workpiece attribute data, so as to obtain the workpiece usage of the assembly line in the target place in real time, thereby facilitating the subsequent determination of the target workpiece to be transported. In this way, the workpiece attribute data in the semantic map or knowledge graph can be automatically updated according to the update time, ensuring that the workpiece attribute data in the semantic map or knowledge graph can truly reflect the real-time scene data of the target place, ensuring the authenticity of the target workpiece to be transported determined based on each workpiece attribute data, and thereby improving the operation efficiency of the factory.

[0131] Step 810: Determine the target workpiece to be transported based on the remaining amount of the reference workpiece at each workstation.

[0132] In some embodiments of the present application, when executing step 810, the remaining amount of the reference workpiece at each workstation is compared with the remaining amount threshold to determine whether the remaining amount of the reference workpiece at each workstation is less than the remaining amount threshold. If so, the reference workpiece is determined as the target workpiece.

[0133] In this application, by setting a remaining quantity threshold, the remaining situation of workpieces at each station is considered, and a certain amount of time is reserved for the workpiece transportation, so as to ensure that the assembly line can not be stopped due to lack of workpieces, wherein the remaining quantity threshold can be selected as 40% of the maximum number of workpieces that can be held by the container for holding workpieces at the station. In specific implementation, the remaining quantity threshold of the workpieces at each station can also be specifically set based on actual conditions, and this application does not make specific limitations.

[0134] In other embodiments of the present application, if it is determined that the remaining amount of the reference workpiece at each workstation is greater than or equal to a remaining amount threshold, a first image of the workpiece at each workstation in the assembly line area in the target location is obtained; the attributes of the workpiece in each first image are identified respectively, and the workpiece attribute data of the corresponding workpiece in the semantic map or knowledge graph is updated based on the identification results.

[0135] As a specific embodiment, the robot can perform inspection work on the assembly line area by default. At this time, the robot still inspects the assembly line area according to the original inspection path, obtains the first image of the workpiece of each workstation in the assembly line area in the target place, and identifies the attributes of the workpiece in each first image respectively, to obtain the name of the workpiece, the remaining quantity of the workpiece, etc., and uses the identified remaining quantity of the workpiece to update the workpiece attribute data associated with the name of the workpiece in the assembly line area in the semantic map or knowledge graph, so as to update the workpiece attribute data of part or all of the workpieces, and realize the integration of the real-time scene data of the target place into the semantic map or knowledge graph, so as to formulate and implement the corresponding workpiece handling plan based on the attribute data of each workpiece in real time.

[0136] As another specific embodiment, the background server may control the robot to perform inspection work on the assembly line area by default. For example, the background server formulates and sends an inspection task to the robot based on the position of each workstation in any assembly line in the assembly line area, wherein the inspection task includes an inspection path and obtaining an image of each workstation in the inspection path. After receiving the inspection task, the robot moves along the inspection path and obtains images of each workstation along the way. Then, the robot still inspects the assembly line area according to the original inspection path to obtain the first image of the workpiece of each workstation in the assembly line area in the target place, and uploads the obtained first images to the background server, which identifies the attributes of the workpiece in each received first image to obtain the name of the workpiece, the remaining amount of the workpiece, etc. As mentioned above, based on the first prompt instruction, the attributes of the workpiece in each first image are identified through the VLM large model; then, the identified remaining amount of the workpiece is used to update the workpiece attribute data associated with the name of the workpiece in the assembly line area in the semantic map or knowledge graph, so as to update the workpiece attribute data of part or all of the workpieces, so as to realize the fusion of the real-time scene data of the target place in the semantic map or knowledge graph, so as to formulate and implement the corresponding workpiece handling plan based on the attribute data of each workpiece in real time.

[0137] It should be noted that the attribute identification process for the workpiece in this application can be completed by the background server or by the robot itself, and this application does not make any specific limitations.

[0138] In the present application, when it is determined that there is currently no target workpiece to be transported, the first image of the workpiece at each workstation in the assembly line area can be obtained to obtain the number of workpieces at each workstation, and the workpiece attribute data recorded in the semantic map or knowledge graph can be updated, thereby ensuring that the workpiece attribute data in the semantic map or knowledge graph can truly reflect the real-time scene data of the target location, ensuring the authenticity of the target workpiece to be transported determined based on the attribute data of each workpiece, and thereby improving the operation efficiency of the factory.

[0139] In other embodiments, if there are multiple assembly lines in the target location, the aforementioned process can be executed for each assembly line to perform robot scheduling planning based on real-time scene data, guide the robot to take action, and improve the operating efficiency of the factory.

[0140] In some embodiments of the present application, the workpiece attribute data of any reference workpiece also includes a workpiece consumption rate; when executing step 810, after comparing the workpiece remaining amount of the reference workpiece at each workstation with the remaining amount threshold, it can be determined whether the number of reference workpieces whose workpiece remaining amount is less than the remaining amount threshold is greater than a preset value; if so, based on the workpiece remaining amount and the workpiece consumption rate of each reference workpiece, the remaining usage time of each reference workpiece is estimated respectively; according to the remaining usage time of each reference workpiece, a preset value of reference workpieces are selected from each reference workpiece as the target workpiece.

[0141] Optionally, the remaining usage time of each reference workpiece obtained can be sorted in ascending order, and the reference workpieces with a preset value of the first sorting are selected as the target workpieces. For example, the preset value is 5, then, from the reference workpieces whose remaining amount is less than the remaining amount threshold, the first 5 reference workpieces are selected as the target workpieces.

[0142] In some embodiments of the present application, when it is determined that the number of target workpieces to be transported is greater than a preset value, the remaining usage time of the existing workpieces can be estimated based on the remaining workpieces and the workpiece consumption rate, so that each target workpiece is selected from the reference workpieces whose remaining workpieces are less than the remaining threshold, and each selected target workpiece is transported first, thereby ensuring uninterrupted operation of the factory to the greatest extent possible, thereby improving the production efficiency of the assembly line.

[0143] Step 820: Obtain, from the workpiece attribute data of the target workpiece, a first position of the workstation where the target workpiece is located, and a second position of the warehouse where the target workpiece is located.

[0144] Based on the aforementioned process of constructing a semantic map or knowledge graph, it can be known that the workpiece attribute data of each workpiece in the assembly line also includes the first position of the workstation where the workpiece is located and the second position of the warehouse where the workpiece is located. Therefore, after executing step 810 to determine the target workpiece, step 820 can be executed to obtain the first position of the workstation where the target workpiece is located and the second position of the warehouse where the target workpiece is located from the workpiece attribute data of the target workpiece, so as to facilitate subsequent path planning based on the current equipment position, first position and second position of the robot, and formulate a scheduling strategy for the transportation plan.

[0145] Step 830: Perform path planning according to the current device position, the first position and the second position of the robot to obtain a transport path.

[0146] In some embodiments of the present application, when executing step 830, a multi-objective path planning algorithm can be called to directly plan the transportation path of the entire transportation process from the current equipment position of the robot, through the second position, and finally to the first position. Alternatively, a first transportation path from the current equipment position of the robot to the second position can be planned first, that is, the movement path of the robot from the current equipment position to the second position of the warehouse where the target workpiece is located, and then, a second transportation path from the second position to the first position can be planned, that is, the movement path from the warehouse to the workstation; this application does not make specific limitations.

[0147] In some other embodiments of the present application, if there are multiple target workpieces, such as the aforementioned preset value of target workpieces, when executing step 830, refer to Fig. 9 As shown, the following processes may be specifically performed but are not limited to:

[0148] Step 8301: Determine the constraint duration based on the remaining usage duration of a preset number of target workpieces.

[0149] In some embodiments of the present application, when executing step 8301, the target workpiece with the shortest remaining usage time can be selected from a preset number of target workpieces as the first target workpiece, and the remaining usage time of the first target workpiece can be used as the constraint time of this path planning.

[0150] Step 8302: Path planning is performed based on the constraint duration, the current device position, the second position of the warehouse where a preset number of target workpieces are located, and the first position of the workstation where a preset number of target workpieces are located.

[0151] In some embodiments of the present application, when executing step 8302, based on the constraint duration, the current device position of the robot, the second position of the warehouse where a preset value of target workpieces are located, and the first position of the workstation where a preset value of target workpieces are located, a multi-objective path planning algorithm and / or optimization algorithm is called to perform path planning.

[0152] The multi-objective path planning algorithm may include but is not limited to the A* algorithm, the sampling-based path planning algorithm (Rapidly-exploring Random Tree, RRT), etc. Taking the A* algorithm as an example, its basic principle is to select the node with the minimum cost to expand at each step until the target node is found or there is no node to expand. The evaluation function of the A* algorithm can be expressed by the following formula:

[0153] f*(n)=g*(n)+h*(n)

[0154] Among them, f*(n) is the evaluation function, which represents the estimated value from the starting point to the target node through the current node n; g*(n) is the actual cost function, which represents the actual cost from the starting point to the current node n. It is related to two factors. One is the length of the path from the starting point to the current node n, and the other is whether it is an obstacle. If it is an obstacle, the g*(n) value of this node is relatively large. Therefore, g*(n) can be used to measure the length of the planned path; h*(n) is the heuristic function, which represents the heuristic value from the current node n to the end point (i.e., the target node), which can be expressed using the Euclidean distance: It should be noted that the above-mentioned node is each grid obtained after gridding the map.

[0155] The following is a brief description of the processing flow of the A* algorithm.

[0156] Step 1, A* initialization:

[0157] (1) Initialize an open list, add the starting node to the open list, and set the g*(n) set and h*(n) set for the open list to store the g*(n) value and h*(n) value of each node in the open list, respectively. The starting node can be the current device position of the robot in this application; the open list is used to store unvisited nodes. Each time, select the node with the smallest f*(n) value from the open list as the current node, remove it from the open list, and then add its neighbor nodes (if they have not been visited) to the open list. It should be noted that this process will be repeated until the target node is found or the open list is empty. The target node can be any of the second positions of the warehouse where the preset value of target workpieces are located and the first positions of the workstations where the preset value of target workpieces are located in this application.

[0158] (2) Create a closed list and ensure that the starting node is not in the closed list. The closed list is used to save the node information removed from the open list to ensure that the open list does not repeatedly search for the removed nodes when searching for nodes. After finding the target node, the entire closed list is backtracked, that is, starting from the target node, searching for the previous parent node to find a feasible path.

[0159] Step 2, loop search: when the open set is not empty, perform the following operations: operation a, select the node with the smallest f*(n) value from the open set as the current node; operation b, remove the original current node from the open set and add it to the closed set. When operation a is executed for the first time, since there is no previous current node, there is no need to execute operation a. Operation a will be executed in subsequent processing.

[0160] After obtaining the current node, determine whether the current node is the target node.

[0161] Step 3, explore the neighboring nodes of the current node. If the current node is not the target node, traverse all the neighboring nodes of the current node:

[0162] (1) If the neighbor node is already in the closed set, skip processing the next neighbor node;

[0163] (2) If the neighbor node is not in the open set, add the neighbor node to the open set, calculate the f*(n) value of the neighbor node based on the above formula, and set the parent node of the neighbor node as the current node; calculate the f*(n) value for the neighbor node so that when the node with the smallest f*(n) value is selected from the open set later, the neighbor node can be used as the new current node. At the same time, set the parent node of the neighbor node as the current node, that is, record which node the neighbor node is found through, so that after finding the path, the complete path can be obtained by backtracking the parent node;

[0164] (3) If the neighbor node is already in the open set, but the g*(n) value from the starting node to the neighbor node through the current node is smaller, then update the g*(n) value and f*(n) value of the neighbor node, and change its parent node; since the neighbor node is already in the open set, that is, the neighbor node has been processed, which means that the calculated f*(n) value from the starting node to the neighbor node is greater than the f*(n) value from the starting node to the current node, but a new path is found to reach the neighbor node through the current node with a smaller cost, then update the g*(n) value and f*(n) value of the neighbor node to reflect this new and better path. At the same time, update the parent node of the neighbor node, because the optimal path is now through the current node to reach it, that is, use the current node as the parent node of the neighbor node.

[0165] Step 4: After each addition or update of a neighbor node, the nodes in the open set must be reordered according to the f*(n) value to ensure that the node with the smallest f*(n) value is at the front of the open set.

[0166] Step 5, path reconstruction. If the current node is the target node, then in the closed set, start from the target node and trace back to the start node through the parent node to reconstruct and record the complete path, so as to obtain the subsequent transport path.

[0167] In a specific implementation, it is possible to first plan the movement from the current equipment position of the robot to the warehouse and obtain the path to each target workpiece from the warehouse. Then, based on the remaining usage time of the multiple target workpieces at this time, secondary path planning is performed in order of the remaining usage time from small to large, so as to realize the transportation of the multiple target workpieces from the warehouse to the assembly line. Of course, it is also possible to plan the path from the current equipment position, through the second position of each target workpiece in the warehouse, and then back to the first position of each target workpiece, which will not be repeated here.

[0168] Step 8303: Determine whether a path is planned, if so, execute step 8304; if not, execute step 8305.

[0169] Since there are many destinations for path planning and the positions of each target workpiece in the assembly line or warehouse are uncertain, it is very likely that a path cannot be planned using the aforementioned multi-objective path planning algorithm under the aforementioned constraint time. Therefore, after executing step 8302, execute step 8303 to determine whether a path is planned. If a path is not planned, execute step 8305; if a path is planned, execute step 8304.

[0170] It should be noted that in the present application, when performing the above-mentioned multi-objective path planning, it may be necessary to set constraints according to actual conditions. At this time, when executing the above-mentioned process, path planning is not limited to being performed based on the constraint duration.

[0171] Step 8304: Determine the planned path as the transport path.

[0172] Step 8305: Repeat the following operations until a path is planned: select an alternative workpiece from each target workpiece of the currently planned path, and perform path planning based on the constraint duration, the current device position, and the first position and the second position of each target workpiece except the alternative workpiece, wherein the alternative workpiece is a target workpiece whose remaining usage time meets the preset conditions.

[0173] In some embodiments of the present application, if a path is not planned, step 8305 is executed. When executing step 8305, first, an alternative workpiece is selected from the preset value target workpieces, wherein the alternative workpiece can be the target workpiece with the longest remaining use time among the target workpieces of the current planned path, and in this case, it is the target workpiece with the longest remaining use time among the preset value target workpieces; then, based on the constraint time, the current device position, and the first position and the second position of the preset value target workpieces other than the alternative workpiece, a multi-objective path planning algorithm and / or an optimization algorithm is called to perform path planning, and it is determined whether a path is planned, and if not, the alternative workpiece is continuously selected from the target workpieces of the current planned path, that is, the alternative workpiece is selected again from the preset value target workpieces other than the alternative workpiece, and the path planning is performed again for the remaining target workpieces until a path is planned.

[0174] In some other embodiments of the present application, if it is determined that the number of target workpieces of the currently planned path is less than or equal to 2, it is necessary to delete the target workpiece with the longest remaining usage time among the two target workpieces. Then, the next round of path planning only needs to be performed on one target workpiece. At this time, workpiece handling assistance information can also be output to prompt that there is currently a workpiece handling task that requires assistance, and based on the workpiece handling collaborative information, a collaborative robot is determined from other robots in the target location, and the workpiece handling is completed in collaboration with the collaborative robot, wherein the collaborative robot can be a robot whose working state is a specified working state, or a robot located at a specified position.

[0175] In some embodiments of the present application, the robot may receive responses from other robots after outputting workpiece handling assistance information, and based on the received responses, determine a collaborative robot and complete the workpiece handling together with the collaborative robot.

[0176] In some other embodiments of the present application, the background server can determine the working status of other robots in the target location, adjust the other robots in the specified working status to the workpiece handling working status, use the other robots as collaborative robots, and control the collaborative robot and the robot to collaboratively complete the workpiece handling, wherein the other robot can perform the handling task of the target workpiece with the longest remaining usage time among the two target workpieces, and can also perform the handling task of the target workpiece with the shorter remaining usage time among the two target workpieces.

[0177] In some other embodiments of the present application, the background server can determine whether there are other robots in the target location that are located near the second position of the warehouse where any target workpiece is located. If so, the background server can control the other robots to perform the task of transporting the target workpiece.

[0178] In some embodiments of the present application, collaborative work of multiple robots in a target location can be achieved through workpiece handling assistance information, thereby improving work efficiency and further improving the operation efficiency of the factory.

[0179] Step 840: Execute the transport task for the target workpiece according to the transport path.

[0180] In some embodiments of the present application, when executing step 840, the robot moves along the transport path. After the robot reaches the second position of the warehouse where the target workpiece is located, it grabs the target workpiece and returns to the first position along the transport path, thereby realizing the robot's transport of the target workpiece.

[0181] In some other embodiments of the present application, when executing step 840, the background server may send the transport path to the robot, and the robot performs the transport task for the target workpiece according to the transport path.

[0182] In some embodiments of the present application, when executing step 840, while the robot is performing the transport task, that is, while moving along the transport path, it can also obtain a second image of the workpiece at each workstation and / or each shelf on the transport path, identify the attributes of the workpiece in each second image respectively, and update the workpiece attribute data of the corresponding workpiece in the semantic map or knowledge graph based on the identification results.

[0183] In some other embodiments of the present application, after the robot obtains the second image of the workpiece of each station and / or each shelf on the transport path, the second image can also be uploaded to the background server, and the background server can identify the attributes of the workpiece in the second image. Specifically, for the second image of the workpiece in the container of each station on the transport path, the background server can identify the attributes of the workpiece in each second image obtained based on the first prompt instruction of the assembly line area, obtain the name and remaining amount of the workpiece of each station, and update the remaining amount of the workpiece corresponding to the workpiece in the semantic map or knowledge graph based on the name and remaining amount of each workpiece; similarly, for the workpiece site image of the workpiece on each shelf on the transport path, the background server can identify the attributes of the workpiece in each workpiece site image obtained based on the second prompt instruction of the warehouse area, obtain the name and remaining amount of the workpiece on each shelf, and update the remaining amount of the workpiece corresponding to the workpiece on the shelf in the semantic map or knowledge graph based on the name and remaining amount of each workpiece, thereby realizing the real-time update of the workpiece attribute data of the workpiece.

[0184] In this way, in the present application, it is also possible to obtain a second image of the workpieces at each workstation and / or on each shelf on the transport path during the execution of the transport task, thereby obtaining the number of workpieces on each shelf, and updating the workpiece attribute data recorded in the semantic map or knowledge graph, thereby ensuring that the workpiece attribute data in the semantic map or knowledge graph can truly reflect the real-time scene data of the target location, ensuring the authenticity of the target workpiece to be transported determined based on the attribute data of each workpiece, and thereby improving the operation efficiency of the factory.

[0185] As a specific embodiment, the complete process of the robot-based workpiece handling method in some embodiments of the present application may include but is not limited to the following operations:

[0186] Operation 1: Read the 3D map of the target location.

[0187] Operation 2: Use the VLM large model to identify the assembly line area and warehouse area in the 3D map to obtain the assembly line area and warehouse area.

[0188] Operation 3: Create inspection routes for the robot in the assembly line area and warehouse area respectively.

[0189] Operation 4: Conduct inspections along the inspection route and build a semantic map of the target location.

[0190] Operation 5: Assign tasks to the robot based on the attribute data of each workpiece in the semantic map.

[0191] Operation 6: Determine whether the robot is located in the assembly line area; if so, perform operation 7; if not, perform operation 8.

[0192] Operation 7: Control the robot to inspect the assembly line.

[0193] Operation 8: Determine whether the robot has stopped working. If not, perform operation 5. If yes, end the process.

[0194] When executing operation 7, you may also specifically execute sub-operations including but not limited to the following:

[0195] Sub-operation 1: traverse the attribute data of each workpiece in the semantic map to obtain the remaining quantity and remaining usage time of the reference workpiece at each station in each assembly line.

[0196] Sub-operation 2: Determine whether the remaining number of workpieces is less than 40% of the maximum number of workpieces that the container can hold. If so, and the number of reference workpieces whose remaining number of workpieces is less than 40% of the maximum number of workpieces that the container can hold is greater than 5, execute sub-operation 5; if not, do nothing.

[0197] Sub-operation 3: Sort the reference workpieces according to their remaining usage time from small to large, and select the first five reference workpieces as target workpieces;

[0198] Sub-operation 4: Path planning for 5 target workpieces;

[0199] Sub-operation 5: When a path is planned, the path is determined as a transport path, and a transport task for the target workpieces is performed according to the transport path, so as to transport the five target workpieces from the warehouse to corresponding workstations.

[0200] Sub-operation 6: before a path is planned, determine whether the number of target workpieces of the currently planned path is greater than 2, if so, execute sub-operation 7, if not, execute sub-operation 8.

[0201] Sub-operation 7: Delete the target workpiece with the longest remaining usage time, and perform path planning for each remaining target workpiece.

[0202] Sub-operation 8: Notify the worker or identify the collaborative robot to assist in completing the task.

[0203] Based on the same inventive concept, refer to Fig.10 As shown, an embodiment of the present application provides a robot-based workpiece handling device, comprising:

[0204] The obtaining module 1010 is used to obtain the remaining quantity of the reference workpieces at each station in the assembly line in the target place based on the attribute data of each workpiece in the semantic map or knowledge graph of the target place, wherein the semantic map or the knowledge graph identifies the attributes of the workpieces in the target place and is constructed based on the attributes of each identified workpiece, including multiple workpiece attribute data, each workpiece attribute data including the name of the workpiece and the remaining quantity of the workpiece;

[0205] A decision module 1020 is used to determine a target workpiece to be transported based on the remaining amount of the reference workpieces at each workstation, and obtain a first position of the workstation where the target workpiece is located and a second position of the warehouse where the target workpiece is located from workpiece attribute data of the target workpiece;

[0206] The path planning and execution module 1030 is used to perform path planning according to the current device position of the robot, the first position and the second position to obtain a transport path; and execute the transport task for the target workpiece according to the transport path.

[0207] In a possible implementation, the workpiece attribute data of any reference workpiece further includes a workpiece consumption rate, and the decision module 1020 is specifically configured to:

[0208] respectively comparing the workpiece remaining quantity of the reference workpiece at each workstation with a remaining quantity threshold;

[0209] Determine whether the number of reference workpieces whose remaining quantity is less than the remaining quantity threshold is greater than a preset value;

[0210] If yes, then estimating the remaining usage time of each reference workpiece based on the remaining amount of each reference workpiece and the workpiece consumption rate;

[0211] According to the remaining usage time of each reference workpiece, a preset number of reference workpieces are selected from the reference workpieces as the target workpieces.

[0212] In a possible implementation, the path planning and execution module 1030 is specifically used to:

[0213] Determine a constraint duration based on the preset value and the remaining usage duration of the target workpiece;

[0214] Performing path planning based on the constraint duration, the current device position, the preset number of second positions of the warehouse where the target workpieces are located, and the preset number of first positions of the workstations where the target workpieces are located;

[0215] When a path is planned, the planned path is determined as the transport path;

[0216] When a path is not planned, the following operations are repeatedly performed until a path is planned: selecting an alternative workpiece from each of the target workpieces of the currently planned path, and performing path planning based on the constraint duration, the current device position, and the first position and the second position of each of the target workpieces except the alternative workpiece, wherein the alternative workpiece is a target workpiece whose remaining usage duration meets a preset condition among the target workpieces.

[0217] In a possible implementation, the path planning and execution module 1030 is further configured to:

[0218] If it is determined that the number of each of the target workpieces in the currently planned path is less than or equal to 2, then outputting workpiece handling assistance information to indicate that there is currently a workpiece handling task that requires assistance;

[0219] Based on the workpiece handling assistance information, an assisting robot is determined from other robots in the target location, and the workpiece handling is completed in collaboration with the assisting robot.

[0220] In a possible implementation, the workpiece attribute data of any reference workpiece further includes update time, and the obtaining module 1010 is further used to:

[0221] Based on the update time, the current time and the workpiece consumption rate of the any reference workpiece, updating the remaining amount of the workpiece in the workpiece attribute data of the any reference workpiece;

[0222] The obtaining module 1010 is also used for:

[0223] If it is determined that the remaining amount of the reference workpiece at each workstation is greater than or equal to the remaining amount threshold, acquiring a first image of the workpiece at each workstation in the assembly line area in the target location;

[0224] Respectively identifying the attributes of the artifacts in each first image, and updating the artifact attribute data of the corresponding artifact in the semantic map or knowledge graph based on the identification results;

[0225] The obtaining module 1010 is also used for:

[0226] During the execution of the transport task, a second image of the workpiece at each workstation and / or each shelf on the transport path is acquired;

[0227] The attributes of the artifacts in each second image are identified respectively, and the artifact attribute data of the corresponding artifacts in the semantic map or the knowledge graph are updated based on the identification results.

[0228] In a possible implementation, the obtaining module 1010 is further configured to construct a semantic map of the target place in the following manner:

[0229] Acquire a workpiece scene image and a first position of each workstation in the assembly line in the assembly line area in the target location, and based on a first prompt instruction in the assembly line area, respectively identify the attributes of the workpiece in each acquired workpiece scene image through a preset language model to obtain the name, remaining quantity and consumption rate of each workpiece in the assembly line;

[0230] Acquire a workpiece site image and a second position of each workpiece on a shelf in a warehouse area in the target location, and based on a second prompt instruction in the warehouse area, identify the attributes of each workpiece in the acquired workpiece site image by using the language model to obtain a name of each workpiece on the shelf and a remaining amount of the workpiece;

[0231] The first position, name, remaining quantity and consumption rate of each workpiece in the assembly line, and the second position, name and remaining quantity of each workpiece on the shelf are respectively associated with the map data of the planar map to construct a semantic map of the target location.

[0232] In a possible implementation, the obtaining module 1010 is further configured to:

[0233] Constructing entity types and entity relationship types of the knowledge graph of the target location, wherein the entity types include workpieces, containers, warehouses, shelves, and assembly lines, and the relationship types include assembly line workpiece-container, container-location, shelf-assembly line, warehouse workpiece-location, and workpiece-workpiece attribute data;

[0234] According to the entity type and the entity relationship type, based on the identification, name, remaining quantity, workpiece consumption rate and first position of each workpiece in the assembly line area, and the name, remaining quantity and second position of each workpiece in the warehouse area, a knowledge graph of the target place is constructed.

[0235] Based on the same inventive concept, refer to Fig.11 As shown, an electronic device is provided in an embodiment of the present application, including:

[0236] A memory 111, used for storing computer programs or instructions;

[0237] The processor 112 is used to execute the computer program or instructions in the memory 111 so that any one of the methods in the above embodiments is executed.

[0238] The processor 112 may include one or more central processing units (CPUs) or digital processing units, etc., for executing computer programs or instructions in the memory 111, so that any of the methods in the above embodiments is executed;

[0239] It should be noted that the specific connection medium between the memory 111 and the processor 112 is not limited in the embodiment of the present application. Fig.11 In the embodiment, the memory 111 and the processor 112 are connected via a bus 113, and the connection between other components is only for illustrative purposes and is not intended to be limiting. The bus 113 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0240] Based on the same inventive concept, the embodiment of the present application provides a computer-readable storage medium. When the instructions in the storage medium are executed by a processor, the processor is enabled to execute any one of the methods in the above embodiments. Since the principle of solving the problem by the above computer-readable storage medium is similar to that of a robot-based workpiece handling method, the implementation of the above computer-readable storage medium can refer to the implementation of the method, and the repeated parts will not be repeated.

[0241] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0242] The present application is described with reference to the flowchart and / or block diagram of the method, device (system), and computer program product according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the process and / or box in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0243] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A robot-based workpiece handling method, characterized in that: include: Based on the attribute data of each workpiece in the semantic map or knowledge graph of the target place, the remaining amount of the reference workpiece in each workstation in the assembly line in the target place is obtained, wherein the semantic map or the knowledge graph identifies the attributes of the workpieces in the target place and is constructed based on the attributes of each identified workpiece, including multiple workpiece attribute data, each workpiece attribute data including the name of the workpiece and the remaining amount of the workpiece; Determine a target workpiece to be transported based on the remaining amount of the reference workpieces at each workstation, and obtain a first position of the workstation where the target workpiece is located and a second position of the warehouse where the target workpiece is located from workpiece attribute data of the target workpiece; Path planning is performed according to the current device position of the robot, the first position and the second position to obtain a transport path; and a transport task for the target workpiece is performed according to the transport path.

2. The method according to claim 1, characterized in that The workpiece attribute data of any reference workpiece also includes a workpiece consumption rate, and the method of determining the target workpiece to be transported based on the remaining amount of the reference workpieces at each station includes: respectively comparing the workpiece remaining quantity of the reference workpiece at each workstation with a remaining quantity threshold; Determine whether the number of reference workpieces whose remaining quantity is less than the remaining quantity threshold is greater than a preset value; If yes, then estimating the remaining usage time of each reference workpiece based on the remaining amount of each reference workpiece and the workpiece consumption rate; According to the remaining usage time of each reference workpiece, a preset number of reference workpieces are selected from the reference workpieces as the target workpieces.

3. The method according to claim 2, characterized in that The performing path planning based on the current device position of the robot, the first position and the second position to obtain a transport path includes: Determine a constraint duration based on the preset value and the remaining usage duration of the target workpiece; Performing path planning based on the constraint duration, the current device position, the preset number of second positions of the warehouse where the target workpieces are located, and the preset number of first positions of the workstations where the target workpieces are located; When a path is planned, the planned path is determined as the transport path; When a path is not planned, the following operations are repeatedly performed until a path is planned: selecting an alternative workpiece from each of the target workpieces of the currently planned path, and performing path planning based on the constraint duration, the current device position, and the first position and the second position of each of the target workpieces except the alternative workpiece, wherein the alternative workpiece is a target workpiece whose remaining usage duration meets a preset condition among the target workpieces.

4. The method according to claim 3, characterized in that The method further comprises: If it is determined that the number of each of the target workpieces in the currently planned path is less than or equal to 2, then outputting workpiece handling assistance information to indicate that there is currently a workpiece handling task that requires assistance; Based on the workpiece handling assistance information, an assisting robot is determined from other robots in the target location, and the workpiece handling is completed in collaboration with the assisting robot.

5. The method according to claim 2, characterized in that The workpiece attribute data of any reference workpiece further includes an update time, and the method further includes: Based on the update time, the current time and the workpiece consumption rate of the any reference workpiece, updating the remaining amount of the workpiece in the workpiece attribute data of the any reference workpiece; The method further comprises: If it is determined that the remaining amount of the reference workpiece at each workstation is greater than or equal to the remaining amount threshold, acquiring a first image of the workpiece at each workstation in the assembly line area in the target location; Respectively identifying the attributes of the artifacts in each first image, and updating the artifact attribute data of the corresponding artifact in the semantic map or knowledge graph based on the identification results; The method further comprises: During the execution of the transport task, a second image of the workpiece at each workstation and / or each shelf on the transport path is acquired; The attributes of the artifacts in each second image are identified respectively, and the artifact attribute data of the corresponding artifacts in the semantic map or the knowledge graph are updated based on the identification results.

6. The method according to any one of claims 1 to 5, characterized in that: The semantic map of the target place is constructed in the following way: Acquire a workpiece scene image and a first position of each workstation in the assembly line in the assembly line area in the target location, and based on a first prompt instruction in the assembly line area, respectively identify the attributes of the workpiece in each acquired workpiece scene image through a preset language model to obtain the name, remaining quantity and consumption rate of each workpiece in the assembly line; Acquire a workpiece site image and a second position of each workpiece on a shelf in a warehouse area in the target location, and based on a second prompt instruction in the warehouse area, identify the attributes of each workpiece in the acquired workpiece site image by using the language model to obtain a name of each workpiece on the shelf and a remaining amount of the workpiece; The first position, name, remaining quantity and consumption rate of each workpiece in the assembly line, and the second position, name and remaining quantity of each workpiece on the shelf are respectively associated with the map data of the planar map to construct a semantic map of the target location.

7. The method according to claim 6, characterized in that The method further comprises: Constructing entity types and entity relationship types of the knowledge graph of the target location, wherein the entity types include workpieces, containers, warehouses, shelves, and assembly lines, and the relationship types include assembly line workpiece-container, container-location, shelf-assembly line, warehouse workpiece-location, and workpiece-workpiece attribute data; According to the entity type and the entity relationship type, based on the identification, name, remaining quantity, workpiece consumption rate and first position of each workpiece in the assembly line area, and the name, remaining quantity and second position of each workpiece in the warehouse area, a knowledge graph of the target place is constructed.

8. A robot-based workpiece handling device, characterized in that: include: An acquisition module is used to obtain the remaining quantity of the reference workpieces at each station in the assembly line in the target place based on the attribute data of each workpiece in the semantic map or knowledge graph of the target place, wherein the semantic map or the knowledge graph identifies the attributes of the workpieces in the target place and is constructed based on the attributes of each identified workpiece, including multiple workpiece attribute data, each workpiece attribute data including the name of the workpiece and the remaining quantity of the workpiece; A decision module, configured to determine a target workpiece to be transported based on the remaining amount of the reference workpieces at each workstation, and obtain a first position of the workstation where the target workpiece is located and a second position of the warehouse where the target workpiece is located from workpiece attribute data of the target workpiece; A path planning and execution module is used to perform path planning according to the current device position of the robot, the first position and the second position to obtain a transport path; and execute a transport task for the target workpiece according to the transport path.

9. An electronic device, characterized in that: include: Memory, used to store computer programs or instructions; A processor is used to execute the computer program or instructions in the memory so that the method according to any one of claims 1 to 7 is executed.

10. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the method according to any one of claims 1 to 7.

Images