Visual feedback-based robot grabbing method and device in disordered environment

Abstract

The invention provides a disordered environment robot grabbing method and device based on visual feedback, and the method comprises the steps: determining first communication paths from an input port to an output port in a target service grid system, generating first test service requests corresponding to all the first communication paths, and enabling all the first test service requests to be connected with the output port of the target service grid system; inputting the target service grid system from the input ports corresponding to the first test service requests to obtain first test service results output by the target service grid system, and sending the first test service results to the target service grid system under the condition that the first test service results are matched with service reference results corresponding to the first communication paths. According to the method, the target service grid system is determined to pass the test, a more comprehensive test process can be executed on the target service grid system, the problem that the test process is not comprehensive due to the difference of the service grid system is avoided, and the accuracy of the test result of the service grid system is improved to a certain extent.

Description

Technical Field

[0001] This application relates to the field of automation technology, and in particular to a method and apparatus for robot grasping in cluttered environments based on visual feedback. Background Technology

[0002] Robotic picking and placing in cluttered environments is a common task in industrial automation and warehouse management. Mastering this task involves complex perception, planning, and control mechanisms, making it challenging to grasp items in chaotic scenarios.

[0003] In related technologies, robot object grasping can be achieved through forward and inverse kinematics algorithms based on dynamic models and kinematics. The above method can model the robot arm and obtain the joint angles of each joint of the robot arm from the Cartesian coordinates of the end of the object to be grasped, thereby realizing the robot arm grasping the target object.

[0004] However, the above methods have low adaptability to cluttered environments, and the robot often fails to perform the object grasping task correctly. The grasping accuracy and efficiency of the above methods in cluttered environments are not high. Summary of the Invention

[0005] In view of the above problems, embodiments of this application provide a method, apparatus, electronic device, and readable storage medium for robot grasping in cluttered environments based on visual feedback, so as to overcome the above problems or at least partially solve the above problems.

[0006] In a first aspect, embodiments of this application provide a method for robot grasping in cluttered environments based on visual feedback, the method comprising: Determine the first connected path from the input port to the output port in the target service mesh system; Generate the first test service request corresponding to each of the first connected paths; Each of the first test service requests is input into the target service mesh system from the input port corresponding to each of the first test service requests, and the first test service results output by the target service mesh system are obtained. If the results of each first test service match the service reference results corresponding to each of the first connected paths, the target service mesh system is determined to have passed the test.

[0007] Optionally, the method further includes: Based on the first satisfiability modulo theoretical conditions corresponding to each first condition node in each first connected path, the first input message information corresponding to each first connected path is determined. Based on the message modification actions corresponding to each action node in each first connected path, determine the first modification function corresponding to each first connected path. Based on the first input message information and the first modification function, the service reference results corresponding to each of the first connected paths are determined.

[0008] Optionally, determining the first connectivity path from the input port to the output port in the target service mesh system includes: Determine the first control flow graph of the target service mesh system; Determine the first service request configuration information of each second action node and the second satisfiability modulo theoretical condition of each second condition node contained in the first control flow graph; Based on the first service request configuration information and the second satisfiability modulus theoretical conditions, the first connectivity path from the input port to the output port in the target service mesh system is determined.

[0009] Optionally, determining the first connectivity path from the input port to the output port in the target service mesh system based on the first service request configuration information and the second satisfiability modulus theoretical condition includes: Based on the first service request configuration information, determine each input port of the target service mesh system and the corresponding output port of each input port; Based on the second satisfiability module theory conditions, the third action node and the third condition node between the input port and the corresponding output port are determined, as well as the upstream and downstream relationships corresponding to the third action node and the third condition node respectively. Based on the upstream and downstream relationship, the input port, the output port corresponding to the input port, the third action node, and the third condition node are connected to obtain the first connected path.

[0010] Optionally, determining the first connectivity path from the input port to the output port in the target service mesh system based on the first service request configuration information and the second satisfiability modulus theoretical condition includes: Based on the first service request configuration information, determine each input port of the target service mesh system and the corresponding output port of each input port; Based on the symbolic execution method, the fourth action node and the fourth condition node corresponding to each input port are determined from the second action node and the second condition node, respectively; Based on the symbolic execution method, the input port, the corresponding output port, the fourth action node, and the fourth condition node are connected in the order of the fourth action node and the fourth condition node to obtain the first connected path.

[0011] Optionally, generating the first test service request corresponding to each of the first connected paths includes: Determine the second service request configuration information corresponding to each of the first connected paths from the action nodes contained in each of the first connected paths; Based on the configuration information of each of the second service requests, a first test service request corresponding to each of the first connectivity paths is generated.

[0012] Optionally, the step of inputting each of the first test service requests into the target service mesh system from the input port corresponding to each of the first test service requests, and obtaining the first test service results output by the target service mesh system, includes: Determine the name of the first service mesh resource corresponding to each of the first connected paths; The first service mesh resource name is matched with the standard resource name to obtain the first matching result; Based on the first matching result, determine the test priority of each of the first connected paths; According to the order of test priority from high to low, each of the first test service requests is input into the target service mesh system from the input port corresponding to each of the first test service requests, and the first test service results output by the target service mesh system are obtained.

[0013] Optionally, determining the test priority of each of the first connected paths based on the first matching result includes: If the first matching result is an exact match, the test priority of the first connected path is determined to be the first priority; If the first matching result is a regular expression match, the test priority of the first connected path is determined to be the second priority; wherein the second priority is lower than the first priority.

[0014] Optionally, the method further includes: In the first connected path, if there are at least two second connected paths where the first matching result is either an exact match or a regular expression match, and the standard resource names corresponding to the second connected paths are the same, the creation time of the second connected path is determined. The test priority of the second connected path created earlier is determined as the third priority, and the test priority of the second connected path created later is determined as the fourth priority; wherein the third priority is higher than the fourth priority.

[0015] Secondly, embodiments of this application provide a visual feedback-based robotic grasping device for cluttered environments, the device comprising: The first determining module is used to determine the first connected path from the input port to the output port in the target service mesh system; The generation module is used to generate the first test service request corresponding to each of the first connected paths. The input module is used to input each of the first test service requests from the input ports corresponding to each of the first test service requests into the target service mesh system, and to obtain the results of each of the first test services output by the target service mesh system. The second determining module is used to determine that the target service mesh system has passed the test if the results of each first test service match the service reference results corresponding to each of the first connected paths.

[0016] Optionally, the device further includes: The third determining module is used to determine the first input message information corresponding to each of the first connected paths based on the first satisfiability modulo theoretical conditions corresponding to each of the first condition nodes in each of the first connected paths. The fourth determining module is used to determine the first modification function corresponding to each of the first connected paths based on the message modification actions corresponding to each action node in each of the first connected paths. The fifth determining module is used to determine the service reference result corresponding to each of the first connected paths based on the first input message information and the first modification function.

[0017] Optionally, the first determining module includes: The first determination submodule is used to determine the first control flow graph of the target service mesh system; The second determining submodule is used to determine the first service request configuration information of each second action node and the second satisfiability model theoretical condition of each second condition node contained in the first control flow graph. The third determining submodule is used to determine the first connectivity path from the input port to the output port in the target service mesh system based on the first service request configuration information and the second satisfiability model theoretical conditions.

[0018] Optionally, the third determining submodule includes: The first determining unit is configured to determine, based on the first service request configuration information, each input port of the target service mesh system and the corresponding output port of each input port; The second determining unit is used to determine, based on the second satisfiability modulus theoretical conditions, the third action node and the third condition node between the input port and the output port corresponding to the input port, as well as the upstream and downstream relationships corresponding to the third action node and the third condition node respectively; The first connection unit is used to connect the input port, the output port corresponding to the input port, the third action node and the third condition node based on the upstream and downstream relationship to obtain a first connection path.

[0019] Optionally, the third determining submodule includes: The fourth determining unit is used to determine, based on the first service request configuration information, each input port of the target service mesh system and the corresponding output port of each input port; The fifth determining unit is used to determine the fourth action node and the fourth condition node corresponding to each input port from the second action node and the second condition node, respectively, based on the symbolic execution method. The second connection unit is used to connect the input port, the output port corresponding to the input port, the fourth action node, and the fourth condition node based on the order in which the symbolic execution method passes through the fourth action node and the fourth condition node, to obtain a first connected path.

[0020] Optionally, the generation module includes: The fourth determining submodule is used to determine the second service request configuration information corresponding to each of the first connected paths from the action nodes contained in each of the first connected paths. The generation submodule is used to generate the first test service request corresponding to each of the first connectivity paths based on the configuration information of each of the second service requests.

[0021] Optionally, the input module includes: The fifth determining submodule is used to determine the name of the first service mesh resource corresponding to each of the first connected paths; The matching submodule is used to match the first service mesh resource name with the standard resource name to obtain the first matching result; The sixth determining submodule is used to determine the test priority of each of the first connected paths based on the first matching result; The input submodule is used to input each of the first test service requests into the target service mesh system from the input port corresponding to each of the first test service requests in descending order of test priority, so as to obtain the first test service results output by the target service mesh system.

[0022] Optionally, the sixth determining submodule includes: The sixth determining unit is used to determine the test priority of the first connected path as the first priority when the first matching result is an exact match; The seventh determining unit is used to determine the test priority of the first connected path as the second priority when the first matching result is a regular expression matching; wherein the second priority is lower than the first priority.

[0023] Optionally, the device further includes: The sixth determining module is used to determine the creation time of the second connected path when there are at least two second connected paths in the first connected path where the first matching result is either an exact match or a regular expression match, and the standard resource names corresponding to the second connected paths are the same. The seventh determining module is used to determine the test priority of the second connected path with the earlier creation time as the third priority, and to determine the test priority of the second connected path with the later creation time as the fourth priority; wherein the third priority is higher than the fourth priority.

[0024] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the visual feedback-based robot grasping method for cluttered environments as described in any of the above embodiments.

[0025] Fourthly, embodiments of this application provide a readable storage medium storing a program or instructions, which, when executed by a processor, implement the visual feedback-based robot grasping method for cluttered environments as described above.

[0026] The specific beneficial effects are as follows: This application embodiment determines the first connectivity path from the input port to the output port in the target service mesh system, generates a first test service request corresponding to each first connectivity path, and inputs each first test service request into the target service mesh system from its corresponding input port to obtain the first test service results output by the target service mesh system. If each first test service result matches the service reference result corresponding to each first connectivity path, the target service mesh system is deemed to have passed the test. This approach allows for input testing of the first connectivity paths in the target service mesh system using the first test service requests corresponding to each first connectivity path. Furthermore, if the test results of each first connectivity path match their respective service reference results, the target service mesh system is deemed to have passed the test. This enables a more comprehensive testing process for the target service mesh system, avoiding the problem of incomplete testing processes caused by differences in service mesh systems, and improving the accuracy of the test results to a certain extent. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating a visual feedback-based robot grasping method for cluttered environments provided in an embodiment of this application. Figure 2 This is a flowchart illustrating another method for robot grasping in cluttered environments based on visual feedback, provided in an embodiment of this application. Figure 3 This is a logic block diagram of a visual feedback-based robotic grasping device for cluttered environments provided in an embodiment of this application. Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0030] Reference Figure 1, Figure 1 This application provides a flowchart illustrating a visual feedback-based robot grasping method for cluttered environments, the method comprising: Step 101: Determine the first connected path from the input port to the output port in the target service mesh system.

[0031] In the embodiments of this application, a service mesh is a method for handling service communication in a microservice architecture. It is an infrastructure layer located between services, providing functions such as service discovery and load balancing, traffic control and timeout handling, security and authentication, and monitoring and authentication. The service mesh can run on a Kubernetes (k8s) cluster within Kind software. Accordingly, the execution environment of the service mesh test oracle method in this application can be the same as the service mesh's running environment. A service mesh can include two components: a data plane and a control plane. The data plane handles the actual request and response traffic. It resides between services and is responsible for request routing, load balancing, and traffic control. The control plane manages the service mesh's configuration and policies, and is responsible for service discovery, routing rules, and security settings. The target service mesh system can be the service mesh system to be tested. In the target service mesh system, there can be multiple input ports and multiple output ports used to execute different types of communication services. In the target service mesh system, an input port can form a connection path with at least one output port. This connection path is directed, pointing from the input port to the output port, and each node on this path is also a directed node, indicating the direction of signal flow. Based on this, after obtaining the target service mesh system, it can be analyzed to determine the first connection path from the input port to the output port within the target service mesh system. This first connection path can include all possible connection paths within the target service mesh system.

[0032] Step 102: Generate the first test service request corresponding to each of the first connected paths.

[0033] In the embodiments of this application, after obtaining the first connectivity path of the target service mesh system, a first test service request corresponding to each first connectivity path can be generated according to the service type corresponding to each first connectivity path. The first test service request can correspond to configuration information such as service host, service request header, service security settings, and service input and output ports. The service host, service request header, and service security are set as request fields, and the service input and output ports are the metadata of the service request. Since the first connectivity path covers all possible connectivity paths of the target service mesh system, the first test service request can cover all possible service types of the target service mesh system. Furthermore, the format of the first test service request can be set independently, as long as there is a correspondence between the first test service request and the service type, without needing to include all the aforementioned configuration information in the request command.

[0034] In one possible embodiment, the first test service request can also be encoded by mapping the string of the first test service request to integers using a global string table, thereby converting the first test service request into a list of integers. This reduces the complexity of the test service request, and the list of integers can also include magic numbers to distinguish test requests from real requests.

[0035] Step 103: Input each of the first test service requests into the target service mesh system from the input port corresponding to each of the first test service requests, and obtain the first test service results output by the target service mesh system.

[0036] In the embodiments of this application, each first test service request can be input into the target service mesh system from its corresponding input port. After entering the target service mesh system, the first test service request can undergo corresponding data processing, and the first test service result can be output at its corresponding output port. The data flow path of the first test service request after entering the target service mesh system can be the first connectivity path corresponding to that first test service request.

[0037] Step 104: If each of the first test service results matches the service reference results corresponding to each of the first connected paths, the target service mesh system is determined to have passed the test.

[0038] In the embodiments of this application, service reference results corresponding to each first connected path can be pre-set according to the service type corresponding to the first connected path. If each first test service result matches the service reference result corresponding to each first connected path, the test of the target service mesh system can be determined to have passed. Conversely, if there is a mismatch between the first test service result and the service reference result, the test of the target service mesh system can be determined to have failed. In this way, the test prediction of the target service mesh system is completed.

[0039] In the embodiments of this application, by determining the first connectivity path from the input port to the output port in the target service mesh system, first test service requests corresponding to each first connectivity path are generated. Each first test service request is input into the target service mesh system from the input port corresponding to each first test service request, and the first test service results output by the target service mesh system are obtained. If each first test service result matches the service reference result corresponding to each first connectivity path, the target service mesh system is determined to have passed the test. By using the first test service requests corresponding to each first connectivity path to perform input tests on the first connectivity paths existing in the target service mesh system, and if the test results of each first connectivity path match their respective service reference results, the target service mesh system can be determined to have passed the test. This allows for a more comprehensive testing process for the target service mesh system, avoiding the problem of incomplete testing processes caused by differences in service mesh systems, and improving the accuracy of the test results of the service mesh system to a certain extent.

[0040] Reference Figure 2 , Figure 2 A flowchart illustrating another visual feedback-based robot grasping method for cluttered environments provided in this application embodiment, the method may include: Step 201: Determine the first connected path from the input port to the output port in the target service mesh system.

[0041] In the embodiments of this application, the implementation of this step can be referred to the embodiment of step 101, and will not be repeated here.

[0042] Optionally, step 201 may include the following sub-steps: Sub-step 2011: Determine the first control flow graph of the target service mesh system.

[0043] In the embodiments of this application, when an application is decomposed into a service mesh system, each connection path in the service mesh system corresponds to a specific service function of the application. Therefore, the data flow within the service mesh system can form a control flow graph, and the data flow paths in the control flow graph are the connection paths of the service mesh system. Based on this, the first control flow graph of the target service mesh system can be determined according to its structure.

[0044] Sub-step 2012: Determine the first service request configuration information of each second action node and the second satisfiability modal theoretical condition of each second condition node contained in the first control flow graph.

[0045] In embodiments of this application, the first control flow graph may include two types of nodes: second action nodes and second condition nodes. An action node may contain one or more first service request configuration information entries. This first service request configuration information may include information related to the service functions corresponding to all connected paths flowing through the action node, such as service host, service request header, service security settings, and service input and output ports, used to receive and process service requests related to the aforementioned information. Among adjacent action nodes, the output of the upstream action node is the input of the adjacent downstream action node. This corresponds to the continuous processing of input signals in the source application, where each function can correspond to an action node. A condition node may include satisfiability modulo theories (SMT) conditions, used to determine whether subsequent nodes are suitable for requests flowing through that condition node. Based on this, the first control flow graph can be parsed to determine the first service request configuration information of each second action node and the second satisfiability modulo theory conditions of each second condition node contained in the first control flow graph.

[0046] Sub-step 2013: Based on the first service request configuration information and the second satisfiability model theoretical conditions, determine the first connectivity path from the input port to the output port in the target service mesh system.

[0047] In the embodiments of this application, a first connectivity path from an input port to an output port in the target service mesh system can be determined based on the first service request configuration information and the second satisfiability modulus theoretical condition. In the target service mesh system, the second action node and the second condition node can be spaced apart. Thus, the second satisfiability modulus theoretical condition in the second condition node downstream of the second action node can be used to determine whether a downstream second action node and an upstream second action node can be in the same connectivity path. During the determination, the first service request configuration information can be considered. If the first service request configuration information of the downstream second action node is the same as that of the upstream second action node, and satisfies the second satisfiability modulus theoretical condition of the second condition node, then the two second action nodes and the second condition node can be in the same connectivity path. By traversing all the second action nodes and second condition nodes using the above method, the first connectivity path from the input port to the output port in the target service mesh system can be obtained.

[0048] Optionally, sub-step 2013 may include the following sub-steps: Sub-step A1: Based on the first service request configuration information, determine each input port of the target service mesh system and the corresponding output port of each input port.

[0049] In the embodiments of this application, each input port of the target service mesh system and the corresponding output port of each input port can be determined in the first service request configuration information.

[0050] Sub-step A2: Based on the second satisfiability model theory conditions, determine the third action node and the third condition node between the input port and the corresponding output port, as well as the upstream and downstream relationships corresponding to the third action node and the third condition node.

[0051] In the embodiments of this application, based on the second satisfiability modulus theoretical condition, a third action node between an input port and its corresponding output port can be obtained from the second action nodes, and a third condition node between an input port and its corresponding output port can be obtained from the second condition nodes. Furthermore, since the second satisfiability modulus theoretical condition can be used to determine whether a subsequent node is suitable for a request flowing through that condition node, the upstream and downstream relationships corresponding to the third action node and the third condition node can also be obtained.

[0052] Sub-step A3: Based on the upstream and downstream relationship, connect the input port, the output port corresponding to the input port, the third action node, and the third condition node to obtain the first connected path.

[0053] In embodiments of this application, an input port, its corresponding output port, a third action node, and a third condition node can be connected according to upstream and downstream relationships to obtain a first connected path. The flow direction of this first connected path is consistent with the upstream and downstream relationship, pointing from upstream to downstream and from the input port to the output port.

[0054] In the embodiments of this application, based on the first service request configuration information, each input port of the target service mesh system and the corresponding output port of each input port are determined. Based on the second satisfiability modulus theory, the third action node and the third condition node between the input port and the corresponding output port are determined, as well as the upstream and downstream relationships between the third action node and the third condition node. Based on the upstream and downstream relationships, the input port, the corresponding output port, the third action node, and the third condition node are connected to obtain the first connected path. The upstream and downstream relationships between each third action node and the third condition node can be determined according to the second satisfiability modulus theory. The input port and its corresponding output port can be determined according to the first service request configuration information. Then, the input port, the output port, the third action node, and the third condition node can be connected according to the upstream and downstream relationships to obtain the first connected path. The target mesh system can be comprehensively analyzed according to the first control flow graph corresponding to the target service mesh system, further improving the accuracy and reliability of the first connected path.

[0055] Sub-step A4: Based on the first service request configuration information, determine each input port of the target service mesh system and the corresponding output port of each input port.

[0056] In the embodiments of this application, the implementation of this step can be referred to the embodiment of step A1, and will not be repeated here.

[0057] Sub-step A5: Based on the symbolic execution method, determine the fourth action node and the fourth condition node corresponding to each input port from the second action node and the second condition node, respectively.

[0058] In the embodiments of this application, symbolic execution is a program analysis technique that can obtain input requests to execute specific code regions by analyzing the program. The main goal of symbolic execution is to generate a set of inputs within a given time and explore as many different execution paths of the program as possible using these inputs. Symbolic execution uses symbolic values ​​(rather than concrete values) as program inputs and represents program variables as symbolic expressions. When encountering branch statements, symbolic execution explores each branch and adds branch conditions to the corresponding path constraints. Finally, the solvability of the path constraints is verified by a constraint solver (such as an SMT solver) to determine whether the path is reachable. Symbolic execution can be used to start from each input port. After symbolic execution is completed, the fourth action nodes corresponding to each input port can be obtained from the second action nodes, and the fourth condition nodes corresponding to each input port can be obtained from the second condition nodes.

[0059] Sub-step A6: Based on the symbolic execution method, the input port, the output port corresponding to the input port, the fourth action node, and the fourth condition node are connected in the order of passing through the fourth action node and the fourth condition node to obtain the first connected path.

[0060] In the embodiments of this application, for each input port, the symbolic execution method passes through the fourth action node and the fourth condition node corresponding to that input port in a specific order. Based on this order, the input port, the corresponding output port, the fourth action node, and the fourth condition node can be connected to obtain a first connected path. In this first connected path, the pointers of the input port, the corresponding output port, the fourth action node, and the fourth condition node correspond to the order in which the symbolic execution method passes through these nodes.

[0061] In the embodiments of this application, based on the first service request configuration information, each input port of the target service mesh system and the corresponding output port of each input port are determined. Based on the symbolic execution method, the fourth action node and the fourth condition node corresponding to each input port are determined from the second action node and the second condition node, respectively. Based on the symbolic execution method, the input port, the corresponding output port, the fourth action node and the fourth condition node are connected according to the order of the fourth action node and the fourth condition node to obtain the first connected path. All the first connected paths contained in the target service mesh system can be obtained by the symbolic execution method, which further improves the accuracy and reliability of the first connected path.

[0062] In the embodiments of this application, by determining the first control flow graph of the target service mesh system, determining the first service request configuration information of each second action node and the second satisfiability modulo theoretical condition of each second condition node contained in the first control flow graph, and based on the first service request configuration information and the second satisfiability modulo theoretical condition, determining the first connectivity path from the input port to the output port in the target service mesh system, multiple first connectivity paths contained in the target service mesh system can be obtained by analyzing the control flow graph corresponding to the target service mesh system, thereby improving the matching degree between the first connectivity path and the target service mesh system and improving the accuracy of the first connectivity path.

[0063] Step 202: Generate the first test service request corresponding to each of the first connected paths.

[0064] In the embodiments of this application, the implementation of this step can be referred to the embodiment of step 102, and will not be repeated here.

[0065] Optionally, step 202 may include the following sub-steps: Sub-step 2021: Determine the second service request configuration information corresponding to each of the first connected paths from the action nodes contained in each of the first connected paths.

[0066] In the embodiments of this application, the first connection path may contain multiple action nodes, and each action node may contain second service request configuration information such as service host, service request header, service security settings, and service input and output ports. Therefore, the second service request configuration information corresponding to each first connection path can be determined from the action nodes contained in each first connection path.

[0067] Sub-step 2022: Based on the configuration information of each of the second service requests, generate the first test service requests corresponding to each of the first connectivity paths.

[0068] In the embodiments of this application, first test service requests corresponding to each first connectivity path can be generated based on the second service request configuration information. The information contained in the first test service request is the same as that in the second service request configuration information, but the data types are different. The first test service request can be a string command that can be recognized by the target service mesh system, thereby enabling the service flow corresponding to the first test service request to be started.

[0069] In the embodiments of this application, by determining the second service request configuration information corresponding to each first connected path from the action nodes contained in each first connected path, and generating the first test service request corresponding to each first connected path based on the second service request configuration information, the first test service request corresponding to each first connected path can be generated according to the second service request configuration information contained in the action nodes in each first connected path. Since the information contained in the first test service request is more specific, its corresponding request format also has higher granularity, which can improve the reliability and executability of the first test service request to a certain extent.

[0070] Step 203: Input each of the first test service requests into the target service mesh system from the input port corresponding to each of the first test service requests, and obtain the first test service results output by the target service mesh system.

[0071] In the embodiments of this application, the implementation of this step can be referred to the embodiment of step 103, and will not be repeated here.

[0072] Optionally, step 203 may include the following sub-steps: Sub-step 2031: Determine the first service mesh resource name corresponding to each of the first connected paths.

[0073] In the embodiments of this application, the service mesh resource name can be the name of a service function in the target service mesh system. This service mesh resource name can correspond to a specific service function in the target service mesh system. Since each first connected path corresponds to a service function, each first connected path can also correspond to a first service mesh resource name. The first service mesh resource name can be included in the service request header of the action node in the first connected path. In the code, it can be represented as a function name.

[0074] Sub-step 2032: Match the first service mesh resource name with the standard resource name to obtain the first matching result.

[0075] In the embodiments of this application, the first service mesh resource name can be matched with a standard resource name to obtain a first matching result. The standard resource name can be preset manually. The first matching result can include exact matching and regular expression matching. Exact matching means that the first service mesh name is the same as the standard resource name, while regular expression matching means that the first service mesh name is different from the standard resource name and satisfies the regular expression corresponding to the standard resource name. The regular expression corresponding to the standard resource name can be preset manually.

[0076] Sub-step 2033: Based on the first matching result, determine the test priority of each of the first connected paths.

[0077] In embodiments of this application, the testing priority of each first connected path can be determined based on the first matching result. The testing priority can correspond to the order in which the first connected paths are tested, with higher priority paths being tested first.

[0078] Optionally, sub-step 2033 may include the following sub-steps: Sub-step B1: If the first matching result is an exact match, determine the test priority of the first connected path as the first priority.

[0079] In the embodiments of this application, if the first matching result is an exact match, the test priority of the first connected path corresponding to the first matching result can be determined as the first priority.

[0080] Sub-step B2: If the first matching result is a regular expression match, determine the test priority of the first connected path as the second priority; wherein the second priority is lower than the first priority.

[0081] In the embodiments of this application, if the first matching result is a regular expression match, the test priority of the first connected path corresponding to the first matching result can be determined to be the second priority. The second priority may be lower than the first priority.

[0082] Sub-step B3: If, in the first connected path, there are at least two second connected paths where the first matching result is either an exact match or a regular expression match, and the standard resource names corresponding to the second connected paths are the same, then determine the creation time of the second connected path.

[0083] In the embodiments of this application, if there are at least two second connected paths in the first connected path that are both exact matches or regular matches of the first matching result of the standard resource name, and the standard resource names corresponding to the second connected paths are the same, then the creation time of each second connected path can be further determined.

[0084] Sub-step B4: Determine the test priority of the second connected path with the earlier creation time as the third priority, and determine the test priority of the second connected path with the later creation time as the fourth priority; wherein, the third priority is higher than the fourth priority.

[0085] In the embodiments of this application, the test priority of the second connected path created earlier can be determined as the third priority, and the priority of the path created later can be determined as the fourth priority, wherein the third priority can be higher than the fourth priority. The third priority and the fourth priority are the next level of priority after the first priority or the second priority.

[0086] For example, if there exists a third connected path with first and fourth priorities, and a fourth connected path with second and third priorities, then during testing, the third connected path is tested first, followed by the fourth connected path.

[0087] In the embodiments of this application, when the first matching result is an exact match, the test priority of the first connected path is determined to be the first priority; when the first matching result is a regular expression match, the test priority of the first connected path is determined to be the second priority. The second priority is lower than the first priority. If, in the first connected path, there are at least two second connected paths where both first matching results are exact matches or both are regular expression matches, and the standard resource names corresponding to the second connected paths are the same, the creation time of the second connected path is determined. The second connected path with the earlier creation time is determined to have the third priority, and the second connected path with the later creation time is determined to have the fourth priority. The third priority is higher than the fourth priority. Dividing the test priority of each first connected path according to the order of the first matching results and creation times can make the test results more organized and logical, making it easier for testers to debug based on the test results, and further improving the convenience and operability of the target service mesh system.

[0088] Sub-step 2034: According to the test priority from high to low, each of the first test service requests is input into the target service mesh system from the input port corresponding to each of the first test service requests, so as to obtain the first test service results output by the target service mesh system.

[0089] In the embodiments of this application, each first test service request can be input into the target service mesh system from the input port corresponding to each first test service request in descending order of test priority, so as to obtain the results of each first test service output by the target service mesh system.

[0090] In the embodiments of this application, by determining the first service mesh resource name corresponding to each first connected path, matching the first service mesh resource name with the standard resource name to obtain a first matching result, and determining the test priority of each first connected path based on the first matching result, and inputting each first test service request into the target service mesh system from the input port corresponding to each first test service request in descending order of test priority, the target service mesh system outputs each first test service result. The first connected paths can be tested according to the test priority of the first connected paths, so that the first test service results have a high degree of orderliness and logic, which makes it easier for testers to debug the target service mesh system based on the test results, and can improve the convenience and operability of the target service mesh system to a certain extent.

[0091] Step 204: Based on the first satisfiability modulo theoretical conditions corresponding to each first condition node in each first connected path, determine the first input message information corresponding to each first connected path.

[0092] In the embodiments of this application, symbolic execution can be used to generate first input message information corresponding to each first connected path based on the first satisfiability modulo theoretical conditions corresponding to each first condition node in the first connected path. The content of this first input message information is the same as the first service request configuration information, but the data format is different. The input message can be in text format. The first service request configuration information can be in the form of a registry.

[0093] Step 205: Based on the message modification actions corresponding to each action node in each of the first connected paths, determine the first modification function corresponding to each of the first connected paths.

[0094] In the embodiments of this application, a message modification action can refer to the data processing action of an action node, that is, the process of converting an input signal into an output signal. The modification function can be obtained based on the correlation between the input and output signals. For example, if the input signal is a character and the output signal is a number, then the modification function can be an encoding function. Based on this, a symbolic execution method can be used to determine the first modification function corresponding to each first connected path based on the message modification actions corresponding to each action node in the first connected path. A first connected path can contain multiple modification functions, and these modification functions can be sequentially combined to implement the service of the first connected path.

[0095] Step 206: Based on the first input message information and the first modification function, determine the service reference results corresponding to each of the first connected paths.

[0096] In the embodiments of this application, the symbolic execution method can deduce the service reference results corresponding to each first connected path based on the first input message information and the first modification function.

[0097] In the embodiments of this application, by determining the first input message information corresponding to each first connected path based on the first satisfiability modulo theoretical conditions corresponding to each first condition node in each first connected path, and by determining the first modification function corresponding to each first connected path based on the message modification actions corresponding to each action node in each first connected path, and by determining the service reference result corresponding to each first connected path based on the first input message information and the first modification function, the service reference result corresponding to each first connected path can be obtained by symbolic execution method according to the first input message information and the first modification function, which improves the accuracy and reliability of the service reference result to a certain extent.

[0098] Step 207: If each of the first test service results matches the service reference results corresponding to each of the first connected paths, the target service mesh system is determined to have passed the test.

[0099] In the embodiments of this application, if each first test service result matches the service reference result corresponding to each first connected path, then the target service mesh system test can be determined to be passed.

[0100] Reference Figure 3 , Figure 3 This application provides a logic block diagram of a visual feedback-based robotic grasping device for cluttered environments. The device 300 may include: The first determining module 301 is used to determine the first connected path from the input port to the output port in the target service mesh system; The generation module 302 is used to generate the first test service requests corresponding to each of the first connected paths; The input module 303 is used to input each of the first test service requests from the input ports corresponding to each of the first test service requests into the target service mesh system, and to obtain the results of each of the first test services output by the target service mesh system; The second determining module 304 is used to determine that the target service mesh system has passed the test if the results of each first test service match the service reference results corresponding to each of the first connected paths.

[0101] Optionally, the device 300 further includes: The third determining module is used to determine the first input message information corresponding to each of the first connected paths based on the first satisfiability modulo theoretical conditions corresponding to each of the first condition nodes in each of the first connected paths. The fourth determining module is used to determine the first modification function corresponding to each of the first connected paths based on the message modification actions corresponding to each action node in each of the first connected paths. The fifth determining module is used to determine the service reference result corresponding to each of the first connected paths based on the first input message information and the first modification function.

[0102] Optionally, the first determining module 301 includes: The first determination submodule is used to determine the first control flow graph of the target service mesh system; The second determining submodule is used to determine the first service request configuration information of each second action node and the second satisfiability model theoretical condition of each second condition node contained in the first control flow graph. The third determining submodule is used to determine the first connectivity path from the input port to the output port in the target service mesh system based on the first service request configuration information and the second satisfiability model theoretical conditions.

[0103] Optionally, the third determining submodule includes: The first determining unit is configured to determine, based on the first service request configuration information, each input port of the target service mesh system and the corresponding output port of each input port; The second determining unit is used to determine, based on the second satisfiability modulus theoretical conditions, the third action node and the third condition node between the input port and the output port corresponding to the input port, as well as the upstream and downstream relationships corresponding to the third action node and the third condition node respectively; The first connection unit is used to connect the input port, the output port corresponding to the input port, the third action node and the third condition node based on the upstream and downstream relationship to obtain a first connection path.

[0104] Optionally, the third determining submodule includes: The fourth determining unit is used to determine, based on the first service request configuration information, each input port of the target service mesh system and the corresponding output port of each input port; The fifth determining unit is used to determine the fourth action node and the fourth condition node corresponding to each input port from the second action node and the second condition node, respectively, based on the symbolic execution method. The second connection unit is used to connect the input port, the output port corresponding to the input port, the fourth action node, and the fourth condition node based on the order in which the symbolic execution method passes through the fourth action node and the fourth condition node, to obtain a first connected path.

[0105] Optionally, the generation module 302 includes: The fourth determining submodule is used to determine the second service request configuration information corresponding to each of the first connected paths from the action nodes contained in each of the first connected paths. The generation submodule is used to generate the first test service request corresponding to each of the first connectivity paths based on the configuration information of each of the second service requests.

[0106] Optionally, the input module 303 includes: The fifth determining submodule is used to determine the name of the first service mesh resource corresponding to each of the first connected paths; The matching submodule is used to match the first service mesh resource name with the standard resource name to obtain the first matching result; The sixth determining submodule is used to determine the test priority of each of the first connected paths based on the first matching result; The input submodule is used to input each of the first test service requests into the target service mesh system from the input port corresponding to each of the first test service requests in descending order of test priority, so as to obtain the first test service results output by the target service mesh system.

[0107] Optionally, the sixth determining submodule includes: The sixth determining unit is used to determine the test priority of the first connected path as the first priority when the first matching result is an exact match; The seventh determining unit is used to determine the test priority of the first connected path as the second priority when the first matching result is a regular expression matching; wherein the second priority is lower than the first priority.

[0108] Optionally, the device 300 further includes: The sixth determining module is used to determine the creation time of the second connected path when there are at least two second connected paths in the first connected path where the first matching result is either an exact match or a regular expression match, and the standard resource names corresponding to the second connected paths are the same. The seventh determining module is used to determine the test priority of the second connected path with the earlier creation time as the third priority, and to determine the test priority of the second connected path with the later creation time as the fourth priority; wherein the third priority is higher than the fourth priority.

[0109] The visual feedback-based robotic grasping device for cluttered environments in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a GPU box, mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the specific implementation.

[0110] The visual feedback-based robotic grasping device for cluttered environments in this application embodiment can be a device with an operating system. This operating system can be Android, Linux, Windows, or other possible operating systems; this application embodiment does not specifically limit it.

[0111] The visual feedback-based robotic grasping device for cluttered environments provided in this application embodiment can achieve... Figures 1 to 2 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0112] This application provides an electronic device, see [link to relevant documentation] Figure 4 The electronic device 40 includes a processor 401, a memory 402, and a computer program 4021 stored in the memory 402 and executable on the processor 401. When the processor 401 executes the program, it implements the visual feedback-based robot grasping method for cluttered environments described in the foregoing embodiments.

[0113] This application also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps in the visual feedback-based robot grasping method for cluttered environments disclosed in this application.

[0114] This application also provides a computer program product that, when run on an electronic device, enables a processor to execute the steps of the visual feedback-based robot grasping method for cluttered environments disclosed in this application.

[0115] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0116] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices, and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0117] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0118] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0119] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0120] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0121] The above provides a detailed description of a visual feedback-based robot grasping method for cluttered environments. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A robot grasping method in a cluttered environment based on visual feedback, characterized in that: The method comprises: Acquire a target point cloud image of a target cluttered scene; Inputting the target point cloud image into a trained GraspNet model to obtain a plurality of first robot grasping postures output by the GraspNet model; Performing a simulated grasping test on the plurality of first robot grasping postures to obtain a second robot grasping posture that passes the test; Based on the grasping posture of the second robot, the target robot is controlled to grasp the object in the target cluttered scene to obtain a first grasping result.

2. The method according to claim 1, characterized in that The step of performing a simulated grasping test on the grasping postures of the plurality of first robots to obtain grasping postures of the second robot that pass the test comprises: Inputting the target point cloud image into a trained GraspNet model to obtain first posture scores corresponding to each of the first robot grasping postures output by the GraspNet model; A simulated grasping test is performed on the plurality of first robot grasping postures in a descending order of the first posture scores, and a first robot grasping posture that passes the first test is determined as the second robot grasping posture.

3. The method according to claim 1, characterized in that The step of performing a simulated grasping test on the grasping postures of the plurality of first robots to obtain grasping postures of the second robot that pass the test comprises: In the simulation scenario, the gripper of the target robot is controlled to move to a first target position corresponding to the grasping posture of the first robot; wherein the reference center point of the first target position is the K-means clustering center point of the target point cloud image; Moving the clamping jaw upward in a vertical direction by a first preset distance; Controlling the clamping jaw to rotate with the vertical direction as the rotation axis; In a case where the gripper does not collide with the first target grasped object corresponding to the first target position, the first robot grasping posture is determined as a second robot grasping posture that has passed the test.

4. The method according to claim 1, characterized in that: The step of performing a simulated grasping test on the grasping postures of the plurality of first robots to obtain grasping postures of the second robot that pass the test comprises: In the simulation scenario, controlling the gripper of the target robot to move to a first target position corresponding to the grasping posture of the first robot; When there is an overlap between the base of the gripper and the object point cloud image corresponding to the first target position, the gripper is moved upward in a vertical direction to obtain a third robot grasping posture; When the gripper is in the third robot grasping posture and there is no overlapping portion between the base of the gripper and the object point cloud image, the third robot grasping posture is determined to be the second robot grasping posture that has passed the test.

5. The method according to claim 3, characterized in that: The determining, based on the first service request configuration information and the second satisfiability model theoretical condition, a first connectivity path from an input port to an output port in the target service grid system includes: Determine, based on the first service request configuration information, each input port of the target service grid system and each output port corresponding to each input port; Based on the symbolic execution method, determining the fourth action node and the fourth condition node corresponding to each of the input ports from the second action node and the second condition node respectively; Based on the symbolic execution method, the input port, the output port corresponding to the input port, the fourth action node and the fourth condition node are connected in sequence to obtain a first connected path.

6. The method according to claim 1, characterized in that The generating of first test service requests corresponding to the first communication paths respectively includes: Determining second service request configuration information corresponding to each of the first communication paths from action nodes respectively included in each of the first communication paths; Based on each of the second service request configuration information, a first test service request corresponding to each of the first communication paths is generated.

7. The method according to claim 1, characterized in that The step of inputting each of the first test service requests into the target service grid system from an input port corresponding to each of the first test service requests, and obtaining each of the first test service results output by the target service grid system, includes: Determine a first service grid resource name corresponding to each of the first connectivity paths; Matching the first service grid resource name with the standard resource name to obtain a first matching result; Determining a test priority of each of the first connectivity paths based on the first matching result; In the order of the test priority from high to low, each of the first test service requests is input into the target service grid system from the input port corresponding to each of the first test service requests, so as to obtain each of the first test service results output by the target service grid system.

8. The method according to claim 7, characterized in that The determining, based on the first matching result, a test priority of each of the first connectivity paths includes: When the first matching result is an exact match, determining the test priority of the first connection path to be a first priority; When the first matching result is a regular match, the test priority of the first connection path is determined to be a second priority; wherein the second priority is lower than the first priority.

9. The method according to claim 8, characterized in that The method further comprises: In the first communication path, if there are at least two second communication paths whose first matching results are both exact matches or both regular matches, and the standard resource names corresponding to the second communication paths are the same, determining the creation time of the second communication paths; The test priority of the second communication path created earlier is determined to be the third priority, and the test priority of the second communication path created later is determined to be the fourth priority; wherein the third priority is higher than the fourth priority.

10. A robot grasping device for a cluttered environment based on visual feedback, characterized in that: The device comprises: A first determination module, configured to determine a first connectivity path from an input port to an output port in a target service grid system; A generating module, configured to generate first test service requests corresponding to each of the first connecting paths; An input module, used to input each of the first test service requests into the target service grid system from an input port corresponding to each of the first test service requests, to obtain each of the first test service results output by the target service grid system; The second determination module is used to determine that the target service grid system test has passed when each of the first test service results matches the service reference results respectively corresponding to each of the first connection paths.

Images