System and method for collaborative operation of humanoid robot and robot dog

Abstract

The invention discloses a collaborative operation system and method for a humanoid robot and a robot dog. The system comprises a server, the humanoid robot and the robot dog. The server, the humanoid robot and the robot dog are interconnected through a hybrid communication network; the server receives a task request sent by an external system; creating a task execution instruction of the humanoid robot and a task search instruction of the robot dog; sending a task execution instruction to the humanoid robot, and sending a task search instruction and parameters carried by the task request to the robot dog; the robot dog responds to the task search instruction and carries out traversal search on a target site corresponding to the operation map; sending the searched to-be-processed object information and self operation parameters to the humanoid robot; and the humanoid robot responds to the task execution instruction and performs operation on the target site based on the to-be-processed object information shared by the robot dog. By adopting the embodiment of the invention, the collaborative operation of the humanoid robot and the robot dog can be realized, and the task execution efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of intelligent control technology, and particularly to a collaborative operation system and method for a humanoid robot and a quadruped robot. Background Art

[0002] With the increasingly wide application of humanoid robots and quadruped robots, humanoid robots are known for their high flexibility and operation ability, while quadruped robots are known for their excellent mobility and environmental adaptability. For example, in collaborative operations on factory production lines, humanoid robots can complete tasks such as high-precision component assembly and quality inspection, while quadruped robots can be responsible for material transportation, equipment inspection, etc. The cooperation of the two improves production efficiency. In related technologies, the research on humanoid robots and quadruped robots mainly focuses on the optimization of their respective single performances and the execution ability of single tasks. Although significant achievements have been made in the single performances of humanoid robots and quadruped robots, there is currently no information interaction system and coordination mechanism between humanoid robots and quadruped robots, and the collaborative operation of humanoid robots and quadruped robots cannot be achieved, resulting in low task execution efficiency. Summary of the Invention

[0003] An embodiment of this application provides a collaborative operation system for a humanoid robot and a quadruped robot. To provide a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary part is not a general review, nor is it intended to identify key / important constituent elements or delineate the protection scope of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the subsequent detailed description.

[0004] In a first aspect, an embodiment of this application provides a collaborative operation system for a humanoid robot and a quadruped robot, the system comprising: A server, a humanoid robot, and a quadruped robot; wherein, The server, the humanoid robot, and the quadruped robot are interconnected through a hybrid communication network, and the hybrid communication network is set up using a 5G communication module and a multi-channel LoRa gateway; The server receives a task request sent by an external system, and the task request carries an operation map and task requirements; activates the hybrid communication network; creates a task execution instruction for the humanoid robot and a task search instruction for the quadruped robot; sends the task execution instruction to the humanoid robot through the activated hybrid communication network, and sends the task search instruction, the operation map, and the task requirements to the quadruped robot; The quadruped robot, in response to the task search instruction, traverses and searches the target site corresponding to the operation map in real time according to the task requirements; sends the information of the object to be processed and its own operation parameters found in the search to the humanoid robot through the activated hybrid communication network; The humanoid robot, in response to a task execution instruction, collaborates with the robotic dog to maintain within a preset following distance; when receiving the information of the object to be processed fed back by the robotic dog, it ends the following and caches the information of the object to be processed; based on the cached information of the object to be processed, it performs operations in the target site; and sends the results of the operations to the server and the robotic dog respectively through the activated hybrid communication network.

[0005] In a second aspect, a collaborative operation method for a humanoid robot and a robotic dog, the method includes: Receiving a task request sent by an external system through a server, the task request carrying an operation map and task requirements; activating a hybrid communication network; creating a task execution instruction for the humanoid robot and a task search instruction for the robotic dog; sending the task execution instruction to the humanoid robot through the activated hybrid communication network, and sending the task search instruction, the operation map and the task requirements to the robotic dog; Through the robotic dog, in response to the task search instruction, traversing and searching the target site corresponding to the operation map in real time according to the task requirements; sending the information of the object to be processed and its own operation parameters found to the humanoid robot through the activated hybrid communication network; Through the humanoid robot, in response to the task execution instruction, collaborating with the robotic dog to maintain within a preset following distance; when receiving the information of the object to be processed fed back by the robotic dog, it ends the following and caches the information of the object to be processed; based on the cached information of the object to be processed, it performs operations in the target site; and sends the results of the operations to the server and the robotic dog respectively through the activated hybrid communication network.

[0006] In the embodiments of the present application, on the one hand, the server, the humanoid robot and the robotic dog are interconnected through a hybrid communication network, and the hybrid communication network is set up by using a 5G communication module and a multi-channel LoRa gateway. This interaction system ensures that even in the case of unstable signals, reliable transmission of information can be guaranteed through redundant transmission. This information interaction system provides a guarantee for the humanoid robot and the robotic dog to collaboratively complete complex tasks. On the other hand, the robotic dog traverses and searches the target site in real time according to the task requirements, and sends the information of the object to be processed and its own operation parameters to the humanoid robot. The humanoid robot collaborates with the robotic dog to maintain a preset following distance based on this information, and ends the following and performs operations when receiving the information of the object to be processed. The robotic dog quickly searches for the target with its excellent mobility and environmental adaptability, while the humanoid robot performs fine operations with its flexibility and operation ability. This coordination mechanism enables the humanoid robot and the robotic dog to efficiently collaborate in a complex environment, thereby improving the task execution efficiency.

[0007] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. Description of the Drawings

[0008] The accompanying drawings herein are incorporated into and constitute a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0009] Figure 1 It is a schematic structural diagram of a collaborative operation system for a humanoid robot and a quadruped robot provided by an embodiment of the present application; Figure 2 It is a schematic diagram of the model processing process of a pre-trained object detection model provided by an embodiment of the present application; Figure 3 It is a schematic flowchart of a training method for a multi-modal object detection network provided by an embodiment of the present application; Figure 4 It is a schematic flowchart of a collaborative operation method for a humanoid robot and a quadruped robot provided by an embodiment of the present application; Figure 5 It is a schematic structural diagram of a quadruped robot provided by an embodiment of the present application; Figure 6 It is a schematic structural diagram of a humanoid robot provided by an embodiment of the present application; Figure 7 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0010] The following description and the accompanying drawings fully illustrate the specific implementation manners of the present application so that those skilled in the art can practice them.

[0011] It should be clear that the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts belong to the scope of protection of the present application.

[0012] When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all the implementation manners consistent with the present application. On the contrary, they are merely examples of systems and methods consistent with some aspects of the present application as detailed in the appended claims.

[0013] In the description of the present application, it should be understood that terms such as "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to specific circumstances. In addition, in the description of the present application, unless otherwise specified, "a plurality of" means two or more. "And / or" describes the association relationship of associated objects and indicates that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after.

[0014] Currently, the research on humanoid robots and quadruped robots mainly focuses on the optimization of their respective single performances and the execution ability of single tasks.

[0015] The present application realizes that although remarkable achievements have been made in the unilateral performances of humanoid robots and quadruped robots, there is currently no information interaction system and coordination mechanism between humanoid robots and quadruped robots, and the collaborative operation of humanoid robots and quadruped robots cannot be achieved, resulting in low task execution efficiency.

[0016] In the embodiment of the present application, on the one hand, the server, the humanoid robot, and the quadruped robot are interconnected through a hybrid communication network. The hybrid communication network is set up with a 5G communication module and a multi-channel LoRa gateway. This interaction system ensures that even in the case of unstable signals, reliable transmission of information can be guaranteed through redundant transmission. This information interaction system provides a guarantee for the humanoid robot and the quadruped robot to jointly complete complex tasks. On the other hand, the quadruped robot traverses and searches the target site in real time according to the task requirements, and sends the information of the objects to be processed and its own operating parameters found to the humanoid robot. The humanoid robot cooperates with the quadruped robot to maintain a preset following distance according to this information, and ends the following and performs operations when receiving the information of the objects to be processed. The quadruped robot quickly searches for the target with its excellent mobility and environmental adaptability, while the humanoid robot uses its flexibility and operation ability to perform fine operations. This coordination mechanism enables the humanoid robot and the quadruped robot to efficiently cooperate in a complex environment, thereby improving the task execution efficiency. The following will be described in detail with exemplary embodiments.

[0017] Please refer to Figure 1 , Figure 1 which is a schematic structural diagram of a collaborative operation system for a humanoid robot and a quadruped robot provided by an embodiment of the present application. The system includes: a server, a humanoid robot, and a quadruped robot; wherein, the server, the humanoid robot, and the quadruped robot are interconnected through a hybrid communication network. The hybrid communication network is set up with a 5G communication module and a multi-channel LoRa gateway.

[0018] Among them, the server refers to a computer system or software that provides specific services in a computer network. In the collaborative operation system of humanoid robots and quadruped robots, the server is the control center of the entire system. A humanoid robot is a robot with a human-like appearance, possessing high flexibility and operational capabilities. A quadruped robot is a four-legged robot known for its excellent mobility and environmental adaptability. A hybrid communication network refers to a network system that combines multiple communication technologies for efficient data transmission between different devices. A 5G communication module is a high-speed, low-latency wireless communication technology capable of supporting large-scale data transmission. A multi-channel LoRa gateway is a low-power, long-distance wireless communication technology characterized by a wide coverage range and low power consumption. The multi-channel LoRa gateway can support multiple LoRa communication channels simultaneously to provide redundant communication guarantees in complex environments.

[0019] In the embodiments of this application, by combining 5G and LoRa technologies, the hybrid communication network can flexibly switch communication modes under different environmental conditions to ensure stable and reliable data transmission between the humanoid robot and the quadruped robot.

[0020] In some embodiments of this application, the server receives a task request sent by an external system, and the task request carries a job map and task requirements; activates the hybrid communication network; creates a task execution instruction for the humanoid robot and a task search instruction for the quadruped robot; sends the task execution instruction to the humanoid robot through the activated hybrid communication network, and sends the task search instruction, job map, and task requirements to the quadruped robot; the quadruped robot, in response to the task search instruction, traverses and searches the target site corresponding to the job map in real time according to the task requirements; sends the information of the object to be processed and its own operating parameters found during the search to the humanoid robot through the activated hybrid communication network; the humanoid robot, in response to the task execution instruction, cooperates with the quadruped robot to maintain within a preset following distance; when receiving the information of the object to be processed fed back by the quadruped robot, ends the following and caches the information of the object to be processed; based on the cached information of the object to be processed, performs operations at the target site; sends the results of the operations to the server and the quadruped robot respectively through the activated hybrid communication network.

[0021] Among them, the task request refers to the instruction sent by an external system to the server to initiate the collaborative operation of the humanoid robot and the robotic dog. The task request carries the operation map and task requirements. The operation map is the detailed geographical information of the target site, including the layout of the site, the positions of obstacles, key areas, etc. The task requirements refer to the specific task objectives and requirements to be completed, such as searching for trapped people, clearing debris, equipment operation, etc. The task execution instruction is generated by the server according to the task requirements and sent to the humanoid robot. It is used to guide the humanoid robot on how to operate in the target site, including the specific steps of the operation, the objects to be operated, etc. The task search instruction is generated by the server according to the task requirements and specifically sent to the robotic dog. It is used to guide the robotic dog on how to search in the target site, including the search path, search range, etc. The information of the object to be processed refers to the relevant information of the target object discovered by the robotic dog during the search process, such as the position of the trapped person, the status of the damaged equipment, etc. This information will be sent to the humanoid robot through the hybrid communication network for the humanoid robot to perform subsequent operations. The self-running parameters refer to some status information of the robotic dog during operation, such as the current coordinates, heading angle, linear velocity, angular velocity, etc., which are used for the humanoid robot to predict the movement trajectory of the robotic dog and thus maintain the collaborative operation. The preset following distance refers to the fixed distance maintained between the humanoid robot and the robotic dog during the collaborative operation. This distance is preset according to the task requirements and environmental conditions to ensure the coordination and safety of the two during the operation.

[0022] For example, a collaborative operation system for humanoid robots and robotic dogs is used in earthquake rescue scenarios. The rescue command center sends a task request to the server through an external system. The task request carries a map of the ruins area and task requirements (such as searching for trapped people and carrying out rescue operations). After receiving the task request, the server activates a hybrid communication network to ensure stable communication in complex environments. Based on the task requirements, the server creates task execution instructions for the humanoid robot (such as clearing the ruins and performing delicate rescue operations) and task search instructions for the robotic dog (such as searching for trapped people in the ruins). The server sends the task execution instructions to the humanoid robot through the activated hybrid communication network, and at the same time sends the task search instructions, operation map, and task requirements to the robotic dog. After receiving the task search instructions, the robotic dog traverses and searches the ruins area in real time according to the operation map and task requirements. It uses its own sensor array (such as cameras and lidar) to collect environmental data and identify the positions of trapped people. During the search process, the robotic dog sends the positions of the trapped people found (information on objects to be processed) and its own operating parameters (such as current coordinates, heading angle, linear velocity, and angular velocity) to the humanoid robot through the hybrid communication network. After receiving the task execution instructions, the humanoid robot cooperates with the robotic dog and maintains a preset following distance. The humanoid robot predicts the movement trajectory of the robotic dog by receiving its operating parameters and controls its own movement to maintain the cooperative state. When the humanoid robot receives the position information of the trapped people fed back by the robotic dog, it ends the following and caches the information. The humanoid robot goes to the location of the trapped people according to the cached information and performs delicate rescue operations such as clearing the ruins and carrying the trapped people. After completing the rescue operation, the humanoid robot sends the operation results (such as the trapped people have been successfully rescued and the situation of the ruins cleared) back to the server and the robotic dog respectively through the hybrid communication network. The server receives the feedback information from the humanoid robot and the robotic dog in real time, monitors the progress of task execution, and feeds it back to the external system. The client interface of the external system is, for example Figure 2 as shown. If new situations are discovered or the task strategy needs to be adjusted, the server can dynamically adjust the task instructions and resend them.

[0023] In some embodiments of the present application, the server, humanoid robot, and robotic dog are provided with 5G communication modules and multi-channel LoRa gateways.

[0024] Specifically, the specific process of activating the hybrid communication network includes: determining the bandwidth requirements between the humanoid robot and the robotic dog according to the task requirements; configuring a quality of service strategy for preset video bandwidth transmission for the 5G communication module according to the bandwidth requirements; and configuring LoRa channel parameters for the multi-channel LoRa gateway according to the bandwidth requirements.

[0025] Among them, the bandwidth requirement refers to the minimum network bandwidth required for data transmission between a humanoid robot and a quadruped robot in a specific task scenario. The Quality of Service (QoS) policy is a network technology used to optimize network resource allocation and ensure the performance of critical applications such as real-time video transmission. LoRa channel parameters include channel frequency, spreading factor (SF), bandwidth (BW), etc. These parameters determine the transmission distance, data rate, and power consumption of LoRa communication.

[0026] In the embodiments of this application, by dynamically adjusting the bandwidth requirement between the humanoid robot and the quadruped robot according to the task requirements and optimizing the configuration of the 5G communication module and the multi-channel LoRa gateway, the communication efficiency and reliability of collaborative operations can be significantly improved. This flexible communication configuration method can ensure the stability and real-time performance of data transmission in complex environments, thereby enhancing the overall performance and task success rate of collaborative operations.

[0027] For example, the server configures the QoS policy for the 5G communication module according to the bandwidth requirement. Assign a high priority to the high-definition video stream to ensure its stable transmission in a high-load network environment. Configure the bandwidth resources of the 5G module and reserve sufficient resource blocks (RBs) to support high-definition video transmission.

[0028] For example, configure the channel parameters of the LoRa gateway according to the bandwidth requirement. Select a suitable channel frequency (such as 470 MHz), spreading factor (such as SF7, suitable for short-distance high-speed data transmission), and bandwidth (such as 125 kHz). Set the management parameters of the LoRa gateway, including the address and port of the LoRa Server and the power parameters of the radio frequency interface, to ensure the stability and reliability of LoRa communication.

[0029] Among them, by dynamically configuring the 5G and LoRa communication parameters, the humanoid robot and the quadruped robot can efficiently collaborate to complete inspection and maintenance tasks. The stable transmission of high-definition video ensures the real-time performance of remote monitoring, while the low-power and long-distance transmission characteristics of LoRa guarantee the reliable transmission of sensor data. This flexible communication configuration method significantly improves the efficiency and reliability of equipment inspection and maintenance in industrial parks.

[0030] Specifically, according to the task requirements, the specific process of determining the bandwidth requirements between the humanoid robot and the robot dog includes: extracting the task type from the task requirements; according to the task type, searching for the corresponding data flow entries from the pre-established mapping relationship between the task type and the task data flow to obtain the basic data flow set; querying the unit bandwidth coefficients of each basic data flow in the basic data flow set; combining each basic data flow with its unit bandwidth coefficient to obtain the original bandwidth requirement matrix; reading the environmental monitoring data in the target site and determining the bandwidth correction coefficient according to the environmental monitoring data; multiplying the original bandwidth requirement matrix by the bandwidth correction coefficient to obtain the compensated final bandwidth requirement value, which is used as the bandwidth requirement between the humanoid robot and the robot dog.

[0031] For example, the task types extracted from the task requirements are inspection and equipment maintenance. According to the task types (inspection and equipment maintenance), search for the corresponding data flow entries from the pre-established mapping relationship between the task type and the task data flow. For example: high-definition video stream: 8Mbps, sensor data: 100kbps, control instruction: 50kbps, to obtain the basic data flow set: {high-definition video stream 8Mbps, sensor data 100kbps, control instruction 50kbps}. Query the unit bandwidth coefficients of each basic data flow in the basic data flow set. The unit bandwidth coefficients are: 1.2 for the high-definition video stream (considering video compression and transmission redundancy), 1.1 for the sensor data (considering data packet headers and checksums), and 1.0 for the control instruction (simple data transmission). Based on the above parameters, the original bandwidth requirement matrix can be combined, and this matrix is shown in Table 1 for example.

[0032] Table 1

[0033] Then read the environmental monitoring data of the target site (industrial park), including signal strength, interference situation, etc., and determine the bandwidth correction coefficient according to the environmental monitoring data. Assume that the signal strength is good, but there is slight interference, and the bandwidth correction coefficient is 1.1.

[0034] Finally, multiply the original bandwidth requirement matrix by the bandwidth correction coefficient to obtain the compensated final bandwidth requirement value, as shown in Table 2 for example.

[0035] Table 2

[0036] It should be noted that the pre-established mapping relationship between the task type and the task data flow and the unit bandwidth coefficient are pre-set based on experience.

[0037] In some embodiments of the present application, the specific process of sending the searched information of the object to be processed and its own operating parameters to the humanoid robot through the activated hybrid communication network includes: real-time monitoring of the 5G signal strength between the humanoid robot; when the 5G signal strength is greater than or equal to a preset threshold, sending the searched information of the object to be processed to the humanoid robot through the preset quality of service strategy in the 5G communication module; or, when the 5G signal strength is less than the preset threshold, switching to LoRa redundant transmission and sending the searched information of the object to be processed to the humanoid robot through the preset LoRa channel parameters in the multi-channel LoRa gateway.

[0038] In some embodiments of the present application, the specific process of cooperating with the robot dog to maintain within a preset following distance includes: receiving the own operating parameters sent by the robot dog through the activated hybrid communication network, where the own operating parameters include the current coordinates of the robot dog in the global coordinate system, the current heading angle of the robot dog, the current linear velocity of the robot dog, and the current angular velocity of the robot dog; predicting the predicted coordinates of the robot dog at the next moment according to the preset robot dog kinematic model and the current coordinates of the robot dog, the current heading angle of the robot dog, the current linear velocity of the robot dog, and the current angular velocity of the robot dog; controlling its own movement according to the predicted coordinates of the robot dog at the next moment to cooperate with the robot dog to maintain within a preset following distance; where the preset robot dog kinematic model for maintaining cooperation with the robot dog is:

[0039]

[0040] where, , is the predicted coordinate of the robot dog at the next moment ; , is the current coordinate of the robot dog at the current moment ; is the current linear velocity of the robot dog at the current moment ; is the current heading angle of the robot dog at the current moment ; is the current angular velocity of the robot dog at the current moment ; is the control period.

[0041] In some embodiments of the present application, the specific process of controlling its own movement according to the predicted coordinates of the robotic dog at the next moment includes: calculating the position where the humanoid robot needs to arrive through a preset following distance and the predicted coordinates at the next moment; obtaining the current position of the humanoid robot; calculating the actual distance deviation and actual direction deviation between the position where the robot needs to arrive and the current position of the robot; generating a speed control instruction based on the actual distance deviation and actual direction deviation; executing the speed control instruction to control its own movement to the position where it needs to arrive; where the calculation formula for the position where the robot needs to arrive is:

[0042] Wherein, is the position where the robot needs to arrive, is the coordinate and heading angle of the robotic dog in the global coordinate system, is the preset following distance, is the current heading angle of the robotic dog at the current moment of the robotic dog, is the preset azimuth offset angle.

[0043] In some embodiments of the present application, the specific process of traversing and searching the target site corresponding to the operation map in real time according to the task requirements includes: planning a movement path based on the operation map in combination with the SLAM algorithm, and sending the movement path to the humanoid robot through the activated hybrid communication network; autonomously moving according to the movement path and turning on the sensor array; during the movement, collecting environmental data in real time through the multi-modal sensor; inputting the task requirements and environmental data into a pre-trained target detection model, and outputting the objects to be processed that meet the task requirements; creating a description information of the objects to be processed to obtain the information of the objects to be processed searched.

[0044] Wherein, SLAM (Simultaneous Localization and Mapping) is a technology that enables a robot to build a map in real time and determine its own position in an unknown environment. The movement path refers to the planned route of the robot from the starting point to the target point, usually generated by the SLAM algorithm. The sensor array refers to the collection of various sensors carried by the robot, including visual sensors (cameras), lidar, environmental sensors (temperature, humidity, gas sensors), etc. The multi-modal sensor refers to a combination of sensors that can collect different types of data, such as image data, point cloud data, environmental data, etc. The target detection model is an algorithm model based on deep learning, used to identify and locate target objects in image or point cloud data. The objects to be processed refer to the target objects that need to be processed by the robot during the task execution, such as trapped persons, equipment to be repaired, etc.

[0045] For example, based on the operation map (map of the industrial park) and task requirements (inspecting equipment), the server uses the SLAM algorithm to plan an optimal path for the robotic dog from the starting point to the target equipment. The planned path is sent to the humanoid robot through a hybrid communication network (5G and LoRa). The robotic dog moves autonomously along the planned path and activates the sensor array (such as cameras and lidar). During the movement, environmental data is collected in real time. The collected environmental data (such as images and point clouds) and task requirements are input into a pre-trained object detection model. The model identifies the location and status of the target equipment and creates descriptive information. The robotic dog sends the identified target equipment information to the humanoid robot, and the humanoid robot goes to the target location for maintenance operations according to this information.

[0046] For example Figure 2 as shown Figure 2 is a schematic diagram of the model processing process of a pre-trained object detection model provided by this application. The pre-trained object detection model includes a feature extraction module, an encoder, and a decoder. The robotic dog collects environmental data in real time through multi-modal sensors. This environmental data includes visual data, point cloud data, and lidar data. The feature extraction module provided by the pre-trained object detection model and a preset 3D reference point are used to extract features from the visual data, point cloud data, and lidar data, obtaining image features, point cloud features, and radar point features. The extracted image features, point cloud features, and radar point features are input into the encoder of the pre-trained object detection model for position encoding and feature sampling, obtaining an encoded result. Finally, the encoded result is input into the transformer decoder to output the information of the object to be processed searched. Among them, the encoder specifically includes a first-stage detection network and a second-stage detection network.

[0047] Specifically, the specific process of generating the pre-trained object detection model includes: collecting the historical environmental data fed back by the sensor array carried by the robotic dog when performing tasks in each historical site; using the category library defined in advance through task requirements to frame the target location in the historical environmental data and associate category labels, obtaining a labeled structured data set; constructing a multi-modal feature vector according to the labeled structured data set; constructing a multi-modal object detection network. The multi-modal object detection network includes a first-stage detection network and a second-stage detection network. The first-stage detection network is used to generate candidate regions, and the second-stage detection network is used to classify and regress the target location; inputting the multi-modal feature vector into the multi-modal object detection network for machine learning and outputting a model loss value; when the model loss value reaches the minimum, obtaining the pre-trained object detection model.

[0048] Among them, the labeled structured data set includes labeled image data, labeled point cloud data, and labeled environmental data.

[0049] Specifically, the specific process of constructing the multi-modal feature vector based on the labeled structured data set includes: using a pre-trained convolutional neural network to extract the image features of the labeled image data; using the PointNet network to extract the point cloud features of the labeled point cloud data; using the LSTM network to extract the environmental features of the labeled environmental data; fusing the image features, point cloud features, and environmental features to obtain a fused multi-modal feature vector; where the loss function of the multi-modal object detection network is:

[0050] Wherein, is the loss value, is the th vector of the image features of the labeled image data, is the th vector of the point cloud features of the labeled point cloud data, is the th vector of the environmental features of the labeled environmental data, is the number of the labeled structured data set.

[0051] In the embodiment of the present application, on the one hand, the server, the humanoid robot, and the robot dog are interconnected through a hybrid communication network. The hybrid communication network is set up with a 5G communication module and a multi-channel LoRa gateway. This interaction system ensures that even in the case of unstable signals, reliable transmission of information can be guaranteed through redundant transmission. This information interaction system provides a guarantee for the humanoid robot and the robot dog to cooperate to complete complex tasks. On the other hand, the robot dog traverses and searches the target site in real time according to the task requirements, and sends the information of the to-be-processed object and its own operating parameters searched to the humanoid robot. The humanoid robot cooperates with the robot dog to maintain a preset following distance according to this information, and ends following and performs operations when receiving the information of the to-be-processed object. The robot dog quickly searches for the target with its excellent mobility and environmental adaptability, while the humanoid robot uses its flexibility and operation ability to perform fine operations. This coordination mechanism enables the humanoid robot and the robot dog to efficiently cooperate in a complex environment, thereby improving the task execution efficiency.

[0052] Please refer to Figure 3 , which shows a schematic flowchart of a training method for a multi-modal object detection network provided by an embodiment of the present application. As Figure 3 shown, this process includes the following steps: S201, collecting historical environmental data fed back by the sensor array carried by the robot dog when performing tasks in each historical site; S202, use the category library defined in advance through task requirements to frame the target location in the historical environmental data and associate category labels, so as to obtain the labeled structured data set; S203, construct multi-modal feature vectors according to the labeled structured data set; S204, construct a multi-modal object detection network, which includes a first-stage detection network and a second-stage detection network. The first-stage detection network is used to generate candidate regions, and the second-stage detection network is used to classify and regress the target location; S205, input the multi-modal feature vectors into the multi-modal object detection network for machine learning, and output the model loss value; S206, when the model loss value reaches the minimum, obtain the pre-trained object detection model.

[0053] In the embodiments of the present application, by collecting and annotating historical environmental data, constructing multi-modal feature vectors, and using a staged detection network for training, the accuracy and robustness of object detection can be effectively improved. This method makes full use of various sensor data, enhances the adaptability of the model to complex environments, and at the same time ensures the efficiency and reliability of the model in actual tasks by optimizing the model loss value.

[0054] Please refer to Figure 4 , which provides a schematic flowchart of a collaborative operation method for a humanoid robot and a robotic dog in the embodiments of the present application. As Figure 4 shown, the detection method in the embodiments of the present application may include the following steps: S101, receive a task request sent by an external system through the server. The task request carries a job map and task requirements; activate the hybrid communication network; create a task execution instruction for the humanoid robot and a task search instruction for the robotic dog; send the task execution instruction to the humanoid robot through the activated hybrid communication network, and send the task search instruction, job map, and task requirements to the robotic dog; S102, through the robotic dog in response to the task search instruction, traverse and search the target site corresponding to the job map in real time according to the task requirements; send the information of the object to be processed and its own operating parameters found to the humanoid robot through the activated hybrid communication network; S103, through the humanoid robot in response to the task execution instruction, cooperate with the robotic dog to maintain within a preset following distance; when receiving the information of the object to be processed fed back by the robotic dog, end the following and cache the information of the object to be processed; perform operations on the target site based on the cached information of the object to be processed; send the results of the operations to the server and the robotic dog respectively through the activated hybrid communication network.

[0055] In the embodiment of the present application, on the one hand, the service end, the humanoid robot and the robot dog are interconnected through a hybrid communication network, which is set up with a 5G communication module and a multi-channel LoRa gateway. The interactive system ensures that even in the case of unstable signals, the reliable transmission of information can be guaranteed through redundant transmission. The information interaction system provides a guarantee for the humanoid robot and the robot dog to collaborate in completing complex tasks. On the other hand, the robot dog traverses and searches the target site in real time according to the task requirements, and sends the searched information of the object to be processed and its own operating parameters to the humanoid robot. The humanoid robot cooperates with the robot dog to maintain a preset following distance based on this information, and ends the following and performs the operation when receiving the information of the object to be processed. The robot dog relies on its excellent mobility and environmental adaptability to quickly search for the target, while the humanoid robot uses its flexibility and operational ability to perform fine operations. This coordination mechanism enables the humanoid robot and the robot dog to work efficiently and collaboratively in a complex environment, thereby improving the efficiency of task execution.

[0056] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0057] See also Figure 5 , which shows a schematic diagram of the structure of a robot dog provided by an exemplary embodiment of the present application. The robot dog can be implemented as all or part of an electronic device through software, hardware or a combination of both. The robot dog 1 includes a traversal search module 10 and an information sending module 20.

[0058] The traversal search module 10 is used to respond to the task search instruction and perform a traversal search on the target site corresponding to the operation map in real time according to the task requirements; The information sending module 20 is used to send the searched information of the object to be processed and its own operating parameters to the humanoid robot through the activated hybrid communication network.

[0059] It should be noted that the collaborative operation device of the humanoid robot and the robot dog provided in the above embodiment only uses the division of the above functional modules as an example when executing the collaborative operation method of the humanoid robot and the robot dog. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the collaborative operation device of the humanoid robot and the robot dog provided in the above embodiment belongs to the same concept as the collaborative operation method embodiment of the humanoid robot and the robot dog. The implementation process is detailed in the method embodiment, which will not be repeated here.

[0060] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0061] In an embodiment of the present application, on the one hand, the server, the humanoid robot, and the robot dog are interconnected through a hybrid communication network. The hybrid communication network is set up using a 5G communication module and a multi-channel LoRa gateway. This interaction system ensures that even in the case of unstable signals, reliable transmission of information can be guaranteed through redundant transmission. This information interaction system provides a guarantee for the humanoid robot and the robot dog to cooperate to complete complex tasks. On the other hand, the robot dog traverses and searches the target site in real time according to the task requirements, and sends the information of the object to be processed and its own operating parameters found during the search to the humanoid robot. The humanoid robot cooperates with the robot dog based on this information to maintain a preset following distance, and ends the following and performs operations when receiving the information of the object to be processed. The robot dog quickly searches for the target with its excellent mobility and environmental adaptability, while the humanoid robot uses its flexibility and operating ability to perform fine operations. This coordination mechanism enables the humanoid robot and the robot dog to cooperate efficiently in a complex environment, thereby improving the task execution efficiency.

[0062] The following is an embodiment of the device of the present application, which can be used to execute the method embodiment of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the method embodiment of the present application.

[0063] Please refer to Figure 6 , which shows a schematic structural diagram of a humanoid robot provided by an exemplary embodiment of the present application. This humanoid robot can be implemented as all or part of an electronic device through software, hardware, or a combination of both. The humanoid robot 2 includes a response following module 30, an end following module 40, an operation module 50, and an operation result sending module 60.

[0064] The response following module 30 is configured to cooperate with the robot dog to maintain within a preset following distance in response to a task execution instruction; The end following module 40 is configured to end following and cache the information of the object to be processed when receiving the information of the object to be processed fed back by the robot dog; The operation module 50 is configured to perform operations on the target site based on the cached information of the object to be processed; The operation result sending module 60 is configured to send the results of the operation to the server and the robot dog respectively through the activated hybrid communication network.

[0065] It should be noted that when the collaborative operation device of the humanoid robot and the quadruped robot provided in the above embodiments executes the collaborative operation method of the humanoid robot and the quadruped robot, only the above division of each functional module is used for illustration. In actual application, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the collaborative operation device of the humanoid robot and the quadruped robot provided in the above embodiments belongs to the same concept as the embodiments of the collaborative operation method of the humanoid robot and the quadruped robot. For the specific implementation process, please refer to the method embodiments and will not be elaborated here.

[0066] The serial numbers of the embodiments of the present application above are only for description and do not represent the advantages and disadvantages of the embodiments.

[0067] In the embodiments of the present application, on the one hand, the server, the humanoid robot and the quadruped robot are interconnected through a hybrid communication network, which is set up with a 5G communication module and a multi-channel LoRa gateway. This interaction system ensures that even in the case of unstable signals, reliable transmission of information can be guaranteed through redundant transmission. This information interaction system provides a guarantee for the humanoid robot and the quadruped robot to collaboratively complete complex tasks. On the other hand, the quadruped robot traverses and searches the target site in real time according to the task requirements, and sends the information of the objects to be processed and its own operating parameters found to the humanoid robot. The humanoid robot collaborates with the quadruped robot to maintain a preset following distance based on this information, and ends the following and performs operations when receiving the information of the objects to be processed. The quadruped robot quickly searches for the target with its excellent mobility and environmental adaptability, while the humanoid robot uses its flexibility and operation ability to perform fine operations. This coordination mechanism enables the humanoid robot and the quadruped robot to efficiently collaborate in complex environments, thereby improving the task execution efficiency.

[0068] The present application also provides a computer-readable medium, on which program instructions are stored, and when the program instructions are executed by a processor, the collaborative operation method of the humanoid robot and the quadruped robot provided in each of the above method embodiments is implemented.

[0069] The present application also provides a computer program product containing instructions, which when running on a computer, causes the computer to execute the collaborative operation method of the humanoid robot and the quadruped robot in each of the above method embodiments.

[0070] Please refer to Figure 7 which is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As Figure 7 shown, the electronic device 1000 may include: at least one processor 1001, at least one network interface 1004, a user interface 1003, a memory 1005, and at least one communication bus 1002.

[0071] Among them, the communication bus 1002 is used to realize the connection and communication between these components.

[0072] Among them, the optional user interface 1003 may further include a standard wired interface and a wireless interface.

[0073] Among them, the network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).

[0074] Among them, the processor 1001 may include one or more processing cores. The processor 1001 uses various interfaces and circuits to connect various parts within the entire electronic device 1000, and by running or executing instructions, programs, code sets or instruction sets stored in the memory 1005, as well as calling data stored in the memory 1005, it executes various functions of the electronic device 1000 and processes data. Optionally, the processor 1001 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The processor 1001 may integrate one or a combination of several of a central processing unit (CPU), a graphics processing unit (GPU), and a modem, etc. Among them, the CPU mainly processes the operating system, user interface, application programs, etc.; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communication. It can be understood that the above-mentioned modem may not be integrated into the processor 1001 and may be implemented separately by a single chip.

[0075] Among them, the memory 1005 may include a Random Access Memory (RAM), or may also include a Read-Only Memory. Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, codes, code sets, or instruction sets. The memory 1005 may include a program storage area and a data storage area. Among them, the program storage area can store instructions for implementing the operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-mentioned method embodiments, etc.; the data storage area can store the data involved in the above-mentioned method embodiments. Optionally, the memory 1005 can also be at least one storage system located far from the aforementioned processor 1001. As Figure 7 shown, the memory 1005, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a collaborative operation application program for humanoid robots and robotic dogs.

[0076] In Figure 7 the electronic device 1000 shown, the user interface 1003 is mainly used to provide an input interface for the user to obtain the data input by the user; while the processor 1001 can be used to call the collaborative operation application program for humanoid robots and robotic dogs stored in the memory 1005 and specifically perform the following operations: In response to a task search instruction, traverse and search the target site corresponding to the operation map in real time according to the task requirements; Send the information of the object to be processed and its own operation parameters searched to the humanoid robot through the activated hybrid communication network.

[0077] Or perform the following operations: In response to a task execution instruction, cooperate with the robotic dog to maintain within a preset following distance; When receiving the information of the object to be processed fed back by the robotic dog, end the following and cache the information of the object to be processed; Based on the cached information of the object to be processed, perform operations in the target site; Send the results of the operations to the server and the robotic dog respectively through the activated hybrid communication network.

[0078] In the embodiments of the present application, on the one hand, the server, the humanoid robot, and the robot dog are interconnected through a hybrid communication network, which is set up using a 5G communication module and a multi-channel LoRa gateway. This interaction system ensures reliable information transmission through redundant transmission even in the case of unstable signals, providing a guarantee for the humanoid robot and the robot dog to cooperate to complete complex tasks. On the other hand, the robot dog traverses and searches the target site in real time according to the task requirements, and sends the information of the objects to be processed and its own operating parameters found during the search to the humanoid robot. The humanoid robot cooperates with the robot dog to maintain a preset following distance based on this information, and ends the following and performs operations when receiving the information of the objects to be processed. The robot dog quickly searches for the target with its excellent mobility and environmental adaptability, while the humanoid robot uses its flexibility and operation ability to perform fine operations. This coordination mechanism enables the humanoid robot and the robot dog to efficiently cooperate in complex environments, thus improving the task execution efficiency.

[0079] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing relevant hardware through a computer program. The program for the collaborative operation of the humanoid robot and the robot dog can be stored in a computer-readable storage medium. When this program is executed, it can include the processes of the above-mentioned method embodiments. Among them, the storage medium for the program of the collaborative operation of the humanoid robot and the robot dog can be a magnetic disk, an optical disk, a read-only memory, or a random access memory, etc.

[0080] The above-disclosed are only the preferred embodiments of the present application, and of course, the scope of the rights of the present application cannot be limited thereby. Therefore, equivalent changes made according to the claims of the present application still fall within the scope covered by the present application.

Claims

1. A collaborative operation system for a humanoid robot and a quadruped robot, characterized in that, The system includes: a server, a humanoid robot, and a robotic dog; where the server, the humanoid robot, and the robotic dog are interconnected through a hybrid communication network, and the hybrid communication network is set up using a 5G communication module and a multi-channel LoRa gateway; the server receives a task request sent by an external system, and the task request carries an operation map and task requirements; activates the hybrid communication network; creates a task execution instruction for the humanoid robot and a task search instruction for the robotic dog; and sends the task execution instruction to the humanoid robot and sends the task search instruction, the operation map, and the task requirements to the robotic dog through the activated hybrid communication network; the robotic dog, in response to the task search instruction, traverses and searches the target site corresponding to the operation map in real time according to the task requirements; and sends the information of the object to be processed and its own operation parameters found in the search to the humanoid robot through the activated hybrid communication network; the humanoid robot, in response to the task execution instruction, cooperates with the robotic dog to maintain within a preset following distance; when receiving the information of the object to be processed fed back by the robotic dog, ends following and caches the information of the object to be processed; performs operations on the target site based on the cached information of the object to be processed; and sends the results of the operations to the server and the robotic dog respectively through the activated hybrid communication network.

2. The system according to claim 1, wherein The server, the humanoid robot, and the robotic dog are provided with the 5G communication module and the multi-channel LoRa gateway; Activating the hybrid communication network includes: determining the bandwidth requirements between the humanoid robot and the robotic dog according to the task requirements; configuring a quality of service policy for preset video bandwidth transmission for the 5G communication module according to the bandwidth requirements; configuring LoRa channel parameters for the multi-channel LoRa gateway according to the bandwidth requirements.

3. The system according to claim 2, wherein Determining the bandwidth requirements between the humanoid robot and the robotic dog according to the task requirements includes: extracting the task type from the task requirements; searching for the corresponding data flow entry from the pre-established mapping relationship between the task type and the task data flow according to the task type to obtain a basic data flow set; querying the unit bandwidth coefficients of each basic data flow in the basic data flow set; combining each basic data flow with its unit bandwidth coefficient to obtain an original bandwidth requirement matrix; reading the environmental monitoring data in the target site and determining a bandwidth correction coefficient according to the environmental monitoring data; multiplying the original bandwidth requirement matrix by the bandwidth correction coefficient to obtain a compensated final bandwidth requirement value as the bandwidth requirement between the humanoid robot and the robotic dog.

4. The system according to claim 1, wherein Sending the information of the object to be processed and its own operation parameters found in the search to the humanoid robot through the activated hybrid communication network includes: real-time monitoring of the 5G signal strength between itself and the humanoid robot; When the 5G signal strength is greater than or equal to a preset threshold, send the searched information of the object to be processed to the humanoid robot through the service quality policy preset in the 5G communication module; or, When the 5G signal strength is less than the preset threshold, switch to LoRa redundant transmission, and send the searched information of the object to be processed to the humanoid robot through the LoRa channel parameters preset in the multi-channel LoRa gateway.

5. The system according to claim 1, wherein The cooperation with the robotic dog to maintain within a preset following distance includes: Receive the own operation parameters sent by the robotic dog through the activated hybrid communication network, where the own operation parameters include the current coordinates of the robotic dog in the global coordinate system, the current heading angle of the robotic dog, the current linear velocity of the robotic dog, and the current angular velocity of the robotic dog; Predict the predicted coordinates of the robotic dog at the next moment according to the preset robotic dog kinematic model and the current coordinates of the robotic dog, the current heading angle of the robotic dog, the current linear velocity of the robotic dog, and the current angular velocity of the robotic dog; Control its own movement according to the predicted coordinates of the robotic dog at the next moment to cooperate with the robotic dog to maintain within a preset following distance; Among them, the preset robotic dog kinematic model for cooperating with the robotic dog is: Among them, , is the predicted coordinate of the robotic dog at the next moment , , is the current coordinate of the robotic dog at the current moment , is the current linear velocity of the robotic dog at the current moment , is the current heading angle of the robotic dog at the current moment , is the current angular velocity of the robotic dog at the current moment , is the control period.

6. The system according to claim 5, wherein The controlling its own movement according to the predicted coordinates of the robotic dog at the next moment includes: Calculate the required arrival position of the humanoid robot through the preset following distance and the predicted coordinates at the next moment; Obtain the current position of the humanoid robot; Calculate the actual distance deviation and actual direction deviation between the required arrival position of the robot and the current position of the robot; Generate a speed control command based on the actual distance deviation and actual direction deviation; Execute the speed control command to control its own movement to the required arrival position; where the calculation formula for the required arrival position of the robot is: Among them, is the position where the robot needs to reach, is the coordinates and heading angle of the robotic dog in the global coordinate system, is the preset following distance, is the robotic dog at the current moment of the current heading angle of the robotic dog, is the preset azimuth offset angle.

7. The system according to claim 1, wherein The traversing and searching the target site corresponding to the operation map in real time according to the task requirements includes: Based on the operation map, plan a movement path by combining the SLAM algorithm, and send the movement path to the humanoid robot through the activated hybrid communication network; Autonomously move according to the movement path and turn on the sensor array; During the movement, collect environmental data in real time through multi-modal sensors; Input the task requirements and the environmental data into a pre-trained target detection model, and output the objects to be processed that meet the task requirements; Create the description information of the object to be processed to obtain the searched information of the object to be processed.

8. The system according to claim 7, wherein Generate a pre-trained target detection model according to the following steps, including: Collect the historical environmental data fed back by the sensor array carried by the robotic dog when performing tasks in each historical site; Use the category library defined in advance by the task requirements to frame the target positions in the historical environmental data and associate category labels to obtain a labeled structured data set; Construct multi-modal feature vectors according to the labeled structured data set; Construct a multi-modal object detection network, where the multi-modal object detection network includes a first-stage detection network and a second-stage detection network. The first-stage detection network is used to generate candidate regions, and the second-stage detection network is used to classify and regress the target positions; Input the multi-modal feature vector into the multi-modal object detection network for machine learning, and output the model loss value; When the model loss value reaches the minimum, obtain the pre-trained object detection model.

9. The system according to claim 8, characterized in that, The labeled structured data set includes labeled image data, labeled point cloud data, and labeled environmental data; Constructing a multi-modal feature vector according to the labeled structured data set includes: Using a pre-trained convolutional neural network to extract the image features of the labeled image data; Using the PointNet network to extract the point cloud features of the labeled point cloud data; Using the LSTM network to extract the environmental features of the labeled environmental data; Fuse the image features, the point cloud features, and the environmental features to obtain a fused multi-modal feature vector; where the loss function of the multi-modal object detection network is: Among them, is the loss value, is the vector of the image features of the th annotated image data, is the vector of the point cloud features of the th annotated point cloud data, is the vector of the environmental features of the th annotated environmental data, is the number of the annotated structured data sets.

10. A collaborative operation method for a humanoid robot and a robot dog implemented by the system according to any one of claims 1-9, characterized in that, The method includes: Receive a task request sent by an external system through the server. The task request carries a job map and task requirements; activate the hybrid communication network; create a task execution instruction for the humanoid robot and a task search instruction for the robot dog; send the task execution instruction to the humanoid robot through the activated hybrid communication network, and send the task search instruction, the job map, and the task requirements to the robot dog; Through the robot dog in response to the task search instruction, according to the task requirements, traverse and search the target site corresponding to the job map in real time; send the information of the object to be processed and its own operation parameters found to the humanoid robot through the activated hybrid communication network; Through the humanoid robot in response to the task execution instruction, cooperate with the robot dog to maintain within a preset following distance; when receiving the information of the object to be processed fed back by the robot dog, end the following and cache the information of the object to be processed; based on the cached information of the object to be processed, perform operations in the target site; send the results of the operations to the server and the robot dog respectively through the activated hybrid communication network.

Images