Collaborative operation system and method of humanoid robot and robot dog

Abstract

The present application discloses a collaborative operation system and method for a humanoid robot and a robot dog, the system comprising: a server, a humanoid robot, and a robot dog; the server, the humanoid robot, and the robot dog are interconnected via a hybrid communication network; the server receives a task request sent by an external system; creates a task execution instruction for the humanoid robot and a task search instruction for the robot dog; sends the task execution instruction to the humanoid robot, and sends the task search instruction and the parameters carried by the task request to the robot dog; the robot dog, in response to the task search instruction, traverses and searches the target site corresponding to the operation map; sends the searched information about the object to be processed and its own operating parameters to the humanoid robot; the humanoid robot, in response to the task execution instruction, performs operations at the target site based on the information about the object to be processed shared by the robot dog. By adopting the embodiments of the present application, the collaborative operation of the humanoid robot and the robot dog can be realized, thereby improving the efficiency of task execution.

Description

Technical Field

[0001] The present application relates to the field of intelligent control technology, and in particular to a collaborative operation system and method of a humanoid robot and a robot dog. Background Art

[0002] With the increasing application of humanoid robots and robot dogs, humanoid robots are known for their high flexibility and operational capabilities, while robot dogs are known for their excellent mobility and environmental adaptability. For example, in collaborative operations on factory production lines, humanoid robots can complete high-precision parts assembly and quality inspection, while robot dogs can handle tasks such as material transportation and equipment inspection. The two work together to improve production efficiency. In related technologies, research on humanoid robots and robot dogs has mainly focused on optimizing their respective single performance and the ability to execute single tasks. Although significant progress has been made in the unilateral performance of humanoid robots and robot dogs, there is currently no information exchange system or coordination mechanism between humanoid robots and robot dogs, making it impossible for humanoid robots and robot dogs to work together, resulting in low task execution efficiency. Summary of the Invention

[0003] The embodiments of this application provide a collaborative operation system for a humanoid robot and a robot dog. To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is provided below. This summary is not intended to be a comprehensive review, identify key or important elements, or delineate the scope of protection for these embodiments. Its sole purpose is to present some concepts in a simplified form, serving as a prelude to the detailed description that follows.

[0004] In a first aspect, an embodiment of the present application provides a collaborative operation system of a humanoid robot and a robot dog, the system comprising:

[0005] Server, humanoid robot and robot dog; among them,

[0006] The server, humanoid robot, and robot dog are interconnected through a hybrid communication network, which is set up using a 5G communication module and a multi-channel LoRa gateway.

[0007] The server receives task requests from external systems, which carry a work map and task requirements; activates the hybrid communication network; creates task execution instructions for the humanoid robot and task search instructions for the robot dog; sends the task execution instructions to the humanoid robot via the activated hybrid communication network, and sends the task search instructions, work map, and task requirements to the robot dog;

[0008] The robot dog responds to the task search instruction and searches the target site corresponding to the operation map in real time according to the task requirements; it sends the searched object information and its own operating parameters to the humanoid robot through the activated hybrid communication network;

[0009] The humanoid robot responds to task execution instructions and cooperates with the robot dog to maintain a preset following distance; when receiving the information of the object to be processed fed back by the robot dog, the humanoid robot ends following and caches the information of the object to be processed; based on the cached information of the object to be processed, the robot performs operations at the target site; and sends the results of the operations to the server and the robot dog respectively through the activated hybrid communication network.

[0010] In a second aspect, a collaborative operation method of a humanoid robot and a robot dog is provided, the method comprising:

[0011] Receive task requests from external systems through the server, which carry a work map and task requirements; activate the hybrid communication network; create task execution instructions for the humanoid robot and task search instructions for the robot dog; send the task execution instructions to the humanoid robot through the activated hybrid communication network, and send the task search instructions, work map, and task requirements to the robot dog;

[0012] The robot dog responds to the task search instruction and searches the target site corresponding to the operation map in real time according to the task requirements; the searched object information and its own operating parameters are sent to the humanoid robot through the activated hybrid communication network;

[0013] The humanoid robot responds to the task execution instructions and cooperates with the robot dog to maintain a preset following distance; when receiving the information of the object to be processed fed back by the robot dog, the following is ended and the information of the object to be processed is cached; based on the cached information of the object to be processed, the operation is performed at the target site; and the results of the operation are sent to the server and the robot dog respectively through the activated hybrid communication network.

[0014] In an embodiment of the present application, on the one hand, the server, the humanoid robot, and the robot dog are interconnected through a hybrid communication network. The hybrid communication network is set up using a 5G communication module and a multi-channel LoRa gateway. This interactive system ensures that even in the case of unstable signals, information can be reliably transmitted through redundant transmission. This information interaction system provides a guarantee for the humanoid robot and the robot dog to collaboratively complete complex tasks. On the other hand, the robot dog traverses and searches the target site in real time according to the task requirements, and sends the searched object information and its own operating parameters to the humanoid robot. The humanoid robot coordinates with the robot dog based on this information to maintain a preset following distance, and ends the following and performs the operation when the object information is received. The robot dog uses its excellent mobility and environmental adaptability to quickly search for the target, while the humanoid robot uses its flexibility and operational capabilities to perform delicate operations. This coordination mechanism enables the humanoid robot and the robot dog to work together efficiently in complex environments, thereby improving the efficiency of task execution.

[0015] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0017] Figure 1 This is a schematic diagram of the system structure of a collaborative operation system of a humanoid robot and a robot dog provided in an embodiment of the present application;

[0018] Figure 2 This is a schematic diagram of the model processing process of a pre-trained target detection model provided in an embodiment of the present application;

[0019] Figure 3 This is a flow chart of a method for training a multimodal target detection network provided in an embodiment of the present application;

[0020] Figure 4 This is a flow chart of a collaborative operation method of a humanoid robot and a robot dog provided in an embodiment of the present application;

[0021] Figure 5 This is a schematic structural diagram of a robot dog provided in an embodiment of the present application;

[0022] Figure 6 This is a schematic structural diagram of a humanoid robot provided in an embodiment of the present application;

[0023] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The following description and the drawings sufficiently illustrate specific embodiments of the application to enable those skilled in the art to practice them.

[0025] It should be clear that the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0026] When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of systems and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0027] In the description of this application, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances. In addition, in the description of this application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship.

[0028] At present, research on humanoid robots and robot dogs mainly focuses on the optimization of their respective single performance and the ability to perform single tasks.

[0029] The present applicant recognizes that, although remarkable achievements have been made in the unilateral performance of humanoid robots and robot dogs, there is currently no information interaction system and coordination mechanism between the humanoid robots and robot dogs, making it impossible to achieve collaborative operation of the humanoid robots and robot dogs, resulting in inefficient task execution.

[0030] In an embodiment of the present application, on the one hand, the server, the humanoid robot, and the robot dog are interconnected through a hybrid communication network. The hybrid communication network is set up using a 5G communication module and a multi-channel LoRa gateway. This interactive system ensures that even in the case of unstable signals, information can be reliably transmitted through redundant transmission. This information interaction system provides a guarantee for the humanoid robot and the robot dog to collaboratively complete complex tasks. On the other hand, the robot dog traverses and searches the target site in real time according to the task requirements, and sends the searched object information and its own operating parameters to the humanoid robot. The humanoid robot coordinates with the robot dog based on this information to maintain a preset following distance, and ends the following and performs the operation when the object information is received. The robot dog uses its excellent mobility and environmental adaptability to quickly search for the target, while the humanoid robot uses its flexibility and operational capabilities to perform delicate operations. This coordination mechanism enables the humanoid robot and the robot dog to work together efficiently in complex environments, thereby improving the efficiency of task execution. The following uses an exemplary embodiment to explain in detail.

[0031] See Figure 1 , Figure 1 This is a system structure diagram of a collaborative operation system of a humanoid robot and a robot dog provided in an embodiment of the present application. The system includes: a server, a humanoid robot and a robot dog; wherein the server, the humanoid robot and the robot dog are interconnected through a hybrid communication network, and the hybrid communication network is set up using a 5G communication module and a multi-channel LoRa gateway.

[0032] The server refers to a computer system or software that provides specific services within a computer network. In the collaborative operation system of a humanoid robot and a robot dog, the server serves as the control center for the entire system. A humanoid robot is a humanoid robot with a high degree of flexibility and operational capabilities. A robot dog is a quadruped robot known for its excellent mobility and environmental adaptability. A hybrid communication network is a network system that combines multiple communication technologies to achieve efficient data transmission between different devices. The 5G communication module is a high-speed, low-latency wireless communication technology capable of supporting large-scale data transmission. The multi-channel LoRa gateway is a low-power, long-range wireless communication technology with wide coverage and low power consumption. The multi-channel LoRa gateway can simultaneously support multiple LoRa communication channels to provide redundant communication support in complex environments.

[0033] In the embodiment of the present application, by combining 5G and LoRa technologies, the hybrid communication network can flexibly switch communication modes under different environmental conditions, ensuring stable and reliable data transmission between the humanoid robot and the robot dog.

[0034] In some embodiments of the present application, the server receives a task request sent by an external system, the task request carries an operation map and task requirements; activates the hybrid communication network; creates a task execution instruction for the humanoid robot and a task search instruction for the robot dog; sends the task execution instruction to the humanoid robot through the activated hybrid communication network, and sends the task search instruction, the operation map and the task requirements to the robot dog; the robot dog, in response to the task search instruction, traverses and searches the target site corresponding to the operation map in real time according to the task requirements; sends the searched information of the object to be processed and its own operating parameters to the humanoid robot through the activated hybrid communication network; the humanoid robot, in response to the task execution instruction, cooperates with the robot dog to maintain a preset following distance; upon receiving the information of the object to be processed fed back by the robot dog, ends following and caches the information of the object to be processed; performs operations at the target site based on the cached information of the object to be processed; and sends the results of the operation to the server and the robot dog respectively through the activated hybrid communication network.

[0035] A task request is a command sent by an external system to the server to initiate collaborative operations between the humanoid robot and the robot dog. The task request carries a task map and task requirements. The task map provides detailed geographic information about the target site, including its layout, obstacle locations, and critical areas. Task requirements refer to the specific task objectives and requirements to be completed, such as searching for trapped personnel, clearing debris, and operating equipment. Task execution instructions are generated by the server based on the task requirements and sent to the humanoid robot. They instruct the humanoid robot on how to perform operations at the target site, including specific steps and objects to be operated. Task search instructions are generated by the server based on the task requirements and sent specifically to the robot dog. They instruct the robot dog on how to search the target site, including search paths and ranges. Object information to be processed refers to information about the target objects discovered by the robot dog during the search process, such as the location of trapped personnel and the status of damaged equipment. This information is transmitted to the humanoid robot via the hybrid communication network for subsequent operations. Self-operational parameters refer to information about the robot dog's status during operation, such as its current coordinates, heading angle, linear velocity, and angular velocity. These parameters are used by the humanoid robot to predict the robot dog's trajectory and maintain collaborative operation. The preset following distance refers to the fixed distance maintained between the humanoid robot and the robot dog during collaborative operation. This distance is pre-set based on task requirements and environmental conditions to ensure coordination and safety between the two during operation.

[0036] For example, a collaborative operation system involving a humanoid robot and a robot dog is used in earthquake rescue scenarios. The rescue command center sends a task request to the server through an external system. The task request includes a map of the ruined area and the task requirements (e.g., locating and rescuing trapped individuals). Upon receiving the task request, the server activates the hybrid communication network to ensure stable communication in complex environments. Based on the task requirements, the server creates task execution instructions for the humanoid robot (e.g., clearing the ruins or performing precision rescue operations) and task search instructions for the robot dog (e.g., searching for trapped individuals in the ruins). The server sends the task execution instructions to the humanoid robot via the activated hybrid communication network, and simultaneously sends the task search instructions, the operation map, and the task requirements to the robot dog. After receiving the task search instructions, the robot dog conducts a real-time traversal search of the ruined area based on the operation map and task requirements. It uses its own sensor array (e.g., cameras and lidar) to collect environmental data and identify the location of trapped individuals. During the search, the robot dog transmits the location of the trapped individuals (information about the object to be processed) and its own operating parameters (e.g., current coordinates, heading angle, linear velocity, and angular velocity) to the humanoid robot via the hybrid communication network. After receiving the task execution instruction, the humanoid robot cooperates with the robot dog to maintain the preset following distance. The humanoid robot receives the operating parameters of the robot dog, predicts the movement trajectory of the robot dog, and controls its own movement to maintain a collaborative state. When the humanoid robot receives the position information of the trapped person fed back by the robot dog, it ends the following and caches the information. Based on the cached information, the humanoid robot goes to the location of the trapped person and performs fine rescue operations, such as clearing rubble, moving trapped people, etc. After completing the rescue operation, the humanoid robot sends the operation results (such as whether the trapped person has been successfully rescued, the rubble clearing situation, etc.) back to the server and the robot dog respectively through the hybrid communication network. The server receives feedback information from the humanoid robot and the robot dog in real time, monitors the progress of task execution and feeds back to the external system. The client interface of the external system is, for example, Figure 2 If new situations are discovered or the task strategy needs to be adjusted, the server can dynamically adjust the task instructions and resend them.

[0037] In some embodiments of the present application, the server, humanoid robot and robot dog are equipped with a 5G communication module and a multi-channel LoRa gateway.

[0038] Specifically, the specific process of activating the hybrid communication network includes: determining the bandwidth requirements between the humanoid robot and the robot dog based on the task requirements; configuring the service quality policy for the 5G communication module for preset video bandwidth transmission based on the bandwidth requirements; and configuring the LoRa channel parameters for the multi-channel LoRa gateway based on the bandwidth requirements.

[0039] Bandwidth requirements refer to the minimum network bandwidth required for data transmission between the humanoid robot and the robot dog in a specific mission scenario. Quality of Service (QoS) is a network technology used to optimize network resource allocation and ensure the performance of critical applications such as real-time video transmission. LoRa channel parameters include channel frequency, spreading factor (SF), and bandwidth (BW). These parameters determine the transmission range, data rate, and power consumption of LoRa communications.

[0040] In the embodiments of this application, by dynamically adjusting the bandwidth requirements between the humanoid robot and the robot dog based on task requirements and optimizing the configuration of the 5G communication module and multi-channel LoRa gateway, the communication efficiency and reliability of collaborative operations can be significantly improved. This flexible communication configuration ensures the stability and real-time performance of data transmission in complex environments, thereby improving the overall performance of collaborative operations and the success rate of tasks.

[0041] For example, the server configures QoS policies for the 5G communication module based on bandwidth requirements. It assigns high priority to high-definition video streams to ensure stable transmission even in a high-load network environment. It also configures bandwidth resources for the 5G module, reserving sufficient RBs (radio resource blocks) to support HD video transmission.

[0042] For example, configure the LoRa gateway's channel parameters based on bandwidth requirements. Select an appropriate channel frequency (such as 470MHz), spreading factor (such as SF7, suitable for short-range, high-speed data transmission), and bandwidth (such as 125kHz). Set the LoRa gateway's management parameters, including the LoRa server's address and port, and the power parameters of the RF interface, to ensure the stability and reliability of LoRa communications.

[0043] By dynamically configuring 5G and LoRa communication parameters, a humanoid robot and a robot dog can efficiently collaborate to complete inspection and maintenance tasks. The stable transmission of high-definition video ensures real-time remote monitoring, while LoRa's low power consumption and long-distance transmission characteristics guarantee reliable transmission of sensor data. This flexible communication configuration significantly improves the efficiency and reliability of equipment inspection and maintenance in industrial parks.

[0044] Specifically, the specific process of determining the bandwidth requirements between the humanoid robot and the robot dog based on the task requirements includes: extracting the task type from the task requirements; based on the task type, searching for the corresponding data flow entry from the pre-established mapping relationship between the task type and the task data flow to obtain a basic data flow set; querying the unit bandwidth coefficient of each basic data flow in the basic data flow set; combining each basic data flow with the unit bandwidth coefficient of each basic data flow to obtain an original bandwidth requirement matrix; reading the environmental monitoring data in the target site, and determining the bandwidth correction coefficient based on the environmental monitoring data; multiplying the original bandwidth requirement matrix with the bandwidth correction coefficient to obtain the compensated final bandwidth requirement value as the bandwidth requirement between the humanoid robot and the robot dog.

[0045] For example, if the task type is inspection and equipment maintenance, extracted from the task requirements, the corresponding data stream entry is searched from the pre-established mapping between task types and task data streams. For example, if the data rates are: HD video stream: 8 Mbps, sensor data: 100 kbps, control instructions: 50 kbps, the basic data stream set is obtained: {HD video stream 8 Mbps, sensor data 100 kbps, control instructions 50 kbps}. The unit bandwidth coefficient of each basic data stream in the basic data stream set is queried: 1.2 for HD video stream (accounting for video compression and transmission redundancy), 1.1 for sensor data (accounting for packet headers and checksums), and 1.0 for control instructions (simple data transmission). Based on these parameters, the original bandwidth requirement matrix can be obtained, as shown in Table 1.

[0046] Table 1

[0047]

[0048] Next, we read the environmental monitoring data of the target site (industrial park), including signal strength and interference. Based on this data, we determine the bandwidth correction factor. Assuming good signal strength but slight interference, the bandwidth correction factor is 1.1.

[0049] Finally, the original bandwidth requirement matrix is ​​multiplied by the bandwidth correction coefficient to obtain the final bandwidth requirement value after compensation, as shown in Table 2.

[0050] Table 2

[0051]

[0052] It should be noted that the pre-established mapping relationship between task types and task data flows and the unit bandwidth coefficient are pre-set based on experience.

[0053] In some embodiments of the present application, the specific process of sending the searched information of the object to be processed and its own operating parameters to the humanoid robot through the activated hybrid communication network includes: real-time monitoring of the 5G signal strength between the robot and the humanoid robot; when the 5G signal strength is greater than or equal to a preset threshold, sending the searched information of the object to be processed to the humanoid robot through the service quality policy preset in the 5G communication module; or, when the 5G signal strength is less than the preset threshold, switching to LoRa redundant transmission, and sending the searched information of the object to be processed to the humanoid robot through the LoRa channel parameters preset in the multi-channel LoRa gateway.

[0054] In some embodiments of the present application, a specific process of collaboratively maintaining a preset following distance with a robot dog includes: receiving operating parameters of the robot dog itself sent via an activated hybrid communication network, the operating parameters of the robot dog itself including the robot dog's current coordinates, the robot dog's current heading angle, the robot dog's current linear velocity, and the robot dog's current angular velocity in a global coordinate system; predicting the robot dog's predicted coordinates at a next moment based on a preset robot dog kinematic model and the robot dog's current coordinates, the robot dog's current heading angle, the robot dog's current linear velocity, and the robot dog's current angular velocity; and controlling the robot dog's own movement based on the robot dog's predicted coordinates at the next moment to collaboratively maintain the robot dog within the preset following distance; wherein the preset robot dog kinematic model for collaboratively maintaining the robot dog is:

[0055]

[0056]

[0057] in, , It is the robot dog in the next moment The predicted coordinates of , Is the robot dog at this moment The current coordinates of the robot dog, Is the robot dog at this moment The current linear speed of the robot dog, Is the robot dog at this moment The current heading angle of the robot dog, Is the robot dog at this moment The current angular velocity of the robot dog, is the control period.

[0058] In some embodiments of the present application, the specific process of controlling the movement of the robot dog according to the predicted coordinates at the next moment includes: calculating the desired arrival position of the humanoid robot by presetting the following distance and the predicted coordinates at the next moment; obtaining the current position of the humanoid robot; calculating the actual distance deviation and the actual direction deviation between the desired arrival position and the current position of the robot; generating a speed control instruction based on the actual distance deviation and the actual direction deviation; executing the speed control instruction to control the robot dog to move to the desired arrival position; wherein, the calculation formula of the desired arrival position of the robot is:

[0059]

[0060] in, is the position the robot needs to reach, are the robot dog coordinates and heading angle of the robot dog in the global coordinate system, is the preset following distance, Is the robot dog at this moment The current heading angle of the robot dog, is the preset azimuth offset angle.

[0061] In some embodiments of the present application, according to task requirements, the specific process of traversing and searching the target site corresponding to the work map in real time includes: planning the movement path based on the work map in combination with the SLAM algorithm, and sending the movement path to the humanoid robot through the activated hybrid communication network; autonomously moving according to the movement path and turning on the sensor array; during the movement, collecting environmental data in real time through multimodal sensors; inputting the task requirements and environmental data into a pre-trained target detection model, and outputting the objects to be processed that meet the task requirements; creating descriptive information of the objects to be processed, and obtaining the searched information of the objects to be processed.

[0062] Among them, SLAM (Simultaneous Localization and Mapping) is a technology that allows robots to build maps and determine their own positions in unknown environments in real time. The movement path refers to the planned route of the robot from the starting point to the target point, which is usually generated by the SLAM algorithm. The sensor array refers to the collection of various sensors carried by the robot, including visual sensors (cameras), lidar, environmental sensors (temperature, humidity, gas sensors), etc. Multimodal sensors refer to a combination of sensors that can collect different types of data, such as image data, point cloud data, environmental data, etc. The target detection model is an algorithm model based on deep learning, which is used to identify and locate target objects in images or point cloud data. Objects to be processed refer to target objects that need to be processed by the robot during the execution of the task, such as trapped people, equipment that needs to be repaired, etc.

[0063] For example, based on the work map (a map of the industrial park) and task requirements (inspection equipment), the server uses a SLAM algorithm to plan an optimal path for the robot dog from its starting point to the target device. This planned path is transmitted to the humanoid robot via a hybrid communication network (5G and LoRa). The robot dog moves autonomously along the planned path, activating its sensor array (such as cameras and LiDAR). During this movement, it collects environmental data in real time. This collected environmental data (such as images and point clouds) and task requirements are input into a pre-trained object detection model. The model identifies the location and status of the target device and creates a description. The robot dog then transmits this target device information to the humanoid robot, which then proceeds to the target location based on this information to perform maintenance operations.

[0064] For example Figure 2 As shown, Figure 2 This is a schematic diagram of the model processing process of a pre-trained target detection model provided by the present application. The pre-trained target detection model includes a feature extraction module, an encoder, and a decoder. The robot dog collects environmental data in real time through a multimodal sensor. The environmental data includes visual data, point cloud data, and lidar data. The feature extraction module provided by the pre-trained target detection model and the preset 3D reference points are used to extract features of the visual data, point cloud data, and lidar data to obtain image features, point cloud features, and radar point features. The extracted image features, point cloud features, and radar point features are input into the encoder of the pre-trained target detection model for position encoding and feature sampling to obtain an encoding result. Finally, the encoding result is input into the transformer decoder to output the searched object information to be processed. Among them, the encoder specifically includes a first-stage detection network and a second-stage detection network.

[0065] Specifically, the specific process of generating a pre-trained target detection model includes: collecting historical environmental data fed back by the sensor array carried by the robot dog when the robot dog performs tasks in various historical sites; using a category library pre-defined by task requirements, selecting target positions in the historical environmental data and associating category labels to obtain a labeled structured data set; constructing a multimodal feature vector based on the labeled structured data set; constructing a multimodal target detection network, which includes a first-stage detection network and a second-stage detection network. The first-stage detection network is used to generate candidate areas, and the second-stage detection network is used to classify and regress target positions; inputting the multimodal feature vector into the multimodal target detection network for machine learning, and outputting a model loss value; when the model loss value reaches the minimum, a pre-trained target detection model is obtained.

[0066] The annotated structured dataset includes annotated image data, annotated point cloud data, and annotated environment data.

[0067] Specifically, the specific process of constructing a multimodal feature vector based on the annotated structured data set includes: using a pre-trained convolutional neural network to extract image features of the annotated image data; using a PointNet network to extract point cloud features of the annotated point cloud data; using an LSTM network to extract environmental features of the annotated environmental data; fusing image features, point cloud features, and environmental features to obtain a fused multimodal feature vector. The loss function of the multimodal object detection network is:

[0068]

[0069] in, is the loss value, It is The vector of image features of the labeled image data, It is The vector of point cloud features of the annotated point cloud data, It is A vector of environmental features of labeled environmental data, is the number of structured datasets after annotation.

[0070] In an embodiment of the present application, on the one hand, the server, the humanoid robot, and the robot dog are interconnected through a hybrid communication network. The hybrid communication network is set up using a 5G communication module and a multi-channel LoRa gateway. This interactive system ensures that even in the case of unstable signals, information can be reliably transmitted through redundant transmission. This information interaction system provides a guarantee for the humanoid robot and the robot dog to collaboratively complete complex tasks. On the other hand, the robot dog traverses and searches the target site in real time according to the task requirements, and sends the searched object information and its own operating parameters to the humanoid robot. The humanoid robot coordinates with the robot dog based on this information to maintain a preset following distance, and ends the following and performs the operation when the object information is received. The robot dog uses its excellent mobility and environmental adaptability to quickly search for the target, while the humanoid robot uses its flexibility and operational capabilities to perform delicate operations. This coordination mechanism enables the humanoid robot and the robot dog to work together efficiently in complex environments, thereby improving the efficiency of task execution.

[0071] See Figure 3 , provides a flow chart of a training method for a multimodal target detection network according to an embodiment of the present application. Figure 3 As shown, the process includes the following steps:

[0072] S201, collecting historical environmental data fed back by the sensor array carried by the robot dog when the robot dog performs tasks in various historical sites;

[0073] S202, using a category library pre-defined by task requirements, select target locations from the historical environmental data and associate category labels to obtain a labeled structured dataset;

[0074] S203, constructing a multimodal feature vector based on the annotated structured data set;

[0075] S204: constructing a multimodal target detection network. The multimodal target detection network includes a first-stage detection network and a second-stage detection network. The first-stage detection network is used to generate candidate regions, and the second-stage detection network is used to classify and regress target locations.

[0076] S205, inputting the multimodal feature vector into a multimodal object detection network for machine learning, and outputting a model loss value;

[0077] S206: When the model loss value reaches the minimum, a pre-trained target detection model is obtained.

[0078] In the embodiments of this application, by collecting and annotating historical environmental data, constructing multimodal feature vectors, and using a phased detection network for training, the accuracy and robustness of target detection can be effectively improved. This approach fully utilizes a variety of sensor data, enhances the model's adaptability to complex environments, and at the same time ensures the model's efficiency and reliability in actual tasks by optimizing the model loss value.

[0079] See Figure 4 , provides a flowchart of a collaborative operation method of a humanoid robot and a robot dog for an embodiment of the present application. Figure 4 As shown, the detection method of the embodiment of the present application may include the following steps:

[0080] S101, receiving a task request sent by an external system through a server, the task request carrying a work map and task requirements; activating a hybrid communication network; creating a task execution instruction for the humanoid robot and a task search instruction for the robot dog; sending the task execution instruction to the humanoid robot through the activated hybrid communication network, and sending the task search instruction, work map, and task requirements to the robot dog;

[0081] S102, the robot dog responds to the task search instruction and, based on the task requirements, traverses and searches the target site corresponding to the operation map in real time; and sends the searched information of the object to be processed and its own operating parameters to the humanoid robot via the activated hybrid communication network;

[0082] S103, the humanoid robot responds to the task execution instruction and cooperates with the robot dog to maintain a preset following distance; when receiving the information of the object to be processed fed back by the robot dog, the following is ended and the information of the object to be processed is cached; based on the cached information of the object to be processed, the operation is performed at the target site; and the results of the operation are sent to the server and the robot dog respectively through the activated hybrid communication network.

[0083] In an embodiment of the present application, on the one hand, the server, the humanoid robot, and the robot dog are interconnected through a hybrid communication network. The hybrid communication network is set up using a 5G communication module and a multi-channel LoRa gateway. This interactive system ensures that even in the case of unstable signals, information can be reliably transmitted through redundant transmission. This information interaction system provides a guarantee for the humanoid robot and the robot dog to collaboratively complete complex tasks. On the other hand, the robot dog traverses and searches the target site in real time according to the task requirements, and sends the searched object information and its own operating parameters to the humanoid robot. The humanoid robot coordinates with the robot dog based on this information to maintain a preset following distance, and ends the following and performs the operation when the object information is received. The robot dog uses its excellent mobility and environmental adaptability to quickly search for the target, while the humanoid robot uses its flexibility and operational capabilities to perform delicate operations. This coordination mechanism enables the humanoid robot and the robot dog to work together efficiently in complex environments, thereby improving the efficiency of task execution.

[0084] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0085] See Figure 5 , which shows a schematic diagram of the structure of a robot dog provided by an exemplary embodiment of the present application. The robot dog can be implemented as all or part of an electronic device through software, hardware, or a combination of both. The robot dog 1 includes a traversal search module 10 and an information sending module 20.

[0086] The traversal search module 10 is used to respond to the task search instruction and perform a traversal search on the target site corresponding to the operation map in real time according to the task requirements;

[0087] The information sending module 20 is used to send the searched information of the object to be processed and its own operating parameters to the humanoid robot through the activated hybrid communication network.

[0088] It should be noted that the collaborative operation device for a humanoid robot and a robot dog provided in the above embodiment, when executing the collaborative operation method for a humanoid robot and a robot dog, only uses the division of the above-mentioned functional modules as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the collaborative operation device for a humanoid robot and a robot dog provided in the above embodiment and the collaborative operation method embodiment for a humanoid robot and a robot dog are based on the same concept. The implementation process is detailed in the method embodiment and will not be repeated here.

[0089] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0090] In an embodiment of the present application, on the one hand, the server, the humanoid robot, and the robot dog are interconnected through a hybrid communication network. The hybrid communication network is set up using a 5G communication module and a multi-channel LoRa gateway. This interactive system ensures that even in the case of unstable signals, information can be reliably transmitted through redundant transmission. This information interaction system provides a guarantee for the humanoid robot and the robot dog to collaboratively complete complex tasks. On the other hand, the robot dog traverses and searches the target site in real time according to the task requirements, and sends the searched object information and its own operating parameters to the humanoid robot. The humanoid robot coordinates with the robot dog based on this information to maintain a preset following distance, and ends the following and performs the operation when the object information is received. The robot dog uses its excellent mobility and environmental adaptability to quickly search for the target, while the humanoid robot uses its flexibility and operational capabilities to perform delicate operations. This coordination mechanism enables the humanoid robot and the robot dog to work together efficiently in complex environments, thereby improving the efficiency of task execution.

[0091] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0092] See Figure 6 , which shows a schematic diagram of the structure of a humanoid robot provided by an exemplary embodiment of the present application. The humanoid robot can be implemented as all or part of an electronic device through software, hardware, or a combination of both. The humanoid robot 2 includes a response follow-up module 30, an end follow-up module 40, an operation module 50, and an operation result sending module 60.

[0093] a response following module 30, for responding to a task execution instruction and cooperating with the robot dog to stay within a preset following distance;

[0094] The ending following module 40 is used to end the following and cache the information of the object to be processed when receiving the information of the object to be processed fed back by the robot dog;

[0095] An operation module 50 is used to perform an operation at a target site based on the cached information of the object to be processed;

[0096] The operation result sending module 60 is used to send the operation result to the server and the robot dog respectively through the activated hybrid communication network.

[0097] It should be noted that the collaborative operation device for a humanoid robot and a robot dog provided in the above embodiment, when executing the collaborative operation method for a humanoid robot and a robot dog, only uses the division of the above-mentioned functional modules as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the collaborative operation device for a humanoid robot and a robot dog provided in the above embodiment and the collaborative operation method embodiment for a humanoid robot and a robot dog are based on the same concept. The implementation process is detailed in the method embodiment and will not be repeated here.

[0098] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0099] In an embodiment of the present application, on the one hand, the server, the humanoid robot, and the robot dog are interconnected through a hybrid communication network. The hybrid communication network is set up using a 5G communication module and a multi-channel LoRa gateway. This interactive system ensures that even in the case of unstable signals, information can be reliably transmitted through redundant transmission. This information interaction system provides a guarantee for the humanoid robot and the robot dog to collaboratively complete complex tasks. On the other hand, the robot dog traverses and searches the target site in real time according to the task requirements, and sends the searched object information and its own operating parameters to the humanoid robot. The humanoid robot coordinates with the robot dog based on this information to maintain a preset following distance, and ends the following and performs the operation when the object information is received. The robot dog uses its excellent mobility and environmental adaptability to quickly search for the target, while the humanoid robot uses its flexibility and operational capabilities to perform delicate operations. This coordination mechanism enables the humanoid robot and the robot dog to work together efficiently in complex environments, thereby improving the efficiency of task execution.

[0100] The present application also provides a computer-readable medium having program instructions stored thereon, which, when executed by a processor, implement the collaborative operation method of the humanoid robot and the robot dog provided in the above-mentioned various method embodiments.

[0101] The present application also provides a computer program product containing instructions, which, when executed on a computer, enables the computer to execute the collaborative operation method of the humanoid robot and the robot dog of each of the above-mentioned method embodiments.

[0102] See Figure 7 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7 As shown, the electronic device 1000 may include: at least one processor 1001 , at least one network interface 1004 , a user interface 1003 , a memory 1005 , and at least one communication bus 1002 .

[0103] The communication bus 1002 is used to implement the connection and communication between these components.

[0104] The optional user interface 1003 may also include a standard wired interface or a wireless interface.

[0105] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0106] The processor 1001 may include one or more processing cores. The processor 1001 utilizes various interfaces and circuits to connect various components within the electronic device 1000. It executes instructions, programs, code sets, or instruction sets stored in the memory 1005, and accesses data stored in the memory 1005 to perform various functions and process data within the electronic device 1000. Optionally, the processor 1001 may be implemented in hardware using at least one of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 1001 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing content displayed on the display; and the modem handles wireless communications. It is understood that the modem may also be implemented independently of the processor 1001 and implemented on a separate chip.

[0107] Among them, the memory 1005 may include a random access memory (RAM) or a read-only memory (Read-Only Memory). Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1005 may also be optionally at least one storage system located away from the aforementioned processor 1001. As Figure 7 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a collaborative operation application program for the humanoid robot and the robot dog.

[0108] exist Figure 7 In the electronic device 1000 shown, the user interface 1003 is mainly used to provide an input interface for the user and obtain user input data; and the processor 1001 can be used to call the collaborative operation application of the humanoid robot and the robot dog stored in the memory 1005 and specifically perform the following operations:

[0109] In response to the task search instruction, according to the task requirements, the target site corresponding to the operation map is searched in real time;

[0110] The searched information of the object to be processed and the robot's own operating parameters are sent to the humanoid robot through the activated hybrid communication network.

[0111] Alternatively, do the following:

[0112] In response to the task execution instruction, the robot dog cooperates with the robot dog to maintain a preset following distance;

[0113] When receiving the object information to be processed fed back by the robot dog, the following is ended and the object information to be processed is cached;

[0114] Based on the cached information of the objects to be processed, operations are performed at the target site;

[0115] The results of the job are sent to the server and the robot dog respectively through the activated hybrid communication network.

[0116] In an embodiment of the present application, on the one hand, the server, the humanoid robot, and the robot dog are interconnected through a hybrid communication network. The hybrid communication network is set up using a 5G communication module and a multi-channel LoRa gateway. This interactive system ensures that even in the case of unstable signals, information can be reliably transmitted through redundant transmission. This information interaction system provides a guarantee for the humanoid robot and the robot dog to collaboratively complete complex tasks. On the other hand, the robot dog traverses and searches the target site in real time according to the task requirements, and sends the searched object information and its own operating parameters to the humanoid robot. The humanoid robot coordinates with the robot dog based on this information to maintain a preset following distance, and ends the following and performs the operation when the object information is received. The robot dog uses its excellent mobility and environmental adaptability to quickly search for the target, while the humanoid robot uses its flexibility and operational capabilities to perform delicate operations. This coordination mechanism enables the humanoid robot and the robot dog to work together efficiently in complex environments, thereby improving the efficiency of task execution.

[0117] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The program for the collaborative operation of the humanoid robot and the robot dog can be stored in a computer-readable storage medium. When executed, the program can include the processes in the above-described method embodiments. The storage medium for the program for the collaborative operation of the humanoid robot and the robot dog can be a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0118] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A collaborative operation system of a humanoid robot and a robot dog, characterized in that: The system comprises: Server, humanoid robot and robot dog; among them, The server, the humanoid robot, and the robot dog are interconnected via a hybrid communication network, which is configured using a 5G communication module and a multi-channel LoRa gateway; The server receives a task request sent by an external system, the task request carrying a work map and task requirements; activates the hybrid communication network; creates a task execution instruction for the humanoid robot and a task search instruction for the robot dog; sends the task execution instruction to the humanoid robot via the activated hybrid communication network, and sends the task search instruction, the work map and task requirements to the robot dog; The robot dog, in response to the task search instruction and according to the task requirements, traverses and searches the target site corresponding to the operation map in real time; and sends the searched information of the object to be processed and its own operating parameters to the humanoid robot via the activated hybrid communication network; The humanoid robot, in response to the task execution instruction, cooperates with the robot dog to maintain a preset following distance; upon receiving information about an object to be processed fed back by the robot dog, stops following and caches the information about the object to be processed; performs an operation at the target site based on the cached information about the object to be processed; and transmits the result of the operation to the server and the robot dog respectively via the activated hybrid communication network; The server, the humanoid robot, and the robot dog are provided with the 5G communication module and the multi-channel LoRa gateway; activating the hybrid communication network includes: determining the bandwidth requirement between the humanoid robot and the robot dog according to the task requirement; configuring a quality of service policy for preset video bandwidth transmission on the 5G communication module according to the bandwidth requirement; and configuring LoRa channel parameters on the multi-channel LoRa gateway according to the bandwidth requirement; Determining the bandwidth requirements between the humanoid robot and the robot dog based on the task requirements includes: extracting a task type from the task requirements; searching for corresponding data stream entries from a pre-established mapping relationship between task types and task data streams based on the task type to obtain a basic data stream set; querying the unit bandwidth coefficient of each basic data stream in the basic data stream set; combining each basic data stream with the unit bandwidth coefficient of each basic data stream to obtain an original bandwidth requirement matrix; reading environmental monitoring data in the target site, and determining a bandwidth correction coefficient based on the environmental monitoring data; and multiplying the original bandwidth requirement matrix by the bandwidth correction coefficient to obtain a compensated final bandwidth requirement value as the bandwidth requirement between the humanoid robot and the robot dog.

2. The system according to claim 1, wherein: The step of sending the searched information of the object to be processed and the robot's own operating parameters to the humanoid robot via the activated hybrid communication network includes: monitoring the 5G signal strength between the robot and the humanoid robot in real time; When the 5G signal strength is greater than or equal to a preset threshold, the searched information of the object to be processed is sent to the humanoid robot through the service quality policy preset in the 5G communication module; or When the 5G signal strength is less than a preset threshold, it switches to LoRa redundant transmission, and sends the searched information of the object to be processed to the humanoid robot through the LoRa channel parameters preset in the multi-channel LoRa gateway.

3. The system according to claim 1, wherein: The step of cooperating with the robot dog to maintain a preset following distance includes: receiving self-operation parameters of the robot dog sent through the activated hybrid communication network, wherein the self-operation parameters include the robot dog's current coordinates, the robot dog's current heading angle, the robot dog's current linear velocity, and the robot dog's current angular velocity in the global coordinate system; Predicting the predicted coordinates of the robot dog at the next moment based on a preset robot dog kinematic model and the robot dog's current coordinates, the robot dog's current heading angle, the robot dog's current linear velocity, and the robot dog's current angular velocity; Controlling its own movement based on the predicted coordinates of the robot dog at the next moment to coordinate with the robot dog to stay within a preset following distance; The kinematic model of the robot dog that keeps within the preset following distance is: in, , It is the robot dog in the next moment The predicted coordinates of , Is the robot dog at this moment The current coordinates of the robot dog, Is the robot dog at this moment The current linear speed of the robot dog, Is the robot dog at this moment The current heading angle of the robot dog, Is the robot dog at this moment The current angular velocity of the robot dog, is the control period.

4. The system according to claim 3, characterized in that The controlling the movement of the robot dog according to the predicted coordinates of the robot dog at the next moment includes: Calculating the desired arrival position of the humanoid robot by using a preset following distance and the predicted coordinates at the next moment; Obtaining a current position of the humanoid robot; Calculating the actual distance deviation and actual direction deviation between the desired position of the robot and the current position of the robot; generating a speed control instruction based on the actual distance deviation and the actual direction deviation; Execute the speed control command to control the robot to move to the desired arrival position; wherein, the calculation formula for the desired arrival position of the robot is: in, is the position the robot needs to reach, It is the robot dog in the next moment The predicted coordinates of is the preset following distance, Is the robot dog at this moment The current heading angle of the robot dog, is the preset azimuth offset angle.

5. The system according to claim 1, wherein: The step of traversing and searching the target site corresponding to the operation map in real time according to the task requirements includes: Based on the operation map, a movement path is planned in combination with a SLAM algorithm, and the movement path is sent to the humanoid robot through an activated hybrid communication network; autonomously moving along the movement path and activating the sensor array; During the movement, environmental data is collected in real time through multimodal sensors; Inputting the task requirements and the environmental data into a pre-trained target detection model, and outputting an object to be processed that meets the task requirements; Create description information of the object to be processed and obtain the searched information of the object to be processed.

6. The system according to claim 5, characterized in that Follow these steps to generate a pre-trained object detection model, including: Collect historical environmental data fed back by the sensor array carried by the robot dog when the robot dog performs tasks in various historical venues; Using a category library pre-defined by task requirements, target locations are selected from the historical environmental data and associated with category labels to obtain a labeled structured dataset; Construct a multimodal feature vector based on the annotated structured dataset; Constructing a multimodal target detection network, the multimodal target detection network comprising a first-stage detection network and a second-stage detection network, wherein the first-stage detection network is used to generate candidate regions, and the second-stage detection network is used to classify and regress target locations; Inputting the multimodal feature vector into the multimodal target detection network for machine learning, and outputting a model loss value; When the model loss value reaches a minimum, a pre-trained object detection model is obtained.

7. The system according to claim 6, characterized in that The annotated structured data set includes annotated image data, annotated point cloud data, and annotated environment data; The multimodal feature vector is constructed based on the labeled structured data set, including: Use pre-trained convolutional neural networks to extract image features from labeled image data; Use the PointNet network to extract point cloud features from the labeled point cloud data; Use LSTM network to extract environmental features of labeled environmental data; The image features, the point cloud features, and the environmental features are fused to obtain a fused multimodal feature vector; wherein the loss function of the multimodal object detection network is: in, is the loss value, It is The vector of image features of the labeled image data, It is The vector of point cloud features of the annotated point cloud data, It is A vector of environmental features of labeled environmental data, is the number of structured datasets after annotation.

8. A collaborative operation method of a humanoid robot and a robot dog implemented by the system according to any one of claims 1 to 7, characterized in that: The method comprises: Receiving, through a server, a task request sent by an external system, the task request carrying a work map and task requirements; activating the hybrid communication network; creating a task execution instruction for the humanoid robot and a task search instruction for the robot dog; sending the task execution instruction to the humanoid robot through the activated hybrid communication network, and sending the task search instruction, the work map, and task requirements to the robot dog; The robot dog responds to the task search instruction and, according to the task requirements, traverses and searches the target site corresponding to the operation map in real time; and sends the searched information of the object to be processed and its own operating parameters to the humanoid robot through the activated hybrid communication network; The humanoid robot responds to the task execution instruction and cooperates with the robot dog to maintain a preset following distance; when receiving the information of the object to be processed fed back by the robot dog, the humanoid robot ends following and caches the information of the object to be processed; based on the cached information of the object to be processed, the operation is performed at the target site; and the results of the operation are sent to the server and the robot dog respectively through the activated hybrid communication network.

Images