Humanoid robot state monitoring method and device

Abstract

The invention discloses a humanoid robot state monitoring method and device. The method comprises the following steps: acquiring multi-modal information data of a humanoid robot; determining physical examination result information according to the multi-modal information data, wherein the physical examination result information is used for representing the anthropomorphic health state of the humanoid robot; when a state inquiry instruction is detected, a state updating strategy is determined according to the physical examination result information and the state inquiry instruction, and the state updating strategy comprises interaction confirmation operation, state updating operation and health maintenance operation; and executing the state updating strategy to update the current running state of the humanoid robot. Therefore, the physical examination result information representing the anthropomorphic health state of the humanoid robot is determined according to the multi-modal information data, so that when the state inquiry instruction is detected, the state updating strategy is determined according to the physical examination result information to perform interaction with higher anthropomorphic degree and more intelligent updating and maintenance; therefore, the state monitoring of the humanoid robot is more intelligent, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of humanoid robot state monitoring technology, specifically to a humanoid robot state monitoring method and device. Background Technology

[0002] Operational data monitoring is a crucial part of supporting the normal operation of humanoid robots. In existing technologies, the operational status is updated and the user is notified when the robot's operational data reaches abnormal ranges. This decision-making process is simple and the notification is mechanical, resulting in low intelligence and a poor user experience. Therefore, improving the intelligence level of humanoid robot status detection and enhancing the user experience has become a technical problem that needs to be addressed. Summary of the Invention

[0003] This application proposes a humanoid robot state monitoring method and device to solve the problems of low intelligence level and poor user experience in humanoid robot state monitoring, thereby improving the intelligence level of humanoid robot state monitoring and enhancing the user experience.

[0004] In a first aspect, embodiments of this application provide a humanoid robot state monitoring method, applied to a controller in a humanoid robot, the method comprising: The multimodal information data of the humanoid robot is acquired, and the multimodal information data is used to characterize the multidimensional physical information perceived by the humanoid robot in the current anthropomorphic functional scenario; The physical examination results are determined based on the multimodal information data, and the physical examination results are used to characterize the anthropomorphic health status of the humanoid robot. When a status inquiry command is detected, a status update strategy is determined based on the physical examination result information and the status inquiry command. The status update strategy includes interactive confirmation operation, status update operation and health maintenance operation. The state update strategy is executed to update the current operating state of the humanoid robot.

[0005] Secondly, embodiments of this application provide a humanoid robot state monitoring device, applied to a controller in a humanoid robot, the device comprising: The first receiving unit is used to acquire multimodal information data of the humanoid robot, wherein the multimodal information data is used to characterize the multidimensional physical information perceived by the humanoid robot in the current anthropomorphic functional scenario; The first processing unit is configured to determine physical examination result information based on the multimodal information data, wherein the physical examination result information is used to characterize the anthropomorphic health status of the humanoid robot; when a status inquiry command is detected, a status update strategy is determined based on the physical examination result information and the status inquiry command, wherein the status update strategy includes interactive confirmation operation, status update operation and health maintenance operation; and the status update strategy is executed to update the current operating status of the humanoid robot.

[0006] Thirdly, embodiments of this application provide a controller including a processor, a memory, and one or more programs stored in the memory and configured to be executed by the processor, the programs including instructions for performing steps as described in any of the first aspects.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implement the steps of the method described in any of the first aspects.

[0008] Fifthly, embodiments of this application provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement some or all of the steps of the method described in any of the first aspects of embodiments of this application.

[0009] As can be seen, in this application, the controller acquires multimodal information data of the humanoid robot, which is used to characterize the multidimensional physical information perceived by the humanoid robot in the current anthropomorphic functional scenario; it determines physical examination result information based on the multimodal information data, which is used to characterize the humanoid robot's anthropomorphic health status; when a status inquiry command is detected, it determines a status update strategy based on the physical examination result information and the status inquiry command, which includes interactive confirmation operations, status update operations, and health maintenance operations; and it executes the status update strategy to update the current operating status of the humanoid robot. Thus, by determining the physical examination result information characterizing the humanoid robot's anthropomorphic health status based on the multimodal information data, and thereby determining a status update strategy based on the physical examination result information when a status inquiry command is detected, it can perform more human-like interactions, more intelligent updates and maintenance, making the status monitoring of the humanoid robot more intelligent and improving the user experience. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a structural schematic diagram of a humanoid robot provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a controller in a humanoid robot provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a controller in another humanoid robot provided in an embodiment of this application; Figure 4 This is a flowchart illustrating a humanoid robot state monitoring method provided in an embodiment of this application; Figure 5 This is a flowchart illustrating another humanoid robot state monitoring method provided in an embodiment of this application; Figure 6 This is a schematic diagram of a scenario for a humanoid robot state monitoring method provided in an embodiment of this application; Figure 7 This is a functional unit block diagram of a humanoid robot state monitoring device provided in an embodiment of this application; Figure 8 This is a functional unit block diagram of another humanoid robot state monitoring device provided in this application embodiment; Figure 9 This is a structural block diagram of a controller provided in an embodiment of this application. Detailed Implementation

[0012] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0013] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0014] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0015] In the embodiments of this application, "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent the following three situations: A exists alone; A and B exist simultaneously; B exists alone. Among them, A and B can be singular or plural.

[0016] In this embodiment, the symbol " / " can indicate that the preceding and following objects are in an "or" relationship. Alternatively, the symbol " / " can also represent a division sign, i.e., performing a division operation. For example, A / B can mean A divided by B.

[0017] In the embodiments of this application, "at least one item" or its similar expression refers to any combination of these items, including any combination of a single item or a plurality of items. "One or more" means one or more, while "multiple" means two or more. For example, "at least one item" of a, b, or c can represent the following seven cases: a, b, c; a and b; a and c; b and c; a, b, and c. Each of a, b, and c can be an element or a set containing one or more elements.

[0018] In the embodiments of this application, "equal to" can be used with "greater than" and is applicable to technical solutions used when "greater than" is used; it can also be used with "less than" and is applicable to technical solutions used when "less than" is used. When "equal to" is used with "greater than", it is not used with "less than"; when "equal to" is used with "less than", it is not used with "greater than".

[0019] To better understand the solutions of the embodiments of this application, the terminal devices, related concepts and background that may be involved in the embodiments of this application will be introduced below.

[0020] Humanoid robots: Intelligent robots with human-like appearance and limb structure (such as head, torso, and limbs) that use sensors, drive systems, and AI algorithms to simulate human actions and interactions to complete human-like tasks.

[0021] Operational data monitoring is a crucial part of supporting the normal operation of humanoid robots. In existing technologies, the operational status is updated and the user is notified when the robot's operational data reaches abnormal ranges. This decision-making process is simple and the notification is mechanical, resulting in low intelligence and a poor user experience. Therefore, improving the intelligence level of humanoid robot status detection and enhancing the user experience has become a technical problem that needs to be addressed.

[0022] To address the aforementioned issues, this application provides a method and apparatus for monitoring the state of a humanoid robot. This method obtains custom actions edited by the user for the humanoid robot through a visual configuration interface for action editing. The server then updates the action library based on these custom actions, enabling the humanoid robot to call these custom actions from the server's action library during operation. This achieves real-time updates to the humanoid robot's action library simply by configuring custom actions on the terminal device. This not only simplifies operation and improves the efficiency of action library configuration but also allows for updates during use without downtime, enhancing the flexibility of action library configuration.

[0023] Please see Figure 1 , Figure 1 This is a structural schematic diagram of a humanoid robot provided in an embodiment of this application. Figure 1 As shown, the humanoid robot 100 includes multiple sensors 110 and a controller 120. The multiple sensors 110 are communicatively connected to the controller 120. The multiple sensors 110 can be multiple sensors of different or the same type. The multiple sensors 110 are distributed in any number of places on the humanoid robot 100. The controller 120 can be a single controller or a group of controllers.

[0024] In the daily use of the humanoid robot 100, the controller 120 acquires multimodal information data of the humanoid robot, which is used to characterize the multidimensional physical information perceived by the humanoid robot in the current anthropomorphic functional scenario; determines physical examination result information based on the multimodal information data, which is used to characterize the anthropomorphic health status of the humanoid robot; when a status inquiry command is detected, a status update strategy is determined based on the physical examination result information and the status inquiry command, which includes interactive confirmation operation, status update operation and health maintenance operation; and executes the status update strategy to update the current operating status of the humanoid robot.

[0025] Please see Figure 2 , Figure 2This is a schematic diagram of the structure of a controller in a humanoid robot provided in an embodiment of this application. Figure 2 As shown, the controller 120 includes a multimodal information acquisition module 210, a health analysis module 220, a cognitive decision-making module 230, and a strategy execution module 240. The multimodal information acquisition module 210 is communicatively connected to the health analysis module 220, the health analysis module 220 is communicatively connected to the cognitive decision-making module 230, and the cognitive decision-making module 230 is communicatively connected to the strategy execution module 240.

[0026] The multimodal information acquisition module 210 is used to acquire multimodal information data of the humanoid robot, and the multimodal information data is used to characterize the multidimensional physical information perceived by the humanoid robot in the current anthropomorphic functional scenario.

[0027] The health analysis module 220 is used to determine the physical examination result information based on the multimodal information data, and the physical examination result information is used to characterize the anthropomorphic health status of the humanoid robot.

[0028] The cognitive decision module 230 is used to determine a status update strategy based on the physical examination result information and the status inquiry command when a status inquiry command is detected. The status update strategy includes interactive confirmation operation, status update operation and health maintenance operation.

[0029] The strategy execution module 240 is used to execute the state update strategy to update the current operating state of the humanoid robot.

[0030] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a controller in another humanoid robot provided in an embodiment of this application. Figure 3 As shown, the controller 120 includes a processor 310 and a memory 320, with the processor 310 communicatively connected to the memory 320. The memory 320 stores one or more programs, which are configured to be executed by the processor 310. The functions of these programs are: acquiring multimodal information data of the humanoid robot, which characterizes the multidimensional physical information perceived by the humanoid robot in the current anthropomorphic functional scenario; determining physical examination result information based on the multimodal information data, which characterizes the humanoid robot's anthropomorphic health status; determining a status update strategy based on the physical examination result information and the status inquiry command when a status inquiry command is detected, the status update strategy including interactive confirmation operations, status update operations, and health maintenance operations; and executing the status update strategy to update the current operating status of the humanoid robot.

[0031] The following describes a method for monitoring the state of a humanoid robot provided by an embodiment of this application.

[0032] Please see Figure 4 , Figure 4 This is a flowchart illustrating a humanoid robot state monitoring method provided in an embodiment of this application, applicable to, for example... Figure 1 The humanoid robot 100 shown includes a controller 120. The humanoid robot 100 comprises multiple sensors 110 and a controller 120. The multiple sensors 110 are communicatively connected to the controller 120. The multiple sensors 110 can be multiple sensors of different or the same type, and are distributed at any number of locations throughout the humanoid robot 100. The controller 120 can be a single controller or a group of controllers. Figure 4 As shown, the method includes the following steps: Step S401: Obtain the multimodal information data of the humanoid robot.

[0033] The multimodal information data is used to characterize the multidimensional physical information perceived by the humanoid robot in the current anthropomorphic functional scenario.

[0034] The multimodal information data may include energy state data, attitude dynamics data, and external environment data.

[0035] Specifically, the energy state data may include battery remaining capacity data, internal impedance change data, charge / discharge cycle data, etc. The posture dynamics data may include plantar pressure distribution data, joint motion data, motor temperature data, current ripple data, etc. The external environment data may include ground tilt angle data, external ambient temperature, external ambient humidity, etc.

[0036] Step S402: Determine the physical examination result information based on the multimodal information data.

[0037] The physical examination results are used to characterize the anthropomorphic health status of the humanoid robot.

[0038] The physical examination results information is a structured description of the anthropomorphic health status of the humanoid robot generated by the health analysis module.

[0039] In one possible embodiment, the multimodal information data includes energy state data, external environment data, and posture dynamics data. The energy state data characterizes the energy usage of the humanoid robot, the external environment data characterizes the external environment conditions of the humanoid robot, and the posture dynamics data characterizes the body movements of the humanoid robot. Determining the physical examination result information based on the multimodal information data includes: determining environmental impact information based on the external environment data, whereby the environmental impact information characterizes the impact of the external environment conditions on the anthropomorphic health status of the humanoid robot; determining motion impact information based on the posture dynamics data, whereby the motion impact information characterizes the impact of the body movements of the humanoid robot on the anthropomorphic health status of the humanoid robot; and determining the physical examination result information based on the energy state data, the environmental impact information, and the motion impact information.

[0040] The environmental impact information includes heat dissipation impact information and joint torque impact information. The heat dissipation impact information characterizes the effect of external ambient temperature and humidity on the heat dissipation of the humanoid robot's motors. The joint torque impact information characterizes the effect of ground tilt angle on the joint torque of the humanoid robot. Determining the environmental impact information based on the external environment data includes: determining the heat dissipation impact information based on external ambient temperature, external ambient humidity, and a preset motor heat dissipation efficiency attenuation curve; and determining the joint torque impact information based on the ground tilt angle data and preset joint torque data. For example, when the external ambient temperature is 40°C and the external ambient humidity is 80%, the heat dissipation impact information includes a 28% decrease in heat dissipation performance; when the ground tilt angle is 15°, the joint torque impact information includes a 75% limitation on the joint motor torque.

[0041] The motion impact information includes plantar force fluctuation impact information and joint wear impact information. The plantar force fluctuation impact information is used to characterize the influence of the plantar pressure distribution on the structural wear of the humanoid robot, and the joint wear impact information is used to characterize the influence of joint load on the bearing wear of the humanoid robot. Determining the motion impact information based on the posture dynamics data includes: determining the plantar force fluctuation impact information based on the plantar pressure distribution data and a preset gait symmetry index calculation model; and determining the joint wear impact information based on the joint motion data and a joint load calculation model. For example, plantar pressure distribution data includes the pressure borne by the left forefoot, left heel, right forefoot, and right heel. This data is imported into the gait symmetry index calculation model to obtain the left-right gait symmetry index and the front-back gait symmetry index. The plantar force fluctuation impact information includes these left-right and front-back gait symmetry indices. Smaller left-right and front-back gait symmetry indices have a greater impact on the humanoid robot's balance ability. Joint motion data includes the motion angles and angular velocities of the knee and ankle joints. This data is imported into the joint load calculation model to obtain the knee joint load fluctuation index and the ankle joint load fluctuation index. Joint wear impact information includes these knee joint load fluctuation indices and the ankle joint load fluctuation index. Larger knee and ankle joint load fluctuation indices cause greater bearing wear to the humanoid robot.

[0042] As can be seen, in this example, environmental impact information is determined based on external environmental data, and motion impact information is determined based on posture dynamics data. This quantifies the impact of the humanoid robot's external environment and its own motion on the humanoid robot's anthropomorphic health status. Then, based on the environmental impact information and motion image information, the physical examination results information characterizing the humanoid robot's anthropomorphic health status is determined. Thus, when a status inquiry command is detected, a status update strategy is determined based on the physical examination results information to perform more human-like interaction, more intelligent updates and maintenance, making the status monitoring of the humanoid robot more intelligent and improving the user experience.

[0043] In one possible embodiment, determining the physical examination result information based on the energy state data, the environmental impact information, and the motion impact information includes: determining fatigue information based on the energy state data, the environmental impact information, and the motion impact information, wherein the fatigue information characterizes the humanoid robot's ability to maintain operation; determining abnormal information based on the environmental impact information, the motion impact information, and the fatigue information, wherein the abnormal information characterizes a malfunction or abnormality occurring in the humanoid robot; and determining the physical examination result information based on the fatigue information and the abnormal information.

[0044] Specifically, determining fatigue information based on the energy state data, the environmental impact information, and the motion impact information can be achieved by: determining an environmental energy consumption factor based on the environmental impact information, wherein the environmental energy consumption factor is used to indicate the basic energy consumption rate of the humanoid robot; determining a motion energy consumption factor based on the motion impact information, wherein the motion energy consumption factor is used to indicate the motion energy consumption rate of the humanoid robot; and determining the fatigue information based on the energy state data, the environmental energy consumption factor, and the motion energy consumption factor.

[0045] The fatigue information includes a fatigue index and an estimated operating time. The fatigue index is the ratio of the humanoid robot's activated operating capacity to its maximum achievable capacity. The estimated operating time is the current sustainable operating time of the humanoid robot. Determining the fatigue information based on the energy state data, the environmental energy consumption factor, and the motion energy consumption factor can specifically involve: inputting the environmental energy consumption factor and the motion energy consumption factor into a preset fatigue fusion model to obtain the fatigue index; and determining the estimated operating time based on the fatigue index and the energy state data.

[0046] Specifically, determining the abnormal information based on the environmental impact information, the motion impact information, and the fatigue information may involve: determining a first fault probability based on the environmental impact information and the fatigue information, whereby the first fault probability characterizes the probability that the humanoid robot will experience an environment-induced fault under the coupled influence of the environmental impact information and the fatigue information; determining a second fault probability based on the motion impact information and the fatigue information, whereby the second fault probability characterizes the probability that the humanoid robot will experience a motion-induced fault under the coupled influence of the motion impact information and the fatigue information; and determining the abnormal information based on the first fault probability and the second fault probability.

[0047] Specifically, determining the abnormal information based on the first fault probability and the second fault probability can be: obtaining a preset fault decision tree model; and determining the abnormal information based on the first fault probability, the second fault probability, and the fault decision tree model.

[0048] Specifically, determining the physical examination result information based on the fatigue information and the abnormal information can be as follows: determining a physical examination conclusion based on the fatigue information and the abnormal information. The physical examination conclusion indicates the anthropomorphic health status of the humanoid robot, which includes physical strength status and disease status. The physical examination conclusion, the fatigue information, and the abnormal information are then determined as the physical examination result information. For example, the fatigue information could include a fatigue index of 0.95 and an expected operating time of 5 minutes. The abnormal information could include overheating of the right knee joint for 3 minutes and a failure probability of the cooling system exceeding a preset probability threshold. In this case, the physical strength status indicated by the corresponding physical examination conclusion would be physical exhaustion, and the disease status would be hot right knee and fever. The physical examination result information would then be an information package composed of the fatigue information, the abnormal information, and the physical examination conclusion.

[0049] As can be seen in this example, fatigue information is determined based on environmental and motion impact information, and then abnormal information is determined based on these environmental, motion, and fatigue information. Thus, physical examination results representing the anthropomorphic health status of the humanoid robot are determined based on the fatigue and abnormal information. When a status inquiry command is detected, a status update strategy is determined based on the physical examination results to perform more human-like interaction, more intelligent updates and maintenance, making the status monitoring of the humanoid robot more intelligent and improving the user experience.

[0050] Step S403: When a status inquiry command is detected, a status update strategy is determined based on the physical examination result information and the status inquiry command.

[0051] The status update strategy includes interactive confirmation operations, status update operations, and health maintenance operations.

[0052] In one possible embodiment, determining a state update strategy based on the physical examination result information and the state inquiry instruction when a state inquiry instruction is detected includes: determining a target operating state based on the physical examination result information; determining autonomous dynamic declaration information based on the physical examination result information and the state inquiry instruction, wherein the autonomous dynamic declaration information is information describing the operating state of the humanoid robot in language describing human physiological states; and determining the state update strategy based on the state inquiry instruction, the target operating state, and the autonomous dynamic declaration information.

[0053] The target operating state is used to indicate the health status of the humanoid robot in the current time window.

[0054] The target operating state includes normal operation, fatigue operation, and malfunction. Determining the target operating state based on the physical examination results can be, for example: if the physical condition indicated by the physical examination conclusion is not physical exhaustion and the abnormal information is empty, the target operating state is determined to be normal operation; if the physical condition indicated by the physical examination conclusion is physical exhaustion and the abnormal information is empty, the target operating state is determined to be fatigue operation; if the abnormal information is not empty, the target operating state is determined to be malfunction.

[0055] As can be seen in this example, the target's operating status is determined based on the physical examination results, and then the autonomous dynamic declaration information is determined based on the physical examination results and the status inquiry command. Thus, based on the status inquiry command, the target's operating status, and the autonomous dynamic declaration information, a status update strategy is determined to achieve more human-like interaction, more intelligent updates and maintenance, making the status monitoring of the humanoid robot more intelligent and improving the user experience.

[0056] In one possible embodiment, determining the autonomous dynamic declaration information based on the physical examination results and the status inquiry instruction includes: determining replenishment requirement information based on the fatigue information, the replenishment requirement information being used to characterize the humanoid robot's energy requirements; determining anomaly warning information based on the anomaly information, the anomaly warning information being used to characterize a severe situation where the humanoid robot has malfunctioned; obtaining a preset initial declaration model; using the replenishment requirement information and the anomaly warning information as training data to incrementally learn the initial declaration model to obtain a target declaration model; and determining the autonomous dynamic declaration information based on the status inquiry instruction and the target declaration model.

[0057] The replenishment demand information includes energy gap, replenishment type, and replenishment priority. The energy gap is used to indicate the replenishment demand information determined based on the fatigue information. Specifically, this can be done by: determining the energy gap and replenishment type based on the fatigue index and the expected operating time; obtaining the task arrangement information of the humanoid robot; and determining the replenishment priority based on the task arrangement information and the energy gap.

[0058] The process of determining the autonomous dynamic declaration information based on the status query instruction and the target declaration model can be, for example, receiving a status query instruction of "Are you tired?" and importing the status query instruction into the target declaration model to obtain the autonomous dynamic declaration information output by the model as "I am physically exhausted, my right knee hurts, I have a fever, and my physical condition is getting worse. Please allow me to return to the station immediately to recover my strength and perform maintenance."

[0059] As can be seen in this example, the replenishment requirement information is determined based on fatigue information, and the abnormality warning information is determined based on abnormality information. Then, the replenishment requirement information and the abnormality warning information are used as incremental learning training data for the initial declaration model to train the target declaration model. Then, autonomous dynamic declaration information is determined based on the state query command and the target declaration model. Finally, based on the state query command, the target running state, and the autonomous dynamic declaration information, a state update strategy is determined to achieve more human-like interaction, more intelligent updates and maintenance, making the state monitoring of the humanoid robot more intelligent and improving the user experience.

[0060] In one possible embodiment, determining the anomaly warning information based on the anomaly information includes: acquiring a preset anomaly prediction twin model; performing a pre-run of the anomaly prediction twin model based on the anomaly information to obtain a prediction development result; and determining the anomaly warning information based on the prediction development result.

[0061] The anomaly prediction twin model is a digital twin model that predicts the development of the fault anomaly.

[0062] The predicted development results include predicted development trends and predicted development characteristics. The predicted development trends are used to characterize the direction and speed of movement of the fault anomaly, and the predicted development characteristics are used to characterize the movement characteristics of the fault anomaly.

[0063] Specifically, determining the abnormal warning information based on the predicted development results can involve: determining an abnormality risk index based on the predicted development trend, whereby the abnormality risk index indicates the severity of the impact of the malfunction on the health status of the humanoid robot; determining a warning level based on the abnormality risk index; and determining the abnormal warning information based on the warning level, the abnormality risk index, and the predicted development characteristics. For example, if the predicted development trend of the humanoid robot shows that the right knee is heating up in a direction of continuous temperature increase at an average rate of 1 degree Celsius per minute, the calculated abnormality risk index is 0.6, and the predicted development characteristic is that the temperature increase may accelerate after five minutes and affect other surrounding components, and referring to a preset warning level mapping table, the abnormality risk index of 0.6 corresponds to a warning level of three. Therefore, the abnormal warning information includes a warning level of three for the right knee heating and the possibility of accelerated temperature increase and impact on other surrounding components after five minutes.

[0064] As can be seen in this example, abnormal information is pre-simulated using a pre-defined anomaly prediction twin model to obtain the predicted development results. Then, anomaly warning information is determined based on the predicted development results. The replenishment demand information and anomaly warning information are used as incremental learning training data for the initial declaration model to train the target declaration model. Thus, autonomous dynamic declaration information is determined based on the state query command and the target declaration model. Based on the state query command, the target running state, and the autonomous dynamic declaration information, a state update strategy is determined to achieve more human-like interaction, more intelligent updates and maintenance, making the state monitoring of the humanoid robot more intelligent and improving the user experience.

[0065] In one possible embodiment, determining the status update strategy based on the target operating state and the autonomous dynamic declaration information includes: determining the interactive confirmation operation based on the autonomous dynamic declaration information; determining the status update operation based on the target operating state; and performing the health maintenance operation based on the interactive confirmation instruction and the target operating state when an interactive confirmation instruction is obtained.

[0066] The interactive confirmation operation is used to remind the user of the anthropomorphic health status of the humanoid robot and obtain the user's interactive confirmation command for the anthropomorphic health status of the humanoid robot; the status update operation is used to update the operating status of the humanoid robot; and the health maintenance operation is used to execute the processing operation corresponding to the interactive confirmation command to maintain the health status of the humanoid robot.

[0067] Specifically, determining the interactive confirmation operation based on the autonomous dynamic declaration information can be: determining voice reminder data based on the autonomous dynamic declaration information; and determining the interactive confirmation operation based on the voice reminder data, so that the humanoid robot can output the autonomous dynamic declaration information to the user and obtain the interactive confirmation command issued by the user by executing the interactive confirmation operation.

[0068] Specifically, the interactive confirmation operation may include: outputting the voice reminder data; and obtaining the user's interactive confirmation command.

[0069] Specifically, the state update operation can be: updating the current running state of the humanoid robot to the target running state.

[0070] Specifically, when an interactive confirmation instruction is received, the health maintenance operation based on the interactive confirmation instruction and the target running state may be: determining multiple health maintenance steps based on the interactive confirmation instruction; and generating the health maintenance operation based on the multiple health maintenance steps.

[0071] Please refer to Figure 5 , Figure 5 This is a flowchart illustrating another humanoid robot state monitoring method provided in an embodiment of this application. Figure 5 As shown, the humanoid robot state monitoring method specifically includes the following steps: Step S401: Obtain the multimodal information data of the humanoid robot.

[0072] The multimodal information data is used to characterize the multidimensional physical information perceived by the humanoid robot in the current anthropomorphic functional scenario.

[0073] The multimodal information data includes external environment data and posture dynamics data. The external environment data is used to characterize the external environment of the humanoid robot, and the posture dynamics data is used to characterize the body movement of the humanoid robot.

[0074] Step S402a: Determine environmental impact information based on the external environment data.

[0075] The environmental impact information is used to characterize the influence of the external environmental conditions on the anthropomorphic health status of the humanoid robot.

[0076] Step S402b: Determine motion influence information based on the attitude dynamics data.

[0077] The motion impact information is used to characterize the influence of the humanoid robot's body motion on the humanoid robot's anthropomorphic health status.

[0078] Step S402c: Determine fatigue information based on the environmental impact information and the motion impact information.

[0079] The fatigue information is used to characterize the humanoid robot's ability to maintain operation.

[0080] Step S402d: Determine abnormal information based on the environmental impact information, the motion impact information, and the fatigue information.

[0081] The abnormal information is used to characterize the malfunction or abnormality that has occurred in the humanoid robot.

[0082] Step S402e: Determine the physical examination result information based on the fatigue information and the abnormal information.

[0083] The physical examination results are used to characterize the anthropomorphic health status of the humanoid robot.

[0084] Step S403a: When a status inquiry command is detected, the target operating status is determined based on the physical examination result information.

[0085] Step S403b: Determine the autonomous dynamic declaration information based on the physical examination result information and the status inquiry instruction.

[0086] The autonomous dynamic declaration information is information describing the operating state of the humanoid robot in a language that describes human physiological states.

[0087] Step S403c: Determine the state update strategy based on the target running state and the autonomous dynamic declaration information.

[0088] The status update strategy includes interactive confirmation operations, status update operations, and health maintenance operations.

[0089] Step S404: Execute the state update strategy to update the current operating state of the humanoid robot.

[0090] As can be seen in this example, the interaction confirmation operation is determined based on the autonomous dynamic declaration information, the status update operation is determined based on the target running state, and the health maintenance operation is determined based on the interaction confirmation instruction and the target running state when the interaction confirmation instruction is obtained. This results in a status update strategy for more human-like interaction, more intelligent updates and maintenance, making the status monitoring of humanoid robots more intelligent and improving the user experience.

[0091] Step S404: Execute the state update strategy to update the current operating state of the humanoid robot.

[0092] Please refer to Figure 6 , Figure 6 This is a schematic diagram illustrating a scenario of a humanoid robot state monitoring method provided in an embodiment of this application. For example... Figure 6As shown, after the humanoid robot 100 continuously transports material boxes for 2 hours in a 40°C factory environment, the energy status data obtained includes a remaining battery power of 5%, external environment data includes an external ambient temperature of 40°C and an external ambient humidity of 80%, and posture dynamics data includes foot pressure distribution data and joint movement data. Based on the energy status data, external environment data, and posture dynamics data, the humanoid robot 100 determines the physical examination results, including physical exhaustion, disease status of right knee burning, fever, fatigue index of 0.95, expected maintenance duration of 5 minutes, right knee joint overheating lasting 3 minutes, and the probability of heat dissipation system failure exceeding a preset probability threshold. Therefore, based on the real-time physical examination results, a status update strategy including interactive confirmation operation, status update operation, and health maintenance operation is generated in real time. When the user asks "Are you tired?", the humanoid robot 100 performs an interactive confirmation operation and answers, "I am physically exhausted, my right knee hurts, and I have a fever. My physical condition is getting worse and worse. Please allow me to return to the station immediately to recover my strength and perform maintenance." It then performs a status update operation. After the user replies, "Okay, you can go back," the humanoid robot immediately performs a health maintenance operation.

[0093] As can be seen, in this application, the controller acquires multimodal information data of the humanoid robot, which is used to characterize the multidimensional physical information perceived by the humanoid robot in the current anthropomorphic functional scenario; it determines physical examination result information based on the multimodal information data, which is used to characterize the humanoid robot's anthropomorphic health status; when a status inquiry command is detected, it determines a status update strategy based on the physical examination result information and the status inquiry command, which includes interactive confirmation operations, status update operations, and health maintenance operations; and it executes the status update strategy to update the current operating status of the humanoid robot. Thus, by determining the physical examination result information characterizing the humanoid robot's anthropomorphic health status based on the multimodal information data, and thereby determining a status update strategy based on the physical examination result information when a status inquiry command is detected, it can perform more human-like interactions, more intelligent updates and maintenance, making the status monitoring of the humanoid robot more intelligent and improving the user experience.

[0094] The above primarily describes the solutions of the embodiments of this application from the perspective of the method execution process. It is understood that, in order to achieve the above functions, the server includes the corresponding hardware structure and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in the embodiments provided herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0095] For embodiments consistent with those shown above, please refer to... Figure 7 , Figure 7 This is a functional unit block diagram of a humanoid robot state monitoring device provided in an embodiment of this application, such as... Figure 7 As shown, the humanoid robot state monitoring device 700 includes: a first receiving unit 701, used to acquire multimodal information data of the humanoid robot, the multimodal information data being used to characterize the multidimensional physical information perceived by the humanoid robot in the current anthropomorphic functional scenario; a first processing unit 702, used to determine physical examination result information based on the multimodal information data, the physical examination result information being used to characterize the anthropomorphic health status of the humanoid robot; when a status inquiry command is detected, a status update strategy is determined based on the physical examination result information and the status inquiry command, the status update strategy including interactive confirmation operation, status update operation, and health maintenance operation; and the status update strategy is executed to update the current operating status of the humanoid robot.

[0096] In one possible embodiment, the multimodal information data includes energy state data, external environment data, and posture dynamics data. The energy state data characterizes the energy usage of the humanoid robot, the external environment data characterizes the external environment conditions of the humanoid robot, and the posture dynamics data characterizes the body movements of the humanoid robot. Regarding the determination of the physical examination result information based on the multimodal information data, the first processing unit 702 is specifically configured to: determine environmental impact information based on the external environment data, wherein the environmental impact information characterizes the impact of the external environment conditions on the anthropomorphic health status of the humanoid robot; determine motion impact information based on the posture dynamics data, wherein the motion impact information characterizes the impact of the body movements of the humanoid robot on the anthropomorphic health status of the humanoid robot; and determine the physical examination result information based on the energy state data, the environmental impact information, and the motion impact information.

[0097] In one possible embodiment, in determining the physical examination result information based on the energy state data, the environmental impact information, and the motion impact information, the first processing unit 702 is specifically configured to: determine fatigue information based on the energy state data, the environmental impact information, and the motion impact information, wherein the fatigue information characterizes the humanoid robot's ability to maintain operation; determine abnormal information based on the environmental impact information, the motion impact information, and the fatigue information, wherein the abnormal information characterizes a malfunction or abnormality occurring in the humanoid robot; and determine the physical examination result information based on the fatigue information and the abnormal information.

[0098] In one possible embodiment, regarding the determination of a state update strategy based on the physical examination result information and the state inquiry instruction when a state inquiry instruction is detected, the first processing unit 702 is specifically configured to: determine a target operating state based on the physical examination result information; determine autonomous dynamic declaration information based on the physical examination result information and the state inquiry instruction, wherein the autonomous dynamic declaration information is information describing the operating state of the humanoid robot in language describing human physiological states; and determine the state update strategy based on the state inquiry instruction, the target operating state, and the autonomous dynamic declaration information.

[0099] In one possible embodiment, in determining the autonomous dynamic declaration information based on the physical examination results and the status inquiry instruction, the first processing unit 702 is specifically configured to: determine replenishment demand information based on the fatigue information, the replenishment demand information being used to characterize the energy demand of the humanoid robot; determine anomaly warning information based on the anomaly information, the anomaly warning information being used to characterize the severe situation of the humanoid robot malfunctioning; obtain a preset initial declaration model; use the replenishment demand information and the anomaly warning information as training data to incrementally learn the initial declaration model to obtain a target declaration model; and determine the autonomous dynamic declaration information based on the status inquiry instruction and the target declaration model.

[0100] In one possible embodiment, in determining the abnormal warning information based on the abnormal information, the first processing unit 702 is specifically configured to: acquire a preset abnormal prediction twin model; perform a pre-run of the abnormal prediction twin model based on the abnormal information to obtain a prediction development result; and determine the abnormal warning information based on the prediction development result.

[0101] In one possible embodiment, in determining the state update strategy based on the target operating state and the autonomous dynamic declaration information, the first processing unit 702 is specifically configured to: determine the interactive confirmation operation based on the autonomous dynamic declaration information; determine the state update operation based on the target operating state; and, when an interactive confirmation instruction is obtained, perform the health maintenance operation based on the interactive confirmation instruction and the target operating state.

[0102] It is understood that since the method embodiments and the device embodiments are different presentations of the same technical concept, the content of the method embodiment section in this application should be adapted to the device embodiment section in a synchronous manner, and will not be repeated here.

[0103] When using integrated units, such as Figure 8 As shown, Figure 8 This is a functional unit block diagram of another humanoid robot state monitoring device provided in this application embodiment. Figure 8 The humanoid robot state monitoring device 700 includes a processing module 812 and a communication module 811. The processing module 812 controls and manages the actions of the humanoid robot state monitoring device 700, for example, executing the steps of the first receiving unit 701 and the first processing unit 702, and / or performing other processes described herein. The communication module 811 supports interaction between the humanoid robot state monitoring device 700 and other devices. Figure 8 As shown, the humanoid robot state monitoring device 700 may also include a storage module 813, which is used to store the program code and data of the humanoid robot state monitoring device 700.

[0104] The processing module 812 can be a processor or controller, such as a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. The communication module 811 can be a transceiver, RF circuitry, or a communication interface, etc. The storage module 813 can be a memory.

[0105] All relevant content in each scenario involved in the above method embodiments can be referenced from the functional descriptions of the corresponding functional modules, and will not be repeated here. The above-mentioned humanoid robot state monitoring device 700 can all perform the above-mentioned... Figure 4 The method for monitoring the state of a humanoid robot is shown.

[0106] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0107] Figure 9 This is a structural block diagram of a controller provided in an embodiment of this application. Figure 9 As shown, the controller 120 may include one or more of the following components: a processor 310 and a memory 320 coupled to the processor 310, wherein the memory 320 may store one or more computer programs 321, which may be configured to implement the methods described in the above embodiments when executed by one or more processors 310.

[0108] Processor 310 may include one or more processing cores. Processor 310 connects to various parts within the controller 120 using various interfaces and lines, and executes various functions and processes data of the controller 120 by running or executing instructions, programs, code sets, or instruction sets stored in memory 320, and by calling data stored in memory 320. Optionally, processor 310 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 310 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 310 and may be implemented separately using a communication chip.

[0109] The memory 320 may include random access memory (RAM) or read-only memory (ROM). The memory 320 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 320 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described above. The data storage area may also store data created by the controller 120 during use.

[0110] It is understood that the controller 120 may include more or fewer structural elements than those shown in the above block diagram, and this is not limited thereto. Embodiments of this application provide a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by processor 310, implement the steps of the method described in any possible embodiment.

[0111] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0112] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatuses, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and other division methods may exist in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0113] The unit described as a separate component may or may not be physically separate. The component shown as a unit may or may not be a physical unit; that is, it may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0114] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can be physically comprised separately, or two or more units can be integrated into one unit. The integrated unit described above can be implemented in hardware or in the form of hardware plus software functional units.

[0115] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute some steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, volatile memory, or non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DRRAM), etc., which are various media that can store program code.

[0116] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can easily conceive of variations or substitutions without departing from the spirit and scope of the present invention, and various modifications and alterations can be made, including combinations of the different functions and implementation steps described above, as well as software and hardware implementation methods, all of which are within the protection scope of the present invention.

Claims

1. A method for monitoring the state of a humanoid robot, characterized in that, The controller applied in a humanoid robot, the method comprising: The multimodal information data of the humanoid robot is acquired, and the multimodal information data is used to characterize the multidimensional physical information perceived by the humanoid robot in the current anthropomorphic functional scenario; The physical examination results are determined based on the multimodal information data, and the physical examination results are used to characterize the anthropomorphic health status of the humanoid robot. When a status inquiry command is detected, a status update strategy is determined based on the physical examination result information and the status inquiry command. The status update strategy includes interactive confirmation operation, status update operation and health maintenance operation. The state update strategy is executed to update the current operating state of the humanoid robot.

2. The method according to claim 1, characterized in that, The multimodal information data includes energy state data, external environment data, and posture dynamics data. The energy state data characterizes the energy usage of the humanoid robot, the external environment data characterizes the external environment of the humanoid robot, and the posture dynamics data characterizes the body movement of the humanoid robot. Determining the physical examination results based on the multimodal information data includes: Environmental impact information is determined based on the external environment data, and the environmental impact information is used to characterize the impact of the external environment on the anthropomorphic health status of the humanoid robot; Motion influence information is determined based on the posture dynamics data, and the motion influence information is used to characterize the impact of the humanoid robot's body motion on the humanoid robot's anthropomorphic health status. The physical examination results are determined based on the energy state data, the environmental impact information, and the motion impact information.

3. The method according to claim 2, characterized in that, The step of determining the physical examination result information based on the energy state data, the environmental impact information, and the exercise impact information includes: Fatigue information is determined based on the energy state data, the environmental impact information, and the motion impact information. The fatigue information is used to characterize the humanoid robot's ability to maintain operation. Anomaly information is determined based on the environmental impact information, the motion impact information, and the fatigue information, and the anomaly information is used to characterize the malfunction or anomaly that occurs in the humanoid robot. The physical examination results are determined based on the fatigue information and the abnormal information.

4. The method according to claim 3, characterized in that, When a status inquiry command is detected, the status update strategy is determined based on the physical examination results information and the status inquiry command, including: The target operating status is determined based on the aforementioned physical examination results. The autonomous dynamic declaration information is determined based on the physical examination results and the status inquiry command. The autonomous dynamic declaration information is information describing the operating status of the humanoid robot in language that describes human physiological state. The status update strategy is determined based on the target operating status and the autonomous dynamic declaration information.

5. The method according to claim 4, characterized in that, The process of determining the autonomous dynamic declaration information based on the physical examination results and the status inquiry instruction includes: Based on the fatigue information, replenishment demand information is determined, which is used to characterize the energy demand of the humanoid robot. Anomaly warning information is determined based on the anomaly information, and the anomaly warning information is used to characterize the severe situation of the humanoid robot malfunctioning or malfunctioning; Obtain the preset initial declaration model; Using the supply demand information and the anomaly warning information as training data, the initial declaration model is incrementally learned to obtain the target declaration model; The autonomous dynamic declaration information is determined based on the state query instruction and the target declaration model.

6. The method according to claim 5, characterized in that, The step of determining the abnormal warning information based on the abnormal information includes: Obtain a pre-defined anomaly prediction twin model; Based on the abnormal information, the anomaly prediction twin model is pre-run to obtain the predicted development result; The abnormal early warning information is determined based on the predicted development results.

7. The method according to claim 6, characterized in that, The step of determining the state update strategy based on the target operating state and the autonomous dynamic declaration information includes: The interactive confirmation operation is determined based on the autonomous dynamic declaration information; The status update operation is determined based on the target operating state; When an interactive confirmation instruction is received, the health maintenance operation is performed according to the interactive confirmation instruction and the target running state.

8. A humanoid robot state monitoring device, characterized in that, A controller for use in humanoid robots, the device comprising: The first receiving unit is used to acquire multimodal information data of the humanoid robot, wherein the multimodal information data is used to characterize the multidimensional physical information perceived by the humanoid robot in the current anthropomorphic functional scenario; The first processing unit is configured to determine physical examination result information based on the multimodal information data, wherein the physical examination result information is used to characterize the anthropomorphic health status of the humanoid robot; when a status inquiry command is detected, a status update strategy is determined based on the physical examination result information and the status inquiry command, wherein the status update strategy includes interactive confirmation operation, status update operation and health maintenance operation; and the status update strategy is executed to update the current operating status of the humanoid robot.

9. A controller, characterized in that, It includes a processor, a memory, and one or more programs, said one or more programs being stored in the memory and configured to be executed by the processor, said programs including instructions for performing the steps of the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a computer program / instruction thereon, which, when executed by a processor, implements the steps of the method as described in any one of claims 1-7.

Images