Robot elevator interaction method and device, electronic equipment and readable storage medium

Abstract

The application provides a robot elevator interaction method and device, electronic equipment and readable storage medium. The method comprises the following steps: after the robot enters the elevator, a target button corresponding to a target floor is marked; the robot presses the target button based on the mark corresponding to the target button; floor selection information on an elevator button panel is acquired, and a predicted elevator running state corresponding to the floor selection information is determined; the running state of the elevator is monitored to obtain an actual elevator running state; and the predicted elevator running state and the actual elevator running state are compared to determine the robot's elevator exit timing. The target button of the target floor is marked, so that the position of the target button can be accurately located based on the mark in the subsequent process, thereby accurately controlling the robot to press the target button. Meanwhile, whether the target floor is reached is determined in combination with the running state of the elevator, thereby improving the accuracy of the determination of the elevator exit timing, so that the reliability of the robot riding the elevator is improved.

Description

Technical Field

[0001] This invention relates to the field of robotics, and more particularly to a robot elevator interaction method, device, electronic device, and readable storage medium. Background Technology

[0002] In existing technologies, robots need to solve two core problems to achieve automatic elevator access: first, accurately identify and press the target floor button on the elevator control panel; and second, accurately determine whether the elevator has reached the target floor in order to decide whether to exit the elevator.

[0003] However, in practical applications, the complex environment inside elevators may affect the robot's recognition of buttons or floors, leading to problems such as button selection failure or elevator exit failure. Summary of the Invention

[0004] The main objective of this invention is to provide a robot elevator interaction method, device, electronic device, and readable storage medium, aiming to solve the problem of robot elevator failure in the prior art.

[0005] To achieve the above objectives, the present invention provides a robot elevator interaction method, the method comprising the following steps: After the robot enters the elevator, it marks the target button corresponding to the target floor; The robot is controlled to press the target button based on the marker corresponding to the target button. Obtain the floor selection information on the elevator button panel and determine the predicted elevator operating status corresponding to the floor selection information; The actual elevator operating status is obtained by monitoring the elevator's operating status; The robot's departure timing is determined by comparing the predicted elevator operating state with the actual elevator operating state.

[0006] Optionally, marking the target button corresponding to the target floor after the robot enters the elevator includes: After the robot enters the elevator, it captures images of the elevator button panel to obtain panel images; Determine the target button corresponding to the target floor within the panel image; In the virtual overlay, the target box corresponding to the target button is set to a non-operation color to mark the target button. The non-operation color is different from the component color in the elevator button panel.

[0007] Optionally, determining the predicted elevator operating status corresponding to the floor selection information includes: Obtain the lit button from the floor selection information; Determine the number of buttons lit between the current floor and the target floor; Determine the predicted number of door openings corresponding to the number of lights; Generate the predicted elevator operating status, which includes the predicted number of door openings.

[0008] Optionally, determining the robot's departure timing by comparing the predicted elevator operating state with the actual elevator operating state includes: Obtain multiple predicted sub-states from the predicted elevator operating state, and obtain multiple actual sub-states from the actual elevator operating state; For the actual sub-state, the actual sub-state is compared with the corresponding predicted sub-state to determine the target floor confidence level corresponding to the actual sub-state; Determine whether the target floor has been reached by combining the confidence levels of each target floor; If the target floor is reached, the dwell time on the current floor is determined as the robot's exit time.

[0009] Optionally, determining whether the target floor has been reached by comprehensively considering the confidence levels of each target floor includes: Obtain the state weights corresponding to each of the actual sub-states; The total confidence level is calculated based on the corresponding state weights and the target floor confidence level. Determine whether the total confidence level is greater than the confidence threshold; If the total confidence level is greater than the confidence level threshold, then it is determined that the target floor has been reached.

[0010] Optionally, obtaining the state weights corresponding to each actual sub-state includes: Obtain the actual door opening sub-state and the predicted door opening sub-state corresponding to the number of door openings; Determine whether the actual door opening sub-state does not match the predicted door opening sub-state; If the actual sub-state of door opening does not match the predicted sub-state of door opening, the state weight of the actual sub-state corresponding to the accelerometer will be adjusted from the first weight to the second weight, wherein the first weight is less than the second weight.

[0011] Optionally, determining whether the target floor has been reached by comprehensively considering the confidence levels of each target floor includes: Determine whether the number of times the actual sub-state and the predicted sub-state do not match has reached a preset failure threshold; If the number of times the actual sub-state and the predicted sub-state do not match reaches a preset failure threshold, then the confidence level of the target floor corresponding to the accelerometer is obtained. Determine whether the confidence level of the target floor corresponding to the accelerometer is greater than the independent judgment threshold; If the confidence level of the target floor corresponding to the accelerometer is greater than the independent judgment threshold, then it is determined that the target floor has been reached.

[0012] To achieve the above objectives, the present invention also provides a robot elevator interaction device, the robot elevator interaction device comprising: The first marking module is used to mark the target button corresponding to the target floor after the robot enters the elevator; The first control module is used to control the robot to press the target button based on the mark corresponding to the target button; The first acquisition module is used to acquire floor selection information on the elevator button panel and determine the predicted elevator operating status corresponding to the floor selection information. The first monitoring module is used to monitor the elevator's operating status and obtain the actual elevator operating status. The first comparison module is used to compare the predicted elevator operating state with the actual elevator operating state to determine the robot's exit timing.

[0013] To achieve the above objectives, the present invention also provides an electronic device, the electronic device including a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the robot elevator interaction method as described above.

[0014] To achieve the above objectives, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the robot elevator interaction method described above.

[0015] This invention proposes a robot elevator interaction method, device, electronic device, and readable storage medium. After the robot enters the elevator, a target button corresponding to the target floor is marked. Based on the marked target button, the robot is controlled to press the target button. Floor selection information on the elevator button panel is acquired, and the predicted elevator operating state corresponding to the floor selection information is determined. The elevator operating state is monitored to obtain the actual elevator operating state. The predicted elevator operating state is compared with the actual elevator operating state to determine the robot's exit timing. By marking the target button for the target floor, the location of the target button can be accurately located based on the mark, thereby accurately controlling the robot to press the target button. At the same time, combining the elevator operating state to determine whether the target floor has been reached improves the perception ability of the elevator floor, thereby improving the accuracy of determining the exit timing and improving the reliability of the robot riding the elevator. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the first embodiment of the robot elevator interaction method of the present invention; Figure 2 This is a detailed flowchart of the robot elevator interaction method of the present invention; Figure 3 This is a schematic diagram of the module structure of the electronic device of the present invention. Detailed Implementation

[0019] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0020] This invention provides a robot elevator interaction method, applied to a robot elevator interaction device. The robot elevator interaction state can be a robot controller or a control terminal for controlling the robot; see reference. Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the robot elevator interaction method of the present invention. The method includes the following steps: Step S10: After the robot enters the elevator, mark the target button corresponding to the target floor; The robot is one that requires elevator control; the specific type of robot can be set based on actual needs.

[0021] The target floor is the floor the robot needs to reach by taking the elevator; the target floor is determined by the robot's actual movement needs; for example, if the robot is a food delivery robot, after picking up the food, the food delivery robot determines the room where the food needs to be delivered and determines the floor where the room is located as the target floor.

[0022] The target button is the button on the elevator button panel inside the elevator that corresponds to the target floor; for example, if the target floor is the 3rd floor, then the target button is the button corresponding to "3".

[0023] The labeling refers to marking the target button with specific elements in the robot vision system. The specific elements can be set according to actual needs, such as using color, pattern, etc. as labeling elements. It can be understood that after being marked in the vision system, the label always indicates the position of the target button, regardless of whether the robot's vision acquisition can obtain an image of the target button, the position of the target button can be determined based on the label. Specifically, the position of the target button can be marked in the visual space coordinate system of the vision system.

[0024] Step S20: Control the robot to press the target button based on the mark corresponding to the target button; After the target button is marked, the robot is controlled to press the target button based on the mark.

[0025] The coordinates in the robotic arm coordinate system are determined by marking the coordinates in the visual space coordinate system. Specifically, the two-dimensional coordinates corresponding to the markers are first determined, and then the three-dimensional coordinates corresponding to the markers are obtained by combining the coordinates of the markers with those of the depth camera on the robot. This determines the coordinates in the robotic arm coordinate system, and the robot is controlled to extend its robotic arm to press the target button based on the coordinates in the robotic arm coordinate system. At the same time, to improve the pressing accuracy, the coordinates of the target button are collected in real time by the robot's depth camera during the pressing process, and the three-dimensional coordinates are transmitted through ROS. The topic is transmitted, the coordinates are in JSON format, and the transmission delay is required to be <10ms. If the transmission times out, it will be retransmitted. If the retransmission exceeds 3 times, an error will be reported. After transmission, Zhang's calibration method is used to correct distortion. When the detected error is greater than the preset error threshold, the visual servoing closed loop is triggered. That is, the position of the target button is corrected by the robot's vision system. At the same time, the control of the robotic arm is adjusted based on the target button to eliminate the deviation between the robotic arm's pressing path and the target button. The convergence time of the visual servoing closed loop can be set according to actual needs, such as requiring less than 0.5s. This loop continues until the robotic arm completes the pressing of the target button. The specific value of the preset error threshold can be set according to actual needs, such as 2mm.

[0026] Step S30: Obtain the floor selection information on the elevator button panel and determine the predicted elevator operating status corresponding to the floor selection information; The floor selection information indicates whether the floor buttons on the elevator button panel have been pressed.

[0027] For example, after the robot presses the target button, the target button lights up. At the same time, other people in the elevator press other floor buttons. Therefore, the elevator button panel will indicate all the floor buttons that have been pressed.

[0028] The specific method for obtaining floor selection information can be set based on the actual application scenario; for example, it can be set based on the specific indication method of the elevator, such as the elevator button panel lighting up the floor button that is pressed. Then, the elevator button panel can be image captured and the lit floor button can be identified to determine the floor selection information based on the lit floor button. The floor information can specifically include the floor indicated by the lit floor button.

[0029] The predicted elevator operating status is based on floor selection information, which indicates the door opening status of the elevator as it reaches the target floor. This means the floor selection information indicates the pressed floor button, which in turn indicates the floor the elevator needs to stop at. Therefore, the floor selection information can determine the door opening status of the elevator during operation. For example, if the robot's starting floor is 1 and the target floor is 10, and the floor selection information indicates that the pressed floor button corresponds to floors 3 and 5, then based on the floor selection information, it can be predicted that the elevator will need to stop at floors 3, 5, and 10 respectively during its journey from floor 1 to floor 10. This means the elevator will need to open its doors at floors 3, 5, and 10, a total of 3 times. The predicted elevator operating status can include the floor where the doors are opened and the number of times the doors are opened.

[0030] Step S40: Monitor the elevator's operating status to obtain the actual elevator operating status; The actual elevator operating status is the status obtained by monitoring the elevator's condition during actual operation.

[0031] The actual elevator operating status can include the number of times the door opens, the floor corresponding to the door opening, the direction of travel, and the elevator height. The specific parameters in the actual elevator operating status can be monitored by setting specific monitoring methods based on its type. For example, the door opening action can be monitored by image acquisition to obtain the number of door openings and the floor corresponding to the door opening. Alternatively, the direction of travel and the elevator height can be obtained by using an accelerometer installed inside the robot.

[0032] Step S50: Compare the predicted elevator operating state with the actual elevator operating state to determine the robot's exit timing.

[0033] The predicted elevator operating status is based on the floor buttons to indicate the elevator's stopping status, while the actual elevator operating status indicates the actual stopping status. Therefore, by comparing the predicted elevator operating status with the actual elevator operating status, it is possible to determine whether the elevator has actually reached the target floor, and when it is determined that the target floor has been reached, the robot can be controlled to exit the elevator.

[0034] The elevator exit timing is the scenario where the robot can be controlled to perform the elevator exit operation when the elevator reaches the target floor; specifically, the elevator exit timing can be the period from after the elevator reaches the target floor and opens the door until the elevator door closes again.

[0035] This embodiment marks the target button on the target floor, enabling accurate location of the target button based on the mark, thus accurately controlling the robot to press the target button. At the same time, it combines the elevator's operating status to determine whether the target floor has been reached, improving the elevator floor perception capability and thus improving the accuracy of determining the timing of exiting the elevator, thereby enhancing the reliability of the robot riding the elevator.

[0036] Furthermore, in the second embodiment of the robot elevator interaction method of the present invention based on the first embodiment of the present invention, step S10 includes the following steps: Step S11: After the robot enters the elevator, the elevator button panel is image captured to obtain a panel image; Step S12: Determine the target button corresponding to the target floor within the panel image; Step S13: In the virtual overlay, the target box corresponding to the target button is set to a non-operation color to mark the target button. The non-operation color is different from the component color in the elevator button panel.

[0037] After the robot enters the elevator, the image acquisition module of the robot's vision system acquires images of the elevator button panel. To improve the accuracy of button recognition, a high-resolution, high-frame-rate camera can be used as the image acquisition module. The specific camera parameters can be set according to the needs of the situation, such as ensuring the camera's frame rate covers the elevator button response time. The button response time is determined based on the elevator settings; for example, a 0.3s button response time can correspond to a camera with a frame rate of 30fps.

[0038] After obtaining the panel image, the panel image is identified to determine the target button in the panel image. In order to improve the accuracy of the target button identification, the panel image can be enhanced after obtaining it. The specific enhancement method can be set according to actual needs. For example, the enhancement method can include Gaussian filtering for noise reduction, histogram equalization to enhance contrast, Kalman filtering to remove noise, etc.

[0039] The specific method for recognizing the target buttons can be set based on actual needs. For example, a recognition model can be used to recognize panel images. The specific type of recognition model can also be set based on actual needs. For instance, a CLIP (Contrastive Language-Image Pretraining) model can be used to perform semantic matching on panel images to recognize target buttons. The recognition model can be trained using collected elevator panel images as training samples. To improve adaptability to different scenarios, the training samples need to cover diverse elevator scenarios, such as low light, panel wear, and dense button scenes. Specifically, a certain number of elevator panel images covering low light, panel wear, and dense button scenes can be acquired, and the recognition model can be trained after fine-tuning the elevator panel images. The specific number can be set based on actual needs, such as 1000 images. To ensure accuracy, the fine-tuning loss of the elevator panel images needs to be less than the loss threshold. The loss threshold can be set based on actual needs, such as 0.05.

[0040] The panel image is semantically recognized using a recognition model to identify the target button corresponding to the target floor. For example, if the target floor is the third floor, the model performs semantic recognition on the number 3 in the panel image to locate the button containing that number, thus identifying the target button. Semantic recognition yields the similarity between each elevator button and the target button, and the button with the highest similarity is selected as the target button. When multiple elevator buttons have high similarity (e.g., all exceeding 0.9), a positional prior can be introduced to further improve the accuracy of target button identification. For instance, the buttons located at the top of the panel image are prioritized as the target buttons. For example, if the elevator buttons are arranged from smallest to largest and from bottom to top, such as B2, B1, 1 at the bottom row; 2, 3, 4 at the second to last row; 5, 6, 7 at the third to last row, and so on, then the similarity between buttons 2 and 5 is greater than 0.9. Since button 5 is located at the top, it is selected as the target button.

[0041] A virtual overlay is a plane within the vision system, positioned at the elevator button panel location, used to overlay marker elements; the resolution of the virtual overlay can be set according to actual needs, such as 1280×720.

[0042] The target box is the part of the virtual overlay where the target button is located.

[0043] The non-operation color is different from the component colors in the elevator button panel; mainstream elevator button panels usually use several high-contrast colors such as red, green, yellow, white, orange, and blue; while marking the target box with a non-operation color can distinguish it from the color of the elevator button panel itself, thereby improving the perception of the target button; the specific color of the non-operation color can be set according to actual needs, such as purple being used as the non-operation color in industrial HMI (Human-Machine Interface).

[0044] By marking the target box corresponding to the target button in a non-operational color in the virtual overlay, it can be distinguished from the color of the elevator button panel, which facilitates the rapid mask extraction of the target button, improves the segmentation accuracy of the target button, and thus improves the accuracy of the robotic arm pressing the target button; the segmentation method can be set according to actual needs, such as U-Net.

[0045] Furthermore, in the third embodiment of the robot elevator interaction method of the present invention based on the first embodiment of the present invention, step S30 includes the following steps: Step S31: Obtain the lit button from the floor selection information; Step S32: Determine the number of buttons lit between the current floor and the target floor; Step S33: Determine the predicted number of door openings corresponding to the number of lights; Step S34: Generate the predicted elevator operating status, which includes the predicted number of door openings.

[0046] The lit button is the floor button that is lit after being selected in the floor selection information; the lit number is the number of lit buttons on the elevator button panel.

[0047] Understandably, the lit buttons indicate the floors the elevator needs to stop at. Therefore, the door opening status of the elevator during operation can be determined by lighting the buttons. For example, if the robot's starting floor is the 1st floor and the target floor is the 10th floor, and the floors corresponding to the lit buttons include the 3rd, 5th, and 10th floors, then based on the floor selection information, it can be predicted that the elevator will need to stop at the 3rd, 5th, and 10th floors respectively during its journey from the 1st floor to the 10th floor. That is, it will need to open the door at the 3rd, 5th, and 10th floors, resulting in a predicted number of door openings of 3. The predicted number of door openings is the number of lit buttons.

[0048] By generating a predicted elevator operating status that includes the predicted number of door openings, the predicted elevator operating status can indicate the number of door openings during the process of reaching the target floor. Therefore, when the actual number of door openings of the elevator reaches the predicted number of door openings, it is considered that there is a high probability of reaching the target floor.

[0049] Furthermore, in the fourth embodiment of the robot elevator interaction method of the present invention proposed based on the first embodiment of the present invention, step S50 includes the following steps: Step S51: Obtain multiple predicted sub-states in the predicted elevator operating state, and obtain multiple actual sub-states in the actual elevator operating state. Step S52: For the actual sub-state, compare the actual sub-state with the corresponding predicted sub-state to determine the target floor confidence level corresponding to the actual sub-state; Step S53: Determine whether the target floor has been reached by combining the confidence levels of each target floor; Step S54: If the target floor is reached, the dwell time of the current floor is determined as the time for the robot to exit the elevator.

[0050] The predicted sub-state is the state for specific parameters in the prediction of elevator operation status; such as the predicted number of door openings, predicted button status, predicted running direction, predicted height, etc.

[0051] The actual sub-state is the state obtained by the elevator in actual operation corresponding to the predicted sub-state. For example, the actual sub-state corresponding to the predicted number of door openings is the actual number of door openings, the actual sub-state corresponding to the predicted button state is the actual button state, the actual sub-state corresponding to the predicted running direction is the actual running direction, and the actual sub-state corresponding to the predicted height is the actual height.

[0052] The target floor confidence level indicates the consistency between the actual sub-state and the predicted sub-state; the higher the consistency between the actual and predicted sub-states, the higher the target floor confidence level. Both the predicted and actual sub-states indicate the elevator's operating status; therefore, it is necessary to comprehensively consider the target floor confidence levels of all corresponding sub-states to determine whether the target floor has been reached. The specific confidence level can be set based on actual needs. For the sub-state of button status, when the predicted button status is the same as the actual button status, such as when the actual button status indicates that the target button is lit before the number of door openings reaches the predicted number of door openings, the confidence level of the target floor corresponding to the button status is set to the first confidence level. After the number of door openings reaches the predicted number of door openings, if the actual button status indicates that the target button is still lit, it is considered that the actual operating status does not match the prediction. At this time, the confidence level of the target floor is reduced to the second confidence level, where the second confidence level is less than the first confidence level. The specific values ​​of the first and second confidence levels can be set according to actual needs, such as the first confidence level being 0.9 and the second confidence level being 0.4. For the sub-state of door opening count, when the predicted door opening count is the same as the actual door opening count, the confidence level of the target floor corresponding to the door opening count is set to the third confidence level. For example, if the predicted door opening count indicates a door opening sequence of 3, 5, 10, and the actual door opening count also indicates a door opening sequence of 3, 5, 10, then the predicted door opening count is considered to be the same as the actual door opening count. When the door opening sequence of the actual door opening count skips a floor, such as the second door opening in the actual count corresponding to the 6th floor, then the predicted door opening count is considered to be different from the actual door opening count. In this case, the confidence level of the target floor is reduced to the fourth confidence level, where the fourth confidence level is less than the third confidence level. The specific values ​​of the third and fourth confidence levels can be set according to actual needs, such as the third confidence level being 0.8 and the fourth confidence level being 0.3. For the sub-state of running direction, when the predicted running direction is the same as the actual running direction, the corresponding target floor confidence level is set to the fifth confidence level. The specific value of the fifth confidence level can be set according to actual needs, such as 0.7. For the sub-state of height, when the predicted height is the same as the actual height obtained based on the accelerometer, the corresponding target floor confidence level is set to the sixth confidence level. The specific value of the sixth confidence level can be set according to actual needs, such as 0.75. After determining the confidence level of the target floor corresponding to each actual sub-state, the confidence level of the target floor is combined to comprehensively determine whether the target floor has been reached. Once it is determined that the target floor has been reached, the dwell period of the target floor, that is, the time period from when the elevator arrives at the target floor and opens the door until the elevator closes the door again, is the time for the robot to exit the elevator.

[0053] Further, step S53 includes: Step S531: Obtain the state weights corresponding to each actual sub-state; Step S532: Calculate the total confidence score based on the corresponding state weights and the target floor confidence score; Step S533: Determine whether the total confidence level is greater than the confidence level threshold; Step S534: If the total confidence level is greater than the confidence level threshold, then it is determined that the target floor has been reached.

[0054] The method for calculating the total confidence score can be set according to actual needs, such as when building a Bayesian model:

[0055] Among them, P(F) i |E) represents the total confidence level; P(E|F) i P(F) represents the evidence likelihood for the target floor; i P(E|F) represents the prior probability that the current floor is the target floor; j) represents the evidence likelihood at level j; P(F) j ) represents the prior probability that the current floor is the j-th floor.

[0056] The aforementioned prior probabilities are determined by the relationship between the height values ​​obtained from the accelerometer and the heights of each floor.

[0057] The evidence likelihood of the target floor is calculated by weighting the confidence level of the target floor with state weights.

[0058] The state weight indicates the credibility of a specific actual sub-state; the sum of the state weights corresponding to each actual sub-state is 1; the specific value of the state weight corresponding to each actual sub-state can be set according to actual needs, such as the state weight corresponding to the button state being 0.4; the state weight corresponding to the number of times the door is opened being 0.3; the state weight corresponding to the running direction being 0.15; and the state weight corresponding to the height being 0.15.

[0059] After obtaining the total confidence score, it is compared with the confidence score threshold. When the total confidence score is greater than the confidence score threshold, it is considered that the current floor is more likely to be the target floor, and therefore, the target floor is determined to have been reached. When the total confidence score is less than or equal to the confidence score threshold, it is considered that the current floor is less likely to be the target floor, and therefore, the target floor is determined not to have been reached. The specific value of the confidence score threshold can be set based on actual needs, such as 0.8.

[0060] Further, step S531 includes the following steps: Step S5311: Obtain the actual sub-state of door opening and the predicted sub-state of door opening corresponding to the number of times the door is opened; Step S5312: Determine whether the actual door opening sub-state does not match the predicted door opening sub-state; Step S5313: If the actual sub-state of door opening does not match the predicted sub-state of door opening, the state weight of the actual sub-state corresponding to the accelerometer is adjusted from the first weight to the second weight, wherein the first weight is less than the second weight.

[0061] The predicted elevator door opening sequence can be obtained from the number of door openings. For example, if the elevator buttons on floors 3, 5, and 10 are lit, the number of door openings is 3, and the elevator stops at floors 3, 5, and 10 in sequence. If the actual door opening sub-state differs from the corresponding predicted door opening sub-state at any given time, such as the predicted door opening sub-state indicating floor 5 during the second door opening, while the actual door opening sub-state indicates floor 6, then the actual door opening sub-state does not match the predicted door opening sub-state, indicating that the elevator has skipped floors. The predicted door opening sequence is disrupted, making it difficult to use the original door opening sequence as a basis for floor determination. Therefore, the reliability of using the actual door opening number as the actual sub-state for determining the target floor is reduced. To avoid the error in the number of door openings affecting the determination of the target floor, the weight of the accelerometer is increased, that is, the weight of the actual sub-state of height is increased, while the weight of the number of door openings is reduced. For example, the first weight for height is 0.15, and the second weight is 0.15 × 1.5.

[0062] If the actual sub-state of the door opening matches the predicted sub-state of the door opening, then the state weight of the actual sub-state corresponding to the accelerometer will be maintained as the first weight.

[0063] Further, step S53 includes the following steps: Step S535: Determine whether the number of times the actual sub-state and the predicted sub-state do not match has reached a preset failure threshold. Step S536: If the number of times the actual sub-state and the predicted sub-state do not match reaches a preset failure threshold, then obtain the target floor confidence level corresponding to the accelerometer. Step S537: Determine whether the confidence level of the target floor corresponding to the accelerometer is greater than the independent judgment threshold; Step S538: If the confidence level of the target floor corresponding to the accelerometer is greater than the independent judgment threshold, then it is determined that the target floor has been reached.

[0064] To avoid misjudgments caused by accidental factors, multi-level degradation can be adopted when there is a mismatch in the number of door openings. For example, a three-level degradation can be set. When the elevator running time is greater than 30 seconds or there is a mismatch between the actual door opening sub-state and the predicted door opening sub-state, the first level of degradation is executed first, which is a retry operation. The retry operation involves re-acquiring the floor selection information and executing subsequent operations. When the number of retry operations reaches the preset failure threshold, the second level of degradation is executed, which is to perform independent floor estimation based on the accelerometer. The specific value of the preset failure threshold can be set according to actual needs, such as 3.

[0065] Understandably, when there are multiple instances where the actual sub-state does not match the predicted sub-state, the floor reference provided by the button state and the number of times the door is opened is relatively limited. Therefore, in this case, the target floor is determined independently based on the accelerometer by judging the height, which avoids interference from other erroneous factors and determines the target floor based on the most direct height judgment.

[0066] Since the accuracy of determination based solely on accelerometer readings decreases, a more lenient threshold needs to be set to achieve the desired result; the independent judgment threshold should be lower than the confidence threshold; the specific value of the independent judgment threshold can be set based on actual needs, such as 0.6.

[0067] If the confidence level of the target floor corresponding to the accelerometer is greater than the independent judgment threshold, it is determined that the target floor has been reached; if the confidence level of the target floor corresponding to the accelerometer is less than or equal to the independent judgment threshold, it is considered that the target floor has not been reached.

[0068] The following is based on Figure 2 The overall implementation process of this application is described below: 1. Initialization and Data Acquisition: After entering the elevator, the robot initializes its state machine to the "data acquisition" state. It acquires images of the control panel and internal environment using a high-resolution RGB camera (frame rate sufficient to cover the elevator's 0.3-second button response time, typically 30fps). Simultaneously, it monitors the elevator's operating status (start, stop, running time) using an accelerometer (sampling rate 100Hz). Images are denoised using Gaussian filtering (standard deviation σ=1.5) and contrast-enhanced using histogram equalization. Sensor data undergoes noise removal using Kalman filtering, with an error <0.1m / s².

[0069] Image enhancement: Gaussian filtering (σ=1.5) for noise reduction, histogram equalization to enhance contrast, Canny edge detection (threshold 50-150) to extract contours, adapting to illumination <50 lux.

[0070] Region segmentation: The U-Net model segments the button area (accuracy > 95%), excluding interference from billboard numbers (such as "10%).

[0071] Real-time optimization: MobileNet optimized processing, single frame time <100ms.

[0072] Visual positioning decoupling: 2. Semantic Understanding and Button Recognition: A domain-adaptive CLIP model (pre-trained and fine-tuned with 1000 elevator panel images, covering low-light, wear, and dense button scenarios, with a fine-tuning loss <0.05) is used for "semantic matching" to identify target buttons. If there is ambiguity in the matching (Top-2 similarity is >0.9), a positional prior is introduced (top row buttons are prioritized over bottom row buttons, reducing the mismatch rate to 1%).

[0073] Semantic matching: Input the target floor text (e.g., "5th floor"), extract the button area features (convolutional feature vector, dimension 512), and calculate the cosine similarity with the text (threshold 0.9). If the Top-2 similarity is >0.9, a positional prior is introduced (top row buttons are prioritized, based on panel geometric distribution), reducing the mismatch rate to 1%.

[0074] 3. Virtual Overlay and Positioning: In a 1280×720 virtual overlay layer, the target button is marked in purple (RGB: 128, 0, 128; mainstream elevator panels use several high-contrast colors such as red, green, yellow, white, orange, and blue (a small amount). Purple is recognized as a "non-operational color" in industrial HMIs. Coloring the target box in purple can avoid confusion with the real button color and facilitates rapid mask extraction in post-processing). The button area is isolated by U-Net segmentation (accuracy > 95%), and the three-dimensional coordinates (x, y, z, error < 2 mm) are calculated and transmitted to the robotic arm through a ROS topic (delay < 10 ms, automatic retransmission upon timeout).

[0075] Target floor marker: The target floor buttons are identified using a domain-adaptive CLIP multimodal model. Specific steps: Fine-tuning training: The CLIP multimodal model was fine-tuned using 1000 elevator panel images (covering low light <50 lux, wear >50%, and dense buttons), resulting in a loss convergence to <0.05 and an improvement in recognition rate to >95%.

[0076] Output: Generate the bounding box of the target button ([x, y, w, h]), and mark it as the object to be processed.

[0077] In the virtual overlay (1280×720), mark the target button in purple, without modifying the physical button. Specific steps: Virtual Overlay: Creates an overlay layer in the image memory, with the target button area pixels set to purple (RGB: 128, 0, 128) and a contrast of >50% with other buttons.

[0078] 4. Button Pressing and Verification: The robotic arm presses the button, and the camera verifies the lighting status of the target button. When the brightness change of the target button is >50%, it is considered to be successfully lit. If the lighting fails, the visual servo closed loop is triggered (adjusting the positioning until the error converges to <2mm) and retrying (up to 3 times).

[0079] Coordinate calculation: 2D coordinates (x, y) are calculated based on the purple area, and then converted into 3D coordinates (x, y, z) using a depth camera (accuracy ±5mm).

[0080] Coordinate transmission: transmitted via ROS topic (JSON format, delay <10ms, retransmission 3 times upon timeout).

[0081] Error correction: Zhang's calibration method is used to correct distortion (k1, k2 < 0.01). If the error is > 2 mm, the visual servo closed loop is triggered (adjust the robotic arm, convergence time < 0.5 s).

[0082] Button activation feedback: The robotic arm presses the light to verify its illumination status: Feedback collection: The camera detects changes in brightness (>50%) in the purple area.

[0083] Verification logic: If it fails, retry 3 times or switch to the backup button.

[0084] Log recording: Saves JPEG images (640x480) and timestamps.

[0085] 5. Operation Status Analysis and Floor Prediction: By switching to the "Floor Prediction" state via a state machine, the system analyzes button states, door opening counts, running direction, and accelerometer data to construct a Bayesian probability model to predict the current floor. If the number of door openings does not match the prediction, such as skipping a floor, the system reverts to the "Semantic Matching" state, with dynamic weight adjustments (accelerometer weight increased by 1.5 times).

[0086] Floor prediction using state machines and Bayesian models, including anomaly rollback: Button state inference: Record the lit buttons (e.g., 3rd floor, 5th floor, brightness > 80), and generate the docking sequence ([3, 5]).

[0087] Door opening count determination: Record the number of door openings using image difference (change > 50%) or accelerometer (< 0.1 m / s²) and match the sequence.

[0088] Special case handling: Skip to floor / reverse docking: If the number of door openings is not equal to the prediction (e.g., the second door opening matches the 5th floor), revert to the "semantic matching" state and increase the accelerometer weight by 1.5 times.

[0089] Digital interference: U-Net excludes non-button numbers (such as "10%" on billboards) and combines them with accelerometer height estimation (3 meters per floor, error <0.5m).

[0090] Elevator exit decision and anomaly handling: The decision on whether to exit the elevator is based on the total confidence level. An abnormal scenario triggers a three-level degradation strategy: retry, accelerometer calculation, and safe shutdown. If the running time is >30s or the door opening is mismatched, the three-level degradation is triggered: ① retry 3 times; ② independent accelerometer calculation (threshold 0.6); ③ stop in place, flashing lights for help, and uploading fault codes.

[0091] Constructing a Bayesian model:

[0092] P(E│F i Evidence likelihood includes: Button lights up: (P) b =0.9), the number of times the door was opened but the fire did not go out decreased to 0.4.

[0093] Number of door openings: when matching sequences (P) d =0.8), the skipping level drops to 0.3.

[0094] Operating direction: Upward, higher floors prioritized, (P) r =0.7).

[0095] Accelerometer: Altitude estimation, (P) s =0.75).

[0096] Weights: (w1=0.4) (button state), (w2=0.3) (number of door openings), (w3=0.15) (direction of movement), (w4=0.15) (height); 100 simulations, MSE (Mean Squared Error) < 0.05 Dynamic adjustment: When there is digital interference or dead zone, (w1→ 0.2), (w4→ 0.35).

[0097] Decision: Confidence threshold P(F) t = 0.8 out-of-step rate, false out-of-step rate <3%.

[0098] 6. Closed-loop optimization: The state machine runs in a loop and records logs, including JPEG images and fault codes, supporting real-time debugging and performance optimization.

[0099] This application ensures a button recognition rate of >95% and a floor prediction error rate of <3% under conditions of 10–3000 lux illumination, dense buttons, digital interference, and complex operating scenarios through module decoupling (separation of vision and robotic arm), state machine rollback, and dynamic weight adjustment.

[0100] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0101] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0102] This application also provides a robot elevator interaction device for implementing the above-described robot elevator interaction method, the robot elevator interaction device comprising: The first marking module is used to mark the target button corresponding to the target floor after the robot enters the elevator; The first control module is used to control the robot to press the target button based on the mark corresponding to the target button; The first acquisition module is used to acquire floor selection information on the elevator button panel and determine the predicted elevator operating status corresponding to the floor selection information. The first monitoring module is used to monitor the elevator's operating status and obtain the actual elevator operating status. The first comparison module is used to compare the predicted elevator operating state with the actual elevator operating state to determine the robot's exit timing.

[0103] This robotic elevator interaction device marks the target button on the target floor, enabling accurate location of the target button based on the mark, thus accurately controlling the robot to press the target button. At the same time, it combines the elevator's operating status to determine whether the target floor has been reached, improving the elevator's floor perception capability and thereby improving the accuracy of determining the timing of exiting the elevator, thus enhancing the reliability of the robot riding the elevator.

[0104] It should be noted that the first marking module in this embodiment can be used to execute step S10 in this application embodiment, the first control module in this embodiment can be used to execute step S20 in this application embodiment, the first acquisition module in this embodiment can be used to execute step S30 in this application embodiment, the first monitoring module in this embodiment can be used to execute step S40 in this application embodiment, and the first comparison module in this embodiment can be used to execute step S50 in this application embodiment.

[0105] Further, the first marking module includes: The first acquisition submodule is used to acquire images of the elevator button panel after the robot enters the elevator to obtain panel images; The first determining submodule is used to determine the target button corresponding to the target floor within the panel image; The first setting submodule is used to set the target box corresponding to the target button in the virtual overlay to a non-operation color to mark the target button, wherein the non-operation color is different from the component color in the elevator button panel.

[0106] Furthermore, the first acquisition module includes: The first acquisition submodule is used to acquire the lit button from the floor selection information; The second determining submodule is used to determine the number of buttons lit between the current floor and the target floor; The third determining submodule is used to determine the predicted number of door openings corresponding to the number of lights; The first generation submodule is used to generate the predicted elevator operating status, which includes the predicted number of door openings.

[0107] Furthermore, the first comparison module includes: The second acquisition submodule is used to acquire multiple predicted sub-states in the predicted elevator operating state and multiple actual sub-states in the actual elevator operating state. The first comparison submodule is used to compare the actual substate with the corresponding predicted substate for the actual substate in order to determine the target floor confidence level corresponding to the actual substate. The first comprehensive submodule is used to determine whether the target floor has been reached by integrating the confidence levels of each target floor. The fourth determining submodule is used to determine the robot's exit timing based on the dwell period on the current floor if the target floor is reached.

[0108] Furthermore, the first integrated submodule includes: The first acquisition unit is used to acquire the state weights corresponding to each of the actual sub-states; The first calculation unit is used to calculate the total confidence level based on the corresponding state weights and the target floor confidence level. The first judgment unit is used to determine whether the total confidence level is greater than the confidence threshold. The first determining unit is configured to determine that the target floor has been reached if the total confidence level is greater than the confidence level threshold.

[0109] Further, the first acquisition unit includes: The first acquisition subunit is used to acquire the actual door opening sub-state and the predicted door opening sub-state corresponding to the number of door openings. The first judgment subunit is used to determine whether the actual door opening sub-state does not match the predicted door opening sub-state. The first adjustment subunit is used to adjust the state weight of the actual substate corresponding to the accelerometer from the first weight to the second weight if the actual substate of the door opening does not match the predicted substate of the door opening, wherein the first weight is less than the second weight.

[0110] Furthermore, the first integrated submodule includes: The second judgment unit is used to determine whether the number of times the actual sub-state and the predicted sub-state do not match has reached a preset failure threshold. The second acquisition unit is used to acquire the target floor confidence level corresponding to the accelerometer if the number of times the actual sub-state and the predicted sub-state do not match reaches a preset failure threshold. The third judgment unit is used to determine whether the confidence level of the target floor corresponding to the accelerometer is greater than the independent judgment threshold. The second determining unit is used to determine that the target floor has been reached if the confidence level of the target floor corresponding to the accelerometer is greater than the independent judgment threshold.

[0111] Reference Figure 3 In terms of hardware structure, the electronic device may include components such as a communication module 10, a memory 20, and a processor 30. In the electronic device, the processor 30 is connected to both the memory 20 and the communication module 10. The memory 20 stores a computer program, which is executed by the processor 30. When the computer program is executed, it implements the steps of the above-described method embodiments.

[0112] The communication module 10 can connect to external communication devices via a network. The communication module 10 can receive requests from the external communication devices and can also send requests, instructions, and information to the external communication devices. The external communication devices can be other electronic devices, servers, or IoT devices, such as televisions, etc.

[0113] The memory 20 can be used to store software programs and various data. The memory 20 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as marking the target button corresponding to the target floor after the robot enters the elevator), etc.; the data storage area may include a database, and may store data or information created based on system usage. Furthermore, the memory 20 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0114] The processor 30 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 20, and by calling data stored in the memory 20, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. The processor 30 may include one or more processing units; optionally, the processor 30 may integrate an application processor and a modem processor. The application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 30.

[0115] although Figure 3 Not shown, but the above-described electronic device may further include a circuit control module for connecting to a power supply to ensure the normal operation of other components. Those skilled in the art will understand that... Figure 3 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0116] The present invention also proposes a computer-readable storage medium having a computer program stored thereon. The computer-readable storage medium may be... Figure 3 The memory 20 in the electronic device may also be at least one of ROM (Read-Only Memory) / RAM (Random Access Memory), magnetic disk, optical disk, etc. The computer-readable storage medium includes a number of instructions to cause a terminal device with a processor (which may be a television, automobile, mobile phone, computer, server, terminal, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0117] In this invention, the terms "first," "second," "third," "fourth," and "fifth" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0118] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0119] Although embodiments of the present invention have been shown and described above, the scope of protection of the present invention is not limited thereto. It is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, and substitutions to the above embodiments within the scope of the present invention, and such changes, modifications, and substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A robot elevator interaction method, characterized in that, The robot elevator interaction method includes: After the robot enters the elevator, it marks the target button corresponding to the target floor; The robot is controlled to press the target button based on the marker corresponding to the target button. Obtain the floor selection information on the elevator button panel and determine the predicted elevator operating status corresponding to the floor selection information; The actual elevator operating status is obtained by monitoring the elevator's operating status; The robot's departure timing is determined by comparing the predicted elevator operating state with the actual elevator operating state.

2. The robot elevator interaction method as described in claim 1, characterized in that, The step of marking the target button corresponding to the target floor after the robot enters the elevator includes: After the robot enters the elevator, it captures images of the elevator button panel to obtain panel images; Determine the target button corresponding to the target floor within the panel image; In the virtual overlay, the target box corresponding to the target button is set to a non-operation color to mark the target button. The non-operation color is different from the component color in the elevator button panel.

3. The robot elevator interaction method as described in claim 1, characterized in that, Determining the predicted elevator operating status corresponding to the floor selection information includes: Obtain the lit button from the floor selection information; Determine the number of buttons lit between the current floor and the target floor; Determine the predicted number of door openings corresponding to the number of lights; Generate the predicted elevator operating status, which includes the predicted number of door openings.

4. The robot elevator interaction method as described in claim 1, characterized in that, The step of comparing the predicted elevator operating state with the actual elevator operating state to determine the robot's departure timing includes: Obtain multiple predicted sub-states from the predicted elevator operating state, and obtain multiple actual sub-states from the actual elevator operating state; For the actual sub-state, the actual sub-state is compared with the corresponding predicted sub-state to determine the target floor confidence level corresponding to the actual sub-state; Determine whether the target floor has been reached by combining the confidence levels of each target floor; If the target floor is reached, the dwell time on the current floor is determined as the robot's exit time.

5. The robot elevator interaction method as described in claim 4, characterized in that, The step of determining whether the target floor has been reached by comprehensively considering the confidence levels of each target floor includes: Obtain the state weights corresponding to each of the actual sub-states; The total confidence level is calculated based on the corresponding state weights and the target floor confidence level. Determine whether the total confidence level is greater than the confidence threshold; If the total confidence level is greater than the confidence level threshold, then it is determined that the target floor has been reached.

6. The robot elevator interaction method as described in claim 5, characterized in that, The step of obtaining the state weights corresponding to each actual sub-state includes: Obtain the actual door opening sub-state and the predicted door opening sub-state corresponding to the number of door openings; Determine whether the actual door opening sub-state does not match the predicted door opening sub-state; If the actual sub-state of door opening does not match the predicted sub-state of door opening, the state weight of the actual sub-state corresponding to the accelerometer will be adjusted from the first weight to the second weight, wherein the first weight is less than the second weight.

7. The robot elevator interaction method as described in claim 4, characterized in that, The step of determining whether the target floor has been reached by comprehensively considering the confidence levels of each target floor includes: Determine whether the number of times the actual sub-state and the predicted sub-state do not match has reached a preset failure threshold; If the number of times the actual sub-state and the predicted sub-state do not match reaches a preset failure threshold, then the confidence level of the target floor corresponding to the accelerometer is obtained. Determine whether the confidence level of the target floor corresponding to the accelerometer is greater than the independent judgment threshold; If the confidence level of the target floor corresponding to the accelerometer is greater than the independent judgment threshold, then it is determined that the target floor has been reached.

8. A robot elevator interaction device, characterized in that, The robot elevator interaction device includes: The first marking module is used to mark the target button corresponding to the target floor after the robot enters the elevator; The first control module is used to control the robot to press the target button based on the mark corresponding to the target button; The first acquisition module is used to acquire floor selection information on the elevator button panel and determine the predicted elevator operating status corresponding to the floor selection information. The first monitoring module is used to monitor the elevator's operating status and obtain the actual elevator operating status. The first comparison module is used to compare the predicted elevator operating state with the actual elevator operating state to determine the timing of the robot's exit from the elevator.

9. An electronic device, characterized in that, The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the robot elevator interaction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the robot elevator interaction method as described in any one of claims 1 to 7.

Images