Method and device for guiding a robot into an elevator in a multi-elevator scenario

Abstract

The application discloses a method and device for guiding a robot to enter an elevator in a multi-elevator scenario. The method comprises: obtaining a plurality of sets of reference coordinate data of a plurality of position reference points in a current floor scenario provided with a plurality of elevators in a floor reference coordinate system; converting the plurality of sets of reference coordinate data into a plurality of sets of first coordinate data in a robot coordinate system; converting the plurality of sets of first coordinate data into a plurality of sets of second coordinate data in an image sensor coordinate system; determining an elevator expected position of each of the plurality of elevators in an image obtained by the image sensor based on the plurality of sets of second coordinate data; performing dynamic target detection on the image obtained by the image sensor based on the elevator expected position to obtain an elevator door state of each of the plurality of elevators; and determining a target elevator among the plurality of elevators based on the elevator door state of each of the plurality of elevators, and controlling the robot to enter the target elevator based on the plurality of sets of first coordinate data.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robots, and in particular to a method and device for guiding a robot to enter an elevator in a multi-elevator scenario. BACKGROUND

[0002] In a multi-elevator scenario, a robot moves across floors usually in a manner of binding the robot with an elevator, i.e., a designated elevator is controlled by a lift control device for the robot to take. This manner requires the lift control device to be deployed in advance, which is difficult to promote. SUMMARY

[0003] The method for guiding a robot to enter an elevator in a multi-elevator scenario according to an embodiment of the present application comprises: obtaining a plurality of sets of reference coordinate data of a plurality of position reference points in a current floor scenario provided with a plurality of elevators in a floor reference coordinate system; converting the plurality of sets of reference coordinate data into a plurality of sets of first coordinate data in a robot coordinate system based on a conversion relationship between the robot coordinate system and the floor reference coordinate system, the robot coordinate system being capable of describing a positional relationship between the current floor scenario and a robot body; converting the plurality of sets of first coordinate data into a plurality of sets of second coordinate data in an image sensor coordinate system based on a conversion relationship between the image sensor coordinate system and the robot coordinate system, the image sensor coordinate system being capable of describing a positional relationship between the current floor scenario and an image sensor provided on the robot body; determining an elevator expected position of each of the plurality of elevators in an image acquired by the image sensor based on the plurality of sets of second coordinate data; performing dynamic target detection on the image acquired by the image sensor based on the elevator expected position to obtain an elevator door state of each of the plurality of elevators; and determining a target elevator among the plurality of elevators based on the elevator door state of each of the plurality of elevators, and controlling the robot to enter the target elevator based on the plurality of sets of first coordinate data.

[0004] The device for guiding a robot to enter an elevator in a multi-elevator scenario according to an embodiment of the present application comprises: a processor; and a memory having computer executable instructions stored thereon, wherein the computer executable instructions, when executed by the processor, cause the processor to perform the method for guiding a robot to enter an elevator in a multi-elevator scenario.

[0005] The computer readable storage medium according to an embodiment of the present application has computer executable instructions stored thereon, wherein the computer executable instructions, when executed by a processor, cause the processor to perform the method for guiding a robot to enter an elevator in a multi-elevator scenario.

[0006] The computer program product according to an embodiment of the present application comprises computer executable instructions, wherein the computer executable instructions, when executed by a processor, cause the processor to perform the method for guiding a robot to enter an elevator in a multi-elevator scenario. BRIEF DESCRIPTION OF DRAWINGS

[0007] The present application can be better understood with reference to the following examples and the appended drawings, of which:

[0008] Figure 1 A schematic flow chart of a method for guiding a robot into an elevator in a multi-elevator scenario according to an embodiment of the present application is shown.

[0009] Figure 2 A structural schematic diagram of a robot according to an embodiment of the present application is shown.

[0010] Figure 3 A structural block diagram of a controller of a robot body according to an embodiment of the present application is shown.

[0011] Figure 4 An example flow chart of a method for guiding a robot into an elevator in a multi-elevator scenario according to an embodiment of the present application is shown.

[0012] Figure 5 A schematic diagram of a computer system that can implement a method and apparatus for guiding a robot into an elevator in a multi-elevator scenario according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0013] Features and exemplary embodiments of various aspects of the present application will be described in detail below. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one of ordinary skill in the art that the present application can be practiced without some or all of these specific details. The description of the embodiments is merely intended to provide a better understanding of the present application. The present application is not limited to any particular configuration and algorithm disclosed below, but covers any modifications, equivalents, and alternatives falling within the spirit of the present application. In the drawings and the following description, well-known structures and techniques have not been shown in order to avoid unnecessary obscuring of the present application.

[0014] The robot cross-floor movement is realized by binding the robot with the elevator, which requires the pre-deployment of the elevator control equipment, and the promotion is difficult. The method and device for guiding the robot to enter the elevator in the multi-elevator scenario according to the embodiment of the application are provided, wherein the conversion relationship between the coordinate systems is used to convert the multiple sets of reference coordinate data of the multiple position points in the current floor scenario provided with multiple elevators in the floor reference coordinate system into multiple sets of primary coordinate data in the robot coordinate system and multiple sets of secondary coordinate data in the image sensor coordinate system, wherein the multiple sets of secondary coordinate data can determine the elevator expected position of each elevator in the image obtained by the image sensor, that is, the positioning of the elevator in the image; the elevator door state of each elevator in the image can be accurately determined based on the elevator expected position, and the target elevator can be determined based on the elevator door state, and then the robot can be controlled to enter the target elevator based on the multiple sets of primary coordinate data. Here, the positioning of the multiple elevators with the same shape in the image by the multiple sets of secondary coordinate data effectively distinguishes the multiple elevators, overcomes the problem that similar objects are easily confused in image recognition, the robot can take the elevator by itself, does not need to be bound with the robot, does not need to be deployed in relation to the elevator control equipment, and the implementation cost of the scheme is reduced.

[0015] Figure 1 A schematic flowchart of the method for guiding the robot to enter the elevator in the multi-elevator scenario according to the embodiment of the application is shown. As shown in Figure 1 The method 100 for guiding the robot to enter the elevator in the multi-elevator scenario according to the embodiment of the application comprises:

[0016] S101: Obtain multiple sets of reference coordinate data of multiple position reference points in the current floor scenario provided with multiple elevators in the floor reference coordinate system; S102: Convert the multiple sets of reference coordinate data into multiple sets of primary coordinate data in the robot coordinate system based on the conversion relationship between the robot coordinate system and the floor reference coordinate system, wherein the robot coordinate system can describe the positional relationship between the current floor scenario and the robot body; S103: Convert the multiple sets of primary coordinate data into multiple sets of secondary coordinate data in the image sensor coordinate system based on the conversion relationship between the image sensor coordinate system and the robot coordinate system, wherein the image sensor coordinate system can describe the positional relationship between the current floor scenario and the image sensor arranged on the robot body; S104: Determine the elevator expected position of each elevator in the image obtained by the image sensor based on the multiple sets of secondary coordinate data; S105: Perform dynamic target detection on the image obtained by the image sensor based on the elevator expected position, to obtain the elevator door state of each elevator in the multiple elevators; and S106: Determine the target elevator in the multiple elevators based on the elevator door state of each elevator in the multiple elevators, and control the robot to enter the target elevator based on the multiple sets of primary coordinate data.

[0017] In the method according to an embodiment of the present invention, multiple elevators are provided in the current floor scene, and each elevator operates between the current floor scene and other floor scenes. Multiple location reference points in the current floor scene are key location points used to form a map of the current floor scene required for the robot to enter the elevator. For example, one or more location reference points corresponding to each elevator (e.g., the center point or apex of each elevator door, a preset boarding point within each elevator car). Here, the map formed by the multiple location reference points in the current floor scene is used to describe the positional relationships between the location reference points. The map itself remains unchanged when the location reference points do not change substantially, but the coordinate data describing each location reference point will be determined according to the corresponding coordinate system.

[0018] In the method according to an embodiment of the present invention, the floor reference coordinate system is a coordinate system that is fixed relative to the current floor scene and does not change with the robot. The coordinate system can take a specified location point in the current floor scene (e.g., the entrance location point or center point of the elevator hall) as the origin. The coordinate system can describe the positional relationship between the specified location point and other location reference points.

[0019] In the method according to an embodiment of the present invention, the robot coordinate system is a coordinate system with the robot body as the origin, and the positional relationship between the current floor scene and the robot body changes dynamically with the robot's pose. Controlling the robot's movement can be based on primary coordinate data in the robot coordinate system.

[0020] In some embodiments, a set of reference coordinate data for a location reference point in a floor reference coordinate system is: It can be based on the formula Convert to a set of linear coordinate data in the robot coordinate system Here, a first-order rotation matrix And a first-order translation matrix Determined based on the robot's current pose in the floor reference coordinate system.

[0021] In the method according to an embodiment of the present invention, the image sensor coordinate system is a coordinate system with the image sensor located on the robot body as the origin, and the positional relationship between the current floor scene and the image sensor changes dynamically with the robot's pose and the installation position of the image sensor relative to the robot.

[0022] In some embodiments, a set of primary coordinate data in the robot coordinate system It can be based on formula Converted into a set of quadratic coordinate data in the image sensor coordinate system Here, the quadratic rotation matrix and the second translation matrix The installation position of the image sensor relative to the robot is determined.

[0023] In the method according to the embodiments of the present application, there is a corresponding relationship between the image sensor coordinate system and the image acquired by the image sensor, specifically a projection mapping relationship between the three-dimensional space and the two-dimensional plane. Due to the existence of the mapping relationship, the expected position of each elevator in the image acquired by the image sensor can be determined based on the secondary coordinate data.

[0024] Specifically, in some embodiments, determining the expected position of each elevator in the image acquired by the image sensor in the plurality of elevators includes: converting the plurality of sets of secondary coordinate data into a plurality of sets of image coordinate data in a two-dimensional image coordinate system based on the projection mapping relationship between the two-dimensional image coordinate system in which the image acquired by the image sensor is located and the image sensor coordinate system; and determining the expected position of each elevator in the image acquired by the image sensor in the plurality of elevators based on the plurality of sets of image coordinate data.

[0025] Specifically, in some embodiments, the projection mapping relationship between the two-dimensional image coordinate system in which the image acquired by the image sensor is located and the image sensor coordinate system is determined by the camera focal length f of the image sensor. For example, a set of secondary coordinate data can be converted into a set of image coordinate data according to the following formula:

[0026] ,

[0027] where f is the camera focal length of the image sensor.

[0028] Specifically, in some embodiments, the two-dimensional image coordinate system is a coordinate system that ideally describes a normalized image plane. In actual applications, the image acquired by the image sensor is usually a collection of a plurality of pixel points arranged in order, and the position of each pixel point can be described by a set of pixel sequence values. For example, a set of image coordinate data can be converted into a set of pixel sequence values (u, v) according to the following formula:

[0029] , ;

[0030] where the pixel sequence value (u, v) represents the u-th horizontal and v-th vertical pixel in the collection of pixel points of the image acquired by the image sensor, K is the camera intrinsic matrix of the image sensor, and is the focal length of the image sensor in the horizontal and vertical axis directions in units of pixels, , is the camera principal point coordinate of the image sensor, which is usually close to the image center.

[0031] Further, the method according to the embodiments of the present application can further comprise: before converting the plurality of sets of secondary coordinate data into a plurality of sets of image coordinate data, screening out secondary coordinate data in the plurality of sets of secondary coordinate data which is not in the image capturing range of the image sensor. Since the pose and the angle of view of the image sensor are limited, the acquired image does not necessarily contain all the position reference points. In order to reduce the amount of data processing and avoid interference of irrelevant position reference points on the process of elevator positioning, the secondary coordinate data of these irrelevant position reference points (i.e. the secondary coordinate data not belonging to the image capturing range) can be screened out in advance.

[0032] Specifically, the image capturing range includes two aspects. On the one hand, the image acquisition of the image sensor is only directed to the front of the image sensor (i.e. the direction of the arrow in FIG. 1), and on the other hand, the image acquisition angle of the image sensor is limited. Even if the position reference point is located in the front of the image sensor, the image acquired by the image sensor does not include the object beyond the image framing frame. Therefore, the screening action can be performed in two steps. In some embodiments, the method can further comprise: before converting the plurality of sets of secondary coordinate data into a plurality of sets of image coordinate data, screening out secondary coordinate data in the plurality of sets of secondary coordinate data which is not in the front of the image sensor; and before determining the expected position of each of the plurality of elevators in the image acquired by the image sensor based on the plurality of sets of image coordinate data, screening out image coordinate data in the plurality of sets of image coordinate data which is not in the image framing frame of the image acquired by the image sensor.

[0033] In some embodiments, screening out image coordinate data in the plurality of sets of image coordinate data which is not in the image framing frame of the image acquired by the image sensor comprises: converting the plurality of sets of image coordinate data into a plurality of sets of pixel sequence values in the image acquired by the image sensor by using the camera intrinsic matrix of the image sensor, and screening out pixel sequence values in the plurality of sets of pixel sequence values which are not in the effective pixel range corresponding to the image framing frame of the image acquired by the image sensor; and determining the expected position of each of the plurality of elevators in the image based on the plurality of sets of image coordinate data comprises: determining the expected position of each of the plurality of elevators in the image acquired by the image sensor based on the plurality of sets of pixel sequence values. Here, the effective pixel range corresponding to the image framing frame includes the pixel number width corresponding to the width of the image framing frame and the pixel number height corresponding to the height of the image framing frame. Assuming that the pixel sequence values are counted from 0, the effective pixel range can be expressed as:

[0034] , ,

[0035] If the pixel sequence value is not in this range, it means that the corresponding position reference point is beyond the angle of view of the image sensor, and is considered as an invalid value.

[0036] ​In the method according to the embodiments of the present application, a unique number can be assigned to each elevator in advance to distinguish each elevator, and after the expected positions of the elevators in the image acquired by the image sensor are determined, dynamic target detection can be performed on the image acquired by the image sensor. Specifically, the process of dynamic target detection includes: performing target detection using a deep learning target detection algorithm (such as the YOLO series) to identify elevators, comparing the target detection result with the expected positions of the elevators to determine the local image of each elevator; for the local image of each elevator, using a visual recognition algorithm to analyze the door state of a single frame image as fully open, opening and closing, or fully closed, dynamically analyzing the states of multiple single frame images corresponding to consecutive frame images in time sequence, and determining the elevator door state of the elevator as one of fully closed, opening, fully open, or closing.

[0037] In the method according to the embodiments of the present application, the target elevator is the elevator that stays at the current scene floor and provides sufficient time for the robot to enter. When the elevator door state is opening, fully open, or closing, the elevator stays in the current floor scene. To ensure that the time is sufficient for the robot to enter the elevator, the elevator door state of the target elevator is usually opening or fully open.

[0038] Further, in some embodiments, determining the target elevator from the plurality of elevators can include: performing passenger target detection on the image acquired by the image sensor based on the expected positions of the elevators to obtain elevator occupancy values indicating the number of passengers in front of the doors and inside the cars of each elevator in the plurality of elevators; and determining the target elevator based on the elevator door state and the elevator occupancy values of each elevator in the plurality of elevators. Here, the elevator occupancy value can be determined according to the passengers identified by the passenger target detection and the image intersection over union (IoU) of the elevators identified above. When the image intersection over union exceeds a preset threshold, it can be considered that there are passengers in front of the doors or inside the cars, and further, the remaining space of the elevator can be determined according to the amount of intersection over union. When the number of passengers in front of the doors or inside the cars of the elevator indicated by the elevator occupancy value is large, the remaining space of the elevator is small, and it is difficult for the robot to enter the elevator or the robot will affect the comfort of other passengers after entering the elevator, so the elevator cannot be the target elevator.

[0039] Further, in some embodiments, multiple elevators in the plurality of elevators can simultaneously meet the analysis conditions regarding the elevator door state and the elevator occupancy value, and the closest elevator to the robot can be selected from the multiple elevators as the only target elevator.

[0040] Generally in the use scenario of multiple elevators, a call instruction needs to be first issued, and the elevator responding to the call instruction will reach the current floor scene, open the elevator door for a short time, wait for the caller to enter the elevator, then close the elevator door and run in the direction of the call instruction. The method of the embodiment of the present application can consider the call response mechanism to control and state detect the control and display module related to multiple elevators.

[0041] In some embodiments, the multiple position reference points include one or more position reference points corresponding to each of the multiple elevators and one or more position reference points corresponding to the control and display module in the current floor scene, the control and display module being used to call any of the multiple elevators and display the running state of the multiple elevators. Here, the calling function in the control and display module can be realized by an up or down external call button, and the pressed external call button issues a call instruction from the current floor scene to up or down to each elevator. The running state displayed by the control and display module at least includes whether there is an elevator responding to the call instruction, and further can include the current elevator running direction and the floor scene where the current elevator is located. The display function in the control and display module can be realized by the on-off state of the external call button (for example, lighting or changing from lighting to extinguishing indicating that there is an elevator responding to the call instruction), and can also be realized by the state light corresponding to each elevator (for example, the state light indicating that the elevator responds to the call instruction), and can also be realized by the electronic display screen displaying the running state of the elevator, and the display function can also be realized by the combination of the above multiple devices.

[0042] In some embodiments, the method further includes: before determining the target elevator in the multiple elevators, determining the module expected position of the control and display module in the image acquired by the image sensor based on the multiple sets of secondary coordinate data, and performing dynamic target detection on the image acquired by the image sensor based on the module expected position to obtain the running state of the multiple elevators displayed by the control and display module; when the running state of the multiple elevators displayed by the control and display module indicates that none of the multiple elevators meets the robot's demand for taking the elevator, controlling the robot to perform a corresponding point pressing operation on the control and display module to call any of the multiple elevators, and the demand for taking the elevator is to take the elevator at the current floor scene and go to another floor scene through the elevator.

[0043] Taking the control and display module as an external call button, the on state or the change from the on state to the off state of the external call button indicates that there is an elevator meeting the robot's demand for taking the elevator. Here, if there is an elevator meeting the robot's demand for taking the elevator, i.e., there is an elevator responding to the call instruction of the external call button, the external call button does not need to be repeatedly pressed, and if there is no elevator meeting the demand for taking the elevator, the point pressing operation needs to be performed to issue a call instruction to the elevator.

[0044] In some embodiments, determining the target elevator from the plurality of elevators comprises: when the operation state of the plurality of elevators displayed by the control and display module indicates that there is at least one elevator in the plurality of elevators that meets the boarding demand, determining, from the at least one elevator that meets the boarding demand, an elevator whose door state is in the opening state or the fully open state as the target elevator. If there is no elevator in the elevators that meet the boarding demand whose door state is in the opening state or the fully open state, waiting is maintained and continuous detection is continued until an elevator that meets the boarding demand and whose door state is in the opening state or the fully open state appears.

[0045] In some embodiments, the method further comprises: after the robot is controlled to perform the corresponding point pressing operation on the control and display module, controlling the robot to move to a preset observation point in the current floor scene, which can be located 1 meter directly in front of the call point, so that the image sensor can collect images including the plurality of elevators.

[0046] Figure 2 A structural schematic diagram of a robot according to an embodiment of the present application is shown. As Figure 2 shown, the robot 200 applying the method of the embodiment of the present application comprises: a robot body 201 including a mobile chassis, a controller and a power module; a mechanical arm 202 provided on the robot body, used for performing a point pressing operation on a call button; an image sensor including a button state detector 203 and an elevator door state and passenger state detector 204, the button state detector 203 is implemented by a high dynamic and high precision camera, installed at the fingertip position of the mechanical arm, used for detecting the on-off state of the call button, the elevator door state and passenger state detector 204 includes at least one wide-angle camera, installed on the robot body, used for detecting the elevator door state and the pedestrian state, three wide-angle cameras are used in the embodiment, respectively located at the left side, the right side and the middle position of the robot; a laser radar 205 provided on the robot body, used for distance detection.

[0047] Figure 3 A structural block diagram of the controller of the robot body according to an embodiment of the present application is shown. As Figure 3As shown, the controller 300 in the robot body 201 is used to implement the method of the embodiment of the application, and the controller 300 specifically comprises: a key state detection module 301, configured to determine the key state according to the image of the key state detector 203; an elevator identification module 302, configured to identify the elevator in the image of the elevator door and passenger state detector 204; an elevator door state detection module 303, configured to determine the elevator door state based on the identification result of the elevator identification module 302; a passenger identification module 304, configured to identify the passenger in the image of the elevator door and passenger state detector 204; a SLAM positioning module 305, configured to quickly determine the position of the robot body through the SLAM technology; a master control module 306, configured to generate an execution strategy according to the key state, the elevator door state, the identified elevator door and passenger, and the position of the robot body; a navigation module 307, configured to plan an optimal path according to the execution strategy and control the robot to move according to the path; and a mechanical arm operation module 308, configured to control the mechanical arm to perform corresponding actions according to the execution strategy.

[0048] Figure 4 An example flowchart of the method of guiding the robot to enter the elevator in the multi-elevator scenario according to the embodiment of the application is shown. As shown in FIG. 4, the example flowchart 400 of guiding the robot to enter the elevator by the controller 300 comprises: Figure 4

[0049] (1) Navigate to the call-out point. After receiving the elevator call instruction, call the current floor map (i.e., multiple sets of reference coordinate data), perform autonomous positioning based on the SLAM technology, plan the optimal path to the call-out point through the navigation system, and control the robot body 201 to move along the optimal path. The call-out point is preset to be a position point 0.4-0.5 meters away from the call-out button outside the elevator door and convenient for mechanical arm operation.

[0050] (2) Press the call-out button. Control the mechanical arm 202 to start, and based on the call-out button image collected by the key state detector 203 (resolution 720x1280), use the deep learning target detection algorithm to locate the position coordinates of the up and down buttons. The mechanical arm 202 performs the press operation of the call-out button of the up or down elevator according to the position coordinates to send the call instruction to the elevator. Here, it is assumed that the call-out button is d meters (0.4-0.5 meters) in front of the call-out point and h meters in height, and based on the pose rotation matrix and the translation matrix of the call-out point based on the floor reference coordinate system, the coordinate data of the call-out button in the call-out point coordinate system is converted to the floor reference coordinate system according to the following formula :

[0051] .

[0052] ​(3) Moving to a preset observation point. After the point pressing operation is completed, the robot body 201 is controlled to move along a preset path to a preset observation point. The preset observation point can be located 1 meter directly in front of the calling point, covering multiple elevator doors and the surrounding range in the field of view.

[0053] (4) Real-time obstacle avoidance. During the movement of the robot body 201, real-time obstacle avoidance is performed based on the laser radar 205, the speed is controlled at 0.3 m / s, and the robot is ensured to arrive stably, while the elevator door is always within the detection range of the image sensor, so that the robot can also observe the state of the elevator door during movement.

[0054] (5) Key state detection. The key state detector 203 on the mechanical arm continuously detects the on-off state of the calling key, with a sampling frequency of 10 Hz. The on-off state of the calling key is identified using image recognition technology and matched with the operation of the robot. When a certain elevator door is opened, if the calling key of the previous point pressing operation is not turned off, it is determined that the running direction of the current opened elevator is inconsistent with the required direction; if a certain elevator door is opened and the corresponding key light is turned off, it is confirmed that the elevator is the target elevator in the required direction. When the key light is turned off and the target elevator is not locked (for example, no elevator door is opened at this time), it is considered that this time of calling fails, and the operation of step (2) is performed after moving to the calling point again.

[0055] (6) Elevator door state and passenger state detection. The multiple elevator door state and passenger state detectors 204 are respectively directed towards different elevator doors, so as to simultaneously collect image information of multiple elevator door areas. Based on the expected position of the elevator and the images of all elevator door state and passenger state detectors 204, dynamic target monitoring and passenger target detection are performed.

[0056] (7) Determining a unique target elevator. After the robot triggers the up or down calling key, when it is detected that the elevator door is being opened or completely opened, there is no passenger in front of the elevator door and the elevator is empty, and the calling key is turned off, it is confirmed that the elevator is the target elevator in the required direction. When multiple target elevators appear, the elevator closest to the current robot body 201 is selected as the final target elevator.

[0057] (8) Navigating to the boarding point. Based on the current position of the robot body 201 and the preset boarding point inside the target elevator, the optimal path of the robot from the current position to the elevator door, then into the car and to the boarding point is planned, and the robot body 201 is controlled to move along the optimal path to the boarding point. During the movement, the laser radar 205 scans the inside environment of the car in real time to observe whether it is empty, and the elevator door state and passenger state detector 204 identifies the passenger data in the car. When it is detected that the car is not empty or the number of passengers reaches a threshold, the current boarding is cancelled, and the next boarding operation is performed at a preset avoiding point.

[0058] Figure 5 A schematic diagram of a computer system that can implement the method and apparatus for guiding a robot to take an elevator in a multi-elevator scenario according to embodiments of the application is shown. It should be understood that Figure 5 The computer system 500 shown is only one example of a computer system and should not be construed to limit the scope of the method and apparatus for guiding a robot to take an elevator in a multi-elevator scenario according to embodiments of the application.

[0059] As Figure 5 shown, the computer system 500 can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 502 or programs loaded from a storage device 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the computer system 500 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0060] Generally, the following devices can be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a camera, an accelerometer, a gyroscope, a sensor, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, a motor, an electronic speed controller, etc.; a storage device 508 including, for example, a flash card, etc.; and a communication device 509. The communication device 509 can allow the computer system 500 to communicate with other devices wirelessly or through wires to exchange data. Although Figure 5 The computer system 500 is shown with various devices, but it should be understood that not all of the devices shown are required to implement or possess the method and apparatus for guiding a robot to take an elevator in a multi-elevator scenario according to embodiments of the application. More or fewer devices can alternatively be implemented or possessed. Figure 5 Each block shown in the flowcharts in the description

[0061] In particular, processes described above with reference to the flowcharts can be implemented as a computer program according to some embodiments of the application. For example, a computer readable medium is provided having stored thereon a computer program comprising instructions for performing Figure 1 The program code of the method for guiding a robot to take an elevator in a multi-elevator scenario is shown. In such embodiments, the computer program can be downloaded and installed from a network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functional units defined in the apparatus for guiding a robot to take an elevator in a multi-elevator scenario according to embodiments of the application are implemented.

[0062] Note that a computer readable medium according to an embodiment of the present application can be a computer readable signal medium or a computer readable storage medium or any combination thereof. The computer readable storage medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. The computer readable storage medium of an embodiment of the present application can be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. In addition, a computer readable signal medium can include a computer readable program code carried in a baseband or as part of a carrier wave in a propagated signal, where the computer readable program code can be transmitted or propagated using any suitable medium, including but not limited to wireless media or wired media. The computer readable signal medium of an embodiment of the present application can also be any computer readable medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer readable medium can be transmitted or propagated using any suitable medium, including but not limited to wireless media or wired media, or any suitable combination of the foregoing.

[0063] Computer program code for carrying out operations of embodiments of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0064] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0065] This invention can be implemented in other specific forms without departing from its spirit and essential characteristics. For example, the algorithm described in a particular embodiment can be modified without departing from the basic spirit of the invention. Therefore, the present embodiments are to be regarded as exemplary rather than limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and all changes falling within the meaning and scope of the claims and their equivalents are thus included within the scope of the invention.

Claims

1. A method for guiding a robot to enter an elevator in a multi-elevator scenario, comprising: obtaining a plurality of sets of reference coordinate data of a plurality of position reference points in a current floor scenario provided with a plurality of elevators in a floor reference coordinate system; converting the plurality of sets of reference coordinate data into a plurality of sets of first coordinate data in a robot coordinate system based on a conversion relationship between the robot coordinate system and the floor reference coordinate system, the robot coordinate system being capable of describing a positional relationship between the current floor scenario and a robot body; converting the plurality of sets of first coordinate data into a plurality of sets of second coordinate data in an image sensor coordinate system based on a conversion relationship between the image sensor coordinate system and the robot coordinate system, the image sensor coordinate system being capable of describing a positional relationship between the current floor scenario and an image sensor provided on the robot body; determining an elevator expected position of each of the plurality of elevators in an image acquired by the image sensor based on the plurality of sets of second coordinate data; performing dynamic target detection on the image acquired by the image sensor based on the elevator expected position to obtain an elevator door state of each of the plurality of elevators; and determining a target elevator among the plurality of elevators based on the elevator door state of each of the plurality of elevators and controlling the robot to enter the target elevator based on the plurality of sets of first coordinate data. The determining of the elevator expected position of each of the plurality of elevators in the image acquired by the image sensor comprises:

2. The method of claim 1, wherein, converting the plurality of sets of second coordinate data into a plurality of sets of image coordinate data in a two-dimensional image coordinate system based on a projection mapping relationship between the two-dimensional image coordinate system and the image sensor coordinate system; and determining the elevator expected position of each of the plurality of elevators in the image acquired by the image sensor based on the plurality of sets of image coordinate data. Further comprising:

3. The method of claim 2, wherein, before converting the plurality of sets of second coordinate data into the plurality of sets of image coordinate data, filtering out second coordinate data in the plurality of sets of second coordinate data that is not within an image acquisition range of the image sensor. Further comprising:

4. The method of claim 2, wherein, before converting the plurality of sets of second coordinate data into the plurality of sets of image coordinate data, filtering out second coordinate data in the plurality of sets of second coordinate data that is not in front of the image sensor; before determining the elevator expected position of each of the plurality of elevators in the image acquired by the image sensor based on the plurality of sets of image coordinate data, filtering out image coordinate data in the plurality of sets of image coordinate data that is not within an image framing box of the image acquired by the image sensor. The filtering out of image coordinate data in the plurality of sets of image coordinate data that is not within the image framing box of the image acquired by the image sensor comprises: converting the plurality of sets of image coordinate data into a plurality of sets of pixel sequence values in the image acquired by the image sensor through an intrinsic parameter matrix of the image sensor; filtering out pixel sequence values in the plurality of sets of pixel sequence values that are not within an effective pixel range corresponding to the image framing box of the image acquired by the image sensor; and 5. The method of claim 4, wherein, ​ Determining the elevator expected position of each of the plurality of elevators in the image based on the plurality of sets of image coordinate data comprises determining the elevator expected position of each of the plurality of elevators in the image acquired by the image sensor based on the plurality of sets of pixel sequence values.

6. The method of claim 2, wherein, The projection mapping relationship between the two-dimensional image coordinate system in which the image acquired by the image sensor is located and the image sensor coordinate system is determined by a camera focal length of the image sensor.

7. The method of claim 1, wherein, The plurality of position reference points comprise one or more position reference points corresponding to each of the plurality of elevators and one or more position reference points corresponding to a control and display module arranged in the current floor scene, the control and display module being configured to call any of the plurality of elevators and display the operating status of the plurality of elevators.

8. The method of claim 7, wherein, Further comprising: Before determining the target elevator in the plurality of elevators, determining a module expected position of the control and display module in the image acquired by the image sensor based on the plurality of sets of secondary coordinate data, and performing dynamic target detection on the image acquired by the image sensor based on the module expected position to obtain the operating status of the plurality of elevators displayed by the control and display module; And When the operating status of the plurality of elevators displayed by the control and display module indicates that none of the plurality of elevators meets the elevator riding demand of the robot, controlling the robot to perform a corresponding point press operation on the control and display module to call any of the plurality of elevators, the elevator riding demand being to take an elevator at the current floor scene and go up or down to another floor scene by the elevator.

9. The method of claim 8, wherein, Determining the target elevator in the plurality of elevators comprises: When the operating status of the plurality of elevators displayed by the control and display module indicates that at least one of the plurality of elevators meets the elevator riding demand, determining one of the at least one elevator whose elevator door state is in the process of opening or is completely open as the target elevator.

10. The method of claim 8, wherein, The control and display module is an outbound call button, and the outbound call button is in a lighted state or changes from a lighted state to an extinguished state to indicate that there is an elevator that meets the elevator riding demand of the robot.

11. The method of claim 8, wherein, Further comprising: After controlling the robot to perform a corresponding point press operation on the control and display module, controlling the robot to move to a preset observation point in the current floor scene to enable the image sensor to acquire an image including the plurality of elevators.

12. The method of claim 1, wherein, Determining the target elevator in the plurality of elevators comprises: Performing passenger target detection on the image acquired by the image sensor based on the elevator expected position to obtain an elevator occupancy value indicating the number of passengers in front of the door and in the car of each of the plurality of elevators; and Determining the target elevator based on the elevator door state and the elevator occupancy value of each of the plurality of elevators.

13. An apparatus for guiding a robot to take an elevator in a multi-elevator scene, comprising: a processor; and A memory having computer-executable instructions stored thereon, wherein the computer-executable instructions, when executed by the processor, cause the processor to perform the method of any one of claims 1-12.

14. A computer-readable storage medium having stored thereon computer- executable instructions, wherein, The computer-executable instructions, when executed by the processor, cause the processor to perform the method of any one of claims 1-12.

15. A computer program product comprising computer-executable instructions, wherein, The computer-executable instructions, when executed by the processor, cause the processor to perform the method of any one of claims 1-12.

Images