A calibration-free elevator operation status recognition method supporting multiple perspectives

Abstract

This invention discloses a calibration-free method for identifying the operating status of an elevator that supports multiple perspectives. The method includes the following steps: S1: After enabling recognition, if the device is carried by a human, a voice prompt is issued to the user to align the camera with the elevator panel; if the device is a component of a mobile robot, the robot is notified to rotate the camera toward the elevator panel; S2: At a certain sampling frequency, the camera is controlled to capture photos or videos and continuously send single-frame images to a display recognition model; S3: The display recognition model eliminates interference from advertisements and other factors and outputs screenshots of candidate display screens; S4: If no candidate display screens are found, the method returns to step 1. This method can individually recognize each character, achieving higher recognition accuracy, stronger generalization capabilities, and effectively combating data imbalance. It can also solve the problem of two elevators sharing a single external display screen, thereby improving the accuracy of the method.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a calibration-free elevator operating state recognition method supporting multiple viewing angles. Background Art

[0002] By analyzing images or videos captured inside and outside the elevator car and using the information on the display screen to obtain the elevator's current operating status in real time, this technology is of great significance for developing robots that can automatically ride elevators and assisting visually impaired people in using elevators. However, because cameras may be mounted on mobile robots, held by people, or hung on people, and need to shoot while moving, the images or videos captured may come from multiple perspectives: from outside or inside the elevator, at different heights and angles, and may be subject to shaking.

[0003] The diversity of shooting angles causes the following problems in elevator status recognition:

[0004] 1. The display screen may not be captured from the current viewing angle. In this case, the camera needs to be moved to reselect the viewing angle. To reduce camera movement, a camera with a wider viewing angle is usually selected, but this will make the image larger and increase the difficulty of real-time processing.

[0005] 2. Shooting from different angles can cause display distortion, color aberration, reflections, and occlusions. A shaky viewing angle can cause "ghosting" in the image. These problems can complicate the recognition of the display and its characters. The shooting environments inside and outside elevators vary significantly, making it difficult to use a single method for recognition.

[0006] In addition to the diversity of viewing angles, elevator display screens and on-screen characters are also diverse, which further increases the difficulty of recognition.

[0007] 1. Display screens inside and outside elevators are often different. They range in type from LCD to LED, and in shape from rectangular to rounded-corner rectangular to semicircular. Background colors include black, gray, and blue. Furthermore, captured images often contain a variety of panels, advertisements, and interior decoration, which can interfere with display recognition. Therefore, traditional computer vision methods, such as OpenCV, struggle to recognize these diverse displays. To address this, a common approach involves posting QR codes, checkerboard patterns, and other methods near the display for calibration, which increases the cost of identifying elevator status.

[0008] 2. Display screens have varying character styles. Floor characters may be seven-segment lines or dot-matrix numbers, and up and down arrows may have varying shapes. Furthermore, a single display screen on the elevator's exterior may simultaneously display the operating status of two elevators, requiring differentiation. Display screens may also display characters and patterns unrelated to the elevator, such as dates, times, and advertisements, interfering with elevator status recognition. Consequently, common optical character recognition (OCR) techniques and tools, as well as traditional computer vision methods such as template matching, perform poorly in identifying elevator operating status.

[0009] Finally, due to the limitations of the elevator environment and the real-time requirements, the elevator status recognition function cannot be executed remotely through the network. It must be executed by a removable device carried by a robot or a person. This makes it difficult to ensure the accuracy of identifying the elevator status, and the real-time performance and computing power are also insufficient. Summary of the Invention

[0010] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of this application to avoid obscuring the purpose of this section, the abstract and the title of the invention, and such simplifications or omissions should not be used to limit the scope of the present invention.

[0011] In view of the problems existing in the above-mentioned existing method for identifying the running status of an elevator without calibration and supporting multiple perspectives, the present invention is proposed.

[0012] Therefore, the purpose of the present invention is to provide a calibration-free elevator operation status recognition method that supports multiple perspectives. The method is suitable for solving the problems of difficulty in ensuring the accuracy of elevator status recognition when performed by a movable device carried by a robot or a person, and insufficient real-time performance and computing power.

[0013] To solve the above technical problems, the present invention provides the following technical solution: a calibration-free elevator operation status recognition method supporting multiple perspectives, the recognition method comprising the following steps:

[0014] S1: After the recognition function is turned on, if the device is carried by a person, the user will be prompted by voice to point the camera at the elevator panel; if the device is a component of a mobile robot, the robot will be notified to turn the camera toward the elevator panel;

[0015] S2: Control the camera to take photos or videos at a certain sampling frequency and continuously send single-frame images to the display recognition model;

[0016] S3: The display recognition model eliminates interference such as advertisements and outputs candidate display screenshots;

[0017] S4: If there is no candidate display screen, return to step 1;

[0018] S5: Analyze each candidate display screen in turn and filter out candidate displays that are too small;

[0019] S6: Send the candidate display screen screenshots to the elevator state recognition model;

[0020] S7: Calculate the elevator status based on the arrangement of the characters, including the elevator floor and up and down status;

[0021] S8: After obtaining the elevator status, when the elevator status is initially recognized or changes, the elevator status is output to the outside world: if the device is carried by a person, a voice message is output; if the device is a mobile robot component, the elevator operation status message is released to other components.

[0022] As a preferred solution of the multi-view, calibration-free elevator operation status recognition method described in the present invention, in step S7, since the same external display screen may simultaneously display information of two elevators on the left and right, it is necessary to divide the characters into two groups, one corresponding to the two elevators, and guide the grouping by the arrow position and the character spacing. Specifically, the following method is used to calculate the elevator status based on the arrangement of the characters:

[0023] Step 1: Get the characters and positions on the elevator display screen from the elevator state model;

[0024] Step 2: Sort the characters from left to right according to the position information;

[0025] Step 3: Separate the arrow from the character;

[0026] Step 4: If the number of arrows is 1 and the number of characters is not greater than 2, then all characters are considered as a group;

[0027] Step 5: If the number of arrows is 1 and the number of characters is greater than 2, calculate the distance between the characters and divide the characters into two groups based on the maximum separation;

[0028] Step 6: If the number of arrows is 2, calculate the perpendicular line passing through the center points of the two arrows and divide the characters into two groups on the left and right sides according to the perpendicular line;

[0029] Step 7: Connect the characters in the same group from left to right to get the number of floors;

[0030] Step 8: Output the elevator's operating status, including the elevator floor number and up and down directions.

[0031] As a preferred solution of the multi-view-supporting calibration-free elevator operation status recognition method of the present invention, in step S3, the same image can also have multiple candidate display screens.

[0032] As a preferred solution of the multi-view uncalibrated elevator operation status identification method described in the present invention, in which: in step S5, the candidate display screen is too small, which usually means that either the candidate display screen is not a real floor display screen, or the current viewing angle cannot cover the entire display screen or the display screen is blocked.

[0033] As a preferred solution of the method for uncalibrated elevator operation status recognition supporting multiple perspectives described in the present invention, in step S6, the model eliminates interference from irrelevant information such as advertising text and outputs labels and position information of characters related to the elevator status.

[0034] As a preferred solution of the method for identifying elevator operation status without calibration and supporting multiple perspectives described in the present invention, in step five, it is based on the assumption that ordinary buildings rarely have three-digit floors.

[0035] As a preferred solution of the method for identifying elevator operation status without calibration and supporting multiple perspectives described in the present invention, in step 6, if the number of arrows exceeds 2, it indicates an identification error and no processing is performed.

[0036] As a preferred solution of the method for identifying elevator operation status without calibration and supporting multiple perspectives described in the present invention, in step seven, the direction of the arrow represents the up and down direction of the elevator on the same side of the arrow.

[0037] As a preferred solution of the present invention's method for identifying the operation status of an elevator without calibration and supporting multiple perspectives, the identification method uses a state recognition system, which includes an elevator state recognition model, a camera, a voice module, a camera moving component and a communication module.

[0038] As a preferred solution of the multi-view, calibration-free elevator operation status recognition method described in the present invention, the elevator status recognition model is used to receive image information from a camera and provide the elevator status;

[0039] Camera: provides elevator image information from multiple perspectives;

[0040] Voice module: used only in manual carrying scenarios, prompting users to adjust the camera position to capture elevator images;

[0041] Camera movement component: only used to move the robot scene and adjust the camera position to capture the elevator image;

[0042] Communication module: This module is used only in mobile robot scenarios. It publishes the elevator status as a message to the components of the mobile robot, including the camera movement component, to guide the camera position adjustment or guide the mobile robot to take the elevator.

[0043] The beneficial effects of the present invention are as follows: compared with the traditional computer vision method of manually extracting features, the end-to-end target recognition method based on convolutional neural networks is more universal; compared with the use of general OCR technology to recognize all characters on the elevator display screen, the end-to-end target recognition method based on convolutional neural networks can recognize each character individually, with higher recognition accuracy, stronger generalization ability, and can effectively combat the problem of data imbalance; it can solve the problem of two elevators sharing one external display screen, thereby improving the accuracy of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort. Among them:

[0045] Figure 1 This is a flowchart of a multi-view elevator state recognition method supporting multi-view without calibration proposed by the present invention;

[0046] Figure 2 The present invention proposes a flow chart of calculating the elevator state of a method for recognizing elevator operation states without calibration and supporting multiple perspectives. DETAILED DESCRIPTION

[0047] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0048] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0049] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0050] Furthermore, the present invention is described in detail with reference to schematic diagrams. For ease of illustration, when describing the embodiments of the present invention, cross-sectional views illustrating device structures may be partially enlarged and not to scale. Furthermore, the schematic diagrams are merely illustrative and should not limit the scope of protection of the present invention. Furthermore, in actual production, the three-dimensional dimensions of length, width, and depth should be included.

[0051] Example

[0052] Reference Figure 1-Figure 2 , as an embodiment of the present invention, provides a calibration-free elevator operation status recognition method supporting multiple perspectives, the recognition method comprising the following steps:

[0053] S1: After the recognition function is turned on, if the device is carried by a person, the user will be prompted by voice to point the camera at the elevator panel; if the device is a component of a mobile robot, the robot will be notified to turn the camera toward the elevator panel;

[0054] S2: Control the camera to take photos or videos at a certain sampling frequency and continuously send single-frame images to the display recognition model;

[0055] S3: The display recognition model eliminates interference from advertisements and outputs candidate display screenshots. The same image may have multiple candidate displays.

[0056] S4: If there is no candidate display screen, return to step 1;

[0057] S5: Analyze each candidate display screen in turn and filter out candidate displays that are too small. A candidate display screen that is too small usually means that either the candidate display screen is not a real floor display screen, or the current viewing angle cannot cover the entire display screen, or the display screen is blocked;

[0058] S6: Send the candidate display screen screenshots to the elevator status recognition model, which eliminates irrelevant information such as advertising text and outputs the labels and position information of the characters related to the elevator status;

[0059] S7: Calculate the elevator status based on the arrangement of the characters, including the elevator floor and up and down status;

[0060] In step S7, since the same external display screen may display information for two elevators at the same time, the characters need to be divided into two groups, one for each elevator. The arrow positions and character spacing are used to guide the grouping. Specifically, the following method is used to calculate the elevator status based on the arrangement of the characters:

[0061] Step 1: Get the characters and positions on the elevator display screen from the elevator state model;

[0062] Step 2: Sort the characters from left to right according to the position information;

[0063] Step 3: Separate the arrow from the character;

[0064] Step 4: If the number of arrows is 1 and the number of characters is not greater than 2, then all characters are considered as a group;

[0065] Step 5: If the number of arrows is 1 and the number of characters is greater than 2, calculate the distance between the characters and divide the characters into two groups based on the maximum separation, based on the assumption that ordinary buildings rarely have three-digit floors.

[0066] Step 6: If the number of arrows is 2, calculate the perpendicular line passing through the center points of the two arrows and divide the characters into two groups on the left and right according to the perpendicular line. If the number of arrows exceeds 2, it means that the recognition is wrong and will not be processed;

[0067] Step 7: Connect the characters in the same group from left to right to get the floor number. The direction of the arrow indicates the up and down direction of the elevator on the same side of the arrow.

[0068] Step 8: Output the elevator's operating status, including the elevator floor number and up and down directions;

[0069] S8: After obtaining the elevator status, when the elevator status is initially recognized or changes, the elevator status is output externally: if the device is carried by a person, a voice message is output; if the device is a mobile robot component, the elevator operation status message is released to other components.

[0070] The recognition method uses a state recognition system, which includes an elevator state recognition model, a camera, a voice module, a camera movement component and a communication module.

[0071] Elevator status recognition model: used to receive image information from the camera and give the elevator status;

[0072] Camera: provides elevator image information from multiple perspectives;

[0073] Voice module: only used in manual carrying scenarios, prompting users to adjust the camera position to capture elevator images;

[0074] Camera movement component: only used to move the robot scene and adjust the camera position to capture the elevator image;

[0075] Communication module: This module is used only in mobile robot scenarios and publishes elevator status messages to components of the mobile robot, including the camera movement component, to guide camera position adjustment or instruct the mobile robot to take the elevator.

[0076] During use, compared to the single-perspective elevator status recognition method, we proposed a multi-perspective elevator status recognition method that uniformly supports elevator status recognition from both internal and external perspectives. It supports multi-angle recognition of various display screens and does not require auxiliary means such as QR codes and chessboard diagrams to calibrate the display screen positions. Our method has low computing power requirements and low latency, meeting the real-time requirements for elevator status recognition.

[0077] Compared with traditional computer vision methods that manually extract features, our end-to-end object recognition method based on convolutional neural networks is more versatile. Compared with using general-purpose OCR technology to recognize all characters on an elevator display, our end-to-end object recognition method based on convolutional neural networks can recognize each character individually, with higher recognition accuracy, stronger generalization ability, and can effectively combat data imbalance problems. It can also solve the problem of two elevators sharing a single external display, thereby improving the method's accuracy.

[0078] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A calibration-free elevator operation status recognition method supporting multiple perspectives, characterized in that: The identification method comprises the following steps: S1: After the recognition function is turned on, if the device is carried by a person, the user will be prompted by voice to point the camera at the elevator panel; if the device is a component of a mobile robot, the robot will be notified to turn the camera toward the elevator panel; S2: Control the camera to take photos or videos at a certain sampling frequency and continuously send single-frame images to the display recognition model; S3: The display recognition model eliminates advertising interference and outputs candidate display screenshots; S4: If there is no candidate display screen, return to step 1; S5: Analyze each candidate display screen in turn and filter out candidate displays that are too small; S6: Send the candidate display screen screenshots to the elevator state recognition model; S7: Calculate the elevator status based on the arrangement of the characters, including the elevator floor and up and down status; In step S7, since the same external display screen displays information for the left and right elevators simultaneously, the characters need to be divided into two groups, one for each elevator. The arrow positions and character spacing are used to guide the grouping. Specifically, the following method is used to calculate the elevator status based on the arrangement of the characters: Step 1: Obtain the characters and positions on the elevator display screen from the elevator status recognition model; Step 2: Sort the characters from left to right according to the position information; Step 3: Separate the arrow from the character; Step 4: If the number of arrows is 1 and the number of characters is not greater than 2, then all characters are considered as a group; Step 5: If the number of arrows is 1 and the number of characters is greater than 2, calculate the distance between the characters and divide the characters into two groups based on the maximum separation; In the step 5, it is based on the assumption that ordinary buildings rarely have three-digit floors; Step 6: If the number of arrows is 2, calculate the perpendicular line passing through the center points of the two arrows and divide the characters into two groups on the left and right sides according to the perpendicular line; In step 6, if the number of arrows exceeds 2, it indicates a recognition error and will not be processed; Step 7: Connect the characters in the same group from left to right to get the number of floors; In step 7, the direction of the arrow indicates the up and down direction of the elevator on the same side of the arrow; Step 8: Output the elevator's operating status, including the elevator floor number and up and down directions; S8: After obtaining the elevator status, when the elevator status is initially recognized or changes, the elevator status is output externally: if the device is carried by a person, a voice message is output; if the device is a mobile robot component, the elevator operation status message is released to other components.

2. The method for identifying elevator operation status without calibration and supporting multiple perspectives according to claim 1, characterized in that: In step S3, there are multiple candidate display screens for the same image.

3. The method for identifying elevator operation status without calibration and supporting multiple perspectives according to claim 1, characterized in that: In step S5, the candidate display screen being too small means that the candidate display screen is not a real floor display screen, or the current viewing angle cannot cover the entire display screen, or the display screen is blocked.

4. The method for identifying elevator operation status without calibration and supporting multiple viewpoints according to claim 1, characterized in that: In step S6, the model eliminates the interference of irrelevant information of the advertisement text and outputs the label and position information of the characters related to the elevator status.

5. The method for identifying elevator operation status without calibration and supporting multiple perspectives according to claim 1, characterized in that: The identification method uses a state identification system, which includes an elevator state identification model, a camera, a voice module, a camera moving component and a communication module.

6. The method for identifying elevator operation status without calibration and supporting multiple viewpoints according to claim 5, characterized in that: Elevator status recognition model: used to receive image information from the camera and give the elevator status; Camera: provides elevator image information from multiple perspectives; Voice module: only used in manual carrying scenarios, prompting users to adjust the camera position to capture elevator images; Camera movement component: only used to move the robot scene and adjust the camera position to capture the elevator image; Communication module: This module is used only in mobile robot scenarios and publishes the elevator status as a message to the components of the mobile robot, including the camera movement component, to guide the camera position adjustment or guide the mobile robot to take the elevator.

Images