Uncalibrated elevator running state recognition method supporting multiple view angles

Abstract

The invention discloses an uncalibrated elevator running state recognition method supporting multiple view angles, and the recognition method comprises the following steps: S1, after recognition is started, if equipment is carried manually, a user is reminded to align a camera to an elevator panel through voice; if the equipment is the assembly of the mobile robot, the robot is notified to rotate a camera to face an elevator panel; s2, controlling a camera to take photos or videos at a certain sampling frequency, and continuously sending single-frame images to the display screen identification model; s3, eliminating interference of advertisements and the like by the display screen identification model, and outputting candidate display screen screenshots; and S4, if no candidate display screen exists, returning to the step 1. According to the method, each character can be independently recognized, the recognition accuracy is higher, the generalization ability is higher, and the problem of data imbalance can be effectively solved; the problem that two elevators share one external display screen can be solved, and the accuracy of the method is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a calibration-free elevator operation state recognition method supporting multiple viewing angles. Background Art

[0002] By analyzing the images or videos shot inside and outside the elevator car, the information on the display screen can be used to obtain the current operating status of the elevator in real time, which is of great significance for the development of robots that automatically take elevators and assisting visually impaired people to take elevators. However, since the camera may be installed on a movable robot, held by a person or hung on a person, it needs to be shot while moving, so the images or videos shot may come from multiple perspectives: they may be located outside or inside the elevator, and the height and angle of the shot may be different, and there may be shaking.

[0003] The diversity of shooting angles causes the following problems in elevator status recognition:

[0004] 1. The display screen may not be captured from the current viewing angle. In this case, the camera needs to be moved to reselect the viewing angle. In order to reduce the number of camera movements, a camera with a wider viewing angle is usually selected, but this will make the image larger and increase the difficulty of real-time processing.

[0005] 2. Shooting at different angles will cause distortion, color difference, reflection, occlusion and other problems of the display screen, and shaking angles will cause "ghosting" of the image, all of which will make it difficult to recognize the display screen and the characters on the screen. In particular, the shooting environment inside and outside the elevator is quite different, making it difficult to use the same method for recognition.

[0006] In addition to the diversity of viewing angles, elevator display screens and on-screen characters are also diverse, which further increases the difficulty of recognition.

[0007] 1. The display screens inside and outside the elevator are usually different. The types of display screens include LCD screens, LED screens, etc. The shapes of the display screens include rectangles, rounded rectangles, semicircles, etc. The background colors of the display screens include black, gray, blue, etc. In addition, the captured images often contain a variety of panels, advertisements, interiors, etc., which interfere with the recognition of the display screen. Therefore, traditional computer vision methods such as OpenCV are difficult to recognize diverse display screens. To this end, the usual method is to post QR codes, checkerboards, etc. near the display screen for calibration, which increases the recognition cost of the elevator status.

[0008] 2. The characters on the display screen have different styles. The characters indicating the floor may be seven-segment line numbers, dot matrix numbers, etc., and the shapes of the arrows indicating up and down are diverse. In addition, a display screen outside the elevator may display the operating status of two elevators at the same time, which needs to be distinguished. The display screen may display characters and patterns unrelated to the elevator, such as date, time, and advertisements, which interfere with the recognition of the elevator status. Therefore, common OCR technologies and tools and traditional computer vision methods such as template matching perform poorly in identifying the operating status of elevators.

[0009] Finally, due to the limitations of the elevator environment and the real-time requirements, the elevator status recognition function cannot be executed remotely through the network, but must be executed by a removable device carried by a robot or a person. It is difficult to ensure the accuracy of identifying the elevator status, and the real-time performance and computing power are also insufficient. Summary of the invention

[0010] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0011] In view of the problems existing in the above-mentioned existing method for identifying the running status of an elevator without calibration and supporting multiple perspectives, the present invention is proposed.

[0012] Therefore, the purpose of the present invention is to provide a calibration-free elevator operation status recognition method that supports multiple perspectives, which is suitable for solving the problem that it is difficult to ensure the accuracy of elevator status recognition when it is performed by a movable device carried by a robot or a person, and the real-time performance and computing power are also insufficient.

[0013] In order to solve the above technical problems, the present invention provides the following technical solutions: a calibration-free elevator operation status recognition method supporting multiple perspectives, the recognition method comprising the following steps:

[0014] S1: After the recognition is turned on, if the device is carried by a person, the user will be reminded by voice to point the camera at the elevator panel; if the device is a component of a mobile robot, the robot will be notified to turn the camera toward the elevator panel;

[0015] S2: Control the camera to take photos or videos at a certain sampling frequency, and continuously send single-frame images to the display recognition model;

[0016] S3: The display screen recognition model eliminates interference from advertisements and outputs candidate display screen screenshots;

[0017] S4: If there is no candidate display screen, return to step 1;

[0018] S5: Analyze each candidate display screen in turn and filter out candidate display screens that are too small;

[0019] S6: Send the candidate display screen screenshots to the elevator state recognition model;

[0020] S7: Calculate the elevator status according to the arrangement of characters, including the elevator floor and up and down status;

[0021] S8: After obtaining the elevator status, when the elevator status is first recognized or changes, the elevator status is output to the outside: if the device is carried by a person, a voice message is output; if the device is a mobile robot component, the elevator operation status message is published to other components.

[0022] As a preferred solution of the method for identifying the operation status of an elevator without calibration and supporting multiple viewpoints described in the present invention, in step S7, since the same external display screen may display information of two elevators on the left and right at the same time, it is necessary to divide the characters into two groups on the left and right, corresponding to the two elevators respectively, and the grouping is guided by the arrow position and the character interval. Specifically, the following method is used to calculate the elevator status according to the arrangement of the characters:

[0023] Step 1: Get the characters and positions on the elevator display screen from the elevator status model;

[0024] Step 2: Sort the characters from left to right according to the position information;

[0025] Step 3: Separate the arrow from the character;

[0026] Step 4: If the number of arrows is 1 and the number of characters is not greater than 2, all characters are considered as a group;

[0027] Step 5: If the number of arrows is 1 and the number of characters is greater than 2, calculate the distance between the characters and divide the characters into left and right groups according to the maximum isolation;

[0028] Step 6: If the number of arrows is 2, calculate the vertical line passing through the center points of the two arrows and divide the characters into two groups on the left and right according to the vertical line;

[0029] Step 7: Connect the characters in the same group from left to right to get the number of floors;

[0030] Step 8: Output the operating status of the elevator, including the number of elevator floors and up and down directions.

[0031] As a preferred solution of the method for identifying elevator operation status without calibration and supporting multiple viewing angles described in the present invention, in step S3, the same image can also have multiple candidate display screens.

[0032] As a preferred solution of the method for identifying elevator operation status without calibration and supporting multiple perspectives described in the present invention, in which: in step S5, the candidate display screen is too small, which usually means that either the candidate display screen is not a real floor display screen, or the current perspective cannot cover the entire display screen or the display screen is blocked.

[0033] As a preferred solution of the method for uncalibrated elevator operation status recognition supporting multiple perspectives described in the present invention, in step S6, the model eliminates interference from irrelevant information such as advertising text, and outputs labels and position information of characters related to the elevator status.

[0034] As a preferred solution of the method for identifying elevator operation status without calibration and supporting multiple perspectives described in the present invention, in step five, it is based on the assumption that ordinary buildings rarely have three-digit floors.

[0035] As a preferred solution of the method for identifying the operation status of an elevator without calibration and supporting multiple perspectives described in the present invention, in step six, if the number of arrows exceeds 2, it indicates an identification error and no processing is performed.

[0036] As a preferred solution of the method for identifying the operation status of an elevator without calibration and supporting multiple perspectives described in the present invention, in step seven, the direction of the arrow indicates the up and down directions of the elevator on the same side of the arrow.

[0037] As a preferred solution of the method for uncalibrated elevator operation status recognition supporting multiple perspectives described in the present invention, the recognition method uses a status recognition system, which includes an elevator status recognition model, a camera, a voice module, a camera moving component and a communication module.

[0038] As a preferred solution of the method for identifying the operation status of an elevator without calibration and supporting multiple viewpoints described in the present invention, the elevator status recognition model is used to receive the image information of the camera and give the elevator status;

[0039] Camera: provides elevator image information from multiple perspectives;

[0040] Voice module: only used in manual carrying scenarios, prompting users to adjust the camera position to capture elevator images;

[0041] Camera moving component: only used in the mobile robot scene to adjust the camera position to capture the elevator image;

[0042] Communication module: Only used in mobile robot scenarios, it publishes the elevator status as a message to the components of the mobile robot, including the camera movement component, to guide the camera position adjustment or guide the mobile robot to take the elevator.

[0043] The beneficial effects of the present invention are as follows: compared with the traditional computer vision method of manually extracting features, the end-to-end target recognition method based on the convolutional neural network is more universal; compared with the use of general OCR technology to recognize all characters on the elevator display screen, the end-to-end target recognition method based on the convolutional neural network can recognize each character individually, with higher recognition accuracy, stronger generalization ability, and can effectively combat the problem of data imbalance; it can solve the problem of two elevators sharing one external display screen, and improve the accuracy of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0045] Figure 1 A flowchart of multi-view elevator state recognition of a method for identifying elevator operation states without calibration and supporting multiple views proposed by the present invention;

[0046] Figure 2 The present invention provides a flowchart of calculating the elevator state of a method for identifying elevator operation states without calibration and supporting multiple perspectives. DETAILED DESCRIPTION

[0047] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0048] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0049] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0050] Secondly, the present invention is described in detail with reference to the schematic diagram. When describing the embodiments of the present invention in detail, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0051] Example

[0052] Reference Figure 1-Figure 2 , is an embodiment of the present invention, and provides a calibration-free elevator operation status recognition method supporting multiple perspectives, the recognition method comprising the following steps:

[0053] S1: After the recognition is turned on, if the device is carried by a person, the user will be reminded by voice to point the camera at the elevator panel; if the device is a component of a mobile robot, the robot will be notified to turn the camera toward the elevator panel;

[0054] S2: Control the camera to take photos or videos at a certain sampling frequency, and continuously send single-frame images to the display recognition model;

[0055] S3: The display recognition model eliminates interference from advertisements and outputs candidate display screenshots. The same image may have multiple candidate displays.

[0056] S4: If there is no candidate display screen, return to step 1;

[0057] S5: Analyze each candidate display screen in turn and filter out candidate display screens that are too small. A candidate display screen that is too small usually means that either the candidate display screen is not a real floor display screen, or the current viewing angle cannot cover the entire display screen or the display screen is blocked;

[0058] S6: Send the candidate display screen screenshots to the elevator status recognition model, which excludes the interference of irrelevant information such as advertising text and outputs the labels and position information of the characters related to the elevator status;

[0059] S7: Calculate the elevator status according to the arrangement of characters, including the elevator floor and up and down status;

[0060] In step S7, since the same external display screen may display information of two elevators at the same time, the characters need to be divided into two groups, corresponding to the two elevators respectively, and the grouping is guided by the arrow position and the character interval. Specifically, the following method is used to calculate the elevator status according to the arrangement of the characters:

[0061] Step 1: Get the characters and positions on the elevator display screen from the elevator status model;

[0062] Step 2: Sort the characters from left to right according to the position information;

[0063] Step 3: Separate the arrow from the character;

[0064] Step 4: If the number of arrows is 1 and the number of characters is not greater than 2, all characters are considered as a group;

[0065] Step 5: If the number of arrows is 1 and the number of characters is greater than 2, the distance between characters is calculated and the characters are divided into left and right groups according to the maximum isolation, which is based on the assumption that ordinary buildings rarely have three-digit floors;

[0066] Step 6: If the number of arrows is 2, calculate the vertical line passing through the center points of the two arrows, and divide the characters into two groups on the left and right according to the vertical line. If the number of arrows exceeds 2, it means that the recognition is wrong and will not be processed;

[0067] Step 7: Connect the characters in the same group from left to right to get the floor number. The direction of the arrow indicates the up and down direction of the elevator on the same side of the arrow.

[0068] Step 8: Output the running status of the elevator, including the number of elevator floors and up and down directions;

[0069] S8: After obtaining the elevator status, when the elevator status is first recognized or changes, the elevator status is output to the outside: if the device is carried by a person, a voice message is output; if the device is a mobile robot component, the elevator operation status message is published to other components.

[0070] The recognition method uses a state recognition system, which includes an elevator state recognition model, a camera, a voice module, a camera moving component and a communication module.

[0071] Elevator status recognition model: used to receive image information from the camera and give the elevator status;

[0072] Camera: provides elevator image information from multiple perspectives;

[0073] Voice module: only used in manual carrying scenarios, prompting users to adjust the camera position to capture elevator images;

[0074] Camera moving component: only used in the mobile robot scene to adjust the camera position to capture the elevator image;

[0075] Communication module: only used in mobile robot scenarios, it publishes the elevator status as a message to the components of the mobile robot, including the camera movement component, to guide the camera position adjustment or guide the mobile robot to take the elevator;

[0076] During use, compared with the single-view elevator status recognition method, we proposed a multi-view elevator status recognition method that uniformly supports elevator status recognition from both the internal and external perspectives of the elevator; it supports multi-angle recognition of multiple display screens, and does not require auxiliary means such as QR codes and chessboards to calibrate the display screen position; our method has low computing power requirements and low latency, meeting the real-time requirements for identifying elevator status;

[0077] Compared with traditional computer vision methods that manually extract features, our end-to-end object recognition method based on convolutional neural networks is more general; compared with the use of general OCR technology to recognize all characters on the elevator display, our end-to-end object recognition method based on convolutional neural networks can recognize each character individually, with higher recognition accuracy, stronger generalization ability, and can effectively combat the problem of data imbalance; it can solve the problem of two elevators sharing one external display, thereby improving the accuracy of the method.

[0078] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A calibration-free elevator operation status recognition method supporting multiple perspectives, characterized in that: The identification method comprises the following steps: S1: After the recognition is turned on, if the device is carried by a person, the user will be reminded by voice to point the camera at the elevator panel; if the device is a component of a mobile robot, the robot will be notified to turn the camera toward the elevator panel; S2: Control the camera to take photos or videos at a certain sampling frequency, and continuously send single-frame images to the display recognition model; S3: The display screen recognition model eliminates interference from advertisements and outputs candidate display screen screenshots; S4: If there is no candidate display screen, return to step 1; S5: Analyze each candidate display screen in turn and filter out candidate display screens that are too small; S6: Send the candidate display screen screenshots to the elevator state recognition model; S7: Calculate the elevator status according to the arrangement of characters, including the elevator floor and up and down status; S8: After obtaining the elevator status, when the elevator status is first recognized or changes, the elevator status is output to the outside: if the device is carried by a person, a voice message is output; if the device is a mobile robot component, the elevator operation status message is published to other components.

2. The method for identifying elevator operation status without calibration and supporting multiple perspectives according to claim 1, characterized in that: In step S7, since the same external display screen may display information of two elevators on the left and right at the same time, the characters need to be divided into two groups on the left and right, corresponding to the two elevators respectively, and the grouping is guided by the arrow position and the character interval. Specifically, the following method is used to calculate the elevator status according to the arrangement of the characters: Step 1: Get the characters and positions on the elevator display screen from the elevator status model; Step 2: Sort the characters from left to right according to the position information; Step 3: Separate the arrow from the character; Step 4: If the number of arrows is 1 and the number of characters is not greater than 2, all characters are considered as a group; Step 5: If the number of arrows is 1 and the number of characters is greater than 2, calculate the distance between the characters and divide the characters into left and right groups according to the maximum isolation; Step 6: If the number of arrows is 2, calculate the vertical line passing through the center points of the two arrows and divide the characters into two groups on the left and right according to the vertical line; Step 7: Connect the characters in the same group from left to right to get the number of floors; Step 8: Output the operating status of the elevator, including the number of elevator floors and up and down directions.

3. The method for identifying elevator operation status without calibration and supporting multiple perspectives according to claim 1, characterized in that: In step S3, the same image may have multiple candidate display screens.

4. The method for identifying elevator operation status without calibration and supporting multiple perspectives according to claim 1, characterized in that: In step S5, the candidate display screen being too small usually means that either the candidate display screen is not a real floor display screen, or the current viewing angle cannot cover the entire display screen or the display screen is blocked.

5. The method for identifying elevator operation status without calibration and supporting multiple perspectives according to claim 1, characterized in that: In step S6, the model eliminates interference from irrelevant information such as advertising text, and outputs labels and position information of characters related to the elevator status.

6. The method for identifying elevator operation status without calibration and supporting multiple perspectives according to claim 2, characterized in that: In the step 5, it is based on the assumption that ordinary buildings rarely have three-digit floors.

7. The method for identifying elevator operation status without calibration and supporting multiple perspectives according to claim 2, characterized in that: In step six, if the number of arrows exceeds 2, it indicates a recognition error and will not be processed.

8. The method for identifying elevator operation status without calibration and supporting multiple perspectives according to claim 2, characterized in that: In step seven, the direction of the arrow indicates the up and down directions of the elevator on the same side of the arrow.

9. The method for identifying elevator operation status without calibration and supporting multiple perspectives according to claim 1, characterized in that: The identification method uses a state identification system, which includes an elevator state identification model, a camera, a voice module, a camera moving component and a communication module.

10. The method for identifying elevator operation status without calibration and supporting multiple viewing angles according to claim 9, characterized in that: Elevator status recognition model: used to receive image information from the camera and give the elevator status; Camera: provides elevator image information from multiple perspectives; Voice module: only used in manual carrying scenarios, prompting users to adjust the camera position to capture elevator images; Camera moving component: only used in the mobile robot scene to adjust the camera position to capture the elevator image; Communication module: Only used in mobile robot scenarios, it publishes the elevator status as a message to the components of the mobile robot, including the camera movement component, to guide the camera position adjustment or guide the mobile robot to take the elevator.

Images