Contactless elevator service based on enhanced dataset

Abstract

A system, method, and apparatus for real-time contactless elevator service operation for an elevator includes a trained neural network (TNN) model. The TNN model is trained using a training processor with an augmented data set as a synthetic training data set to later perform elevator identifier recognition. An augmented data set is generated from a synthetic text image, which is augmented to a predetermined number of variations in appearance of a training elevator identifier set with different geometric parameters and visual parameters. A camera captures a user image. A text image portion from the user image is extracted using the TNN model, and an elevator identifier in the extracted text image portion is detected using the extracted text image portion and the TNN model. The detected elevator identifier is displayed for user confirmation or user cancellation, and upon user confirmation, a control command is generated based on the detected elevator identifier associated with the elevator service.

Description

Technical Field

[0001] The present disclosure relates generally to visual recognition systems and, more particularly, to visual recognition systems that enable elevator service without touching elevator panels / buttons. Background Art

[0002] Traditional elevator call buttons utilize micro-stroke mechanical switches, other traditional methods use proximity sensing, and other traditional methods use optical patterns. However, traditional call buttons are complex mechanical structures with a large number of parts, which are expensive to manufacture compared to the functions they serve, and are subject to mechanical wear. These traditional buttons have ridges and gaps, which not only accumulate dirt and bacteria, but also make them difficult to clean. The above problems are exacerbated in environments such as doctors' offices and hospitals, and in factories where work is carried out, and bacteria and dirt are thus transferred from one passenger to other passengers via elevator buttons. Traditional buttons require some type of human contact, even if it is the so-called proximity sensing type. In order to make an elevator call, some type of human contact is required, that is, the thumb, index finger or other human body parts of equal size must actually be aligned directly on the button. This can be extremely difficult when a person is carrying groceries or other packages and the view of his or her hands is blocked, or when the fingers are not free. Similarly, for the visually impaired group, locating the exact position of the call button so that the person presses or touches the elevator button is a very difficult task. Buttons with moving parts are also very expensive to maintain and can break due to vandalism and abuse.Another problem faced by conventional elevator buttons is that the ability to customize the aesthetic appearance of the elevator car call button to fit the building design can be extremely difficult when it becomes necessary to change its mechanical structure.

[0003] When the activation of the circuit is to be achieved without the use of a manual switch, a conventional non-personal contact electrical switch is used. These types of conventional manual switches are a source of transmission of infectious diseases or transmission of harmful substances to people. An infected or contaminated person may press a particular conventional elevator switch at one point in time, and at a later point in time, another person may be infected or contaminated by manually touching the particular elevator switch. Therefore, in order to prevent infection or contamination of people using switches in public environments, it would be desirable to provide a non-personal contact electrical switch in a public environment. Other public environments using conventional manual switches include light switches in public buildings.

[0004] Some conventional impersonal contact electrical switches depend on proximity to an electrical conductor, and a change in capacitance enables the switch. These conventional capacitance-based impersonal contact electrical switches are not practical in public environments. This is because, for example, one person may wear gloves and another person may not wear gloves. It may be difficult to account for capacitance changes between a person wearing gloves and another person not wearing gloves in capacitance-based impersonal contact electrical switches. In this regard, there is a need in the field of elevator contactless technology for an impersonal contact method that can be used in public environments and does not rely on capacitance sensing technology.

[0005] Another conventional elevator button-related technology in non-personal contact electrical switches is based on the interruption of a sensing light beam. This conventional technology does not depend on whether a person is wearing gloves. In this regard, there is a need for a non-personal contact method for a public environment in the field of elevator contactless technology, which uses a non-contact-enabled technology based on an elevator button. For example, U.S. Patent No. 4,042,067 discloses an elevator system using a capacitance-based proximity switch or a touch contact switch. However, as described above, some problems associated with capacitance-based switches and touch contact switches for switches used in public environments are problematic. U.S. Patent No. 4,716,992 discloses an operating panel device for an elevator system, which includes a two-dimensional matrix array of six light emitters and six light receivers to provide switches for eight switch selections. However, there are multiple problems with this conventional method, such that if only one of the six light emitters or one of the six light receivers fails, the entire system is endangered. For example, depending on whether a switch selection of a row or column in the two-dimensional matrix fails, the failure may cause two or four switch selections to be disabled. To avoid multiple switch failures if only one phototransmitter or one photoreceiver fails, it would be desirable to provide non-personal contact electrical switches in public environments such as elevators that do not employ a two-dimensional matrix array of phototransmitters and photoreceivers to actuate multiple switches.

[0006] The present disclosure addresses the need for elevator contactless elevator button technology in today's elevator industry and other related technology industries by solving the traditional problems of contactless operation of elevator systems by producing a visual recognition system that can enable elevator service without touching the elevator panel / button. Summary of the invention

[0007] The present disclosure relates generally to visual recognition systems and, more particularly, to visual recognition systems that enable elevator service without touching elevator panels / buttons.

[0008] Some embodiments of the present disclosure use a visual recognition system to enable elevator service without touching the elevator panel / button. The user displays the letter and / or number of the destination floor to a camera associated with the elevator to enable the corresponding elevator button. A letter / number classifier and a user confirmation / cancel mechanism are provided so that the visual recognition system is trained using synthesized training data, eliminating the need to manually collect training data. The training data can be synthesized with a range of customized categories in terms of font color, perspective transformation, position, scale, font style, noise modeling, etc. to make the trained visual recognition system more robust with respect to these categories.

[0009] However, in order to construct the embodiments of the present disclosure, the experiments included many failed experimental methods, from which many valuable insights were gained. For example, some important components of these test experimental methods that are considered important for developing robust touchless elevator services include: (1) reliably identifying the destination floor of the elevator user without imposing many prerequisites on the user and / or the building where the elevator is deployed. For example, some prerequisites may include installing a specific application on the user's mobile device or a strong wifi signal in the elevator; (2) training the recognition system at low cost, including the cost associated with collecting training data for training the recognition system. Among them, at least one insight is that the embodiments of the present disclosure require a recognition system that robustly identifies the user's destination floor with minimal cost and manual effort, while minimizing the prerequisites on the user and / or the building where the elevator is deployed.

[0010] In order to develop these embodiments of the present disclosure, it was necessary to better understand some of the problems that needed to be overcome to achieve the goal of developing a robust touchless elevator service. For example, some of these recognized problems include finding methods to train and deploy a recognition system for a touchless elevator service that can robustly recognize a user's destination floor with minimal cost and manual effort while minimizing the prerequisites for the user and / or the building where the elevator is deployed. After a good understanding of some of the problems to be overcome, a number of test experimental methods were tested in order to develop embodiments of the present disclosure.

[0011] From extensive experiments, some embodiments of the present disclosure include a solution that includes a user can enter the floor number from a dial pad or any text-based cell phone application (e.g., notepad, Google Docs, text messaging, calculator, etc.), where the floor text may include L, LL, B1, etc. and a number, so that the user then presents the phone screen (or any artifact at hand that includes the floor text) to the camera.

[0012] Another aspect recognized from extensive experimentation is the realization that none of the experimental approaches tested provide a mechanism for the user to confirm or cancel their initial input in the event that the recognition system misinterpreted the user's initial input, or the user may have changed their mind (i.e., the user wants to deactivate an already enabled button). It is believed that the cause of this particular problem is that these experimental approaches do not take into account at least one of the following:

[0013] (1) The burdens these experimental methods impose on elevator users and the buildings in which they are deployed;

[0014] (2) the overall cost of building a reliable identification system for touchless elevator services; and

[0015] (3) These experimental methods fail to provide users with a mechanism to confirm or cancel their input in the recognition system.

[0016] Some embodiments of the present disclosure address the above recognition by developing a method including an apparatus for contactless elevator service operation of an elevator. The apparatus may include an image sensor or multiple sensors to capture images from a user. A memory is used to store data, wherein the data may include a training neural network model and elevator training data. The elevator training data may have elevator identifiers of a set of elevators and a set of elevator templates associated with the elevator training data. Such that each elevator includes a set of elevator identifiers associated with an elevator service corresponding to the elevator template. In addition, the apparatus may include a control circuit for generating a synthetic text image for each training elevator identifier in a training elevator identifier set corresponding to a training elevator template in a training elevator template set. Wherein, each synthetic text image in the synthetic text image is a variant of at least one typeface parameter of at least one training elevator identifier in the training elevator identifier set. The control circuit may also generate an enhanced data set of the synthetic text image by image transformation of the synthetic text image of each training elevator identifier in the training elevator identifier set corresponding to the training elevator template. Wherein, the synthetic text image is enhanced to a predetermined number of variants in appearance of the training elevator identifier set with different geometric parameters and visual parameters. The enhanced data set may be used as a synthetic training data set to train the neural network model. The neural network model is trained to perform elevator identifier recognition. After training, the trained neural network model can perform elevator identifier recognition directly on the captured image without the need for an enhanced data set. In other words, after the trained neural network model is deployed, no enhanced data set is required. Then, the trained neural network model can extract a text image portion from the captured image. To detect at least one elevator identifier in the captured image based on the extracted text image portion and the trained neural network model. Wherein, the extracted text image portion includes a training elevator identifier from a training elevator identifier set in the elevator training data. The display can be used to display the detected elevator identifier on the display for user confirmation or user cancellation, and when the user confirms, the control circuit can generate a control command based on the detected elevator identifier associated with the elevator service. Wherein, the control circuit can output the control command so as to enable or disable the elevator service based on the control command.

[0017] The experiment provides knowledge depth to assist in developing some embodiments of the present disclosure, and assist in better understanding some problems in traditional contactless methods. For example, some reasons why some embodiments of the present disclosure can solve problems in traditional contactless methods are that some embodiments of the present disclosure do not require users to install specific QR code generation applications on their mobile devices. Among them, these embodiments do not require the user's mobile device or building to access or provide wifi so that the specific QR code generation application can be accessed.

[0018] Another reason why some embodiments of the present disclosure can solve the problems in the traditional contactless method is that some embodiments of the present disclosure do not require the user to know and find the correspondence between the destination floor and the gesture. Among them, these embodiments also do not require the user to perform any gesture other than displaying the floor text to the camera. In addition, displaying the floor text to the camera does not take up a lot of space or require a lot of space, which happens to make these embodiments more practical, especially when the elevator is crowded or close to full.

[0019] Another reason why some embodiments of the present disclosure can solve the problems in the traditional contactless method is due to the number of sensors required to operate these embodiments. For example, a sensor such as a camera is required, and more cameras can be imagined, however, in order to operate these embodiments, a single camera is sufficient. In addition, the number of sensors of these embodiments of the present disclosure is independent of the number of floors, making these embodiments easier to expand to large-scale commercial applications, such as skyscrapers with a large number of floors. In addition, due to the confirmation mechanism characteristics, compared with traditional contactless elevator methods, users are less likely to accidentally enable other unrelated floor buttons when using these embodiments. In fact, these embodiments of the present disclosure provide users with a mechanism to confirm or cancel their input when the user changes his mind or the recognition system (i.e., classifier) ​​misclassifies the user input.

[0020] experiment

[0021] As described above, several aspects were learned from the experiments, including that none of the test experiments provided a way for the user to confirm or cancel their input in the event that the recognition system misinterpreted the user's input or in the event that the user changed their mind (the user wanted to deactivate an already enabled button). It was later recognized that the reason for this problem may be that in conducting these test experiments or in developing these experiments, at least one of the following was considered: (1) the burden that the test experiments imposed on elevator users and buildings where elevators were deployed; (2) the amount of cost to construct a reliable recognition system for touchless elevator service; and (3) the way or mechanism incorporated into the recognition system to provide or allow users to confirm or cancel their input.

[0022] For example, some test experimental methods include a touchless elevator ride that requires the user to install a specific QR code generating application on the user's personal mobile device (either offline or on-site via wifi) to use a QR code reader to identify a QR code associated with the destination floor presented by the user. However, there are many problems with this experimental method, including that if the user does not have a mobile device on which the specific application can be installed, or there is no wifi connection / weak wifi connection on-site (i.e., the building does not provide wifi or any connection issues on the user's mobile device), the user cannot operate the elevator touchlessly. Therefore, no further testing or research has been conducted on this test experimental method.

[0023] Another test experimental method includes developing touchless gesture recognition for elevator services, which requires the user to know the unique gesture corresponding to the specific elevator service (i.e., requesting the destination floor) and to perform this unique gesture associated with the destination floor within the limited space of the elevator. Among them, the unique gesture performed can be recognized by the gesture recognition system and enable the corresponding elevator service. However, many key problems of this experimental method were later recognized, namely, due to the limited space in the elevator, it was problematic for users to perform unique gestures when the elevator was close to half or full. In fact, users found it very difficult to try to figure out how to perform the unique gesture correctly, or even failed to perform the correct unique gesture at all. Other major problems recognized by this test experimental method include that disabled or injured users cannot perform unique gestures, or that users carrying personal items cannot place their personal items on the floor due to the large user capacity in the elevator. Based on many recognized problems, this experimental solution leads to impracticality and, for example, for hygiene reasons, may not achieve the purpose of touchless elevator service.

[0024] Another experimental approach included developing a contactless button for an elevator that required ultrasonic and infrared proximity sensors, wherein the installed buttons represented buttons associated with elevator services. For example, a user could enable a particular button by moving close to a corresponding proximity sensor. However, it was recognized that problems that could not be overcome by this particular test experiment included implementation costs and the inability of users to use the method. For example, since the number of proximity sensors required to operate the experimental approach was proportional to the number of elevator buttons, this proved to be not easily scalable and resulted in a very high cost to implement the solution. It was also recognized that this particular experimental approach was impractical when the elevator was crowded due to users in front of the proximity sensors accidentally enabling more buttons than the users intended. Therefore, for these and other reasons, no further testing of this particular test experimental approach was performed.

[0025] Practical Application

[0026] Some practical applications of embodiments of the present disclosure may include contactless elevator operation for elevators, including but not limited to elevators for passengers and / or cargo / food.

[0027] According to an embodiment of the present disclosure, a device for real-time contactless elevator service operation of an elevator. The device includes a memory to store data. The data includes a training neural network (TNN) model. The TNN model is trained via a training processor using an enhanced data set as a synthetic training data set to perform elevator identifier recognition. An enhanced data set is generated from a synthetic text image by image transformation of a synthetic text image of each training elevator identifier in a training elevator identifier set of an elevator corresponding to a training elevator template in a training elevator template set. The synthetic text image is enhanced to a predetermined number of variations in appearance of the training elevator identifier set with different geometric parameters and visual parameters. An image sensor captures one or more real-time images from a user. The processor extracts a text image portion from the captured one or more images using a TNN model in real time. At least one elevator identifier in the extracted text image portion is detected using the extracted text image portion and the TNN model. The detected elevator identifier is displayed on a display for user confirmation or user cancellation, and when the user confirms, a control command is generated based on the detected elevator identifier associated with the elevator service. The control command is output to enable or disable the elevator service based on the control command.

[0028] According to another embodiment of the present disclosure, a device for real-time contactless elevator service operation of an elevator. The device including a control circuit is configured to access a training neural network (TNN) model from a cloud computer via a communication network connected to the device, and store the TNN in a memory. Wherein, the TNN uses a training processor to train with elevator training data. The elevator training data includes an elevator training identifier of a training elevator set and an elevator training template set associated with the elevator training data, and each training elevator in the training elevator set includes a training elevator identifier set associated with an elevator training service corresponding to the elevator training template. One or more user images are received in real time via an image sensor. A text image portion is extracted from one or more user images using a TNN model. At least one elevator identifier in one or more user images is detected based on the extracted text image portion and the TNN model. The detected elevator identifier is displayed on a display for user confirmation or user cancellation, and when the user confirms, a control command is generated based on the detected elevator identifier associated with the elevator service. The control command is output to enable or disable the elevator service based on the control command.

[0029] According to another embodiment of the present disclosure, a method for real-time contactless elevator service operation for an elevator. The method includes: accessing a memory storing data, the stored data including a training neural network (TNN) model. Wherein, the TNN model is trained via a training processor using an enhanced data set as a synthetic training data set to perform elevator identifier recognition. An enhanced data set is generated from a synthetic text image by image transformation of a synthetic text image of each training elevator identifier in a training elevator identifier set of an elevator corresponding to a training elevator template in a training elevator template set. Wherein, the synthetic text image is enhanced to a predetermined number of variations in appearance of the training elevator identifier set with different geometric parameters and visual parameters. One or more user images are received in real time. A text image portion is extracted from the captured one or more user images using a TNN model. At least one elevator identifier in the captured one or more user images is detected based on the extracted text image portion and the TNN model. The detected elevator identifier is displayed on a display for user confirmation or user cancellation, and when the user confirms, a control command is generated based on the detected elevator identifier associated with the elevator service. The control command is output to enable or disable the elevator service based on the control command.

[0030] According to another embodiment of the present disclosure, a device for contactless transportation service operation for a transportation system operated in real time, the transportation system being associated with one of a food elevator, a moving sidewalk or a wheelchair elevator. The device includes a transceiver that receives a training neural network (TNN) model from a cloud-based computer and stores the TNN model in a memory. Wherein, the TNN model is trained by a cloud-based computer using an enhanced data set as a synthetic training data set so as to perform transportation identifier recognition later after the training is completed. An enhanced data set is generated from a synthetic text image by image transformation of a synthetic text image of each training transportation identifier in a training transportation identifier set of a transportation system corresponding to a training transportation template in a training transportation template set. Wherein, the synthetic text image is enhanced to a predetermined number of variations in appearance of the training transportation identifier set with different geometric parameters and visual parameters. The user guidance mechanism has a display to assist the user in locating the text information within a target area of ​​the image sensor before the image sensor captures the user image. Wherein, the user locates the text image in the target area, and then the image sensor captures one or more real-time user images. The processor extracts the text image portion from the captured one or more real-time user images using the TNN model in real time. At least one transport identifier in a text image portion extracted from one or more captured real-time user images is detected based on the extracted text image portion and the TNN model. The detected transport identifier is displayed on a display for user confirmation or user cancellation, and upon user confirmation, a control command is generated based on the detected transport identifier associated with the transport service. The control command is output to enable or disable the transport service based on the control command. Wherein the transport system is associated with a food elevator, an escalator, a moving sidewalk, a wheelchair lift, etc.

[0031] According to another embodiment of the present disclosure, a non-transitory machine-readable medium including instructions stored thereon, the instructions configuring the processing circuit to perform operations for real-time contactless elevator service operations for elevators when executed by a processing circuit. The non-transitory machine-readable medium includes: accessing a memory storing data, the stored data including a training neural network (TNN) model; and storing the TNN model in the memory. Wherein, the TNN model is trained using a training processor, and the training processor uses an enhanced data set as a synthetic training data set to generate a distribution of classified text images. So that when the training is completed, the TNN model is configured to perform elevator identifier recognition. Wherein, an enhanced data set is generated from a synthetic text image by image transformation of a synthetic text image of each training elevator identifier in a training elevator identifier set of an elevator corresponding to a training elevator template in a training elevator template set. Wherein, the synthetic text image is enhanced to a predetermined number of variations in appearance of the training elevator identifier set with different geometric parameters and visual parameters. Receive one or more user images. Extract a text image portion from one or more user images using a TNN model. Detect at least one elevator identifier in one or more user images based on the extracted text image portion and the training neural network model. The extracted text image portion includes an elevator identifier from a set of training elevator identifiers in the elevator training data. The detected elevator identifier is displayed on a display for user confirmation or user cancellation, and upon user confirmation, a control command is generated based on the detected elevator identifier associated with the elevator service. The control command is output to enable or disable the elevator service based on the control command.

[0032] According to another embodiment of the present disclosure, a computerized system includes an image sensor to capture an image from a user. One or more processors and one or more computer storage media storing computer usable instructions, which when used by the one or more processors cause the one or more processors to access a memory having a training neural network (TNN) model. The TNN model is trained using a training processor, and the training processor uses an enhanced data set as a synthetic training data set to generate a distribution of classified text images, so that when the training is completed, the TNN model is configured to perform elevator identifier recognition, wherein the enhanced data set is generated from the synthetic text image by image transformation of the synthetic text image of each training elevator identifier in the training elevator identifier set of the elevator corresponding to the training elevator template in the training elevator template set. Wherein, the synthetic text image is enhanced to a predetermined number of variations in appearance of the training elevator identifier set with different geometric parameters and visual parameters. A feature representation is extracted from the captured image using the TNN model. At least one elevator identifier in the captured image is detected based on the extracted feature representation and the TNN model. The detected elevator identifier is displayed on a display for user confirmation or user cancellation, and when the user confirms, a control command is generated based on the detected elevator identifier associated with the elevator service. A control command is output to enable or not enable the elevator service based on the control command.

[0033] The presently disclosed embodiments will be further described with reference to the accompanying drawings. The drawings shown are not necessarily to scale, with emphasis instead generally being placed upon illustrating the principles of the presently disclosed embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] [ Figure 1A ] Figure 1A is a block diagram illustrating an apparatus for contactless elevator service operation of an elevator according to some embodiments of the present disclosure;

[0035] [ Figure 1B ] Figure 1B Some embodiments of the present disclosure are shown in Figure 1A A schematic diagram of the method steps of the device;

[0036] [ Figure 1C ] Figure 1C Some embodiments of the present disclosure are shown in Figure 1A A schematic diagram of the method steps of the device;

[0037] [ Figure 1D ] Figure 1D Some embodiments of the present disclosure are shown in Figure 1A A schematic diagram of the method steps of the device;

[0038] [ Figure 1E ] Figure 1E is a schematic diagram illustrating a method for contactless elevator service operation of an elevator according to some embodiments of the present disclosure;

[0039] [ Figure 2 ] Figure 2 is a schematic diagram showing some processing steps of an apparatus according to some embodiments of the present disclosure;

[0040] [ Figure 3A ] Figure 3A is a schematic diagram showing some steps for implementing a user guidance display mechanism according to some embodiments of the present disclosure, which may be located in an elevator car or at an opening of an elevator floor in a multi-story building;

[0041] [ Figure 3B ] Figure 3B is a schematic diagram showing some steps for implementing a user guidance display mechanism according to some embodiments of the present disclosure, which may be located in an elevator car or at an opening of an elevator floor in a multi-story building;

[0042] [ Figure 3C ] Figure 3C is a schematic diagram showing some steps for implementing a user guidance display mechanism according to some embodiments of the present disclosure, which may be located in an elevator car or at an opening of an elevator floor in a multi-story building;

[0043] [ Figure 4 ] Figure 4 Some embodiments of the present disclosure are shown in Figure 1A A block diagram of a method that may be implemented using an alternative computer or processor;

[0044] [ Figure 5A ] Figure 5A is a schematic diagram illustrating a food elevator according to some embodiments of the present disclosure;

[0045] [ Figure 5B ] Figure 5B is a schematic diagram showing a moving walkway according to some embodiments of the present disclosure; and

[0046] [ Figure 5C ] Figure 5C is a schematic diagram illustrating a wheelchair lift according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0047] Although the above-mentioned drawings illustrate the embodiments of the present disclosure, other embodiments are also conceivable as pointed out in the discussion. The present disclosure presents exemplary embodiments as representations and not limitations. Those skilled in the art may design many other modifications and embodiments that fall within the scope and spirit of the principles of the embodiments of the present disclosure.

[0048] Figure 1A is a block diagram showing an apparatus for contactless elevator service operation of an elevator according to some embodiments of the present disclosure. The apparatus 40 may include a control circuit 41, which includes a processor 42, an adapter 44 (e.g., a display adapter, a mouse adapter, a printer adapter, etc.), a memory 46, and a transceiver 48 (operating as an input device and an output device) connected via a bus 43.

[0049] The memory 46 may be configured to store executable instructions. The executable instructions may be stored or organized in any manner at any level of abstraction, such as in combination with one or more applications, processes, routines, procedures, methods, functions, etc. The instructions stored in the memory 46 may be executed by one or more processors such as the processor 42. The processor 42 may be connected to the transceiver 48 or one or more input / output (I / O) devices (not shown). The transceiver 48 (or I / O device) may be configured to provide an interface to allow a user to interact with the device 40. The bus 43 may be connected to other devices 69 (which may include I / O devices, keyboards or keypads, touch screens or touch panels, display screens, microphones, speakers, mice, buttons, remote controls, joysticks, printers, phones, mobile devices (e.g., smart phones), sensors, alternative computer systems, etc.). Such other devices may also include adapters for the adapter 44 if necessary. The memory 46 may store data from one or more sensors (e.g., two-dimensional (2D) or three-dimensional (3D) sensors). The data may be processed by the processor 42 to control one or more parameters associated with the operating device 40. The data may include data indicative of an environment or scene captured by one or more sensors, including temperature, humidity, noise data that may be included in the environment / scene.

[0050] A storage device 50 may also be connected to the bus 43, and the storage device 50 may include the trained neural network 55 and other data. In addition, a controller 49 may be connected to the bus 43 according to the user's needs.

[0051] The device 40 is connected to a communication network 7. The communication network may be connected to one or more cloud computers 3 (which may include a storage unit). Other components connected to the communication network may include a storage unit 6, one or more laptop computers 4, and one or more mobile devices 5.

[0052] The device 40 may also be connected to a controller 60, a transmitter 62, a power source or multiple power sources 64, a receiver 66, and a transceiver 68 via a bus 43. It is contemplated that the controller 60, the transmitter 62, the power source 64, the receiver 66, and the transceiver 68 may all be operated using the transceiver 48 via the bus 43. The controller 60 is also connected to the elevator panel 1A via a connection 71, either wired or wireless, to send control signals to the elevator panel 1A. Other components disclosed in the present disclosure may be connected to the elevator panel 1A and to other elevator components (not shown) of the elevator, which may depend on the specific operational goals and requirements of the user. It is contemplated that one or more servers (not shown) may be configured to communicate with the device 40 via the communication network 7. Some examples of the one or more servers (not shown) may include one or more application servers, one or more cloud servers, one or more network servers, one or more database servers, one or more file servers, one or more host servers.

[0053] The device 40 is connected to the elevator access panel 1A, the image sensor 10, and the user display guidance camera 80. Optionally, the camera 10 and the user display guidance camera may be covered by a privacy cover 12A. The image sensor 10 may be a camera associated with an elevator to enable elevator service corresponding to an elevator button, and the camera 10 includes a target area 14A, and when presented by a user who positions text related to the elevator service in the target area 14A of the camera 10, the camera 10 captures a real-time image of the text. In some embodiments, the image sensor 10 such as a camera may be implemented as an integrated unit of the device 40 or as a separate device.

[0054] The user mobile device 20 may display one of letters, numbers, symbols, alphanumeric patterns specific to an identifier associated with an elevator service, including one of an elevator floor number, opening or closing an elevator door, accessing an elevator phone, keeping an elevator door open for a period of time, or some other elevator service related actions.

[0055] The camera 10 may be located in an elevator (not shown), in a corridor (not shown) near an elevator door (not shown) on a floor (not shown) of a building (not shown), or in both. Among them, the elevator service may include a user activating a button from the corridor on the floor to request the elevator to travel from an initial position to the user's position along the X-axis direction or along the Y-axis direction, and then when the user enters the elevator and the user indicates the destination position, the elevator travels to the requested user destination floor. In addition, the image captured from the target area 14A of the camera 10 may be a high dynamic range image and a JPEG image. The image sensor or camera 10 may be configured to send or otherwise communicate the captured image as an input to the device 40 for processing.

[0056] The user inputs text information 31A into the mobile device 20 and displays the text information 31A to the target area 14A of the camera 10. The user can extend the user's hand 22 to a nearby area within the target area 14A so that the camera 10 captures the user text information 31A as a captured image, i.e., captured image data 47.

[0057] The user guidance display mechanism (see FIG. 3 ) may include a display 16B (see FIG. 3 ) mounted on top of the privacy cover 12A to display a pre-captured real-time image (see X83 in FIG. 3 ). Figure 1B ) to assist the user in aligning his / her text information 31A in the target area 14A before the camera 10 acquires the user text information 31A in the target area 14A as a captured real-time image 47.

[0058] Alternatively, the user may handwrite the non-electronic text information on a non-electronic surface configured to display text information before presenting the non-electronic text information related to the elevator service to the target area 14A of the camera 10. Then, before the camera 10 acquires the user text information 31A in the target area 14A, the user may utilize the user guidance display mechanism (see FIG. 3 ) to ensure that the user text information 31A is in the target area 14A.

[0059] The target area 14A of the camera 10 can be configured to capture only the real-time image associated with the user text information 31A, so that the camera 10 can be pointed to a floor position (e.g., an elevator floor or a building floor) to obtain a camera target area that avoids capturing user privacy data or other data not associated with the user text information. In addition, it is possible that in order to ensure user privacy data, the camera focus depth amount of the camera focus level of the camera 10 can be adjusted to ensure that the camera 10 only captures the real-time image associated with the user text, and does not capture the user privacy data. In other words, any object beyond the camera focus level will be out of focus of the camera 10, so that the focus depth level can be set or controlled by the operator's setting of the camera controller (not shown).

[0060] Alternatively or optionally, the device 40 may include a feedback device (not shown) to provide a positive indication to the user that the user text information 31A has been recognized. The feedback device may include an audio or visual device. In some embodiments, the feedback may simply indicate that the gesture has been recognized. In some embodiments, the feedback may indicate the type of action recognized by the device 40. Before providing elevator resources for the requested service / function, the user can confirm the feedback. In this regard, the feedback device can communicate bidirectionally and can be used to provide interaction between the device 40 and the user of the device 40.

[0061] It is contemplated that the device 40 may also include a video display adapter 44 for communicating displayable images to a display device (e.g., display device 69). Examples of display device 69 may include a liquid crystal display (LCD), a cathode ray tube (CRT), a plasma display, a light emitting diode (LED) display, and any combination thereof. Display adapter 44 and display device 69 may be used in combination with processor 42 to provide graphical representations of various aspects of the present disclosure. In addition to display device 69, alternative computer system 69 may include one or more other peripheral output devices (not shown), including audio speakers, printers, and any combination thereof. In addition, other devices 69 may include peripheral output devices connected to bus 43 via peripheral interface 51. Examples of peripheral interface 51 include serial ports, USB connections, FIREWIRE connections, parallel connections, and any combination thereof.

[0062] Figure 1B , Figure 1C and Figure 1D Some embodiments of the present disclosure are shown in Figure 1A Schematic diagram of the method steps of the device. Figure 1B The user is shown presenting user text information 31B (in this example, floor "3") in the target area 14B.

[0063] Figure 1C The display 16C is shown displaying the detected user text information so that the user can verify whether his or her text information 31B is correctly recognized. The display 16C may also display instructions to the user to guide the user on what to do next. Example instructions may be (but are not limited to): If the user's text information 31B is the same as that displayed on the display 16C, please present a predefined reserved symbol (e.g., a "#" sign) 35 in the target area 18 to confirm, such as Figure 1D If the user chooses to confirm the recognized text information displayed on 16C, then Figure 1A The control circuit 41 in will light up the corresponding button on the panel 1D and initiate the elevator operation corresponding to the recognized text information displayed on 16C. If the corresponding button on the panel 1D has been lit before the user presents the confirmation symbol 35, then after the user presents the confirmation symbol 35, the corresponding button on the panel 1D will go out and the corresponding elevator operation will be cancelled.

[0064] In other words, the user can deactivate the already enabled button and elevator operation by presenting the corresponding text information 31B again. If the user chooses not to confirm the recognized text information displayed on 16C (in other words, the user does not present the predefined reserved symbol 35 in the target area 18 within a predefined time period (e.g., a few seconds), the recognized text information will no longer be displayed on 16C. The method we have just described here can be one of the mechanisms for the user to confirm / enable or cancel / deactivate the elevator operation.

[0065] Figure 1E is a block diagram illustrating a method for contactless elevator service operation of an elevator according to some embodiments of the present disclosure.

[0066] Step 110 of the method may include the user inputting elevator service information (ie, a floor destination as a user elevator identifier) ​​into the user mobile device and displaying it in a target area of ​​the image sensor.

[0067] Optionally, Figure 1E Step 115 may include a user guiding display mechanism to assist the user in viewing the text before the camera acquires the user text in the target area as a captured real-time image.

[0068] Figure 1E Step 120 may include accessing a memory having a trained neural network (TNN) model, the TNN model being trained using the augmented dataset as a synthetic training dataset to perform elevator identifier recognition, i.e., the TNN performs elevator identifier recognition directly on the captured image. Furthermore, it will be understood that after the TNN is deployed, access to the training dataset is not required after the TNN is deployed.

[0069] Figure 1E Step 125 may extract the text image portion from the captured image, which is done by the TNN without access to the augmented dataset.

[0070] Figure 1E Step 130 may include detecting an elevator identifier in the captured image based on the extracted text image portion and the TNN model, such that the extracted text image portion includes a training elevator identifier from a set of training elevator identifiers in the elevator training data.

[0071] Figure 1E Step 135 may include displaying the detected elevator identifier on a display for user confirmation or user cancellation, and upon user confirmation, generating a control command based on the detected elevator identifier.

[0072] Figure 1E Step 140 may include outputting a control command to enable or deactivate the elevator service based on the control command.

[0073] Figure 2 is a schematic diagram showing some processing steps of an apparatus according to some embodiments of the present disclosure. Figure 2 A technician 200 is shown controlling a classifier training system 201 in an external computing facility (e.g., a cloud computing facility), a classifier computer 204 on which a trained classifier 205 may be deployed, and a relay and elevator mechanical system 206 that performs elevator service. Before the elevator begins operation, the technician 200 first starts the classifier training system 201 so that a synthetic data set is generated by a data synthesis module 202, and the classifier is trained using the synthetic data set in a classifier training service 203 in the external computing facility. After the classifier is trained, the technician 200 deploys the classifier in the classifier computer 204 as a classifier 205. After the elevator begins operation, the LCD display 207 may display information and instructions to guide the user 210 on how to start the elevator service (e.g., requiring the user 210 to display text information associated with the destination floor to the camera 208). After the user 210 presents the text information to the camera 208, the captured image is fed as input to the classifier 205 in the classifier computer 204. The classifier computer 204 will send the text information and associated instructions recognized by the classifier 205 to the LCD display 207 so that the LCD display 207 can display them to the user 210 along with the image captured by the camera 208. The classifier computer 204 will also send commands associated with the text information recognized by the classifier 205 to the relay and elevator mechanical system 206 so that the elevator service can be performed. The elevator panel 209 connected to the relay and elevator mechanical system 206 will display to the user 210 which elevator operations are enabled (e.g., all of the user's destination floors). Unless interrupted by the technician 200, Figure 2 The entire system will operate continuously.

[0074] Figure 3A , Figure 3B and Figure 3C is a schematic diagram showing some steps for implementing a user guidance display mechanism according to some embodiments of the present disclosure, which may be located in an elevator car or at the opening of an elevator floor in a multi-story building.

[0075] For example, Figure 3AA user (shown as the user's hand 322A) is shown presenting text information 331A (i.e., displayed as the number "3") depicted by the user's phone 320A in the target area 314A of the camera 310A. The camera 310A may be protected by a privacy cover 312A, such that the privacy cover 312A also protects the user display instruction camera 380A. Optionally, when the user presents the text information 331A (displayed as the number "3") in the target area 314A of the camera 310A via the user's phone 320A, the user display instruction camera 380A may be enabled. The camera 310A, the user display instruction camera 380A, and the privacy cover 312A may be located near the elevator panel 301A, near an elevator door on a floor of a multi-story building (not shown), or some other convenient location for user access to elevator operations.

[0076] Figure 3B It is shown that after the user display guide camera 380A is enabled, the user's text information 331B (ie, displayed as the number "3") is reproduced or displayed on Figure 3C in the display 383. Figure 3C The display 383 can be mounted on top of the privacy cover 312C or some other convenient location for the user to view while presenting the text information to the camera 310B. The user can view the text information (i.e., displayed as the number "3") in the Figure 3C How the display 383 is positioned in the display 383 is used to adjust the placement of the user phone 320B so that the camera 310B correctly records the user text message to be read as input by the device to confirm the user command, that is, in this case, the user command is to request the elevator to take the user to floor 3.

[0077] Figure 3C It is shown that the user did not correctly present the user text information 331C because the user text information 331C was not fully displayed in the display 383. Therefore, the user needs to further adjust the placement of the user text information displayed as the number "3" so that the number "3" can be fully seen in the display 383. When the user text information 331B is correctly centered within the display 383, the camera 310B can capture an image of the user text information 331B, which can be used as an input to the device. The device then accepts the captured image for processing so that the device can understand the user's command (in this particular case, that the user wants to go to floor 3 from the initial floor position). Figure 3C The display acts as an aid to the user to guide the placement of the user's phone 320B in the correct position for the camera 310B to capture an image that the device can receive and understand to figure out the solution to the user input (ie, text message 331B).

[0078] As described above, the device extracts a text image portion from one or more captured images of the user text information 331B (displayed as the number "3" in this case), and based on the similarity between the text image portion and at least one enhanced text image, the TNN model detects at least one elevator identifier in the captured one or more images (i.e., based on the extracted text image portion and the TNN model). Wherein, the extracted text image portion includes a training elevator identifier from a training elevator identifier set in the elevator training data, which is detected. The detected elevator training identifier is displayed for user confirmation or user cancellation, and upon user confirmation, a control command is generated based on the detected elevator identifier associated with the elevator service. Wherein, the control command is output to enable or disable the elevator service based on the control command.

[0079] feature

[0080] A device for real-time contactless elevator service operation of an elevator. The device, including a control circuit, is configured to access a training neural network (TNN) model from a cloud computer via a communication network connected to the device, and store the TNN in a memory. The TNN is trained using a training processor with elevator training data. The elevator training data includes an elevator training identifier of a training elevator set and an elevator training template set associated with the elevator training data, each training elevator in the training elevator set including a training elevator identifier set associated with an elevator training service corresponding to the elevator training template. One or more user images are received in real time via an image sensor. A text image portion is extracted from one or more user images using a TNN model. At least one elevator identifier in one or more user images is detected based on the extracted text image portion and the TNN model. The detected elevator identifier is displayed on a display for user confirmation or user cancellation, and when the user confirms, a control command is generated based on the detected elevator identifier associated with the elevator service. The control command is output to enable or disable the elevator service based on the control command. It is conceivable that the system may include any combination of the different aspects listed below with respect to the above-mentioned device. Specifically, the following aspects are intended to create one or more embodiments based on one or more combinations of the aspects listed below for the above-mentioned apparatus, alone or in combination.

[0081] On the one hand, the image sensor may be a camera having a target area for capturing one or more images, such that a portion of the camera is covered by a privacy cover, and wherein the user guidance display mechanism has a user guidance display mounted on an outer surface of the privacy cover to assist a user in positioning text information within the target area of ​​the camera before the camera captures one or more images.

[0082] Another aspect may include a training processor that generates a synthetic text image during training based on the step of accessing stored elevator training data from a training memory. The stored elevator training data may include training elevator identifiers of a set of elevators and a set of training elevator templates associated with the elevator training data. Wherein each training elevator includes a set of training elevator identifiers associated with an elevator service corresponding to a training elevator template. A synthetic text image is generated for each training elevator identifier in the set of training elevator identifiers corresponding to a training elevator template in the set of training elevator templates. Wherein each synthetic text image in the synthetic text image is a variation of at least one typeface parameter of at least one training elevator identifier in the set of training elevator identifiers so as to obtain an enhanced data set. Wherein the extracted text image portion includes an elevator identifier from the set of training elevator identifiers in the elevator training data, such that the elevator identifier and the set of training elevator identifiers include one of a letter, a number, a symbol, an alphanumeric pattern specific to an identifier associated with an elevator service, wherein the elevator service includes one of an elevator floor number, opening or closing an elevator door, accessing an elevator phone, keeping an elevator door open for a period of time, or some other elevator service related action.

[0083] In one aspect, the elevator identifier and the set of training elevator identifiers include one of letters, numbers, symbols, alphanumeric patterns specific to an identifier associated with an elevator service, wherein the elevator service includes one of an elevator floor number, opening or closing an elevator door, accessing an elevator phone, keeping an elevator door open for a period of time, or some other elevator service related action.

[0084] On the one hand, the image sensor may be a camera including a target area, and when the user positions text in the target area of ​​the camera, it captures a real-time image so that a portion of the camera is covered by a privacy cover, wherein the privacy cover also covers a portion of the user guidance camera, so that the privacy cover includes a user guidance display mechanism having a user guidance display mounted on an outer surface of the privacy cover, to assist the user in positioning text information within the target area so that the camera can capture one or more images.

[0085] Additionally, one aspect is that an image sensor is located in the elevator, near an elevator door on a building floor, or both, so as to capture an image associated with a user requesting elevator service, such that the elevator service includes enabling operation of the elevator to travel from an initial location to a destination location (e.g., from a user initial location to a destination floor) in one of an X-axis direction or in a Y-axis direction.

[0086] As an aspect, before presenting the text related to the elevator service into the target area of ​​the camera, the user may input the text into a mobile device, a calculator, or one of electronic devices configured to display text.

[0087] Another aspect is that the user handwrites the text related to the elevator service on one of a non-electronic surface configured to display text or an electronic surface configured to display text before presenting the text into the target area of ​​the camera.

[0088] For example, on the one hand, the target area of ​​the image sensor is configured to capture only real-time images associated with user text, so that the image sensor is a camera that: (1) points to a floor location (e.g., an elevator floor or a building floor) to obtain a target area that avoids capturing user privacy data or other data not associated with the user text; or (2) includes a camera focus level. The depth of focus amount causes the camera to capture only real-time images associated with the user text so that any data or objects beyond the camera focus level are out of focus, wherein the depth of focus level is controlled by the operator's setting of the camera controller.

[0089] In one aspect, the control circuit extracts a set of text image parts from one or more captured images through a TNN model, wherein the control circuit searches the extracted set of text image parts for a plurality of alphanumeric patterns present in a set of elevator identifiers specified by elevator training data after applying optical character recognition to the extracted set of text image parts.

[0090] Another aspect is that the geometric parameters applied to enhance the synthetic text image used during training correspond to one or a subset of a rotation parameter set, a scaling parameter set, a font style parameter set, a perspective transformation parameter set, a shape parameter set, or an orientation parameter set.

[0091] Another possible aspect is that the visual parameters applied to enhance the synthetic text image used during training correspond to a set or a subset of a color parameter set, a contrast parameter set, an external lighting parameter set, a brightness parameter set, a shadow parameter set, a noise modeling parameter set, a background parameter set including a color parameter set, a noise modeling parameter set or a visual image variable parameter set.

[0092] Another aspect is that the TNN model includes two or more neural networks or subsets of different types of artificial neural networks, such that the two or more neural networks and subsets of different types of artificial neural networks include at least one or a combination of a convolutional neural network (CNN), a feedforward neural network, a recurrent neural network, or a modular neural network. Wherein the control circuit segments a plurality of alphanumeric characters arranged in a specific order for each enhanced text image in the enhanced data set.

[0093] Another aspect also includes outputting an alarm control command to an elevator operator in real time or near real time by the control circuit in response when the detected elevator identifier is associated with an elevator service for accessing the elevator phone, such that an alarm control command is generated, wherein the generated alarm control command is output to a communication network associated with the elevator operator and connected to the control circuit.

[0094] It is contemplated that one aspect of the training is that each of the synthetic text images is enhanced by adding different types of noise (including but not limited to Gaussian noise) to the synthetic text image to simulate the noise in the captured image.

[0095] One aspect of the training is that the captured text images include an image background, so that each of the synthetic text images is enhanced by filling the image background of the synthetic text image with a gradient that simulates a shadow effect.

[0096] Another aspect in the training is that each of the synthetic text images is enhanced by randomly perturbing the synthetic text image by an affine transformation.

[0097] Another aspect in the training is that each of the synthetic text images is augmented by resizing the synthetic text image to different heights and widths at a plurality of different aspect ratios.

[0098] Another aspect in the training is that each of the synthetic text images is enhanced by adding a random blur effect (including but not limited to Gaussian blur) to the synthetic text image.

[0099] Another aspect in the training also includes predicting an identity of at least one elevator identifier in the captured image using optical character recognition (OCR), wherein the prediction is based at least in part on a particular distribution of the classified text images in the distribution of classified text images.

[0100] On the one hand, alternatively, Figure 1A The processor 42 may be configured to use the enhanced data set as a synthetic training data set to train the neural network model. Figure 1A The processor 42 may then use the TNN model to extract the text image portion from the captured image. Figure 1A Examples of processor 42 may be an application specific integrated circuit (ASIC) processor, a complex instruction set computing (CISC) processor, and / or other hardware.

[0101] Another optional aspect of the device may include a network interface, which may include an antenna, a frequency modulation (FM) transceiver, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identity module (SIM) card, and / or a local buffer. The network interface may communicate with a network such as the Internet, an intranet, and / or a wireless network (e.g., a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN)) via wireless communication. Wireless communication may use any of a variety of communication standards, protocols, and technologies, such as Long Term Evolution (LTE), Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.120g., IEEE 802.11a, IEEE802.11b, IEEE 802.11g, and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, protocols for email, instant messaging, and / or short message service (SMS). These components of device 40 may depend on specific user goals and requirements. In addition, depending on the user's specific design architecture, the device may include I / O devices that must receive input and output from the device. Examples of input devices may include a touch screen, a keyboard, a mouse, a joystick, a microphone, and / or an image sensor. Examples of output devices may include a display screen (e.g., a liquid crystal display (LCD) or a light emitting diode (LED) display) and / or a speaker.

[0102] Another aspect may include that the image sensor 40 may be implemented via charge coupled device technology or complementary metal oxide semiconductor technology. Some implementation examples of the image sensor 10 may include CCTV cameras, digital cameras, video cameras, and action cameras.

[0103] definition

[0104] According to various aspects of the present disclosure, and based on experiments, the following definitions have been established, and are certainly not complete definitions of the individual phrases or terms. Wherein, based on the knowledge from experiments, the definitions provided are provided only as examples, where other explanations, definitions and other aspects may be relevant. However, at least for only a basic preview of the phrases or terms presented, these definitions are provided. In addition, the definitions below cannot be regarded as prior art because the knowledge obtained is only from experiments.

[0105] Convolutional Neural Network (CNN): CNN is a deep learning neural network designed to process structured arrays of data such as images. CNN is widely used in computer vision (e.g., image classification). CNN is very good at picking up patterns in the input image, such as lines, gradients, circles, or even eyes and faces. It is this property that makes CNN powerful for computer vision, allowing CNN to operate directly on raw image data without any preprocessing. CNN is a feed-forward neural network, often with up to 20 or 30 layers, where the power of convolutional neural networks comes from a special kind of layer, called convolutional layers. CNN contains many convolutional layers stacked on top of each other, and each convolutional layer is able to recognize more complex shapes. With three or four convolutional layers, handwritten numbers can be recognized, and with 25 layers, faces can be distinguished.

[0106] The architecture of a CNN consists of a multi-layer feed-forward neural network, formed by stacking many hidden layers on top of each other. It is this sequential design that allows CNNs to learn hierarchical features. Hidden layers are usually convolutional layers, followed by activation layers, some of which are followed by pooling layers. A simple CNN that helps understand the core design principles is able to recognize handwritten characters. At least one key building block in a CNN is a convolutional layer. We can visualize a convolutional layer as many small square templates, called convolution kernels, which slide over an image and find patterns. In the case where the portion of the image matches the kernel's pattern, the kernel returns a large positive value, and when it does not match, the kernel returns zero or a small value. Mathematically, a kernel is a weight matrix. Despite the simplicity of CNNs, the ability to detect vertical or horizontal lines, corners, curves, and other simple features is an extremely powerful property of convolution kernels. We remember that a convolutional layer consists of a series of convolution kernels. Typically, the first layer of a convolutional neural network contains a vertical line detector, a horizontal line detector, and various diagonal line, curve, and corner detectors. These feature detector kernels are not programmed by humans, but are in fact learned by the neural network during training and are used as the first stage of the image recognition process.

[0107] Some of the activation functions in CNNs are, after passing the image through the convolutional layers, the output is usually passed through an activation function. Common activation functions include the sigmoid function and the ReLU function (i.e., also known as the rectified linear unit, which is the same as taking the positive component of the input). The activation function has the effect of adding nonlinearity to the convolutional neural network. If the activation function does not exist, all layers of the neural network can be condensed into a single matrix multiplication. In the case of the cat image above, applying the ReLU function to the first layer output results in a stronger contrast that highlights the vertical lines and removes noise from other non-vertical features.

[0108] One aspect of CNNs is the repetitive structure, so that a basic convolutional neural network can be viewed as a series of convolutional layers, followed by an activation function, followed by a pooling (downscaling) layer, repeated many times. Through repeated combinations of these operations, the first layer detects simple features in the image (e.g., edges), and the second layer begins to detect higher-level features. By the tenth layer, the convolutional neural network is able to detect more complex shapes (e.g., eyes). By the twentieth layer, it is generally able to distinguish faces from one another. This ability comes from the repeated layering of operations, with each layer detecting slightly higher-order features than the previous layer. Later layers in the neural network are able to build on features detected by earlier layers and recognize more complex shapes.

[0109] Typeface vs. Font: A typeface is a collection of characters of the same design. These characters include letters, numbers, punctuation, and symbols. Some popular typefaces include Arial, Helvetica, Times, and Verdana. Although most computers come with a few dozen typefaces installed, there are thousands available. Because they are vector-based (rather than bitmapped), typefaces can be scaled very large and still look sharp. The term "typeface" is often confused with "font," which is a typeface of a specific size and style. For example, Verdana is a typeface, while Verdana 10pt Bold is a font. It's a subtle difference, but good to know.

[0110] Figure 4 The present invention is shown in the embodiment of the present invention. Figure 1A The block diagram of the method can be implemented using an alternative computer or processor. The computer 411 includes a processor 440, a computer readable memory 412, a storage system 458, and a user interface 449 having a display 452 and a keyboard 451, which are connected via a bus 456. For example, when a user receives input from the surface of the user interface 457 (keyboard 453), the user interface 457 in communication with the processor 440 and the computer readable memory 412 acquires image data and stores it in the computer readable memory 412.

[0111] The computer 411 may include a power supply 454, which may optionally be external to the computer 411, depending on the application. A user input interface 457 adapted to connect to a display device 448 may be linked via the bus 456, wherein the display device 448 may include a computer monitor, camera, television, projector, or mobile device, etc. A printer interface 459 may also be connected via the bus 456 and adapted to connect to a printing device 432, wherein the printing device 432 may include a liquid inkjet printer, a solid ink printer, a large commercial printer, a thermal printer, a UV printer, or a dye sublimation printer, etc. A network interface controller (NIC) 434 is adapted to connect to a network 436 via the bus 456, wherein image data or other data, etc. may be presented on a third party display device, a third party imaging device, and / or a third party printing device external to the computer 411. The computer / processor 411 may include a GPS 401 connected to the bus 456.

[0112] Still refer to Figure 4 , image data or other data, etc. may be sent via the communication channel of the network 436 and / or stored in the storage system 458 for storage and / or further processing. In addition, time series data or other data may be received wirelessly or hardwired from the receiver 446 (or external receiver 438), or sent wirelessly or hardwired via the transmitter 447 (or external transmitter 439), both of which are connected by the bus 456. The computer 411 may be connected to the external sensing device 444 and the external input / output device 441 via the input interface 408. The input interface 408 may be connected to one or more input / output devices 441, the external memory 406, and the external sensor 404, which may be connected to the machine-like device 402. For example, the external sensing device 444 may include a sensor that collects data before-during-after the time series data of the collected machine. The computer 411 may be connected to other external computers 442. The output interface 409 may be used to output the processed data from the processor 440. Note that upon receiving user input from the surface 452 of the user interface 449 , the user interface 449 in communication with the processor 440 and the non-transitory computer-readable storage medium 412 retrieves the region data and stores it in the non-transitory computer-readable storage medium 412 .

[0113] Figure 5A , Figure 5B and Figure 5C is a schematic diagram showing different embodiments in which the device may be incorporated into different systems, e.g. Figure 5A It is a transport system that can be associated with a food elevator. Figure 5B It's a moving sidewalk. Figure 5C It's a wheelchair lift.

[0114] The description provides only exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. Instead, the following description of the exemplary embodiments will provide a feasible description for implementing one or more exemplary embodiments to those skilled in the art. It is contemplated that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of the disclosed subject matter set forth in the appended claims.

[0115] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be appreciated by those of ordinary skill in the art that the embodiments may be practiced without these specific details. For example, the systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagram form to avoid obscuring the embodiments in terms of unnecessary details. In other cases, well-known processes, structures, and techniques may be shown without unnecessary details to avoid obscuring the embodiments. In addition, similar reference numerals and designations in the various drawings indicate similar elements.

[0116] In addition, various embodiments may be described as processes, which are depicted as flow charts, data flow diagrams, structure diagrams, or block diagrams. Although flow charts may describe operations as sequential processes, many operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process may terminate when its operations are completed, but may have additional steps that are not discussed or included in the accompanying drawings. In addition, not all operations in any specifically described process may appear in all embodiments. A process may correspond to a method, a function, a program, a subroutine, a subprogram, etc. When a process corresponds to a function, the termination of the function may correspond to the function returning to the calling function or the main function.

[0117] In addition, embodiments of the disclosed subject matter may be implemented at least in part manually or automatically. Manual or automatic implementation may be performed or at least assisted by the use of a machine, hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments that perform the desired tasks may be stored in a machine-readable medium. A processor may perform the desired tasks.

[0118] The above embodiments of the present disclosure may be implemented in any of many ways. For example, the embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code may be executed on any suitable processor or set of processors (whether arranged in a single computer or distributed between multiple computers).

[0119] In addition, the various methods or processes outlined herein may be encoded as software that can be executed on one or more processors using any of a variety of operating systems or platforms. In addition, such software may be written using any of a variety of suitable programming languages ​​and / or programming or scripting tools, and may also be compiled into executable machine language code or intermediate code that is executed on a framework or virtual machine. Typically, in various embodiments, the functionality of the program modules may be combined or distributed as desired.

[0120] In addition, the embodiments of the present disclosure can be embodied as a method, and examples thereof have been provided. The actions performed as a part of the method can be ordered in any suitable manner. Therefore, an embodiment that performs actions in an order different from that shown can be constructed, although shown as sequential actions in the illustrative embodiments, it may include performing some actions simultaneously. In addition, the use of ordinal numbers such as "first" and "second" in the claims to modify claim elements does not itself imply any priority or order of one claim element over another claim element or the time order of the execution of the method actions, but is only used as a label to distinguish a claim element with a specific name from another element with the same name (but using ordinal numbers) to distinguish these claim elements.

[0121] Although the present disclosure has been described with reference to certain preferred embodiments, it will be understood that various other adaptations and modifications may be made within the spirit and scope of the present disclosure. Therefore, the aspects of the appended claims cover all such changes and modifications that fall within the true spirit and scope of the present disclosure.

Claims

1. A device for real-time contactless elevator service operation of an elevator, the device comprising: a memory storing data, the data comprising a training neural network (TNN) model, wherein the TNN model is trained via a training processor using an augmented data set as a synthetic training data set to later perform elevator identifier recognition, such that the augmented data set is generated from a synthetic text image by image transformation of a synthetic text image of each training elevator identifier in a set of training elevator identifiers of elevators corresponding to a training elevator template in a set of training elevator templates, wherein the synthetic text image is augmented to a predetermined number of variations in appearance of the set of training elevator identifiers with different geometrical and visual parameters; an image sensor that captures one or more images from a user in real time; A processor that in real time: extracting a text image portion from the captured one or more images using the TNN model, and detecting at least one elevator identifier in the extracted text image portion using the extracted text image portion and the TNN model; displaying the detected elevator identifier on a display for user confirmation or user cancellation, and upon user confirmation, generating a control command based on the detected elevator identifier associated with the elevator service; and The control command is output to enable or not enable the elevator service based on the control command.

2. The device according to claim 1, wherein: The image sensor is a camera having a target area for capturing the one or more images, such that a portion of the camera is covered by a privacy cover, and wherein the user guidance display mechanism has a user guidance display mounted on an outer surface of the privacy cover to assist the user in positioning text information within the target area of ​​the camera before the camera captures the one or more images.

3. The device according to claim 1, wherein: The training processor generates the synthetic text image during training based on the following steps: accessing stored elevator training data from the memory, the stored elevator training data comprising training elevator identifiers of a set of elevators and the set of training elevator templates associated with the elevator training data, wherein each training elevator comprises a set of training elevator identifiers associated with an elevator service corresponding to a training elevator template; and generating a synthetic text image for each of the training elevator identifiers of the training elevator corresponding to a training elevator template of the training elevator template set, wherein each of the synthetic text images is a variation of at least one typeface parameter of at least one training elevator identifier of the training elevator identifier set, so as to obtain the enhanced data set.

4. The device according to claim 1, wherein: The extracted text image portion includes an elevator identifier from the set of training elevator identifiers in the elevator training data, such that the elevator identifier and the set of training elevator identifiers include one of letters, numbers, symbols, alphanumeric patterns specific to an identifier associated with the elevator service, the elevator service including one of an elevator floor number, opening or closing an elevator door, accessing an elevator phone, keeping the elevator door open for a period of time, or some other elevator service related actions.

5. A device for real-time contactless elevator service operation of an elevator, the device comprising: control circuit, the control circuit is configured to accessing a trained neural network TNN model from a cloud computer via a communication network connected to the device, and storing the TNN model in a memory, wherein the TNN model is trained using an elevator training data using a training processor, the elevator training data comprising elevator training identifiers of a training elevator set and a set of elevator training templates associated with the elevator training data, each training elevator in the training elevator set comprising a set of training elevator identifiers associated with an elevator training service corresponding to an elevator training template; receiving one or more user images in real time via an image sensor; extracting a text image portion from the one or more user images using the TNN model, and detecting at least one elevator identifier in the one or more user images based on the extracted text image portion and the TNN model; displaying the detected elevator identifier on a display for user confirmation or user cancellation, and upon user confirmation, generating a control command based on the detected elevator identifier associated with the elevator service; and The control command is output to enable or not enable the elevator service based on the control command.

6. The device according to claim 5, wherein: The training processor generating a synthetic text image for each training elevator identifier in the set of training elevator identifiers of a training elevator corresponding to an elevator training template in the set of elevator training templates, wherein each synthetic text image in the synthetic text images is a variation of at least one typeface parameter of at least one training elevator identifier in the set of training elevator identifiers; generating an enhanced dataset of synthetic text images by image transformation of the synthetic text images of respective training elevator identifiers in the set of training elevator identifiers of the training elevators corresponding to the elevator training template, wherein the synthetic text images are enhanced to a predetermined number of variations in appearance of the set of training elevator identifiers with different geometric and visual parameters; and The enhanced data set is used as a synthetic training data set to train a neural network model, so that when the neural network model is trained, the trained neural network TNN model can perform elevator identifier recognition on captured non-training images, so that the TNN model is stored in the memory or in the memory of the cloud computer.

7. The device according to claim 5, wherein: The elevator identifier and the set of training elevator identifiers include one of letters, numbers, symbols, alphanumeric patterns specific to an identifier associated with the elevator service, the elevator service including one of an elevator floor number, opening or closing an elevator door, accessing an elevator phone, keeping the elevator door open for a period of time, or some other elevator service related action.

8. The device according to claim 5, wherein: The image sensor is a camera including a target area, and when a user positions text in the target area of ​​the camera, the camera captures a real-time image so that a portion of the camera is covered by a privacy cover, wherein the privacy cover also covers a portion of a user guidance camera, so that the privacy cover includes a user guidance display mechanism having a user guidance display mounted on an outer surface of the privacy cover, and the user guidance display mechanism assists the user to position text information within the target area so that the camera can capture one or more images.

9. The device according to claim 5, wherein: The image sensor is located in the elevator, near an elevator door on a floor of a building, or both to capture an image associated with a user requesting elevator service, such that the elevator service includes enabling operation of the elevator to travel from an initial location to a destination location in one of an X-axis direction or in a Y-axis direction, such as from a user initial location to a destination floor.

10. The device according to claim 5, wherein: Prior to presenting the text related to the elevator service into a target area of ​​a camera, the user inputs the text into one of a mobile device, a calculator, or an electronic device configured to display text.

11. The device according to claim 5, wherein: Prior to presenting the text related to the elevator service into the target area of ​​the camera, the user handwrites the text onto one of a non-electronic surface configured to display text or an electronic surface configured to display text.

12. The device according to claim 5, wherein: The target area of ​​the image sensor is configured to capture only real-time images associated with user text, such that the image sensor is a camera that: (1) is pointed at a floor location such as an elevator floor or a building floor, resulting in the target area avoiding capturing user privacy data or other data not associated with the user text; or (2) includes a camera focus level, resulting in the camera capturing only real-time images associated with the user text so that any data or objects beyond the camera focus level are out of focus, wherein the level of depth of focus is controlled by an operator setting of a camera controller.

13. The device according to claim 5, wherein: The control circuit extracts a set of text image parts from the captured one or more images through a TNN model.

14. The device according to claim 13, wherein: The control circuit searches the extracted set of text image portions for a plurality of alphanumeric patterns present in the set of elevator identifiers specified by the elevator training data after applying optical character recognition to the extracted set of text image portions.

15. The device according to claim 5, wherein: The geometric parameters applied to enhance the synthetic text image used during training correspond to one or a subset of a rotation parameter set, a scaling parameter set, a font style parameter set, a perspective transform parameter set, a shape parameter set, or an orientation parameter set.

16. The device according to claim 5, wherein: The visual parameters applied to enhance the synthetic text image used during training correspond to a set or a subset of a color parameter set, a contrast parameter set, an external lighting parameter set, a brightness parameter set, a shadow parameter set, a noise modeling parameter set, a background parameter set including a color parameter set, a noise modeling parameter set or a visual image variable parameter set.

17. The device according to claim 5, wherein: The TNN model includes two or more neural networks or subsets of different types of artificial neural networks, so that the two or more neural networks and the subsets of different types of artificial neural networks include at least one or a combination of a convolutional neural network CNN, a feedforward neural network, a recurrent neural network or a modular neural network.

18. The device according to claim 17, wherein: The control circuit segments a plurality of alphanumeric characters arranged in a specific order for each enhanced text image in the enhanced data set.

19. The device according to claim 5, further comprising: When the detected elevator identifier is associated with the elevator service for accessing the elevator phone, an alarm control command is outputted by the control circuit to the elevator operator in real time or near real time as a response, so that the alarm control command is generated, wherein, The generated alarm control command is output to a communication network associated with the elevator operator and connected to the control circuit.

20. A method for real-time contactless elevator service operation of an elevator, the method comprising the steps of: accessing a memory storing data, the stored data comprising a training neural network (TNN) model, wherein the TNN model is trained via a training processor using an augmented data set as a synthetic training data set to perform elevator identifier recognition, such that the augmented data set is generated from a synthetic text image by image transformation of a synthetic text image of each training elevator identifier in a set of training elevator identifiers of elevators corresponding to a training elevator template in a set of training elevator templates, wherein the synthetic text image is augmented to a predetermined number of variations in appearance of the set of training elevator identifiers with different geometric parameters and visual parameters; receiving one or more user images in real time; extracting a text image portion from one or more captured user images using the TNN model; detecting at least one elevator identifier in the captured one or more user images based on the extracted text image portion and the TNN model; displaying the detected elevator identifier on a display for user confirmation or user cancellation, and upon user confirmation, generating a control command based on the detected elevator identifier associated with the elevator service; and The control command is output to enable or not enable the elevator service based on the control command.

21. An apparatus operating in real time for contactless transportation service operation of a transportation system associated with one of a food elevator, a moving walkway, or a wheelchair elevator, the apparatus comprising: a transceiver receiving a training neural network (TNN) model from a cloud-based computer and storing the TNN model in a memory, wherein the TNN model is trained by the cloud-based computer using an augmented data set as a synthetic training data set for later performing transport identifier recognition after the training is completed, such that the augmented data set is generated from a synthetic text image by image transformation of a synthetic text image of each training elevator identifier in a set of training transport identifiers for a transport system corresponding to a training transport template in a set of training transport templates, and wherein the synthetic text image is augmented to a predetermined number of variations in appearance of the set of training transport identifiers with different geometric parameters and visual parameters; a user guidance mechanism having a display to assist a user in positioning text information within a target area of ​​the image sensor before the image sensor captures an image of the user, wherein the user positions the text image within the target area and then the image sensor captures one or more real-time user images; A processor that in real time: extracting a text image portion from the one or more captured real-time user images using the TNN model, and detecting at least one transportation identifier in the text image portion extracted from the one or more captured real-time user images based on the extracted text image portion and the TNN model; displaying the detected system identifier on the display for user confirmation or user cancellation, and upon user confirmation, generating a control command based on the detected transportation identifier associated with the transportation service; and The control command is output to enable or not enable the transportation service based on the control command.

22. A non-transitory machine-readable medium comprising instructions stored thereon, the instructions, when executed by a processing circuit, configure the processing circuit to perform operations for real-time contactless elevator service operations for an elevator, the operations comprising: accessing a memory storing data, the stored data comprising a training neural network (TNN) model, and storing the TNN model in the memory, wherein the TNN model is trained using a training processor, the training processor using an augmented data set as a synthetic training data set to generate a distribution of classified text images, such that upon completion of the training, the TNN model is configured to perform elevator identifier recognition, wherein the augmented data set is generated from the synthetic text images by image transformation of the synthetic text images of respective training elevator identifiers in a set of training elevator identifiers of elevators corresponding to training elevator templates in a set of training elevator templates, and wherein the synthetic text images are augmented to a predetermined number of variations in appearance of the set of training elevator identifiers with different geometric parameters and visual parameters; receiving one or more user images; extracting text image portions from the one or more user images using the TNN model; detecting at least one elevator identifier in the one or more user images based on the extracted text image portion and the trained neural network model, wherein the extracted text image portion includes an elevator identifier from the set of training elevator identifiers in the elevator training data; displaying the detected elevator identifier on a display for user confirmation or user cancellation, and upon user confirmation, generating a control command based on the detected elevator identifier associated with the elevator service; and The control command is output to enable or not enable the elevator service based on the control command.

23. A computerized system, the computerized system comprising: an image sensor that captures an image from a user; one or more processors; as well as One or more computer storage media storing computer usable instructions that, when used by the one or more processors, cause the one or more processors to: accessing a memory having a training neural network (TNN) model, the TNN model being trained using a training processor, the training processor using an augmented data set as a synthetic training data set to generate a distribution of classified text images, such that upon completion of the training, the TNN model is configured to perform elevator identifier recognition, wherein the augmented data set is generated from the synthetic text images by image transformation of the synthetic text images of respective training elevator identifiers in a set of training elevator identifiers of elevators corresponding to training elevator templates in a set of training elevator templates, and wherein the synthetic text images are augmented to a predetermined number of variations in appearance of the set of training elevator identifiers with different geometric and visual parameters; extracting a feature representation from the captured image using the TNN model, and detecting at least one elevator identifier in the captured image based on the extracted feature representation and the TNN model; displaying the detected elevator identifier on a display for user confirmation or user cancellation, and upon user confirmation, generating a control command based on the detected elevator identifier associated with the elevator service; and The control command is output to enable or not enable the elevator service based on the control command.

24. The computerized system of claim 23, wherein: The TNN model is trained by the training processor through the following steps: accessing elevator training data from a training memory connected to the training processor, the elevator training data comprising training elevator identifiers for a set of training elevators and the set of training elevator templates associated with the elevator training data, wherein each training elevator comprises the set of training elevator identifiers associated with an elevator service corresponding to a training elevator template; generating, using control circuitry, a synthetic text image for each of the training elevator identifiers in the set of training elevators corresponding to a training elevator template in the set of training elevator templates, wherein each of the synthetic text images is a variation of at least one typeface parameter of at least one training elevator identifier in the set of training elevator identifiers; and The enhanced dataset of synthetic text images is generated by image transformation of the synthetic text images of respective training elevator identifiers of the set of training elevator identifiers of the training elevators corresponding to the training elevator templates, wherein the synthetic text images are enhanced to a predetermined number of variations in appearance of the set of training elevator identifiers with different geometrical and visual parameters.

25. The computerized system of claim 23, wherein: Each of the synthetic text images is enhanced by adding different types of noise to the synthetic text image to simulate noise in the captured image, the different types of noise including but not limited to Gaussian noise.

26. The computerized system of claim 23, wherein: The captured text images include an image background, such that each of the composite text images is enhanced by filling the image background of the composite text image with a gradient simulating a shading effect.

27. The computerized system of claim 23, wherein: Each of the synthetic text images is enhanced by randomly perturbing the synthetic text image by an affine transformation.

28. The computerized system of claim 23, wherein: Each of the composite text images is enhanced by adjusting the composite text image to different heights and widths at a plurality of different aspect ratios.

29. The computerized system of claim 23, wherein: Each of the synthetic text images is enhanced by adding a random blur effect to the synthetic text image, wherein the random blur effect includes but is not limited to Gaussian blur.

30. The computerized system of claim 23, further comprising: Optical character recognition (OCR) is used to predict an identity of at least one elevator identifier in the captured image, wherein the prediction is based at least in part on a particular distribution of the classified text images in the distribution of classified text images.

Images