Determination of at least one characteristic associated with at least one object in a limited space

Abstract

According to an aspect, there is provided a system. The system is configured to receive image data associated with an interior of a limited space, determine surface normal data based on the image data, and determine at least one characteristic associated with at least one object in the limited space based on the surface normal data.

Description

[0001] DETERMINATION OF AT LEAST ONE CHARACTERISTIC ASSOCIATED WITH AT LEAST ONE OBJECT IN A LIMITED SPACE

[0002] TECHNICAL FIELD

[0003] Various examples generally may relate to the field of elevator systems . In particular, some examples relate to a solution for determining at least one characteristic associated with at least one obj ect in a limited space , for example , in an elevator car .

[0004] BACKGROUND

[0005] Elevators are essential for transporting people , goods and other obj ects in buildings . In many buildings , due to the height of the buildings , elevators are in constant use as the use of stairs is not a practical option . Elevator systems often face challenges , for example , long waiting times , overcrowding and poor user experience , when operated . At the same time , the elevator systems may not be used ef ficiently . For example , the capacity usage of an elevator car may not be optimal .

[0006] One solution that can be applied to address the above challenges at least partially is people counting . People counting is the process of detecting and tracking the number o f people entering and exiting a speci fic area, such as an elevator car . People counting can help to optimi ze elevator operations , improve safety and security, and enhance customer satis faction . However, various people counting solutions often face issues relating to a lack of reliability and accuracy . Another challenge is that these solutions may not always be used to identi fy other obj ects than people in the elevator car .

[0007] To address the above issues , there is a need for a solution that provides accurate and reliable data on the number and behavior of people and also on other obj ects transported using elevators .

[0008] SUMMARY

[0009] The scope of protection sought for various example embodiments of the disclosure is set out by the independent claims . The example embodiments and features , i f any, described in this speci fication that do not fall under the scope of the independent claims are to be interpreted as examples useful for understanding various example embodiments of the disclosure .

[0010] According to a first aspect, there is provided a system comprising at least one processor and at least one memory storing instructions that , when executed by the at least one processor, cause the system, to at least perform : receiving image data associated with an interior of a limited space , determining surface normal data based on the image data, and determining at least one characteristic associated with at least one obj ect in the limited space based on the surface normal data .

[0011] In an implementation form of the first aspect , the image data comprises two-dimensional image data .

[0012] In an implementation form of the first aspect , the image data comprises three-dimensional image data .

[0013] In an implementation form of the first aspect , the three-dimensional image data is obtained from a sensor configured to provide depth data, and the at least one memory stores instructions that , when executed by the at least one processor, cause the system to at least perform : obtaining partial derivatives of a depth image with respect to pixel coordinates by computing a gradient of the depth image along the hori zontal and vertical directions , converting the pixel coordinates to sensor coordinates by using an intrinsic matrix of the sensor, calculating a normal vector for each pixel value , and mapping the normal vectors to a correct color range to provide the surface normal data .

[0014] In an implementation form of the first aspect , when determining at least one obj ect in the limited space based on the surface normal map, and the at least one memory stores instructions that , when executed by the at least one processor, cause the system to at least perform : inputting the surface normal data to an analyzing module , and receiving a result associated with the least one obj ect from the analyzing module .

[0015] In an implementation form of the first aspect , the analyzing module comprises a neural network model .

[0016] In an implementation form of the first aspect , the analyzing module comprises a heuristic algorithm .

[0017] In an implementation form of the first aspect , determining at least one characteristic associated with at least one obj ect in the limited space comprises determining the number of passengers in an elevator car based on the result .

[0018] In an implementation form of the first aspect , the method further comprises observing changes in the surface normal data over time , and using the observed changes in determining the at least one characteristic associated with at least one obj ect .

[0019] In an implementation form of the first aspect , and the at least one memory stores instructions that , when executed by the at least one proces sor, cause the system to at least perform : determining at least one characteristic associated with at least one obj ect in the limited space comprises determining the number of passengers in the limited space based on the result .

[0020] According to a second aspect , there is provided an elevator system comprising a system according to the first aspect , and a sensor configured to provide image data as sociated with an interior o f a limited space to the system .

[0021] In an implementation form of the second aspect , the sensor comprises a depth camera .

[0022] According to a third aspect , there is provided an elevator system comprising a system according to the first aspect , and a sensor arranged in an elevator car and configured to provide image data about the interior of the elevator car to the system .

[0023] According to a fourth aspect , there is provided a computer-implemented method comprising : receiving image data associated with an interior of a limited space , determining surface normal data based on the image data ; and determining at least one characteristic associated with at least one obj ect in the limited space based on the surface normal data .

[0024] In an implementation form of the fourth aspect , the image data comprises two-dimensional image data .

[0025] In an implementation form of the fourth aspect , the image data comprises three-dimensional image data .

[0026] In an implementation form of the fourth aspect , the three-dimensional image data is obtained from a sensor configured to provide depth data, and the method further comprises : obtaining partial derivatives of a depth image with respect to pixel coordinates by computing a gradient of the depth image along the hori zontal and vertical directions , converting the pixel coordinates to sensor coordinates by using an intrinsic matrix of the sensor, calculating a normal vector for each pixel value , and mapping the normal vectors to a correct color range to provide the surface normal data .

[0027] In an implementation form of the fourth aspect , determining at least one obj ect in the limited space based on the surface normal map comprises : inputting the surface normal data to an analyzing module , and receiving a result associated with the least one obj ect from the analyzing module .

[0028] In an implementation form of the fourth aspect , the analyzing module comprises a neural network model .

[0029] In an implementation form of the fourth aspect , the analyzing module comprises a heuristic algorithm .

[0030] In an implementation form of the fourth aspect , determining at least one characteristic associated with at least one obj ect in the limited space comprises determining the number of passengers in an elevator car based on the result .

[0031] In an implementation form of the fourth aspect , the method further comprises observing changes in the surface normal data over time , and using the observed changes in determining the at least one characteristic associated with at least one obj ect .

[0032] According to a fi fth aspect , there is provided a computer program comprising instructions which, when the program is executed by at least one processor, cause a system to perform the method of the fourth aspect . According to a sixth aspect , there is provided a computer-readable medium comprising a computer program comprising instructions which, when the program is executed by at least one processor, cause a system to perform the method of the fourth aspect .

[0033] According to a seventh aspect , there is provided a system comprising means for : receiving image data associated with an interior of a limited space , determining surface normal data based on the image data, and determining at least one characteristic associated with at least one obj ect in the limited space based on the surface normal data .

[0034] BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings , which are included to provide a further understanding of the invention and constitute a part of this speci fication, illustrate examples of the invention and together with the description help to explain the principles of the invention . In the drawings :

[0036] FIG . 1 illustrates a flow diagram of a computer- implemented method for determining at least one characteristic associated with at least one obj ect in a limited space according to an example embodiment .

[0037] FIG . 2 illustrates a block diagram of a system according to an example embodiment .

[0038] FIG . 3 illustrates an elevator system according to an example embodiment .

[0039] FIG . 4 illustrates an elevator car according to an example embodiment . FIG. 5A illustrates three-dimensional data from a time- of-f light sensor according to an example embodiment.

[0040] FIG. 5B illustrates two-dimensional data from a time- of-f light sensor according to an example embodiment.

[0041] FIG. 5C illustrates surface normal data according to an example embodiment.

[0042] FIG. 5D illustrates filtered surface normal data according to an example embodiment.

[0043] DETAILED DESCRIPTION

[0044] Various examples and embodiments discussed below illustrate a solution in which surface normal data can be used to determine one or more characteristics for at least one object in a limited space, for example, in an elevator car.

[0045] FIG. 1 illustrates a flow diagram of a computer- implemented method for determining at least one characteristic associated with at least one object in a limited space according to an example embodiment. The method may be implemented, for example, by a system or device associated with the elevator car, for example, an elevator controller or an elevator group controller. In another example, the system or device may be an external entity from an elevator system.

[0046] At 100, image data associated with an interior of a limited space may be received. The limited space may be, for example, an elevator car, a waiting area, a room or an elevator lobby. The image data may originate from a sensor configured to provide depth data arranged in the limited space. The sensor may be, for example, a 3D camera, a camera, a radar, a 3D radar, a stereo camera or any sensor providing depth data. In an example embodiment, the image data may originate from multiple cameras. The camera may be configured to provide two- dimensional image data or three-dimensional image data. In case of the three-dimensional image data, the camera may be, for example, a depth camera or a 3D time of flight (ToF) camera. A ToF camera is a 3D camera that uses pulses of invisible infrared laser light to illuminate a subject, and measures the time it takes for the reflected light to reach an image sensor to determine depth information. ToF cameras can illuminate the entire scene and record the depth information in it with a single pulse of laser light.

[0047] At 102, surface normal data may be determined based on the image data. A surface normal is a vector that is perpendicular to a surface at a specific point, i.e. it points directly away from the surface.

[0048] At 104, at least one characteristic associated with at least one object in the limited space based on the surface normal data. In an example, determining at least one characteristic associated with at least one object in the limited space comprises determining the number of passengers in the elevator car based on the result. In other words, the surface normal data may be used in determining the number of passengers in the elevator car. In other examples, the at least one object may comprise one or more objects that are not people. For example, the object may be a robot, hospital bed, a wheelchair, a suitcase, a bike, an electric bike etc.

[0049] In an example embodiment, the three-dimensional image data may be obtained from a sensor configured to provide depth data, for example, a depth camera. Then, partial derivatives of a depth image with respect to pixel coordinates may be obtained by computing a gradient of the depth image along the horizontal and vertical directions. The pixel coordinates may then be converted to sensor coordinates by using an intrinsic matrix of the sensor, and a normal vector may be calculated for each pixel value. Finally, the normal vectors may be mapped to a correct color range to provide the surface normal data.

[0050] In an example embodiment, the surface normal data may be input to an analyzing module, and a result associated with the least one object may be received from the analyzing module. In an example, the analyzing module may comprise a neural network model. The neural network model may have been taught with surface normal data obtained based on 3D image data to determine at least one characteristic, for example, the number of people in the image data. When the surface normal data is then input to the neural network model, the neural network model outputs the characteristic, for example, the number of people. The calculated surface normal can be used with object detection models. A model can be trained to detect any object about which image data is available. The object may be, for example, a wheelchair, a bed, an electric scooter or any other object that can be transported in an elevator car. In an example, training data for the neural network model may be generated using artificial intelligence. For example, it is possible to use a few, for example, 20-40 pictures of real data to teach a neural network model to generate similar training data. The result of this (i.e., thousands of pictures) may then ube used to teach the neural network model used for inferencing.

[0051] In another example, the analyzing module may comprise a heuristic algorithm. For example, a two-dimensional image may represent a height field. For example, grayscale intensity values may be regarded as heights. The gradient of the height values in may then be computed both in x and y directions , and the cross product of these gradients provides the surface normal at each pixel .

[0052] In an example embodiment , the method may further comprise observing changes in the surface normal data over time , and using the observed changes in determining the at least one characteristic associated with at least one obj ect . The surface normal changes may be observed, for example , when elevator car doors of an elevator car are closed and / or during the ride of the elevator car . As people normally slightly move when being in the elevator, the changes observed in the surface normal data may enhance the accuracy of determining the number of people in the elevator car . In another example , based the changes observed in the surface normal data it may be possible to detect a person or person entering or existing the elevator car .

[0053] FIG . 2 illustrates a block diagram of a system 200 according to an example embodiment . The system 200 comprises one or more proces sors 202 , and one or more memories 204 that comprise computer program code 206 , and / or a communication interface 208 for wired and / or wireless communication . Although the system 200 is depicted to include only one processor 202 , the system 200 may include more than one processor . In an example , the memory 204 is capable of storing instructions , such as an operating system and / or various applications .

[0054] Furthermore , the processor 202 is capable of executing the stored instructions . In an example embodiment , the processor 202 may be embodied as a multi-core processor, a single core processor, or a combination of one or more multi-core processors and one or more single core processors . For example , the processor 202 may be embodied as one or more of various process ing devices , such as a coprocessor, a microprocessor, a controller, a digital signal processor ( DSP ) , a processing circuitry with or without an accompanying DSP, or various other processing devices including integrated circuits such as , for example , an application speci fic integrated circuit (AS IC ) , a field programmable gate array ( FPGA) , a microcontroller unit (MCU) , a hardware accelerator, a special-purpose computer chip, or the like . In an example embodiment, the processor 202 may be configured to execute hard-coded functionality . In an example embodiment , the processor 202 is embodied as an executor of software instructions , wherein the instructions may speci fically configure the processor 202 to perform the algorithms and / or operations described herein, when the instructions are executed, for example , the steps discussed relating to FIG . 1 .

[0055] The memory 204 may be embodied as one or more volatile memory devices , one or more non-volatile memory devices , and / or a combination of one or more volatile memory devices and non-volatile memory devices . For example , the memory 204 may be embodied as semiconductor memories ( such as mask ROM, PROM (programmable ROM) , EPROM ( erasable PROM) , flash ROM, RAM ( random access memory) , etc . ) .

[0056] The at least one memory 204 may store program instructions that , when executed by the at least one processor 202 , cause the system 200 to perform the functionality of the various embodiments discussed herein . Further, in an embodiment , at least one o f the processor 202 and the memory 204 may constitute means for implementing the discussed functionality . Further, the system 200 may be configured to receive image data associated with an interior of a limited space , determine surface normal data based on the image data, and determine at least one characteristic associated with at least one obj ect in the limited space based on the surface normal data .

[0057] A computer program may comprise instructions which, when the program is executed by the at least one processor 202 , cause the system 200 to perform any of the methods described above . Furthermore , a computer-readable medium may comprise the computer program .

[0058] FIG . 3 illustrates how the system 200 may be utili zed in an example system . In FIG . 3 , a sensor 300 may be configured to provide image data, for example , depth data associated with an interior of a limited space , for example , an elevator car . The sensor 300 may be , for example , a camera, a time-of- f light ( ToF) camera, a radar or any other sensor providing depth data to the system 200 . The system 200 may be installed in any other suitable location in an elevator system . In another example , the system 200 may be an external entity from the elevator system and the system 200 may be communicatively connected to an elevator controller 302 or an elevator group controller 304 . In the example illustrated in FIG . 3 , the system 200 may be configured to determine surface normal data based on the image data and determine at least one characteristic associated with at least one obj ect in the limited space based on the surface normal data . The system 300 may be configured to transmit the determined characteristic, for example , the number of people in the limited space or in the elevator car, to the elevator control ler 302 or to the elevator group controller 304 .

[0059] FIG . 4 illustrates an example of an elevator car 400 , comprising the sensor 300 , for example , a ToF camera, installed in a ceiling of the elevator car 400 . The ToF camera 300 is communicatively connected to the system 200 . The connection may be a wired or wireless connection .

[0060] FIG . 5A illustrates three-dimensional data from a time- of- f light sensor according to an example embodiment .

[0061] FIG . 5B illustrates two-dimensional data from a time- of- f light sensor according to an example embodiment .

[0062] FIG . 5C illustrates surface normal data according to an example embodiment . The surface normal data has been generated based on two-dimensional data from a time-of- flight sensor . More speci fically, FIG . 5C illustrates surface normal data of people inside an elevator car .

[0063] FIG . 5D illustrates filtered surface normal data according to an example embodiment . More speci fically, FIG . 5D illustrates surface normal data of people inside an elevator car .

[0064] One or more of the above discussed examples and example embodiments may enable a solution in which surface normal can help to distinguish between a person and other elements in a limited space , for example , an elevator car by providing detailed geometric information . This is especially useful in crowded situations where people are closely aligned or partially occluded . One or more of the above discussed examples and example embodiments may enable a solution in which by observing changes in the surface normals over time , it may be possible to detect movements or actions , such as a person entering or exiting the limited space or the elevator car . Thus , the surface normals can enhance the performance and accuracy of people detection models used in elevator systems . The examples discussed above may be implemented in software, hardware, application logic or a combination of software, hardware and application logic. The example devices can store information relating to various methods described herein. This information can be stored in one or more memories, such as a hard disk, a solid state drive (SSD) , an optical disk, a magneto-optical disk, an RAM, and the like. One or more databases can store the information used to implement the examples. The databases can be organized using data structures (e.g., records, tables, arrays, fields, graphs, trees, lists, and the like) included in one or more memories or storage devices listed herein. The methods described with respect to the examples can include appropriate data structures for storing data collected and / or generated by the methods of the devices and subsystems of the examples in one or more databases.

[0065] The components of the examples may include computer readable medium or memories for holding instructions programmed according to the teachings and for holding data structures, tables, records, and / or other data described herein. In an example, the application logic, software or an instruction set is maintained on any one of various conventional computer-readable media. In the context of this document, a "computer-readable medium" may be any media or means that can contain, store, communicate, propagate or transport the instructions for use by or in connection with an instruction execution system, apparatus, or device, such as a computer. A computer-readable medium may include a computer- readable storage medium that may be any media or means that can contain or store the instructions for use by or in connection with an instruction execution system, apparatus, or device, such as a computer. A computer readable medium can include any suitable medium that participates in providing instructions to a processor for execution . Such a medium can take many forms , including but not limited to , non-volatile media, volatile media, transmission media, and the like .

[0066] While there have been shown and described and pointed out fundamental novel features as applied to preferred examples thereof , it will be understood that various omissions and substitutions and changes in the form and details of the devices and methods described may be made by those skilled in the art without departing from the spirit o f the disclosure . For example , it is expres sly intended that all combinations of those elements and / or method steps which perform substantially the same function in substantially the same way to achieve the same results are within the scope of the disclosure . Moreover, it should be recogni zed that structures and / or elements and / or method steps shown and / or described in connection with any disclosed form or example may be incorporated in any other disclosed or described or suggested form as a general matter of design choice . Furthermore , in the claims means-plus- function clauses are intended to cover the structures described herein as performing the recited function and not only structural equivalents , but also equivalent structures .

[0067] The applicant hereby discloses in isolation each individual feature described herein and any combination of two or more such features , to the extent that such features or combinations are capable of being carried out based on the present speci fication as a whole , in the light of the common general knowledge of a person skilled in the art , irrespective of whether such features or combinations of features solve any problems disclosed herein, and without limitation to the scope of the claims . The applicant indicates that the disclosed aspects / embodiments may consist of any such individual feature or combination of features . In view of the foregoing description it will be evident to a person skilled in the art that various modi fications may be made within the scope of the disclosure .

Claims

CLAIMS1. A system (200) , comprising: at least one processor (202) ; and at least one memory (204) storing instructions that, when executed by the at least one processor (200) , cause the system (200) to at least perform: receiving image data associated with an interior of a limited space; determining surface normal data based on the image data; and determining at least one characteristic associated with at least one object in the limited space based on the surface normal data.

2. The system (200) according to claim 1, wherein the image data comprises two-dimensional image data .

3. The system (200) according to claim 1, wherein the image data comprises three-dimensional image data .

4. The system (200) according to claim 3, wherein the three-dimensional image data is obtained from a sensor configured to provide depth data, and the at least one memory (204) stores instructions that, when executed by the at least one processor (200) , cause the system (200) to at least perform: obtaining partial derivatives of a depth image with respect to pixel coordinates by computing a gradient of the depth image along the horizontal and vertical directions; converting the pixel coordinates to sensor coordinates by using an intrinsic matrix of the sensor; calculating a normal vector for each pixel value; andmapping the normal vectors to a correct color range to provide the surface normal data.

5. The system (200) according to any one of claims 1 - 4, wherein when determining at least one object in the limited space based on the surface normal map, and the at least one memory (204) stores instructions that, when executed by the at least one processor (200) , cause the system (200) to at least perform: inputting the surface normal data to an analyzing module; and receiving a result associated with the least one object from the analyzing module.

6. The system (200) according to claim 5, wherein the analyzing module comprises a neural network model .

7. The system (200) according to claim 5, wherein the analyzing module comprises a heuristic algorithm.

8. The system (200) according to any one of claims 1 - 7, wherein determining at least one characteristic associated with at least one object in the limited space comprises determining the number of passengers in an elevator car based on the result.

9. The system (200) according to any one of claims 1 - 8, the at least one memory (204) stores instructions that, when executed by the at least one processor (200) , cause the system (200) to at least perform: observing changes in the surface normal data over time; andusing the observed changes in determining the at least one characteristic associated with at least one ob j ect .

10. An elevator system comprising: a system (200) according to any of claims 1 - 9 ; and a sensor (300) configured to provide image data associated with an interior of a limited space to the system (200) .

11. The elevator system according to claim 10, wherein the sensor (300) comprises a depth camera.

12. A computer-implemented method comprising: receiving image data associated with an interior of a limited space; determining surface normal data based on the image data; and determining at least one characteristic associated with at least one object in the limited space based on the surface normal data.

13. The computer-implemented method) according to claim 12, wherein the image data comprises two- dimensional image data.

14. The computer-implemented method according to claim 12, wherein the image data comprises three- dimensional image data.

15. The computer-implemented method according to claim 14, wherein the three-dimensional image data is obtained from a sensor configured to provide depth data, and the method further comprises: obtaining partial derivatives of a depth image with respect to pixel coordinates by computing agradient of the depth image along the horizontal and vertical directions; converting the pixel coordinates to sensor coordinates by using an intrinsic matrix of the sensor; calculating a normal vector for each pixel value; and mapping the normal vectors to a correct color range to provide the surface normal data.

16. The computer-implemented method according to any one of claims 12 - 15, wherein determining at least one object in the limited space based on the surface normal map comprises: inputting the surface normal data to an analyzing module; and receiving a result associated with the least one object from the analyzing module.

17. The computer-implemented method according to claim 16, wherein the analyzing module comprises a neural network model.

18. The computer-implemented method according to claim 16, wherein the analyzing module comprises a heuristic algorithm.

19. The system (200) according to any one of claims 12 - 18, wherein determining at least one characteristic associated with at least one object in the limited space comprises determining the number of passengers in the limited space based on the result.

20. The method according to any one of claims 12 - 19, further comprising: observing changes in the surface normal data over time; andusing the observed changes in determining the at least one characteristic associated with at least one ob j ect . 21 . A computer program comprising instructions which, when the program is executed by at least one processor, cause a system to perform the method of any of claims 12 - 20 . 22 . A computer-readable medium compri sing a computer program comprising instructions which, when the program is executed by at least one processor, cause a system to perform the method of any of claims 12 - 20 .

Images