System, camera device, and method for displaying collected sensor data together with images

Abstract

To provide a mechanism for displaying collected sensor data together with images.SOLUTION: A camera device comprises an image capturing unit, a communication interface, and processing circuitry. The image capturing unit is configured with a variable zoom level. Images depicting a scene are captured by the image capturing unit. By the communication interface, communication is performed with a user interface for displaying the image and with a sensor device for obtaining sensor data collected from the sensor device for display on the user interface together with the images. By the processing circuitry and via the communication interface, the sensor device is configured to provide the collected sensor data at a higher frequency of occurrence when the image capturing unit captures images depicting the sensor device with a zoom level above a zoom threshold value than that when the image capturing unit captures images with a zoom level below the zoom threshold value.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The embodiments presented herein relate respectively to a system, a camera device, a method, a computer program, and a computer program product for displaying collected sensor data along with an image. [Background technology]

[0002] In general, a sensor device is a device, module, machine, or subsystem whose purpose is to detect events or changes in its environment and transmit that information to other electronic equipment, such as a computer processor, for further processing or to display to a user in a user interface. To do so, the sensor device consumes power. There can be various ways to provide power to a sensor device. In some deployments, it can be beneficial to use a sensor device that consumes as little power as possible. This can be the case in deployments with multiple sensor devices and / or deployments where sensors are battery operated, where overall power consumption therefore needs to be limited, or in other deployments where sensors have access to only very limited power resources but should still have a long lifespan. Furthermore, even if not required by limited power resources, lowering power consumption may be desirable from an environmental or cost perspective.

[0003] As a non-limiting example for illustrative purposes, assume that the sensor device is a Z-Wave type sensor device. As such, the Z-Wave device is battery-powered and more energy-efficient. This is because the Z-Wave device is configured with a sleep state, and once in the sleep state, the Z-Wave device does not accept control from an external controller. After a period of time, the Z-Wave device wakes up from sleep, exchanges commands with the controller, and then re-enters the sleep state. The Z-Wave device may be configured with minimum, maximum, and default values ​​for the duration of the sleep state. Similarly, the wake time can be set within a range from a minimum to a maximum value.

[0004] When a user accesses a Z Wave device, the user wants an immediate response from the device. For this purpose, it is necessary to set a short sleep time (i.e., the duration of the sleep state is short). However, if the sleep time is set too short, the power consumption of the device increases, and in some cases, the power of the sensor device is lost faster than desired. On the one hand, for the convenience of the user, the sleep time should be short, but on the other hand, for the convenience of extending the operating life of the sensor device, the sleep time should be long. This creates a conflict of interest in how the sleep time should be selected.

[0005] Similar considerations regarding the conservation of sensor device power resources relative to a user's need to access sensor data exist for other types of sensor devices besides Z-Wave devices. Thus, selecting a sleep time or similar parameter to enable the sensor device to extend its operational life while still allowing sensor data to be provided to the user as needed can be cumbersome. Summary of the Invention

[0006] The purpose of each of the embodiments presented herein is to address the above-mentioned problems.

[0007] Generally, in accordance with the inventive concepts disclosed herein, a sensor device is monitored by an imaging unit, and the rate at which the sensor device provides sensor data is determined as a function of the zoom level used by the imaging unit, which allows the sensor data collected by the sensor device to be variably displayed on a user interface.

[0008] According to a first aspect, a related concept of the present invention is defined by a system for displaying collected sensor data together with images. The system includes a camera device. The camera device includes an imaging unit configured to capture images depicting a scene. The imaging unit is configured with a variable zoom level. The camera device is configured to communicate with a user interface to display the images. The system includes a sensor device configured to collect sensor data and provide the collected sensor data to the camera device for display together with the images on the user interface at a variable frequency of occurrence. The sensor device is configured by the camera device to provide the collected sensor data with a higher frequency of occurrence when the camera device acquires images depicting the sensor device from the imaging unit that were taken at a zoom level above a zoom threshold than when the camera device acquires images from the imaging unit that were taken at a zoom level below the zoom threshold.

[0009] According to a second aspect, the present invention provides a camera device for displaying collected sensor data together with images. The camera device includes an imaging unit configured to capture images depicting a scene. The imaging unit is configured with a variable zoom level. The camera device includes a communication interface configured to communicate with a user interface to display the images and to obtain collected sensor data from the sensor device for display on the user interface together with the images. The camera device includes a processing circuit configured, via the communication interface, to configure the sensor device to provide the collected sensor data with a higher frequency of occurrence when the imaging unit captures images depicting the sensor device at a zoom level above a zoom threshold than when the imaging unit captures images at a zoom level below the zoom threshold.

[0010] According to a third aspect, the present invention provides a method for displaying collected sensor data together with images. The method is performed by a camera device. The camera device includes an imaging unit, a communication interface, and a processing circuit. The imaging unit is configured with a variable zoom level. The method includes capturing images depicting a scene by the imaging unit. The method includes communicating with a user interface by the communication interface to display the images and to obtain collected sensor data from the sensor device for display with the images on the user interface. The method includes configuring the sensor device by the processing circuit via the communication interface to provide the collected sensor data with a higher frequency of occurrence when the imaging unit captures images depicting the sensor device at a zoom level above a zoom threshold than when the imaging unit captures images at a zoom level below the zoom threshold.

[0011] According to a fourth aspect, the inventive concept is defined by a computer program for displaying collected sensor data together with an image, the computer program comprising computer program code which, when executed on a camera device, causes the camera device to perform the method according to the third aspect.

[0012] According to a fifth aspect, the inventive concept is defined by a computer program product including a computer program according to the fourth aspect and a computer-readable storage medium on which the computer program is stored, which may be a non-transitory computer-readable storage medium.

[0013] Advantageously, the inventive concept allows a sleep time or similar parameter to be selected to allow a long operational life for the sensor device, while at the same time allowing sensor data to be provided to the user as needed.

[0014] Advantageously, the inventive concept allows for a balance between the user's need to access sensor data and the preservation of power resources of the sensor device.

[0015] Advantageously, the inventive concept reduces the power consumption of a sensor device without reducing the information provided to a user interacting with the user interface, since a user is more likely to be interested in the sensor data of a sensor device if the camera is zoomed in to depict an area that includes the sensor device.

[0016] It should be noted that any feature of the first, second, third, fourth, and fifth aspects may be applied to any of the other aspects, where appropriate. Similarly, any advantage of the first aspect may equally be applied to the second, third, fourth, and / or fifth aspects, respectively, and vice versa. Other objects, features, and advantages of the embodiments encompassed herein will become apparent from the following detailed disclosure, from the appended dependent claims, as well as from the accompanying drawings.

[0017] In general, all terms used in the claims are to be interpreted according to their ordinary meaning in the art unless expressly defined herein. Any reference to "a / an / the element, apparatus, component, means, module, step, etc." is openly interpreted as referring to at least one instance of that element, apparatus, component, means, module, step, etc., unless expressly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless expressly stated.

[0018] The concepts relating to the present invention will now be described, by way of example only, with reference to the accompanying drawings. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 is a schematic diagram illustrating a system according to one embodiment. [Figure 2] 10A and 10B are schematic diagrams illustrating user interfaces according to the respective embodiments. [Figure 3] 10A and 10B are schematic diagrams illustrating user interfaces according to the respective embodiments. [Figure 4] 1 is a flowchart of a method according to each of the embodiments. [Figure 5] 1 is a flowchart of a method according to each of the embodiments. [Figure 6] 1 is a flowchart of a method according to each of the embodiments. [Figure 7]1 is a flowchart of a method according to each of the embodiments. [Figure 8] 1 is a flowchart of a method according to each of the embodiments. [Figure 9] FIG. 1 is a schematic diagram illustrating functional units of a camera device according to one embodiment. [Figure 10] 1 illustrates an example of a computer program product including a computer-readable storage medium according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0020] The inventive concepts are described in more detail below with reference to the accompanying drawings, which illustrate specific embodiments of the inventive concepts. The inventive concepts may, however, be embodied in many different forms and should not be understood as limited to the embodiments set forth below. Rather, these embodiments are provided for illustrative purposes so that this disclosure will be thorough and complete, and will fully convey the scope of the inventive concepts to those skilled in the art. Like reference numerals refer to like elements throughout this specification. Any steps indicated by dotted lines should be considered optional.

[0021] As noted above, selecting a sleep time or similar parameter to enable the sensor device to extend its operational life while still allowing sensor data to be provided to the user as needed can be cumbersome.

[0022] Each of the embodiments disclosed herein therefore relates to a mechanism for variably displaying collected sensor data. To achieve such a mechanism, there is provided a system 100, a camera device 110, a method performed by the camera device 110, and a computer program product including code, such as in the form of a computer program, that, when executed on the camera device 110, causes the camera device 110 to perform the method.

[0023] As an example, consider a scenario in which sensor data collected from a sensor device is overlaid or otherwise superimposed, and an image of the sensor device is displayed on a user interface, e.g., according to the setup of FIG. 1 . FIG. 1 is a schematic diagram illustrating a system 100 to which the embodiments presented herein can be applied. The system 100 includes a camera device 110. The camera device 110 includes an imaging unit 120. The imaging unit 120 is configured to capture digital images depicting a scene 160. The imaging unit 120 is configured with a variable zoom level. That is, the imaging unit 120 can capture images using a variable zoom level. Different examples of how the zoom level can be changed are disclosed below. In FIG. 1 , reference numeral 150a schematically illustrates the field of view of the imaging unit 120 when the zoom level is fully zoomed out, i.e., the widest field of view that the imaging unit 120 can achieve. Reference numeral 150b schematically illustrates the field of view of imaging unit 120 when the zoom level is fully zoomed in, i.e., corresponding to the imaging unit's 120's narrowest field of view. Fields of view 150a and 150b indicate that imaging unit 120 points in the same direction. In this regard, there may be various ways in which the zoom level can be changed. In one non-limiting example, the zoom level can be changed using an optical zoom of an optical lens system that is part of or attached to imaging unit 120 and controlled by camera device 110. In one non-limiting example, the zoom level can be changed using digital zoom, and the zoom level is therefore implemented in software provided to either imaging unit 120 or camera device 110. In some examples, imaging unit 120 can (digitally) pan, tilt, and zoom (PTZ) and thus can be considered a (digital) PTZ imaging unit. Camera device 110 is configured to communicate with user interface 130 to display images. Furthermore, camera device 110 is configured to encode images according to any known video coding standard, such as H.264 or H.265.In this regard, the encoding may occur directly in conjunction with the imaging unit 120 capturing the image, or at a separate entity, such as the computing server 114, followed by at least temporary storage in the data storage 116. The computing server 114 and the data storage 116 are therefore considered to be part of the camera device 110. The computing server 114 and the data storage 116 are operatively connected to the imaging unit 120 via the network 112. The network 112 may be wired or wireless, or may be partly wired and partly wireless.

[0024] In some embodiments, camera device 110 includes a communication interface 920 configured to communicate with user interface 130 to display those images and with sensor devices 170a, 170b to acquire sensor data collected from sensor devices 170a, 170b for display along with those images on user interface 130. If those images are encoded, corresponding decoding follows before the images are displayed on user interface 130. In some aspects, a video management system (VMS) may be provided in association with user interface 130, configured to decode the received sequence of encoded images. Further aspects of communication interface 920 are disclosed below with reference to FIG. 9 . Reference numeral 140 denotes an example connection between camera device 110 and user interface 130. Connection 140 may be wired or wireless, or may be partially wired and partially wireless. A user 180 may interact with user interface 130. It is understood that user interface 130 is, at least in part, a visual user interface. 1, as it is configured to display images captured by the imaging unit 120. Further aspects of the user interface 130 are respectively disclosed below with reference to Figure 2. Further aspects of the user 180 that may interact with the interface 130 are respectively disclosed below with reference to Figure 3.

[0025] System 100 further includes at least one sensor device 170a, 170b. Each of sensor devices 170a, 170b is configured to collect sensor data and provide the collected sensor data to camera device 110 for display together with the images on user interface 130. The sensor data may then be added as an overlay to the acquired images when they are displayed on user interface 130, for example, at the location of the sensor device in the image. The overlay may be added directly to the sequence of images or may be transmitted separately as metadata and added to the images only when they are displayed on user interface 130, or at least after they have been decoded. The overlay may then be encoded together with and / or separately from the images.

[0026] As part of the conceptual development of the present invention, the inventors of the respective embodiments disclosed herein recognized that when a user 180 monitors the user interface 130 in a situation where images are captured with a wide field of view, the user 180 is more likely to be interested in sensor data from nearby sensor devices than from sensor devices that are farther away. Conversely, when the user 180 zooms in on a sensor device (nearby or far away), the user 180 is more likely to be interested in the sensor data from the sensor device depicted at the zoomed-in location than in sensor data from sensor devices that are not zoomed in on. In the latter case, even if sensor data acquired from sensor devices that the user 180 is not interested in is displayed, the zoomed-out view on the user interface 130 becomes cluttered and difficult to read. Furthermore, there is a risk of unnecessary power loss to the sensor devices.

[0027] Generally, the inventive concepts disclosed herein are therefore based on addressing the above-mentioned problems and providing an efficient system 100 of sensor devices 170a, 170b and camera device 110 that allows for efficient overlay of sensor data onto captured images. According to the inventive concepts, this is achieved by having the sensor devices 170a, 170b monitored by the imaging unit 120, and determining the rate at which the sensor devices 170a, 170b provide sensor data to the camera device 110 as a function of the zoom level used by the imaging unit 120.

[0028] In particular, sensor devices 170a, 170b are configured to provide collected sensor data, respectively, to camera device 110 when camera device 110 acquires from imaging unit 120 images depicting sensor devices 170a, 170b taken at a zoom level above the zoom threshold, with a higher frequency of occurrence than when camera device 110 acquires from imaging unit 120 images taken at a zoom level below the zoom threshold. In some embodiments, camera device 110 includes processing circuitry 910 configured to configure sensor devices 170a, 170b via communications interface 920. Further aspects of processing circuitry 910 are disclosed below with reference to FIG. 9 . Each of sensor devices 170a, 170b can thus collect sensor data and provide the collected sensor data to camera device 110 for display along with the images on user interface 130 with a variable frequency of occurrence.

[0029] In other words, the frequency with which the sensor devices 170a, 170b provide the collected sensor data to the camera device 110 thus depends on the current zoom level used by the imaging unit 120. This dependency is implemented using a zoom threshold. When the imaging unit 120 uses a zoom level above the zoom threshold (i.e., towards the telephoto end), this is an indication that the depicted sensor devices 170a, 170b will be enlarged. As a result, the update frequency is set to a higher value and the sensor data is added as an overlay. Conversely, when the imaging unit 120 uses a zoom level below the zoom threshold (i.e., towards the wide-angle end), this is an indication that the depicted sensor devices 170a, 170b will be reduced. As a result, the update frequency is set to a lower value and the sensor data is not added as an overlay, as an optional feature.

[0030] FIG. 1 illustrates an example of a system 100 having two sensor devices 170a, 170b. Each of the embodiments disclosed herein is not limited to any particular number of sensor devices 170a, 170b. However, some embodiments contemplate that there are at least two sensor devices 170a, 170b, each associated with its own zoom threshold. That is, in some embodiments, the system 100 includes at least two sensor devices 170a, 170b, each configured to collect sensor data and provide the collected sensor data, at a respective variable frequency of occurrence, for display along with their images on the user interface 130. Each of the at least two sensor devices 170a, 170b may then be associated with its own zoom threshold. How each of the sensor devices 170a, 170b may be configured with or associated with its own zoom threshold is disclosed below.

[0031] FIG. 2 schematically illustrates user interface 130 according to one embodiment when displaying images captured by imaging unit 120 of scene 160. FIG. 2 illustrates user interface 130 as a screen, which may be a touchscreen that allows a user to interact with user interface 130. However, there may be various ways for a user to interact with user interface 130. For example, user interface 130 may be configured to allow a user to move a cursor on user interface 130. In this regard, user interface 130 may be provided with or operably connected to a navigation tool, such as a mouse, keyboard, joystick, etc., to allow a user to move a cursor on user interface 130. As described above, user interface 130 is operably connected to camera device 110 via connection 140. As disclosed further below, this allows system 100 to detect a user click on user interface 130 (either using touch input on the touchscreen or via an indication from the navigation tool). Additionally, the user interface 130 may be operably connected to a computer, a terminal, and / or a communications network.

[0032] Reference numeral 132a denotes a view of scene 160 when imaging unit 120 is zoomed out to the maximum (i.e., corresponding to field of view 150a). Reference numeral 132b denotes a view of scene 160 when imaging unit 120 is zoomed in to the maximum (i.e., corresponding to field of view 150b). Sensor device 170b is closer to imaging unit 120 than sensor device 170a, and therefore sensor device 170b appears larger than sensor device 170a when displayed on user interface 130. Furthermore, because sensor device 170b is outside the narrowest field of view of imaging unit 120, sensor device 170b is not displayed when the zoom level is at the maximum zoom-in.

[0033] Further detailed embodiments of system 100, particularly camera device 110 and sensor devices 170a, 170b, are disclosed below with successive reference to FIGS. 1 and 2, respectively. Reference is also made to FIG. 3, and particularly FIG. 3(a). In the illustrative example of FIG. 3, sensor devices 170a, 170b each represent a temperature sensor, and thus the sensor data represents the temperature readings of sensor devices 170a, 170b. That is, sensor devices 170a, 170b in FIG. 3 represent a certain type of weather sensor. However, those skilled in the art will understand that there may be other types of sensor devices with corresponding types of sensor data, such as, but not limited to, light, voltage, current, pressure, humidity, air pollution, fire, gas, etc., and that the embodiments disclosed herein are not limited to any particular type of sensor device or sensor data. In some embodiments, sensor devices 170a, 170b are battery-powered Z-Wave devices. Other embodiments include sensor devices that communicate via Bluetooth, ZigBee, or any other communication protocol suitable for other applications, respectively.

[0034] In some aspects, the sensor data is displayed as overlays 172a, 172b only when the zoom level is higher than the zoom threshold for the corresponding sensor device 170a, 170b. In particular, in some embodiments, when camera device 110 acquires images depicting sensor devices 170a, 170b taken at a zoom level that exceeds the zoom threshold, camera device 110 is configured to provide the sensor data as overlays 172a, 172b on the images. That is, the first condition is that the zoom level is higher than the zoom threshold for sensor devices 170a, 170b, and the second condition is that sensor devices 170a, 170b are depicted in those images (and therefore displayed on user interface 130). In the example of FIG. 3(a), for purposes of explanation, it is assumed that the zoom level exceeds the zoom threshold for sensor device 170a but is less than the zoom threshold for sensor device 170b. Thus, the sensor data is displayed as an overlay 172a (showing a temperature value of 24.6° C.) for sensor device 170a.

[0035] In some aspects, representations of sensor devices 170a, 170b are displayed when the zoom level is below the zoom threshold for the corresponding sensor device 170a, 170b. In particular, in some embodiments, when camera device 110 captures an image depicting sensor device 170a, 170b taken at a zoom level below the zoom threshold, camera device 110 is configured to display illustrations 174 representing sensor device 170a, 170b on user interface 130 as overlays 172a, 172b on the image. Non-limiting examples of illustrations include icons and symbols. In the example of FIG. 3(a), illustration 174 in the form of an icon for sensor 170b is displayed because, as disclosed above, it is assumed that the zoom level is below the zoom threshold for sensor device 170b (and that sensor device 170b is displayed on user interface 130).

[0036] As previously disclosed, the system 100 may be configured to detect a click by a user on the user interface 130. Assume that this click occurs on a location on the user interface 130. A comparison may then be made to the saved image positions of the sensor devices 170a, 170b to determine which sensor device 170a, 170b (if any) was selected by the user. That is, the sensor devices 170a, 170b have a location in the scene 160 in the image. Because this location is in the image (as opposed to the actual location of the sensor devices 170a, 170b), this location may be referred to as an image location. The image locations of the sensor devices 170a, 170b in the scene 160 have a mapping to locations on the user interface 130 when the sensor devices 170a, 170b are displayed on the user interface 130. In this regard, there may be one mapping for each zoom level, pan direction, and tilt direction of the imaging unit 120 (depending on whether the imaging unit 120 is capable of panning and / or tilting). In some embodiments, the camera device 110 is configured to determine that a user input identifies a representation 174 of the sensor devices 170a, 170b by verifying that the location of the user input on the user interface 130 corresponds to the image location of the sensor devices 170a, 170b on the user interface 130. This therefore allows the system 100 to determine whether the user has “clicked” on one of the sensor devices 170a, 170b displayed on the user interface 130. FIG. 3(a) shows an example in which the hand 182 of the user 180 clicks on the depicted sensor device 170b. Actions taken by the system 100 upon receiving an input representing a user clicking on the depicted sensor device 170b will be further disclosed below with reference to FIGS. 3(b) and 3(c).

[0037] In some aspects, user 180 clicking on illustration 174 of sensor device 170b causes sensor data for that sensor device 170b to be acquired and displayed. In particular, in some embodiments, user interface 130 is configured to receive user input, which identifies representation 174 of sensor device 170a, 170b, and camera device 110 is configured to responsively display the sensor data on user interface 130 as overlays 172a, 172b on the image. This is illustrated in FIGS. 3(a) and 3(b). It is assumed in FIG. 3(a) that hand 182 of user 180 clicks on sensor device 170b, resulting in a transition from FIG. 3(a) to FIG. 3(b), as indicated by arrow 190a. FIG. 3(b) illustrates the sensor data acquired by sensor device 170b as overlay 172b on sensor device 170b (showing a temperature value of 20.1°C). Thus, sensor data for sensor device 170b is displayed in response to user input even though the zoom level is below the zoom threshold for sensor device 170b.

[0038] In some aspects, user 180 clicking on illustration 174 of sensor device 170b causes imaging unit 120 to zoom in on sensor device 170b to at least the zoom threshold to display sensor data for that sensor device 170b. In particular, in some embodiments, user interface 130 is configured to receive user input identifying representation 174 of sensor device 170a, 170b, and camera device 110 is configured to responsively cause imaging unit 120 to zoom in to the zoom threshold. This is illustrated in FIGS. 3(a) and 3(c). It is assumed that in FIG. 3(a), hand 182 of user 180 clicks on sensor device 170b, resulting in a transition from FIG. 3(a) to FIG. 3(c), as indicated by arrow 190b. In FIG. 3(c), imaging unit 120 has zoomed in on sensor device 170b to the zoom threshold of sensor device 170b. As a result, sensor data 172b for sensor device 170b is displayed as an overlay 172b (showing a temperature value of 20.1° C.) for sensor device 170b.

[0039] Aspects of how camera device 110 may configure sensor devices 170a, 170b, respectively, are disclosed below.

[0040] In some aspects, a calibration procedure is performed to determine the zoom threshold for each of sensor devices 170a, 170b. That is, in some embodiments, camera device 110 is configured to establish the zoom threshold during a calibration procedure. An example of such a calibration procedure is disclosed below.

[0041] In some aspects, the presence of a given sensor device 170a, 170b within a given field of view of the imaging unit 120 is listed by referencing position information acquired and stored during a calibration procedure. In particular, in some embodiments, the imaging unit 120 has a geometric relationship to the sensor devices 170a, 170b established during the calibration procedure. This geometric relationship includes zoom-level-dependent position information for the sensor devices 170a, 170b. The zoom-level-dependent position information specifies whether the sensor devices 170a, 170b are visible in an image captured using a given zoom level. Thus, the presence of a given sensor device 170a, 170b within a given field of view of the imaging unit 120 can be established without object recognition or other types of image analysis of images captured by the imaging unit 120.

[0042] As previously disclosed, sensor devices 170a, 170b are configured to provide sensor data collected by camera device 110 at a frequency of occurrence that depends on whether the zoom level of imaging unit 120 is above or below a zoom threshold. Further in this regard, camera device 110 may be configured to configure an update frequency value for sensor devices 170a, 170b to provide sensor data to camera device 110. In particular, in some embodiments, camera device 110 is configured to instruct sensor devices 170a, 170b to provide sensor data at the frequency of occurrence of sensor devices 170a, 170b. In some aspects, the update frequency value of sensor devices 170a, 170b can be switched between at least two predetermined values. In particular, in some embodiments, the frequency of occurrence can be varied by being selectable between a first predetermined value and a second predetermined value, the first predetermined value producing a higher frequency of occurrence than that of the second predetermined value. That is, the frequency of occurrence can be selectably set to either a first predetermined value or a second predetermined value, i.e., the frequency of occurrence can be selectably set to either a high frequency of occurrence or a low frequency of occurrence. There are various ways in which the predetermined values ​​can be selected. In some embodiments, the first predetermined value is provided by a wake time configuration for sensor devices 170a, 170b, and the second predetermined value is provided by a sleep time configuration for sensor devices 170a, 170b. This may be the case when sensor devices 170a, 170b are Z Wave devices. Here, the wake time configuration is defined by the value of a wake state parameter for the Z Wave device, and the sleep time configuration is defined by the value of a sleep state parameter. Generally, the wake state parameter is a parameter that assumes a value associated with the Z Wave being in a wake state, and the sleep state parameter is a parameter that assumes a value associated with the Z Wave being in a sleep state.In this regard, the value of the wake state parameter defines the frequency or rate of occurrence at which sensor data is collected by sensor devices 170a, 170b and provided to camera device 110 when the Z Wave device is in a wake state. Correspondingly, the value of the sleep state parameter defines the frequency or rate of occurrence at which sensor data is collected by sensor devices 170a, 170b and provided to camera device 110 when the Z Wave device is in a sleep state. In other words, when the frequency of occurrence is set to a first predetermined value, sensor devices 170a, 170b collect sensor data and provide the collected sensor data to camera device 110 according to the wake time configuration, which in turn provides the collected sensor data at a rate that may be defined by the value of the wake state parameter. Correspondingly, when the frequency of occurrence is set to a second predetermined value, sensor devices 170a, 170b collect sensor data and provide the collected sensor data to camera device 110 according to the sleep time configuration, which in turn provides the collected sensor data at a rate that may be defined by the value of the sleep state parameter. In some examples, when the frequency of occurrence is set to the second predetermined value, sensor data is not collected, and therefore no sensor data is provided, because the Z Wave device subsequently enters its sleep state. Further, in this regard, as described above, when the Z Wave device enters its sleep state, the Z Wave device may not accept control from an external controller until it next wakes up again. Thus, if sensor device 170a, 170b is a Z Wave device, updates to the value of the frequency of occurrence of sensor device 170a, 170b (i.e., the frequency of occurrence, either from high to low or from low to high) may be effective only when the Z Wave device is not in its sleep state. Thus, even if camera device 110 acquires from imaging unit 120 an image depicting sensor device 170a, 170b taken at a zoom level above the zoom threshold, sensor device 170a, 170b may still temporarily provide collected sensor data at a lower frequency of occurrence until it exits its sleep state.

[0043] In general, any of the previously disclosed aspects, embodiments, and examples of camera device 110 also apply to the methods disclosed below.

[0044] 4 is a flow chart illustrating one embodiment of a method for variably displaying collected sensor data, performed by camera device 110. The method is preferably provided as computer program 1020. As previously disclosed, camera device 110 includes imaging unit 120, communication interface 920, and processing circuitry 910, with imaging unit 120 configured with variable zoom levels.

[0045] S102: The imaging unit 120 captures an image depicting the scene 160.

[0046] S104: The communication interface 920 communicates with the user interface 130 to display those images and with the sensor devices 170a, 170b to obtain sensor data collected from the sensor devices 170a, 170b to display together with those images on the user interface 130.

[0047] S106: The processing circuit 910 configures the sensor devices 170a, 170b via the communication interface 920 to provide the collected sensor data at a frequency that is greater when the imaging unit 120 captures images depicting the sensor devices 170a, 170b at a zoom level that exceeds the zoom threshold than when the imaging unit 120 captures images at a zoom level that is less than the zoom threshold.

[0048] Further details of the variably displaying collected sensor data performed by camera device 110 are disclosed in each of the following embodiments.

[0049] 5 is a flowchart illustrating one embodiment of a method for establishing a relationship between the imaging unit 120 and the sensor devices 170a, 170b in a global or common coordinate system. This relationship can be used to determine whether the imaging unit 120 is pointing at or directed toward a given sensor device 170a, 170b (for a given zoom level). The relative positions of each of the sensor devices 170a, 170b in relation to the imaging unit 120 are represented and stored, for example, in the storage medium 930 of the camera device 110 (step S202). The zoom level is then moved toward the wide-angle end, and user input is received defining the furthest distance for displaying sensor data for each of the sensor devices 170a, 170b. The determined distances for each of the sensor devices 170a, 170b are stored, for example, in the storage medium 930 of the camera device 110, as respective first set values, one for each of the sensor devices 170a, 170b (step S204). Respective physical distances between the imaging unit 120 and each of the sensor devices 170a, 170b (for each of the sensor devices 170a, 170b) are calculated or predicted from the stored device positions. A value of the apparent distance when the zoom level is at the wide-angle end is calculated from the angle-of-view information of the imaging unit 120 (step S206). If the calculated value is greater than the first set value, a zoom level at which the value of the apparent distance is equal to the first set value is calculated and stored, for example, in the storage medium 930 of the camera device 110 as a zoom threshold for the sensor devices 170a, 170b (step S208). Through the above process, the zoom thresholds for each of the sensor devices 170a and 170b are calculated and stored.

[0050] 6 is a flowchart illustrating one embodiment of a method for variably displaying collected sensor data, based on at least some of the embodiments disclosed above. The camera device 110 obtains the current zoom level used by the imaging unit 120 (step S302). Any of the sensor devices 170a, 170b present within the corresponding field of view of the imaging unit 120 are listed (step S304), for example, by referring to the location information established and stored according to FIG. 5. A zoom threshold for the listed sensor devices 170a, 170b is obtained (step S306). The method branches depending on whether the current zoom value exceeds or is less than the zoom threshold. It is then checked whether the current zoom value exceeds or is less than the zoom threshold (step S308). If the current zoom value exceeds the zoom threshold, the update frequency value of the sensor device 170a, 170b is set to a high value (corresponding to a short sleep time) (step S310). Sensor data is returned from the sensor devices 170a, 170b displayed on the user interface 130 (step S312). If the current zoom value is less than the zoom threshold, the update frequency value of the sensor devices 170a, 170b is set to a low value (corresponding to a long sleep time) (step S314). An icon or other representation of the sensor device is displayed as an overlay at the location of the sensor device 170a, 170b on the image displayed on the user interface 130 (step S316).

[0051] FIG. 7 is a flowchart illustrating one embodiment of a method for variably displaying collected sensor data based on the embodiment of FIGS. 3(a) and 3(b). First, assume that the zoom level currently being used by the imaging unit 120 for a given sensor device 170b under consideration is below a zoom threshold. Thus, the situation is as depicted in FIG. 3(a). A representation 174, such as an icon, of the sensor device 170b is displayed as an overlay at the location of the sensor device 170b on the image displayed in the user interface 130. It is assumed that a user 180 interacts with the user interface 130 (e.g., by the user's hand 182 performing a touch action on the user interface 130, or by the user using a navigation tool to select the sensor device 170b) to identify the sensor device 170b (step S402). The location specified on the user interface 130 is compared with the saved image locations of the sensor devices 170a, 170b to determine that the sensor device 170b has been selected (step S404). Therefore, the update frequency value of the selected sensor device 170b is set to a high value (corresponding to the set short sleep time) (step S406). Sensor data collected by the sensor device 170b is acquired (step S408). Therefore, the acquired sensor data is displayed as an overlay 172b at the location of the sensor device 170b on the user interface 130 (step S410). This corresponds to the transition made from FIG. 3(a) to FIG. 3(b) according to the arrow 190a.

[0052] FIG. 8 is a flowchart illustrating one embodiment of a method for variably displaying collected sensor data based on the embodiment of FIGS. 3(a) and 3(c). As shown in FIG. 8, it is initially assumed that the zoom level currently being used by the imaging unit 120 for a given sensor device 170b under consideration is below a zoom threshold. Thus, the situation is as depicted in FIG. 3(a). A representation 174, such as an icon, of the sensor device 170b is displayed as an overlay at the location of the sensor device 170b on the image displayed in the user interface 130. It is assumed that a user 180 interacts with the user interface 130 (e.g., by the user's hand 182 performing a touch action on the user interface 130, or by the user using a navigation tool to select the sensor device 170b) to identify the sensor device 170b (step S502). The location specified on the user interface 130 is compared with the saved locations of the sensor devices 170a, 170b to determine that the sensor device 170b has been selected (step S504). It is checked whether the current zoom level for the sensor device 170b exceeds or is less than the zoom threshold for the sensor device 170b (step S506). If the current zoom level is less than the zoom threshold, the camera device 110 instructs the imaging unit 120 to zoom in at least to the zoom threshold for the sensor device 170b (step S508). As a result, the update frequency value of the thus selected sensor device 170b is set to a high value (corresponding to the set short sleep time) (step S510). The sensor data collected by the sensor device 170b is acquired, and the acquired sensor data is then displayed as an overlay 172b at the location of the sensor device 170b on the user interface 130 (step S512). This corresponds to the transition made from Figure 3(a) to Figure 3(c) following arrow 190b.

[0053] 9 illustrates, in terms of a number of functional units, components of camera device 110 according to one embodiment. Processing circuitry 910 is provided using any combination of one or more suitable central processing units (CPUs), multiprocessors, microcontrollers, digital signal processors (DSPs), etc., capable of executing software instructions stored on computer program product 1010 (such as in FIG. 10), e.g., in the form of storage medium 930. Processing circuitry 910 may further be provided as at least one application-specific integrated circuit (ASIC) or field-programmable gate array (FPGA). Processing circuitry 910 may perform the functions of computing server 114.

[0054] In particular, the processing circuitry 910 is configured to cause the camera device 110 to perform a set of operations or steps as disclosed above. For example, the storage medium 930 may store the set of operations, and the processing circuitry 910 may be configured to retrieve the set of operations from the storage medium 930 and cause the camera device 110 to perform the set of operations. These set of operations may be provided as a set of executable instructions. The storage medium 930 may perform the functionality of the data storage 116.

[0055] Thus, processing circuitry 910 is thereby configured to perform the methods disclosed herein. Storage medium 930 may include persistent storage, which may be, for example, one or a combination of magnetic memory, optical memory, solid-state memory, or remotely mounted memory. Camera device 110 may further include a communication interface 920 configured at least for communicating with other entities, functions, nodes, and devices of system 100. As such, communication interface 920 may include one or more transmitters and receivers, including analog and digital components. Processing circuitry 910 controls the general operation of camera device 110, for example, by sending data and control signals to communication interface 920 and storage medium 930, by receiving data and reports from communication interface 920, and by retrieving data and instructions from storage medium 930. Other components of camera device 110, as well as related functions, are omitted to avoid obscuring the concepts presented herein.

[0056] 10 illustrates an example of a computer program product 1010 including a computer-readable storage medium 1030. The computer-readable storage medium 1030 may have stored thereon a computer program 1020 that can cause the processing circuitry 910, and thus entities and devices operatively connected thereto, such as the communication interface 920 and the storage medium 930, to perform methods according to each of the embodiments described herein. The computer program 1020 and / or the computer program product 1010 may thus provide means for performing any of the steps disclosed herein.

[0057] 10, computer program product 1010 is shown as an optical disc, such as a compact disc (CD) or digital versatile disc (DVD) or Blu-ray disc. Computer program product 1010 may also be embodied as memory, such as random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or electrically erasable programmable read-only memory (EEPROM), or as the non-volatile storage medium of the device, such as external memory, such as a universal serial bus (USB) memory, or flash memory, such as a compact flash memory, among others. Thus, while computer program 1020 is shown here diagrammatically as tracks on the depicted optical disc, computer program 1020 may be stored in any manner suitable for computer program product 1010.

[0058] The inventive concepts have been described above with reference to certain embodiments. However, as will be readily apparent to one skilled in the art, other embodiments different from those disclosed above are equally possible within the scope of the inventive concepts as defined by the claims.

Claims

1. A system (100) for displaying collected sensor data together with an image, A camera device (110) including an imaging unit (120) configured to capture an image depicting a scene (160) and configured to have a variable zoom level, the camera device (110) being configured to communicate with a user interface (130) for displaying the image, A sensor device (170a, 170b) configured to collect sensor data and provide the collected sensor data to the camera device (110) at a variable frequency of occurrence for display on the user interface (130) together with the image, comprising, The sensor device (170a, 170b) is configured such that when the camera device (110) obtains an image captured by the imaging unit (120) at a zoom level exceeding a zoom threshold while depicting the sensor device (170a, 170b), the camera device (110) provides the collected sensor data at a higher frequency of occurrence than when the camera device (110) obtains an image captured at a zoom level below the zoom threshold from the imaging unit (120), System (100).

2. The system (100) according to claim 1, wherein when the camera device (110) obtains an image captured by the imaging unit (120) at a zoom level exceeding the zoom threshold while depicting the sensor device (170a, 170b), the camera device (110) is configured to provide the sensor data as an overlay (172a, 172b) for the image.

3. When the camera device (110) acquires an image taken in a state of depicting the sensor device at a zoom level less than the zoom threshold, the camera device (110) is configured to display, on the user interface (130), an illustration (174) representing the sensor device (170a, 170b) as an overlay (172a, 172b) for the image. The system (100) according to claim 1.

4. The user interface (130) is configured to receive user input, The user input identifies something (174) representing the sensor device (170a, 170b), In response thereto, the camera device (110) is configured to display the sensor data on the user interface (130) as an overlay (172a, 172b) for the image. The system (100) according to claim 3.

5. The user interface (130) is configured to receive user input, The user input identifies something (174) representing the sensor device (170a, 170b), In response thereto, the camera device (110) is configured to cause the imaging unit (120) to zoom in up to the zoom threshold. The system (100) according to claim 3.

6. The sensor device (170a, 170b) has a position in the scene (160), The position of the sensor device (170a, 170b) in the scene (160) has a mapping to a position on the user interface (130) when the sensor device (170a, 170b) is displayed on the user interface (130). The camera device (110) is configured to determine that the user input is identified as the one (174) representing the sensor device (170a, 170b) by confirming that the position of the user input on the user interface (130) corresponds to the position of the sensor device (170a, 170b) on the user interface (130). The system (100) according to claim 1.

7. The camera device (110) of the system (100) according to claim 1 is configured to establish the zoom threshold during a calibration procedure.

8. The imaging unit (120) has a geometric relationship with respect to the sensor device (170a, 170b) established during the calibration procedure. The geometric relationship includes position information that depends on the zoom level of the sensor device (170a, 170b). The position information depending on the zoom level specifies whether the sensor device (170a, 170b) is visible in an image taken using a given zoom level. The system (100) according to claim 7.

9. The variable occurrence frequency can be changed by being selectable between a first predetermined value and a second predetermined value. The first predetermined value produces a higher occurrence frequency than the second predetermined value. The system (100) according to claim 1.

10. The first predetermined value is given by the wake time configuration of the sensor device (170a, 170b), and the second predetermined value is given by the sleep time configuration of the sensor device (170a, 170b). The system (100) according to claim 9.

11. The system (100) includes at least two sensor devices (170a, 170b) each configured to collect sensor data and provide the collected sensor data at respective variable occurrence frequencies for display on the user interface (130) together with the image. Each of the at least two sensor devices (170a, 170b) is associated with its own zoom threshold. The system (100) according to claim 1.

12. The system (100) according to claim 1, wherein the sensor devices (170a, 170b) are Z-wave devices driven by a battery.

13. A camera device (110) for displaying collected sensor data together with an image, an imaging unit (120) configured to capture an image depicting a scene (160) and configured to have a variable zoom level, a communication interface (920) configured to communicate with the user interface (130) for displaying the image and to obtain sensor data collected from the sensor devices (170a, 170b) for display on the user interface (130) together with the image, and a processing circuit (910) configured to configure the sensor devices (170a, 170b) via the communication interface (920) to provide the collected sensor data at a higher occurrence frequency when the imaging unit (120) captures an image depicting the sensor devices (170a, 170b) at a zoom level exceeding the zoom threshold than when the imaging unit (120) captures an image at a zoom level below the zoom threshold. and a camera device (110) including the same.

14. A method for displaying collected sensor data together with an image, the method being executed by a camera device (110) including an imaging unit (120), a communication interface (920), and a processing circuit (910), the imaging unit (120) being configured to have a variable zoom level, the method comprising: capturing (S102), by the imaging unit (120), an image depicting a scene (160); communicating (S104), by the communication interface (920), with a user interface (130) to display the image and acquiring, from sensor devices (170a, 170b), sensor data collected by the sensor devices (170a, 170b) for display on the user interface (130) together with the image; configuring (S106), by the processing circuit (910) via the communication interface (920), the sensor devices (170a, 170b) such that the collected sensor data is provided at a higher frequency when the imaging unit (120) captures an image depicting the sensor devices (170a, 170b) at a zoom level exceeding a zoom threshold than when the imaging unit (120) captures an image at a zoom level below the zoom threshold; A method comprising the above steps.

15. A non-transitory computer-readable storage medium storing a computer program (1020) for displaying collected sensor data together with an image, the computer program including computer code that, when executed on a camera device (110), causes the camera device (110) to perform the method according to claim 14.

Images