Portable display device with overlaid virtual information

Abstract

A method for locating a virtual object (10, 12, 14) in an image (20) of a scene including a real object (30) is disclosed. The method includes receiving (620) a signal from a short-range wireless transmitter (40) arranged at a predetermined position relative to the real object and determining (640) a position and orientation of a portable device relative to the short-range wireless transmitter based on the received signal. An image of the scene is displayed (650) with the virtual object, wherein the position of the virtual object in the image is based on the determined position and orientation of the portable device relative to the short-range wireless transmitter and on the predetermined position of the short-range wireless transmitter relative to the real object. A system (500), a portable device (100) and a computer program product (710) are also disclosed.

Description

Technical Field

[0001] The present invention relates generally to the field of inspection and maintenance of equipment of power transmission and / or distribution networks and, more particularly, to a method and a device for locating a virtual object in an image of a scene comprising such equipment. Background Art

[0002] A technician or field operator (e.g., at a substation of an electric power transmission and / or distribution network) needs to interact with systems or devices in installed equipment and be able to obtain information about the installed equipment. The information may, for example, relate to the identity of various components and parts of the installed equipment, thereby directing the technician or field operator to the correct part of the equipment. The information may also include the characteristics or status of various components and parts, for example, including maintenance history and previous repair reports.

[0003] Augmented reality is an example of a technology that has been used to try to assist technicians or field operators in obtaining information about installed equipment. Augmented reality is a method of overlaying a real-world representation with computer-generated graphics (also called virtual objects) that include additional information of the real objects in the image. Thus, with augmented reality, the user gains a perception of the real-world environment that is improved by the presence of the added virtual objects.

[0004] Augmented reality technology is now being implemented in handheld display devices, including, for example, tablet computers and smartphones, which include cameras for capturing images of the environment, which can be reproduced on a display or touch screen of the handheld display device along with additional information in the form of computer-generated graphical features.

[0005] However, the aforementioned augmented reality techniques are known to suffer from problems associated with poor accuracy in positioning of virtual objects relative to real objects in a real-world representation. As a result, these techniques provide little or no help to technicians or field operators working in environments requiring high precision and positioning accuracy. Summary of the invention

[0006] It would be advantageous to achieve a technique that overcomes or at least mitigates the above mentioned disadvantages.In particular, it would be desirable to be able to improve the positioning of virtual objects in an image of a scene comprising real objects.

[0007] In order to better solve one or more of these problems, methods, systems, devices and computer program products are provided having the features defined in the independent claims. Preferred embodiments are defined in the dependent claims.

[0008] Therefore, according to a first aspect, a method for locating a virtual object in an image of a scene including a real object is provided, wherein the real object may form at least part of an equipment of a power transmission network and / or a power distribution network. The method comprises acquiring an image of the scene by a portable device, and receiving a signal at the portable device from a short-range wireless transmitter arranged at a predetermined position relative to the object. The method also comprises determining the position and orientation of the portable device relative to the short-range wireless transmitter based at least in part on the received signal. The method may also comprise: displaying the acquired image and the virtual object on the portable device; and locating the virtual object in the acquired image based at least in part on the determined position and orientation of the portable device relative to the short-range wireless transmitter and based on the predetermined position of the short-range wireless transmitter relative to the real object.

[0009] According to a second aspect, a system for locating a virtual object in an image of a scene including a real object is provided. The system may include a portable device for inspection and maintenance of equipment of a power transmission network and / or a power distribution network, a short-range wireless transmitter arranged at a predetermined position relative to the real object, and a processing unit. The portable device may be configured to acquire an image of the scene and receive a signal from the short-range wireless transmitter. The processing unit may be configured to determine the position and orientation of the portable device relative to the short-range wireless transmitter based at least in part on the received signal. Further, the portable device may be configured to display the acquired image and the virtual object, and locate the virtual object in the acquired image, based at least in part on the determined position and orientation of the portable device relative to the short-range wireless transmitter. The position of the virtual object in the acquired image may be based on the determined position and orientation of the portable device relative to the short-range wireless transmitter and based on the predetermined position of the short-range wireless transmitter relative to the real object.

[0010] According to a third aspect, a portable device for inspection and maintenance of equipment of a power transmission network and / or a power distribution network is provided. The portable device includes an imaging device configured to acquire an image of a scene including a real object, and a display device configured to display the acquired image. The portable device can be configured to receive a signal from a short-range wireless transmitter arranged at a predetermined position relative to the real object, and be communicatively connected to a processing unit, which is configured to determine the position and orientation of the portable device relative to the short-range wireless transmitter based at least in part on the received signal. The display device can be configured to display the acquired image with a virtual object, and locate the virtual object in the acquired image based at least in part on the determined position and orientation of the portable device relative to the short-range wireless transmitter and based on the predetermined position of the short-range wireless transmitter relative to the real object.

[0011] According to a fourth aspect, there is provided a computer program product comprising instructions for causing a system according to the second aspect to perform the steps of the method according to the first aspect.

[0012] The present invention utilizes signals from a short-range wireless transmitter to determine the position and orientation of a portable device relative to a short-range wireless transmitter. This allows virtual objects to be more accurately positioned in the acquired image of a real object, especially in view of the technology that relies only on the acceleration and gyroscope data from the inertial measurement unit (IMU) of the portable device, and the technology that relies only on the received signal strength indicator (RSSI) data obtained from the signal. Although the technology based on IMU data is prone to cumulative errors or drift errors, resulting in increased offset and poor repeatability, the present invention provides a more reliable and more accurate positioning based on the predetermined position of the transmitter relative to the real object. By using the signal from the transmitter, the position and orientation of the portable device can be repeatedly (or even continuously) calculated when the portable device moves relative to the transmitter. Since the transmitter is arranged at a predetermined position relative to the real object, the position of the portable device can also be established relative to the known position of the transmitter when the portable device is being moved. This can alleviate the problem of cumulative errors associated with positioning based on data from an IMU inside a portable (and possibly mobile) device. Further, while techniques that rely on RSSI data can indicate the distance between a transmitter and a receiver of a portable device, such techniques are generally not particularly accurate when it comes to determining the direction in which the signal is being transmitted. Therefore, an advantage of the present invention is that both distance and direction can be obtained from the signal of the transmitter. This allows for a more accurate determination of the absolute position and also the orientation of the portable device relative to the transmitter (and therefore the real object).

[0013] Orientation is generally understood as the angular position, attitude or direction of a portable device, which together with its position describes how the portable device is placed in space. Thus, the orientation can be changed by rotating the portable device around its own axis, while the position can be changed by translating the portable device. In the context of the present disclosure, the position of a portable device can be described as the geospatial location where it is placed, while the orientation can be described by the direction in which the optical axis of its imaging device is pointing.

[0014] Assuming that the position of the display device relative to the portable device is known, the determined position and orientation of the portable device (and therefore the display device) relative to the real object can be used to determine the position of the real object in the image displayed on the display device. The position of the real object in the image can then be used to correctly position the virtual object in the image.

[0015] A short-range wireless transmitter (also referred to hereinafter as a "transmitter" for brevity) generally refers to a radio frequency transmitter device utilizing technologies such as Bluetooth, Wi-Fi, near field communication, ultra-wideband, and IEE 802.15.4. A short-range transmitter is typically a low-power transmitter limited to 25 to 100 mW effective radiated power, thereby limiting its useful range to a few hundred meters. Preferably, the transmitter is capable of transmitting a signal from which the direction of propagation of the signal relative to a receiver of a portable device can be determined, as described in further detail below.

[0016] A virtual object generally refers to a graphical object or feature that is displayed on a display device and is at least partially generated by a computer. In other words, a virtual object can be considered a graphical feature that includes additional information and that can overlay an image of the real world. Examples of virtual objects can include labels, logos, text boxes, symbols, numbers, letters, etc.

[0017] Therefore, according to an embodiment, a virtual object may be a graphical feature associated with the identity of a real object. A virtual object may, for example, assist a user (such as a field operator or technician) in identifying various components or parts of a real object. Additionally or alternatively, a graphical feature may be associated with a feature of a real object. The feature may, for example, relate to the state or function of a component or part of a real object, such as an applied voltage or a functional connection to other components or devices of the installed equipment, or a maintenance history of a component. In another example, a virtual object may include instructions that guide a user through a process (such as an inspection or maintenance process) at the equipment.

[0018] In another example, a virtual object may be associated with a property or category to which a real object belongs. Although the identity of a real object may be unique to each real object, multiple individual real objects may be grouped according to common characteristics based on their properties or categories.

[0019] According to an embodiment, the identity and / or characteristics may be retrieved from the real object. Thus, the object or a device associated with the object may be configured to transmit data about the identity or characteristics to the portable device or a receiver associated with the portable device. In an example, the identity or characteristics may be transmitted by a short-range wireless transmitter, preferably via the same signal transmission used to determine the position and orientation of the portable device relative to the short-range wireless transmitter.

[0020] According to an embodiment, the identity and / or characteristics may be retrieved from a remote server, i.e., a server communicatively connected to the portable device and arranged at a physical location separate from the portable device. The server may be configured to house or access a database storing information associated with one or more devices or components of the equipment and to provide the information to be displayed as virtual information in the acquired image.

[0021] The information from the server can be associated with an identity or category of a real object and provided to the portable device when requesting information associated with the identity. The identity can be obtained, for example, as input from a user, which can indicate a desired identity or category for which the user wishes to retrieve information from the server. In another example, the identity or category can be obtained from the real object (e.g., via a signal as discussed above) and included in the request to the server. In another example, the identity or category of the real object can be obtained by a computer-assisted image analysis method.

[0022] Thus, according to an embodiment, the identity or category of the real object may be determined. The determination of the identity may include processing of the acquired image, for example by computer-assisted analysis, in which features or objects in the acquired image may be compared with features of known objects and features in previously captured images. This processing may also be referred to as an object recognition method. The identity or category of the real object may be determined automatically, i.e. without involving a user, or presented as a suggestion to be confirmed by the user.

[0023] The identification of real objects in the image can also be used to further refine or verify the position of the real objects in the image. Thus, the positioning of the virtual object in the acquired image can be further based on the output of the object recognition method. Thus, the processing for determining the identity or category can also allow for more accurate positioning of the virtual object in the acquired image.

[0024] According to an embodiment, the identity (or category) can be determined using an artificial intelligence (AI) algorithm that is trained on previously acquired images of real objects and the corresponding identities (or categories) of these objects. Therefore, the present embodiment uses an artificial intelligence algorithm (which may also be referred to as a neural network technique or machine learning) as a tool to improve the processing of acquired images to obtain information about real objects. The artificial intelligence algorithm can be trained on previously captured images in which real objects have been manually identified by humans (such as operators of portable devices).

[0025] According to an embodiment, the artificial intelligence algorithm can be updated by communicating with a server configured to receive images and information about the identity (or category) of real objects identified in the acquired images. This information can be used to further improve and train the artificial intelligence algorithm and improve the accuracy of the recognition of real objects.

[0026] The server can house the artificial intelligence algorithm. Additionally or alternatively, the artificial intelligence algorithm can be stored locally in the portable device. It should be understood that in some examples, the artificial intelligence algorithm can be downloaded to the portable device from a remote location (e.g., the server mentioned above or a remote server of an Internet cloud infrastructure). Further, the artificial intelligence algorithm stored on the portable device can be replaced with a newer version, which in some examples can be improved by input received from one or more portable devices. Therefore, data generated by the use of the portable device can be provided as input for training the artificial intelligence algorithm and ultimately improving the method and device.

[0027] Examples of artificial intelligence algorithms include probabilistic graphical models, such as Bayesian networks, random forests, probabilistic classifiers and / or controllers (e.g., Gaussian mixture models). Additional examples include neural networks, such as recurrent networks, and reinforcement learning gated recurrent units, long short-term memory architectures, and attention networks.

[0028] According to an embodiment, the position of the portable device relative to the short-range wireless transmitter can also be determined based on the signal strength of the short-range wireless transmitter. The measurement of the strength or density of the received signal can be combined with the propagation model to determine the distance between the short-range wireless transmitter and the receiver of the portable device. The determined difference can then be used with the direction information associated with the signal to determine the position and orientation of the portable device.

[0029] According to an embodiment, the position of the portable device relative to the short-range wireless transmitter may also be determined based on signals received from multiple short-range wireless transmitters. In an embodiment, this determination may include triangulation of the signals. Thus, in the case where several short-range wireless transmitters are available and arranged at predetermined positions relative to the real object, the determination of the position and orientation of the portable object may be further improved by including the signals from these transmitters in the calculation.

[0030] Determining the direction of the portable device relative to the short-range wireless transmitter may include determining the direction of propagation of a signal transmitted by the short-range wireless transmitter and incident on an antenna of the portable device. This may be accomplished, for example, by determining the maximum signal strength during rotation of the antenna, or by measuring the time differences of arrival between various elements of an antenna array of the portable device.

[0031] According to an embodiment, the portable device may include an array of antenna elements that allow the angle of arrival (AoA) of the received signal to be determined. The AoA may be determined by measuring the difference in phase of the signals received at the various antenna elements in the array, wherein the determined AoA indicates the orientation of the portable device relative to the short-range wireless transmitter. Preferably, the AoA determination may be combined with a distance determination based on the received signal strength to obtain a measure of both the orientation and position of the portable device relative to the short-range wireless transmitter.

[0032] An advantage of this embodiment is that it allows the orientation and position of a portable device to be determined using signals from a single antenna (in this case a transmitter). Several transmitters can indeed be used to verify the determined orientation and position, but unlike, for example, triangulation-based techniques, this is not necessary in the context of the present disclosure.

[0033] According to an embodiment, the short-range wireless transmitter may include an array of antenna elements that allow the angle of departure (AoD) of the received signal to be determined. The AoD can be determined by measuring the phase difference in a manner similar to that described above for the AoD calculation. Thus, the receiving antenna of the portable device can collect samples of the phase of the signal transmitted by multiple antenna elements in the array. Based on the phase sample data, the portable device can determine the relative signal direction, and therefore determine the relative orientation of the portable device and the transmitter. Preferably, these calculations are performed using the calculation of the distance based on the received signal strength data of the signal, so that the orientation and position of the portable device relative to the transmitter can be determined.

[0034] Similar to the embodiments related to AoA calculation, this embodiment allows the relative position and orientation of a portable device to be determined using a single antenna (in this case a portable device). It is possible to use more than one antenna, but this is not necessary in the context of the present disclosure.

[0035] According to some embodiments, the portable device may include a positioning device, such as an internal positioning device, configured to provide data to the processing unit. The data may be used by the processing unit to determine the position and / or orientation of the portable device relative to the short-range wireless transmitter. The positioning device may, for example, include a short-range wireless receiver configured to collect data from the short-range wireless transmitter, or a global navigation satellite system GNSS receiver that allows the geospatial position of the portable device to be determined in a global coordinate system. Alternatively or additionally, the positioning device includes an inertial measurement unit IMU, or an inertial sensor, for providing acceleration and gyroscope data. When determining the direction and position of the portable device relative to the short-range wireless transmitter, IMU data may be provided to further improve accuracy.

[0036] According to an embodiment, a real object may be or form part of the equipment of a substation. A substation may act as a connection point between different transmission lines of an electrical system, and may additionally convert voltage from high to low, or vice versa, or perform any of several other important functions in a transmission network or a distribution network. Substations and the equipment installed in various substations are usually designed differently. The advantage of this embodiment is that they allow one or several short-range wireless transmitters to be associated with components and equipment of a substation, for assisting a field operator or technician in identifying the correct components and equipment, and guiding him or her to approach, inspect, and perform actions on the equipment. Examples of components or assets may include connectors and cables for their application, where virtual objects may indicate signals transmitted through cables, the status of cable leads, and data values ​​associated with cables. Further, virtual objects may indicate identification of cable types and numbering and routing of cables. Virtual objects may also identify several components or assets of the same type, such as multiple fans, and visualize the condition or status of each of the assets (e.g., indicating whether the fan is rotating).

[0037] According to some embodiments, the portable device may be a handheld display device, such as a tablet computer or a smartphone, or a head mounted display device.

[0038] It should be understood that other embodiments than those described above are also possible. It should also be understood that any of the features in the embodiments described above for the method according to the first aspect can be combined with systems, devices and products according to other aspects, and vice versa.

[0039] Other objects, features and advantages of the attached embodiments will become apparent from the following detailed description, from the attached dependent claims as well as from the drawings. Those skilled in the art realize that different features of the invention may be combined in embodiments other than those described below, even if recited in different claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Exemplary embodiments will now be described in more detail with reference to the following drawings, in which:

[0041] Figure 1 A schematic diagram showing a portable device for inspection and maintenance of equipment of a power transmission network and / or a power distribution network according to some embodiments;

[0042] Figure 2 shows an example of an acquired image displayed with a virtual object on a display device of a portable device according to some embodiments;

[0043] Figure 3 shows the capture of an image of a scene including a real object according to some embodiments;

[0044] Figure 4a and Figure 4b shows examples of transmitters and receivers for determining the position and orientation of a portable device relative to a real object in accordance with some embodiments;

[0045] Figure 5 shows a schematic diagram of a system according to some embodiments;

[0046] Figure 6 is a flow chart illustrating the method of the present disclosure; and

[0047] Figure 7 An example of a computer program product including a computer-readable storage medium according to an embodiment is shown.

[0048] As shown in the figures, the sizes of elements and regions may be exaggerated for illustrative purposes, and thus, the sizes of these elements and regions are provided to illustrate the overall structure of the embodiments.Throughout the text, the same reference numerals refer to the same elements. DETAILED DESCRIPTION

[0049] Exemplary embodiments will now be described more fully hereinafter with reference to the accompanying drawings, in which currently preferred embodiments are shown. However, the present invention may be implemented in many different forms and should not be construed as limited to the embodiments set forth herein; on the contrary, these embodiments are provided for thoroughness and completeness and to fully convey the scope of the invention to the skilled person.

[0050] refer to Figure 1 , describes a portable device 100 according to some embodiments of the present disclosure. Figure 1A portable device 100, such as a handheld tablet computer, is shown, which can be used for inspection or maintenance of equipment 30 (such as primary equipment or auxiliary equipment) of a power transmission network or distribution network. The portable device 100 may include an imaging device 110, a positioning device or receiver 120, a processing unit 130, a display device 140 and a transceiver 150.

[0051] The display device 140 may be adapted to display one or more images captured by the imaging device 110 and virtual objects for assisting the user in obtaining information about the real object 30, as will be described with reference to FIG. Figure 2 The display device 140 may also assist the user in capturing an image of a scene including the real object 30 within the field of view of the imaging device 110 .

[0052] The virtual object may include information associated with a real object 30, which may be, for example, an equipment device or asset of a substation. In one example, the real object 30 may be a current transformer or a current sensor. The virtual object may indicate to the user the identified transformer or sensor and the output signal type and data value.

[0053] The portable device 100 may be provided with an antenna element for receiving signals from a short-range wireless transmitter 40 arranged at a predetermined position relative to the real object 30. In some examples, the antenna may form part of the positioning device or receiver 120. Therefore, the positioning device or receiver 120 may have one or several antenna elements (not shown) for receiving one or several signals from one or several short-range wireless transmitters 40. In the event that multiple signals are available (e.g., from a transmitter 40 associated with the real object 30 and from additional transmitters associated with the same object 30 or other objects), the multiple signals may be subjected to triangulation to further determine the position and / or orientation of the portable device relative to the real object 30.

[0054] In an example, the positioning device 120 may be configured to receive radio frequency signals, such as Bluetooth signals. Data obtained from the received signals may then be provided to the processing unit 130 for determining the position of the portable device 100 relative to the real object 30. Optionally, the positioning device may include or be configured to receive data from an inertial measurement unit IMU 170, which may provide acceleration and gyroscope data related to the movement and position of the portable device 100.

[0055] In the present embodiment, the processing unit 130 is arranged in the portable device 100. However, it should be understood that the processing unit 130 may also be arranged outside the portable device, such as in an Internet cloud structure. The processing unit 130 may communicate with the positioning device 120, the display unit 140, and the optional IMU 170. In some examples, the processing unit 130 may also be configured to receive signals from a global navigation satellite system GNSS.

[0056] The processing unit 130 may be adapted to control the imaging device 110 so as to capture one or more images of a scene including the real object 30. The processing unit 130 may also be adapted to receive images captured by the imaging device 110, or at least receive data corresponding to the images, and further control the images displayed by the display device 140.

[0057] As will be described below, the position of a virtual object in an image displayed by display device 140 may be determined based on data collected by positioning device 120 .

[0058] The imaging device 110 of the portable device 100 can be, for example, a digital camera including an image sensor such as a semiconductor charge-coupled device (CCD), a complementary metal-oxide-semiconductor (CMOS) sensor, or another active digital pixel sensor. The imaging device 110 is arranged at a specific position relative to the antenna of the positioning device 120.

[0059] The signal incident on the antenna of the positioning device 120 can be used to determine the distance between the antenna and the short-range wireless transmitter 40 (referred to as the transmitter) that sent the signal. The distance can be calculated, for example, based on the strength of the received signal combined with a path loss model for the signal. The signal incident on the antenna of the positioning device 120 can also be used to determine the direction of the received signal, which can be defined by the angle of arrival or the angle of departure of the signal. The direction can be calculated, for example, based on a phase measurement of the incident signal, such as referring to Figure 4a and Figure 4b As further described.

[0060] The antenna of the positioning device 120 can have a specific position and orientation relative to the portable device 100, so that the direction of the incident signal from the transmitter 40 can be defined relative to the position and orientation of the portable device 100. In this example, the direction of the incident signal is illustrated by the angle α between the axis 122 parallel to the optical axis 114 of the imaging device 110 and the line of sight between the antenna of the positioning device 120 and the transmitter 40.

[0061] Based on the direction of the incident signal and the distance to the transmitter 40, and because the position of the antenna of the positioning device 120 in the portable device 100 is known, the position and orientation of the portable device 100 can be estimated relative to the transmitter 40. The position and orientation of the portable device 100 relative to the real object 30 can then be calculated based on the predetermined position of the transmitter 40 relative to the real object 30.

[0062] Figure 1 A remote device is also shown, such as a server 160 in communication with the portable device 100. The communication may be achieved, for example, by the transceiver 150 of the portable device 100. The server may be configured to receive an image and information about the identity or category of a real object 30 in the image acquired by the imaging device 110. The server 160 may store information associated with the real object 30, such as the identity, state, or characteristics of the object. In an example, the identity of the real object 30 may be provided to the server 160, which may respond by providing information associated with the identity of the real object 30 to the portable device 100. The information associated with the identity may then be displayed as a virtual object in the acquired image.

[0063] Further, the server may house an artificial intelligence algorithm that may be used to identify real objects 30 in images captured by imaging device 110. This identification may be used to further improve the positioning of virtual objects in the image, and to retrieve information associated with the identity of real objects 30. In some examples, the artificial intelligence algorithm model may be downloaded from server 160 and stored locally on portable device 100, where in other examples, the artificial intelligence algorithm may be operated from server 190.

[0064] If the user finds that the identification of the real object 30 is incorrect, the user can correct the information and the artificial intelligence algorithm is updated accordingly.

[0065] Although the server 160 is shown as being at a remote location, such as a remote server of an Internet cloud infrastructure, the server 160 may be an integral part of the portable device 100 .

[0066] Figure 2 Shown according to Figure 1The display device 140 of the portable device 100 of the illustrated embodiment. In this example, an image 20 of a scene including a real object in the form of equipment device 30 is displayed on the display device 140 together with information in the form of virtual objects 10, 12, 14. The virtual objects 10, 12, 14 may be computer-generated graphical features, such as symbols and text fields - in this example illustrated by the warning symbol 10, arrow 12, and indicated voltage level 14. The virtual objects 10, 12, 14 are overlaid on the image 20 and positioned in such a way that they convey information to the user about a particular part or component 35 of the equipment device 30 shown. The position of the virtual objects 10, 12, 14 in the image is determined based on the position and orientation of the portable device 100 relative to the real object 30, and allows the user to gain knowledge about, for example, the identity, status, and characteristics of a particular part or component 35 of the equipment 30 being imaged.

[0067] In an embodiment, the portable device includes an input device for receiving input from a user. The input device may, for example, be configured to allow a user to select a specific part or component or a virtual object in an image to retrieve additional information associated with the selected item. The input device may, for example, include a touch screen 140 or a graphical navigation tool, such as a user-controlled pointer. In an example, the input is sent as a request to a server 160, which may respond by providing stored information associated with the selected item. The stored information may be presented on the display device 140 as a virtual object in the image.

[0068] Figure 3 1 shows a scenario where a portable device 100 is placed in a position where an imaging device 110 views a real object 30, while data from a signal from a short-range wireless transmitter 40 is collected together with data from the imaging device 110. The imaging device 110 may capture an image (or sequence of images, or video) of a scene or surface where the real object 30 is located, as represented by an area 116 of the field of view of the sensor 112 of the imaging device 110. The location where the image is captured may correspond to the location of a projection center 117 of the imaging device 110 (the lens is not in the Figure 3 It should be appreciated that another point of the portable device 110 may be used as a reference, such as the center of the imaging sensor 112.

[0069] The portable device 100, or more precisely the processing unit 130, can determine the position of the projection center 117 relative to the real object 30 according to the coordinate system X, Y, Z based on the signal from the emitter 40 and the predetermined position of the emitter 40 relative to the real object 30, as discussed above in conjunction with the previous figures. Therefore, the processing unit 130 can further determine the relative orientation of the portable device 100 based on the same signal. The orientation can be defined, for example, by the angle α between the optical axis 114 of the imaging device 110 and the incident light ray 32 propagating from the real object 30.

[0070] Now refer to Figure 4a and Figure 4b Examples of transmitting and receiving signals from a short-range wireless transmitter in accordance with some embodiments are discussed.

[0071] Figure 4a and Figure 4b A short-range wireless transmitter 40 and a receiver or positioning device 120 according to an embodiment are shown, wherein the positioning device 120 is configured to receive a signal transmitted by the short-range wireless transmitter 40. The short-range wireless transmitter 40 and the positioning device may, for example, be configured to support Bluetooth 5.1 direction finding functionality. As discussed above in conjunction with the previous embodiments, the transmitter 40 may be arranged at a predetermined position relative to the real object 30, and the positioning device 120 may form part of the portable device 100, or communicate with the portable device 100, for inspection and maintenance of equipment including the real object 30.

[0072] Figure 4a A transmitter 40 including a single antenna 42 for transmitting radio frequency signals and a positioning device or receiver 120 including an array 124 of antenna elements for determining the angle of arrival (AoA) α of the signal are shown. By using multiple antenna elements 124, phase differences of the incident signals can be observed and used to determine the AoA, and therefore the orientation of the portable device 100 relative to the real object 30. In this example, the angle of arrival α is indicated as a plane defined by the array of antenna elements 124 (defined by Figure 4a The angle α between the line 122 in FIG. 1 and the line of sight between the array of antenna elements 124 and the antenna 42 of the transmitter 40 is indicated.

[0073] Figure 4b An alternative configuration is shown in FIG, in which the transmitter 40 includes an array of antenna elements 42 and the positioning device or receiver 120 includes a single antenna 124. Figure 4a In the configuration, the direction of the transmitted signal can be determined by observing the phase. Figure 4b, the direction can be defined as an angle of departure (AoD) β and can be described by the angle β between the plane defined by the array of antenna elements 42 of the transmitter 40 (indicated by line 44 in this figure) and the line of sight between the array of antenna elements 42 and the antenna 124 of the positioning device 120.

[0074] Figure 5 is a schematic diagram of a system 500 according to some embodiments. The system 500 may include a portable device 100, a plurality of short-range wireless transmitters 40 associated with a real object 30, a processing unit 130, and a server 50, which may be configured similarly to the embodiments described with reference to the previous figures.

[0075] The portable device 100 may be a handheld device 101 such as a tablet or a smartphone or a head mounted device 102 such as an augmented reality headset. Two examples are indicated in this figure, showing a user 60 holding the portable device in his hand or wearing it as a headset.

[0076] The user 60 may be, for example, a field operator or service technician approaching and / or performing actions on installed equipment (such as equipment 30 of a substation). In order to assist the user 60 in obtaining information about components of the equipment, one or more transmitters 40 may be provided at the equipment to facilitate display of the information in an image captured by the portable device 100. Thus, when the user 60 directs the portable device toward the equipment 30 so that an object of interest (i.e., the equipment 30 or a part of the equipment) is viewed by the imaging device 110 of the portable device 100, a signal from the transmitter 40 is received by the positioning device and data is provided to the processing unit 130 for determining the orientation and position of the portable device 100 relative to the equipment 30. Further, components or equipment of the equipment may be identified and information associated with the components or equipment may be requested from the remote server 50. The information may then be presented to the user as a virtual object in the captured image displayed on the portable device 100.

[0077] Figure 66 is a flow chart illustrating a method for locating a virtual object in an image according to one embodiment, which may be similar to the embodiments described with reference to the previous figures. The method may include: acquiring 610 an image 20 of a scene by a portable device 100, wherein the scene includes a real object 30; and receiving 620 a signal at the portable device 100 from a short-range wireless transmitter 40 arranged at a predetermined position relative to the real object 30. In some embodiments, the method may optionally include a step of retrieving 630 an identity or characteristic of the real object 30 from the real object 30 itself, preferably via the received signal. Alternatively or additionally, the identity may be determined by processing 635 the acquired image 20, preferably using an artificial intelligence algorithm, which may be updated 637 or trained based on the image and information about the identity of the object determined in the image.

[0078] Further, the method may include: determining 640 a position and orientation of the portable device 100 relative to the short-range wireless transmitter 40 based on the received signal; displaying 650 the acquired image 20 and the virtual object 10 on the portable device 100; and positioning 660 the virtual object 10 in the acquired image 20 based on the determined position and orientation of the portable device relative to the short-range wireless transmitter 40 and based on a predetermined position of the short-range wireless transmitter relative to the real object 30.

[0079] Figure 7 An example of a computer program product 710 including a computer-readable storage medium 730 is shown. On this computer-readable storage medium 730, a computer program 720 may be stored, which may cause the system 500 or the portable device 100 and entities and devices operatively coupled thereto (such as the server 50) to perform methods according to the embodiments described herein. Thus, the computer program 720 and / or the computer program product 510 may provide means for performing the actions disclosed herein.

[0080] exist Figure 7In the example of , the computer program product 710 is shown as an optical disc, such as a compact disc (CD) or a digital versatile disc (DVD) or a Blu-ray disc. The computer program product may also be implemented as a memory, such as a random-access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or an electrically erasable programmable read-only memory (EEPROM), more particularly as a non-volatile storage medium of a device in an external memory, such as a Universal Serial Bus (USB) memory or a flash memory, such as a compact flash. Thus, although the computer program 720 is schematically shown here as a track on the depicted optical disc, the computer program 720 may be stored in any manner suitable for the computer program product 710.

[0081] The inventive concept has mainly been described above with reference to a few embodiments. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the inventive concept as defined by the appended patent claims.

Claims

1. A method for locating a virtual object (10, 12, 14) in an image (20) of a scene comprising a real object (30), the real object forming at least part of an equipment of a power transmission network and / or a power distribution network, the method comprising: acquiring (610) an image of the scene via the portable device (100); and receiving (620) a signal at the portable device from a short-range wireless transmitter (40) arranged at a predetermined position relative to the real object; determining (640) a position and orientation of the portable device relative to the short-range wireless transmitter based at least in part on the received signal; displaying (650) the acquired image and the virtual object on the portable device; as well as positioning (660) the virtual object in the acquired image based at least in part on the determined position and orientation of the portable device relative to the short-range wireless transmitter and based on a predetermined position of the short-range wireless transmitter relative to the real object, in: determining the angle of arrival AoA (α) of the received signal by using an array of antenna elements (124) of the portable device; and / or determining an angle of departure AoD(β) of the received signal using an array of antenna elements (42) of the short-range wireless transmitter; and Further based on the AoA and / or the AoD, a position and orientation of the portable device relative to the short-range wireless transmitter is determined. 2 . The method of claim 1 , wherein the virtual object is a graphical feature associated with the identity and / or characteristics of the real object.

3. The method according to claim 2, further comprising retrieving (630) the identity and / or characteristics of the real object.

4. The method of claim 3, further comprising determining (635) the identity of the real object by an object recognition method.

5. The method of claim 4, wherein the object recognition method comprises an artificial intelligence algorithm trained based on previously acquired images of real objects and the corresponding identities of these objects.

6. The method of claim 5, further comprising updating (637) the artificial intelligence algorithm by communicating with a server configured to receive images and information about the identities of objects identified in the received images.

7. The method of claim 4, wherein the positioning of the virtual object in the acquired image is also based on an output of the object recognition method.

8. The method of any one of claims 1-7, wherein the position of the portable device relative to the short-range wireless transmitter is also determined based on a signal strength of the short-range wireless transmitter.

9. The method of any one of claims 1-7, wherein the position of the portable device relative to the short-range wireless transmitter is also determined based on triangulation.

10. The method according to any one of claims 3-7, wherein the identity and / or characteristics of the real object are retrieved from the received signal.

11. A system (500) for locating a virtual object (10, 12, 14) in an image (20) of a scene including a real object (30), comprising: A portable device (100) for inspection and repair of equipment of a power transmission network and / or a power distribution network; a short-range wireless transmitter (40) arranged at a predetermined position relative to the real object; as well as A processing unit (130); The portable device is configured to: acquiring an image of the scene, and receiving a signal from the short-range wireless transmitter; wherein the processing unit is configured to determine a position and orientation of the portable device relative to the short-range wireless transmitter based at least in part on the received signal; and The portable device is further configured to: displaying the acquired image and the virtual object, and positioning the virtual object in the acquired image based at least in part on the determined position and orientation of the portable device relative to the short-range wireless transmitter and based on a predetermined position of the short-range wireless transmitter relative to the real object, in: The portable device comprises an array of antenna elements (124) allowing the angle of arrival AOA (α) of the received signal to be determined, and / or The short-range wireless transmitter comprises an array of antenna elements (42) allowing the angle of departure AoD(β) of the received signal to be determined; as well as The processing unit is configured to determine a position and an orientation of the portable device relative to the short-range wireless transmitter further based on the AoA and / or the AoD.

12. A system according to claim 11, wherein the portable device includes a positioning device (120), and wherein the processing unit is further configured to determine the position and / or orientation of the portable device relative to the short-range wireless transmitter further based on data from the positioning device, wherein the positioning device includes a global navigation satellite system GNSS, a receiver and / or an inertial sensor.

13. A portable device (100) for inspection and maintenance of equipment of a power transmission network and / or a power distribution network; comprising: an imaging device (110) configured to acquire an image of a scene including a real object (30); as well as A display device (140) is configured to display the acquired image; wherein: The portable device is configured to receive a signal from a short-range wireless transmitter (40) arranged at a predetermined position relative to the real object; The portable device is communicatively connected to a processing unit (130), the processing unit (130) being configured to determine a position and orientation of the portable device relative to the short-range wireless transmitter based at least in part on the received signal; and The display device (140) is further configured to display a virtual object in the acquired image, wherein a position of the virtual object in the acquired image is based at least in part on the determined position and orientation of the portable device relative to the short-range wireless transmitter and on a predetermined position of the short-range wireless transmitter relative to the real object, in: The portable device comprises an array of antenna elements (124) allowing the angle of arrival AOA (α) of the received signal to be determined, and / or The portable device is configured to determine an angle of departure AoD(β) of a received signal from an array of antenna elements (42) of the short-range wireless transmitter; and The processing unit is configured to determine a position and an orientation of the portable device relative to the short-range wireless transmitter based on the AoA and / or the AoD.

14. The portable device according to claim 13, wherein the portable device is a handheld display device (101) or a head mounted display device (102).

15. A computer program product (710) comprising instructions for causing the system (500) according to any one of claims 11 to 12 to perform the steps of the method according to claims 1 to 10.

Images