Enhanced Virtual Tour

Abstract

The present invention relates to a computer-implemented method including obtaining previously captured virtual tour data (610) of a location, receiving real-time image data (607) of a target object located at the location, overlaying the real-time image data (607) over an area of ​​a virtual tour view (610) to form mixed image virtual tour data, receiving real-time acoustic data (600) captured by an audio sensor within the location, and playing the real-time acoustic data (600) when a user traveling through the mixed image virtual tour is within a certain predefined distance of the audio sensor. The present invention further relates to a system for performing the method, and a computer program product including instructions for performing the method.

Description

[Technical Field]

[0001] FIELD OF THE INVENTION The present invention relates to a computer-implemented method for extending a virtual tour.

[0002] The invention also relates to a system and a computer program product for causing an apparatus to perform the method. [Background technology]

[0003] Background of the Invention In many environments, for example, industrial environments and production plants, there are situations where monitoring of a process or environment is necessary, for example to identify changes or developing faults. This is predictive maintenance. Monitoring can be used to schedule maintenance or take other measures to prevent consequential damage and avoid its consequences. However, industry is not the only environment that can benefit from monitoring; monitoring the condition or status of a building also increases the owner's peace of mind.

[0004] For example, in an industrial environment, there are multiple different types of elements, features, or factors that need to be monitored. For example, in the pulp and paper manufacturing process in the paper industry, the process needs to be monitored to detect possible failures. In many cases, monitoring the situation by looking at measurements, monitoring image data generated by cameras from various locations on the production floor on a screen, or relying on the analysis results of image processing devices is not always sufficient; information can be obtained more efficiently when experts visit the production floor to check the status of the process, e.g., instruments, machines, and other essential production components. However, this is not always possible, for example, depending on the location of the production floor, or is not always advisable due to difficult conditions on the production floor.

[0005] Another example of a place where monitoring is desired is a home or a holiday home. People want to monitor their buildings remotely, for example, because a household appliance may malfunction, or a water tap or door may be open. In many cases, it is not appropriate to monitor the situation by looking at measurements from a measuring device received over the Internet, or to relay information about whether a door is open or closed, and the required information can be obtained more efficiently when a person enters the building and checks the situation for themselves. However, this is not always possible, for example, due to the location of a holiday home, or when the person is traveling or at work. Summary of the Invention

[0006] overview An improved method and technical equipment for implementing said method have now been invented. Various aspects of the invention include a computer-executable method, a camera system with at least one image sensor, and a computer-readable medium with a computer program stored therein, which are characterized in that they are set out in the independent claims. Various embodiments of the invention are disclosed in the dependent claims.

[0007] According to a first aspect of the present invention, there is provided a computer-implemented method including: acquiring previously captured virtual tour data of a location; receiving real-time image data of a target object located within a field of view of a camera, the target object being located at the same location as the location where the virtual tour data was captured; and overlaying the real-time image data onto a region of a virtual tour view to form mixed image virtual tour data, the region of the virtual tour view corresponding to the field of view.

[0008] According to one embodiment, the virtual tour data is static image data of the real environment of the location. According to one embodiment, the previously captured virtual tour data includes images during a walk through the location, the previously captured virtual tour data being acquired from a camera, memory, or cloud storage, and the method further includes constructing the acquired virtual tour data as a virtual tour by a data processing unit. According to one embodiment, the previously captured virtual tour data is a previously constructed virtual tour. According to one embodiment, the method further includes receiving real-time acoustic data captured by an audio sensor at the location and playing the acoustic data in a region of the mixed image virtual tour view corresponding to the location of the audio sensor or surroundings of the location. According to one embodiment, the method further includes acquiring additional information about the location and overlaying the additional information on the mixed image virtual tour view. According to one embodiment, the real-time image data is a real-time video stream or frequently captured real-time images. According to one embodiment, the method further includes capturing real-time image data by two or more cameras, which is overlaid on a region of the virtual tour view corresponding to a field of view. According to one embodiment, the point of view of the mixed image virtual tour can be changed to an earlier point in time.

[0009] According to a second aspect of the present invention, there is provided a system comprising at least one camera and a data processing unit, wherein the data processing unit is configured to acquire previously captured virtual tour data of a location, wherein the at least one camera is configured to capture real-time image data of a target object located within a field of view of the at least one camera, the target object being located at the same location as the location where the virtual tour data was captured, and to transmit the captured image data to the data processing unit, and the data processing unit is further configured to overlay the real-time image data on a region of the virtual tour view corresponding to the field of view to form mixed image virtual tour data.

[0010] According to one embodiment, the system further comprises an audio sensor configured to capture real-time audio data at the location and transmit the captured audio data to a data processing unit to be played in a region of the mixed image virtual tour view corresponding to or around the location of the audio sensor in the real environment. According to one embodiment, the data processing unit is further configured to obtain additional information about the location and overlay the additional information on the mixed image virtual tour view. According to one embodiment, the system comprises two or more cameras. According to one embodiment, the time point of the view of the mixed image virtual tour is changeable to an earlier time point.

[0011] According to a third aspect of the present invention there is provided a computer program product stored on a computer readable medium and executable on a computing device, said computer program product comprising instructions for performing a method according to the first aspect and embodiments thereof. [Brief explanation of the drawings]

[0012] BRIEF DESCRIPTION OF THE DRAWINGS The invention will now be described in more detail with reference to the accompanying drawings.

[0013] [Figure 1] 1a-1d illustrate an exemplary embodiment of the present invention in which a mixed image virtual tour of a paper machine environment is formed from static virtual tour data of the paper machine environment and at least one live video stream from the paper machine environment. [Figure 2] 2a-2f show an example of the principle of creating a mixed image virtual tour of a house according to one embodiment of the present invention. [Figure 3] 3a-3d illustrate an exemplary embodiment of the present invention, where two different views of a mixed image virtual tour of a paper machine environment are shown at two different times. [Figure 4] FIG. 4 illustrates an exemplary embodiment of the present invention where additional information is overlaid on a mixed image virtual tour view of a residential environment. [Figure 5] FIG. 5 illustrates an exemplary embodiment of the present invention, in which a block diagram of a computer-implemented method for extending a virtual tour into a mixed image virtual tour is disclosed. [Figure 6] FIG. 6 shows an example of a top view of a user terminal 611 presenting a wet end view of a mixed image virtual tour of a paper machine environment. DETAILED DESCRIPTION OF THE INVENTION

[0014] Detailed Description A virtual tour is a simulation of an existing location, e.g., a mechanical environment, typically consisting of a sequence of image data, e.g., video or still images. It can also combine other multimedia elements, such as sound effects, music, narration, and text within the image data. As used herein, the term "virtual tour" includes static virtual tours or static 3D virtual tours created / constructed using image data from a still or video camera. Static 3D virtual tours can be created using 3D reconstruction.

[0015] A static virtual tour can be created from multiple static images captured from a single imaging point as the camera and lens rotate around the imaging point, or from multiple static, adjacent images captured as the tour progresses, i.e., as the cameraman walks through the location, and then the adjacent images are combined into a single virtual tour image. A virtual tour created using a video camera is a linear walkthrough of a location. For a virtual tour, the location can be imaged, for example, at a walking pace while moving continuously through the location. Video may be captured, for example, every few meters, e.g., every 3 meters. These cameras typically store video data with distance information. Distance measurements may be performed, for example, using laser or lidar. This distance information can then be used to construct a 3D model of the location. The advantage of using this walkthrough method to capture virtual tour data compared to a virtual tour acquired from a single imaging point is that the viewpoint constantly changes as the tour progresses, allowing the person viewing the virtual tour to gain much more location information, especially if the location is large. However, capturing a walk-through virtual tour requires a person or other mobile solution to walk or move through a location. Also, obtaining real-time information from a virtual tour requires a person or other mobile solution to walk or move through a location at all times. A problem with real-time virtual tours, however, is that it is not always possible for a person to walk through a location due to its geographic location or, for example, a dangerous process or mechanical environment, or for some other mobile solution, such as an autonomous mobile robot, to move through the location due to height differences in the floor plan or other obstacles at the location. Additionally, even if an autonomous mobile robot moves through a location and captures images of the surroundings of the location along the way, once the robot reaches the end of its tour, the initial captured images and their associated information, such as measurements, become outdated.This means that the measurements at the start of the tour and at the end of the tour are not taken at the same moment and any synergistic effects, if any, cannot be seen.

[0016] As described above, the virtual tours and virtual tour data used in the present invention are static and typically include image data, e.g., video images, captured at one or more specific points in time in the past, i.e., in the history of a location. The concept of the present invention is to augment a static virtual tour by overlaying at least one or more live image streams of image points, i.e., spots, of a location onto one or more views of the virtual tour to provide an augmented virtual tour, i.e., a mixed image virtual tour. This type of augmented virtual tour provides a more intuitive user experience and also provides up-to-date information, along with real-time image data, to a person remotely monitoring the mixed image virtual tour location. A camera captures this real-time (live) image data of a specific target object, and the camera's field of view is configured to replace the corresponding area in the static virtual tour view, so that when a user views the mixed image virtual tour, this area of ​​view contains the real-time image. Thus, the virtual tour image data provides a kind of frame structure for the real-time image data, which is overlaid on the virtual tour view. The real-time image data may be, for example, real-time video image data, i.e., a real-time video stream or frequently captured real-time still images. For example, if a virtual tour was captured in a machine environment, say, one month ago, a specific region of the virtual tour view can be replaced with real-time image data of the same region, for example, by overlaying a real-time video stream on a static virtual tour view, to form an augmented mixed-image virtual tour. To augment the virtual tour, frequently captured real-time still images can also be used instead of the real-time video stream. As a more specific application example, a region of the display of a measuring device in a measuring device view of a virtual tour captured one month ago can be replaced with real-time image data of the same region of the measuring device display.Thus, as a user moves during the augmented mixed image virtual tour of the machine environment, the measuring device shows the current real-time conditions at the corresponding locations in the augmented mixed image virtual tour as they would in the actual machine environment. Thus, in the augmented mixed image virtual tour, the user can conduct a corresponding tour with an actual measuring device monitoring tour of the machine environment, but here, the monitoring tour is conducted virtually and the measuring device measurements are still up-to-date. And in this mixed image virtual tour, the measuring device readings can be visually analyzed. The number of real-time video streams overlaid on one view of the video tour may depend on the number of cameras capturing image data for the area shown in the view.

[0017] The capture of image data for the virtual tour can be performed by any suitable camera supported by a platform, e.g., a wheeled platform structure, or by a mobile robotic camera, an autonomous monitoring robotic system, a camera supported and used by a person, or any other suitable camera means. The term "camera" in this context includes any image sensor suitable for capturing images and / or video, i.e., image data for the virtual tour, such as a regular camera or a smart camera, a video camera, or any other suitable camera. The images / videos can be assembled into a virtual tour form by a data processing unit, e.g., a server, or by any known computing means, using any known method for constructing a virtual tour. The data processing unit may be a separate device that receives image data directly from the camera capturing the image data for the virtual tour, or it may receive images, e.g., from a memory, a cloud, or another device. The virtual tour data can also be obtained off-the-shelf, e.g., from a memory, cloud storage, or other device.

[0018] The real-time video image data or frequently captured real-time still images, i.e., the real-time image data configured to be added to the virtual tour, may be captured by any suitable camera type as described above, for example, a fixed surveillance camera aimed at a desired target. The real-time image data may be transmitted to a data processing unit, which may overlay the real-time image data on the virtual tour view, i.e., combine the real-time image data with the virtual tour data to provide a mixed image virtual tour. The mixed image virtual tour may then be shown / displayed or transmitted to a user for viewing / display, allowing the user to monitor the machine environment by moving within the mixed image data virtual tour even when the user is away from the machine environment.

[0019] Unlike augmented reality (AR), which shows a real-time environment and overlays portions of it with digital information, e.g., previously captured images, the augmented mixed image virtual tour of the present invention shows a previously captured virtual tour, e.g., a 360° virtual tour, of a location or environment and overlays areas of it with real-time images or video of corresponding areas, e.g., measurements of machine components, valves, belts, or other equipment, or other process measurements, so that the real-time image data is shown on top of the static virtual tour view and in the corresponding locations that would be found in the real-world location and environment.

[0020] 1a-1d illustrate an exemplary embodiment of the present invention in which a mixed-image virtual tour of a papermaking machine environment 100 is formed from image data of a static virtual tour of the papermaking machine environment 100 and a real-time video stream of at least one of target objects 101, 102 from the papermaking machine environment 100. FIG. 1a illustrates two cameras 103, 104, each equipped with an image sensor. The cameras 103, 104 are configured to image, or capture, real-time image data, which in this example is a real-time video stream. The first camera 103 is positioned to capture the real-time video stream from a first target object, which in this embodiment is the wet end 101 of the papermaking machine. The second camera 104 is positioned to image the real-time video stream from a second target object, which in this embodiment is an area of ​​a device 102 displaying measurement results from eight measurement devices that measure different data of the machine environment or the papermaking machine itself. The first and second viewing areas 107, 106 of the cameras 103, 104 are indicated by rectangles. The first and second field of view areas 106, 107 include first and second target objects 101, 102, respectively.

[0021] The cameras 103, 104 are equipped with data transfer means, such as a transmitter or transceiver, for transmitting real-time video image data from the cameras 103, 104 to the data processing unit 109 wirelessly or via a wired connection. The data processing unit 109 comprises at least one processor, at least one memory containing computer program code for one or more program units, and means, such as a receiver or transceiver, for receiving the real-time video image data from the cameras 103, 104 wirelessly or via a wired connection. There may be multiple processors, such as a general-purpose processor, a graphics processor, and / or a DSP processor, and / or multiple different memories, such as volatile memory for storing data and programs during execution and non-volatile memory such as a hard disk for persistently storing data and programs. The data processing unit 109 may be any computing device suitable for handling and processing image data, such as a computer. The data processing unit 109 can be in electronic communication with the cameras 103, 104 via signal lines. The data processing unit 109 may also include a video controller and an audio controller for generating signals that can be generated for a user using computer accessories. The data processing unit 109 may generate an output to a user via output means. The video controller may be connected to a display (not shown). The display may be, for example, a flat panel display, a tablet or laptop display, or a projector for generating a larger image. The audio controller may be connected to an audio source such as a speaker or earphones. The cameras 103, 104 may also include an acoustic sensor such as a microphone.

[0022] The data processing unit 109 is configured to receive real-time video image data from the cameras 103, 104 and to overlay with said real-time video image data at least one position / area of ​​the view of the static virtual tour, including a corresponding area that is an area imaged by the cameras 103, 104. The data processing unit 109 may have received image data of this static virtual tour from the cameras that captured it, and the data processing unit 109 may have constructed the virtual tour from this image data, or the data processing unit 109 may have received a pre-constructed virtual tour, for example from another computing device, from a memory or cloud storage.

[0023] Also, instead of two cameras 103, 104 as in this exemplary embodiment, there can be only one camera positioned to capture a real-time video stream, or there can be three or more cameras, for example 3 to 10 or more cameras, positioned to capture real-time video streams from different parts of the papermaking machine environment.

[0024] 1b shows, by way of example, a first viewing area 107 configured to be overlaid, or superimposed, on a corresponding area of ​​a previously imaged static virtual tour of the paper machine environment 100. Pulp fed onto a wire is shown in the first viewing area 107. Real-time video image data of the second viewing area 106 may also be superimposed on an area of ​​the virtual tour view corresponding to the area of ​​the second viewing area 106, although this is not shown in this example.

[0025] Figure 1c shows a wet end view 110 of a virtual tour of the paper machine environment 100. This view 110 includes the same target object 101 as the first field of view 107, i.e., the wet end 101 of the paper machine.

[0026] 1d shows a top view of a user terminal 111 presenting a static virtual tour view 110 as a mixed image virtual tour view of the paper machine environment 100. In this embodiment, the user terminal 111 is a monitor. In this mixed image virtual tour view 110, the first viewing area 107 is overlaid on a corresponding area of ​​the static virtual tour view 110 to form the mixed image virtual tour view. As can be seen, while the remainder of the view 110 is old and static, the wet end 101 area contains real-time image data that is updated in real time. At this time, the morphology of the pulp on the wire is different than when the virtual tour was captured.

[0027] 2a-2f illustrate the principle of creating a mixed image virtual tour of a house 200 according to an exemplary embodiment. The mixed image virtual tour of the house 200 is created from a previously captured static virtual tour of the house 200 and two real-time video streams. FIG. 2a shows three cameras 201a, 201b, and 201c, each equipped with an image sensor. The cameras 201a, 201b, and 201c are configured to capture real-time image data, which in this example is a real-time video stream. The first camera 201a is positioned to capture the real-time video stream from a first target object, which in this embodiment is the switch for a coffee maker 203. The second camera 201b is positioned to capture the real-time video stream from a second target object, which in this embodiment is a bathroom faucet 204. The third camera 201c is positioned to capture the real-time video stream from a third target object, which in this embodiment is a birdcage 205. The viewing areas 203a, 204a, 205a of the first, second, and third cameras 201a, 201b, 201c are shown as corresponding rectangles. Each viewing area 203a, 204a, 205a includes a corresponding target object 203, 204, 205. In this embodiment, a microphone 201d is also present. The microphone 201d is connected to the birdcage 20. 5 Bird cage 20 is installed so that sound signals, i.e., acoustic data, generated by the birds within the cage can be captured. 5is placed in the vicinity of

[0028] The cameras 201a, 201b, and 201c each include a data transfer means, such as a transmitter or transceiver, for transmitting real-time video image data from the cameras 201a, 201b, and 201c to the data processing unit 202 wirelessly or via a wired connection. The data processing unit 202 may be, for example, a server. The data processing unit 202 includes at least one processor, at least one memory containing computer program code for one or more program units, and means, such as a receiver or transceiver, for receiving real-time video image data from the cameras 201a, 201b, and 201c wirelessly or via a wired connection and audio data from the microphone 201d. There may be multiple processors, such as a general-purpose processor, a graphics processor, and / or a DSP processor, and / or multiple different memories, such as volatile memory for storing data and programs during execution and non-volatile memory such as a hard disk for persistently storing data and programs. The data processing unit 202 may be any computing device, such as a computer, suitable for handling / processing image and audio data. The data processing unit 202 can be in electronic communication with the cameras 201a, 201b, 201c and the microphone 201d via signal lines. The data processing unit 202 can also include a video controller and an audio controller for generating signals that can be generated for a user using a computer accessory. The data processing unit 202 can generate output to a user via its output means. The video controller can be connected to a display (not shown). The display can be, for example, a flat panel display, a laptop display, or a projector that generates a larger image. The audio controller can be connected to an audio source such as speakers or earphones. Physically, the data processing unit 202 can be located inside or outside the home 200. It is also possible that the cameras 201a, 201b, 201c, and the microphone 201d transfer their data to cloud storage, from which the data processing unit 202 can retrieve the data.

[0029] The data processing unit 202 is configured to receive real-time video image data from the cameras 201a, 201b, 201c, receive audio signals from the microphone 201d or cloud storage, add the above-mentioned real-time video image data to a preferred view of the static virtual tour including the corresponding areas imaged by the cameras 201a, 201b, 201c, and add the audio signal to a preferred position of the view of the static virtual tour corresponding to the position where the microphone 201d is located or at a position of the view of the static virtual tour where the audio signal captured by the microphone 201d would be heard in a real-world situation.

[0030] Again, the number of cameras and / or microphones capturing real-time data may vary, and the number of cameras and / or microphones from which data is added to the mixed image virtual tour may be changed as needed or desired. Cameras 201 a, 201 b, 201 c may also include acoustic sensors, such as microphones.

[0031] 2b shows, by way of example, a first field of view 203a of a coffee maker switch configured to be superimposed on a corresponding region of a kitchen view 210 of a previously imaged static virtual tour of a home 200. From the first field of view 203a, it can be seen that the switch is currently off. From the second field of view 203a of a bathroom faucet 204, it can be seen that the switch is currently off. 4a 2b, can be configured to be superimposed on a corresponding area of ​​the bathroom view of the static virtual tour of the house 200. Figure 2b also shows a third viewing area 205a of the birdcage 205 configured to be superimposed on a corresponding area of ​​the living room view of the static virtual tour of the house 200. From the third viewing area 205a, the current status of the bird in the cage 205 can be seen.

[0032] 2c shows a kitchen view 210 of the virtual tour of the home 200. This view 210 shows a first target object, the coffee maker switch, configured to be replaced by the first field of view 203a, and the switch is on.

[0033] 2d shows kitchen view 210 on a handheld user terminal 211, e.g., a tablet, presenting kitchen view 210 of the mixed image virtual tour. In this view 210, a first field of view region 203a, i.e., the coffee maker switch region, is superimposed, or overlaid, on a corresponding region of kitchen view 210. As can be seen, while the rest of view 210 is outdated and static, the coffee maker switch region contains live data, is updated in real time, and is in a different state than when virtual tour kitchen view 210 was captured.

[0034] 2e shows a living room view 212 of the virtual tour of the house 200. This view 212 shows a third target object, namely, a birdcage 205, which is configured to be replaced by a third field of view 205a in the mixed image virtual tour.

[0035] 2f shows a living room view 212 on a user terminal 211. In this view 212, a third viewing area 205a, the birdcage area, is shown as the living room view 211 of the virtual tour. 2 As can be seen, in this expanded view, the virtual tour's living room view 21 2 The bird is positioned in a different position than when it was imaged.

[0036] The audio signal captured by microphone 201d can also be configured to be played when the user enters the living room in the mixed image virtual tour. The audio signal captured by microphone 201d can also be configured to be played when the user exceeds a certain predetermined distance from the birdcage when entering the living room in the mixed image virtual tour, and stopped when the user exceeds a second certain predetermined distance from the birdcage when exiting the living room in the mixed image virtual tour.

[0037] To allow a user to more easily detect which portions of the mixed image virtual tour are real-time data and which portions are not, replaced regions in one or more views of the mixed image virtual tour, i.e., regions where real-time image data is superimposed on a view of the static virtual tour, can be indicated. This indication can be done, for example, by surrounding the replaced regions with color or lighting (which may or may not blink), by using brighter lighting in the replaced regions, or by slightly blurring the periphery of the real-time regions.

[0038] Figures 3a-3d illustrate an exemplary embodiment of the present invention, where these two different views of a mixed image virtual tour of a paper machine environment are shown with overlaid image data, i.e., video streams, captured at two different times. These Figures 3a-3d disclose previously captured static virtual tour views 110, 310 of the paper machine environment and video streams that overlay specific regions of these views.

[0039] FIG. 3 a shows a monitoring room view of a mixed image virtual tour, showing two devices displaying measurement results. In this image, a video camera captures a real-time video stream from the target object, which is the same area as device 102 displaying measurement results in FIG. 1 a. The camera's field of view 106 is then overlaid on a previously captured static virtual tour view 310 and again at the corresponding location where this area of ​​device 102 displaying measurement results is in the actual paper machine environment and in the previously captured static virtual tour monitoring room view 310. The time point of this mixed image virtual tour view is indicated by a time indicator 320 above the view, although this time indicator 320 could also be displayed in some other manner. As can be seen, the time point at which the overlaid image data shown was captured corresponds to the current moment, and the data superimposed on the previously captured static virtual tour view 310 is real-time image data.

[0040] Figure 3b shows a wet-end view of the mixed image virtual tour, showing pulp being fed onto the wire. A real-time video stream from a target object, the same wet-end region as in Figure 1a, is captured by a second video camera. The camera's field of view 107 is then superimposed over a previously captured static virtual tour view 110, showing this region of the wet-end within the actual paper machine environment and the previously captured static virtual tour wet-end view 110. 1 10. The time at which the image data for this mixed image virtual tour view was captured by the second video camera is again indicated by a time indicator 320 above the view. As can be seen, this time also corresponds to the present moment, and the data overlaid on the previously captured static virtual tour view 110 is real-time image data.

[0041] However, according to examples of the present invention, it is possible to reverse the time point of the mixed image virtual tour, i.e., to change the time point of the overlaid image data displayed in the views of the mixed image virtual tour to a time point earlier than the current time point. This earlier time point may be the same in all views of the mixed image data virtual tour. These earlier views of the mixed image virtual tour then show views of the static virtual tour captured previously and views of image data captured at the same earlier time point. This type of presentation is not possible in the case of a typical virtual tour, because the imaging devices that capture or will capture image data for the virtual tour cannot capture the data so that the image capture time is exactly the same in all views. And, as already mentioned above, this ability to show simultaneously captured information in all views of the mixed image virtual tour may be necessary in determining the causality of damage and avoiding its consequences.

[0042] Figures 3c and 3d show the same monitoring room view and wet-end view of the mixed image virtual tour as Figures 3a and 3b, respectively, but at an earlier time point, i.e., one hour earlier. Thus, Figure 3c shows the monitoring room view, with two devices displaying measurement results. The camera's field of view 106 is again overlaid on the previously captured static virtual tour view 310. The time point for capturing the overlaid image data of this monitoring room view of the mixed image virtual tour is indicated by time indicator 320. The image data overlaid on the previously captured static virtual tour view 310 corresponds to video image data captured at the indicated time point of the area of ​​device 102 displaying measurement results from eight measurement devices measuring different data of the machine environment or the papermaking machine itself. Figure 3d shows the wet-end view. The camera's field of view 107 is again overlaid on the previously captured static virtual tour view 110. The time point for this wet-end view of the mixed image virtual tour is again indicated by time indicator 320. The image data overlaid on the previously captured static virtual tour view 110 corresponds to video image data captured at the indicated time point in the wet end of the paper machine and indicated by time indicator 320.

[0043] As can be seen, there are differences between the measurements in viewing area 106 of the monitoring room view and the measurements on the pulp in viewing area 107 of the wet end view between different time points, while the remaining views of the mixed image virtual tour remain the same.

[0044] Changing the time point of the displayed view can be determined, for example, by moving a square portion on the horizontal axis of the time indicator 320 or by any other method. Note that at a selected time point, it is predetermined that all data shown overlaid on the views of the mixed image virtual tour was captured at that same selected time point. In other words, if a user selects to view a mixed image virtual tour captured 10 minutes ago, the images overlaying all views were captured 10 minutes ago, but in the real-time views, all data comprising the images or image streams overlaying all views is real-time image data. However, it is also possible to select the time point so that at least one view shows image data captured at a different time point compared to the image data overlaid on other views of the mixed image virtual tour.

[0045] 4 shows a living room view 212 on the same user terminal 211 as in FIG. 2f. In this view 212, the birdcage viewing area 205a of real-time video image data is combined with the living room view 211 of the previously captured virtual tour so that an augmented virtual tour, i.e., a mixed image virtual tour, is formed by the data processing unit. 2 However, in addition to the real-time image data, additional information is also overlaid on the corresponding area of ​​the living room view 21. 2 The additional information that can be shown in the view of the mixed image virtual tour can be, for example, any information about the environment of the view or other parts of the virtual tour. 410 relates to the birdcage shown in the birdcage viewing area 205a. 410 indicates the temperature around the birdcage 205 and the water level in the water basin inside the birdcage 205. 411 is associated with a lighting device, i.e., a ceiling lamp, and indicates that the lamp is on.

[0046] The additional information shown in FIG. 4 is merely exemplary. The additional information may also relate to, for example, sensor readings or measurements shown in the viewing area, or other information related to those measurements. The sensor readings or measurements may be captured by sensors within the location, such as sensors on a machine and / or other connected devices. In some cases, the additional information indicates captured sensor data related to structures, devices, or other objects within the location. The additional information may be shown, for example, next to, below, or above the viewing area, or in any other location above the view of the mixed image virtual tour. The additional information may be captured simultaneously with the image data overlaying the view or may be historical data, although the additional information data may also be collected before or after the time the image data overlaying the view of the mixed image virtual tour is captured, such as one minute before. Alternatively, any other time period may be used instead of one minute. The time period may depend, for example, on the additional information, or the user may configure the time period.

[0047] The data processing unit can obtain additional information about the location or environment, for example from the sensors or measuring devices mentioned above, and overlay the additional information onto the blended virtual tour data.

[0048] 5 shows a flowchart of a computer-implemented method 500 for extending a virtual tour into a mixed image virtual tour. In step 510, previously captured virtual tour data of a location is obtained. In step 520, real-time image data of a target object located within the field of view of the camera is received, the target object being located at the same location where the virtual tour data was captured. In step 530, the real-time image data is overlaid, i.e., shown, on a region of the virtual tour view corresponding to the field of view to form the mixed image virtual tour data.

[0049] 6 shows an example of a top view of a user terminal 611 presenting a wet-end view 610 of a mixed image virtual tour of a paper machine environment. In this example, the wet-end area of ​​the paper machine is the target object. A real-time video camera is present in the paper machine environment, capturing a real-time video stream of the wet-end area. In this wet-end view 610 of the mixed image virtual tour, real-time video stream data 607 of the real-time video camera's field of view is overlaid on the corresponding wet-end area of ​​the static virtual tour wet-end view 610 to form the mixed image virtual tour. However, in this example, an acoustic sensor, such as a microphone, is also present in the wet-end area of ​​the paper machine.

[0050] Acoustic sensors detect environmental sounds. 600 The recorded environmental sound is placed 600 is arranged to be transmitted as acoustic data to a data processing unit configured to build a mixed virtual tour. The acoustic sensor comprises means for transmitting the acoustic data to the data processing unit. The means for transmitting the acoustic data may for example be a transceiver or transmitter arranged to transmit the acoustic data from the sensor to the data processing unit wirelessly or via a wired connection.

[0051] The data processing unit configured to build a mixed virtual tour comprises at least one processor, at least one memory containing computer program code for one or more program units, and means for receiving acoustic data from an acoustic sensor wirelessly or via a wired connection. The means for receiving acoustic data can be, for example, a receiver or a transceiver. Examples of data processing units are defined more precisely above in connection with the preceding figures.

[0052] Thus, the data processing unit is configured to receive acoustic data from acoustic sensors that record environmental acoustic signals, convert the acoustic signals into electrical acoustic data, and transmit the acoustic data to the data processing unit, and the data processing unit is arranged to play back the acoustic data within a wet-end view of the mixed image virtual tour that corresponds to the locations in the actual paper machine environment where the acoustic sensors were positioned. Thus, the mixed image virtual tour is further extended with audio signal data in addition to the real-time video stream data.

[0053] An audio sensor according to the present invention that captures or records real-time acoustic data comprises at least circuitry and electronics for recording environmental acoustic sounds or data, converting the acoustic data into electrical signals, and transmitting the acoustic data. A camera according to the present invention comprises at least circuitry and electronics for capturing real-time images or video streams and transmitting the image data.

[0054] It should be noted that the number of cameras positioned to provide real-time image data of the target object that is overlaid on the view of the mixed image virtual tour can vary. There can be one camera or multiple cameras. The number of cameras positioned to overlay image data on the view of the mixed image virtual tour can also vary. There can be multiple cameras capturing image data of the target object, but only one, two, etc., of the field of view can be shown on the view. The shown image data or additional information of the field of view can also be changeable by the user or can depend on, for example, the time of day, the season, the user profile, outdoor temperature, indoor temperature, humidity, etc., so that only the most relevant additional information is overlaid on the view.

[0055] It is obvious that the invention is not limited solely to the embodiments presented above, but it can be modified within the scope of the appended claims. The present application provides the following aspects of the invention. (Aspect 1) 1. A computer-implemented method comprising: Obtaining previously captured image data of a virtual tour of a location; receiving real-time image data of a target object located within a field of view of a camera, the target object being located at the same location where the image data was captured; overlaying the real-time image data of the target object onto a corresponding region of the target object in a virtual tour view to form a mixed image virtual tour; receiving real-time acoustic data captured by an audio sensor within the location; and playing the real-time acoustic data when a user traveling through the mixed image virtual tour is within a certain predetermined distance of the audio sensor. (Aspect 2) 2. The computer-implemented method of claim 1, wherein the image data is static image data of a real-world environment of the location. (Aspect 3) 3. The computer-implemented method of claim 1 or 2, wherein the previously captured image data includes images captured during a walk through the location, and the previously captured image data is obtained from a camera, memory, or cloud storage, and the method further includes constructing the image data as a virtual tour by a data processing unit. (Aspect 4) 3. The computer-implemented method of any one of claims 1 to 2, wherein the previously captured image data is a previously constructed virtual tour. (Aspect 5) The method comprises: obtaining additional information about the location; and Aspects 1-4: The computer-implemented method of any one of aspects 1-4, further comprising: overlaying the additional information on the mixed image virtual tour view. (Aspect 6) 6. The computer-implemented method of any one of aspects 1 to 5, wherein the real-time image data is a real-time video stream or frequently captured real-time images. (Aspect 7) The method comprises: Aspects 7. The computer-implemented method of any one of aspects 1-6, further comprising capturing, by two or more cameras, real-time image data configured to be overlaid on a region of the virtual tour view. (Aspect 8) Aspects 8. The computer-implemented method of any one of aspects 1 to 7, wherein the point of view of the mixed image virtual tour is changeable to an earlier point in time. (Aspect 9) 1. A system comprising at least one camera and a data processing unit, wherein the data processing unit is configured to acquire previously captured image data of a virtual tour of a location, wherein the at least one camera is configured to capture real-time image data of a target object located within a field of view of the at least one camera, the target object being located at the same location as the location where the image data was captured, and to transmit the captured image data to the data processing unit, wherein the data processing unit is further configured to overlay the real-time image data of the target object on a corresponding area of ​​a virtual tour view to form a mixed image virtual tour, receive real-time acoustic data captured by an audio sensor within the location, and play back the real-time acoustic data when a user moving through the mixed image virtual tour is within a certain predetermined distance of the audio sensor. (Aspect 10) The system of aspect 9, wherein the data processing unit is further configured to obtain additional information about the location and overlay the additional information on the mixed image virtual tour view. (Aspect 11) 11. The system of claim 9 or 10, wherein the system comprises two or more cameras. (Aspect 12) 12. The system of any one of aspects 9 to 11, wherein the point of view of the mixed image virtual tour is changeable to an earlier point in time. (Aspect 13) 9. A computer program product stored on a computer-readable medium and executable on a computing device, the computer program product comprising instructions for performing the method of any one of aspects 1-8.

Claims

1. 1. A computer-implemented method comprising: Obtaining previously captured image data of a virtual tour of a real-world environment of a location; constructing the previously captured image data as a virtual tour view by a data processing unit if the previously captured image data is not an already constructed virtual tour view; receiving real-time image data of a target object (101, 102) located within a field of view (106, 107) of a camera (103, 104) at a camera position in the real environment of the location; overlaying the real-time image data of the target objects (101, 102) at positions within a virtual tour view to form a mixed image virtual tour view; Including, The method comprises: receiving real-time acoustic data captured by an audio sensor at an audio sensor location in the real environment of the location; playing the real-time acoustic data when a user traveling through the mixed image virtual tour view is within a certain predetermined distance of a position corresponding to the audio sensor position in the real-world environment of the location; The method further comprises:

2. The method of claim 1 , wherein the previously captured image data is static image data of a real-world environment of the location of the virtual tour view.

3. 3. The method of claim 1 or 2, wherein the previously captured image data includes images captured by a person walking through the real environment of the location, and the previously captured image data is obtained from a camera, memory, or cloud storage.

4. The method comprises: - obtaining additional information (410, 411) about the real environment of the location, said additional information (410, 411) being information received from one or more sensors or measuring devices; The method of any one of claims 1 to 3, further comprising overlaying said additional information (410, 411) on the mixed image virtual tour view.

5. The method according to any one of claims 1 to 4, wherein the real-time image data is a real-time video stream or a captured real-time image.

6. The method comprises: The method of any one of claims 1 to 5, further comprising capturing real-time image data by two or more cameras (103, 104).

7. The method according to any one of claims 1 to 6, wherein the point in time from which the mixed image virtual tour view is formed is variable to one or more specific points in time in the past.

8. A system comprising at least one camera and a data processing unit, wherein the data processing unit (109) is configured to acquire previously captured image data of a virtual tour of a real environment of a location, for constructing the previously captured image data as a virtual tour view by the data processing unit if the previously captured image data is not already a constructed virtual tour view, wherein the at least one camera (103, 104) captures real-time image data of a target object (101, 102) located within a field of view (106, 107) of the at least one camera (103, 104) at a camera position of the real environment of the location, and to process the previously captured image data into the data processing unit (109). 9), wherein the data processing unit (109) is further configured to overlay the real-time image data of the target objects (101, 102) at a position of the virtual tour view corresponding to the camera position to form a mixed image virtual tour view, and the data processing unit is further configured to receive real-time acoustic data captured by an audio sensor at an audio sensor position in the real environment of the location, and play back the real-time acoustic data when a user moving through the mixed image virtual tour view is within a certain predetermined distance of a position in the real environment of the location corresponding to the audio sensor position.

9. 9. The system of claim 8, wherein the data processing unit is further configured to: obtain additional information (410, 411) about the real environment of the location, the additional information (410, 411) being information received from one or more sensors or measuring devices; and overlay the additional information on the mixed image virtual tour view.

10. The system according to claim 8 or 9, wherein the system comprises two or more cameras (103, 104).

11. The system according to any one of claims 8 to 10, wherein the point in time at which the mixed image virtual tour view is formed can be changed to one or more specific points in time in the past.

12. A computer program stored on a computer readable medium and executable on a computing device, said computer program comprising instructions for carrying out the method of any one of claims 1 to 7.

Images