Expanded virtual tour

Abstract

The invention relates to a computer-implemented method including acquiring previously captured virtual tour data (501) of a location, receiving real-time image data of a target object located within a viewing area of ​​a camera (503, 504), the target object being located at the same location where the virtual tour data (501) was captured, analyzing the real-time image data (503, 504) to detect deviations, and if deviations are detected, overlaying the real-time image data (503, 504) onto an area of ​​a virtual tour view to form a mixed image virtual tour data, the area of ​​the virtual tour view corresponding to the viewing area (503, 504). The invention further relates to a system for performing the method, and a computer program product including instructions for performing the method.

Description

Technical Field

[0001] Field of the Invention The present invention relates to a computer-implemented method for extending virtual tours.

[0002] The present invention also relates to a system and a computer program product for causing the method to be performed on a device.

Background Art

[0003] Background of the Invention In many environments, such as industrial environments and production plants, there are situations where monitoring of a process or environment is necessary, for example, to identify changes or emerging faults. This is predictive maintenance. By using monitoring, it becomes possible to schedule maintenance or take other measures to prevent resulting damage and avoid its consequences. However, the environments that benefit from monitoring are not only industries, and monitoring the condition or state of a building also enhances the owner's sense of security.

[0004] For example, in industrial environments, there are multiple different types of elements, features, or factors that need to be monitored. For example, in the manufacturing process of pulp and paper in the paper industry, it is necessary to monitor the process to detect possible faults. In many cases, simply monitoring by looking at measurement values, viewing image data generated by cameras from various locations on the manufacturing site on a screen, or relying on the analysis results of image processing devices is not always sufficient. More efficient information acquisition can be achieved when an expert visits the manufacturing site to check the condition of the process, such as instruments, machines, and other essential manufacturing parts. However, this is not always possible depending on the location of the manufacturing site, for example, or is not always recommended depending on the difficult conditions at the manufacturing site.

[0005] Another example of a place where monitoring is desirable is a house or vacation home. For example, people want to remotely monitor a building because of potential malfunctions in household appliances or because a water tap or door might be left open. Often, it's not practical to monitor the situation by viewing measurements from measuring devices received over the internet or relaying door open / closed information; it's more efficient to obtain the necessary information when someone enters the building and checks the situation themselves. However, this isn't always possible, for example, if it's a vacation home, while traveling, or at work. [Overview of the Initiative]

[0006] overview Herein, improved methods and technical devices for implementing such methods are invented. Various aspects of the present invention include a computer-executable method, a camera system comprising at least one image sensor, and a computer-readable medium comprising a computer program stored therein, which are characterized by being described in the independent claims. Various embodiments of the present invention are disclosed in the dependent claims.

[0007] A computer implementation method is provided that, according to a first aspect of the present invention, includes acquiring previously captured virtual tour data of a certain location, receiving real-time image data of a target object located within the field of view of a camera, wherein the target object is located at the same location where the virtual tour data was captured, analyzing the real-time image data to detect deviations, and, if deviations are detected, overlaying the real-time image data onto a region of a virtual tour view to form mixed-image virtual tour data, wherein the region of the virtual tour view corresponds to the field of view.

[0008] According to one embodiment, virtual tour data is static image data of the real environment of the location. According to one embodiment, previously captured virtual tour data includes images taken during a walk through the location, and the previously captured virtual tour data is acquired from a camera, memory, or cloud storage, and the method further includes constructing the acquired virtual tour data as a virtual tour by a data processing unit. According to one embodiment, previously captured virtual tour data is an already constructed virtual tour. According to one embodiment, the method further includes receiving real-time acoustic data captured by an audio sensor in the location, analyzing the received acoustic data to detect machine malfunctions, and, if a malfunction is detected, playing the acoustic data in the area of ​​the audio sensor or the area of ​​the mixed-image virtual tour view corresponding to the location, in order to further extend the mixed-image virtual tour with the real-time acoustic data. According to one embodiment, the method further includes acquiring additional information about the location, analyzing the additional information to detect deviations, and, if a deviation is detected, overlaying the additional information on the mixed-image virtual tour view. According to one embodiment, real-time image data is a real-time video stream or frequently captured real-time images. According to one embodiment, the method further includes capturing real-time image data, which is overlaid on a virtual tour view area corresponding to a field of view area, using two or more cameras. According to one embodiment, the time point of the view of the mixed-image virtual tour is changeable to a previous time point.

[0009] According to a second aspect of the present invention, a camera system is provided comprising at least one camera and a data processing unit, the data processing unit being configured to acquire previously captured virtual tour data of a certain location, the at least one camera being configured to acquire real-time image data of a target object located within the field of view of at least one camera, the target object being located at the same location where the virtual tour data was acquired, and the data processing unit being further configured to analyze the real-time image data to detect deviations, and if a deviation is detected, to overlay the real-time image data onto a region of a virtual tour view corresponding to the field of view to form mixed-image virtual tour data.

[0010] According to one embodiment, the system further comprises a sound sensor configured to acquire acoustic data at a location, transmit the acquired acoustic data to a data processing unit for analysis, and, if a malfunction is detected, to play back the real-time acoustic data in a region of a mixed-image virtual tour view corresponding to the location of the sound sensor in the real environment or its surroundings. According to one embodiment, the data processing unit is further configured to acquire additional information about the location, analyze the additional information, and, if the data processing unit detects a deviation, to overlay the additional information on the mixed-image virtual tour view. According to one embodiment, the system comprises two or more cameras. According to one embodiment, the time point of the view of the mixed-image virtual tour is changeable to a previous time point.

[0011] According to a third aspect of the present invention, a computer program product is provided which is stored on a computer-readable medium and is executable on a computing device, the computer program product including instructions for carrying out the method according to the first aspect and its embodiments. [Brief explanation of the drawing]

[0012] Brief explanation of the drawing The invention will be described in more detail below with reference to the attached drawings. [Figure 1]Figures 1a to 1d illustrate an exemplary embodiment of the present invention in which a mixed-image virtual tour of the papermaking machine environment is formed from static virtual tour data of the papermaking machine environment and at least one live video stream from the papermaking machine environment. [Figure 2] Figures 2a to 2f illustrate an example of the principle of forming a mixed-image virtual tour of a house according to one embodiment of the present invention. [Figure 3] Figures 3a to 3d illustrate an exemplary embodiment of the present invention, showing two different views of a mixed-image virtual tour of a papermaking machine environment at two different points in time. [Figure 4] Figure 4 shows an exemplary embodiment of the present invention, in which additional information is overlaid on a mixed-image virtual tour view of the residential environment. [Figure 5] Figure 5 shows an exemplary embodiment of the present invention, illustrating a view of a mixed-image virtual tour. [Figure 6] Figure 6 shows an exemplary embodiment of the present invention, illustrating a view of a mixed-image virtual tour. [Figure 7] Figure 7 shows an exemplary embodiment of the present invention, illustrating a view of a mixed-image virtual tour. [Figure 8] Figure 8 shows one embodiment of the present invention, disclosing a block diagram of a computer implementation method for extending a virtual tour to a mixed-image virtual tour when a deviation is detected. [Modes for carrying out the invention]

[0013] Detailed explanation A virtual tour is typically a simulation of an existing location, such as a mechanical environment, consisting of image data, such as a sequence of video or still images. Other multimedia elements, such as sound effects, music, narration, and text, can also be combined within the image data. In this specification, the term “virtual tour” includes static virtual tours or static 3D virtual tours created / constructed using image data from a still camera or video camera. Static 3D virtual tours can be created using 3D reconstruction.

[0014] A static virtual tour can be created from multiple static images captured from a single imaging point as the camera and lens rotate around the imaging point, or from multiple static adjacent images captured as the tour progresses, i.e., as the imager walks through the location, and these adjacent images are then combined into a single virtual tour image. A virtual tour created using a video camera is a linear walkthrough of the location. For a virtual tour, for example, the location can be imaged at a walking pace while continuously moving through it. Video may be captured, for example, every few meters, or every three meters. These cameras typically store video data that contains distance information. Distance measurement may be performed, for example, using a laser or lidar. This distance information can then be used to construct a 3D model of the location. Compared to a virtual tour acquired from a single imaging point, the advantage of using this walkthrough method to capture virtual tour data is that the viewpoint constantly changes as the tour progresses, and the person viewing the virtual tour obtains much more information about the location, especially if the location is large. However, incorporating a walkthrough virtual tour requires walking or moving through locations where a person or other mobile solution is present. Furthermore, obtaining real-time information from a virtual tour requires that a person or other mobile solution constantly walk or move through locations where they are present. The problem with real-time virtual tours, however, is that it's not always possible for a person to walk through a location due to its geographical location, or, for example, a dangerous process or mechanical environment, or for another mobile solution, such as an autonomous mobile robot, to move through that location due to elevation differences in the floor plan or other obstacles at that location. In addition, even if an autonomous mobile robot moves through a location and images its surroundings along the way, by the time the robot reaches the end of the tour, the initial images and their information, such as measurement results, become outdated.This means that the measurement results at the start and end of the tour are not taken at the same moment, i.e., the same point in time, and therefore, even if there is some synergistic effect, it cannot be seen.

[0015] As described above, the virtual tours and virtual tour data used in the present invention are static and typically include image data, such as video images, captured at one or more specific points in the past, i.e., in the history of a particular location. The concept of the present invention is to extend a static virtual tour by overlaying at least one or more live image streams of imaging points, i.e., spots, of a location onto one or more views of the virtual tour in order to provide an extended virtual tour, i.e., a mixed-image virtual tour. This type of extended virtual tour provides a more intuitive user experience and also provides up-to-date information, along with real-time image data, to a person remotely monitoring the location of the mixed-image virtual tour. A camera captures this real-time (live) image data of a particular target object, and the camera's field of view is configured to replace the corresponding area in the view of the static virtual tour, so that when a user views the mixed-image virtual tour, the area of ​​this view includes the real-time image. Thus, the virtual tour image data provides a kind of frame structure for the real-time image data, which is overlaid on the view of the virtual tour. The real-time image data may be, for example, real-time video image data, i.e., a real-time video stream or frequently captured real-time still images. For example, if a virtual tour was captured in a mechanical environment, say, one month ago, an extended hybrid image virtual tour can be formed by overlaying, for example, a real-time video stream onto the static virtual tour view, thereby replacing a specific area of ​​the virtual tour view with real-time image data of the same area. Instead of a real-time video stream, frequently captured real-time still images can also be used to extend the virtual tour. Furthermore, as a more specific application example, an area of ​​the measurement device's display in the measurement device view of a virtual tour captured one month ago can be replaced with real-time image data of the same area of ​​the measurement device's display.Therefore, if a user moves during an augmented mixed-image virtual tour of the machine environment, the measurement device will display the current real-time conditions at the corresponding location within the augmented mixed-image virtual tour as if it were the location in the actual machine environment. Thus, in the augmented mixed-image virtual tour, the user can conduct a tour that corresponds to an actual measurement device monitoring tour in the machine environment, but here the monitoring tour is conducted virtually, and the measurement device readings remain up-to-date. And in this mixed-image virtual tour, the readings of the measurement device can be visually analyzed. The number of real-time video streams overlaid on one view of the video tour may depend on the number of cameras capturing image data of the area shown in the view.

[0016] Image data acquisition for a virtual tour may be carried out by any suitable camera supported by a platform, e.g., a wheeled platform structure, or by a mobile robot camera, an autonomous monitoring robot system, a camera supported and used by a person, or any other suitable camera means. In this context, the term "camera" includes any image sensor suitable for acquiring images and / or video, i.e., image data for a virtual tour, e.g., a conventional camera or smart camera, a video camera, or any other suitable camera. Images / videos can be constructed into a virtual tour form by a data processing unit, e.g., a server, or by any known computing means, using any known method of constructing the virtual tour. The data processing unit may be a separate device that receives image data directly from the camera acquiring the image data for the virtual tour, or it may receive images from, for example, memory, the cloud, or another device. Virtual tour data can also be acquired as off-the-shelf products from, for example, memory, cloud storage, or other devices.

[0017] Real-time video image data or frequently captured real-time still images, i.e., real-time image data configured to be added to a virtual tour, may be captured by any preferred camera type described above, for example, a fixed surveillance camera pointed at a desired target. The real-time image data may be transmitted to a data processing unit, which may overlay the real-time image data onto a virtual tour view, i.e., combine the real-time image data with virtual tour data to provide a mixed-image virtual tour. The mixed-image virtual tour may then be shown / displayed or transmitted for a user to view / display, allowing the user to monitor the machine environment by navigating within the mixed-image virtual tour, even when the user is away from the machine environment.

[0018] Unlike augmented reality (AR), which displays a real-time environment and overlays a portion of it with digital information, such as previously captured images, the augmented mixed-image virtual tour of the present invention displays a virtual tour of a previously captured location or environment, such as a 360° virtual tour, and overlays that area with real-time images or videos of a corresponding area, such as measurement results of a mechanical component, valve, belt, or other equipment, or other process measurements, so that the real-time image data is displayed on top of the static virtual tour view and in the corresponding locations that would be seen in the real-world location and environment.

[0019] Figures 1a to 1d show an exemplary embodiment of the present invention in which a mixed image virtual tour of a papermaking machine environment 100 is formed from image data of a static virtual tour of the papermaking machine environment 100 and at least one real-time video stream of target objects 101, 102 from the papermaking machine environment 100. Figure 1a shows two cameras 103, 104 each equipped with an image sensor. The cameras 103, 104 are configured to capture, i.e., capture, real-time image data, which is a real-time video stream in this example. The first camera 103 is arranged to capture a real-time video stream from a first target object, which is the wet end 101 of the papermaking machine in this embodiment. The second camera 104 is arranged to capture a real-time video stream from a second target object, which is, in this embodiment, the area of a device 102 that displays the measurement results of eight measurement devices that measure different data of the machine environment or the papermaking machine itself. The first and second field-of-view regions 107, 106 of the cameras 103, 104 are shown by rectangles. The first and second field-of-view regions 106, 107 correspondingly include the first and second target objects 101, 102.

[0020] Cameras 103 and 104 are provided with data transfer means, e.g., transmitters or transceivers, for transmitting real-time video image data from cameras 103 and 104 to the data processing unit 109 wirelessly or via a wired connection. The data processing unit 109 comprises at least one processor, at least one memory containing computer program code for one or more program units, and means, e.g., receivers or transceivers, for receiving real-time video image data from cameras 103 and 104 wirelessly or via a wired connection. There may be multiple processors, e.g., general-purpose processors, graphics processors and / or DSP processors, and / or multiple different memories, e.g., volatile memory for storing data and programs at runtime, and non-volatile memory such as hard disks for permanently storing data and programs. The data processing unit 109 may be any computing device suitable for handling and processing image data, such as a computer. The data processing unit 109 can communicate electronically with cameras 103 and 104 via signal lines. The data processing unit 109 may also include video and audio controllers for generating signals that can be generated for a user using computer accessories. The data processing unit 109 can generate output to the user via output means. The video controller may be connected to a display (not shown). The display may be, for example, a flat panel display, a tablet, a laptop display, or a projector that produces a larger image. The audio controller may be connected to a sound source such as a speaker or earphones. Cameras 103 and 104 may also include acoustic sensors such as microphones.

[0021] The data processing unit 109 is configured to receive real-time video image data from the cameras 103 and 104, and overlay at least one position / area of the view of the static virtual tour including the area corresponding to the area imaged by the cameras 103 and 104 on the above-mentioned real-time video image data. The data processing unit 109 may receive the image data of this static virtual tour from the camera that captured it. The data processing unit 109 may also construct a virtual tour from this image data, or the data processing unit 109 may receive a constructed virtual tour from, for example, another computing device, from memory, or from cloud storage.

[0022] Also, instead of the two cameras 103 and 104 as in this exemplary embodiment, there may be only one camera arranged to capture a real-time video stream, or there may be three or more cameras arranged to capture real-time video streams from different parts of the paper machine environment, for example, 3 to 10 or more cameras.

[0023] FIG. 1b shows, as an example, a first field of view area 107 configured to be overlaid, i.e., superimposed, on the corresponding area of the previously captured static virtual tour of the paper machine environment 100. The pulp supplied on the wire is shown in the first field of view area 107. Although not shown in this example, the real-time video image data of the second field of view area 106 may also be superimposed on the area of the view of the virtual tour corresponding to the area of the second field of view area 106.

[0024] FIG. 1c shows a wet end view 110 of the virtual tour of the paper machine environment 100. This view 110 includes the first field of view area 107, i.e., the same target object 101 as the wet end 101 of the paper machine.

[0025] Figure 1d shows a plan view of a user terminal 111 that presents a static virtual tour view 110 as a view of a mixed-image virtual tour of the papermaking machine environment 100. In this embodiment, the user terminal 111 is a monitor. In this mixed-image virtual tour view 110, a first field of view area 107 is superimposed on the corresponding area of ​​the static virtual tour view 110 so that a view of the mixed-image virtual tour is formed. As can be seen from the figure, the rest of the view 110 is old and static, but the area of ​​the wet end 101 contains real-time image data that is updated in real time. At this time, the morphology of the pulp on the wire is different from when the virtual tour was captured.

[0026] Figures 2a to 2f illustrate the principle of forming a mixed-image virtual tour of a house 200 according to an exemplary embodiment. The mixed-image virtual tour of house 200 is formed from a previously captured static virtual tour of house 200 and two real-time video streams. Figure 2a shows three cameras 201a, 201b, and 201c, each equipped with an image sensor. Cameras 201a, 201b, and 201c are configured to capture real-time image data, which in this example is a real-time video stream. The first camera 201a is positioned to capture a real-time video stream from a first target object, which is a switch on a coffee maker 203, in this embodiment. The second camera 201b is positioned to capture a real-time video stream from a second target object, which is a bathroom faucet 204, in this embodiment. The third camera 201c is positioned to capture a real-time video stream from a third target object, which is a birdcage 205, in this embodiment. The fields of view 203a, 204a, and 205a of the first, second, and third cameras 201a, 201b, and 201c are correspondingly indicated by rectangles. Each field of view 203a, 204a, and 205a correspondingly includes target objects 203, 204, and 205. In this embodiment, a microphone 201d is also present. The microphone 201d is connected to the birdcage 20 5 Birdcage 20 5It is placed in the vicinity of [the object].

[0027] Cameras 201a, 201b, and 201c are provided with data transfer means, such as a transmitter or transceiver, for transmitting real-time video image data from cameras 201a, 201b, and 201c to the data processing unit 202 wirelessly or via a wired connection. The data processing unit 202 may be, for example, a server. The data processing unit 202 includes at least one processor, at least one memory containing computer program code for one or more program units, and means for receiving real-time video image data from cameras 201a, 201b, and 201c wirelessly or via a wired connection, and for receiving audio data from microphone 201d, such as a receiver or transceiver. There may be multiple processors, such as a general-purpose processor, a graphics processor, and / or a DSP processor, and / or multiple different memories, such as volatile memory for storing data and programs at runtime, and non-volatile memory such as a hard disk for permanently storing data and programs. The data processing unit 202 may be any computing device suitable for handling / processing image and audio data, such as a computer. The data processing unit 202 can electronically communicate with cameras 201a, 201b, 201c and microphone 201d via signal lines. The data processing unit 202 may also include a video controller and an audio controller for generating signals that can be generated for the user using computer accessories. The data processing unit 202 can generate output to the user via its output means. The video controller may be connected to a display (not shown). The display may be, for example, a flat panel display, a laptop display, or a projector that produces a larger image. The audio controller may be connected to a sound source such as a speaker or earphones. Physically, the data processing unit 202 may be located inside or outside the house 200. It is also possible that cameras 201a, 201b, 201c and microphone 201d transfer their data to cloud storage, from which the data processing unit 202 retrieves the data.

[0028] The data processing unit 202 is configured to receive real-time video image data from cameras 201a, 201b, and 201c, receive audio signals from microphone 201d or cloud storage, add the aforementioned real-time video image data to a preferred view of a static virtual tour that includes the corresponding areas captured by cameras 201a, 201b, and 201c, and add audio signals to a preferred position in the static virtual tour view corresponding to the position where microphone 201d is located, or to a position in the static virtual tour view where the audio signals captured by microphone 201d would be audible in real-world conditions.

[0029] In this case as well, the number of cameras and / or microphones capturing real-time data may vary. Furthermore, the number of cameras and / or microphones to which data is added to the mixed-image virtual tour can be changed as needed or desired. Cameras 201a, 201b, and 201c may also include acoustic sensors such as microphones.

[0030] Figure 2b shows, as an example, a first field of view 203a of a coffee maker switch configured to be superimposed on a corresponding area of ​​a kitchen view 210 of a previously imaged static virtual tour of house 200. From the first field of view 203a, it can be seen that the switch is currently off. Second field of view 203a of a bathroom faucet 204 4 Although not shown in Figure 2b, b can also be configured to be superimposed on a corresponding area of ​​the bathroom view in a virtual tour of house 200. Figure 2b also shows a third field of view area 205a of the birdcage 205 configured to be superimposed on a corresponding area of ​​the living room view in a static virtual tour of house 200. From the third field of view area 205a, the current status of the bird inside the cage 205 can be seen.

[0031] Figure 2c shows a kitchen view 210 of a virtual tour of house 200. This view 210 shows a coffee maker switch configured to be replaced by a first target object, namely a first field of view area 203a, and the switch is in the "on" position.

[0032] Figure 2d shows the kitchen view 210 on a user terminal 211, a handheld device, such as a tablet, which presents the kitchen view 210 of a mixed-image virtual tour. In this view 210, the first field of view area 203a, i.e., the coffee maker switch area, is superimposed, or overlaid, on the corresponding area of ​​the kitchen view 210. As can be seen from the figure, the rest of the view 210 is old and static, but the coffee maker switch area contains live data, is updated in real time, and is in a different state than when the kitchen view 210 of the virtual tour was captured.

[0033] Figure 2e shows a living room view 212 of a virtual tour of house 200. This view 212 shows a birdcage 205 configured to be replaced by a third target object, namely a third field of view region 205a, in the mixed-image virtual tour.

[0034] Figure 2f shows the living room view 212 on the user terminal 211. In this view 212, the third field of view area 205a, i.e., the birdcage area, is the living room view 21 of the virtual tour. 2 It is superimposed on the corresponding area. As can be seen from the diagram, in this extended view, the living room view 21 of the virtual tour is shown. 2 The bird is positioned differently than when the image was taken.

[0035] The audio signal captured by microphone 201d may also be configured to be played back when the user enters the living room in the mixed-image virtual tour. The audio signal captured by microphone 201d may also be configured to be played back when the user exceeds a certain predetermined distance from the birdcage when entering the living room in the mixed-image virtual tour, and to be stopped when the user exceeds a second certain predetermined distance from the birdcage when leaving the living room in the mixed-image virtual tour.

[0036] To make it easier for users to detect which parts of a mixed-image virtual tour contain real-time data and which do not, it is possible to indicate replaced areas within one or more views of the mixed-image virtual tour—that is, areas where real-time image data is superimposed on a view of the static virtual tour. This indication can be done, for example, by surrounding the replaced areas with color or illumination (which may or may not be flashing), by using brighter illumination in the replaced areas, or by slightly blurring the area around the real-time areas.

[0037] Figures 3a to 3d illustrate an exemplary embodiment of the present invention, where these two different views of a mixed-image virtual tour of a papermaking machine environment are shown, with overlaid image data, i.e., a video stream, captured at two different points in time. These Figures 3a to 3d disclose previously captured static virtual tour views 110, 310 of the papermaking machine environment, and a video stream overlaid on specific areas of these views.

[0038] Figure 3a shows a monitoring room view of a mixed-image virtual tour, illustrating two devices displaying measurement results. In this image, a real-time video stream from a target object, the same region as device 102 displaying measurement results in Figure 1a, is captured by a video camera. The camera's field of view 106 is then overlaid again onto the previously captured static virtual tour view 310, and onto the corresponding location in the monitoring room view 310 of the previously captured static virtual tour, where this region of device 102 displaying measurement results is located within the actual papermaking machine environment. The time point in this mixed-image virtual tour view is indicated by a time indicator 320 on the view, although this time indicator 320 could be displayed in some other way. As can be seen from the figure, the time point at which the shown overlaid image data was captured corresponds to the current moment, while the data superimposed on the previously captured static virtual tour view 310 is real-time image data.

[0039] Figure 3b shows a wet-end view of a mixed-image virtual tour, showing the pulp being fed to the wire. In this image, a real-time video stream from the target object, which is the same wet-end region as in Figure 1a, is captured by a second video camera. The camera's field of view 107 is the view of the previously captured static virtual tour. 3 Above, this area of ​​the wet end is located within the actual papermaking machine environment, and is a previously captured static virtual tour wet end view. 1 It is overlaid again in the corresponding location within 10. The point in time when the image data for this mixed-image virtual tour view was captured by the second video camera is again indicated by the time indicator 320 above the view. As can be seen from the figure, this point in time also corresponds to the current moment, and the data overlaid on the previously captured static virtual tour view 110 is real-time image data.

[0040] However, according to an example of the present invention, it is possible to reverse the time of a mixed-image virtual tour, that is, to change the time of the overlaid image data displayed in the view of the mixed-image virtual tour to an earlier time than the current time. And that earlier time may be the same in all views of the mixed-image virtual tour. These earlier time views of the mixed-image virtual tour then show the previously captured static virtual tour view and the view of the image data captured at the same earlier time. This type of presentation is not possible in the case of a normal virtual tour because the imaging device that captures or will capture image data for the virtual tour cannot capture data such that the image acquisition time is exactly the same in all views. And, as already mentioned above, this ability to show simultaneously captured information in all views of a mixed-image virtual tour may be necessary in determining the causal relationship of damage and avoiding the consequences of damage.

[0041] Figures 3c and 3d show the monitoring room view and wet end view of the same mixed-image virtual tour as Figures 3a and 3b, respectively, but at an earlier point in time, i.e., one hour earlier. Thus, Figure 3c shows the monitoring room view and shows two devices displaying measurement results. The camera's field of view 106 is overlaid again on the previously captured static virtual tour view 310. The point in time for capturing the overlaid image data of this monitoring room view of the mixed-image virtual tour is indicated by the time indicator 320. The image data overlaid on the previously captured static virtual tour view 310 corresponds to the video image data captured at that indicated point in time in the area of ​​device 102 that displays the measurement results of eight measuring devices that measure different data of the machine environment or the papermaking machine itself. Figure 3d shows the wet end view. The camera's field of view 107 is overlaid again on the previously captured static virtual tour view 3It is overlaid again on 10. The point in time of this wet end view of the mixed image virtual tour is again indicated by the time indicator 320. The image data overlaid on the previously captured static virtual tour view 110 corresponds to the video image data captured at the indicated point in time of the wet end of the papermaking machine and indicated by the time indicator 320.

[0042] As can be seen from the figure, there is a difference between the measurement results in field of view 106 of the monitoring room view and the measurement results in the pulp in field of view 107 of the wet end view between different time points, but the remaining views of the mixed image virtual tour remain the same.

[0043] The time point of the displayed view can be changed, for example, by moving the square portion on the horizontal axis of the time indicator 320, or by any other method. Note that, at the selected time point, it is predetermined that all data shown as overlays on the views of the mixed-image virtual tour was captured at that same selected time point. In other words, if the user chooses to view a mixed-image virtual tour captured 10 minutes ago, the images overlaid on all views will be captured 10 minutes ago, while in the real-time view, all data, including the images or image streams overlaid on all views, will be real-time image data. However, it is also possible to select a time point such that at least one view shows image data captured at a different time point compared to the image data overlaid on other views of the mixed-image virtual tour. The acoustic signal data being played back is also captured at the selected time point.

[0044] Figure 4 shows a living room view 212 on the same user terminal 211 as in Figure 2f. In this view 212, the field of view area 205a of the birdcage of real-time video image data is overlaid on the corresponding area of ​​the previously captured virtual tour living room view 210 so that an extended virtual tour, i.e., a mixed-image virtual tour, is formed by the data processing unit. However, in Figure 4, in addition to the real-time image data, additional information is also overlaid on the living room view 210. Additional information that may be shown in the view of the mixed-image virtual tour may be, for example, any information about the environment of the view or other parts of the virtual tour. In this figure, the first additional information 410 This relates to the birdcage shown in the field of view area 205a of the birdcage. First additional information 410 This indicates the temperature around the birdcage 205 and the water level in the water basin inside the birdcage 205. Second additional information 411 This relates to a lighting device, namely a ceiling-mounted lamp, and indicates that the lamp is on.

[0045] The additional information shown in Figure 4 is merely an example. The additional information may also relate to, for example, sensor readings or measurement results shown in the field of view, or other information related to those measurement results. Sensor readings or measurement results may be captured by sensors in the location, e.g., machine sensors and / or sensors of other connected devices. In some cases, the additional information indicates captured sensor data relating to structures, devices, or other objects in the location. The additional information may be shown, for example, next to, below, or above the field of view, or anywhere else on the view of the mixed-image virtual tour. The additional information may be captured simultaneously with the image data overlaid on the view, or it may be historical data, but the additional information data may also be collected before or after the time when the image data overlaid on the view of the mixed-image virtual tour is captured, for example, one minute prior. Alternatively, any other period may be used instead of one minute. The period may depend, for example, on the additional information, or the user may configure the period. Furthermore, the data processing unit can analyze the additional information by comparing it with a baseline value. If a deviation is detected, for example, if the value of the additional information falls below or exceeds a predetermined baseline value, the additional information is overlaid on the view; otherwise, it is not overlaid.

[0046] The data processing unit can acquire additional information about the location or environment, for example, from the aforementioned sensors or measuring devices, and overlay this additional information onto the mixed-image virtual tour data.

[0047] Figure 5 shows a kitchen view 501 of a mixed-image virtual tour on a user terminal 500. In this view 501, real-time video data from the field of view 503 of a first real-time video camera is overlaid on the kitchen view 501. The field of view 503 substantially corresponds to the coffee maker switch area. Figure 5 also shows another field of view of a second video camera corresponding to the area 504 in front of the dishwasher. The second video camera is positioned to capture image data from the target object, i.e., the area 504 in front of the dishwasher, and transmit the image data to a data processing unit. The real-time video data from the field of view 504 of the second video camera is displayed on the kitchen view 501 of the mixed-image virtual tour only when the camera's data processing unit or an external data processing unit detects a predetermined deviation of the floor when analyzing the image data captured by the second video camera. In this example, the deviation 502 This is water on the floor, specifically a leak from the dishwasher. In other words, the image data in this field of view 504 is a deviation in the image data captured by the second camera. 502 After detection, it is overlaid on kitchen view 501 of the mixed image virtual tour. This type of image data, which appears when a deviation is detected in the image data, is sometimes referred to as pop-up image data.

[0048] Overlaying the real-time video stream of the second video camera onto the area in front of the dishwasher in the kitchen view 501 of the mixed-image virtual tour may be performed by the data processing unit that constructs the mixed-image virtual tour, or the coffee maker switch area of ​​the kitchen view 501 of the mixed-image virtual tour may be overlaid on the real-time video stream of the first video camera, although the real-time video stream of this switch is always overlaid on the kitchen view. The data processing unit that constructs the mixed-image virtual tour may be a data processing unit that analyzes image data, or another data processing unit, for example, one that acquires already constructed virtual tour data, or the same one that constructs the virtual tour. The pop-up real-time video stream of the second video camera may be displayed in the kitchen view 501 until the user indicates to the data processing unit that a deviation has been observed, until the data processing unit analyzing the image data of the second video camera detects that the deviation is no longer detectable, or for a predetermined period of time. The pop-up real-time video stream from the second video camera is displayed again on the kitchen view 504 of the mixed-image virtual tour if a new deviation is detected when analyzing the image data from the second video camera. The data processing unit may be further configured to notify the user that image data related to a given deviation is being displayed in the view of the mixed-image virtual tour, even before the user views the mixed-image virtual tour.

[0049] The data processing unit comprises at least one processor, at least one memory containing computer program code for one or more program units, and means for receiving image data from at least a second video camera via a wireless or wired connection, e.g., a receiver or transceiver. There may be multiple processors, e.g., general-purpose processors, and graphics processors and DSP processors, and / or multiple different memories, e.g., volatile memory for storing data and programs at runtime, and non-volatile memory such as a hard disk for permanently storing data and programs. The data processing unit may be any computing device suitable for processing image data, such as a computer. The data processing unit communicates electronically with at least a second video camera via signal lines. The second video camera may also include a video controller and an audio controller for generating signals that can be generated for a user using computer accessories. The second video camera may generate output to the user via output means. The video controller may be connected to a display. The display may be, for example, a flat panel display or a projector for generating a larger image. The audio controller may be connected to a sound source such as a speaker or earphone. The second video camera may also include an acoustic sensor such as a microphone.

[0050] The rest of the kitchen view 501 is old and static, but the coffee maker switch area 503 and the area in front of the dishwasher 504 include live data received from the camera when deviations are detected.

[0051] The user can also select a point in time other than real time, at which point in time the image data to be overlaid on the kitchen view of the mixed-image virtual tour is captured. In this way, the user can see how or when the deviation progressed, or when the image of the deviation appeared in the kitchen view 501 of the mixed-image virtual tour.

[0052] Figure 6 shows an example of a plan view of a user terminal 611 presenting a wet-end view 610 of a mixed-image virtual tour of a papermaking machine environment. In this example, the wet-end region of the papermaking machine is the target object. The papermaking machine environment includes a real-time video camera that captures a real-time video stream of the wet-end region. In this wet-end view 610 of the mixed-image virtual tour, the real-time video stream data from the field of view 607 of the real-time video camera is superimposed on the corresponding wet-end region of the wet-end view 610 of the static virtual tour to form the mixed-image virtual tour. However, in this example, there are also acoustic sensors such as microphones in the wet-end region of the papermaking machine.

[0053] Acoustic sensors detect ambient sounds. 600 It is positioned to record the ambient sound. 600 The acoustic sensor is positioned to be transmitted as acoustic data to a data processing unit configured to construct a mixed virtual tour for analysis. The acoustic sensor includes means for transmitting the acoustic data to the data processing unit. The means for transmitting the acoustic data may be, for example, a transceiver or transmitter configured to transmit the acoustic data from the sensor to the data processing unit wirelessly or via a wired connection.

[0054] A data processing unit configured to construct a mixed virtual tour comprises at least one processor, at least one memory containing computer program code for one or more program units and pre-stored audio profiles for different operating states of a papermaking machine, and means for receiving audio data from an acoustic sensor wirelessly or via a wired connection. The pre-stored audio profiles may include, for example, audio profiles for various operating states of the papermaking machine, relating to the normal and / or abnormal operating states of various machines of the process machine, and / or audio profiles for one of the machines. The pre-stored audio profiles may include, for example, normal state, startup state, stopped state, idle state, or malfunction state. The means for receiving audio data may be, for example, a receiver or a transceiver. An example of a data processing unit is defined more precisely above in relation to the aforementioned figure.

[0055] Therefore, the data processing unit is configured to receive acoustic data from an acoustic sensor that records ambient acoustic signals, convert the acoustic signals into electroacoustic data, and transmit the acoustic data to the data processing unit. The data processing unit is configured to determine the acoustic spectrum of the received acoustic data and to form an audio profile of the acoustic data.

[0056] After forming an audio profile, the data processing unit analyzes the acoustic data by comparing the formed audio profile with pre-stored audio profiles of various operating states. Based on this comparison, the data processing unit can determine the operating state of the machine around the acoustic sensor. If it determines that the formed audio profile is equivalent to an operating state indicating a machine malfunction, the data processing unit positions the acoustic data to play back within the wet-end view of a mixed-image virtual tour corresponding to the actual location of the acoustic sensor within the papermaking machine environment where it is positioned. Thus, the mixed-image virtual tour is further enhanced by audio signal data in addition to real-time video stream data when the data processing unit analyzes the acoustic sensor data and defines, i.e., determines, that there may be a malfunction in the papermaking machine environment. The data processing unit can also inform the user about the malfunction. A malfunction is an example of a deviation.

[0057] Figure 7 shows an example of a living room view 710 of a mixed-image virtual tour on a user terminal 711. A first real-time video camera is positioned to capture image data of a first target object, which in this example is a birdcage. The captured image data of the birdcage is sent to a data processing unit that constructs a mixed-image virtual tour. In this view 710, the real-time video stream of the field of view 701 of the first real-time video camera is overlaid on the corresponding birdcage area of ​​the living room view 710 of the virtual tour, and as a result, the real-time video stream of the real-time video camera is displayed on the living room view 710 of the mixed-image virtual tour.

[0058] In a real living room, a second real-time video camera is also positioned to capture image data. The second real-time video camera captures image data of a second target object, which in this example is a couch. The image data captured by the second video camera is sent to a data processing unit that constructs a mixed-image virtual tour. This real-time video stream from the second field of view 702 of the second real-time video camera is displayed on the living room view 710 of the mixed-image virtual tour only when the data processing unit detects the presence of a person or something else that does not belong on the couch, i.e., a deviation is detected because the couch should not be occupied, and the deviation in this example is something present on the couch. If a deviation is detected, the real-time image data from the second field of view 702 of the second video camera is overlaid on the living room view 710 of the mixed-image virtual tour, more precisely, overlaid on the couch as pop-up image data.

[0059] The rest of the living room view 710 is old and static, but the birdcage and couch areas include live image data received from cameras positioned to capture image data within the real living room environment when deviations are detected.

[0060] Deviations that may trigger the display of image data from one or more cameras on top of one or more views of a mixed-image virtual tour are not limited to the examples above, and deviations may be any changes, anomalies, or defects that can be detected when analyzing the image data.

[0061] Figure 8 shows a flowchart of a computer implementation method 800 for extending a virtual tour to a mixed-image virtual tour when a deviation is detected. In step 810, previously captured virtual tour data for a certain location is acquired; in step 820, real-time image data of a target object located within the camera's field of view and at the same location where the virtual tour data was acquired is received; in step 830, the real-time image data is analyzed for deviation detection; and in step 840, if a deviation is detected, the real-time image data is overlaid on the area of ​​the virtual tour view corresponding to the field of view to form mixed-image virtual tour data.

[0062] The audio sensor according to the present invention, which acquires, i.e., records, real-time acoustic data, comprises at least a circuit and electronic equipment for recording ambient acoustic sound, i.e., data, converting the acoustic data into an electrical signal, and transmitting the acoustic data. The camera according to the present invention comprises at least a circuit and electronic equipment for acquiring real-time images or video streams and transmitting image data.

[0063] It should be noted that the number of cameras positioned to provide real-time image data of target objects overlaid on the view of the mixed-image virtual tour may vary, regardless of whether deviation detection is enabled. There may be one camera or multiple cameras. The number of cameras positioned so that image data is overlaid on the view of the mixed-image virtual tour may also vary. There may be multiple cameras capturing image data of target objects, but it is possible that image data from only one, two, or other fields of view may be displayed on the view. Furthermore, the displayed image data or additional information within the field of view may be user-modifiable, or may depend on, for example, time of day, season, user profile, outdoor temperature, indoor temperature, humidity, etc., resulting in only the most relevant additional information being overlaid on the view.

[0064] It is clear that the present invention is not limited to the embodiments presented above, but can be modified within the scope of the appended claims. This application provides the invention in the following embodiments. (Aspect 1) A computer implementation method, This involves obtaining previously captured image data for a virtual tour of a certain location, and Receiving real-time image data of a target object located within the camera's field of view, wherein the target object is located at the same location where the image data was captured, To detect deviations, the real-time image data is analyzed, The computer implementation method, comprising: overlaying the real-time image data of the target object onto a corresponding region of the target object in a virtual tour view to form mixed image virtual tour data only when the deviation is detected; and showing the virtual tour view of the target object when no deviation is detected. (Aspect 2) The computer implementation method according to Embodiment 1, wherein the image is static image data of the actual environment of the location. (Aspect 3) The computer implementation method according to embodiment 1 or 2, wherein the previously acquired image data includes images acquired during walking through the location, the previously acquired image data is obtained from a camera, memory, or cloud storage, and the method further comprises constructing the image data as a virtual tour by a data processing unit. (Aspect 4) The computer implementation method according to embodiment 1 or 2, wherein the previously acquired image data is an already constructed virtual tour. (Appendix 5) The aforementioned method, Receiving real-time acoustic data captured by an audio sensor within the aforementioned location, To detect malfunctions in the machine, the received acoustic data is analyzed, A computer implementation method according to any one of embodiments 1 to 4, further comprising: in order to further extend the mixed-image virtual tour with real-time acoustic data, playing the acoustic data in the area of ​​the mixed-image virtual tour view corresponding to the location of the sound sensor or its surroundings at the location when a malfunction is detected. (Aspect 6) The aforementioned method, To obtain additional information regarding the aforementioned location, To detect deviations, the aforementioned additional information is analyzed, A computer implementation method according to any one of embodiments 1 to 5, further comprising overlaying the additional information on the mixed image virtual tour view if a deviation is detected. (Aspect 7) The computer implementation method according to any one of embodiments 1 to 6, wherein the real-time image data is a real-time video stream or a frequently captured real-time image. (Pattern 8) The aforementioned method, The computer implementation method according to any one of embodiments 1 to 7, further comprising capturing real-time image data configured to be overlaid on the area of ​​the virtual tour view by two or more cameras. (Aspect 9) The computer implementation method according to any one of embodiments 1 to 8, wherein the point in time of viewing the mixed image virtual tour can be changed to a previous point in time. (Aspect 10) A camera system comprising at least one camera and a data processing unit, wherein the data processing unit is configured to acquire previously captured image data of a virtual tour of a location, the at least one camera is configured to acquire real-time image data of a target object located within the field of view of the at least one camera, the target object being located at the same location where the image data was acquired, The camera system is further configured such that the data processing unit analyzes the real-time image data to detect deviations, and only if a deviation is detected, overlays the real-time image data of the target object onto a corresponding region of a virtual tour view to form a mixed-image virtual tour, and if no deviation is detected, the virtual tour view of the target object is shown. (Aspect 11) The system according to embodiment 10, further comprising a sound sensor configured to acquire acoustic data at the location, transmit the acquired acoustic data to an analysis data processing unit, and, if a malfunction is detected, to play back real-time acoustic data in the area of ​​the mixed image virtual tour view corresponding to the location of the sound sensor in the real environment or its surroundings. (Aspect 12) The system according to embodiment 10 or 11, wherein the data processing unit is further configured to acquire additional information relating to the location, analyze the additional information, and, if the data processing unit detects a deviation, to overlay the additional information on the mixed image virtual tour view. (Aspect 13) The system is the system according to any one of embodiments 10 to 12, comprising two or more cameras. (Aspect 14) The system according to any one of embodiments 10 to 13, wherein the point in time of viewing the mixed image virtual tour can be changed to a previous point in time. (Aspect 15) A computer program product stored on a computer-readable medium and executable on a computing device, wherein the computer program product includes instructions for performing the method described in any one of embodiments 1 to 9.

Claims

1. A computer implementation method, This involves obtaining previously captured image data for a virtual tour of a real-world location, If the previously acquired image data is not already a constructed virtual tour, the data processing unit constructs the previously acquired image data as a virtual tour. Receiving real-time image data of target objects (101, 102) located within the field of view (106, 107) of cameras (103, 104) at camera positions in the aforementioned real-world environment, The real-time image data is analyzed to detect a predetermined deviation from the image data, Only when the predetermined deviation is detected, the real-time image data is overlaid on the position in the virtual tour view corresponding to the camera position in the real-world location in order to form mixed image virtual tour data; if the predetermined deviation is not detected, the virtual tour view of the position corresponding to the camera position in the real-world location is shown. Receiving real-time acoustic data captured by an audio sensor located at the audio sensor position in the aforementioned real-world environment, This includes analyzing the received acoustic data to detect malfunctions of the machine by comparing the received acoustic data with at least one pre-stored audio profile of various operating states, The aforementioned method, The computer implementation method further includes, in order to further extend the mixed-image virtual tour with real-time acoustic data, when a malfunction is detected, playing the acoustic data when a user navigating the mixed-image virtual tour is at a location in the real environment corresponding to the location of the sound sensor or at a specific predetermined distance from the location in the real environment corresponding to the location of the sound sensor.

2. The computer implementation method according to claim 1, wherein the previously acquired image data is static image data of the locations in the real environment of the virtual tour.

3. The computer implementation method according to claim 1 or 2, wherein the previously acquired image data includes images taken by a person walking through the location in the real environment, and the previously acquired image data is obtained from a camera, memory, or cloud storage.

4. The aforementioned method, To obtain additional information (410, 411) regarding the location of the aforementioned real environment, The additional information (410, 411) is analyzed in order to detect deviations, A computer implementation method according to any one of claims 1 to 3, further comprising overlaying the additional information (410, 411) on a mixed image virtual tour view if a deviation is detected.

5. The computer implementation method according to any one of claims 1 to 4, wherein the real-time image data is a real-time video stream from a location in the real environment or a frequently captured real-time image.

6. The computer implementation method according to any one of claims 1 to 5, further comprising acquiring real-time image data by two or more cameras (103, 104).

7. The computer implementation method according to any one of claims 1 to 6, wherein the point in time of viewing the mixed image virtual tour can be changed to a previous point in time.

8. A camera system comprising at least one camera and a data processing unit (109), wherein the data processing unit is configured to acquire previously captured image data of a virtual tour of a real-world location, and if the previously captured image data is not already constructed as a virtual tour, to construct the previously captured image data as a virtual tour by the data processing unit, and the at least one camera (103, 104) is configured to capture real-time image data of target objects (101, 102) located within the field of view (106, 107) of the at least one camera (103, 104) at the camera position of the real-world location, and to transmit the previously captured image data to the data processing unit. The data processing unit (109) is further configured to analyze the real-time image data to detect a predetermined deviation, and only if the predetermined deviation is detected, to overlay the real-time image data of the target objects (101, 102) at the position of a virtual tour view corresponding to the camera position of the real-world location to form a mixed-image virtual tour; and if the predetermined deviation is not detected, to show the virtual tour view of the target objects (101, 102); the camera system further comprises a sound sensor configured to acquire acoustic data at the sound sensor position of the real-world location and transmit the acquired acoustic data to the data processing unit for analysis; and the real-time acoustic data is configured to be played back when a malfunction is detected by comparing the received acoustic data with at least one pre-stored sound profile of various operating states, in order to further extend the mixed-image virtual tour with the real-time acoustic data, if the received acoustic data is at the position corresponding to the sound sensor position of the real-world location or at a specific predetermined distance from the position corresponding to the sound sensor position of the real-world location.

9. The camera system according to claim 8, wherein the data processing unit is further configured to acquire additional information (410, 411) relating to the location of the real environment, which is information received from one or more sensors or measuring devices, analyze the additional information, and overlay the additional information (410, 411) on a mixed image virtual tour view when the data processing unit (109) detects a predetermined deviation.

10. The camera system according to claim 8 or 9, comprising two or more cameras (103, 104).

11. The camera system according to any one of claims 8 to 10, wherein the point in time of the view of the mixed image virtual tour can be changed to a previous point in time.

12. A computer program comprising instructions for carrying out the method described in any one of claims 1 to 7.

Images