Remote monitoring system, remote monitoring method, and control program for remote monitoring system

Abstract

To provide a remote monitoring system capable of zooming and displaying a region of interest with high definition and confirming a peripheral region other than the zoomed region.SOLUTION: The system includes a plurality of cameras, an edge server, and at least one terminal. The plurality of cameras are installed so as to face different photographing directions and can acquire omnidirectional photographed video images, and each camera includes a lens, a zoom control unit, and a first communication unit. The edge server includes a second communicator, a camera controller configured to control the zoom controller, and a synthesis processor configured to synthesize at least two captured videos of the captured videos to generate a synthesized video. The terminal includes a third communication unit, a display unit, and an attention designation unit that designates an attention region in a displayed video. Then, the edge server provides, to the terminal, an omnidirectional synthesized video image generated by synthesizing a zoom video image of a camera whose image pickup region is a region corresponding to the region of interest designated by the interest designation unit and a picked up video image of a camera other than the zoomed camera.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] An embodiment of the present invention relates to a remote monitoring system, a remote monitoring method, and a control program for the remote monitoring system. [Background technology]

[0002] Conventionally, remote monitoring systems have been put into practical use to monitor the operating status of equipment installed in remote locations or the status of work at remote locations from an office or other location far from the monitored object. To achieve such remote monitoring, cameras are often installed at the desired locations, such as the equipment or work site, and the images are transmitted to a remote site, such as an office, for remote viewing. In such cases, 360-degree cameras capable of capturing a wider range of situations are sometimes used. This allows for omnidirectional capture, allowing a remote user (such as a supervisor) to grasp the surrounding environment of the site with a single camera. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-142934 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional 360-degree cameras use the number of pixels inherent in the lens to expand the image quality to 360 degrees, resulting in coarse image quality compared to regular flat cameras, which can make discrimination (judgment) difficult due to insufficient resolution. Furthermore, when performing inspection work, such as recognizing text from a remote site, there is a need to zoom in on only the area of ​​interest (area of ​​interest) for high-resolution confirmation. However, conventional 360-degree cameras generally create 360-degree images by combining images from multiple lenses within the camera, making it structurally difficult to zoom in on only a specific lens. Furthermore, when users want to view a specific area, they must cut out that specific part from the 360-degree image and enlarge it, further degrading the resolution.

[0005] Therefore, if a remote monitoring system could be provided that could zoom in and display a high-resolution area of ​​interest within the monitored object, while also enabling confirmation of the surrounding area outside the zoomed area, it would be useful as it would improve the accuracy of remote monitoring. [Means for solving the problem]

[0006] The remote monitoring system according to the embodiment includes multiple cameras, an edge server, and at least one terminal device. The edge server is connected to the multiple cameras via a first network, performs processing according to images captured by the multiple cameras, and outputs the images. The terminal device is connected to the edge server via a second network and is capable of displaying at least images provided by the edge server. The multiple cameras are installed facing different shooting directions to capture omnidirectional images, and each camera includes a lens, a zoom control unit that adjusts the zoom of the lens, and a first communication unit that communicates with the edge server. The edge server includes a second communication unit that communicates between the multiple cameras and the terminal device, a camera control unit that independently controls the zoom control units of the cameras, and a synthesis processing unit that synthesizes at least two of the captured images to generate a synthesized image. The terminal device includes a third communication unit that communicates with the edge server, a display unit that displays the images provided by the edge server, and an attention designation unit that designates an area of ​​interest in the image displayed on the display unit. The edge server then provides the terminal device with an omnidirectional composite image generated by combining a zoomed image of a camera whose shooting area is an area corresponding to the attention area specified by the attention specification unit with an image shot by a camera other than the zoomed camera. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is an exemplary schematic explanatory diagram showing the configuration of a remote monitoring system according to an embodiment, and also showing the transmission of an omnidirectional composite video generated based on video images captured by a plurality of cameras. [Figure 2] FIG. 2 is an exemplary schematic explanatory diagram showing the configuration of a remote monitoring system according to an embodiment, and also showing how zoomed video is transmitted from a camera that includes a region of interest in its shooting region. [Figure 3] FIG. 3 is an exemplary schematic explanatory diagram showing the configuration of a remote monitoring system according to an embodiment, and also showing the transmission of an omnidirectional composite image generated based on captured images other than zoomed images. [Figure 4]FIG. 4 is an exemplary schematic explanatory diagram showing the imaging range of one camera of the remote monitoring system according to the embodiment. [Figure 5] FIG. 5 is an exemplary schematic explanatory diagram showing the shooting directions of four cameras of the remote monitoring system according to the embodiment and a case where a plurality of shot videos are combined to generate an omnidirectional composite video. [Figure 6] FIG. 6 is an exemplary schematic explanatory diagram showing a case where an omnidirectional composite image is generated by combining images captured by cameras other than the zoomed camera in the remote monitoring system according to the embodiment. [Figure 7] FIG. 7 is an exemplary schematic explanatory diagram showing the display layout of a zoomed video and an omnidirectional composite video (composite image) in a terminal device of a remote monitoring system according to an embodiment. [Figure 8] FIG. 8 is an exemplary schematic explanatory diagram showing another display layout of a zoomed image and an omnidirectional composite image (composite image) in a terminal device of a remote monitoring system according to an embodiment. [Figure 9] FIG. 9 is an exemplary explanatory diagram illustrating the relationship between video quality according to video content, network slicing resource allocation, and bandwidth guarantee in the remote monitoring system according to the embodiment. [Figure 10] FIG. 10 is an exemplary schematic explanatory diagram illustrating an image of network slicing resource allocation according to video content in the remote monitoring system according to the embodiment. [Figure 11] FIG. 11 is an exemplary flowchart showing the flow of processing in the remote monitoring system according to the embodiment. [Figure 12] FIG. 12 is an exemplary schematic explanatory diagram showing a configuration in which a plurality of terminal devices are present in a remote monitoring system according to an embodiment, and also showing the transmission of a specified zoom image and the transmission of an omnidirectional composite image generated based on images captured by a plurality of cameras. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The configurations of the embodiments described below, as well as the actions and results (effects) brought about by the configurations, are merely examples and are not limited to the following description.

[0009] FIG. 1 is an exemplary schematic explanatory diagram showing the configuration of a remote monitoring system 10 according to an embodiment, and also showing the transmission of an omnidirectional composite video generated based on video images captured by a plurality of cameras 12.

[0010] The remote monitoring system 10 comprises multiple cameras 12, an edge server 14, and a terminal device 16. The multiple cameras 12 and the edge server 14 are connected via a first network NA. The edge server 14 is also connected to the terminal device 16 via a second network NB. The edge server 14 processes the images captured by the multiple cameras 12 and can output images based on the captured images to the terminal device 16. The terminal device 16 can display the images provided by the edge server 14. Note that, as an example, FIG. 1 shows the remote monitoring system 10 including four cameras 12 (first camera 12a to fourth camera 12d) and one terminal device 16; however, the number of cameras 12 may be two or more. Preferably, if the number of cameras 12 is four or more, when generating a 360-degree omnidirectional composite image described below, the capturing area assigned to each camera 12 can be prevented from becoming too large, making it possible to generate an image with less distortion. Furthermore, the number of terminal devices 16 may be multiple.

[0011] The first network NA connecting the multiple cameras 12 (first camera 12a to fourth camera 12d) and the edge server 14 is a network capable of low latency and bandwidth allocation (bandwidth limitation). The first network NA may be a low latency wired network or may be connected via low latency wireless communication such as 5G or local 5G, and the type of network is not limited. The second network NB connecting the edge server 14 and the terminal device 16 can be a network capable of low latency and bandwidth allocation (bandwidth limitation) like the first network NA. The second network NB may be a general network.

[0012] Camera 12 includes lens 18 for capturing images, zoom control unit 20 for controlling the zoom of lens 18, and video communication unit 22 (first communication unit) for transmitting video data of captured images to edge server 14 via first network NA. Note that first camera 12a to fourth camera 12d basically have the same configuration and performance, and are cameras that can be controlled (capture images) independently. In the example shown in FIG. 1, four cameras, first camera 12a to fourth camera 12d, are used, and therefore each camera 12 is positioned with its capture direction shifted by 90 degrees.

[0013] The lens 18 is, for example, an ultra-wide-angle lens, a fisheye lens, or a wide-angle lens, and is a lens that can capture a wider range of images than the omnidirectional angle (360 degrees) divided by the number of cameras 12. The zoom control unit 20 may control the optical zoom of the lens 18 of the camera 12, or may control the digital zoom of the captured video data. The video communication unit 22 communicates with the video communication unit of the edge server 14. Details of the lens 18, the zoom control unit 20, and the video communication unit 22 will be described later.

[0014] The edge server 14 may be configured with computer resources such as a general-purpose PC. The edge server 14 may be configured with, for example, a central processing unit (CPU), read-only memory (ROM), random access memory (RAM), and a storage unit such as a hard disk drive (HDD) or solid-state drive (SSD). The CPU constituting the edge server 14 reads a control program installed and stored in a nonvolatile storage unit such as a ROM or SSD, and realizes various modules in accordance with the control program. For example, modules such as a video communication unit 24 (second communication unit), a digital synthesis processing unit 26, a past video data accumulation unit 28 (video storage unit), a boundary adjustment control unit 30, a network monitoring unit 32, a network control unit 34, and a camera control unit 36 ​​are realized.

[0015] The video communication unit 24 receives video data from the camera 12, performs necessary processing, and transmits the generated omnidirectional composite video to the terminal device 16. Furthermore, when the video communication unit 24 receives zoomed video from the camera 12, it transmits the zoomed video to the terminal device 16 as is.

[0016] The digital synthesis processing unit 26 digitally synthesizes at least two pieces of video data (captured videos) received from the cameras 12 to construct (generate) an omnidirectional synthesized video (360-degree video).

[0017] The past video data storage unit 28 holds for a certain period of time the video data received from the camera 12. When storing video data, the past video data storage unit 28 stores the identification information of the camera 12 that captured the video to be stored, as well as information about the time of shooting, etc.

[0018] The boundary adjustment control unit 30 controls the boundary area of ​​adjacent images to be joined when the digital composition processing unit 26 performs image composition. As will be described later, if some of the cameras 12 capture zoomed images, those images cannot be used to generate the omnidirectional composite image. On the other hand, the cameras 12 use wide-angle lenses 18 to capture images of a range wider than the omnidirectional angle (360 degrees) divided by the number of cameras 12. Therefore, when using images captured by four cameras 12, 90-degree images are cut out from each image to generate the 360-degree omnidirectional composite image. On the other hand, if there is a camera 12 that has taken zoomed images, unusable images will be generated when generating the omnidirectional composite image. Therefore, the boundary adjustment control unit 30 adjusts the boundary area of ​​the images captured by the non-zoom cameras 12 that sandwich the zoom-target camera 12 to generate the omnidirectional composite image. In this case, the cutout range of the images adjacent to the unusable images is expanded, that is, boundary adjustment is performed, to complete the omnidirectional composite image. Details of boundary adjustment will be described later.

[0019] The network monitoring unit 32 monitors, for example, the communication load of the first network NA. The network control unit 34 performs bandwidth control by allocating resources to the first network NA. The network monitoring unit 32 and the network control unit 34 will be described in detail later.

[0020] The camera control unit 36 ​​acquires viewpoint information (attention information) indicating the area of ​​interest from the terminal device 16, and performs zoom control for the camera 12 and parameter adjustment for the boundary adjustment control unit 30. The camera control unit 36 ​​can independently control the zoom control unit 20 of each camera 12.

[0021] In addition, at least one of the modules such as the above-mentioned video communication unit 24, digital synthesis processing unit 26, past video data storage unit 28, boundary adjustment control unit 30, network monitoring unit 32, network control unit 34, and camera control unit 36 ​​may be configured in hardware.

[0022] The terminal device 16 includes a video communication unit (third communication unit) , a display unit 40, and an attention designation unit .

[0023] The terminal device 16 receives data of the omnidirectional composite video (360-degree video) and data of the zoom video from the edge server 14.

[0024] The display unit 40 performs display based on the received video data. In addition to the video data received from the edge server 14, the display unit 40 can display operation instructions for various processes, operation details, processing results, etc. Furthermore, the display unit 40 may be covered with a transparent operation unit such as a touch panel and integrated with an input device.

[0025] The attention designation unit 42 designates, for example, an image area that a user (monitor, etc.) wants to focus on, such as an important monitoring (inspection) area or an area where a malfunction is suspected, as an attention area, and transmits attention information indicating the attention area to the edge server 14. For example, if the terminal device 16 is a VR goggle, the attention area can be designated by well-known gaze tracking. Also, if the terminal device 16 is a smartphone or a tablet device, the attention area can be designated by performing a touch operation on the display unit 40. Also, if the terminal device 16 is configured as a general PC, etc., the attention area can be designated by inputting a position of attention on the display unit 40 with an input device such as a mouse or a touch pen.

[0026] FIG. 1 shows an example of a normal transmission path when no zoomed image is requested, that is, the transmission of an omnidirectional composite image generated based on images captured by a plurality of cameras 12.

[0027] The first camera 12a to the fourth camera 12d capture images through the lenses 18, and the image data is sequentially transmitted from the image communication unit 22 (first communication unit) to the image communication unit 24 (second image communication unit) of the edge server 14 via the network NA. The image communication unit 24 of the edge server 14 provides the images received from the first camera 12a to the fourth camera 12d to the digital synthesis processing unit 26. When generating a omnidirectional composite image, the digital synthesis processing unit 26 acquires parameters indicating the boundary area of ​​adjacent images to be joined from the boundary adjustment control unit 30. In the example shown in FIG. 1, the attention designation unit 42 of the terminal device 16 has not designated an area of ​​interest. In other words, since no instruction has been issued to the camera 12 to zoom in, all of the images captured by the first camera 12a to the fourth camera 12d are usable. Therefore, the boundary adjustment control unit 30 instructs the images captured by the first camera 12a to the fourth camera 12d to be cropped within a 90-degree range, 45 degrees to the left and 45 degrees to the right of the shooting direction. The digital synthesis processing unit 26 synthesizes the images cut out at 90 degrees each to generate (construct) a 360-degree omnidirectional synthesized image.

[0028] As mentioned above, when a command to zoom a camera 12 is issued, the image captured by the camera 12 that performed the zoom cannot be used to generate the omnidirectional composite image. In this case, the image is synthesized by adjusting the boundaries during image synthesis, but some parts of the image may be missing. To secure images to compensate for these missing parts, the past image data storage unit 28 of the edge server 14 temporarily stores the images received from each camera 12 as past images, associating them with the camera 12's identification information, shooting time information, etc. The past image data storage unit 28 stores the images for a certain period of time and updates them with images provided by the same camera 12 after a certain period of time has passed. In this case, depending on the timing of the image capture, the image may be unsuitable for missing part compensation due to unwanted objects or light shining in. Therefore, past images captured at multiple times may be stored for a certain period of time.

[0029] Then, the video communication unit 24 transmits the generated omnidirectional composite video to the terminal device 16 via the second network NB. The terminal device 16 displays on the display unit 40 an image based on the video received by the video communication unit 38 (third communication unit).

[0030] The network monitoring unit 32 of the edge server 14 monitors the communication load of the first network NA when each camera 12 transmits video data captured by the camera 12 to the edge server 14, and instructs the network control unit 34 to control the network bandwidth if the bandwidth usage exceeds a predetermined reference value. The network control unit 34 determines the priority of the video data based on the video from each camera 12 and allocates network bandwidth resources according to the priority. For example, if zoomed video is captured, it can be assumed that high-quality (high-resolution) video is requested. In other words, zoomed video is a high-priority video, and a high allocation of network bandwidth resources is made. Furthermore, if there is no difference in the priority of the video transmitted from each camera 12, for example, if there is no request for zoomed video, network bandwidth resources are allocated equally.

[0031] 2 is an exemplary schematic diagram illustrating the configuration of the remote monitoring system 10, and also illustrating the transmission of zoomed video from a camera 12 that includes a region of interest in its shooting region. Fig. 2 shows a case where zoomed video is requested from the first camera 12a, and illustrates an example of the communication path of the zoomed first camera 12a.

[0032] When the attention specifying unit 42 of the terminal device 16 operated by a user (such as a supervisor) located remotely from the camera 12 has an eye-tracking function, the user adjusts the viewpoint to a desired location while viewing the video on the display unit 40 of the terminal device 16. In the example of FIG. 2, it is assumed that the user is focusing on the area captured by the first camera 12a in the omnidirectional composite video provided by the edge server 14 and displayed on the display unit 40 as shown in FIG. 1. As a result, the attention specifying unit 42 of the terminal device 16 provides the acquired position information of the viewpoint as attention information (viewpoint information) to the camera control unit 36 ​​of the edge server 14. The camera control unit 36, having acquired the user's attention information, controls the zoom control unit 20 of the first camera 12a to zoom the lens 18 to match the position indicated by the attention information. Furthermore, the camera control unit 36 ​​of the first camera 12a controls the video communication unit 22 to transmit the video at a lower compression rate or transmit the video uncompressed in order to transmit the video at a high resolution. At this time, the video communication unit 22 of the first camera 12a performs communication conversion to transmit the zoomed image captured by the lens 18 of the first camera 12a to the edge server 14, and transmits the image to the edge server 14 with flag information attached indicating that the image is a zoomed image and is not subject to digital synthesis.

[0033] The video communication unit 24 of the edge server 14 transmits the received zoomed video to the terminal device 16 as is, since it is not subject to digital synthesis. Then, the video communication unit 38 of the terminal device 16 displays the received zoomed video on the display unit 40, thereby providing the user with a high-definition video of the area to which the user wants to focus (area of ​​interest). Furthermore, if the focus designation unit 42 detects a viewpoint shift (gazing at another viewpoint position for a predetermined period of time or longer) while the user is viewing the video, it provides new focus information (viewpoint information) to the edge server 14, and continues to control each camera 12 based on the movement of the point of interest (viewpoint).

[0034] Furthermore, the camera control unit 36 ​​of the edge server 14 issues a command to the network control unit 34 to increase the communication priority of the camera 12 (first camera 12a in the case of FIG. 2) that performs zoom control. The network monitoring unit 32 monitors the communication load of the first network NA, and when the bandwidth becomes congested, performs network control to guarantee the communication bandwidth for the zoomed video and lower the priority of other communications. As a result, control can be achieved that enables stable transmission of high-definition zoomed video. Details regarding guaranteeing the communication bandwidth will be described later.

[0035] 3 is an exemplary schematic diagram illustrating the configuration of remote monitoring system 10, and also illustrating the transmission of omnidirectional composite video generated based on captured video other than zoomed video. Fig. 3 is a diagram illustrating an example of communication paths for second camera 12b, third camera 12c, and fourth camera 12d other than zoomed first camera 12a when zooming in on first camera 12a.

[0036] As in FIG. 2, the user adjusts the viewpoint to a desired location while viewing the video on the display unit 40 of the terminal device 16. As a result, the focus designation unit 42 provides position information of the focus point (viewpoint) as focus information (viewpoint information) to the camera control unit 36 ​​of the edge server 14. The camera control unit 36 ​​of the edge server 14, having acquired the user's focus information, changes the setting of the boundary adjustment control unit 30 so that digital composition processing is performed using the video from the cameras 12 other than the zoom target, in order to exclude the first camera 12a, which is the zoom target, from the digital composition processing. In other words, the boundary adjustment control unit 30 sets the video from the first camera 12a not to be used in the digital composition, and also changes the boundary area when performing digital composition using the video from the second camera 12b to the fourth camera 12d so that the 360-degree omnidirectional composite video does not collapse even if the first camera 12a is excluded.

[0037] The second camera 12b to the fourth camera 12d transmit the images captured by their respective lenses 18 to the edge server 14 via the video communication unit 22. The video communication unit 24 of the edge server 14 provides the received images from the second camera 12b to the fourth camera 12d to the digital synthesis processing unit 26. The digital synthesis processing unit 26 applies the boundary area settings changed by the boundary adjustment control unit 30 to synthesize (generate) a 360-degree omnidirectional composite image using only the images from the second camera 12b to the fourth camera 12d, excluding the image from the first camera 12a. The synthesized omnidirectional composite image is then transmitted to the terminal device 16 via the video communication unit 24. The terminal device 16 displays the image received by the video communication unit 38 on the display unit 40, providing it as a 360-degree omnidirectional composite image excluding the zoomed area.

[0038] Furthermore, when digital composition processing unit 26 of edge server 14 combines images from second camera 12b to fourth camera 12d to generate a 360-degree omnidirectional composite image, it may perform image complementation to generate the 360-degree omnidirectional composite image using past images from camera 12 (e.g., first camera 12a) when not zoomed, which are stored in past image data storage unit 28. By performing such image complementation, it becomes possible to generate a pseudo 360-degree omnidirectional composite image even if there is image of a location that is not included in the shooting areas of second camera 12b to fourth camera 12d.

[0039] Specific synthesis processing will be described with reference to Figures 4 to 6. Figure 4 is an exemplary and schematic explanatory diagram showing the shooting range of one camera 12 included in the remote monitoring system 10. Figure 5 is an exemplary and schematic explanatory diagram showing the shooting directions of four cameras 12 and a case where a plurality of captured images are synthesized to generate an omnidirectional composite image. Figure 6 is an exemplary and schematic explanatory diagram showing a case where an omnidirectional composite image is generated by synthesizing images captured by cameras 12 other than the zoomed camera 12 (for example, the first camera 12a).

[0040] As shown in Fig. 4, when generating a 360-degree omnidirectional composite video, cameras 12 generally use fisheye lenses or ultra-wide-angle lenses with a wide viewing angle W. In this case, multiple cameras 12 are arranged so that they face in different directions from the installation location. Therefore, each camera 12 is arranged so that the shooting areas overlap and it is possible to shoot a 360-degree video around it. In this embodiment, an example is shown in which four cameras 12 capable of shooting 180 degrees as shown in Fig. 4 are used and arranged facing perpendicularly in all four directions to generate a 360-degree omnidirectional composite video.

[0041] First, the generation of an omnidirectional composite video when there is no request for zoom shooting (no area of ​​interest is specified) will be described with reference to FIG.

[0042] In this case, the digital composition processing unit 26 and the boundary adjustment control unit 30 digitally combine the images captured by each camera 12 (first camera 12a to fourth camera 12d) so that they are uniform, as shown in FIG. 5. As described above, the viewing angle of the camera 12 is greater than 90 degrees, but the boundaries for cutting out each image are set so that the image area M (viewing angle) of each camera 12 is 90 degrees around the center of the image capture. For example, images captured with a fisheye lens become more distorted the farther they are from the center of the image capture of the camera 12. As shown in FIG. 5, by cutting out and combining images at 90 degrees around the center of the image capture of each image, it becomes easier to generate an omnidirectional composite image with less distortion.

[0043] It should be noted that the color tone of each camera 12 may vary slightly when capturing an image, and the amount of light may vary depending on the capturing direction. Therefore, for example, if images are combined with clearly defined boundaries, the impression of the images may change at the transition point (boundary), creating an unnatural feeling. Therefore, the boundary adjustment control unit 30 may cut out adjacent images so that they overlap by a predetermined amount when combined, and apply image processing to the overlapping portion to reduce the unnatural feeling of the combined image.

[0044] Next, the generation of an omnidirectional composite video when zoom shooting is requested (a region of interest is specified) will be described with reference to FIG.

[0045] When user attention information (e.g., viewpoint information) is obtained from the attention designation unit 42 of the terminal device 16, the camera 12 (e.g., first camera 12a) capturing images in the direction F that matches the attention point (viewpoint) indicated by the attention information performs zoom control of the lens 18. As a result, the first camera 12a, which is the zoom target, zooms its image to match the user's attention point (viewpoint). At this time, as described above, the zoomed image is not targeted when synthesizing the 360-degree omnidirectional composite image. As a result, if the digital synthesis processing unit 26 performs synthesis processing as is, the resulting 360-degree image will have a blank image area for the image area of ​​the camera 12 (first camera 12a) that is the zoom target.

[0046] Therefore, as shown in FIG. 6, the boundary adjustment control unit 30 expands the boundary position of the images from the cameras 12 (second camera 12b and fourth camera 12d) adjacent to the image from the camera to be zoomed (first camera 12a) by an adjustment area N, generates a 270-degree composite image using the images from the second camera 12b and the fourth camera 12d, and generates a 360-degree omnidirectional composite image by combining this with the image from the third camera 12c. In other words, the boundary during digital composition is reset so that the images from the cameras 12 other than the camera 12 that captured the zoomed image approach a 360-degree image. As a result, it is possible to continue generating a 360-degree omnidirectional composite image even if the number of cameras 12 available for composition is reduced.

[0047] However, depending on the limit of the viewing angle of the camera 12, the number and arrangement of the cameras 12, etc., there may be areas without images (missing areas 44) in the 360-degree video, as shown in Fig. 6. In such cases, it is possible to deal with this by generating an omnidirectional composite video using the missing areas 44 as blank images, or by applying video interpolation processing using past videos stored in the past video data storage unit 28. Details of the video interpolation processing will be described later.

[0048] In this way, by generating a 360-degree omnidirectional composite image based on images captured by a camera other than the camera 12 that is the zoom target, both the zoomed image and the omnidirectional composite image can be viewed on the display unit 40, and even if the user's focus (viewpoint) on the terminal device 16 suddenly changes, it becomes possible to check the surrounding situation from a bird's-eye view.

[0049] 7 and 8 are exemplary, schematic explanatory diagrams showing the display layout of zoomed images and omnidirectional composite images (composite images) on the terminal device 16 of the remote monitoring system 10. The examples shown in FIGS. 7 and 8 illustrate a case where the missing area 44 described above is displayed as a blank image. As described above, in the remote monitoring system 10 of this embodiment, both the zoomed image of the camera 12 designated as the zoom target by the user and the 360-degree omnidirectional composite image using the other cameras 12 are transmitted to the terminal device 16. As a result, these images can be used to appropriately display the desired information on the display unit 40 of the terminal device 16. For example, as shown in FIG. 7, a composite image 46 (e.g., an image with a 180-degree angle of view) of the forward direction of the user's direction from the generated omnidirectional composite image is displayed on the entire surface of the display unit 40 so as to provide a wide view of the direction (forward direction) in which the user is facing at the actual monitoring site. Furthermore, a zoomed image 48 of a focus area including the user's focus point in the forward direction is enlarged and displayed at the position of the focus point. In this case, regardless of the position of the attention point on the display unit 40, the attention point, i.e., the zoomed image 48, may be moved to the center of the display unit 40, and the display direction of the composite image 46 may be changed accordingly, so that the composite image 46 is displayed with the attention point at its center. By performing such a display, it is possible to reproduce on the display unit 40 the situation in which the user actually goes to the site and views the attention area and its surroundings. Note that the zoomed image 48 may be configured to be able to be zoomed in and out as appropriate while viewing it. In another embodiment, for example, only the zoomed image 48 may be displayed using the entire display area of ​​the display unit 40, in response to a user operation.

[0050] FIG. 8 illustrates another display layout on the display unit 40. In the example shown in FIG. 8, a composite image 46 (e.g., an image with a 180-degree angle of view) in the forward direction toward which the user is facing and a zoomed image 48 are separately arranged. In the example shown in FIG. 7, the user (viewer) can easily understand the position to which the composite image 46 is zoomed. However, there is a possibility that a portion of the composite image 46 may be covered by the zoomed image 48. On the other hand, when the composite image 46 and the zoomed image 48 are separately arranged as shown in FIG. 8, overlapping of the images is avoided, making it easier to recognize the displayed content. Note that in FIG. 8, the display size of the zoomed image 48 is smaller than that of the composite image 46; however, the display size of the zoomed image 48 may be larger than that of the composite image 46, or the zoomed image 48 may be the same size as the composite image 46. The display size can be selected appropriately by the user.

[0051] In this way, the image data of both the zoomed image and the omnidirectional composite image is transmitted to the terminal device 16, so that the user can freely view the area he or she wants to focus on and the other areas including the surrounding area in the layout he or she wants to see.

[0052] Next, we will explain how the missing area 44 (blank image) is eliminated by the above-mentioned image complementation process. As mentioned above, when a 360-degree omnidirectional composite image is generated (constructed) using cameras 12 other than the camera 12 that is the zoom target, a missing area 44 (area without image) may occur. In this case, the digital composition processing unit 26 of the edge server 14 can determine, by image processing or the like, that the image from the camera 12 that is the zoom target cannot be used to generate the omnidirectional composite image and that a missing area 44 has occurred. In this case, the digital composition processing unit 26 extracts image information of the camera 12 that is the zoom target from the past image data storage unit 28 and combines the past image data into the missing area 44 by image embedding. As a result, the digital composition processing unit 26 can generate a pseudo 360-degree omnidirectional composite image in which the image data of the missing area 44 has been complemented, and provide the generated image to the terminal device 16. As for the method of complementing past video data, it is possible to select a complementing method according to the application, such as a method of synthesizing the past video data itself, a method of synthesizing only video areas (video information of fixed objects) that have little change in the past video data, etc. Also, by using the video to be complemented as the video immediately before zoom shooting begins, for example, it is possible to generate an omnidirectional composite video that is close to real-time video.

[0053] When performing video complementation processing, the generated omnidirectional composite video includes past video and is not real-time video. Therefore, when displaying video based on the omnidirectional composite video complemented with past video on the display unit 40, the inclusion of past video may be indicated. For example, this may be indicated by a text message on the display unit 40 or by audio. Furthermore, only the past video portion may be visually identified by changing the color, changing the transparency, or adding a mark.

[0054] 9 and 10, an example of network control will be described in which video quality control and network bandwidth allocation are performed based on the degree of coincidence with the viewpoint when zooming in on each camera 12 (first camera 12a to fourth camera 12d) in accordance with the user's viewpoint (point of interest). FIG. 9 is an exemplary explanatory diagram 50 showing the relationship between video quality according to video content, network slicing resource allocation, and bandwidth guarantee in the remote monitoring system 10. FIG. 10 is an exemplary schematic explanatory diagram showing an image of network slicing resource allocation according to video content.

[0055] The camera control unit 36 ​​of the edge server 14 controls the zoom of the lens 18 of the camera 12 whose direction matches the administrator's viewpoint information (attention information) and sets the camera 12 to transmit high-quality video (see Figure 2). High-quality video means, for example, sending uncompressed video or sending video data with a lower compression rate that is close to the original video. While such video data provides high-quality video, it requires broadband resources for transmission. Therefore, the network control unit 34 of the edge server 14 increases the network priority for communication with the camera 12 that is the target of zoom shooting control, allocating network resources to ensure stable communication. For example, systems such as 5G communication and local 5G communication use a technology called network slicing, which divides a network into virtual network resources and allocates them for communication. Using this technology, the bandwidth for transmitting video data from the camera 12, which can potentially be high-quality video data, is guaranteed.

[0056] For example, in the examples shown in FIGS. 9 and 10, the camera 12 targeted for zooming that matches the user's viewpoint information (attention information) is the second camera 12b (degree of matching = ○). The camera control unit 36 ​​instructs the second camera 12b to communicate using uncompressed transmission and a fixed bandwidth, and allocates wideband network resources to guarantee bandwidth and set high communication quality. As shown in FIG. 10, for example, slice b is allocated 10 Mbps. Furthermore, for the cameras 12 not targeted for zooming, medium video compression and medium network resource allocation are performed for the first camera 12a and the third camera 12c, which have a medium degree of matching (= △) with the viewpoint information. For example, slices a and c are allocated 5 Mbps each. Furthermore, for the fourth camera 12d, which has a low degree of matching with the viewpoint information, the compression rate is increased to suppress the transmission bandwidth, and minimum network resource allocation is performed. For example, slice d is allocated 2 Mbps. Note that by using best-effort communication for cameras other than the camera 12 to be zoomed (e.g., the second camera 12b), high-quality communication quality is set within an acceptable range, ensuring that the video from the camera 12 to be zoomed (the second camera 12b), which communicates at a fixed bandwidth, is reliably delivered to the terminal device 16. Note that by using best-effort communication for cameras other than the camera 12 to be zoomed (e.g., the second camera 12b), the highest quality communication quality within an acceptable range is set and video data is provided to the edge server 14, so that the boundary adjustment control unit 30 can adjust the boundary area using video data (captured video) of high quality within an acceptable range and generate an omnidirectional composite video.

[0057] In this way, the remote monitoring system 10 of this embodiment controls the camera 12 and the network in cooperation with each other based on the user's attention information (viewpoint information). As a result, it is possible to provide a remote monitoring system 10 that can ensure both high-definition video and stable communication.

[0058] The edge server 14 may store attention information indicating the attention area designated by the attention designation unit 42. In this case, the camera control unit 36 ​​can estimate the high attention area, which is frequently designated, based on the stored past attention information. The digital composition processing unit 26 may then display the camera 12 capturing the high attention area in the omnidirectional composite image. The camera control unit 36 ​​may also store zoom control values ​​for the camera 12 corresponding to the high attention area. The boundary adjustment control unit 30 may also store boundary area adjustment information used when compositing captured images that do not include the high attention area. In this way, by performing control using the high attention area, it is possible to quickly present the user with zoomed images showing areas that the user frequently pays attention to, such as areas that require frequent inspection or frequent maintenance. Furthermore, it is possible to quickly compose and provide the omnidirectional composite image. This contributes to improving the efficiency and quality of surveillance work.

[0059] FIG. 11 is an exemplary flowchart showing the flow of processing in the remote monitoring system 10 according to the embodiment.

[0060] When a remote monitoring start operation is performed on the terminal device 16, shooting control for each camera 12 is performed via the second network NB, the edge server 14, and the first network NA (S100).

[0061] The camera control unit 36 ​​of the edge server 14 checks whether the attention designation unit 42 of the terminal device 16 has specified an area of ​​interest (S102). If an area of ​​interest has been designated (Yes in S102), that is, if a zoom image of the point of interest (viewpoint) is requested by viewpoint tracking or the like, the camera control unit 36 ​​starts zoom control of the camera 12 corresponding to the area of ​​interest (S104). The amount of zoom control can be appropriately controlled by eye movement or the like when the area of ​​interest is designated. Furthermore, the network control unit 34 executes network slicing processing based on information about the camera 12 performing zoom shooting and the monitoring results of the network monitoring unit 32 so that high-quality zoomed images can be transmitted (S106). That is, as described with reference to FIGS. 9 and 10 , the camera control unit 36 ​​instructs the camera 12 to perform uncompressed transmission and fixed-bandwidth communication, and allocates wideband network resources to guarantee bandwidth to the camera 12 that is being zoom-controlled. Furthermore, network resources are allocated to the other cameras 12 according to a priority level corresponding to the degree of match with the viewpoint information (attention information).

[0062] When the video communication unit 24 of the edge server 14 acquires the captured images from each camera 12 (S108), the boundary adjustment control unit 30 sets parameters indicating the boundary area of ​​adjacent images to be joined when generating an omnidirectional composite image using the captured images from the non-zoom camera 12 (S110). Then, the digital composition processing unit 26 generates an omnidirectional composite image using the captured images from the non-zoom camera 12 using the set boundary area. At this time, it detects whether or not a missing area 44 exists in the omnidirectional composite image to be generated (S112). If a missing area 44 exists in the omnidirectional composite image to be generated (Yes in S112), the digital composition processing unit 26 extracts video information of the camera 12 to be zoomed from the past video data storage unit 28 and performs composition by fitting the video of the past video data into the missing area 44 (S114). As a result, the digital composition processing unit 26 performs processing to generate a pseudo 360-degree omnidirectional composite image by complementing the video data of the missing area 44 (S116).

[0063] When the process of generating the omnidirectional composite image is completed, the image communication unit 24 performs a process of transmitting the omnidirectional composite image and the zoom image 48 to the terminal device 16 via the second network NB (S118).

[0064] When the video communication unit 38 of the terminal device 16 acquires the omnidirectional composite image and the zoomed image 48 from the edge server 14, the display unit 40 executes a display process to display the composite image 46 and the zoomed image 48 in the display format requested by the user (S120).

[0065] Then, if an instruction to end the remote monitoring process is given in the terminal device 16 (Yes in S122), this flow is temporarily terminated. On the other hand, if an instruction to end the remote monitoring process has not been given (No in S122), the process returns to S102, where it is determined whether or not a region of interest has been designated, and the subsequent processes are executed. In other words, the monitoring operation can be continued by continuously displaying and allowing the user to view the zoomed image 48 showing the region in which the user wants to focus and the omnidirectional composite image (composite image 46) showing the surrounding situation.

[0066] In addition, if there is no missing area 44 in the processing of S112 (No in S112), that is, if the omnidirectional composite image can be generated using images other than the zoomed image, the processing of S114 is skipped and the omnidirectional composite image is generated in the processing of S116.

[0067] Furthermore, if no area of ​​interest is specified in the process of S102 (No in S102), that is, if the user does not wish to view a specific area in detail and performs monitoring work using the omnidirectional composite image (composite image 46), the process of S104 is skipped. In this case, it is assumed that there is no difference in the priority of the images transmitted from each camera 12, and the network control unit 34 allocates network bandwidth resources evenly, transmits the images captured by each camera 12 to the image communication unit 24 of the edge server 14, and executes the subsequent processes.

[0068] FIG. 12 is an exemplary schematic diagram illustrating a configuration in which a plurality of terminal devices 16 are present in the remote monitoring system 10, and also illustrates the transmission of a specified zoom image and the transmission of an omnidirectional composite image generated based on images captured by a plurality of cameras.

[0069] The configuration of the remote monitoring system 10 shown in Fig. 12 is the same as the configuration shown in Fig. 1, etc., except that a plurality of terminal devices 16 (for example, three terminal devices 16a to 16c) are connected. Therefore, the same components are denoted by the same reference numerals, and detailed descriptions thereof will be omitted.

[0070] 12 shows a case where there are three terminal devices 16, but the number of connected devices can be selected as appropriate, and simultaneous viewing is possible for each terminal device 16. The number of connected cameras 12 can also be selected as appropriate, as long as there are two or more cameras.

[0071] For example, consider a case where the attention information (viewpoint information) of terminal device 16a matches that of first camera 12a, and the attention information (viewpoint information) of terminal device 16b matches that of second camera 12b, and each requests zoom control. In this case, based on the attention information (viewpoint information) of the attention designation unit 42 of terminal device 16a, edge server 14 issues a zoom command to the zoom control unit 20 of first camera 12a and instructs it to transmit uncompressed video data. Similarly, based on the attention information (viewpoint information) of the attention designation unit 42 of terminal device 16b, edge server 14 issues a zoom command to the zoom control unit 20 of second camera 12b and instructs it to transmit uncompressed video data. In this case, the video data of first camera 12a and second camera 12b are not included in the video data used to generate the 360-degree omnidirectional composite video. Then, terminal device 16a and terminal device 16b that requested zooming display zoomed images corresponding to the attention information transmitted via edge server 14 on their respective displays 40. 12, terminal device 16c has not requested zooming in on a specific location. Therefore, terminal device 16c is provided with an omnidirectional composite image generated based on images captured by third camera 12c and fourth camera 12d (non-zooming cameras) via second network NB from edge server 14. As a result, only the omnidirectional composite image (composite image 46) is displayed on display unit 40 of terminal device 16c.

[0072] In addition, if there are multiple cameras 12 to be zoomed and many missing areas 44 (areas without images) are detected when generating 360-degree omnidirectional images, a pseudo 360-degree omnidirectional image can be constructed by performing image interpolation based on past image data stored in the past image data storage unit 28.

[0073] When multiple terminal devices 16 simultaneously view multiple viewpoints, it is conceivable that many cameras 12 cannot be used for digital composition, making it impossible to create a 360-degree omnidirectional composite image. In such cases, the zoom control of the cameras 12 can be switched from optical zoom to digital zoom control, and the 360-degree image can be created by digitally combining the images. The 360-degree image and the zoomed image can then be transmitted by transmitting only the digitally zoomed image of the viewpoint desired to the requesting terminal device 16. Although digital zoom cannot achieve higher image quality than optical zoom, it is possible to provide a remote monitoring system that can accommodate multiple viewpoints or when the user zooms in on different locations on the same camera.

[0074] In the remote monitoring system 10 of this embodiment, the control program that realizes the processing performed by the edge server 14 may be configured to be provided by being recorded in an installable or executable format on a computer-readable recording medium such as a CD-ROM, CD-R, DVD (Digital Versatile Disk), flash memory, or the like.

[0075] Furthermore, in the remote monitoring system 10 of this embodiment, the control program that realizes the processing performed by the edge server 14 may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the control program may be provided or distributed via a network such as the Internet.

[0076] Although the embodiments of the present invention have been described above, the above embodiments are merely examples and are not intended to limit the scope of the invention. The above embodiments can be implemented in various forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. The above embodiments are included within the scope and spirit of the invention, as well as within the scope of the inventions and their equivalents as set forth in the claims. [Explanation of symbols]

[0077] 10 Remote Monitoring System 12 Camera 12a Camera 1 12b Second Camera 12c Third Camera 12d 4th camera 14 Edge Server 16, 16a, 16b, 16c terminal equipment 18 Lenses 20 Zoom control section 22 Video communication unit (first communication unit) 24 Video communication unit (second communication unit) 26 Digital composition processing unit 28 Past video data storage unit (video memory unit) 30 Boundary adjustment control section 32 Network Monitoring Department 34 Network Control Unit 36 Camera control unit 38 Video communication unit (third communication unit) 40 Display section 42 Attention designation section 44 Missing Area 46 Synthetic Images 48 Zoomed Video M Video area N adjustment area NA First Network NB Second Network

Claims

1. A remote monitoring system including a plurality of cameras, an edge server connected to the plurality of cameras via a first network, performing processing according to images captured by the plurality of cameras, and outputting images based on the captured images, and at least one terminal device connected to the edge server via a second network and capable of displaying at least images provided by the edge server, The plurality of cameras are installed facing different shooting directions so as to be able to acquire the shot images in all directions, and each of the cameras is Lenses and a zoom control unit that adjusts the zoom of the lens; a first communication unit that communicates with the edge server; Equipped with The edge server a second communication unit that communicates between the plurality of cameras and the terminal device; a camera control unit that independently controls the zoom control units of the cameras; a synthesis processing unit that synthesizes at least two of the captured images to generate a synthesized image; Equipped with The terminal device a third communication unit that communicates with the edge server; a display unit that displays the video supplied from the edge server; an attention designation unit that designates an attention area in the image displayed on the display unit; Equipped with the edge server provides the terminal device with an omnidirectional composite image generated by combining a zoomed image of the camera, the zoomed image having an area corresponding to the area of ​​interest designated by the attention designation unit as an image capture area, and the image captured by the camera other than the zoomed camera. Remote monitoring system.

2. The lens is a wide-angle lens that can capture a wider range of images than the angle obtained by dividing the omnidirectional angle by the number of the cameras, The edge server a video storage unit that stores the captured videos acquired by the plurality of cameras for a predetermined period of time; a boundary adjustment control unit that adjusts a boundary area between adjacent captured images captured through the wide-angle lens when generating the omnidirectional composite image; The remote monitoring system of claim 1 , comprising:

3. the edge server identifies the camera for which zoom control is performed based on the region of interest that can be acquired from the terminal device, provides a control command to the zoom control unit of the identified camera, and acquires a zoomed image in which the region of interest is zoomed; the boundary adjustment control unit adjusts the boundary area of ​​the images captured by the non-zoom cameras that sandwich the zoom target camera, and generates the omnidirectional composite image. The remote monitoring system of claim 2 .

4. When generating the omnidirectional composite video from the captured video of the non-zoom camera, if there is a missing area that is not projected onto the lens, the edge server complements the missing area using previously captured video stored in the video storage unit to generate a virtual omnidirectional composite video. The remote monitoring system according to claim 3 .

5. the first network and the second network are communication networks capable of allocating bandwidth of network resources with low latency, The edge server includes a network monitoring unit that monitors a load status of a bandwidth of at least one of the first network and the second network; a network control unit that adjusts resources allocated to video data communication of the camera according to a load status of the bandwidth; Equipped with The remote monitoring system of claim 2 .

6. the network control unit sets a communication quality of a high quality equal to or higher than a predetermined value for the image captured by the zoom-controlled camera, and transmits the image by setting a compression rate to a low compression rate or no compression rate. The remote monitoring system according to claim 5 .

7. the network control unit sets a high communication quality within an allowable range for the image captured by the non-zoom controlled camera, the boundary adjustment control unit adjusts the boundary area using the captured video having high quality within the allowable range. The remote monitoring system according to claim 5 .

8. the edge server accumulates attention information indicating the attention area designated by the attention designation unit; The camera control unit estimates a high attention area that is frequently attended to based on the accumulated past attention information, the synthesis processing unit displays the camera capturing the high attention area in the omnidirectional synthesized image; the camera control unit stores a zoom control value of the camera corresponding to the high attention area; the boundary adjustment control unit stores adjustment information of the boundary area when the captured video not including the high attention area is synthesized; The remote monitoring system according to claim 2 .

9. the attention designation unit designates the attention area by detecting a viewpoint through image processing or by a designation operation based on input information from a user interface; The remote monitoring system of claim 1 .

10. When a plurality of terminal devices are operated simultaneously, the edge server identifies the camera to zoom from among the plurality of cameras based on a plurality of attention areas designated by a plurality of attention designation units, and the camera control unit controls the zoom of the identified camera, the synthesis processing unit generates an omnidirectional synthetic image by synthesizing the images captured by the non-zoom cameras, The remote monitoring system of claim 1 .

11. A remote monitoring method including a plurality of cameras, an edge server connected to the plurality of cameras via a first network, performing processing according to images captured by the plurality of cameras, and outputting images based on the captured images, and at least one terminal device connected to the edge server via a second network and capable of displaying at least images provided by the edge server, The plurality of cameras are installed facing different shooting directions so as to be able to acquire the shot images in all directions, and each of the cameras includes a lens, a zoom control unit that adjusts the zoom of the lens, and a first communication unit that communicates with the edge server; the edge server includes a second communication unit that communicates between the plurality of cameras and the terminal device, a camera control unit that independently controls the zoom control units of the cameras, and a synthesis processing unit that synthesizes at least two of the captured images to generate a composite image; the terminal device includes a third communication unit that communicates with the edge server, a display unit that displays the video provided from the edge server, and an attention designation unit that designates an attention area in the video displayed on the display unit; a region-of-interest providing step in which the attention specifying unit provides the specified region of interest to the edge server; a camera control step in which the camera control unit controls the zoom control unit based on the region of interest; a zoomed image providing step in which the edge server provides, via the second communication unit, a zoomed image having an area corresponding to the area of ​​interest acquired from the camera via the first communication unit as a shooting area to the terminal device; an omnidirectional image providing step in which the edge server provides the terminal device with an omnidirectional composite image generated by combining, in the synthesis processing unit, the captured images acquired from the cameras other than the camera zoomed in on the area of ​​interest acquired from the camera via the first communication unit; and Provide a remote monitoring method.

12. A control program for a remote monitoring system including a plurality of cameras, an edge server connected to the plurality of cameras via a first network, performing processing according to images captured by the plurality of cameras, and outputting images based on the captured images, and at least one terminal device connected to the edge server via a second network and capable of displaying images provided by at least the edge server, The plurality of cameras are installed facing different shooting directions so as to be able to acquire the shot images in all directions, and each of the cameras includes a lens, a zoom control unit that adjusts the zoom of the lens, and a first communication unit that communicates with the edge server; the edge server includes a second communication unit that communicates between the plurality of cameras and the terminal device, a camera control unit that independently controls the zoom control units of the cameras, and a synthesis processing unit that synthesizes at least two of the captured images to generate a composite image; the terminal device includes a third communication unit that communicates with the edge server, a display unit that displays the video provided from the edge server, and an attention designation unit that designates an attention area in the video displayed on the display unit; A control unit of the edge server a region-of-interest providing step in which the attention specifying unit provides the specified region of interest to the edge server; a camera control step in which the camera control unit controls the zoom control unit based on the region of interest; a zoomed image providing step in which the edge server provides, via the second communication unit, a zoomed image having an area corresponding to the area of ​​interest acquired from the camera via the first communication unit as a shooting area to the terminal device; an omnidirectional image providing step in which the edge server provides the terminal device with an omnidirectional composite image generated by combining, in the synthesis processing unit, the captured images acquired from the cameras other than the camera zoomed in on the area of ​​interest acquired from the camera via the first communication unit; and A control program for a remote monitoring system that executes the above.

Images