Image processing device, image processing method, and program

Abstract

To provide an image processing device that can grasp the status of a monitored area at shorter time intervals. [Solution] The image processing apparatus according to the present disclosure comprises: an overall image acquisition unit that acquires an overall image of a predetermined region; a partial image acquisition unit that acquires a partial image of a part of the predetermined region taken at a different timing from the overall image; a state change determination unit that determines a state change in a part of the region based on the overall image and the partial image; an influence estimation unit that estimates the range of influence of the state change on the overall image based on the determination result of the state change determination unit; and an image generation unit that generates a pseudo-overall image that pseudoly represents the overall image at the time the partial image was taken, based on the overall image, the partial image, and the influence range.

Description

Technical Field

[0001] The present disclosure relates to an image processing apparatus, an image processing method, and a program.

Background Art

[0002] Techniques for remotely monitoring the situation in a monitored area using images from surveillance cameras and the like are known. By using such techniques for monitoring mountainous areas and river areas, it is possible to respond quickly in the event of a disaster.

[0003] In such techniques, in order to acquire an image, on-site shooting is necessary, but there are physical constraints and cost issues in shooting a wide range of images. In recent years, in order to improve the comprehensiveness and real-time nature of images, posted images on SNS (Social Networking Service) and captured images of a drive recorder may also be used. These images are utilized at the monitoring site by performing predetermined image processing such as pasting together a plurality of images.

[0004] As a related technique, Patent Document 1 discloses an image acquisition device capable of improving the generation frame rate when generating an image with a wide dynamic range by synthesizing a plurality of images. The image acquisition device causes an imaging element to image a full-region image of an observation image over the entire light-receiving surface under a first exposure condition, and causes the imaging element to image a partial-region image of the observation image only for a partial region of the light-receiving surface under a second exposure condition. Further, the image acquisition device synthesizes the full-region image and the partial-region image to obtain a full-region image having a wider dynamic range than the full-region image.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

[0006] In recent years, generative AI (Artificial Intelligence) technology, which generates various types of images, has been developed and is beginning to spread. In such technologies, it is common for a person to input information (prompts / images) into the generative AI to generate images. However, this method makes it difficult to obtain images that reflect the constantly changing situation on site.

[0007] The purpose of this disclosure is to provide an image processing device, an image processing method, and a program that can grasp the status of a monitored area at shorter time intervals, in light of the above-mentioned problems. [Means for solving the problem]

[0008] The image processing apparatus relating to this disclosure is A whole image acquisition unit that acquires a whole image of a predetermined area, A partial image acquisition unit that acquires a partial image of a portion of the predetermined region at a different timing from the overall image, A state change determination unit determines a state change in a partial region based on the overall image and the partial image, An influence estimation unit estimates the range of influence of the state change on the overall image based on the determination result in the state change determination unit, The system includes an image generation unit that generates a pseudo-whole image that pseudo-represents the whole image at the time the partial image was captured, based on the whole image, the partial image, and the affected area.

[0009] The image processing method relating to this disclosure is: A whole image acquisition step to acquire a whole image of a predetermined area, A partial image acquisition step involves acquiring a partial image of a portion of the predetermined region, which is captured at a different timing than the overall image. A state change determination step in which a state change in a part of the region is determined based on the overall image and the partial image, An influence estimation step is performed to estimate the range of influence of the state change on the overall image based on the determination result in the state change determination step, The method includes an image generation step of generating a pseudo-whole image that pseudoly represents the whole image at the time the partial image was captured, based on the whole image, the partial image, and the affected area.

[0010] The program related to this disclosure is A whole image acquisition step to acquire a whole image of a predetermined area, A partial image acquisition step involves acquiring a partial image of a portion of the predetermined region, which is captured at a different timing than the overall image. A state change determination step in which a state change in a part of the region is determined based on the overall image and the partial image, An influence estimation step is performed to estimate the range of influence of the state change on the overall image based on the determination result in the state change determination step, The computer is instructed to perform an image generation step, which involves generating a pseudo-whole image that simulates the whole image at the time the partial image was captured, based on the whole image, the partial image, and the affected area. [Effects of the Invention]

[0011] The image processing apparatus, image processing method, and program described herein enable the status of a monitored area to be grasped at shorter time intervals. [Brief explanation of the drawing]

[0012] [Figure 1] Figure 1 is a block diagram showing the functional configuration of an image processing device. [Figure 2] Figure 2 is a flowchart showing the processes performed by the image processing device. [Figure 3]Figure 3 is a view of the overall configuration of the image processing system. [Figure 4] Figure 4 is a block diagram showing the functional configuration of an image processing device. [Figure 5] Figure 5 is a schematic diagram illustrating the overview of image processing in an image processing system. [Figure 6] Figure 6 is a diagram that explains the image processing shown in Figure 5 in detail. [Figure 7] Figure 7 is a flowchart showing the processes performed by the image processing device. [Figure 8] Figure 8 is a block diagram illustrating the hardware configuration of a computer that implements an image processing device and the like. [Modes for carrying out the invention]

[0013] Embodiments of the present disclosure will be described in detail below with reference to the drawings. In each drawing, the same or corresponding elements are denoted by the same reference numerals. For clarity of explanation, redundant explanations will be omitted where necessary.

[0014] <Embodiment 1> (Image processing device 100) Figure 1 is a block diagram showing the functional configuration of the image processing apparatus 100 according to this disclosure. The image processing apparatus 100 comprises an overall image acquisition unit 101, a partial image acquisition unit 102, a state change determination unit 103, an influence estimation unit 104, and an image generation unit 105.

[0015] The whole image acquisition unit 101 acquires a whole image of a predetermined region. The partial image acquisition unit 102 acquires a partial image of a part of the predetermined region, taken at a different timing than the whole image. The state change determination unit 103 determines a state change in the partial region based on the whole image and the partial image. The influence estimation unit 104 estimates the range of influence of the state change on the whole image based on the determination result from the state change determination unit 103. The image generation unit 105 generates a pseudo-whole image that simulates the whole image at the time the partial image was taken, based on the whole image, the partial image, and the range of influence.

[0016] The image processing device 100 includes a processor, memory, and storage device (not shown). The storage device stores a computer program on which the processing described herein is implemented. The processor can load the computer program from the storage device into memory and execute the computer program. As a result, the processor realizes the functions of the overall image acquisition unit 101, the partial image acquisition unit 102, the state change determination unit 103, the influence estimation unit 104, and the image generation unit 105.

[0017] Alternatively, the overall image acquisition unit 101, the partial image acquisition unit 102, the state change determination unit 103, the influence estimation unit 104, and the image generation unit 105 may each be implemented with dedicated hardware. Furthermore, some or all of each component may be implemented by general-purpose or dedicated circuits, processors, etc., or combinations thereof. These may be configured by a single chip or by multiple chips connected via a bus. Some or all of each component may be implemented by a combination of the aforementioned circuits, etc., and programs.

[0018] (Processing by the image processing device 100) The process performed by the image processing device 100 will be explained with reference to Figure 2. Figure 2 is a flowchart showing the process performed by the image processing device 100.

[0019] First, the overall image acquisition unit 101 acquires an overall image of a predetermined region (S1). Next, the partial image acquisition unit 102 acquires a partial image of a part of the predetermined region, taken at a different timing than the overall image (S2). Subsequently, the state change determination unit 103 determines a state change in the partial region based on the overall image and the partial image (S3).

[0020] Next, the influence estimation unit 104 estimates the extent of the influence of the state change on the overall image based on the determination result in the state change determination unit 103 (S4). Then, the image generation unit 105 generates a pseudo-overall image at the time the partial image was captured, based on the overall image, the partial image, and the influence extent (S5).

[0021] As described above, the image processing device 100 according to this disclosure generates a pseudo-whole image at the time the partial image was captured, based on the whole image, partial image, and affected area, thereby enabling the monitoring of the area to be monitored at shorter time intervals.

[0022] <Embodiment 2> Next, Embodiment 2 will be described. Embodiment 2 is a specific example of Embodiment 1 described above.

[0023] (Image processing system 10) Figure 3 is a block diagram showing the overall configuration of the image processing system 10 according to this disclosure. The image processing system 10 comprises an image processing device 1, a field monitoring system 2, a terminal device 3, a drive recorder 4, and a fixed-point camera 5.

[0024] The image processing device 1, the field monitoring system 2, the terminal device 3, the drive recorder 4, and the fixed-point camera 5 can each communicate with one another via the network N. The number of each component in the image processing system 10 is not limited to those shown in the diagram. For example, there may be two or more terminal devices 3, drive recorders 4, and fixed-point cameras 5.

[0025] The image processing system 10 is a system capable of performing predetermined image processing on images captured in a predetermined area. Specifically, the image processing system 10 acquires images captured by the terminal device 3, the drive recorder 4, and the fixed-point camera 5, and performs predetermined image processing in the image processing device 1. The image processing device 1 transmits a simulated image generated by the image processing to the field monitoring system 2. The field monitoring system 2 receives the simulated image from the image processing device 1 and uses the image to monitor the situation in the predetermined area.

[0026] (Site monitoring system 2) The site monitoring system 2 is a system for monitoring a site. The site is any area that is to be photographed by the image processing system 10. Any area includes the predetermined area described above. The site monitoring system 2 may be installed, for example, at a predetermined monitoring center. The site may be, for example, a mountainous area, a river area, a coastal area, a volcanic area, or a forest area, but is not limited to these.

[0027] Furthermore, the site monitoring system 2 includes a display unit for monitoring the site conditions using images. The display unit is, for example, a display. For example, a monitoring center has a monitoring officer to monitor the site conditions. The monitoring officer monitors the site conditions by looking at the images displayed on the display unit.

[0028] (Fixed-point camera 5) The fixed-point camera 5 is an example of a shooting device that captures a predetermined area to obtain an overall image. The overall image is an image of the entire predetermined area. The predetermined area is the area to be monitored by the on-site monitoring system 2. The area to be monitored may consist of multiple predetermined areas. For example, the image processing system 10 may be equipped with multiple fixed-point cameras 5, and each of the multiple fixed-point cameras 5 may capture a different predetermined area.

[0029] For example, the fixed-point camera 5 is a two-dimensional camera that captures two-dimensional images in a predetermined direction from its installation location. The fixed-point camera 5 may also be an all-around camera capable of capturing a 360-degree field of view around it. The fixed-point camera 5 may be a surveillance camera installed in a mountainous area or an urban area, or an infrastructure camera installed on or around a road. The fixed-point camera 5 captures a predetermined area at predetermined time intervals (e.g., every two hours) and transmits the captured images to the image processing device 1. However, the fixed-point camera 5 may also capture images and transmit them in response to requests from the on-site monitoring system 2, etc.

[0030] Terminal device 3 and drive recorder 4 are examples of imaging devices that capture images from a position different from the overall image capture position. Terminal device 3 and drive recorder 4 capture at least a portion of a predetermined area (the overall image capture range). The images captured by terminal device 3 and drive recorder 4 can be acquired as partial images by the partial image acquisition unit 12 of the image processing device 1.

[0031] (Terminal device 3) Terminal device 3 is a terminal device used by a user of the image processing system 10. Terminal device 3 is equipped with a camera or other imaging device (not shown). Terminal device 3 transmits captured images to the image processing device 1 via a communication unit (not shown). Terminal device 3 may be a portable computer such as a smartphone, PC (Personal Computer), or tablet terminal. The user of terminal device 3 takes pictures of their surroundings at any time. For example, terminal device 3 transmits captured images to the image processing device 1 via a social networking service (SNS) that accepts image submissions. The images may be still images or moving images.

[0032] (Dashcam 4) The drive recorder 4 is a recording device installed in a vehicle. The drive recorder 4 takes pictures of the area around the vehicle and transmits the captured images to the image processing device 1. The drive recorder 4 may take pictures and transmit the captured images at predetermined time intervals, or it may perform these processes when an event such as an accident occurs. The drive recorder 4 may take pictures continuously and transmit the captured images when an abnormality is detected in the captured images. For example, the drive recorder 4 may use any image recognition technology to detect abnormalities. Examples of abnormalities include damage to roads or structures, or landslides. The abnormalities to be detected are not limited to these. The drive recorder 4 may detect various abnormalities that are being monitored.

[0033] Here, terminal device 3, drive recorder 4, and fixed-point camera 5 are used as examples of imaging devices that transmit images to image processing device 1, but the imaging devices are not limited to these. Any device may be used as the imaging device. For example, instead of the fixed-point camera 5, an imaging device mounted on a mobile device may be used. A mobile device may be, for example, a satellite, drone, or camera vehicle. Alternatively, an imaging device carried by the camera operator may be used. Not limited to these, any imaging device capable of comprehensively capturing a predetermined area may be used. Furthermore, instead of terminal device 3 and drive recorder 4, any imaging device capable of capturing at shorter time intervals than the fixed-point camera 5 may be used.

[0034] (Image processing device 1) The image processing device 1 will be described with reference to Figure 4. Figure 4 is a block diagram showing the functional configuration of the image processing device 1. The image processing device 1 is an example of the image processing device 100 described above. The image processing device 1 includes an overall image acquisition unit 11, a partial image acquisition unit 12, a state change determination unit 13, an influence estimation unit 14, an image generation unit 15, a risk level setting unit 16, and a communication unit 17.

[0035] The overall image acquisition unit 11 is an example of the overall image acquisition unit 101 described above. The overall image acquisition unit 11 acquires an overall image of a predetermined area. The overall image acquisition unit 11 acquires an overall image from the fixed-point camera 5. The overall image acquisition unit 11 may acquire the overall image at predetermined time intervals (for example, every 2 hours) or continuously.

[0036] The partial image acquisition unit 12 is an example of the partial image acquisition unit 102 described above. The partial image acquisition unit 12 acquires a partial image of a portion of a predetermined area, captured at a different timing than the overall image. The partial image is captured at a later timing than the overall image. Multiple overall images and partial images may be captured. Therefore, the overall image and the corresponding partial image are appropriately combined according to the timing of the capture.

[0037] For example, the partial image acquisition unit 12 acquires a captured image taken from a position different from the position where the whole image was taken, identifies the overlapping portion between the whole image and the captured image, and acquires that overlapping portion as a partial image.

[0038] Specifically, the partial image acquisition unit 12 first acquires captured images from the terminal device 3 or the drive recorder 4. Here, it is assumed that the partial image acquisition unit 12 acquires captured images from the terminal device 3. The partial image acquisition unit 12 compares (matches) the overall image acquired by the overall image acquisition unit 11 with the captured image acquired from the terminal device 3 and identifies the overlapping portion of the two images. For example, the partial image acquisition unit 12 identifies the overlapping portion using the positional information of each image. The partial image acquisition unit 12 may also identify the overlapping portion using a common object or other marker included in the two images. The partial image acquisition unit 12 may also identify the overlapping portion using any image recognition technology.

[0039] The state change determination unit 13 is an example of the state change determination unit 103 described above. The state change determination unit 13 determines a state change in a part of the region based on the overall image and the partial image. The state change determination may include determining whether or not a state change has occurred. The state change determination may also include determining the content of the state change.

[0040] The state change determination unit 13 may estimate events occurring in a certain area and determine a state change based on those events. For example, the state change determination unit 13 may estimate events that include at least one of the following in a certain area: changes in weather, changes in water level, crustal deformation, changes in the shape of a building, and changes in the position of an object. By determining a state change based on the estimated events in the state change determination unit 13, the image generation unit 15 can generate images that are more accurate and closer to the actual site conditions.

[0041] The state change determination unit 13 may estimate the probability of an event occurring in a certain region based on the location information of that region, and determine a state change based on the probability of occurrence. The location information of a certain region may be information indicating its position in an image, or information indicating its position in real space.

[0042] For example, the state change determination unit 13 estimates the probability of an event occurring using event prediction information associated with location information of a specific area. For example, event prediction information may include weather forecast information, disaster prediction information, hazard maps, crustal survey information, or building specification information for a specific area. However, event prediction information is not limited to these.

[0043] For example, weather forecast information, disaster prediction information, hazard maps, crustal survey information, and building specification information are all collections of information about events that occur in a given area. This event prediction information is associated, for example, with location information (e.g., latitude and longitude) corresponding to map information, and with information about the event itself.

[0044] Event prediction information is provided, for example, from a predetermined information provision server that provides such information. The state change determination unit 13 may acquire event prediction information via the network N. However, the event prediction information may also be stored in a storage unit (not shown) of the image processing device 1.

[0045] For example, the state change determination unit 13 estimates the probability of a landslide occurring in a certain area based on disaster prediction information. The state change determination unit 13 changes the setting value used to determine the state change depending on whether the probability of occurrence is above a predetermined threshold or below a predetermined threshold. By determining the state change using a setting value corresponding to the probability of a landslide occurring, the state change determination unit 13 can make different determinations depending on whether the probability of a landslide occurring is high or low.

[0046] In this way, the state change determination unit 13 can accurately determine which events require particular determination among the state changes in a certain region. Since the state change determination unit 13 can narrow down the target of state change determination, it can determine state changes more quickly and with higher accuracy.

[0047] The influence estimation unit 14 is an example of the influence estimation unit 104 described above. The influence estimation unit 14 estimates the range of influence of the state change on the overall image based on the determination result in the state change determination unit 13. The influence estimation unit 14 estimates the spatially affected image region based on the content of the state change determined by the state change determination unit 13. As a result, the image generation unit 15 can generate an image that reflects the determination content of the state change determination unit 13 for the image region of the affected area.

[0048] For example, the impact estimation unit 14 estimates the extent of impact for each event. This allows the impact estimation unit 14 to estimate the extent of impact for each event. In this way, the image generation unit 15 can generate images with high accuracy (close to the actual situation on site).

[0049] The influence estimation unit 14 may further estimate the degree to which a change in state in a particular area has an impact on the overall image. For example, the influence estimation unit 14 estimates the degree of impact such that the greater the change in state in a particular area, the greater the impact on the overall image, and the smaller the change in state in a particular area, the smaller the impact on the overall image.

[0050] Furthermore, the impact estimation unit 14 may predict future conditions in a predetermined area based on state changes, location information, and event prediction information. This allows the image generation unit 15 to generate an overall image that predicts changes in areas where events have not yet occurred. In this way, the image generation unit 15 can acquire the prediction results as visually perceptible information.

[0051] Furthermore, the state change determination unit 13 and the influence estimation unit 14 may perform their respective processes in parallel if multiple partial images are acquired at the same time by the partial image acquisition unit 12. By performing parallel processing on multiple partially images acquired simultaneously, the image generation unit 15 can use more processing results as input information when generating a pseudo-whole image. As a result, the image generation unit 15 can generate a pseudo-whole image with higher accuracy.

[0052] The image generation unit 15 is an example of the image generation unit 105 described above. Based on the overall image, partial images, and the affected area, the image generation unit 15 generates a pseudo-overall image that simulates the overall image at the time the partial image was captured. The pseudo-overall image is a simulated overall image that was not actually captured.

[0053] The real-world region corresponding to the pseudo-whole image may coincide with the shooting range of the whole image. The pseudo-whole image complements the gaps between multiple whole images actually captured by a fixed-point camera 5, for example. By generating a pseudo-whole image in the image generation unit 15, the field monitoring system 2 can check the site conditions at short time intervals, even when the interval between capturing the whole images is long. As a result, the field monitoring system 2 can check the site conditions during time periods when it was difficult to check the site conditions due to constraints on the shooting interval.

[0054] The image generation unit 15 may generate a pseudo-whole image based on the degree of influence that the state change has on the whole image, which has been estimated by the influence estimation unit 14. For example, the image generation unit 15 generates a pseudo-whole image such that the change from the whole image in the affected area increases as the degree of influence increases. Similarly, the image generation unit 15 generates a pseudo-whole image such that the change from the whole image in the affected area decreases as the degree of influence decreases.

[0055] The image generation unit 15 may also generate a pseudo-whole image with a risk level added to the image area of ​​the affected range. For example, the image generation unit 15 generates a pseudo-whole image with the risk level set by the risk level setting unit 16 added to the vicinity of the image area of ​​the affected range. The placement of the risk level display is arbitrary. The risk level may be placed overlaid on the image area of ​​the affected range.

[0056] The image generation unit 15 may further generate predicted images based on the prediction results from the influence estimation unit 14. For example, the image generation unit 15 may generate predicted images by reflecting the prediction results in the current overall image. The image generation unit 15 may generate predicted images for a range corresponding to the overall image, or it may generate predicted images for a range narrower than that range.

[0057] The risk level setting unit 16 predicts the damage occurring in a predetermined area based on the determination result in the state change determination unit 13, and sets the risk level based on the prediction result. The risk level indicates the magnitude of the risk that the predetermined area will become dangerous. For example, the risk level setting unit 16 sets the risk level based on the event estimated by the state change determination unit 13. This allows the image generation unit 15 to visualize the risk level for the affected area.

[0058] The communication unit 17 is a communication interface for communication via wired or wireless means. The communication unit 17 transmits and receives data between the field monitoring system 2, the terminal device 3, the drive recorder 4, and the fixed-point camera 5.

[0059] The configuration of the image processing system 10 has been described above. Note that the configuration of the image processing system 10 described above is merely an example and can be modified as appropriate. For example, if some or all of the components of the image processing system 10 are implemented by multiple information processing devices or circuits, these devices may be centrally located or distributed. For example, the information processing devices or circuits may be implemented in a form where each is connected via a communication network, such as a client-server system or a cloud computing system. Furthermore, the functions of the image processing device 1 may be provided in SaaS (Software as a Service) format.

[0060] For example, although Figure 4 shows the overall image acquisition unit 11 and the partial image acquisition unit 12 separately, the overall image acquisition unit 11 and the partial image acquisition unit 12 may be configured as a single unit.

[0061] (Specific example) A specific example of the processing of the image processing system 10 will be explained with reference to Figures 5 and 6. Figure 5 is a schematic diagram showing an overview of the image processing in the image processing system 10. Figure 6 is a diagram that explains the image processing b1 shown in Figure 5 in detail.

[0062] First, an overview of the image processing in the image processing system 10 will be explained with reference to Figure 5. Figure 5 (A) in the upper section shows an example of the overall image output when the image processing according to this disclosure is not performed. Figure 5 (B) in the lower section shows examples of the overall image and pseudo-overall image output when the image processing according to this disclosure is performed. In Figure 5, the horizontal axis represents time.

[0063] As shown in Figure 5(A), the overall image acquisition unit 11 of the image processing device 1 acquires the overall image F0 taken at time t0. The overall image acquisition unit 11 also acquires the overall image F0 taken at time t X Overall image F taken at X Obtain the overall image F0 and F Xis an image captured by, for example, the fixed-point camera 5. As described above, there are physical restrictions and the like in photographing the monitoring target area, and it may be difficult to shorten the photographing interval. In such a case, the entire image acquisition unit 11 cannot acquire the entire image at short time intervals.

[0064] For example, assume that the time between time t0 and time t X is 2 hours. The image processing apparatus 1 transmits the entire image F0 to the on-site monitoring system 2, and then 2 hours later, transmits the entire image F X to the on-site monitoring system 2. The on-site monitoring system 2 cannot acquire the on-site images during these 2 hours. Therefore, the monitor at the monitoring center cannot grasp the changing on-site situation moment by moment, and may not be able to respond promptly in case of an event such as a disaster.

[0065] In contrast, in (B) of FIG. 5, at time t X between time t0 and time t X-N to time t X-1 , a plurality of partial images P X-N to P X-1 are being photographed. When performing the image processing according to the present disclosure, the image generation unit 15 generates pseudo-entire images F’ X-N to F’ X-1 for each timing when the partial image is photographed from time t X-N to time t X-1 . Specifically, first, the image generation unit 15 performs image processing b1 using the entire image F0 photographed at time t0 and the partial image P X-N photographed at time t X-N . The image processing b1 will be described later.

[0066] By performing the image processing b1, the image generation unit 15 generates a pseudo-entire image F’ X-N from the entire image F0 and the partial image P X-N . Similarly, the image generation unit 15 uses the entire image F0 photographed at time t0 and the partial image photographed at time t X-1Image processing b1 is performed using partial images taken at multiple timings up to time t, and a pseudo-whole image is generated for each timing when a partial image was taken. For example, the image generation unit 15, at time t X-N+1 , t X-N+2 , t X-N+3 (None of which are shown), partial image P taken at... X-N+1 , P X-N+2 , P X-N+3 Image processing b1 is performed using each of the above and the overall image F0. As a result, the image generation unit 15 generates a pseudo-overall image F' X-N+1 , F' X-N+2 , F' X-N+3 , ...generates.

[0067] The image generation unit 15 processes data from time t0 to time t X During this time, any number of pseudo-whole images may be generated. For example, the image generation unit 15 may use all of the acquired partial images to generate a pseudo-whole image corresponding to each of the acquired partial images, or it may use some of the acquired partial images to generate a pseudo-whole image. If a sufficient number of partial images (for example, more than a predetermined number within a predetermined time) can be acquired, the image generation unit 15 will generate a pseudo-whole image at time t X-1 In the pseudo-whole image F' X-1 Until the desired image is generated, pseudo-whole images may be generated at predetermined time intervals.

[0068] Note: Partial image P X-N From P X-1 These are images captured by, for example, terminal device 3 or drive recorder 4, and are acquired by partial image acquisition unit 12. Partial image P X-N From P X-1 Each of these photos may have a different photographer and shooting location.

[0069] Refer to Figure 6 to explain image processing b1. (1) Identify the overlapping parts First, the partial image acquisition unit 12 acquires the captured image taken from the terminal device 3 or the drive recorder 4. Here, the partial image acquisition unit 12 acquires the captured image at time t X-NIn this case, the captured image is acquired from the terminal device 3. The partial image acquisition unit 12 compares the captured image with the overall image F0 acquired by the overall image acquisition unit 11 and identifies the overlapping portion. For example, the partial image acquisition unit 12 identifies the overlapping portion using any image recognition technology. In the example in Figure 6, the overlapping portion is a partial region f 01 This is shown.

[0070] The partial image acquisition unit 12 captures the overlapping portion of the captured image as a partial image P X-N The image is acquired as follows: If the overall image F0 encompasses the entire captured image, the partial image acquisition unit 12 acquires the entire captured image as partial image P X-N The image is acquired as follows. If the overall image F0 includes a part of the captured image, the partial image acquisition unit 12 acquires the overlapping part as partial image P X-N You may acquire it as such.

[0071] (2) Determine the state change Next, the state change determination unit 13 determines a partial region f of the overall image F0. 01 Corresponding region and partial image P X-N Compare and a portion of region f 01 The state change is determined in a certain region f. The state change determination unit 13 may use AI to determine the state change. For example, the state change determination unit 13 determines a state change in a certain region f. 01 The system determines whether or not there is a state change in the region f. If it is determined that there is no state change, the image processing b1 is terminated. If it is determined that there is a state change, the state change determination unit 13 determines the state of a portion of region f. 01 The content of the state change in this case may be determined.

[0072] The state change determination unit 13 determines a partial region f 01 The system may estimate the events that occurred in a certain region f and determine the state change based on those events. For example, the state change determination unit 13 may determine a state change in a certain region f 01 The system estimates events that include at least one of the following: changes in weather, changes in water level, crustal deformation, changes in the shape of buildings, and changes in the position of objects, and determines changes in state based on the estimated events.

[0073] Furthermore, the state change determination unit 13 determines a portion of region f 01Based on the location information, a portion of area f 01 The probability of an event occurring in a certain region f may be estimated, and a state change may be determined based on that probability. Furthermore, the state change determination unit 13 may determine a state change in a certain region f 01 The probability of an event occurring may be estimated using event prediction information associated with the location information of a certain region f. 01 Weather forecast information, disaster prediction information, hazard maps, crustal survey information, or some area f 01 This includes specifications and other information about the buildings included.

[0074] (3) Estimate the scope of the impact Next, the influence estimation unit 14 determines the influence range f of the state change on the overall image F0 based on the determination result in the state change determination unit 13. 02 The impact estimation unit 14 estimates the impact range f for each event. 02 It is also possible to estimate this.

[0075] The impact estimation unit 14 uses AI to determine the range of influence f 02 It is possible to estimate this. For example, the influence estimation unit 14 performs estimation using a pre-trained model. The pre-trained model is pre-trained to output the influence range by inputting the content of the state change and the position of the partial image.

[0076] The influence estimation unit 14 inputs the content of the state change into the trained model. The influence estimation unit 14 also inputs the partial image P X-N Enter the position of the partial image P. X-N The position of the partial image P in the overall image F0 is X-N It can be the relative position, or the partial image P X-N It may also be the real-world location corresponding to it. The influence estimation unit 14 uses the influence range f output from the trained model. 02 The influence estimation unit 14 may further estimate the degree to which the state change has an effect on the overall image.

[0077] (4) Generate a pseudo-whole image Next, the image generation unit 15 generates the overall image F0 and the partial image P. X-N , and the scope of influence f 02Based on, partial image P X-N The time when it was taken X-N F' is a pseudo-whole image that simulates the overall image in the context of [the system]. X-N Generates.

[0078] The image generation unit 15 uses AI to generate a pseudo-whole image F' X-N It may generate an image. For example, the image generation unit 15 generates an image within the influence range f 02 , the content of the state change, and partial image P X-N By inputting the position into the trained model, a pseudo-whole image F' is generated. X-N The trained model is pre-trained to output a pseudo-whole image by taking the affected area, the content of the state change, and the location of the partial image as input. Pseudo-whole image F' X-N This is a partial image P X-N It can be included as is, or the processed partial image P. X-N It may include.

[0079] Furthermore, the image generation unit 15 generates a pseudo-whole image F' based on the degree of influence. X-N The image generation unit 15 may also generate a pseudo-whole image F' with a risk level added to the image region of the affected area. X-N You may generate this.

[0080] In this way, the image generation unit 15 generates the overall image F0 and F X Between these, the pseudo-whole image F' X-N This generates data. As a result, the field monitoring system 2 can grasp the situation at the site at shorter time intervals, allowing the monitor to respond quickly when an event occurs. For example, if the situation at the site changes significantly, even in a short period of time, the monitor can start taking action immediately.

[0081] In the above description, the image processing device 1 performed some processing using AI, but the image processing device 1 may also perform these processes using methods other than AI. Furthermore, although not explained here, the impact estimation unit 14 may predict future situations, and the image generation unit 15 may generate predicted images. By transmitting the predicted images along with the pseudo-overall image to the on-site monitoring system 2, the on-site situation can be more easily understood.

[0082] (Processing by image processing device 1) Next, with reference to Figure 7, the processes performed by the image processing device 1 will be explained. Figure 7 is a flowchart showing the processes performed by the image processing device 1.

[0083] First, the overall image acquisition unit 11 acquires an overall image of a predetermined area (S11). For example, the overall image acquisition unit 11 acquires an overall image from the fixed-point camera 5 at predetermined time intervals. Next, the partial image acquisition unit 12 acquires a partial image (S12). The partial image acquisition unit 12 may acquire captured images from the terminal device 3 or the drive recorder 4, identify the overlapping portion between the overall image and the captured image, and acquire the overlapping portion as a partial image.

[0084] Next, the state change determination unit 13 determines whether there is a state change in a part of the image based on the overall image and the partial image (S13). The influence estimation unit 14 determines whether or not there is a state change in the part of the image (S14). If it is determined that there is no state change (NO in S14), the process ends. If it is determined that there is a state change (YES in S14), the influence estimation unit 14 estimates the range and degree of influence of the state change on the overall image based on the determination result of the state change determination unit 13 (S15).

[0085] Next, the image generation unit 15 determines whether or not there is an effect outside the range of the partial image (partial area) in the overall image (S16). If it is determined that there is an effect (YES in S16), the image generation unit 15 generates a pseudo-overall image that reflects the state change based on the overall image, the partial image, and the affected area (S17). If it is determined that there is no effect (NO in S16), the image generation unit 15 combines the partial image with the overall image (S18).

[0086] As described above, according to the image processing system 10 of this disclosure, the image processing device 1 acquires an overall image of a predetermined region and a partial image of a part of the predetermined region taken at a different timing than the overall image. Based on the overall image and the partial image, the image processing device 1 determines a change in state in the partial region and estimates the range of influence of the change in state on the overall image based on the determination result. Based on the overall image, the partial image and the range of influence, the image processing device 1 generates a pseudo-overall image that pseudoly represents the overall image at the time the partial image was taken.

[0087] With this configuration, the image processing device 1 can generate a pseudo-whole image at a time when a full image is not captured, for example, using images posted on social media or images from a dashcam. This allows the image processing system 10 to improve the real-time nature of the information (image information) and enhance the comprehensiveness of the information. Therefore, the image processing system 10 can enable the on-site monitoring system 2 to grasp the status of the monitored area at shorter time intervals.

[0088] <Example Hardware Configuration> Each functional component of the image processing device 100 and the image processing device 1 (hereinafter referred to as "image processing device 100, etc.") may be implemented by hardware that realizes each functional component (e.g., hardwired electronic circuits, etc.) or by a combination of hardware and software (e.g., a combination of electronic circuits and a program that controls them, etc.). The case in which each functional component of the image processing device 100, etc. is implemented by a combination of hardware and software will be further explained below.

[0089] Figure 8 is a block diagram illustrating the hardware configuration of a computer 900 that implements the image processing device 100, etc. The computer 900 may be a dedicated computer designed to implement the image processing device 100, etc., or it may be a general-purpose computer. The computer 900 may also be a portable computer such as a smartphone or tablet terminal.

[0090] For example, by installing a predetermined application on the computer 900, the various functions of the image processing device 100 and other devices are realized on the computer 900. The above application consists of a program for realizing the functional components of the image processing device 100 and other devices.

[0091] Computer 900 includes a bus 902, a processor 904, memory 906, a storage device 908, an input / output interface 910, and a network interface 912. The bus 902 is a data transmission path for the processor 904, memory 906, storage device 908, input / output interface 910, and network interface 912 to send and receive data to and from each other. However, the method of connecting the processor 904 and other components to each other is not limited to bus connection.

[0092] The processor 904 is a variety of processors such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field-Programmable Gate Array), or quantum processor (quantum computer control chip). The memory 906 is the main memory, implemented using RAM (Random Access Memory), etc. The storage device 908 is the auxiliary storage, implemented using a hard disk, SSD (Solid State Drive), memory card, or ROM (Read Only Memory), etc.

[0093] The input / output interface 910 is an interface for connecting the computer 900 with input / output devices. For example, input devices such as keyboards and output devices such as display devices are connected to the input / output interface 910.

[0094] Network interface 912 is an interface for connecting computer 900 to a network. This network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0095] The storage device 908 stores programs that implement each functional component of the image processing device 100 (programs that implement the aforementioned applications). The processor 904 reads these programs into memory 906 and executes them to implement each functional component of the image processing device 100.

[0096] Each processor executes one or more programs containing a set of instructions for causing the computer to perform an algorithm. These programs, when loaded into the computer, contain a set of instructions (or software code) for causing the computer to perform one or more functions described in the embodiments. The programs may be stored in various types of non-transitory computer-readable medium or tangible storage medium. Examples, but not limited to, include non-transitory computer-readable medium or tangible storage medium, such as RAM, ROM, flash memory, SSD or other memory technologies, CD-ROM, DVD (Digital Versatile Disc), Blu-ray® disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The programs may also be transmitted over various types of transient computer-readable medium or communication medium. Examples, but not limited to, include transient computer-readable medium or communication medium, such as electrical, optical, acoustic or other forms of propagating signals.

[0097] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure can be made as can be understood by those skilled in the art within the scope of the present disclosure. For example, although the above description has been based primarily on the assumption of monitoring disasters, etc., the present disclosure is not limited to this. The present disclosure can be applied in any environment, and each embodiment can be combined with other embodiments as appropriate.

[0098] Each drawing is merely illustrative to illustrate one or more embodiments. Each drawing may be associated with one or more other embodiments rather than with only one specific embodiment. As those skilled in the art will understand, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings, for example, to create embodiments not explicitly shown or described. Not all features or steps shown in any one drawing to illustrate an exemplary embodiment are necessarily required, and some features or steps may be omitted. The order of steps shown in any of the drawings may be changed as appropriate.

[0099] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0100] (Note 1) A whole image acquisition unit that acquires a whole image of a predetermined area, A partial image acquisition unit that acquires a partial image of a portion of the predetermined region at a different timing from the overall image, A state change determination unit determines a state change in a partial region based on the overall image and the partial image, An influence estimation unit estimates the range of influence of the state change on the overall image based on the determination result in the state change determination unit, The system includes an image generation unit that generates a pseudo-whole image that pseudoly represents the whole image at the time the partial image was captured, based on the whole image, the partial image, and the affected area. Image processing device. (Note 2) The partial image acquisition unit acquires a captured image taken from a position different from the shooting position of the overall image, identifies the overlapping portion between the overall image and the captured image, and acquires the overlapping portion as the partial image. The image processing apparatus described in Appendix 1. (Note 3) The state change determination unit estimates an event that occurred in the partial region and determines the state change based on the event. The influence estimation unit estimates the influence range for each event. The image processing apparatus described in Appendix 1 or 2. (Note 4) The state change determination unit estimates the event which includes at least one of the following in a given area: changes in weather, changes in water level, crustal deformation, changes in the shape of a building, and changes in the position of an object. The image processing apparatus described in Appendix 3. (Note 5) The influence estimation unit further estimates the degree to which the state change has an effect on the overall image. The image generation unit generates the pseudo-whole image based on the degree of the influence. An image processing apparatus as described in any one of the appendices 1 to 4. (Note 6) The state change determination unit estimates the probability of the event occurring in the partial region based on the location information of the partial region, and determines the state change based on the probability of occurrence. An image processing apparatus as described in any one of the items 3 to 5 of the appendix. (Note 7) The state change determination unit estimates the probability of the event occurring using the event prediction information associated with the location information, The aforementioned event prediction information includes at least one of the following: weather forecast information for the aforementioned area, disaster prediction information, hazard maps, crustal survey information, and building specification information. The image processing apparatus described in Appendix 6. (Note 8) The system further includes a risk level setting unit that predicts damage occurring in a predetermined area based on the determination result in the state change determination unit and sets the risk level based on the prediction result, The image generation unit generates the pseudo-whole image by adding the risk level to the image region of the affected area. An image processing apparatus as described in any one of the appendices 1 to 7. (Note 9) The influence estimation unit predicts the future situation in the predetermined region based on the state change, the location information, and the event prediction information. The image generation unit further generates a predicted image based on the prediction results from the influence estimation unit. The image processing apparatus described in Appendix 7 or 8. (Note 10) The state change determination unit and the influence estimation unit perform their respective processes in parallel when multiple partial images are acquired at the same time by the partial image acquisition unit. An image processing apparatus as described in any one of the appendices 1 to 9. (Note 11) A whole image acquisition step to acquire a whole image of a predetermined area, A partial image acquisition step involves acquiring a partial image of a portion of the predetermined region, which is captured at a different timing than the overall image. A state change determination step in which a state change in a part of the region is determined based on the overall image and the partial image, An influence estimation step is performed to estimate the range of influence of the state change on the overall image based on the determination result in the state change determination step, The image generation step includes generating a pseudo-whole image that pseudoly represents the whole image at the time the partial image was captured, based on the whole image, the partial image, and the affected area. Image processing methods. (Note 12) A whole image acquisition step to acquire a whole image of a predetermined area, A partial image acquisition step involves acquiring a partial image of a portion of the predetermined region, which is captured at a different timing than the overall image. A state change determination step in which a state change in a part of the region is determined based on the overall image and the partial image, An influence estimation step is performed to estimate the range of influence of the state change on the overall image based on the determination result in the state change determination step, The computer is instructed to perform an image generation step, which involves generating a pseudo-whole image that pseudoly represents the whole image at the time the partial image was captured, based on the whole image, the partial image, and the affected area. program.

[0101] Some or all of the elements (e.g., configuration and function) described in Appendices 2 to 10 that are dependent on Appendice 1 may also be dependent on Appendices 11 and 12 in the same manner as those described in Appendices 2 to 10. Some or all of the elements described in any appendice may be applicable to various hardware, software, recording means, systems, and methods for recording software. [Explanation of Symbols]

[0102] 1 Image processing device 2. On-site monitoring system 3 Terminal devices 4. Dashcam 5 Fixed-point cameras 10 Image Processing Systems 11 Overall image acquisition unit 12 Partial image acquisition unit 13 State change determination unit 14. Impact Estimation Section 15 Image generation unit 16. Risk Level Setting Section 17 Communications Department 100 Image Processing Devices 101 Overall image acquisition unit 102 Partial image acquisition unit 103 State change determination unit 104 Impact Estimation Section 105 Image generation unit 900 Computers 902 Bus 904 Processor 906 memory 908 Storage Devices 910 Input / Output Interface 912 Network Interface F' X-1 , F' X-NSuspected overall portrait f 01 A field f 02 Influence range F 0、 F X Overall portrait N ネットワーク P X-1、 P X-N partial portrait t 0、 t X t X-1 t X-N Time

Claims

1. A whole image acquisition unit that acquires a whole image of a predetermined area, A partial image acquisition unit that acquires a partial image of a portion of the predetermined region at a different timing from the overall image, A state change determination unit determines a state change in a partial region based on the overall image and the partial image, An influence estimation unit estimates the range of influence of the state change on the overall image based on the determination result in the state change determination unit, The system includes an image generation unit that generates a pseudo-whole image that pseudoly represents the whole image at the time the partial image was captured, based on the whole image, the partial image, and the affected area. Image processing device.

2. The partial image acquisition unit acquires a captured image taken from a position different from the shooting position of the overall image, identifies the overlapping portion between the overall image and the captured image, and acquires the overlapping portion as the partial image. The image processing apparatus according to claim 1.

3. The state change determination unit estimates an event that occurred in the partial region and determines the state change based on the event. The influence estimation unit estimates the influence range for each event. The image processing apparatus according to claim 1 or 2.

4. The influence estimation unit further estimates the degree to which the state change has an effect on the overall image. The image generation unit generates the pseudo-whole image based on the degree of the influence. The image processing apparatus according to claim 1 or 2.

5. The state change determination unit estimates the probability of the event occurring in the partial region based on the location information of the partial region, and determines the state change based on the probability of occurrence. The image processing apparatus according to claim 3.

6. The state change determination unit estimates the probability of the event occurring using the event prediction information associated with the location information, The aforementioned event prediction information includes at least one of the following: weather forecast information for the aforementioned area, disaster prediction information, hazard maps, crustal survey information, and building specification information. The image processing apparatus according to claim 5.

7. The system further includes a risk level setting unit that predicts damage occurring in a predetermined area based on the determination result in the state change determination unit and sets the risk level based on the prediction result, The image generation unit generates the pseudo-whole image by adding the risk level to the image region of the affected area. The image processing apparatus according to claim 1 or 2.

8. The influence estimation unit predicts the future situation in the predetermined region based on the state change, the location information, and the event prediction information. The image generation unit further generates a predicted image based on the prediction results from the influence estimation unit. The image processing apparatus according to claim 6.

9. A whole image acquisition step to acquire a whole image of a predetermined area, A partial image acquisition step involves acquiring a partial image of a portion of the predetermined region, which is captured at a different timing than the overall image. A state change determination step in which a state change in a part of the region is determined based on the overall image and the partial image, An influence estimation step is performed to estimate the range of influence of the state change on the overall image based on the determination result in the state change determination step, The image generation step includes generating a pseudo-whole image that pseudoly represents the whole image at the time the partial image was captured, based on the whole image, the partial image, and the affected area. Image processing methods.

10. A whole image acquisition step to acquire a whole image of a predetermined area, A partial image acquisition step involves acquiring a partial image of a portion of the predetermined region, which is captured at a different timing than the overall image. A state change determination step in which a state change in a part of the region is determined based on the overall image and the partial image, An influence estimation step is performed to estimate the range of influence of the state change on the overall image based on the determination result in the state change determination step, The computer is instructed to perform an image generation step, which involves generating a pseudo-whole image that pseudoly represents the whole image at the time the partial image was captured, based on the whole image, the partial image, and the affected area. program.

Images