Display method, control device, display device, and program

Abstract

This display method, according to one embodiment of the present invention, is for displaying changes in the state of an object, on the basis of a plurality of images which are generated by successive imaging of the object. The display method includes: a step for extracting, from each of the plurality of images, pixels which are located at a predetermined position coordinate; and a step for displaying a pixel group, in which the pixels are arrayed, in the order in which the plurality of images were taken.

Description

[Technical Field]

[0001] The present invention relates to a display method, a control device, a display device, and a program. [Background technology]

[0002] Conventionally, in order to maintain and manage an object such as a plant, it has been common to visually check changes in the state of the object over time by continuously capturing images (video) of the object. For example, Patent Document 1 describes a display device that can display a video of the state of a building after aging. When displaying the video representing the building after aging, the display device generates the video using post-change coloring data corresponding to the pattern or color of each area of the building after aging. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2002-117416 Summary of the Invention [Problem to be solved by the invention]

[0004] When checking changes in the state of an object using video, for example, in order to identify a time period or area where the state change is large, the user must carefully watch the video being played, i.e., multiple images displayed consecutively. Even if the user is able to identify an area or time period where the state change is large in this way, the user must play the video again each time they want to check the state change in that area or time period. As such, checking state changes using video is laborious and inefficient.

[0005] Therefore, an object of the present invention is to provide a display method, a control device, a display device, and a program that enable a change in the state of an object to be easily grasped. [Means for solving the problem]

[0006] A display method according to one aspect of the present invention is a display method for displaying changes in the state of an object based on a plurality of images generated by successively capturing images of the object, and includes the steps of extracting pixels located at predetermined position coordinates from each of the plurality of images, and displaying a set of pixels in which the pixels are arranged according to the order in which the plurality of images were captured.

[0007] According to this aspect, pixels located at predetermined position coordinates are extracted from the multiple images, and these pixels are arranged in the order in which they were captured, making it possible to easily grasp changes in the state of the object at the predetermined position coordinates over time.

[0008] A control device according to another aspect of the present invention is a control device that controls the display of changes in the state of an object based on a plurality of images generated by successively capturing images of the object, and includes a pixel set generation unit that extracts pixels located at predetermined position coordinates from each of the plurality of images and generates a pixel set in which the pixels are arranged according to the order in which the plurality of images were captured, and a display device control unit that displays the generated pixel set on a display device.

[0009] A display device according to another aspect of the present invention displays a set of pixels in which pixels located at predetermined position coordinates in each of a plurality of images generated by successively capturing an object are arranged according to the order in which the plurality of images are captured.

[0010] A program according to another aspect of the present invention causes a computer to execute the steps of extracting pixels located at predetermined position coordinates from each of a plurality of images generated by successively capturing images of an object, and displaying a set of pixels in which the pixels are arranged according to the order in which the plurality of images were captured. [Effects of the Invention]

[0011] According to the present invention, it is possible to provide a display method, a control device, a display device, and a program that enable a change in the state of an object to be easily grasped. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an overview of a display system 1 according to the present embodiment. [Figure 2] 1 is a schematic diagram showing an example of the functional configuration of a display system 1 according to the present embodiment. [Figure 3] FIG. 2 is a schematic diagram showing an example of the functional configuration of an image processing unit 30 according to the present embodiment. [Figure 4] FIG. 2 is a schematic diagram showing an example of the functional configuration of a display control unit 70 according to the present embodiment. [Figure 5] 2 is a schematic diagram showing an example of a state change display screen 200 displayed on the display device 90 according to the present embodiment. FIG. [Figure 6] 4 is an operational flow diagram showing an example of an operational process performed by the display system 1 according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] A preferred embodiment of the present invention will be described with reference to the accompanying drawings. (Note that in each drawing, components with the same reference numerals have the same or similar configurations.)

[0014] [First embodiment] 1 is a diagram illustrating an overview of a display system 1 according to this embodiment. The display system 1 displays changes in the state of an object based on multiple images generated by successively capturing images of the object. Here, the "multiple images" may be, for example, "captured images" that are images obtained by capturing images of the object from a predetermined fixed position, "processed images" that are images obtained by performing predetermined processing such as semantic region division on the captured images, or "difference images" that are images of the differences between multiple captured images or multiple processed images.

[0015] Each of the multiple images is associated with "time information (time stamp)" corresponding to the order of imaging, such as the imaging time, and is stored in a predetermined storage unit. If the multiple images are the above-mentioned captured images or processed images, the time information may be the imaging time of the image. Also, if the multiple images are the above-mentioned difference images, the time information may be time information associated with any of the images that form the basis of the difference image, or may be a value calculated based on time information associated with each of the multiple images that form the basis of the difference image (for example, an average or median of the multiple time information).

[0016] FIG. 1 shows a rectangular parallelepiped C. Here, the rectangular parallelepiped C schematically represents a set of images that are the basis of pixel sets displayed by the display system 1, arranged in the order of time information associated with each image (i.e., in the order of capture). Of the time information associated with the plurality of images (images included in the rectangular parallelepiped C), the oldest time information is designated as the start time T S The latest time information (including time information later than the current time) is set as the end time T E In other words, the time information associated with the plurality of images is the start time T S After that, and end time T E It will be the same as before.

[0017] In Fig. 1, three images P1, P2, and P3 of the multiple images are explicitly depicted. Images P1, P2, and P3 are associated with time information T1, T2, and T3, respectively. For ease of explanation, only images P1, P2, and P3 of the multiple images are explicitly depicted in Fig. 1, and other images are omitted, but the number of images included in the multiple images handled by display system 1 is not limited to three and may be any number. Hereinafter, when individual images such as images P1, P2, and P3 are not particularly distinguished from one another, each of the multiple images may be collectively referred to as image P.

[0018] In Fig. 1, as an example, the multiple images included in a rectangular parallelepiped C are represented as processed images obtained by performing semantic region segmentation processing for detecting damaged regions on captured images obtained by capturing images of an object. Therefore, for example, as shown in Fig. 1, three images P1, P2, and P3 each contain damaged regions D1, D2, and D3 detected by the semantic region segmentation processing.

[0019] An outline of a method for displaying changes in the state of an object using the display system 1 will be described. The axis parallel to the horizontal direction of the image P is the X axis, and the axis parallel to the vertical direction of the image P is the Y axis. That is, the position of each pixel included in the image P is specified by two position coordinates, the X coordinate and the Y coordinate. Each pixel contains information (pixel information) indicating the magnitude of a predetermined parameter that the pixel has, such as luminance information or color difference information. As shown in FIG. 1, the range of the X coordinate of the image P included in the rectangular parallelepiped C is equal to or greater than X1 and equal to or less than X2.

[0020] The display system 1 first extracts, from each image P, pixels located on "pixel extraction coordinates," which are predetermined position coordinates common to each image P. Here, the "pixel extraction coordinates" may be a set of at least one position coordinate specified by an X coordinate and a Y coordinate. The pixel extraction coordinates may also be arbitrarily set by the user. For example, the "pixel extraction coordinates" may be one or more sets of position coordinates that constitute an arbitrary point, line (straight line or curve), or area. In FIG. 1, as an example, the pixel extraction coordinates are (X, Y)=(X1, Y L ) and is set as a straight line parallel to the X-axis. For example, straight line L1 in image P1, straight line L2 in image P2, and straight line L3 in image P3 correspond to pixel extraction coordinates. Then, pixels (pixel rows) arranged on straight line L1, which is the pixel extraction coordinate, are extracted from image P1, pixels (pixel rows) arranged on straight line L2, which is the pixel extraction coordinate, are extracted from image P2, and pixels (pixel rows) arranged on straight line L3, which is the pixel extraction coordinate, are extracted from image P3.

[0021] Next, the display system 1 displays the extracted pixel (pixel row) at the start time TS From end time T E A set of pixels (pixel set) arranged in accordance with the order of capturing each image P is generated and displayed. The symbol S in FIG. 1 indicates the pixel set. The vertical direction of the pixel set S corresponds to the X axis, which is the horizontal axis of the image P, and the horizontal direction of the pixel set S corresponds to the time information (start time T S From end time T E For example, as shown in FIG. 1, pixel strings L1, L2, and L3 extracted from images P1, P2, and P3 are shown at the positions of time information T1, T2, and T3 in pixel set S. Also, pixel set S includes damaged area portions S, which are a set of damaged area portions of each image P. D Contains:

[0022] In this way, pixel set S is constructed by extracting pixels located at predetermined position coordinates (pixel extraction coordinates) from multiple images and arranging these pixels in the order of capture, making it possible to grasp at a glance (from a single pixel set) the change in state of the object at the predetermined position coordinates (pixel extraction coordinates) over time.

[0023] FIG. 2 is a schematic diagram showing an example of the functional configuration of a display system 1 according to this embodiment. The display system 1 is a system that displays changes in the state of an object, for example, based on multiple images generated by successively capturing images of the object. The display system 1 includes, for example, an imaging device control unit 10, an imaging device 20, an image processing unit 30, an actual image DB 40, a predicted image generation unit 50, a predicted image DB 60, a display control unit 70, an operation unit 80, and a display device 90. These functional units included in the display system 1 may be configured by one or more computers. Here, the computer may be, for example, an information processing device configured by a processor, a memory, etc.

[0024] The imaging device control unit 10 may be configured with one or more computers, and controls operations such as movement and imaging of the imaging device 20. The imaging device control unit 10 also acquires, from the imaging device 20, an image of an object generated by the imaging device 20.

[0025] The imaging device 20 may be configured as a camera or the like, and generates an image by capturing an image of an object under the control of the imaging device control unit 10. The object is not particularly limited, but may be, for example, the entirety or part of a large facility such as a furnace wall. When capturing an image, the imaging device 20 may be fixed at a predetermined imaging position or may be held by the operator. The imaging device 20 may also be provided with a moving means for moving under the control of the imaging device control unit 10 or the like, and may be moved by the moving means to change the imaging position. The configuration of the moving means is not particularly limited, but may be, for example, a flying means such as a drone for moving arbitrarily through space, or a traveling means such as a slider for moving along the object.

[0026] The image processing unit 30 performs various processes on the image of the object generated by the imaging device 20, and stores the image in the actual image DB 40. The configuration of the image processing unit 30 will be described later.

[0027] The predicted image generation unit 50 generates predicted images for each future time point based on the multiple images stored in the actual image DB 40, and stores the predicted images in the predicted image DB 60 in association with time information (timestamps) corresponding to the order of image capture, such as future image capture times. The predicted images may be generated by inputting the multiple images stored in the actual image DB 40 into a predetermined prediction model using an algorithm such as multiple regression analysis. The prediction model may be a trained model obtained by performing machine learning using any images, such as the multiple images stored in the actual image DB 40, as training data. The types of predicted images may be the same as the types of images stored in the actual image DB 40. Specifically, the types of predicted images may be captured images (images obtained by continuously capturing images of an object from a predetermined fixed position), images (processed images) obtained by performing a predetermined processing process, such as semantic region segmentation, on captured images, or differential images between multiple captured images or multiple processed images.

[0028] The real image DB 40 stores multiple images generated by successively capturing images of an object in association with time information (time stamps) corresponding to the order of capturing images, such as the time of capturing images. The multiple images may be, for example, images (captured images) obtained by capturing images of the object from a predetermined fixed position, images (processed images) obtained by performing predetermined processing such as semantic region division on the captured images, or differential images between the multiple captured images or the multiple processed images.

[0029] The time information associated with each image may be, for example, the time of capture of the image when the multiple images are the above-mentioned captured images or processed images. Also, when the multiple images are the above-mentioned difference images, the time information associated with each image may be the time information associated with any of the images that form the basis of the difference image, or may be a value calculated based on the time information associated with each of the multiple images that form the basis of the difference image (for example, the average or median of the multiple time information).

[0030] The display control unit 70 is an example of a control device, and generates various types of display data based on images stored in the actual image DB 40 and / or the predicted image DB 60, and supplies the data to the display device 90 to display various screens. The configuration of the display control unit 70 will be described later.

[0031] The operation unit 80 may be any device that allows a user to perform operations (click, tap, swipe, pinch, drag, drop, etc.), such as a touchpad, keyboard, button, mouse, etc. When operated by a user, the operation unit 80 generates a signal corresponding to the operation and supplies the signal to the display control unit 70.

[0032] The display device 90 may be any device capable of displaying videos, images, etc., such as a liquid crystal display, an organic EL (Electro-Luminescence) display, etc. The display device 90 displays various screens based on display data supplied from the display control unit 70.

[0033] 3 is a schematic diagram showing an example of the functional configuration of the image processing unit 30 according to this embodiment. The image processing unit 30 has, for example, a normalization processing unit 31, a processing unit 32, a difference image generation unit 33, and an image composition unit 34. The image processing unit 30 uses at least one of the normalization processing unit 31, the processing unit 32, the difference image generation unit 33, and the image composition unit 34 to perform various processes on the image of the object generated by the imaging device 20 and the images stored in the real image DB 40, and stores the generated images in the real image DB 40.

[0034] The normalization processing unit 31 performs a predetermined normalization process on the image supplied from the imaging device 20. The predetermined normalization process may be performed by, for example, calculating an average value of the luminance value / color difference value of the captured image and correcting the luminance value / color difference value of each captured image data item based on the calculated average value so that the luminance value / color difference value of each captured image data item is approximately the same.

[0035] The processing unit 32 performs a predetermined processing on the normalized image, for example. The type of processing is not particularly limited, but may be, for example, semantic region segmentation processing. The semantic region segmentation processing extracts features from the image and then segments the image into predetermined regions, such as damaged regions, based on the features.

[0036] In addition to semantic region segmentation processing, the processing unit 32 may perform any processing such as histogram conversion, monochrome image conversion, binarization, morphological conversion, noise removal, edge enhancement, enlargement / reduction, filtering, affine transformation, linear transformation, center of gravity measurement, area measurement, circularity measurement, principal axis angle measurement, moment feature calculation, Hough transform, object identification, pattern recognition, template matching, etc.

[0037] The difference image generating unit 33 generates a difference image between multiple images captured at consecutive times (with consecutive timestamps). The multiple images that form the basis of the difference image may be images that have been normalized by the normalization processing unit 31, or may be images that have been processed (such as semantic region segmentation) by the processing unit 32.

[0038] The image composition unit 34 combines multiple images together. For example, multiple images captured at the same time (with the same timestamp) are stitched together based on the position information of each image, thereby generating an image capturing a wide range.

[0039] 4 is a schematic diagram showing an example of the functional configuration of a display control unit 70 according to this embodiment. The display control unit 70 includes, for example, a storage unit 71, an image extraction unit 72, a pixel set generation unit 73, a setting change unit 74, a pixel extraction coordinate determination unit 75, a display device control unit 76, and a storage unit 77.

[0040] The storage unit 71 is configured with a ROM, a RAM, etc., and stores various programs such as driver programs, operating system programs, and application programs, as well as various data. The various programs stored in the storage unit 71 may be installed into the storage unit 71 from a computer-readable portable recording medium such as a CD-ROM or a DVD-ROM using a known setup program. The storage unit 71 stores, for example, programs that enable the display control unit 70 to implement an image extraction unit 72, a pixel set generation unit 73, a setting change unit 74, a pixel extraction coordinate determination unit 75, a display device control unit 76, and a storage unit 77. The storage unit 71 may also store, for example, images (actual images and / or predicted images) extracted by the image extraction unit 72.

[0041] The storage unit 71 also stores any settings related to various screens to be displayed on the display device 90. The setting items are not particularly limited, but may include, for example, the following items: Type of image (captured image, processed image, difference image, etc.) Various parameters of the image (size, aspect ratio, resolution, etc.) Items related to time (start time T S , and the end time T E etc.) Pixel extraction coordinates (may be one or more sets of position coordinates of each point included in a point, line (straight line and curve), area, etc.)

[0042] The image extraction unit 72 extracts actual images and / or predicted images that are the basis of pixel sets from the actual image DB 40 and / or predicted image DB 60. Specifically, the image extraction unit 72 extracts actual images and / or predicted images that are the basis of pixel sets from the actual image DB 40 and / or predicted image DB 60. S From end time T E The image extracting unit 72 extracts images associated with time information before the current time, that is, real images, from the real image DB 40. S From end time T E From among the images, an image associated with time information after the current time, that is, a predicted image, is extracted from the predicted image DB 60.

[0043] The pixel set generation unit 73 generates pixel sets based on the images extracted by the image extraction unit 72. Specifically, the pixel set generation unit 73 extracts pixels arranged on pixel extraction coordinates included in the settings stored in the storage unit 71 from each image (actual image and / or predicted image) extracted by the image extraction unit 72, and then generates pixel sets by arranging the extracted pixels in the order of the time information of the images.

[0044] The setting change unit 74 changes the settings stored in the storage unit 71 in response to operations by the user via the operation unit 80, etc.

[0045] The pixel extraction coordinate determination unit 75 determines pixel extraction coordinates by analyzing the multiple images (actual images and / or predicted images) extracted by the image extraction unit 72. The pixel extraction coordinate determination unit 75 may determine, as pixel extraction coordinates, a portion of the multiple images where the change is greater than other portions. Specifically, for example, the portion where the change in the damaged area identified by the semantic area segmentation process is equal to or greater than a threshold may be determined as pixel extraction coordinates.

[0046] The display device control unit 76 generates display data and supplies it to the display device 90, thereby causing the display device 90 to display various screens. In particular, the display device control unit 76 causes the display device 90 to display a state change display screen. The state change display screen may include, for example, an image (actual image and / or predicted image) extracted by the image extraction unit 72, a pixel set generated by the pixel set generation unit 73, etc. Details of the state change display screen will be described later.

[0047] 5 is a schematic diagram showing an example of a status change display screen 200 displayed on the display device 90 according to this embodiment. As shown in the figure, the status change display screen 200 includes, for example, an image area 210, a first pixel group area 220, a second pixel group area 230, and a save button 240.

[0048] The image area 210 displays one of the multiple actual images stored in the actual image DB 40 or the multiple predicted images stored in the predicted image DB 60, which is associated with the selected time information. In FIG. 5, as an example, the image area 210 displays a processed image (actual image or predicted image) on which semantic region segmentation processing has been performed. Reference numeral 210D denotes a damaged region identified by the semantic region segmentation processing. In this way, the image area 210 makes it possible to grasp at a glance the actual appearance or predicted appearance of the object corresponding to the selected time information.

[0049] Reference numeral 212L denotes a first axis. Here, the first axis 212L is a line parallel to the X-axis and is an example of pixel extraction coordinates. As shown in FIG. 5, the first axis 212L is associated with the image displayed in the image region 210, and more specifically, is displayed superimposed on a position on the image corresponding to the first axis 212L. The first pixel set region 220 displays a first pixel set, which is a pixel set made up of pixels arranged on the first axis 212L.

[0050] The first pixel set is configured by arranging pixel rows arranged on the first axis 212L in each image in the order of the time information associated with each image. Note that the dotted line with the reference numeral 214N indicates the current time T NThe area 220A to the left of the dotted line 214N is made up of pixel strings extracted from actual images stored in the actual image DB 40. The area 220B to the right of the dotted line 214N is made up of pixel strings extracted from predicted images stored in the predicted image DB 60.

[0051] The portion of first pixel set region 220 designated by reference numeral 220D corresponds to damaged region 210D included in image region 210. In this way, first pixel set region 220 makes it possible to grasp at a glance (from a single pixel set) the change over time of the object on first axis 212L, which is the predetermined position coordinate (pixel extraction coordinate).

[0052] Reference numeral 213L denotes a second axis. Here, the second axis 213L is a line parallel to the Y-axis and is an example of pixel extraction coordinates. As shown in FIG. 5, the second axis 213L is associated with the image displayed in the image region 210, and more specifically, is displayed superimposed on a position on the image corresponding to the second axis 213L. The second pixel set region 230 displays a second pixel set, which is a pixel set made up of pixels arranged on the second axis 213L.

[0053] The second pixel set is configured by arranging pixel rows arranged on the second axis 213L in each image in the order of the time information associated with each image. N The region 230A to the left of the dotted line 214N indicating the position is made up of pixel strings extracted from actual images stored in the actual image DB 40. The region 230B to the right of the dotted line 214N is made up of pixel strings extracted from predicted images stored in the predicted image DB 60.

[0054] The portion of second pixel set region 230 designated by reference numeral 230D corresponds to damaged region 210D included in image region 210. In this way, second pixel set region 230 makes it possible to grasp at a glance (from a single pixel set) the change over time of the object on second axis 213L, which is the predetermined position coordinate (pixel extraction coordinate).

[0055] A first axis slide bar 212B extending vertically is displayed on the left side of the image region 210. The first axis slide bar 212B includes a slider 212S, and the user can change the position of the first axis 212L between Y=Y1 and Y=Y2 by sliding the slider 212S vertically along the first axis slide bar 212B via the operation of the operation unit 80. When the position of the first axis 212L is changed by this sliding operation, the setting change unit 74 changes the setting of the first axis 212L stored in the memory unit 71. Then, the pixel set generation unit 73 displays a pixel set based on the pixel column arranged on the changed first axis 212L in the first pixel set region 220.

[0056] A second axis slide bar 213B extending in the horizontal direction is displayed below the image area 210. The second axis slide bar 213B includes a slider 213S, and the user can change the position of the second axis 213L between X=X1 and X=X2 by operating the operation unit 80 to slide the slider 213S in the horizontal direction along the second axis slide bar 213B.

[0057] When the position of the second axis 213L is changed by the slide operation, the setting change unit 74 changes the setting of the second axis 213L stored in the storage unit 71. Then, the pixel set generation unit 73 displays, in the second pixel set area 230, a pixel set based on the pixel row arranged on the changed second axis 213L.

[0058] A time axis slide bar 214B extending horizontally is displayed above the first pixel group region 220. The time axis slide bar 214B includes a slider 214S. A time axis 214L extends vertically from the slider 214S. The user can move the position of the time axis 214L to the start time T by sliding the slider 214S vertically through an operation via the operation unit 80. S and end time T E can be changed between

[0059] When the position of the time axis 214L is changed by the slide operation, the setting change unit 74 changes the setting (time information of the image to be displayed in the image area 210) stored in the storage unit 71. Then, the display device control unit 76 displays in the image area 210 an image (actual image or predicted image) associated with the time information indicated by the changed time axis 214L.

[0060] The user can select the save button 240 by operating the operation unit 80. When the save button 240 is selected, the saving unit 77 saves the image displayed in the image area 210, the first pixel set displayed in the first pixel set area 220, the second pixel set displayed in the second pixel set area 230, etc. in the memory unit 71.

[0061] FIG. 6 is an operational flow diagram showing an example of the operational processing by the display system 1 according to this embodiment.

[0062] (S301) The image extraction unit 72 extracts images from the actual image DB 40 and / or the predicted image DB 60 based on the settings stored in the storage unit 71. The range of time information of the extracted images is from the start time T S From end time T E For example, the setting may be set to an end time T E is the current time T N If the time information is before the start time T S From end time T E The actual images up to the end time T E is the current time T N If the time information is later than the start time T S to the current time T N The actual images up to the current time T are extracted from the actual image DB 40. N From end time T E The predicted images up to are extracted from the predicted image DB 60. Depending on the settings stored in the storage unit 71, the type of image to be extracted may be a captured image, a processed image, a difference image, or the like.

[0063] (S302) Next, the pixel extraction coordinate determination unit 75 may analyze the extracted image (actual image and / or predicted image) depending on the type of the image and determine pixel extraction coordinates. For example, if the extracted image is a processed image that has been subjected to semantic region segmentation processing, the pixel extraction coordinate determination unit 75 may select an axis along which the change in size of the damaged region is equal to or greater than a predetermined threshold.

[0064] (S303) The pixel set generation unit 73 extracts pixel rows arranged on the set pixel extraction coordinates, and generates pixel sets by arranging the extracted pixel rows in accordance with the order of the time information of the image.

[0065] (S304) The display device control unit 76 causes the display device 90 to display a state change display screen 200 as shown in FIG.

[0066] (S305) The setting change unit 74 determines whether the setting of pixel extraction coordinates has been changed in response to a user's operation via the operation unit 80. For example, in the example shown in Fig. 5, when slider 212S of first axis slide bar 212B is operated, it is determined that the setting of first axis 212L, which is an example of pixel extraction coordinates, has been changed, and when slider 213S of second axis slide bar 213B is operated, it is determined that the setting of second axis 213L, which is an example of pixel extraction coordinates, has been changed. If it is determined that the setting of pixel extraction coordinates has not been changed (S305; No), the process proceeds to step S308.

[0067] (S306) If it is determined that the setting of the pixel extraction coordinates has been changed (S305; Yes), the pixel set generation unit 73 extracts pixels arranged on the pixel extraction coordinates whose setting has been changed from each image (actual image and / or predicted image) extracted by the image extraction unit 72, and then generates a pixel set by arranging the extracted pixels in the order of the time information of the image. For example, in the example shown in FIG. 5, if it is determined that the setting of the first axis 212L has been changed, the pixel set generation unit 73 generates a pixel set consisting of pixels arranged on the new first axis 212L. Also, in the example shown in FIG. 5, if it is determined that the setting of the second axis 213L has been changed, the pixel set generation unit 73 generates a pixel set consisting of pixels arranged on the new second axis 213L.

[0068] (S307) The display device control unit 76 updates the display of the pixel set region on the status change display screen with the new pixel set generated in step S306. For example, in the example shown in Fig. 5, if the setting of the first axis 212L is changed, the display device control unit 76 displays the first pixel set generated in step S306 in the first pixel set region 220. Also, in the example shown in Fig. 5, if the setting of the second axis 213L is changed, the display device control unit 76 displays the second pixel set generated in step S306 in the second pixel set region 230.

[0069] (S308) The setting change unit 74 determines whether the time axis setting has been changed in response to a user's operation via the operation unit 80. For example, in the case of the status change display screen 200 shown in Fig. 5, if the slider 214S of the time axis slide bar 214B is operated, it is determined that the setting of the time axis 214L has been changed. If it is determined that the time axis setting has not been changed (S308; No), the processing proceeds to step S310.

[0070] (S309) The display device control unit 76 displays, in the image area on the status change display screen, an image (actual image or predicted image) associated with the time information corresponding to the time axis changed in step S308. For example, in the example shown in Fig. 5, when the setting of the time axis 214L is changed, the display device control unit 76 displays, in the image area 210, an image associated with the time information corresponding to the changed time axis 214L.

[0071] (S310) Next, the saving unit 77 determines whether saving has been selected in response to a user's operation via the operation unit 80. For example, in the case of the status change display screen 200 shown in Fig. 5, if the save button 240 is selected, it is determined that saving has been selected. If it is determined that saving has not been selected (S310; No), the process returns to step S305.

[0072] (S311) Next, the saving unit 77 saves the image displayed in the image region 210, the first pixel set displayed in the first pixel set region 220, and the second pixel set displayed in the second pixel set region 230, etc., in the memory unit 71. Then, the process returns to step S305. Thereafter, steps S305 to S311 are repeated until the user selects "End."

[0073] The above-described embodiments are intended to facilitate understanding of the present invention and are not intended to limit the present invention. The elements of the embodiments, as well as their arrangement, materials, conditions, shapes, sizes, etc., are not limited to those illustrated and can be modified as appropriate. Furthermore, configurations shown in different embodiments can be partially substituted or combined with each other. [Explanation of symbols]

[0074] 1...display system, 10...imaging device control unit, 20...imaging device, 30...image processing unit, 31...normalization processing unit, 32...processing processing unit, 33...differential image generation unit, 34...image synthesis unit, 40...actual image DB, 50...predicted image generation unit, 60...predicted image DB, 70...display control unit, 71...storage unit, 72...image extraction unit, 73...pixel set generation unit, 74...setting change unit, 75...pixel extraction coordinate determination unit, 76...display device control unit, 77...storage unit, 80...operation unit, 90...display device

Claims

1. 1. A display method for displaying a change in state of an object over time based on a plurality of images including a plurality of actual images generated by successively capturing images of the object and a plurality of predicted images associated with future capture times generated based on the plurality of actual images, the method comprising: extracting pixels located on predetermined position coordinates from each of the plurality of images; and displaying a pixel set in which the pixels are arranged according to the order in which the plurality of images were captured.

2. The display method according to claim 1 , wherein the predetermined position coordinates are at least one of a point, a line, or a position coordinate that constitutes an area having an area.

3. The display method according to claim 2 , wherein the predetermined position coordinates form a straight line.

4. The display method according to claim 3 , wherein the straight line is parallel to one of the coordinate axes of each of the plurality of images.

5. The display method according to claim 1 , further comprising the step of determining the predetermined position coordinates based on the plurality of images.

6. The display method according to claim 1 , further comprising the step of displaying at least one image of the plurality of images.

7. The display method according to claim 6 , wherein the step of displaying the at least one image includes a step of displaying the predetermined position coordinates in association with the at least one image.

8. The display method according to claim 1 , further comprising the step of generating the plurality of predicted images based on the plurality of actual images.

9. The display method according to claim 1 , wherein the plurality of images are processed images obtained by performing a predetermined processing process on captured images generated by capturing images of the object.

10. 1. A control device that controls display of a change in state of an object over time based on a plurality of images including a plurality of actual images generated by successively capturing images of the object and a plurality of predicted images associated with future image capture times generated based on the plurality of actual images, a pixel set generation unit that extracts pixels arranged at predetermined position coordinates from each of the plurality of images and generates a pixel set in which the pixels are arranged according to the order in which the plurality of images were captured; a display device control unit that displays the generated pixel set on a display device.

11. A display device that displays changes in the state of an object over time, a display device that displays a pixel set in which pixels located at predetermined position coordinates in each of a plurality of images, the plurality of images including a plurality of actual images generated by successively capturing images of the object and a plurality of predicted images associated with future capture times generated based on the plurality of actual images, are arranged in accordance with the order in which the plurality of images were captured.

12. On the computer, A program for executing a display method for displaying a change in state of an object over time based on a plurality of images including a plurality of actual images generated by successively capturing images of the object and a plurality of predicted images associated with future capture times generated based on the plurality of actual images, the program comprising: extracting pixels located on predetermined position coordinates from each of the plurality of images; and displaying a pixel set in which the pixels are arranged according to the order in which the plurality of images were captured.

Images