Information processing method, information processing device, and program

Abstract

To accurately annotate a state of an object in a photographed image data obtained by photographing an object in a plurality of pieces of photographed image data obtained by photographing with a plurality of photographing cameras.SOLUTION: A plurality of pieces of photographed image data obtained by photographing an identical object with a plurality of pieces of photographing means from different directions, and pieces of time information regarding time points at which the object is photographed are associated with each other. At least a part of the plurality of pieces of photographed image data photographed at the same time points is displayed on a display unit. Prescribed attention areas are set to the displayed pieces of photographed image data. For pieces of photographed image data other than the photographed image data to which the attention areas are set among the plurality of pieces of photographed image data, corresponding areas almost corresponding to the attention areas in the object are set. A corresponding relation between a position of the attention area in a part of the photographed image data and a position of a corresponding area of another photographed image data is generated as a corresponding map, and stored in a storage unit.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] This invention relates to an information processing method, an information processing device, and a program. [Background technology]

[0002] In recent years, in order to improve the operational efficiency of incinerators in waste treatment facilities, there has been a need to develop technologies for understanding the contents of storage pits, such as using machine learning to determine and recognize the mixing state and type of waste within the storage pit. When using supervised learning in machine learning, it is necessary for humans to label, or annotate, the image data of the storage pit that serves as training data. In such annotation, the technology described in Patent Document 1 has been proposed as a method for labeling multiple images of the same subject. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Patent No. 7083189 [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] However, in the technology described in Patent Document 1 mentioned above, it is necessary to use visual markers to associate position and orientation, and it is extremely difficult to use a predetermined range of a predetermined object in each image data to associate it with each image data when multiple different image data obtained by imaging the same object from imaging cameras installed at different positions.

[0005] The present invention has been made in view of these circumstances, and its objective is to provide an information processing method, an information processing device, and a program that can be configured to enable high-precision annotation of the state of an object in image data obtained by imaging an object in multiple image data captured by multiple imaging cameras. [Means for solving the problem]

[0006] To solve the above-mentioned problems and achieve the objective, an information processing method according to one aspect of the present invention is an information processing method executed by a control unit equipped with hardware, comprising: a storage step of storing in a storage unit a plurality of image data obtained by imaging the same object from different directions using a plurality of imaging means, and time information relating to the time when the object was imaged by the plurality of imaging means, associated with each of the plurality of image data; a display step of displaying in a display unit at least some of the image data from the plurality of image data read from the storage unit, the image data being captured at the same time according to the time information; a focus area setting step of setting a predetermined focus area for at least one of the displayed at least some of the image data; a corresponding area setting step of setting a corresponding area in the object substantially the same as the focus area for other image data from the plurality of image data other than the image data from which the focus area has been set; and a map generation step of generating a correspondence map of the correspondence relationship between the position of the focus area in the partial image data and the position of the corresponding area in the other image data, and storing it in the storage unit.

[0007] An information processing method according to one aspect of the present invention includes a labeling step in which an externally set label is set in the area of ​​interest, and based on the correspondence map, the label set in the area of ​​interest is set in the corresponding area corresponding to the area of ​​interest on which the label is set, thereby performing labeling on the plurality of captured image data.

[0008] An information processing method according to one aspect of the present invention involves reading a plurality of imaging image data from the storage unit, wherein the imaging means is the same and the times included in the time information are different from each other; deriving the difference between the two imaging image data as time progresses in the time information for two of the plurality of imaging image data; and setting the region containing the derived difference as the region of interest for the imaging image data with the later time included in the time information of the two imaging image data.

[0009] An information processing method according to one aspect of the present invention involves setting position coordinates for the object in advance, imaging the object with one of the plurality of imaging means to obtain one image data, converting the position coordinates to position coordinates in the image data, and determining the coordinates of the area of ​​interest based on the converted position coordinates in the image data.

[0010] An information processing method according to one aspect of the present invention, in the above invention, converts the position coordinates of a plurality of image data obtained by imaging the object with the plurality of imaging means into the position coordinates of each of the plurality of image data, determines the coordinates of the area of ​​interest and the coordinates of the corresponding area based on the converted position coordinates in the image data, and derives the correspondence between the area of ​​interest and the corresponding area based on the coordinates.

[0011] An information processing method according to one aspect of the present invention involves deriving the correspondence relationship between the area of ​​interest and the corresponding area based on data measured by a distance measuring sensor for the object, in the above invention.

[0012] In one aspect of the present invention, the information processing method is described above, in which the area of ​​interest setting step and the corresponding area setting step are performed based on information input to the control unit from an external source.

[0013] An information processing method according to one aspect of the present invention is characterized in that, in the above invention, the object is waste stored in a storage pit in a waste storage facility.

[0014] An information processing device according to one aspect of the present invention includes a storage unit for storing image data obtained by imaging an object, a display unit capable of displaying the image data, and a control unit for performing image processing on the image data. The control unit stores in the storage unit a plurality of image data obtained by imaging the same object from different directions using a plurality of imaging means, and time information relating to the time when the object was imaged by the plurality of imaging means. The control unit displays in the display unit at least some of the image data from the plurality of image data read from the storage unit, the time at which the imaging was performed according to the time information is the same. A predetermined area of ​​interest is set for at least one of the image data displayed on the display unit. Meanwhile, for other image data from the plurality of image data other than the image data on which the area of ​​interest is set, a corresponding area corresponding to substantially the same area as the area of ​​interest in the object is set. The control unit associates the area of ​​interest with the corresponding area, generates a correspondence map of the correspondence between the position of the area of ​​interest in the partial image data and the position of the corresponding area in the other image data, and stores in the storage unit.

[0015] A program according to one aspect of the present invention causes a control unit having hardware to associate and store in a storage unit a plurality of captured image data obtained by capturing the same object from different directions by a plurality of imaging means and time information regarding the time when the object is imaged by the plurality of imaging means, and causes a display unit to display at least a part of the captured image data among the plurality of captured image data read from the storage unit and having the same time of imaging in the time information, a region-of-interest setting step of setting a predetermined region of interest for at least one of the captured image data among the at least a part of the displayed captured image data, a corresponding region setting step of setting a corresponding region corresponding to a region substantially the same as the region of interest in the object for captured image data other than the captured image data having the region of interest set among the plurality of captured image data, and a map generation step of generating a correspondence relationship between the position of the region of interest in the part of the captured image data and the position of the corresponding region in the other captured image data as a correspondence map and storing the correspondence map in the storage unit.

Advantages of the Invention

[0016] According to the information processing method, information processing apparatus, and program according to the present invention, it is possible to set the state of an object in captured image data obtained by capturing the object in a plurality of captured image data captured by a plurality of imaging cameras so as to be able to annotate the state of the object with high accuracy.

Brief Description of the Drawings

[0017] [Figure 1] FIG. 1 is a block diagram showing an information processing system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing a management apparatus according to an embodiment of the present invention. [Figure 3] FIG. 3 is a diagram showing an example of captured image data obtained by imaging the inside of a storage pit according to an embodiment of the present invention. [Figure 4]Figure 4 shows an example of a pair of image data obtained when imaging the inside of a storage pit with two imaging cameras according to one embodiment of the present invention. [Figure 5A] Figure 5A shows an example of a screen displaying the input / output unit with the target region extracted, showing the captured image data pair 140 shown in Figure 4. [Figure 5B] Figure 5B shows an example in which the region of interest in one of the captured image data in the image display screen shown in Figure 5A is mapped to the corresponding region in the other captured image data. [Figure 6] Figure 6 shows an example in which the area of ​​interest in one image data point in a pair of image data points is sequentially mapped to the corresponding area in the other image data point. [Figure 7A] Figure 7A shows image data at a predetermined point in time, similar to the image data shown in Figure 3. [Figure 7B] Figure 7B shows the state in which the changed region has been extracted from the image data after a predetermined time has elapsed from the image data shown in Figure 7A. [Figure 7C] Figure 7C shows the state in which the changed region has been extracted from the image data after a predetermined time has elapsed from the image data shown in Figure 7B. [Modes for carrying out the invention]

[0018] Hereinafter, one embodiment of the present invention will be described with reference to the drawings. In all the drawings of the following embodiment, the same or corresponding parts will be denoted by the same reference numerals. Furthermore, the present invention is not limited to the embodiment described below.

[0019] First, in describing the information processing device according to one embodiment of the present invention, the inventors' diligent research will be explained. Specifically, according to the inventors' findings, for example, in a facility for storing objects, it is conceivable to install multiple imaging cameras to image the entire facility with high precision. When objects are mixed within the facility, it is difficult for an operator to reliably evaluate the mixing state of the objects by visual inspection. Therefore, even for the same object, it has been difficult for an operator to perform annotation by assigning a unique label to different image data obtained by imaging from different directions with different imaging cameras.

[0020] In other words, when workers visually evaluate and annotate image data of objects within a facility, it is difficult to reliably assess the condition of those objects within the facility. To solve this problem, annotation tools have been proposed that allow for quick switching between multiple images to facilitate comparison.

[0021] However, when capturing image data within a target facility, if the distances between multiple cameras are large, even if the predetermined area for evaluating the target object is the same, it becomes difficult to associate the same predetermined area with each image data captured by the multiple cameras. For example, if the target facility is a waste storage pit and the target object is waste, the storage pit is large, so multiple cameras are often used to image the waste. In this case, even the same waste at the same location within the storage pit will look different depending on the distance from the camera and the direction of imaging. Specifically, if each image data captured by each camera is annotated individually, there is a high possibility of incorrect labeling based on the view from different directions. In other words, in different image data captured at the same time within the storage pit by different cameras, the degree of mixing of waste captured nearby and waste captured at a distance will appear different, even if they are at the same location. Therefore, when annotators perform visual annotation, there is a possibility of assigning different labels to the same waste.

[0022] Therefore, the inventors investigated a method for performing annotation work while simultaneously reviewing multiple image data captured from different directions within a target facility. The inventors devised a method in which, for multiple image data of the same object, a position specified in one image data is set or corrected in the other image data, and the set or corrected correspondence is stored in a memory unit. In other words, the inventors conceived of a method to accurately associate multiple image data by repeatedly matching the positions of multiple image data obtained by capturing the target facility or object from different directions at the same time. This makes it possible to generate with high accuracy a table (hereinafter referred to as a position conversion table) that allows for the conversion of positions between multiple image data obtained by capturing the same object from different directions. In particular, waste materials in a storage pit, which is a target facility, are only stirred to the extent of, for example, by a crane, so rapid and large shape changes are unlikely to occur. Therefore, by generating a position conversion table, the generated position conversion table can be applied to multiple newly acquired image data, reducing the need for corrections and adjustments during annotation work. In other words, by using a position transformation table, annotation can be applied to some of the image data obtained from capturing an object at the same time, specifically to image data of other parts that are relatively difficult to understand and evaluate, thereby reducing the workload of annotation.

[0023] Furthermore, by sequentially acquiring multiple image data obtained by imaging the same object from different directions, and then extracting the differences corresponding to the passage of time, it becomes easier to extract the changed parts of the object, thus further simplifying the annotation process. The embodiment of the present invention described below was devised based on the above diligent research by the inventors.

[0024] (Annotation data generation system) Figure 1 shows an information processing system to which an information processing device according to one embodiment of the present invention is applied. As shown in Figure 1, the annotation data generation system 1, as an information processing system, comprises an information processing device 10, a waste storage facility 20, and a waste incineration facility 30, all of which are able to communicate with each other via a network 2. The waste treatment facility 3 comprises at least the waste storage facility 20 and the waste incineration facility 30. The information processing device 10 may be located externally and able to communicate with the waste treatment facility 3 via the network 2, or it may be part of the waste treatment facility 3. Furthermore, the information processing device 10 may be located inside the waste storage facility 20, and its installation location is not limited.

[0025] Network 2 is composed of a combination of wired and wireless communications as appropriate, and consists of communication networks such as the Internet network and mobile phone network. Network 2 consists of one or more combinations of, for example, dedicated lines, public communication networks such as the Internet, such as LAN (Local Area Network), WAN (Wide Area Network), telephone communication networks and public lines such as mobile phones, and VPN (Virtual Private Network). The information processing device 10, the waste storage facility 20, and the waste incineration facility 30 are connected via Network 2.

[0026] (Waste incineration facility) The waste incineration equipment 30, which serves as the waste incineration unit, has a conventionally known configuration. Specifically, the waste incineration equipment 30 comprises a combustion control device (ACC) 31, a sensor unit 32, and an incinerator 33. The combustion control device 31 controls the amount of combustion air, the amount of cooling air, the waste supply device feed speed, and the grate feed speed as the control variables at each control end, based on the setting of predetermined control variable reference values. The incinerator 33, which is the waste incinerator, comprises a furnace where waste 26a is burned, a waste input port for feeding waste 26a, and a boiler (none of which are shown). The sensor unit 32 consists of, for example, thermometers and pressure gauges installed in various locations. The internal state of the incinerator 33 and various physical quantities such as pressure and velocity in facilities related to the incinerator 33, specifically, in power generation facilities for generating electricity, are measured by the sensor unit 32 and output as sensor information from the sensor unit 32. The sensor information output from the sensor unit 32 is supplied to the combustion control device 31 as parameters. The combustion control device 31 controls the combustion of the incinerator 33 based on the input parameters.

[0027] (Waste storage facility) The waste storage facility 20, which serves as a waste storage unit, comprises a control unit 21, a communication unit 22, an imaging unit 23, a gripping unit 25, and a storage pit 26. The storage pit 26 is movably equipped with a gripping unit 25 and an imaging unit 23. The gripping unit 25 is controlled by the control unit 21 based on a control signal transmitted from the control unit 11 of the information processing device 10. The control unit 21 may also control the imaging unit 23 based on a control signal transmitted from the control unit 11. Alternatively, the control unit 11 of the information processing device 10 may directly control the imaging unit 23 and the gripping unit 25.

[0028] The control unit 21 specifically includes a processor with hardware such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and an FPGA (Field-Programmable Gate Array), as well as main memory such as RAM (Random Access Memory) and ROM (Read Only Memory) (none of which are shown). The control unit 21 controls the imaging unit 23 and the gripping unit 25 based on control signals input from the information processing device 10 via the communication units 13 and 22, according to various programs stored in the main memory such as RAM and ROM.

[0029] The communication unit 22 is, for example, a LAN interface board, a wired communication circuit for wired communication, or a wireless communication circuit for wireless communication. The LAN interface board, wired communication circuit, or wireless communication circuit is connected to the network 2. The communication unit 22, acting as both a transmitter and receiver, is connected to the network 2 and communicates with the information processing device 10.

[0030] The gripping unit 25 grips and moves the waste 26a stored in the storage pit 26. The bucket 252, which acts as an opening and closing unit, can grip the waste 26a. The crane 251, which acts as a moving unit, is configured to be movable by connecting to the bucket 252. The crane 251 is controlled by a control signal transmitted from the pit control unit 113 of the information processing device 10. The storage pit 26 is a space enclosed by walls 26b, and is a pit capable of temporarily storing waste 26a. The waste 26a in the storage pit 26 is gripped by the gripping unit 25 and supplied to the incinerator 33 of the waste incineration equipment 30, where it is incinerated.

[0031] The imaging unit 23 has multiple imaging means, for example, two imaging cameras 231 and 232. The imaging unit 23 is configured to be able to image the waste 26a and walls 26b in the storage pit 26, and in particular the surface of the waste 26a, using the imaging cameras 231 and 232. The imaging cameras 231 and 232 that make up the imaging unit 23 each image the condition of the waste 26a in the storage pit 26 from different directions.

[0032] Figure 3 shows the image data generated by the imaging camera 231. The image data 100, as imaging information shown in Figure 3 and obtained by imaging camera 231, is associated with the time information at the time of imaging and transmitted from imaging camera 231 to the information processing device 10 via the communication unit 22. Similarly, the image data 100 obtained by imaging camera 232 from a different direction than imaging camera 231, capturing the inside of the storage pit 26, is also associated with the time information at the time of imaging and transmitted from imaging camera 232 to the information processing device 10 via the communication unit 22. This allows the information processing device 10 to associate multiple different image data 100 obtained by imaging cameras 231 and 232 capturing the inside of the storage pit 26 from different directions based on the time information. In other words, the information processing device 10 can process multiple image data 100 captured by multiple different imaging cameras 231 and 232, with the same time information, as image data 100 captured simultaneously at the same time. The same applies even if there are three or more imaging cameras.

[0033] (Information processing device) Figure 2 shows details of the information processing device 10 in Figure 1. The information processing device 10 shown in Figure 2 comprises a control unit 11, a storage unit 12, a communication unit 13, and an input / output unit 14. The control unit 11 and the communication unit 13 are physically the same as the control unit 21 and the communication unit 22 described above, respectively. The information processing device 10 according to this embodiment functions as an information processing device that generates an input / output dataset which serves as training data for a learning model capable of measuring the degree of mixing by labeling the mixing state of the waste in the storage pit 26. The learning model is also referred to as a trained model or simply as a model.

[0034] The input / output unit 14 can be composed of, for example, a touch panel display or a speaker microphone. As an input means, the input / output unit 14 includes an interface that receives various information transmitted from, for example, an imaging unit 23 installed in the waste storage facility 20 via the communication unit 22 and outputs it to the control unit 11. The transmission of information from the imaging unit 23 to the input / output unit 14 may be done using either wired or wireless communication. The input / output unit 14 also includes a user interface such as a keyboard, input buttons, levers, a touch panel for manual input superimposed on a display such as an LCD, or a microphone for voice recognition. The input / output unit 14 is configured to allow predetermined information to be input to the control unit 11 by an operator or other person operating it. As an output means, the input / output unit 14 displays images of the storage pit 26 of the waste storage facility 20 on a display monitor, displays characters or figures on the screen of a touch panel display, or outputs sound from a speaker, according to the control unit 11's control. In other words, the input / output unit 14 is configured to be able to notify predetermined information to the outside. The input and output sections of the input / output unit 14 may be configured as separate units.

[0035] The storage unit 12 is composed of a storage medium selected from volatile memory such as RAM, non-volatile memory such as ROM, EPROM (Erasable Programmable ROM), hard disk drive (HDD), and removable media. Removable media include, for example, USB (Universal Serial Bus) memory, or disk recording media such as CD (Compact Disc), DVD (Digital Versatile Disc), or BD (Blu-ray® Disc). Alternatively, the storage unit 12 may be configured using a computer-readable recording medium such as an externally insertable memory card.

[0036] The memory unit 12 can store various programs such as an operating system (OS), image processing applications, various tables, and various databases for executing the operation of the information processing device 10. These various programs can also be recorded on computer-readable recording media such as hard disks, flash memory, CD-ROMs, DVD-ROMs, and flexible disks and widely distributed.

[0037] The memory unit 12 stores image information 121, coordinate information 122, and annotation information 123. All of the image information 121, coordinate information 122, and annotation information 123 are stored in the memory unit 12 as a searchable database.

[0038] Image information 121 includes a plurality of captured image data 100 (see Figure 3) obtained by imaging waste 26a in the storage pit 26 with the imaging unit 23, and time information relating to the time the waste 26a was imaged. The captured image data 100 is associated with the time information in which it was captured. Coordinate information 122 includes coordinate information that has been transformed and associated based on various image processing operations performed by the operator on the captured image data 100 of the image information 121 using a predetermined image processing application and the input / output unit 14 of the information processing device 10. Annotation information 123 includes annotation data (not shown) obtained by applying annotation processing to the captured image data 100 included in the image information 121 obtained as described above.

[0039] The control unit 11 loads programs stored in the memory unit 12 into the work area of ​​the main memory unit and executes them. By controlling each component through program execution, it can realize functions that match a predetermined purpose. Specifically, the control unit 11 can execute the functions of the image processing unit 111, annotation processing unit 112, pit control unit 113, and learning unit 114 by executing various programs loaded from the memory unit 12. The various programs also include programs that realize artificial intelligence and trained models capable of realizing the processing according to this embodiment. Details of the processing by the control unit 11 will be described later.

[0040] (How to generate a dataset) Next, the information processing method executed by the information processing device 10 according to this embodiment, configured as described above, will be explained. In the following explanation, information is transmitted and received between each component via the communication units 13 and 22 and the network 2, but a detailed explanation of this will be omitted each time.

[0041] Figure 4 is a diagram illustrating the method for imaging waste 26a in the storage pit 26 using the information processing method according to this embodiment. As shown in Figure 4, when imaging the inside of the storage pit 26, which is the target facility according to this embodiment, with multiple imaging cameras 231, 232, it is preferable, but not limited, that these imaging cameras 231, 232 be installed in positions that can image as wide an area of ​​waste 26a as possible. The installation positions of the multiple imaging cameras 231, 232 can be appropriately changed according to the shape and dimensions of the storage pit 26.

[0042] As shown in Figure 4, waste 26a is stored in the storage pit 26. In the example shown in Figure 4, one imaging camera 231 is provided in one of the four corners of the storage pit 26, and the imaging camera 231 is capable of imaging at least the surface layer of the waste 26a stored in the storage pit 26 from a predetermined direction, i.e., from a predetermined viewpoint. By imaging the waste 26a in the storage pit 26 with the imaging camera 231, the imaging camera 231 generates image data 141 and transmits it to the information processing device 10. Similarly, the other imaging camera 232 is provided in one of the four corners of the storage pit 26, different from the imaging camera 231, and the imaging camera 232 is capable of imaging at least the surface layer of the waste 26a stored in the storage pit 26 from a direction different from the predetermined direction, i.e., from a viewpoint different from the viewpoint described above. By imaging the waste 26a in the storage pit 26 with the imaging camera 232, the imaging camera 232 generates image data 142 and transmits it to the information processing device 10. The image data 141 and 142, which capture images of the inside of the storage pit 26, are associated with time information and also include information about the waste 26a stored in the storage pit 26 and the walls 26b of the storage pit 26. Although not shown in Figure 4, the crane 251 and bucket 252 used to grasp and release the waste 26a may also be captured in the images.

[0043] The waste 26a stored in the storage pit 26 is composed of various objects, and various states of agitation and mixing of the waste 26a are possible. In this embodiment, as shown in Figure 4, the operator performs annotation processing on multiple image data, in this case for example, two image data sets 141 and 142, which are captured at the same time from different imaging directions using imaging cameras 231 and 232 set at different positions and have the same time information. In this case, the two image data sets 141 and 142 may be output and displayed simultaneously or separately on the input / output unit 14.

[0044] The worker visually inspects the waste 26a in the storage pit 26 using the captured image data 141 and 142 displayed on the input / output unit 14. During annotation processing, the worker uses a predetermined image processing device, in this embodiment, an information processing device 10, to visually evaluate the state of the objects captured in the captured image data 141 and 142, in this case the mixing state of the waste 26a, based on the worker's experience and knowledge. This evaluation of the degree of mixing is also called discrimination or judgment of the degree of mixing.

[0045] The operator selects the area to be evaluated from the captured image data 141 using the input / output unit 14 of the information processing device 10. The image processing unit 111 of the control unit 11 then sets the selected area as image patches 141a and 142a. The operator then sets labels determined based on the evaluation for each image patch 141a and 142a displayed in the captured image data 141 and 142. Here, when setting labels for the mixing state of waste 26a, which is a mixture of multiple objects stored in the storage pit 26, for example, highly mixed areas, medium mixed areas, and low mixed areas, as well as wall areas and bagged waste areas, are pre-set as distinct labels.

[0046] Specifically, for example, an operator uses the input / output unit 14 to input information for setting a predetermined label for an image patch 141a of the captured image data 141. In this case, the annotation processing unit 112 of the control unit 11 sets the predetermined label for the image patch 141a of the captured image data 141 displayed on the input / output unit 14, based on the input information. As a result, annotation processing is performed on the image of the waste 26a in the captured image data 141 shown in Figure 4, and a predetermined label is set for each location of the waste 26a, i.e., for each selected image patch 141a. The image processing unit 111 may also superimpose the input and set labels onto the image patch 141a of the captured image data 141.

[0047] As mentioned above, if annotation processing is performed individually on two image data 141 and 142, which were captured from different directions at the same time, and labels are set for each, the evaluation of the degree of mixing may differ when viewed and evaluated by an operator because the imaging directions of the image data 141 and 142 are different. Specifically, for example, even if image patches 141a and 142a select the same area of ​​waste 26a, the degree of mixing of waste 26a captured relatively far from the imaging camera 232, as in image patch 142a, will appear to be greater and more mixed compared to the degree of mixing of waste 26a captured relatively close, as in image patch 141a.

[0048] Therefore, in this embodiment, the display unit of the input / output unit 14 displays together two image data sets 141 and 142, which were captured simultaneously from different imaging directions of the same waste 26a by imaging cameras 231 and 232. The method of displaying the image data sets 141 and 142 in the input / output unit 14 is not limited to displaying them on the same screen; various display methods can be selected as long as the operator can compare the image data sets 141 and 142 with each other. Hereinafter, the image data sets 141 and 142 will also be referred to as the image data set pair 140. This allows the operator to select one of the image data sets 141 or 142 from the image data set pair 140 that allows for a more accurate evaluation of the mixing degree of the waste 26a, select a predetermined area as an image patch 141a or 142a, and perform the evaluation of the mixing degree.

[0049] (Method for generating correspondence maps) This section describes a method for generating a correspondence map among annotation methods using multiple image data sets 141 and 142. Figure 5A shows an example of a screen displaying the pair of image data sets 140 shown in Figure 4, with the area of ​​interest extracted from the input / output unit. Figure 5B shows an example of mapping the area of ​​interest in one of the image data sets in the screen shown in Figure 5A to the other image data set.

[0050] As shown in Figure 5A, the image processing unit 111 in the control unit 11 of the information processing device 10 stores multiple captured image data captured by multiple imaging cameras 231, 232 as image information 121 in the storage unit 12 (storage step). The image processing unit 111 reads the image information 121 from the storage unit 12 and displays the pair of captured image data 140 contained in the image information 121 on the display screen of the input / output unit 14 (display step).

[0051] Next, the operator evaluates the degree of mixing in an arbitrary part of the waste 26a in the storage pit 26 with respect to the captured image data pair 140 displayed on the display screen of the input / output unit 14. For example, the operator uses the input / output unit 14 to select the captured image data 141 from which it is easy to evaluate the degree of mixing from the captured image data pair 140. Subsequently, the operator selects a region of interest A1 for setting a label in the annotation process for the captured image data 141. After the image processing unit 111 sets the selected region of interest A1 for the captured image data 141, it stores it in the storage unit 12 as image information 121 (region of interest setting step).

[0052] Subsequently, the annotation processing unit 112 reads out the coordinate range of the region of interest A1 from the storage unit 12 as coordinate information 122. Here, as the coordinate range of the region of interest A1 in the coordinate information 122, by designating a predetermined vertex of the region of interest A1, it can be read out as "A1(x 11 ,y 11 )", or as the rectangular range of the region of interest A1, it can be read out as "A1(x 11 ~x 12 ,y 11 ~y 12 )", or as "A1(x 11 ,y 11 )~(x 12 ,y 12 )". In the present embodiment, for example, a predetermined vertex of the region of interest is designated as A1(x 11 ,y 11 ).

[0053] The operator uses the input / output unit 14 to set a label based on the evaluation content of the degree of mixing for the region of interest A1(x 11 ,y 11 ), that is, performs labeling. The label set for the region of interest A1 is associated with the coordinate information 122 of the region of interest A1 and stored in the storage unit 12 as annotation information 123 (labeling step).

[0054] Furthermore, as shown in Figure 5B, the operator selects a corresponding region A2 in the other image data 142 from the pair of image data 140 that was not selected, which corresponds to the region of interest A1 in the image data 141. In other words, the pair of image data 140 consists of images of the same object, waste 26a, taken from multiple different directions, in this case from two different viewpoints. Therefore, if a region of interest A1 is selected in one of the image data 141 of the pair of image data 140, a corresponding region A2 often exists in the other image data 142.

[0055] Therefore, the operator uses the input / output unit 14 to select and set the region in the image data 142 that corresponds to the region of interest A1 in the image data 141 as the corresponding region A2. The corresponding region A2 selected from the image data 142 will indicate approximately the same position and approximately the same region within the storage pit 26 as the region of interest A1 in the image data 141. The image processing unit 111 sets the coordinates (x 21 ,y 21 ) is the area of ​​interest A1(x 11 ,y 11 ) is associated with and stored in the storage unit 12 as coordinate information 122 (corresponding area setting step).

[0056] Furthermore, the annotation processing unit 112 sets the same label for the corresponding region A2 as the label set for the region of interest A1. This makes it possible to perform annotation processing on a portion of the region of interest A1 in one of the captured image data 141 for the other captured image data 142. Conversely, the corresponding region A2(x 21 ,y 21 By selecting ), the region of interest A1(x 11 ,y 11It also becomes possible to extract ). Here, the worker sets a label for one image data 141 from the image data pair 140 that can evaluate the state of the waste 26a with higher accuracy, and performs annotation processing. As a result, even for other image data 142 in which the accuracy of evaluating the state of the waste 26a is relatively lower for the worker, it becomes possible to evaluate the state of the waste 26a with the same high accuracy as with image data 141, set a label, and perform annotation processing.

[0057] Furthermore, various methods other than those described above can be used to determine the correspondence between the area of ​​interest A1 and the corresponding area A2. That is, it is also possible to store the coordinates (x,y) of the top view of the storage pit 26 as coordinate information 122 in the storage unit 12 beforehand. In this case, the image processing unit 111 can map any position in the captured image data 141,142 to the coordinates (x,y) of the top view of the storage pit 26 based on the coordinate information 122 of the installation positions of the imaging cameras 231,232, for example, the coordinate information 122 of two of the four corners of the storage pit 26. As a result, the image processing unit 111 can map the area of ​​interest A1 (x 11 ,y 11 ) is associated with the coordinates (x1, y1) in the storage pit 26, while the corresponding coordinates (x1, y1) in the storage pit 26 are associated with the corresponding region A2(x 21 ,y 21 This allows the system to correspond to the region A1. As a result, the image processing unit 111 can determine the corresponding region A2 via the coordinates in the storage pit 26 from the region of interest A1, and derive the correspondence between the region of interest A1 and the corresponding region A2 through matching. This allows the image processing unit 111 to generate a correspondence map (correspondence map generation step). However, it is possible that a discrepancy may occur between the region of interest A1 and the corresponding region A2 due to the matching. In this case, the operator can correct the position of the corresponding region A2 by moving it to a position corresponding to the position of the region of interest A1 from the input / output unit 14. This allows for higher accuracy in the correspondence between the region of interest A1 and the corresponding region A2, and enables the generation of a correspondence map more accurately, thus enabling more accurate annotation processing.

[0058] Furthermore, the surface layer of the waste 26a in the storage pit 26 can sometimes be measured using a distance measuring sensor (not shown), such as 3D-LiDAR (3-Dimensional Laser Imaging Detection and Ranging). In this case, by associating data such as point cloud data obtained by the distance measuring sensor with the positional information of the waste 26a in the storage pit 26, it is possible to associate the area of ​​interest A1 with the corresponding area A2, derive a correspondence relationship, and generate a correspondence map.

[0059] Furthermore, while the aforementioned areas of interest A1 and corresponding area A2 are defined as, for example, rectangular or square regions, other shapes can be appropriately defined. Specifically, for example, both areas of interest A1 and corresponding area A2 can be defined as circular, elliptical, closed curve, polygonal, or other shapes. In this case, it is preferable that the shapes of areas of interest A1 and corresponding area A2 are similar to each other.

[0060] Figure 6 shows an example in which the area of ​​interest of one image data in the image data pair 140 is sequentially associated with the area of ​​interest of the other image data. As shown in Figure 6, the operator first uses the input / output unit 14 of the information processing device 10 to select and input the image data 141 and 142 from the image data pair 140 that are easier to evaluate with high accuracy regarding the mixing degree of the waste 26a. The image processing unit 111 extracts and selects one image data 141 or 142 from the image data pair 140 based on the input information.

[0061] Next, the operator uses the input / output unit 14 to set areas of interest A1, B1, C1, D1, E1, and F1 for the image data 141 and 142 on the side that is easier to evaluate with high accuracy. In the example shown in Figure 6, areas of interest A1, B1, and C1 are selected on the front side of image data 141 because it is easier to evaluate the mixing state of the waste 26a captured on the front side of image data 141 with high accuracy. Similarly, in image data 142, areas of interest D1, E1, and F1 are selected on the front side of image data 142 because it is easier to evaluate the mixing state of the waste 26a captured on the front side of image data 142 with high accuracy.

[0062] The image processing unit 111 of the control unit 11 reads image information 121 from the storage unit 12 and displays the pair of captured image data 140 contained in the image information 121 on the display screen of the input / output unit 14. The operator inputs the captured image data 141 and 142 from the input / output unit 14 to the pair of captured image data 140 displayed on the display screen, which allows for high-precision evaluation of the degree of mixing, in other words, which allows for appropriate evaluation of the degree of mixing. The operator selects the areas of interest A1 to F1 to be labeled in the annotation process from the respective captured image data 141 and 142. The annotation processing unit 112 reads the coordinate range of the areas of interest A1 to F1 from the storage unit 12 as coordinate information 122.

[0063] The operator uses the input / output unit 14 to label the areas of interest A1 to F1 based on the evaluation of the degree of mixing. The labels set for areas of interest A1 to F1 are associated with the coordinate information 122 of area of ​​interest A1 and stored in the storage unit 12 as annotation information 123 (labeling step).

[0064] Furthermore, the operator uses the input / output unit 14 to select corresponding regions A2, B2, C2, D2, E2, and F2 in the pair of captured image data 140 that correspond to the regions of interest A1 to F1, from the captured image data 141 and 142 that were not selected. The image processing unit 111 stores the coordinates of the selected corresponding regions A2 to F2 as coordinate information 122 in the storage unit 12, associating them with the coordinates of the regions of interest A1 to F1. The annotation processing unit 112 also sets the same label for each of the corresponding regions A2 to F2 as the label set for the regions of interest A1 to F1 (labeling step). This makes it possible to perform annotation processing on multiple regions of interest A1 to F1 in one set of captured image data 141 and 142 for the other set of captured image data 141 and 142.

[0065] By repeatedly performing the above processing on the image data pair 140, it becomes possible to associate the coordinate information 122 between the image data 141 and 142 in the image data pair 140. The image processing unit 111 associates and corresponds the coordinate information 122 of the target region A1 to F1 obtained in this manner with the coordinate information 122 of the corresponding region A2 to F2, and stores them in the storage unit 12. As a result, the annotation processing unit 112 can generate a correspondence map as a correspondence between the two-dimensional positions of the image data 141 and 142 in the coordinate information 122. The generated correspondence map is stored in the storage unit 12 as coordinate information 122.

[0066] Once a correspondence map is generated, the annotation processing unit 112 performs labeling on the corresponding region of the other image data 141, 142, corresponding to the labeling performed on one of the target regions in the captured image data 141, 142. Furthermore, although the correspondence between the target region and the corresponding region changes according to the storage state of the waste 26a in the storage pit 26, the change in the state of the waste 26a caused by one or so operations of the crane 251 is gradual. Therefore, when performing annotation processing on other image data 141, 142 with similar time information, the correspondence map can be largely reused.

[0067] (Time difference) Next, a method for processing time differences performed in the annotation method by the information processing device 10 as an image processing device according to this embodiment will be described. Figure 7A is a diagram showing captured image data at a predetermined time point, similar to the captured image data shown in Figure 3. Figure 7B is a diagram showing the state in which the changed region has been extracted from the captured image data after a predetermined time has elapsed from the captured image data shown in Figure 7A. Figure 7C is a diagram showing the state in which the changed region has been extracted from the captured image data after a predetermined time has elapsed from the captured image data shown in Figure 7B.

[0068] The image processing unit 111 in the control unit 11 of the information processing device reads image information 121 from the storage unit 12 and reads, for example, captured image data 141A taken by the imaging camera 231 at a predetermined time T included in the time information. As shown in Figure 7A, the image processing unit 111 outputs and displays the read image data 141A on the display screen of the input / output unit 14.

[0069] Next, as shown in Figure 7B, the image processing unit 111 reads out the image data 141B from the storage unit 12, which is captured by the same imaging camera 231 and taken at a predetermined time Δt (T+Δt) after a predetermined time T has elapsed. The image processing unit 111 then derives the difference between the image data 141A and the image data 141B taken after the predetermined time Δt has elapsed.

[0070] If the image processing unit 111 detects a difference in the captured image data 141B compared to the captured image data 141A, it extracts the difference in the captured image data 141B compared to the captured image data 141A and extracts the region surrounding the extracted difference as the region of interest. In the example shown in Figure 7B, the image processing unit 111 extracts the difference region of the captured image data 141B compared to the captured image data 141A as regions of interest H and J, and displays them superimposed on the display screen of the input / output unit 14 as, for example, a rectangular region.

[0071] If labeling has already been completed for the entire image data 141A, the operator can complete the labeling process for image data 141B and the annotation process by labeling only the areas of interest extracted as regions H and J. Since labeling is unnecessary for the common parts of image data 141A and 141B, the annotation process for image data 141B can be completed simply by labeling the areas of interest H and J obtained as differences.

[0072] Subsequently, as shown in Figure 7C, the image processing unit 111 reads out the image data 141C from the storage unit 12, which is included in the image information 121 and was captured by the same imaging camera 231, and was obtained after a further time δt has elapsed (T+Δt+δt) from the predetermined time point (T+Δt) shown in Figure 7B. The image processing unit 111 then derives the difference between the image data 141B and the image data 141C obtained after the predetermined time δt has elapsed.

[0073] The image processing unit 111, if it detects a difference between the captured image data 141C and the captured image data 141B, extracts the difference in the captured image data 141C and the region surrounding the extracted difference as the region of interest. In the example shown in Figure 7C, the regions of the difference between the captured image data 141C and the captured image data 141B are extracted as regions of interest K, L, and M. The regions of interest K, L, and M are superimposed on the display screen of the input / output unit 14, for example, as rectangular enclosed regions.

[0074] At this point, labeling has already been completed for the entire image data 141B. In this case, the operator completes the labeling process for the image data 141C and the annotation process by labeling only the areas of interest K, L, and M extracted from the input / output unit 14. Since labeling is unnecessary for the common parts of the image data 141A and 141B, the annotation process for the image data 141B can be completed simply by labeling the areas of interest H and J obtained as differences.

[0075] As described above, the time differences Δt and δt in the captured image data 141A, 141B, and 141C allow us to estimate the location where the waste 26a is moved by the crane 251 and the location where the waste 26a is deposited by the refuse vehicle. Furthermore, by overlaying the change locations in the time-dependent changes of the captured image data 141A to 141C onto the captured image data 141 as areas of interest, we can extract the parts of the captured image data 141 that change over time, thereby reducing the workload of annotation processing.

[0076] (A learning model that evaluates the degree of mixture) Next, a machine learning method using the captured image data 141 and 142 that have undergone annotation processing as described above will be explained. Specifically, the learning unit 114 of the control unit 11 of the information processing device 10 shown in Figure 2 performs machine learning using annotation information 123, which includes the pair of captured image data 140 labeled as described above, as training data. In other words, the learning unit 114, as a learning means, acquires image information 121 and coordinate information 122 from the storage unit 12 as learning input parameters and acquires annotation information 123 as learning output parameters to generate a learning model that evaluates the degree of mixture through machine learning such as supervised learning. Various types of machine learning can be employed, such as deep learning using neural networks, and can be carried out by mini-batch learning, etc.

[0077] By performing annotation processing using multiple image data 141 and 142 from multiple storage pits 26, it becomes possible to assign highly accurate labels, thereby improving the accuracy of machine learning.

[0078] According to the embodiment described above, annotation processing can be performed on the image data 141 and 142 of multiple storage pits 26, enabling high-precision labeling, thereby improving the accuracy of machine learning for generating a learning model. Furthermore, based on the operator's operation history on the image data pair 140, a correspondence map that allows coordinate transformation between one image data 141 and 142 and the other image data 141 and 142 can be automatically generated for multiple image data 141 and 142 captured from multiple different directions at the same time. Therefore, based on the correspondence map, the labeling of some image data 141 can be automatically reflected in other image data 142, thereby reducing the workload of annotation processing on the image data pair 140.

[0079] Although one embodiment of the present invention has been described in detail above, the present invention is not limited to the above-described embodiment, and various modifications are possible based on the technical idea of ​​the present invention. For example, the numerical values ​​given in the above-described embodiment are merely examples, and different numerical values ​​may be used as needed, and the present invention is not limited by the description and drawings that constitute part of the disclosure of the present invention in this embodiment.

[0080] For example, in the embodiment described above, the waste 26a stored in the storage pit 26 is used as the target object for the image data 141 and 142. However, image data of the number of people gathered at a designated venue or the degree of congestion at the venue may also be used. In this case, the learning model can generate input and output datasets for generating a learning model that evaluates the degree of congestion.

[0081] Furthermore, in one embodiment, the terms "part" as described above can be replaced with "circuit" or the like. For example, the control unit can be replaced with a control circuit.

[0082] Further effects and modifications can be readily derived by those skilled in the art. Broader aspects of this disclosure are not limited to the specific details and representative embodiments expressed and described above. Therefore, various modifications are possible without departing from the spirit or scope of the overall concept of the invention as defined by the appended claims and their equivalents. [Explanation of Symbols]

[0083] 1. Annotation Data Generation System 2 Network 3. Waste treatment facilities 10 Information Processing Devices 11,21 Control Unit 12 Storage section 13 Communications Department 14 Input / output section 20 Waste storage facilities 22 Communications Department 23 Imaging Unit 25 Gripping part 26 Storage pit 26a Waste 26b Wall 30 Waste Incineration Facilities 31 Combustion control device 32 Sensor section 33 Incinerator 100 Image data 111 Image Processing Unit 112 Annotation Processing Unit 113 pit control unit 114 Learning Department 121 Image Information 122 Coordinate Information 123 Annotation Information 140 pairs of image data Image data of 141, 141A, 141B, 141C, and 142. 141a, 142a Image Patch 231,232 Imaging cameras 251 Cranes 252 buckets A1 Area of ​​Interest A2 Coverage Area

Claims

1. An information processing method performed by a control unit equipped with hardware, A storage step involves storing in a storage unit multiple image data obtained by imaging the same object from different directions using multiple imaging means, and time information relating to the time when the object was imaged by the multiple imaging means, associated with each of the multiple image data. A display step in which at least some of the captured image data from a plurality of captured image data, whose capture time is the same in the time information read from the storage unit, is displayed on the display unit; A step of setting a focus area to set a predetermined focus area for at least one of the displayed image data images, A corresponding region setting step in which, for imaging image data other than the imaging image data in which the area of ​​interest is set, among the plurality of imaging image data, a corresponding region is set in the object that corresponds to substantially the same area as the area of ​​interest, The process includes a map generation step of generating a correspondence map, which represents the relationship between the position of the region of interest in some of the captured image data and the position of the corresponding region in other captured image data, and storing it in the storage unit. From the storage unit, read out a plurality of image data images taken by the same imaging means, but in which the time information contained in the time information is different from each other. In two of the aforementioned plurality of captured image data, the difference between the two captured image data as time progresses in the time information is derived. For the image data with the later time included in the time information of the two image data sets, the region containing the derived difference is set as the region of interest. Information processing methods.

2. An information processing method performed by a control unit equipped with hardware, A storage step involves storing in a storage unit multiple image data obtained by imaging the same object from different directions using multiple imaging means, and time information relating to the time when the object was imaged by the multiple imaging means, associated with each of the multiple image data. A display step in which at least some of the captured image data from a plurality of captured image data, whose capture time is the same in the time information read from the storage unit, is displayed on the display unit; A step of setting a focus area to set a predetermined focus area for at least one of the displayed image data images, A corresponding region setting step in which, for imaging image data other than the imaging image data in which the area of ​​interest is set, among the plurality of imaging image data, a corresponding region is set in the object that corresponds to substantially the same area as the area of ​​interest, The process includes a map generation step of generating a correspondence map, which represents the relationship between the position of the region of interest in some of the captured image data and the position of the corresponding region in other captured image data, and storing it in the storage unit. Position coordinates are set in advance for the object, and the object is imaged by one of the plurality of imaging means, and the position coordinates are converted to position coordinates in the image data obtained from the image data. Based on the positional coordinates in the converted image data, the coordinates of the region of interest are determined. Information processing methods.

3. For a plurality of image data obtained by imaging the object using the plurality of imaging means, The coordinates of the aforementioned position are converted into the coordinates of the respective positions in the plurality of captured image data. Based on the positional coordinates in the converted image data, the coordinates of the region of interest and the coordinates of the corresponding region are determined. Based on the aforementioned coordinates, the correspondence between the region of interest and the corresponding region is derived. The information processing method according to claim 2.

4. An information processing method performed by a control unit equipped with hardware, A storage step involves storing in a storage unit multiple image data obtained by imaging the same object from different directions using multiple imaging means, and time information relating to the time when the object was imaged by the multiple imaging means, associated with each of the multiple image data. A display step in which at least some of the captured image data from a plurality of captured image data, whose capture time is the same in the time information read from the storage unit, is displayed on the display unit; A step of setting a focus area to set a predetermined focus area for at least one of the displayed image data images, A corresponding region setting step in which, for imaging image data other than the imaging image data in which the area of ​​interest is set, among the plurality of imaging image data, a corresponding region is set in the object that corresponds to substantially the same area as the area of ​​interest, The process includes a map generation step of generating a correspondence map, which represents the relationship between the position of the region of interest in some of the captured image data and the position of the corresponding region in other captured image data, and storing it in the storage unit. The correspondence between the area of ​​interest and the corresponding area is derived based on data measured by a distance measuring sensor on the object. Information processing methods.

5. A label set externally is set to the area of ​​interest, The labeling step includes setting the label set for the area of ​​interest based on the correspondence map to the corresponding area that corresponds to the area of ​​interest on which the label is set, thereby labeling the plurality of captured image data. The information processing method according to any one of claims 1 to 3.

6. The aforementioned step of setting the area of ​​interest and the aforementioned step of setting the corresponding area are performed based on information input to the control unit from an external source. The information processing method according to any one of claims 1 to 3.

7. The object in question is waste stored in a storage pit in a waste storage facility. The information processing method according to any one of claims 1 to 3.

8. A storage unit that stores image data obtained by imaging an object, A display unit capable of displaying the aforementioned captured image data, The system comprises a control unit that performs image processing on the captured image data, The control unit stores in the storage unit, in association with multiple image data obtained by imaging the same object from different directions using multiple imaging means, and time information relating to the time when the object was imaged by the multiple imaging means. The display unit displays at least some of the multiple captured image data, whose capture time is the same in the time information read from the storage unit. If a predetermined area of ​​interest is set for at least one of the captured image data displayed on the display unit, and a corresponding area corresponding to substantially the same area as the area of ​​interest is set for other captured image data among the plurality of captured image data other than the captured image data on which the area of ​​interest is set, the area of ​​interest and the corresponding area are associated, A correspondence map is generated as a correspondence between the position of the area of ​​interest in some of the captured image data and the position of the corresponding area in the other captured image data, and stored in the storage unit. From the storage unit, read out a plurality of image data images taken by the same imaging means, but in which the time information contained in the time information is different from each other. In two of the aforementioned plurality of captured image data, the difference between the two captured image data as time progresses in the time information is derived. For the image data with the later time included in the time information of the two image data sets, the region containing the derived difference is set as the region of interest. Information processing device.

9. A storage unit for storing image data obtained by imaging an object, A display unit capable of displaying the aforementioned captured image data, The system comprises a control unit that performs image processing on the captured image data, The control unit stores in the storage unit, in association with multiple image data obtained by imaging the same object from different directions using multiple imaging means, and time information relating to the time when the object was imaged by the multiple imaging means. The display unit displays at least some of the multiple captured image data, whose capture time is the same in the time information read from the storage unit. If a predetermined area of ​​interest is set for at least one of the captured image data displayed on the display unit, and a corresponding area corresponding to substantially the same area as the area of ​​interest is set for other captured image data among the plurality of captured image data other than the captured image data on which the area of ​​interest is set, the area of ​​interest and the corresponding area are associated, A correspondence map is generated as a correspondence between the position of the area of ​​interest in some of the captured image data and the position of the corresponding area in the other captured image data, and stored in the storage unit. Position coordinates are set in advance for the object, and the object is imaged by one of the plurality of imaging means, and the position coordinates are converted to position coordinates in the image data obtained from the image data. Based on the positional coordinates in the converted image data, the coordinates of the region of interest are determined. Information processing device.

10. A storage unit for storing image data obtained by imaging an object, A display unit capable of displaying the aforementioned captured image data, The system comprises a control unit that performs image processing on the captured image data, The control unit stores in the storage unit, in association with multiple image data obtained by imaging the same object from different directions using multiple imaging means, and time information relating to the time when the object was imaged by the multiple imaging means. The display unit displays at least some of the multiple captured image data, whose capture time is the same in the time information read from the storage unit. If a predetermined area of ​​interest is set for at least one of the captured image data displayed on the display unit, and a corresponding area corresponding to substantially the same area as the area of ​​interest is set for other captured image data among the plurality of captured image data other than the captured image data on which the area of ​​interest is set, the area of ​​interest and the corresponding area are associated, A correspondence map is generated as a correspondence between the position of the area of ​​interest in some of the captured image data and the position of the corresponding area in the other captured image data, and stored in the storage unit. The correspondence between the area of ​​interest and the corresponding area is derived based on data measured by a distance measuring sensor on the object. Information processing device.

11. A control unit equipped with hardware, A storage step involves associating multiple image data obtained by imaging the same object from different directions using multiple imaging means with time information relating to the time when the object was imaged by the multiple imaging means and storing it in a storage unit. A display step in which at least some of the captured image data from a plurality of captured image data, whose capture time is the same in the time information read from the storage unit, is displayed on the display unit; A step of setting a focus area to set a predetermined focus area for at least one of the displayed image data images, A corresponding region setting step in which, for imaging image data other than the imaging image data in which the area of ​​interest is set, among the plurality of imaging image data, a corresponding region is set in the object that corresponds to substantially the same area as the area of ​​interest, The process involves generating a map generation step in which the correspondence between the position of the area of ​​interest in some of the captured image data and the position of the corresponding area in the other captured image data is generated as a correspondence map and stored in the storage unit. A program, wherein the program further includes the control unit, From the storage unit, read out a plurality of image data images taken by the same imaging means, but in which the time information contained in the time information is different from each other. In two of the aforementioned plurality of captured image data, the difference between the two captured image data as time progresses in the time information is derived. For the image data with the later time included in the time information of the two image data sets, the region containing the derived difference is set as the region of interest. A program that performs an action.

12. A control unit comprising hardware, A storage step involves associating multiple image data obtained by imaging the same object from different directions using multiple imaging means with time information relating to the time when the object was imaged by the multiple imaging means and storing it in a storage unit. A display step in which at least some of the captured image data from a plurality of captured image data, whose capture time is the same in the time information read from the storage unit, is displayed on the display unit; A step of setting a focus area to set a predetermined focus area for at least one of the displayed image data images, A corresponding region setting step in which, for imaging image data other than the imaging image data in which the area of ​​interest is set, among the plurality of imaging image data, a corresponding region is set in the object that corresponds to substantially the same area as the area of ​​interest, The process involves generating a map generation step in which the correspondence between the position of the area of ​​interest in some of the captured image data and the position of the corresponding area in the other captured image data is generated as a correspondence map and stored in the storage unit. A program, wherein the program further includes the control unit, Position coordinates are set in advance for the object, and the object is imaged by one of the plurality of imaging means, and the position coordinates are converted to position coordinates in the image data obtained from the image data. Based on the positional coordinates in the converted image data, the coordinates of the region of interest are determined. A program that performs an action.

13. A control unit comprising hardware, A storage step involves associating multiple image data obtained by imaging the same object from different directions using multiple imaging means with time information relating to the time when the object was imaged by the multiple imaging means and storing it in a storage unit. A display step in which at least some of the captured image data from a plurality of captured image data, whose capture time is the same in the time information read from the storage unit, is displayed on the display unit; A step of setting a focus area to set a predetermined focus area for at least one of the displayed image data images, A corresponding region setting step in which, for imaging image data other than the imaging image data in which the area of ​​interest is set, among the plurality of imaging image data, a corresponding region is set in the object that corresponds to substantially the same area as the area of ​​interest, The process involves generating a map generation step in which the correspondence between the position of the area of ​​interest in some of the captured image data and the position of the corresponding area in the other captured image data is generated as a correspondence map and stored in the storage unit. A program, wherein the program further includes the control unit, The correspondence between the area of ​​interest and the corresponding area is derived based on data measured by a distance measuring sensor on the object. A program that performs an action.

Images