Information processing method, information processing device, and program

Abstract

To annotate a state of an object with high accuracy in imaged image data obtained by imaging by an imaging camera.SOLUTION: An information processing method includes: storing, in a storage section, a plurality of pieces of imaged image data obtained by imaging an object stored in a target facility from different directions by imaging means; identifying, for at least one imaged image data read from the storage section, an object area in which the object is imaged and a non-extraction area in which the object is not imaged; setting a planar projection surface that includes the object in the imaged image data and can move in parallel along a predetermined movement line; deriving an evaluation index set due to the number of pixels in the identified object area with respect to the projection surface and selecting the projection surface, as a selected projection surface, when the evaluation index satisfies a predetermined condition; and performing geometric conversion processing on the basis of the selected projection surface to generate converted image data in which the object is projected on the selected projection surface.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] This invention relates to an information processing method, an information processing device, and a program. [Background technology]

[0002] In recent years, in order to improve the operational efficiency of incinerators in waste treatment facilities, there has been a need for the development of technologies to understand the contents of storage pits, such as using machine learning to determine and recognize the mixing state and type of waste within the storage pit. For example, Patent Documents 1, 2, and 3 propose technologies for estimating the mixing state of waste pits using images. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Patent No. 6731680 [Patent Document 2] Patent No. 6603822 [Patent Document 3] Patent No. 6856522 [Overview of the project] [Problems that the invention aims to solve]

[0004] However, the shape of waste deposits in storage pits, such as waste pits, changes constantly due to agitation by cranes. Therefore, even if the waste is in the same storage pit and under similar agitation conditions, the appearance from each camera will be completely different depending on the installation position and imaging angle of the multiple cameras that image the inside of the storage pit. Even when labeling multiple image data obtained by imaging the waste inside the storage pit with such multiple cameras and performing machine learning on the image data, it has been difficult to improve recognition accuracy. This problem occurs not only when imaging waste inside storage pits, but also when performing machine learning on multiple image data obtained by imaging an object with multiple cameras. Therefore, there has been a need for a technology that can improve the recognition accuracy of objects in multiple image data obtained by multiple cameras.

[0005] This invention has been made in view of these circumstances, and its purpose is to provide an information processing method, an information processing device, and a program that can improve the accuracy of object recognition in multiple image data captured by multiple imaging cameras. [Means for solving the problem]

[0006] To solve the above-mentioned problems and achieve the objective, an information processing method according to one aspect of the present invention is an information processing method executed by a control unit equipped with hardware, comprising: a storage step of storing a plurality of image data obtained by imaging an object stored in a target facility from different directions using a plurality of imaging means in a storage unit; an identification step of an object region in which the object is imaged and a non-extracted region in which the object is not imaged for at least one image data read from the storage unit; a setting step of setting a planar projection plane that includes the object in the image data and can be moved in parallel along a predetermined movement line; a selection step of deriving an evaluation index set based on the number of pixels in the identified object region for the projection plane, selecting a projection plane when the evaluation index satisfies predetermined conditions and setting it as a selected projection plane; and a conversion step of generating converted image data in which the object is projected onto the selected projection plane by performing geometric transformation processing based on the selected projection plane.

[0007] An information processing method according to one aspect of the present invention is characterized in that, in the above invention, the object is housed in a predetermined space having a flat bottom surface, and the predetermined projection plane is set to be parallel to the bottom surface or perpendicular to gravity.

[0008] In one aspect of the present invention, the information processing method is derived based on the following equation when λ is a weighting coefficient greater than 0. Evaluation index = (Number of pixels in the object area within the projection plane - Number of pixels in the object area outside the projection plane × λ) / Total number of pixels on the projection plane

[0009] In one aspect of the present invention, the information processing method is characterized in that, in the above invention, the captured image data is associated with time information relating to the time when the object was captured by the imaging means.

[0010] An information processing method according to one aspect of the present invention involves setting the selected projection plane for a portion of the captured image data among a plurality of captured image data, the time information read from the storage unit, where the time of imaging is the same, and applying the selected projection plane to the remaining captured image data.

[0011] An information processing method according to one aspect of the present invention, in the above invention, sets the selected projection plane for each of the multiple captured image data, which are read from the storage unit and whose time information indicates that the time of imaging is the same, independently of each other.

[0012] An information processing method according to one aspect of the present invention is characterized in that, in the above invention, the target facility is a storage pit in a waste storage facility, and the target object is waste stored in the storage pit.

[0013] An information processing apparatus according to one aspect of the present invention comprises a storage unit for storing image data obtained by imaging an object, and a control unit for performing image processing on the image data, wherein the control unit stores multiple image data obtained by imaging an object stored in a target facility from different directions using multiple imaging means in the storage unit, identifies an object region in which the object is imaged and a non-extracted region in which the object is not imaged for at least one image data read from the storage unit, sets a planar projection plane that includes the object in the image data and can be moved in parallel along a predetermined movement line, derives an evaluation index set based on the number of pixels in the identified object region for the projection plane, selects a projection plane when the evaluation index satisfies predetermined conditions and sets it as a selected projection plane, and generates transformed image data in which the object is projected onto the selected projection plane by performing geometric transformation processing based on the selected projection plane and stores it in the storage unit.

[0014] A program according to an aspect of the present invention causes a control unit having hardware to perform a storage step of causing a storage unit to store a plurality of imaging image data obtained by imaging an object stored in a target facility from different directions by a plurality of imaging means, an identification step of identifying, for at least one imaging image data read from the storage unit, an object region in which the object is imaged and a non-extraction region in which the object is not imaged, a setting step of setting a planar projection surface that includes the object in the imaging image data and is translatable along a predetermined movement line, a selection step of deriving an evaluation index set based on the number of pixels of the identified object region with respect to the projection surface, and setting the projection surface that satisfies a predetermined condition as a selected projection surface, and a conversion step of generating converted image data in which the object is projected onto the selected projection surface by performing geometric conversion processing based on the selected projection surface.

Advantages of the Invention

[0015] According to the information processing method, information processing apparatus, and program of the present invention, it is possible to improve the recognition accuracy of an object in a plurality of imaging image data captured by a plurality of imaging cameras.

Brief Description of the Drawings

[0016] [Figure 1] FIG. 1 is a block diagram showing an information processing system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing an information processing apparatus according to an embodiment of the present invention. [Figure 3] FIG. 3 is a diagram showing an example of imaging image data obtained by imaging the inside of a storage pit in an embodiment of the present invention. [Figure 4] FIG. 4 is a diagram showing an example of a pair of imaging image data when the inside of a storage pit is imaged by two imaging cameras in an embodiment of the present invention. [Figure 5] FIG. 5 is a flowchart for explaining an information processing method according to an embodiment of the present invention. [Figure 6A] Figure 6A shows an example of the setting of the bottom surface of a storage pit included in the image data of one embodiment of the present invention. [Figure 6B] Figure 6B shows an example of a state in which the bottom surface of the storage pit included in the image data of one embodiment of the present invention is virtually translated. [Figure 7] Figure 7 shows an example of binarized image data obtained by binarizing the captured image data shown in Figure 3 into the object region and the non-extracted region. [Figure 8A] Figure 8A shows binarized image data at a predetermined evaluation index to explain the method for obtaining the projection plane for the binarized image data shown in Figure 7. [Figure 8B] Figure 8B shows binarized image data at a predetermined evaluation index to explain the method for obtaining the projection plane for the binarized image data shown in Figure 7. [Figure 8C] Figure 8C shows binarized image data at a predetermined evaluation index to explain the method for obtaining the projection plane for the binarized image data shown in Figure 7. [Figure 9] Figure 9 is a diagram illustrating the height from the bottom surface of the projection plane set in the binarized image data in one embodiment of the present invention. [Figure 10A] Figure 10A shows an example of image data and corresponding converted image data captured by one imaging camera inside a storage pit in one embodiment of the present invention. [Figure 10B] Figure 10B shows an example of image data and corresponding converted image data captured by the other imaging camera inside the storage pit in one embodiment of the present invention. [Figure 11A] Figure 11A shows an example of how the projection planes are made relative to each other for a pair of image data obtained by imaging the inside of a storage pit with two imaging cameras according to one embodiment of the present invention. [Figure 11B] Figure 11B shows a pair of converted image data corresponding to the pair of captured image data shown in Figure 11A. [Modes for carrying out the invention]

[0017] Hereinafter, one embodiment of the present invention will be described with reference to the drawings. In all the drawings of the following embodiment, the same or corresponding parts will be denoted by the same reference numerals. Furthermore, the present invention is not limited to the embodiment described below.

[0018] First, in describing the information processing device according to one embodiment of the present invention, the inventors' diligent studies will be explained. Specifically, according to the inventors' findings, for example, in a facility for storing objects, it is conceivable to install multiple imaging cameras at different locations to image the entire facility with high precision. Furthermore, if objects are mixed or changed within the facility, the shape of the objects within the facility may change moment by moment. In addition, in image data captured within the facility, if the distance between multiple imaging cameras is large, even if the predetermined area for evaluating the object region is the same, it becomes difficult to associate the respective image data captured by the multiple imaging cameras with the same predetermined area in each image data.

[0019] For example, if the target facility is a waste storage pit and the target object is waste, the storage pit is large, so multiple cameras are often used to image the waste. In this case, even the same waste at the same location within the storage pit will look different depending on the distance from the camera and the direction of imaging. Specifically, if each image data captured by each camera is annotated individually, there is a high possibility of incorrect labeling due to the different viewing angles. In other words, in different image data captured at the same time within the storage pit by different cameras, even at the same location, the degree of mixing of waste captured nearby and waste captured from a distance will appear different. Therefore, when an operator performs visual annotation, there is a possibility of assigning different labels to the same waste. Furthermore, if the pile shape of the waste within the storage pit changes constantly due to agitation by cranes or other means, even the same waste and the same area captured within the storage pit may look completely different depending on the position and angle from the camera. Therefore, when an object is imaged using multiple different cameras, the label mapping is unstable even when an operator labels the image data obtained from these multiple cameras. As a result, it has been difficult to improve the accuracy of object recognition even when performing machine learning using multiple image data obtained in this way.

[0020] Therefore, the inventors investigated a method for performing annotation work on multiple image data captured from different directions within a target facility. The inventors devised a method to reduce the change in appearance due to the shape of an object by extracting the area in which an object exists (object region) from image data captured within a target facility and applying an appropriate geometric transformation process. This makes it possible to improve the recognition accuracy of the acquired learning model for the image data by including the positional information of the image data when performing machine learning using the image data. In other words, the inventors conceived a method to generate transformed image data by applying a predetermined geometric transformation process to multiple image data obtained by capturing the target facility or object from different directions at the same time, thereby enabling high-precision correspondence between multiple image data. As a result, the operator performing the annotation work can recognize the transformed image data, obtained by geometrically transforming multiple image data obtained from different directions of the same object, as images viewed from the same position, thus reducing the need for corrections and adjustments during the annotation work and reducing the workload of the annotation work. One embodiment of the present invention described below was devised based on the inventors' diligent research described above.

[0021] (Annotation data generation system) Figure 1 shows an information processing system to which an information processing device according to one embodiment of the present invention is applied. As shown in Figure 1, the annotation data generation system 1, as an information processing system, comprises an information processing device 10, a waste storage facility 20, and a waste incineration facility 30, all of which are able to communicate with each other via a network 2. The waste treatment facility 3 comprises at least the waste storage facility 20 and the waste incineration facility 30. The information processing device 10 may be located externally and able to communicate with the waste treatment facility 3 via the network 2, or it may be part of the waste treatment facility 3. Furthermore, the information processing device 10 may be located inside the waste storage facility 20, and its installation location is not limited.

[0022] Network 2 is composed of a combination of wired and wireless communications as appropriate, and consists of communication networks such as the Internet network and mobile phone network. Network 2 consists of one or more combinations of, for example, dedicated lines, public communication networks such as the Internet, such as LAN (Local Area Network), WAN (Wide Area Network), telephone communication networks and public lines such as mobile phones, and VPN (Virtual Private Network). The information processing device 10, the waste storage facility 20, and the waste incineration facility 30 are connected via Network 2.

[0023] (Waste incineration facility) The waste incineration equipment 30, which serves as the waste incineration unit, has a conventionally known configuration. Specifically, the waste incineration equipment 30 comprises a combustion control device (ACC) 31, a sensor unit 32, and an incinerator 33. The combustion control device 31 controls the amount of combustion air, the amount of cooling air, the waste supply device feed speed, and the grate feed speed as the control variables at each control end, based on the setting of predetermined control variable reference values. The incinerator 33, which is the waste incinerator, comprises a furnace where waste 26a is burned, a waste input port for feeding waste 26a, and a boiler (none of which are shown). The sensor unit 32 consists of, for example, thermometers and pressure gauges installed in various locations. The internal state of the incinerator 33 and various physical quantities such as pressure and velocity in facilities related to the incinerator 33, specifically, in power generation facilities for generating electricity, are measured by the sensor unit 32 and output as sensor information from the sensor unit 32. The sensor information output from the sensor unit 32 is supplied to the combustion control device 31 as parameters. The combustion control device 31 controls the combustion of the incinerator 33 based on the input parameters.

[0024] (Waste storage facility) The waste storage facility 20, which serves as a waste storage unit, comprises a control unit 21, a communication unit 22, an imaging unit 23, a gripping unit 25, and a storage pit 26. The storage pit 26 is movably equipped with a gripping unit 25 and an imaging unit 23. The gripping unit 25 is controlled by the control unit 21 based on a control signal transmitted from the control unit 11 of the information processing device 10. The control unit 21 may also control the imaging unit 23 based on a control signal transmitted from the control unit 11. Alternatively, the control unit 11 of the information processing device 10 may directly control the imaging unit 23 and the gripping unit 25.

[0025] The control unit 21 specifically includes a processor with hardware such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and an FPGA (Field-Programmable Gate Array), as well as main memory such as RAM (Random Access Memory) and ROM (Read Only Memory) (none of which are shown). The control unit 21 controls the imaging unit 23 and the gripping unit 25 based on control signals input from the information processing device 10 via the communication units 13 and 22, according to various programs stored in the main memory such as RAM and ROM.

[0026] The communication unit 22 is, for example, a LAN interface board, a wired communication circuit for wired communication, or a wireless communication circuit for wireless communication. The LAN interface board, wired communication circuit, or wireless communication circuit is connected to the network 2. The communication unit 22, acting as both a transmitter and receiver, is connected to the network 2 and communicates with the information processing device 10.

[0027] The gripping unit 25 grips and moves the waste 26a stored in the storage pit 26. The bucket 252, which acts as an opening and closing unit, can grip the waste 26a. The crane 251, which acts as a moving unit, is configured to be movable by connecting to the bucket 252. The crane 251 is controlled by a control signal transmitted from the control unit 11 of the information processing device 10. The storage pit 26 is a space enclosed by an inner wall 26b, and is a pit capable of temporarily storing waste 26a. The waste 26a in the storage pit 26 is gripped by the gripping unit 25 and supplied to the incinerator 33 of the waste incineration equipment 30, where it is incinerated.

[0028] The imaging unit 23 has, for example, two imaging cameras 231 and 232 as multiple imaging means. The imaging unit 23 is configured to be able to image the waste 26a in the storage pit 26 and the surface of the inner wall 26b using the imaging cameras 231 and 232. The imaging cameras 231 and 232 that make up the imaging unit 23 each image the state of the waste 26a in the storage pit 26 from different directions.

[0029] Figure 3 shows the image data generated by the imaging camera 231. The image data 100, as imaging information shown in Figure 3 and obtained by imaging camera 231, is associated with the time information at the time of imaging and transmitted from imaging camera 231 to the information processing device 10 via the communication unit 22. Similarly, image data 100 obtained by imaging camera 232 from a different direction than imaging camera 231, capturing the inside of the storage pit 26, is also associated with the time information at the time of imaging and transmitted from imaging camera 232 to the information processing device 10 via the communication unit 22. This allows the information processing device 10 to associate multiple different image data 100 obtained by imaging cameras 231 and 232 capturing the inside of the storage pit 26 from different directions based on the time information. In other words, the information processing device 10 can process multiple image data 100 captured by multiple different imaging cameras 231 and 232, with the same time information, as image data 100 captured simultaneously at the same time. The same applies even if there are three or more imaging cameras.

[0030] (Information processing device) Figure 2 shows details of the information processing device 10 in Figure 1. As shown in Figure 2, the information processing device 10 comprises a control unit 11, a storage unit 12, a communication unit 13, and an input / output unit 14. The control unit 11 and the communication unit 13 are physically the same as the control unit 21 and the communication unit 22 described above, respectively. The information processing device 10 according to this embodiment functions as an information processing device that performs labeling and machine learning by applying appropriate geometric transformations to captured image data 100 of waste 26a in the storage pit 26 to reduce variations in appearance. The information processing device 10 also functions as an information processing device that generates an input / output dataset that serves as training data for a learning model capable of measuring the degree of mixing by labeling the mixing state of the waste in the storage pit 26. The learning model is also referred to as a trained model or simply a model.

[0031] The input / output unit 14 can be composed of, for example, a touch panel display or a speaker microphone. As an input means, the input / output unit 14 includes an interface that receives various information transmitted from, for example, an imaging unit 23 installed in the waste storage facility 20 via the communication unit 22 and outputs it to the control unit 11. The transmission of information from the imaging unit 23 to the input / output unit 14 may be done using either wired or wireless communication. The input / output unit 14 also includes a user interface such as a keyboard, input buttons, levers, a touch panel for manual input superimposed on a display such as an LCD, or a microphone for voice recognition. The input / output unit 14 is configured to allow predetermined information to be input to the control unit 11 by an operator or other person operating it. As an output means, the input / output unit 14 displays images of the storage pit 26 of the waste storage facility 20 on a display monitor, displays characters or figures on the screen of a touch panel display, or outputs sound from a speaker, according to the control unit 11's control. In other words, the input / output unit 14 is configured to be able to notify predetermined information to the outside. The input and output sections of the input / output unit 14 may be configured as separate units.

[0032] The storage unit 12 is composed of a storage medium selected from volatile memory such as RAM, non-volatile memory such as ROM, EPROM (Erasable Programmable ROM), hard disk drive (HDD), and removable media. Removable media include, for example, USB (Universal Serial Bus) memory, or disk recording media such as CD (Compact Disc), DVD (Digital Versatile Disc), or BD (Blu-ray® Disc). Alternatively, the storage unit 12 may be configured using a computer-readable recording medium such as an externally insertable memory card.

[0033] The memory unit 12 can store various programs such as an operating system (OS), image processing applications, various tables, and various databases for executing the operation of the information processing device 10. These various programs can also be recorded on computer-readable recording media such as hard disks, flash memory, CD-ROMs, DVD-ROMs, and flexible disks and widely distributed.

[0034] The memory unit 12 stores image information 121, coordinate information 122, annotation information 123, and a region extraction model 124. The image information 121, coordinate information 122, and annotation information 123 are all stored in the memory unit 12 as a searchable database. The region extraction model 124 is a learning model that extracts a predetermined region from captured image data.

[0035] Image information 121 includes a plurality of captured image data 100 (see Figure 3) obtained by imaging waste 26a in the storage pit 26 with the imaging unit 23, and time information relating to the time the waste 26a was imaged. The captured image data 100 is associated with the time information in which it was captured. Coordinate information 122 includes coordinate information that has been transformed and associated based on various image processing operations performed by the operator on the captured image data 100 of the image information 121 using a predetermined image processing application and the input / output unit 14 of the information processing device 10. Annotation information 123 includes annotation data (not shown) obtained by applying annotation processing to the captured image data 100 included in the image information 121 obtained as described above.

[0036] The control unit 11 loads programs stored in the memory unit 12 into the work area of ​​the main memory unit and executes them, thereby controlling each component through program execution to realize functions that match a predetermined purpose. Specifically, the control unit 11 can execute the functions of the image processing unit 111, the annotation processing unit 112, and the learning unit 113 by executing various programs loaded from the memory unit 12. The various programs also include programs that realize artificial intelligence and trained models that can realize the processing according to this embodiment. Details of the processing by the control unit 11 will be described later.

[0037] (Information processing methods) Next, we will describe the information processing method performed by the information processing device 10 according to this embodiment, which is configured as described above. In the following description, information is transmitted and received between each component via the communication units 13 and 22 and the network 2, but we will omit further explanation of this each time.

[0038] Figure 4 is a diagram illustrating the method for imaging waste 26a in the storage pit 26 using the information processing method according to this embodiment. As shown in Figure 4, when imaging the inside of the storage pit 26, which is the target facility according to this embodiment, with multiple imaging cameras 231, 232, it is preferable, but not limited, that these imaging cameras 231, 232 be installed in positions that can image as wide an area of ​​waste 26a as possible. The installation positions of the multiple imaging cameras 231, 232 can be appropriately changed according to the shape and dimensions of the storage pit 26.

[0039] As shown in Figure 4, waste 26a is stored in the storage pit 26. In the example shown in Figure 4, one imaging camera 231 is provided in one of the four corners of the storage pit 26, and the imaging camera 231 is capable of imaging at least the surface layer of the waste 26a stored in the storage pit 26 from a predetermined direction, i.e., from a predetermined viewpoint. By imaging the waste 26a in the storage pit 26 with the imaging camera 231, the imaging camera 231 generates image data 141 and transmits it to the information processing device 10. Similarly, the other imaging camera 232 is provided in one of the four corners of the storage pit 26, different from the imaging camera 231, and the imaging camera 232 is capable of imaging at least the surface layer of the waste 26a stored in the storage pit 26 from a direction different from the predetermined direction, i.e., from a viewpoint different from the viewpoint described above. By imaging the waste 26a in the storage pit 26 with the imaging camera 232, the imaging camera 232 generates image data 142 and transmits it to the information processing device 10.

[0040] The image data 141 and 142 obtained by imaging the inside of the storage pit 26 are associated with time information and also include information about the waste 26a stored in the storage pit 26 and the inner wall 26b of the storage pit 26. Although not shown in Figure 4, the crane 251 and bucket 252 used to grasp and release the waste 26a may also be imaged.

[0041] (Conversion process method) Next, a conversion processing method for the captured image data 100 as part of the annotation method executed by the information processing device 10 according to this embodiment will be described. Figure 5 is a flowchart illustrating the conversion processing method executed by the information processing device 10 according to this embodiment. Figures 6A and 6B show examples of the bottom surface of a storage pit and a state in which the bottom surface has been translated, respectively, included in the captured image data 100 according to this embodiment. Figure 7 shows an example of binarized image data 101 obtained by binarizing the captured image data 100 into an object region and a non-extracted region. Figures 8A, 8B, and 8C show binarized image data for each evaluation index to explain the method of obtaining a projection surface for the binarized image data 101, respectively. Figure 9 is a diagram illustrating the height from the bottom surface of the projection surface set in the binarized image data 101 according to this embodiment.

[0042] (Inner shape setting process) As shown in Figure 5, in step ST1, the image processing unit 111 of the control unit 11 of the information processing device 10 acquires image data 100, for example, as shown in Figure 3, obtained by imaging the inside of the storage pit 26 from the imaging cameras 231 and 232. The image processing unit 111 sets the shape of the storage pit 26 based on the acquired image data 100. The shape of the inner surface of the storage pit 26 is usually a rectangular parallelepiped, but it may also be a cylindrical, elliptical, or square parallelepiped, and various shapes can be set as long as the bottom surface can be defined. Furthermore, it is desirable that the bottom surface be flat, but it does not necessarily have to be flat. If the imaging cameras 231 and 232 are fixed in predetermined positions and their field of view is also fixed, the relative position between the imaging cameras 231 and 232 and the inner surface of the storage pit 26 is fixed. Therefore, the image processing unit 111 can set the inner surface of the image data 100 captured by the imaging cameras 231 and 232 based on the coordinate information of the inner surface of the storage pit 26 when no waste 26a as the target object is present.

[0043] Specifically, the image processing unit 111 obtains coordinate information 122 from the storage unit 12, which includes the coordinates of each position on the inner wall 26b and the bottom surface 100a, which are the inner circumferential surfaces within the storage pit 26, and the coordinates of the installation positions of the imaging cameras 231 and 232. Next, as shown in Figure 6A, the image processing unit 111 sets the bottom surface 100a and the movement lines 100b and 100c in the image data 100 based on the obtained coordinates of each position within the storage pit 26 and the installation positions of the imaging cameras 231 and 232. Here, the image processing unit 111 sets the movement lines 100b and 100c along the height direction perpendicular to the bottom surface 100a of the storage pit 26. As a result, if the bottom surface 100a of the storage pit 26 is rectangular, four movement lines corresponding to the four vertices of the rectangle can be virtually set along the height direction perpendicular to the bottom surface 100a. Furthermore, the bottom surface 100a of the storage pit 26 is not limited to a rectangular shape; it can be set to a circular, elliptical, closed curve, polygonal, or other shape.

[0044] In the image data 100 shown in Figure 6A, movement lines 100b and 100c are set at two corners opposite the imaging cameras 231 and 232, and a virtual movement line (not shown) is set on the side where the imaging cameras 231 and 232 are installed, i.e., on the near side in the image data 100. In other words, although the entire interior of the storage pit 26 is not captured in the image data 100, the image processing unit 111 sets a virtual movement line for the uncaptured area. Various methods can be used to set the movement lines, and are not limited to the method described above.

[0045] As described above, the image processing unit 111 can set the shape of the inner surface of the storage pit 26 when no waste 26a is stored inside. In other words, the image processing unit 111 can determine and fix the position of the bottom surface 100a hidden by the waste 26a within the captured image data 100. Furthermore, as shown in Figure 6B, the image processing unit 111 sets a parallel surface 100d parallel to the bottom surface 100a. The parallel surface 100d is parallel to the bottom surface 100a set as described above, and is a surface that can be virtually moved in parallel along the movement lines 100b and 100c. The parallel surface 100d is also set in a state superimposed on the captured image data 100 as a virtual surface parallel to the bottom surface 100a of the storage pit 26. If the bottom surface 100a is not a plane, the parallel surface 100d can be set or defined as a plane perpendicular to gravity. In this case, the movement lines 100b and 100c can be set approximately parallel to the direction of gravity. In this case as well, the parallel plane 100d is set as a virtual plane perpendicular to gravity and perpendicular to the movement lines 100b and 100c, superimposed on the captured image data 100. Furthermore, the above-mentioned perpendicular, perpendicular, and parallel refer to the relationships between lines and planes when the state captured in the captured image data 100 is applied to the actual storage pit 26, and are not limited to perpendicular, perpendicular, and parallel within the captured image data 100.

[0046] (Object region extraction process) Next, when the process moves to step ST2, the annotation processing unit 112 of the control unit 11 determines the boundary between the area where the waste material 26a is located (hereinafter referred to as the "object area") and the area other than the object area (hereinafter referred to as the "non-extracted area"), and identifies the object area in the captured image data 100 (identification step). Specifically, first, the annotation processing unit 112 reads the area extraction model 124 from the storage unit 12. The annotation processing unit 112 inputs the captured image data 100 as an input parameter to the read-out area extraction model 124. The area extraction model 124 is a model capable of classifying the object area and the non-extracted area in the captured image data 100 by machine learning or feature classification. In addition to the area extraction model 124 generated by machine learning, it is also possible to classify the object area and the non-extracted area of ​​the waste material 26a based on frequency characteristics.

[0047] The region extraction model 124 outputs binarized image data 101 as an output parameter, distinguishing between the target region and regions other than the target region, as shown in Figure 7. The binarized image data 101 is image data that has been binarized into the target region 101a (white area in Figure 7) and the unextracted region 101b (hatched region in Figure 7). The annotation processing unit 112 stores the binarized image data 101 as image information 121 in the storage unit 12. In this embodiment, the target region 101a of the binarized image data 101 is the region where waste 26a exists, and the unextracted region 101b is the region where the inner wall 26b of the storage pit 26 exists. Here, if the target object in the captured image data 100 is waste 26a and the unextracted region other than the target object is the inner wall 26b, the region where waste 26a exists and the region of the inner wall 26b have very different characteristics in the captured image data 100, and therefore they appear very different. Therefore, the annotation processing unit 112 can easily and accurately classify the target area 101a from the captured image data 100.

[0048] (Projection plane acquisition process) Next, moving to step ST3, the image processing unit 111 reads the binarized image data 101 as image information 121 from the storage unit 12. Based on the parallel plane 100d shown in Figure 6B, the image processing unit 111 performs the acquisition of a projection plane on the binarized image data 101. Based on the number of pixels in the object region 101a and the non-extracted region 101b, the image processing unit 111 acquires a projection plane 101d for geometric transformation processing. That is, the image processing unit 111 applies the set parallel plane 100d to the binarized image data 101 so as to superimpose it, and sets the projection plane 101d shown in Figure 8A. Next, for example, as shown in Figures 8A to 8C, the image processing unit 111 moves the projection plane 101d little by little in the vertical direction and derives an evaluation index from the following equation (1) based on the number of pixels in the object region 101a. The number of pixels of the projection plane 101d in the denominator corresponds to the number of pixels in the binarized image data 101, and normalization is performed.

[0049] Evaluation index P = (Number of pixels in the object area within the projection plane - Number of pixels in the object area outside the projection plane × λ) / Total number of pixels on the projection surface ……(1) Note that λ is a weighting coefficient (λ>0) for the object region 101a that is outside the range of the projection plane 101d. Furthermore, since the weighting coefficient λ is a penalty for the object region 101a that is outside the range of the projection plane 101d, it can be changed statically or dynamically depending on the accumulation state of waste 26a in the storage pit 26. Moreover, the evaluation index is not limited to equation (1), and other evaluation indices can also be adopted.

[0050] Specifically, we will consider the evaluation index with a weighting coefficient λ of 1 below. In the binarized image data 101A shown in Figure 8A, suppose the number of pixels in the object region 101a included in the projection plane 101d is 100 pixels, and the number of pixels in the object region 101a outside the projection plane 101d is 30 pixels. In this case, since the entire area inside the projection plane 101d is the object region 101a, the evaluation index P from equation (1) is ((100-30×λ) / 100=)0.7.

[0051] Furthermore, in the binarized image data 101B shown in Figure 8B, the number of pixels in the object region 101a included in the projection plane 101d is, for example, 120 pixels, and the number of pixels in the object region 101a outside the projection plane 101d is, for example, 10 pixels. Also, the number of pixels inside the projection plane 101d has increased compared to the binarized image data 101A shown in Figure 8A, and is, for example, about 115 pixels. In this case, from equation (1), the evaluation index is ((120-10×λ) / 115≈)0.9.

[0052] Furthermore, in the binarized image data 101C shown in Figure 8C, the number of pixels in the object region 101a included in the projection plane 101d is, for example, 130 pixels, and the number of pixels in the object region 101a outside the projection plane 101d is, for example, 0 pixels. Also, the number of pixels inside the projection plane 101d is further increased compared to Figure 8B, for example, to about 160. In this case, from equation (1), the evaluation index becomes ((130-0×λ) / 160≈)0.8.

[0053] As described above, the image processing unit 111 derives an evaluation index and selects the projection plane 101d with the highest evaluation index, for example, and sets it as the selected projection plane 101h shown in Figure 9 (selection step). In this case, it is desirable for the image processing unit 111 to select the projection plane 101d as the selected projection plane 101h, where the evaluation index satisfies predetermined conditions set in advance. In this embodiment, the predetermined condition is the condition where the evaluation index is the highest. In this case, in Figures 8A to 8C described above, it is desirable to select the projection plane 101d shown in Figure 8B. It is also possible for the image processing unit 111 to select the projection plane 101d with the lowest evaluation index P and set it as the selected projection plane 101h, or to select the projection plane 101d with the smallest difference of the evaluation index P from a predetermined value or the absolute value of the difference from the predetermined value and set it as the selected projection plane 101h. The image processing unit 111 stores the height information (hereinafter referred to as level information) of the selected projection plane 101h shown in Figure 9 as coordinate information 122 in the storage unit 12.

[0054] Next, the process moves to step ST4, where the image processing unit 111 reads the level information of the selected projection plane 101h from the coordinate information 122 of the storage unit 12. Next, the image processing unit 111 performs a predetermined image transformation process, such as an affine transformation, to project the captured image data 100 onto the selected projection plane 101h. Figures 10A and 10B show the captured image data 141 and 142 captured by the imaging cameras 231 and 232, and the transformed image data 141S and 142S after the image processing unit 111 has performed an affine transformation on these captured image data 141 and 142, respectively.

[0055] Specifically, the image processing unit 111 performs binarization on the captured image data 141 and 142 captured by the imaging cameras 231 and 232. The image processing unit 111 identifies the target region 101a and the non-extracted region 101b, and sets a selected projection plane 101h that gives the highest value to the evaluation index of the target region 101a. For example, the image processing unit 111 performs image conversion processing on the captured image data 141 on the left side of Figure 10A to the selected projection plane 101h, thereby generating the converted image data 141S on the right side. The converted image data 141S is image data output by projecting the captured image data 141 onto a plane parallel to the bottom surface (bottom surface 100a) of the storage pit 26. In other words, the converted image data 141S is image data obtained by converting the image data 141, which is an image of the waste 26a in the storage pit 26 taken from an oblique angle, into an image obtained by projecting it onto a selected projection plane 101h parallel to the bottom surface 100a, that is, a plane perpendicular to the direction of gravity.

[0056] Similarly, as shown in Figure 10B, the image data 142 (left side of Figure 10B) captured by the imaging camera 232 is converted to a selected projection plane 101h by the image processing unit 111 (conversion step). This allows the image processing unit 111 to generate the converted image data 142S on the right from the image data 142 on the left. Note that the setting of the selected projection plane 101h in the image data 141 and 142 may be performed independently of each other, or the average height H from the bottom surface 100a of the selected projection plane 101h in the image data 141 and 142 may be set as the common selected projection plane 101h. The setting of the selected projection plane 101h between multiple image data sets may be related to each other or independent of each other. Furthermore, the selected projection plane 101h set for one of multiple image data sets, for example, two image data sets 141 and 142, such as image data 141, may be set as the selected projection plane 101h for the remaining image data sets, such as image data 142. In this embodiment, the converted image data 141S is image data output by projecting the captured image data 141 onto a set selected projection plane 101h, so as to be projected onto a plane parallel to the bottom surface (bottom surface 100a) of the storage pit 26. In other words, the converted image data 141S is image data obtained by transforming the captured image data 141, which is an image of the waste 26a in the storage pit 26 taken from an oblique angle, so as if it were an image taken as if it were projected onto a selected projection plane 101h, which is a plane perpendicular to the direction of gravity. The converted image data 141S and 142S generated by the image processing unit 111 are stored in the storage unit 12 as image information 121. At least the quantity of these converted image data 141S and 142S necessary for learning is generated and stored in the storage unit 12 as image information 121.

[0057] Next, proceed to steps ST5 and ST6. Note that steps ST5 and ST6, described below, may be executed in parallel or in reverse order; the order of execution does not matter.

[0058] First, in step ST5, annotation processing is performed on the converted image data 141S and 142S, and image processing is carried out. Figure 11A shows the state in which predetermined positions have been selected in the captured image data 141 and 142 acquired from multiple imaging cameras 231 and 232, and Figure 11B shows the corresponding locations of the selected treatment positions in each of the converted image data 141S and 142S.

[0059] In step ST5, annotation processing is first performed on the converted image data 141S and 142S. That is, the waste 26a stored in the storage pit 26 is composed of various objects, and various states of agitation and mixing of the waste 26a are possible. In this embodiment, as shown in Figure 11A, the operator acquires multiple image data with the same time information, captured from different imaging directions at the same time using imaging cameras 231 and 232 set at different positions, capturing the same waste 26a. Then, by performing steps ST2 to ST4 described above, the converted image data 141S and 142S shown in Figure 11B are acquired. The operator performs annotation processing on the two converted image data 141S and 142S. Note that the information processing device 10 may output and display the two converted image data 141S and 142S simultaneously on the input / output unit 14, or output and display them separately.

[0060] The worker visually inspects the converted image data 141S and 142S displayed on the input / output unit 14 to confirm the waste 26a in the storage pit 26. During the annotation process, the worker uses a predetermined image processing device, in this embodiment the information processing device 10, to visually evaluate the state of the object captured in the converted image data 141S and 142S displayed on the input / output unit 14, based on the worker's experience and knowledge, in this case the mixing state of the waste 26a. This evaluation of the degree of mixing is also called discrimination or judgment of the degree of mixing.

[0061] The operator selects the region to be evaluated from the converted image data 141S to be evaluated using the input / output unit 14 of the information processing device 10. The image processing unit 111 of the control unit 11 then sets the selected region as image patch A1. The operator then selects image patch A2 of the converted image data 142S that corresponds to image patch A1 of the converted image data 141S. Alternatively, the annotation processing unit 112 may select the corresponding position in the converted image data 142S for image patch A1 of one of the converted image data 141S based on coordinate information 122. Details of the method for generating a map of corresponding positions using coordinate information 122 will be described later.

[0062] The operator sets labels determined based on the evaluation for each image patch A1 and A2 displayed in the converted image data 141S and 142S. Here, when setting labels for the mixing state of waste 26a, which is a mixture of multiple objects stored in the storage pit 26, for example, high mixing area, medium mixing area, and low mixing area, as well as wall area and bagged waste area, are each set in advance as predetermined labels that can distinguish each other.

[0063] Specifically, for example, an operator uses the input / output unit 14 to input information to set a predetermined label for image patch A1 of the converted image data 141S. In this case, the annotation processing unit 112 of the control unit 11 sets the predetermined label for image patch A1 of the converted image data 141S displayed on the input / output unit 14, based on the input information. As a result, annotation processing is performed on the image of waste 26a in the converted image data 141S shown in Figure 11B, and a predetermined label is set for each location of waste 26a, i.e., for each selected image patch A1. The image processing unit 111 may also overlay the input and set labels onto image patch A1 of the converted image data 141S.

[0064] (Method for generating correspondence maps) Also, in step ST6, a correspondence relationship between a plurality of converted image data is derived. Here, a method for generating a correspondence map regarding the position correspondence will be described. As shown in FIGS. 11A and 11B, an image processing unit 111 in a control unit 11 of the information processing apparatus 10 stores a plurality of captured image data captured by a plurality of imaging cameras 231 and 232 in a storage unit 12 as image information 121 (storage step). The image processing unit 111 generates converted image data 141S and 142S from the captured image data 141 and 142.

[0065] Next, an operator evaluates the mixing degree at an arbitrary portion of the waste 26a in the storage pit 26 with respect to the converted image data 141S and 142S displayed on the display screen of the input / output unit 14. For example, the operator selects one of the converted image data 141S and 142S using the input / output unit 14. Subsequently, the operator selects an image patch A1 for setting a label in the annotation process with respect to the captured image data 141. After the image processing unit 111 sets the selected image patch A1 with respect to the captured image data 141, it stores the image patch A1 in the storage unit 12 as image information 121 (attention area setting step).

[0066] Subsequently, an annotation processing unit 112 reads out the coordinate range of the image patch A1 from the storage unit 12 as coordinate information 122. Here, as the coordinate range of the image patch A1 in the coordinate information 122, it is also possible to read out by designating a predetermined vertex of the image patch A1 as "A1(x 11 ,y 11 )", or as the rectangular range of the image patch A1 as "A1(x 11 ~x 12 ,y 11 ~y 12 )", or as "A1(x 11 ,y 11 )~(x 12 ,y 12 )". In the present embodiment, for example, a predetermined vertex of the attention area is designated as A1(x 11 ,y 11 ).

[0067] The operator uses the input / output unit 14 to input image patch A1(x 11 ,y 11 The degree of mixing is evaluated and a label is set based on that evaluation, i.e., labeling is performed. The label set for image patch A1 is associated with the coordinate information 122 of image patch A1 and stored in the storage unit 12 as annotation information 123 (labeling step).

[0068] Furthermore, as shown in Figure 11B, the operator selects image patch A2 corresponding to image patch A1 in converted image data 141S for the other converted image data 142S that was not selected from converted image data 141S and 142S. If image patch A1 is selected for one of the converted image data 141S, an image patch A2 often exists in the other converted image data 141S.

[0069] Therefore, the operator uses the input / output unit 14 to select and set the area in the converted image data 142S that corresponds to image patch A1 in one of the converted image data 141S as image patch A2. The image patch A2 selected in the converted image data 142S will indicate approximately the same position and area within the storage pit 26 as image patch A1 in the captured image data 141. The image processing unit 111 sets the coordinates (x 21 ,y 21 ) to image patch A1(x 11 ,y 11 ) is associated with and stored in the storage unit 12 as coordinate information 122 (corresponding area setting step).

[0070] Furthermore, the annotation processing unit 112 sets the same label for image patch A2 as the label set for image patch A1. This makes it possible to perform annotation processing on a portion of image patch A1 of one of the converted image data 141S on the other converted image data 142S. Conversely, image patch A2(x 21 ,y 21 By selecting ), image patch A1(x 11,y 11 It also becomes possible to extract ). Here, since the pair of converted image data 141S and 142S are both displayed as an overhead view from above the storage pit 26, the state of the waste 26a can be evaluated with higher accuracy. Therefore, the accuracy of the annotation process can be improved.

[0071] Furthermore, various methods other than those described above can be used to correspond image patch A1 and image patch A2. That is, it is also possible to store the coordinates (x,y) of the top view of the storage pit 26 as coordinate information 122 in the storage unit 12 in advance. In this case, the image processing unit 111 can map any position in the converted image data 141S, 142S after image conversion processing, based on the coordinate information 122 of the installation positions of the imaging cameras 231, 232, for example, the coordinate information 122 of two of the four corners of the storage pit 26, to the coordinates (x,y) of the top view of the storage pit 26. As a result, the image processing unit 111 can map image patch A1(x 11 ,y 11 ) is associated with the coordinates (x1, y1) in the storage pit 26, while the corresponding coordinates (x1, y1) in the storage pit 26 are associated with image patch A2(x 21 ,y 21 ) can be made to correspond to ). Therefore, the image processing unit 111 can determine image patch A2 from image patch A1 via the coordinates in the storage pit 26, and can derive the correspondence between image patch A1 and image patch A2 by matching. As a result, the image processing unit 111 can generate a correspondence map (correspondence map generation step). However, it is possible that a discrepancy may occur between image patch A1 and image patch A2 due to the matching. In this case, the operator can correct the position of image patch A2 by moving it to a position corresponding to the position of image patch A1 from the input / output unit 14, thereby increasing the accuracy of the corresponding position information between image patch A1 and image patch A2 and generating a correspondence map more accurately. Therefore, annotation processing can be performed with even greater accuracy.

[0072] Furthermore, the surface layer of the waste 26a in the storage pit 26 can be measured using a distance measuring sensor (not shown), such as 3D-LiDAR (3 Dimension Laser Imaging Detection and Ranging). In this case, by associating data such as point cloud data obtained by the distance measuring sensor with the positional information of the waste 26a in the storage pit 26, it is possible to associate image patch A1 and image patch A2 to derive a correspondence and generate a correspondence map. Moreover, although the above-mentioned image patches A1 and A2 are, for example, rectangular or square regions, regions of other shapes can be set as appropriate. Specifically, for example, in this case, it is preferable that the shapes of image patch A1 and image patch A2 are similar to each other.

[0073] (A learning model that evaluates the degree of mixture) Next, we will describe a machine learning method using the converted image data 141S and 142S generated and annotated by steps ST2 to ST4. Specifically, the learning unit 113 of the control unit 11 of the information processing device 10 shown in Figure 2 performs machine learning using annotation information 123, which includes the converted image data 141S and 142S labeled as described above, as training data.

[0074] The learning unit 113, as a learning means, generates a learning model for evaluating the degree of mixing by using machine learning such as supervised learning with an input / output dataset, which is obtained by using transformed image data 141S and 142S, which have been previously acquired from the image information 121 of the memory unit 12 and subjected to geometric transformation processing, as learning input parameters, and transformed image data 141S and 142S, on which the degree of mixing of waste 26a has been labeled by annotation processing from the annotation information 123, as learning output parameters. Various machine learning methods can be employed, such as deep learning using neural networks, and can be carried out by mini-batch learning, etc. This concludes the information processing according to this embodiment.

[0075] By performing annotation processing using multiple image data 141 and 142 from multiple storage pits 26, it becomes possible to assign highly accurate labels, thereby improving the accuracy of machine learning.

[0076] According to the embodiment described above, by performing geometric transformation processing on the image data 141 and 142 of multiple storage pits 26, it becomes possible to perform labeling by workers with high accuracy, thereby improving the accuracy of machine learning for generating a learning model. Furthermore, since the transformed image data 141S and 142S can be image data that provides an overhead view of the storage pits 26, a correspondence map that allows coordinate transformation between multiple transformed image data 141S and 142S can be easily generated. As a result, in the evaluation and determination of the mixing degree of actual waste 26a, geometric transformation processing is applied to the image data 141 and 142 to obtain transformed image data 141S and 142S, and these transformed image data 141S and 142S are input to the trained model as input parameters, and the evaluation value of the mixing degree is output as an output parameter, thereby enabling more accurate determination of the mixing degree of waste 26a in the storage pits 26. Furthermore, based on the corresponding map, the labeling for some of the converted image data 141S can be automatically applied to other converted image data 142S, thereby reducing the workload of annotation processing for multiple pairs of captured image data.

[0077] Although one embodiment of the present invention has been described in detail above, the present invention is not limited to the above-described embodiment, and various modifications are possible based on the technical idea of ​​the present invention. For example, the numerical values ​​given in the above-described embodiment are merely examples, and different numerical values ​​may be used as needed, and the present invention is not limited by the description and drawings that constitute part of the disclosure of the present invention in this embodiment.

[0078] For example, in the embodiment described above, the waste 26a stored in the storage pit 26 is used as the target object for the image data 141 and 142. However, image data of the number of people gathered at a designated venue or the degree of congestion at the venue may also be used. In this case, the learning model can generate input and output datasets for generating a learning model that evaluates the degree of congestion.

[0079] Furthermore, in one embodiment, the terms "part" as described above can be replaced with "circuit" or the like. For example, the control unit can be replaced with a control circuit.

[0080] Further effects and modifications can be readily derived by those skilled in the art. Broader aspects of this disclosure are not limited to the specific details and representative embodiments expressed and described above. Therefore, various modifications are possible without departing from the spirit or scope of the overall concept of the invention as defined by the appended claims and their equivalents. [Explanation of symbols]

[0081] 1. Annotation Data Generation System 2 Network 3. Waste treatment facilities 10 Information Processing Devices 11,21 Control Unit 12 Storage section 13,22 Communications Department 14 Input / output section 20 Waste storage facilities 23 Imaging Unit 25 Gripping part 26 Storage pit 26a Waste 26b Inner wall 30 Waste Incineration Facilities 31 Combustion control device 32 Sensor section 33 Incinerator Image data 100, 141, 142 100a bottom surface 100b,100c moving line 100d parallel plane 101a Object area 101b Non-extracted area 101d projection surface 101h Selected projection plane 111 Image Processing Unit 112 Annotation Processing Unit 113 Learning Department 121 Image Information 122 Coordinate Information 123 Annotation Information 124 Region Extraction Models 141S, 142S converted image data 231,232 Imaging cameras 251 Cranes 252 buckets

Claims

1. An information processing method performed by a control unit equipped with hardware, A storage step in which an object stored in a target facility is imaged from different directions using multiple imaging devices, and multiple image data obtained from each image are stored in a storage unit, An identification step of identifying, with respect to at least one image data read from the storage unit, an object region in which the object is imaged and a non-extracted region in which the object is not imaged, A setting step of setting a planar projection plane that includes the object in the captured image data and is movable in parallel along a predetermined movement line, A selection step involves deriving an evaluation index set based on the number of pixels in the identified object region for the projection surface, selecting a projection surface that satisfies predetermined conditions, and setting it as the selected projection surface. The process includes a transformation step of generating transformed image data in which the object is projected onto the selected projection plane by performing a geometric transformation process based on the selected projection plane, The worker performs annotation processing on the converted image data. Information processing methods.

2. The object is stored in a predetermined space whose bottom surface is flat. The projection plane is set to be parallel to the bottom plane or perpendicular to gravity. The information processing method according to claim 1.

3. The aforementioned evaluation index is derived based on the following equation when λ is a weighting coefficient greater than 0. Evaluation index = (Number of pixels in the object area within the projection plane - Number of pixels in the object area outside the projection plane × λ) / Total number of pixels on the projection plane The information processing method according to claim 1.

4. The captured image data is associated with time information relating to the time when the object was captured by the imaging means. The information processing method according to claim 1.

5. The selected projection plane is set for some of the multiple captured image data, whose capture time is the same in the time information read from the storage unit, and the selected projection plane is applied to the remaining captured image data. The information processing method according to claim 4.

6. For multiple image data files read from the storage unit, where the time of capture is the same in the time information, the selected projection plane is set independently for each image data file. The information processing method according to claim 4.

7. The aforementioned facility is a storage pit in a waste storage facility, and the aforementioned object is the waste stored in the storage pit. The information processing method according to claim 1.

8. The system comprises a storage unit for storing image data obtained by imaging an object, and a control unit for performing image processing on the image data. The control unit, Multiple imaging devices are used to image objects stored in the target facility from different directions, and the multiple image data obtained from each image are stored in the storage unit. With respect to at least one image data read from the storage unit, the system identifies the object region in which the object is imaged and the non-extracted region in which the object is not imaged. A planar projection plane is set that includes the object in the captured image data and is movable in parallel along a predetermined line of motion, An evaluation index is derived for the projection surface based on the number of pixels in the identified object region, and the projection surface that satisfies the predetermined conditions is selected and set as the selected projection surface. By performing geometric transformation processing based on the selected projection plane, transformed image data of the object projected onto the selected projection plane is generated and stored in the storage unit. The converted image data is then annotated by the operator. Information processing device.

9. A control unit equipped with hardware, A storage step in which an object stored in a target facility is imaged from different directions using multiple imaging devices, and multiple image data obtained from each image are stored in a storage unit, An identification step of identifying, with respect to at least one image data read from the storage unit, an object region in which the object is imaged and a non-extracted region in which the object is not imaged, A setting step of setting a planar projection plane that includes the object in the captured image data and is movable in parallel along a predetermined movement line, A selection step involves deriving an evaluation index set based on the number of pixels in the identified object region for the projection surface, selecting a projection surface that satisfies predetermined conditions, and setting it as the selected projection surface. A transformation step in which a geometric transformation process is performed based on the selected projection plane to generate transformed image data in which the object is projected onto the selected projection plane, A program that executes, The converted image data is then annotated by the operator. program.

Images