Task execution system, task execution method, and task execution program

Abstract

To provide a technique of executing a task related to a target object robustly against a change in a visual feature.SOLUTION: A task execution system includes: a database management unit configured to record in advance a contextual relationship database indicating a spatial contextual relationship of a plurality of objects including a target object; an imaging unit configured to acquire image data that is data obtained by adding, to an image, position information indicating a position at which the image is captured; an object detection unit configured to detect objects from the image; a segmentation unit configured to separate the objects from the image; a contextual relationship extraction unit configured to extract a spatial contextual relationship of the objects separated from the image; and a task execution unit configured to execute a task related to the target object on the basis of the contextual relationship of the objects separated from the image and the contextual relationship recorded in the contextual relationship database.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a technique for acquiring images to be used in visual inspection, which uses images to inspect the state of an inspection target. [Background technology]

[0002] Due to labor shortages and a decrease in skilled workers, it is recommended to introduce an appearance inspection system using a mobile object that autonomously acquires information to ensure the safety of various facilities. In an appearance inspection system, a mobile object equipped with various sensors moves around the facility to acquire various sensor information, and in certain cases, further photographs the inspection target to acquire an image. Examples of sensors include RGB cameras, infrared cameras, and audio sensors. Examples of mobile objects include mobile vehicles, robo-dogs, and crawler robots.

[0003] A mobile object autonomously travels to an inspection location periodically, acquires images including the object to be inspected, and sends them to an appearance inspection system, which then inspects the condition of the object based on the images. The mobile object travels to an inspection location based on location information from a Global Positioning System (GPS), for example. However, because GPS location information contains an error of 30 cm to 5 meters, the mobile object does not necessarily reach the same location every time, making it difficult to accurately photograph the object to be inspected. Also, GPS may not function well indoors.

[0004] Visual SLAM technology is sometimes used to navigate moving objects in facilities where GPS does not function. However, Visual SLAM technology has a position error of 20-30 cm, and the computation load required to match visual features with high accuracy is high, so it may not be possible to guide a moving object to an inspection target location within the desired inspection cycle.

[0005] Because of these problems, the current common practice is to prepare a template image of the object to be inspected in advance, and then adjust the position of the mobile vehicle and its camera at the inspection site based on GPS location information so that the image it takes matches the visual features of the template image, and then photograph the object to be inspected.

[0006] However, the visual features of the image may change significantly if the object to be inspected is damaged, lost, or obscured by external objects (leaves, snow, dust, birds, flying objects, etc.), if the background of the object to be inspected changes with the seasons (grass, bushes, dynamic objects, environmental changes, etc.), or if a natural disaster occurs (typhoon, flood, fire, earthquake, etc.). If the visual features of the image change significantly, matching by visual features becomes difficult, and not only may it be impossible to photograph the object to be inspected, but it may also be impossible to detect that serious damage has occurred to the object to be inspected.

[0007] Patent Document 1 discloses a technology for recognizing an object from an image, acquiring semantic information such as characters from the object, and using the information to correctly guide a mobile intelligent device such as a service robot.

[0008] Non-Patent Document 1 discloses a method for locating an unmanned flying device by performing scene matching based on semantic information extracted from images captured in real time and images stored in advance. Scene matching is performed using semantic information, object positions, and the topological structure of object arrangement as elements. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] China Patent Publication CN107967473B [Non-patent literature]

[0010] [Non-Patent Document 1] Wang M, Wang H, Wang Z, Li Y, “A UAV Visual Relocalization Method Using Semantic Object Features Based on Internet of Things”, Wireless Communications and Mobile Computing. 2022 Feb 11;2022. Summary of the Invention [Problem to be solved by the invention]

[0011] As described above, an appearance inspection system using a mobile object that autonomously collects information is required to function robustly even in a situation where visual characteristics change significantly, such as when an inspection target or other object is lost. However, the method disclosed in Patent Document 1 does not consider how to identify the position of a mobile intelligent device when an environmental change occurs in the guidance path of the mobile intelligent device. In addition, the method disclosed in Non-Patent Document 1 does not assume the detection of the loss or damage of a specific object.

[0012] One objective of the present disclosure is to provide a technique for performing object-related tasks in a manner that is robust to changes in visual features. [Means for solving the problem]

[0013] A task execution system according to one aspect included in the present disclosure includes a database management unit that pre-records a contextual relationship database indicating spatial contextual relationships of multiple objects including a target object, an imaging processing unit that acquires image data, which is data in which an image is added with location information indicating the location at which the image was taken, an object detection unit that detects an object from the image, a segmentation unit that segments the object from the image, a contextual relationship extraction unit that extracts spatial contextual relationships of the object segmented from the image, and a task execution unit that executes a task related to the target object based on the contextual relationships of the object segmented from the image and the contextual relationships recorded in the contextual relationship database. Effect of the Invention

[0014] According to one aspect of the present disclosure, a task related to an object is performed based on the spatial contextual relationships of the objects, thereby enabling processing that is robust to changes in visual features. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a block diagram of a visual inspection system according to a first embodiment. [Diagram 2] 13 is a flowchart of a context relation database generation process. [Diagram 3] 13 is a flowchart of a process for calculating a physical distance between objects. [Figure 4] FIG. 13 is a diagram for explaining a context relation database. [Diagram 5] 1 is a table illustrating an example of a context relation database. [Figure 6] 1 is a table showing an example of a list of objects to be inspected in a context relation database. [Figure 7] 1 is a table illustrating an example of object relations in a context relation database. [Figure 8] 11 is a flowchart of a first inspection object imaging process. [Figure 9] 13 is a flowchart of a second inspection object imaging process. [Figure 10] FIG. 11 is a block diagram of a visual inspection system according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, an embodiment will be described with reference to the drawings.

[0017] In the present embodiment, an example is shown in which the input data is an image, but the input data is not limited to image data, and may be other media data such as video data.

[0018] In addition, in this embodiment, an appearance inspection system is exemplified that travels around a specified area at a specified inspection cycle to acquire image data and performs an appearance inspection of an inspection target based on the image, but the target task is not limited to appearance inspection. The technology included in the present disclosure can be widely used in systems that perform other tasks, such as generating reports to notify of damage to targets, vehicle navigation, guiding robots that move within facilities, and guiding robots that move in open areas such as cities. EXAMPLES

[0019] A first embodiment will be described with reference to FIGS.

[0020] FIG. 1 is a block diagram of a visual inspection system according to the first embodiment.

[0021] The appearance inspection system 100 is a system that autonomously moves and captures an image at a desired position. The appearance inspection system 100 has an object detection unit 200, a segmentation unit 300, a contextual relationship extraction unit 400, a database management unit 500, a physical distance calculation unit 600, an image capture processing unit 700, a processor 80, a memory 70, an image capture device 90, a GPS 60, and a self-propelled device M.

[0022] The memory 70 is a storage device that stores software programs and data, and includes a RAM (Random Access Memory) and a hard drive.

[0023] The processor 80 is an arithmetic device that executes processing of software programs, and is, for example, a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or both.

[0024] The imaging device 90 is a camera capable of adjusting the angle of view and focal length. The imaging device 90 captures an image at a desired angle of view and focal length. Image data is obtained by adding position information indicating the position at which the image was captured to the image. As an example, the imaging device 90 may be a PTZ camera capable of panning, tilting, and zooming and capturing an image at a wide angle of view.

[0025] The GPS 60 is a GPS receiver that receives position information from the Global Positioning System.

[0026] Position information acquired by the GPS 60 is added to an image captured by the imaging device 90. The position information is used for processing the software program. The object detection unit 200, the segmentation unit 300, the contextual relationship extraction unit 400, the database management unit 500, the physical distance calculation unit 600, and the imaging processing unit 700 are processing units realized by the processor 80 executing the software program.

[0027] The mobile device M is a device that moves the imaging device 90 and the GPS 60 to a desired position. The movement includes various means such as running on land, flying in the air, sailing on water, and diving underwater. In this embodiment, the mobile device is a vehicle device that runs on land. In addition to position information acquired by the GPS 60, the mobile device M may use map data including landmarks to run and move close to the inspection object. Since the position and angle reached by the mobile device M may vary depending on the inspection round due to an error in the GPS 60, the appearance inspection system 100 adjusts the deviation to perform imaging.

[0028] The appearance inspection system 100 is, for example, a system configured to include a computer that performs appearance inspection of an object by a processor 80 executing processing of a software program using a memory 70. The appearance inspection system 100 can also be configured separately into a server device configured with a computer including the processor 80 and the memory 70, and a mobile imaging device including an imaging device 90, a GPS 60, and a mobile device M.

[0029] The database management unit 500 records the physical configuration database 110, which indicates the physical configuration including the physical dimensions of a plurality of objects including the target object, and the contextual relationship database 520, which indicates the spatial contextual relationship of a plurality of objects including the target object. The physical dimensions of the objects, such as the height and width, registered in the physical configuration database 110 can be obtained from the 3D model of the object. If the object is some kind of equipment, the physical dimensions can be known from the equipment specifications disclosed by the manufacturer of the equipment. If the object is a facility with a certain size, the physical dimensions can be known from the design document when the facility was installed or the measurement results of the facility. In the contextual relationship database 520, the contextual relationships between objects in a normal state are registered in advance before starting the appearance inspection cycle. As a result, even if some equipment is damaged due to a natural disaster and the appearance changes, it is possible to estimate or confirm the damaged item from the relative position with respect to the undamaged object. The process of generating the contextual relationship database 520 will be described later.

[0030] The object detection unit 200 detects an object from an image of input data acquired by the image capture processing unit 700 using the image capture device 90. The object detection may be performed by a method using a machine learning model based on deep learning, such as SSD (Single Shot MultiBox Detector), YOLOv4, or YOLOX. The object detected here may include an object to be inspected.

[0031] The physical distance calculation unit 600 calculates the physical distance between objects detected from the input data image.

[0032] The segmentation unit 300 segments the object detected by the object detection unit 200 from the image and identifies the outer edge of the object. The segmentation of the object from the image may be performed using a method using a machine learning model based on deep learning, such as Mask R-CNN or BiSNET.

[0033] The contextual relation extraction unit 400 extracts spatial contextual relations between objects segmented from an image by the segmentation unit 300. In this case, the contextual relation extraction unit 400 may use the physical distance between the objects to extract the spatial contextual relations between the objects, and extract the contextual relations including the physical distance between the objects. In this case, information on the object relations including the physical distance between the objects is also recorded in the contextual relation database 520.

[0034] The imaging processing unit 700 captures an image of an object to be inspected by the imaging device 90 from a position and direction specified based on the contextual relations of the object classified from the image extracted by the contextual relation extraction unit 400 and the contextual relations of the object recorded in the contextual relation database 520 recorded in advance by the database management unit 500, obtains an image for appearance inspection, and estimates whether the object is normal or abnormal based on the image for appearance inspection. Obtaining the image for appearance inspection and estimating the state of the object to be inspected is a task to be performed by the appearance inspection system 100 in this embodiment. The imaging processing unit 700 can be said to be a task execution unit that performs appearance inspection of the object.

[0035] 2 is a flowchart of the context relation database generation process, which is a process for creating and recording a context relation database in advance.

[0036] In step 91, the image capturing processing unit 700 acquires image data of a 360-degree image at a specific position identified by position information from the GPS 60 by stitching together a plurality of images captured from the specific position. The image data stores position information of the specific position in addition to the data of the image itself. Note that, although an example of acquiring a 360-degree image is shown here, the present invention is not limited to this. In order to extract as much contextual information as possible, it is preferable that the image acquired here is an image with a wide angle of view that includes objects including the target object.

[0037] Next, in step 210, the object detection unit 200 detects objects from the image data, and the segmentation unit 300 segments the detected objects from the image and identifies the object's outer edges.

[0038] Next, in step 410, the contextual relationship extraction unit 400 uses the physical distance calculation unit 600 to determine the relative positional relationship, including the physical distance, between the objects based on the physical configurations registered in advance in the physical configuration database 110 and the objects detected and classified from the image.

[0039] 3 is a flowchart of the inter-object physical distance calculation process, which is a process for calculating the physical distance between two objects, and is included in the process of step 410 described above.

[0040] In step 413, the physical distance calculation unit 600 encloses, with a bounding box, an object that is detected from the image and whose physical configuration is registered in the physical configuration database 110. The bounding box is a rectangle that encloses the object.

[0041] In step 414, the physical distance calculation unit 600 performs segmentation to separate the object enclosed by the bounding box from the image and find the exact outer edge of the object.

[0042] In step 415, the physical distance calculation unit 600 calculates the number of pixels that make up the exact width of the outer edge of the object, and calculates the actual physical distance per pixel in the image based on that number of pixels and the width of the object registered in the physical configuration database 110.

[0043] In step 416, the physical distance calculation unit 600 counts the number of pixels between the four corners and the center of the bounding box between the two objects, calculates the average of these five pixel counts, and multiplies the average by the distance per pixel to calculate the physical distance between the two objects.

[0044] 2, next, in step 510, the database management unit 500 identifies spatial contextual relationships between a plurality of objects based on the physical distances and relative positional relationships between the objects, and creates a contextual relationship database indicating the contextual relationships between the objects. The contextual relationships may include information on the physical distances.

[0045] Contextual relationships between objects are expressed, for example, by an overlapping relationship where objects overlap, an adjacent relationship where objects are adjacent to each other, a spatial relationship where objects exist across a space, a dependency relationship where one object depends on another object, a containment relationship where one object is contained in another object, etc. Contextual relationships between objects are expressed by a relationship where one object is above another object, a relationship where one object is below another object, a relationship where one object is to the right of another object, a relationship where one object is to the left of another object, a relationship where one object is diagonally opposite another object, a relationship where one object is inside another object, a relationship where one object supports another object, etc.

[0046] The contextual relationship database 520 may further store map data showing the layout of the objects to be inspected and the facility, information related to the layout of the equipment in the facility, etc. Such information is useful for understanding the positional contextual relationships and functional contextual relationships between objects including the facility and the equipment.

[0047] FIG. 4 is a diagram for explaining the context relation database.

[0048] 4 shows an imaging device 90 located at a specific position and a number of objects (E-1 to E-9) located around the specific position. A name is given to each object as identification information for identifying the object. The name may be determined using the relative positions between the objects and the visual features of the objects. The imaging device 90 may be a self-propelled device capable of determining its own position and traveling by SLAM (Simultaneous Localization and Mapping).

[0049] 5 is a table showing an example of the context relation database 520. In a table 521 showing an example of the context relation database 520, information on each item, such as a primary key, an object, essential, non-essential, and a relationship, is recorded in association with each other.

[0050] In the "primary key" field, location information (GPS location) of the specific location where the image is captured is registered as unique information.

[0051] The "Object" field lists the names of multiple objects of interest. The objects may include an object to be inspected.

[0052] The "required" item lists the names of objects that must be captured in the image (required objects). The required objects may include inspection targets such as factory equipment.

[0053] The "non-essential" item lists the names of objects other than the essential objects (non-essential objects). Non-essential objects may include objects surrounding the inspection target, such as trees.

[0054] In the "Relationship" field, information indicating the contextual relationship between objects is registered. For example, it is registered that objects E2 and E8, and objects E3 and E7 are diagonally opposite each other. It is also registered that objects E1, E2, E3, E4, and E5 are on the opposite side of the road from objects E6, E7, E8, and E9.

[0055] In this embodiment, an example of extracting the contextual relationship from a 360-degree image is shown, but other methods may be used. For example, the contextual relationship may be extracted from a bird's-eye view image or an aerial photograph image. In addition, in the case of a substation, for example, the contextual relationship may be extracted from a facility layout such as an SLD (single-line diagram).

[0056] 6 is a table showing an example of an inspection object list in the context relation database. In the inspection object list, various information related to each inspection object is recorded. In table 522 showing an example of the inspection object list, information on each item, such as GPS position, inspection object, object, and object relation, is recorded in association with each other. In this example, additional information is further added.

[0057] In the "GPS location" field, the location information (GPS location) of the inspection object is recorded.

[0058] The name and / or identification information of the object to be inspected is registered in the "object to be inspected" field.

[0059] In the "Object" field, information on the type of object that is the object to be inspected is registered.

[0060] In the "Object Relation" field, information on the physical positional relationship between the inspection target object and other objects is registered. The object relations may be recorded in the form of a table within a table. Detailed examples of the object relations will be described later.

[0061] The additional information includes information about the object being inspected, such as information indicating the behavior of the object being inspected, predicted change information indicating how the object being inspected is expected to change over time, and information about the risk of danger that arises as the object being inspected changes over time.

[0062] 7 is a table showing an example of object relations in the context relation database. In table 523, information on each item, such as object label, GPS position, relation information, XY coordinates, required, and inspection object, is recorded in association with each other.

[0063] In the "object label" field, identification information of the object is recorded.

[0064] In the "GPS position" field, information expressing the position of the object using position information obtained by the GPS is recorded.

[0065] The "Relationship Information" field records information indicating the relative position of each other object with respect to the object in question. L indicates an object on the left side of the object in question. L indicates an object on the left side of the object in question. Up indicates an object above the object in question. Dw indicates an object below the object in question. D indicates a physical distance. Dia indicates an object diagonally opposite the object in question. For example, object L-1 is located at a distance D-1 from the object in question on the left side of the object in question. Object L-2 is located at a distance D-2 from the object in question on the left side of the object in question. Object L-3 is located at a distance D-3 from the object in question on the left side of the object in question. Object R-1 is located at a distance D-1 from the object in question on the right side of the object in question. Object R-2 is located at a distance D-2 from the object in question on the right side of the object in question. Object U-1 is located at a distance D-1 from the object in question on the top side of the object in question. Object Dw-1 is located at a distance D-1 from the object in question on the bottom side of the object in question. An object Dia-1 is diagonally opposite the object in question at a distance D-1 from the object in question.

[0066] The "XY coordinates" field records information that indicates the position of the object using XY coordinates with a specific device corner as the origin.

[0067] In the "essential" item, information indicating whether the object is a mandatory object that must be captured in an image is recorded.

[0068] In the "inspection object" field, information indicating whether the object is an inspection object or not is recorded.

[0069] 8 is a flowchart of the first inspection object imaging process. The inspection object imaging process is a process for acquiring an image of the inspection object to be used in the inspection (an inspection image).

[0070] At step 92, the image processor 700 captures an image at a particular GPS location.

[0071] In step 211, the object detection unit 200 detects an object from the image, and the segmentation unit 300 segments the detected object from the image and identifies the outer edge of the object.

[0072] Next, in step 411, the contextual relationship extraction unit 400 uses the physical distance calculation unit 600 to identify the contextual relationship and physical distance between the objects based on the physical configurations registered in the physical configuration database 110 and the objects detected and classified from the image.

[0073] In step 530, the database manager 500 sequentially selects from the contextual relation database 520 information of the inspection object at a particular GPS location.

[0074] In step 531, the database management unit 500 determines whether or not information necessary for capturing an inspection image of the inspection object has been obtained based on the contextual relationship information on the inspection object at the specific GPS position selected from the contextual relationship database 520, the physical configuration of the inspection object registered in the physical configuration database 110, and the contextual relationship and physical distance between the objects identified in step 411. The inspection image is an image for inspecting the appearance of the inspection object, and is required to be an image of the inspection object captured accurately from a fixed position.

[0075] At this time, if the relationship between the current position of the imaging device 90 and the position of the object to be inspected can be identified based on the contextual relationship information regarding the object to be inspected at a specific GPS position selected from the contextual relationship database 520, the physical configuration of the object to be inspected registered in the physical configuration database 110, and the contextual relationship and physical distance between the objects identified in step 411, it is determined that the information necessary to capture the inspection image has been obtained.

[0076] If the information necessary to capture an inspection image has not been obtained, the database management unit 500 returns to step 92 .

[0077] If the information required to capture an inspection image has been obtained, then in step 710, the image capture processing unit 700 adjusts various parameters including the position of the image capture device 90 in order to capture a good image of the object to be inspected.

[0078] In step 720, the imaging processing unit 700 captures an inspection image of the inspection object by the imaging device 90. Then, if there are multiple inspection objects to be imaged at the GPS position, the next inspection object is selected and the process is repeated from the beginning.

[0079] FIG. 9 is a flowchart of the second inspection object imaging process.

[0080] In step 93, the image capture device 90 captures an image with a wide angle of view at a particular GPS location. By capturing an image with a wide angle of view, it becomes easier to detect many objects that appear in the contextual relation database 520 from the image and obtain information required to capture an inspection image from the contextual relation database.

[0081] In step 212, the object detection unit 200 detects objects in the image, and the segmentation unit 300 segments the detected objects in the image and identifies the outer edges of the objects. If multiple objects are detected, segmentation is performed on all the objects individually.

[0082] In step 731, the image capturing processing unit 700 selects one of the items detected from the image captured at a specific GPS position, and in accordance with a predetermined priority order, refers to the object-related table 523 included in the table 522 of the inspection target object list, and detects the object label in the table 523. Here, the priority order of the object label may be registered in advance as a priority order database 540 in the appearance inspection system 100.

[0083] In step 732, the imaging processing unit 700 extracts relational information related to the object to be inspected from table 523 of the contextual relational database 520 based on the object label detected in step 731, and estimates the relationship between the current position of the imaging device 90 and the position of the object to be inspected based on information on the relative positions of the objects appearing in the extracted relational information with respect to the object to be inspected.

[0084] In step 733, the imaging processor 700 adjusts the position and focus of the imaging device 90 to the object under inspection or to a device containing the object under inspection based on the estimated positional relationship.

[0085] In step 734, the image capturing and processing unit 700 captures an image of the inspection target and its surroundings or the entire device including the inspection target, and detects the object from the captured image using the object detection unit 200.

[0086] In step 735, the image capture processing unit 700 confirms the relative position between the detected object label and the object detected from the image in step 734 based on the object relationship in table 523 between the detected object label and the object detected from the image.

[0087] In step 736, the imaging processor 700 adjusts the focus and zoom of the imaging device 90 based on the coordinate position of the most recently captured inspection image, and captures an inspection image of the object under inspection.

[0088] As described above, according to the first embodiment, by detecting the objects under inspection and understanding their surrounding objects and their contextual relationships with the objects under inspection, the capture and photographing of the objects under inspection is faster and more robust to changes in visual features that may occur due to weather, environmental changes, natural disasters, accidents, etc., when compared to vision-based feature matching approaches.

[0089] Furthermore, in the visual inspection system 100 of the first embodiment, the database management unit 500 may record in advance object additional information indicating the behavior and / or properties of the inspection object, as exemplified in the table 522, and may chronologically accumulate image data of the inspection image together with information on the date and time when the image data was captured as historical image data. In this case, the imaging processing unit 700 may capture an image at a target point identified based on the contextual relationship of the object classified from an image captured with a wide angle of view and the contextual relationship recorded in the contextual relationship database 520, and may estimate the state of the inspection object based on the acquired target point image, historical image data, and object additional information.

[0090] In the first embodiment, the database management unit 500 may accumulate image data of the inspection images in chronological order as historical image data together with information on the date and time when the image data was captured, and the image capture processing unit 700 may predict the future visual appearance or lifespan of the inspection object based on the change trend of the image of the inspection object in the historical image data. The change trend of the image of the inspection object in the historical image data means how the inspection object has changed over time.

[0091] In addition, in the first embodiment, the database management unit 500 may pre-record connection relationship data representing the arrangement and connection relationships of objects including the object to be inspected, and the object detection unit 200 may detect an object from an image based on the image, the contextual relationship, and the connection relationships.

[0092] In the first embodiment, the visual inspection system 100 may receive operations relating to the execution of visual inspection from an operator on a user interface accessible via a communication network, and may display the results of the visual inspection. EXAMPLES

[0093] FIG. 10 is a block diagram of a visual inspection system according to the second embodiment.

[0094] In the second embodiment, an appearance inspection system 2000 is illustrated, which is expanded by adding an in-facility map 40, a depth camera 50, an inspection report creation unit 800, an automatic scene understanding unit 900, and a functional relationship management unit 1000 to the appearance inspection system 100 of the first embodiment. The appearance inspection system 2000 of the second embodiment has the same configuration as the appearance inspection system 100 of the first embodiment, except for the above-mentioned expanded parts.

[0095] The facility map 40 is map data showing the layout of objects within a facility. The map data may be a single-line map or a three-dimensional (3D) map.

[0096] The depth camera 50 is a camera capable of measuring distance. The depth camera 50 is disposed, for example, near the imaging device 90, and can measure the physical distance from the imaging device 90 to an object imaged by the imaging device 90.

[0097] For example, in step 413, when the physical distance calculation unit 600 identifies an object detected from an image captured by the imaging device 90 and whose physical configuration is registered in the physical configuration database 110, the physical distance calculation unit 600 may use the physical distance from the imaging device 90 to the object in order to identify an object that matches the object in the image captured by the imaging device 90 from among the objects registered in the physical configuration database 110.

[0098] Also, for example, in step 411, when the contextual relationship extraction unit 400 determines the contextual relationship and physical distance between objects based on the physical configuration registered in the physical configuration database 110 and the objects detected and classified from the image, the contextual relationship extraction unit 400 may use the physical distance from the imaging device 90 to the objects in order to determine an object that matches the object in the image from among the objects registered in the physical configuration database 110.

[0099] Furthermore, for example, when creating the contextual relation database 520 in step 510, the contextual relation extraction unit 400 may create and record a 3D facility map 40 that represents the relative positional relations between objects.

[0100] Also, for example, the imaging processing unit 700 may create a 3D map at the time the images captured in step 92 and step 720 were captured, based on the physical distance from the imaging device 90 to the object obtained at the same time as capturing those images.

[0101] Also, for example, the imaging processing unit 700 may create a 3D map at the time the images captured in step 93 and step 736 were acquired based on the physical distance from the imaging device 90 to the object obtained at the same time as capturing those images.

[0102] The inspection report creation unit 800 analyzes the inspection image captured in step 720 or step 736 and creates an inspection report in a human-readable format. For example, an inspection report can be created to inform that damage has occurred to a specific inspection object or a device in which the inspection object is installed in a facility. Also, an inspection report can be created to inform that an oil leak has occurred in the facility, covering approximately 80% of the surface of the specific inspection object or the device in which the inspection object is installed.

[0103] In step 510, the automatic scene understanding unit 900 assists the database management unit 500 in creating a contextual relation database 520 indicating contextual relations between objects by a method using a machine learning model based on deep learning. The automatic scene understanding unit 900 can extract contextual relations of meaning, intention, and content that are closer to human semantics and cannot be directly derived by object detection and segmentation from an image.

[0104] The functional relationship management unit 1000 manages functional relationship information indicating the mutual functional relationships of the objects to be inspected or the devices in which the objects to be inspected are arranged. The functional relationship refers to the relationship between objects, such as the functional order relationship, the dependency relationship, and the inclusion relationship. For example, there is a functional relationship in which device 2 functions normally only when the output from device 1 is given. There is also a functional relationship in which if device 2 breaks down, there is a high possibility that part of device 1 will not be able to operate. For example, it is possible to grasp from the functional relationship that the power of a specific device is turned off and the device cannot be started unless the power button is pressed. The inspection report creation unit 800 may create an inspection report based on the analysis result of the inspection image and the functional relationship information.

[0105] In the second embodiment, the database management unit 500 may pre-record connection relationship data representing the arrangement and connection relationships of objects including the inspection target, and the image capture processing unit 700 may output an inspection report in a human-readable format based on the contextual relationships and the connection relationships.

[0106] The above-described embodiments including Examples 1 and 2 include the following items. However, the items included in the present embodiments are not limited to the items listed below.

[0107] (Item 1) The task execution system a database management unit that prerecords a contextual relationship database indicating spatial contextual relationships of a plurality of objects including a target object; an imaging processing unit that acquires image data in which location information indicating a location where the image was captured is added to the image; an object detection unit that detects an object from the image; a segmentation unit for segmenting the object from the image; a contextual relation extraction unit for extracting spatial contextual relations of the segmented objects from the image; a task execution unit that executes a task related to the object based on the contextual relations of the object segmented from the image and the contextual relations recorded in the contextual relation database; has. This allows tasks related to objects to be performed based on the spatial contextual relationships of the objects, thereby making it possible to realize processing that is robust against changes in visual features.

[0108] (Item 2) In the task execution system according to item 1, The database management unit Recording object additional information indicating the behavior and / or properties of the object in advance; The image data is stored in chronological order as historical image data together with information on the date and time when the image data was captured; The task execution unit takes an image at a target point identified based on the contextual relationship of the object classified from the image and the contextual relationship recorded in the contextual relationship database, and estimates a state of the target object based on the acquired target point image, the historical image data, and the target object additional information. This makes it possible to estimate whether a change in an object is normal or abnormal when it occurs. For example, if the object changes over time in the additional information, the change can be judged as normal, and if an abnormal change occurs, it can be estimated that the object has become abnormal.

[0109] (Item 3) In the task execution system according to item 1, the database management unit accumulates the image data in chronological order as historical image data together with information on the date and time when the image data was captured; The task execution unit predicts a future visual appearance or lifespan of the object based on a change trend of the image of the object in the historical image data. This allows prediction of the future visual appearance or lifespan of an object, thereby enabling, for example, appropriate maintenance of the object.

[0110] (Item 4) In the task execution system according to item 1, the database management unit prerecords connection relationship data representing the arrangement and connection relationship of objects including the target object; The object detection unit detects an object from the image based on the image, the contextual relationship, and the connection relationship. This allows for the detection of objects whose visual characteristics have changed.

[0111] (Item 5) In the task execution system according to item 1, the database management unit prerecords connection relationship data representing the arrangement and connection relationship of objects including the target object; The task execution unit outputs an inspection report in a human readable format based on the contextual relationships and the connectivity relationships. This allows for reports on objects represented using their spatial context and connectivity.

[0112] (Item 6) In the task execution system according to item 1, An operation relating to the execution of a task from an operator is accepted on a user interface accessible via a communication network, and the results of the execution of the task are displayed. This allows an operator to use the inspection system via a communications network to execute tasks related to an object and check the results.

[0113] (Item 7) In the task execution system according to item 1, The task is a visual inspection of the object.

[0114] (Item 8) In the task execution system according to item 1, a physical distance calculation unit that calculates a physical distance between the objects detected by the object detection unit from the image data acquired by the imaging processing unit, The database management unit records a contextual relationship including information on a physical distance between objects in the contextual relationship database, The contextual relation extraction unit extracts the contextual relation including the physical distance; The task execution unit executes the task based on a contextual relationship including a physical distance between objects. According to this, since a task related to an object is executed based on a contextual relationship including the physical distance between objects, it is possible to realize processing that is robust against changes in visual features.

[0115] (Item 9) 9. The task execution system according to claim 8, The database management unit further pre-records a physical configuration database indicating a physical configuration including a physical distance between objects; The physical distance calculation unit selects an object from the image that corresponds to an object whose physical configuration is recorded in the physical configuration database, obtains the number of pixels of the width of the selected object in the image, calculates a physical distance per pixel based on the number of pixels and the physical configuration in the physical configuration database, and calculates the physical distance between the objects classified from the image based on the number of pixels between the objects and the physical distance per pixel. [Explanation of symbols]

[0116] 40...facility map, 50...depth camera, 60...GPS, 70...memory, 80...processor, 90...imaging device, 100...visual inspection system, 110...physical configuration database, 200...object detection unit, 300...segmentation unit, 400...contextual relationship extraction unit, 500...database management unit, 520...contextual relationship database, 540...priority database, 600...physical distance calculation unit, 700...imaging processing unit, 800...inspection report creation unit, 900...automatic scene understanding unit, 1000...functional relationship management unit, 2000...visual inspection system

Claims

1. a database management unit that prerecords a contextual relationship database indicating spatial contextual relationships of a plurality of objects including a target object; an imaging processing unit that acquires image data in which location information indicating a location where the image was captured is added to the image; an object detection unit that detects an object from the image; a segmentation unit for segmenting the object from the image; a contextual relation extraction unit for extracting spatial contextual relations of the segmented objects from the image; a task execution unit that executes a task related to the object based on the contextual relations of the object segmented from the image and the contextual relations recorded in the contextual relation database; A task execution system having the following features:

2. 2. The task execution system according to claim 1, The database management unit Recording object additional information indicating the behavior and / or properties of the object in advance; The image data is stored in chronological order as historical image data together with information on the date and time when the image data was captured; the task execution unit takes an image at a target point identified based on the contextual relation of the object classified from the image and the contextual relation recorded in the contextual relation database, and estimates a state of the target object based on the acquired target point image, the historical image data, and the target object additional information; Task execution system.

3. 2. The task execution system according to claim 1, the database management unit accumulates the image data in chronological order as historical image data together with information on the date and time when the image data was captured; The task execution unit predicts a future visual appearance or a lifespan of the object based on a change trend of an image of the object in the historical image data. Task execution system.

4. 2. The task execution system according to claim 1, the database management unit prerecords connection relationship data representing the arrangement and connection relationship of objects including the target object; The object detection unit detects an object from the image based on the image, the context relationship, and the connection relationship. Task execution system.

5. 2. The task execution system according to claim 1, the database management unit prerecords connection relationship data representing the arrangement and connection relationship of objects including the target object; the task execution unit outputs an inspection report in a human-readable format based on the contextual relationship and the connection relationship. Task execution system.

6. 2. The task execution system according to claim 1, accepting an operation relating to the execution of a task from an operator on a user interface accessible via a communication network, and displaying a result of the execution of the task; Task execution system.

7. 2. The task execution system according to claim 1, the task is visual inspection of the object; Task execution system.

8. 2. The task execution system according to claim 1, a physical distance calculation unit that calculates a physical distance between the objects detected by the object detection unit from the image data acquired by the imaging processing unit, The database management unit records a contextual relationship including information on a physical distance between objects in the contextual relationship database, The contextual relation extraction unit extracts the contextual relation including the physical distance; the task execution unit executes the task based on a contextual relationship including a physical distance between objects; Task execution system.

9. 9. The task execution system according to claim 8, The database management unit further pre-records a physical configuration database indicating a physical configuration including a physical distance between objects; the physical distance calculation unit selects an object corresponding to an object whose physical structure is recorded in the physical structure database from among the objects classified from the image, obtains the number of pixels of the width of the selected object in the image, calculates a physical distance per pixel based on the number of pixels and the physical structure in the physical structure database, and calculates a physical distance between the objects classified from the image based on the number of pixels between the objects and the physical distance per pixel; Task execution system.

10. A contextual relationship database indicating spatial contextual relationships of a plurality of objects including the target object is recorded in advance; acquiring image data that is data in which location information indicating a location where the image was taken is added to the image; Detecting an object from the image; Segmenting the object from the image; Extracting spatial contextual relationships of the segmented objects from the image; performing a task related to the object based on the contextual relations of the object segmented from the image and the contextual relations recorded in the contextual relations database; The way a computer performs tasks.

11. A contextual relationship database indicating spatial contextual relationships of a plurality of objects including the target object is recorded in advance; acquiring image data that is data in which location information indicating a location where the image was taken is added to the image; Detecting an object from the image; Segmenting the object from the image; Extracting spatial contextual relationships of the segmented objects from the image; performing a task related to the object based on the contextual relations of the object segmented from the image and the contextual relations recorded in the contextual relations database; A task execution program that causes a computer to do things.

Images