Information processing apparatus, information processing method, and information processing program

Abstract

To make a user grasp a positional information of an object determined to have abnormality based on an ortho-image generated from an image with the object captured.SOLUTION: The present Invention is directed to an abnormality detecting system having an information processing apparatus and a flying body which communicate with each other through a communication means. A CPU of the information processing apparatus includes: an acquiring unit for acquiring a detection difference image as difference between a reference ortho-image generated from a reference image obtained by capturing an object and a comparison ortho-image generated from a comparison image obtained by capturing the object; a determination unit for inputting an inspection difference image into a teacher model that is trained in advance with using combination of a learning difference image as difference between a learning reference ortho-image and a learning ortho-mage and a label indicating abnormality of the object occurring in the learning difference image as a teacher data, and determining whether abnormality has occurred in the object; and an output unit for, based on positional information of a position at which the comparison image as an origin of the inspection difference image is captured, outputting the positional information of the object with the abnormality.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Patent Document 1 describes an information processing device that can suitably compare data for detecting a change point without using a pre-change reference having absolute accuracy. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-72913 Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, the information processing device according to Patent Document 1 performs information processing using orthoimages, but there is still room for improvement in the information processing performed using orthoimages.

[0005] The present disclosure aims to provide an information processing device, an information processing method, and an information processing program that enable a user to grasp the positional information of an object in which an abnormality has been determined to have occurred based on an orthoimage generated from an image of the object. [Means for solving the problem]

[0006] The information processing device of the first aspect includes an acquisition unit that acquires an inspection difference image which is the difference between a reference ortho-image generated from a reference image in which an object is photographed and a comparison ortho-image generated from a comparison image in which the object is photographed at a different time than the reference image; a determination unit that inputs the inspection difference image acquired by the acquisition unit into a supervised model that has been trained in advance using as teacher data a set of a learning difference image which is the difference between a learning reference ortho-image other than the reference ortho-image and a learning comparison ortho-image other than the comparison ortho-image, and a label indicating an abnormality of the object that appears in the learning difference image, and determines whether the abnormality has occurred in the object; and an output unit that outputs location information of the object in which the abnormality has occurred based on location information of the location where the comparison image that was the basis of the inspection difference image was photographed.

[0007] The information processing device of the second aspect is the information processing device of the first aspect, wherein the label includes the degree of the abnormality, and the acquisition unit further acquires the actual degree of the abnormality, and further includes a learning unit that machine-learns the supervised model using the actual degree of abnormality further acquired by the acquisition unit as teacher data linked to the inspection difference image.

[0008] An information processing device of a third aspect is the information processing device of the first or second aspect, wherein the label includes the degree of abnormality, the supervised model outputs the degree of abnormality of the object appearing in the inspection differential image when the inspection differential image is input, and the acquisition unit further acquires the actual degree of abnormality, and further includes a learning unit that reinforces learning of the supervised model using a reward according to a value that is set to be larger the smaller the difference between the actual degree of abnormality further acquired by the acquisition unit and the degree of abnormality output by the supervised model.

[0009] An information processing device of a fourth aspect is an information processing device of the second or third aspect, wherein the acquisition unit further acquires the actual degree of the abnormality when the judgment unit judges that the abnormality has occurred in the object.

[0010] An information processing device of a fifth aspect is an information processing device of any of the first to fourth aspects, wherein the acquisition unit acquires the reference image and the comparison image, and further includes a generation unit that generates the reference ortho-image from the reference image and the comparison ortho-image from the comparison image.

[0011] An information processing device of a sixth aspect is the information processing device of any one of the first to fifth aspects, in which the determination unit inputs the divided inspection differential image to the supervised model.

[0012] In a seventh aspect of the information processing method, a computer executes a process to obtain an inspection difference image which is the difference between a reference ortho-image generated from a reference image in which an object is photographed and a comparison ortho-image generated from a comparison image in which the object is photographed at a different time than the reference image, input the obtained inspection difference image into a supervised model which has been trained in advance using as teacher data a set of a training difference image which is the difference between a training reference ortho-image other than the reference ortho-image and a training comparison ortho-image other than the comparison ortho-image, and a label indicating an abnormality of the object which appears in the training difference image, determine whether the abnormality has occurred in the object, and output location information of the object in which the abnormality has occurred based on location information of the location where the comparison image which was the basis of the inspection difference image was photographed.

[0013] The information processing program of the eighth aspect causes a computer to execute a process of acquiring an inspection difference image which is the difference between a reference ortho-image generated from a reference image in which an object is photographed and a comparison ortho-image generated from a comparison image in which the object is photographed at a different time than the reference image, inputting the acquired inspection difference image into a supervised model which has been trained in advance using as teacher data a set of a training difference image which is the difference between a training reference ortho-image other than the reference ortho-image and a training comparison ortho-image other than the comparison ortho-image, and a label indicating an abnormality of the object appearing in the training difference image, determining whether the abnormality has occurred in the object, and outputting location information of the object in which the abnormality has occurred based on location information of the location where the comparison image which was the basis of the inspection difference image was photographed. Effect of the Invention

[0014] The information processing device, information processing method, and information processing program disclosed herein enable a user to grasp the location information of an object in which an abnormality has been determined to have occurred based on an orthoimage generated from an image of the object. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a first diagram showing a schematic configuration of an anomaly detection system. [Diagram 2] FIG. 1 is a first explanatory diagram illustrating a specific example of a railroad track that is an object. [Diagram 3] FIG. 2 is a block diagram showing a hardware configuration of the information processing device. [Figure 4] 1 is a block diagram showing an example of a functional configuration of an information processing device. [Diagram 5] 1 is a first flowchart showing the flow of processing executed by an information processing device. [Figure 6] 11 is a second flowchart showing the flow of the process executed by the information processing device. [Figure 7] FIG. 2 is a second diagram showing a schematic configuration of the anomaly detection system. [Figure 8]11 is a third flowchart showing the flow of the process executed by the information processing device. [Figure 9] FIG. 2 is a second explanatory diagram for explaining a specific example of a line that is the target object. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, an example of an embodiment of the present disclosure will be described with reference to the drawings. In each drawing, the same or equivalent components and parts are given the same reference numerals. In addition, the dimensional ratios of the drawings are exaggerated for the convenience of explanation and may differ from the actual ratios.

[0017] (First embodiment) First, a first embodiment of an anomaly detection system 100 according to the present embodiment will be described.

[0018] 1 is a first diagram showing a schematic configuration of an anomaly detection system 100. As shown in FIG.

[0019] The information processing device 10 and the flying object 20 are capable of communicating with each other via a communication means N. In this embodiment, a communication line such as the Internet, a LAN (Local Area Network), or a WAN (Wide Area Network) is applied as the communication means N. Also, in this embodiment, a wireless communication line is applied as the communication means N. However, this is not limited to this example. A wired communication line may be applied as the communication means N, or a combination of wired and wireless communication lines may be applied.

[0020] In this embodiment, a drone, which is a type of unmanned aircraft, is used as the aircraft 20. However, the present invention is not limited to this example. For example, the aircraft 20 may be a radio-controlled or manned aircraft.

[0021] As shown in FIG. 1, the aircraft 20 includes a camera 25 and a communication I / F (Interface) 27.

[0022] The camera 25 is an imaging device that captures an image of a railroad track as an object. In this embodiment, a visible light camera capable of capturing visible light images is used as the camera 25. However, the present invention is not limited to this example. For example, an infrared camera capable of capturing infrared images may be used as the camera 25.

[0023] Fig. 2 is a first explanatory diagram for explaining a specific example of a track that is the target object. In Fig. 2, a "predetermined track 30" is shown as the track. As shown in Fig. 2, the predetermined track 30 includes a rail 32 on which railroad vehicles run, and a plurality of track slabs 34, which are concrete plates laid under the rail 32. Although not shown in Fig. 2, there are also weeds growing around the predetermined track 30.

[0024] Here, in FIG. 2, an arrow D indicates the traveling direction of the aircraft 20. That is, the aircraft 20 photographs the predetermined track 30 while moving from the left side to the right side in the figure. FIG. 2 shows an example in which the aircraft 20 photographs four ranges of the predetermined track 30, ranges R1, R2, R3, and R4, from above while moving. The images of the four ranges of the predetermined track 30 photographed by the aircraft 20 are numbered consecutively (e.g., image of range R1 → 1, image of range R2 → 2, image of range R3 → 3, image of range R4 → 4). Also, as shown in FIG. 2, the four ranges partially overlap each other between adjacent ranges.

[0025] Moreover, location information indicating the respective photographing positions is added to the four images of the ranges R1 to R4 photographed by the flying object 20. Furthermore, photographing date information indicating the same photographing date is added to the four images. Then, the four images are tagged as a predetermined group and transmitted to the information processing device 10.

[0026] Returning to FIG. 1, the communication I / F 27 is an interface for communicating with other devices such as the information processing device 10.

[0027] In addition, the anomaly detection system 100 may use any device having an imaging device for photographing the railroad tracks instead of the aircraft 20.

[0028] The information processing device 10 is a computer such as a personal computer, a mobile terminal (for example, a smartphone, a mobile phone, or a tablet terminal), or a server.

[0029] Fig. 3 is a block diagram showing a hardware configuration of the information processing device 10. As shown in Fig. 3, the information processing device 10 has each of components including a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, a display unit 16, and a communication I / F 17. Each component is connected to each other so as to be able to communicate with each other via a bus 19. The CPU 11 is an example of a "computer."

[0030] The CPU 11 is a central processing unit that executes various programs and controls each part. That is, the CPU 11 reads a program from the ROM 12 or the storage 14, and executes the program using the RAM 13 as a working area. The CPU 11 controls each of the above components and performs various arithmetic processing according to the program stored in the ROM 12 or the storage 14.

[0031] The ROM 12 stores various programs and various data. The RAM 13 serves as a working area for temporarily storing programs or data.

[0032] The storage 14 is configured with a storage device such as a hard disk drive (HDD), a solid state drive (SSD), or a flash memory, and stores various programs and various data. In this embodiment, the storage 14 stores an information processing program 200, an image database 202, and a supervised model 204. The information processing program 200 is an example of an "information processing program", and the supervised model 204 is an example of a "supervised model".

[0033] The information processing program 200 is a program for causing the CPU 11 to function as the functional configuration shown in FIG.

[0034] The image database 202 is a database that stores images captured by the aircraft 20 and transmitted from the aircraft 20. As an example, the image database 202 stores four images of the range R1 to R4 of the predetermined track 30 captured by the aircraft 20, sorted by the date of capture.

[0035] The supervised model 204 is a learning model that has been trained in advance to output the degree of abnormality of the predetermined track 30 that appears in the inspection differential image, when an inspection differential image described later is input. An abnormality means something that is not normal, and in this embodiment, the "abnormality" is the content defined by a machine learning engineer who generates the supervised model 204. For example, an example of an abnormality of a track is damage such as cracks and chipping of the rails 32 and track slabs 34. The supervised model 204 will be described in detail later.

[0036] The display unit 16 is, for example, a liquid crystal display, and displays various types of information. The communication I / F 17 is an interface for communicating with other devices such as the flying object 20.

[0037] Next, a description will be given of the functional configuration of the information processing device 10. FIG 4 is a block diagram showing an example of the functional configuration of the information processing device 10.

[0038] 4, the CPU 11 of the information processing device 10 has, as functional components, an acquisition unit 11A, a generation unit 11B, a determination unit 11C, a learning unit 11D, and an output unit 11E. Each functional component is realized by the CPU 11 reading and executing an information processing program 200 stored in the storage 14.

[0039] The acquisition unit 11A acquires, from the image database 202, a reference image in which the predetermined track 30 is photographed and a comparison image in which the predetermined track 30 is photographed at a different timing from that of the reference image. In this embodiment, the "timing different from that of the reference image" is "a timing later than that of the reference image." As described above, the reference image and the comparison image are images in which the same track is photographed. In the example of FIG. 2, for example, four images in the range R1 to R4 photographed at a first time point are the "reference image," and four images in the range R1 to R4 photographed at a second time point that is a time point after the first time point are the "comparison image." In this case, the acquisition unit 11A acquires, from the image database 202, four images in the range R1 to R4 corresponding to the first time point as the reference image, and four images in the range R1 to R4 corresponding to the second time point as the comparison image.

[0040] The generating unit 11B generates a reference ortho-image from the reference image acquired by the acquiring unit 11A, and generates a comparison ortho-image from the comparison image. The reference ortho-image and the comparison ortho-image are ortho-images. The generating unit 11B generates the reference ortho-image and the comparison ortho-image by performing known image processing such as SfM (Structure From Motion) processing on the reference image and the comparison image.

[0041] The acquisition unit 11A acquires an inspection difference image, which is a difference between the reference ortho-image and the comparison ortho-image generated by the generation unit 11B. The acquisition unit 11A acquires an inspection difference image, which is a difference between the reference ortho-image and the comparison ortho-image generated by the generation unit 11B, for example, by searching the Internet.<URL: https: / / daeudaeu.com / c-diff / > and<URL: https: / / daeudaeu.com / python-diff / > The reference ortho-image and the comparison ortho-image are subjected to known image processing such as those disclosed in the above publications to obtain an inspection difference image.

[0042] The judgment unit 11C inputs the inspection difference image acquired by the acquisition unit 11A into a supervised model 204 that has been trained in advance using as teacher data a pair of a learning difference image, which is the difference between a learning reference orthoimage other than the reference orthoimage and a learning comparison orthoimage other than the comparison orthoimage, and a label indicating an abnormality in the track that appears in the learning difference image (hereinafter referred to as an "abnormality label"), and judges whether an abnormality has occurred in the specified track 30.

[0043] The learning reference ortho-image is, for example, an ortho-image generated from a learning reference image in which a track other than the specified track 30 (hereinafter, "the other track") is photographed. The learning comparison ortho-image is, for example, an ortho-image generated from a learning comparison image in which a track other than the learning reference image is photographed at a time other than the time when the learning reference image was photographed (e.g., at a time later than the time when the learning reference image was photographed). The learning difference image is, for example, an ortho-image generated from a learning comparison image in which a track other than the specified track 30 is photographed at a time other than the time when the learning reference image was photographed (e.g., at a time later than the time when the learning reference image was photographed).<URL: https: / / daeudaeu.com / c-diff / > and<URL: https: / / daeudaeu.com / python-diff / > The images are generated by performing known image processing, such as that disclosed in the above, on the learning reference orthoimage and the learning comparison orthoimage.

[0044] The anomaly label includes the degree and type of anomaly. A numerical parameter ranging from 1 to 3 is used as the degree of anomaly. 1 is the smallest degree of anomaly and 3 is the largest. Types of anomalies include cracks and chips. Various learning difference images are evaluated manually, and the degree and type of anomaly are set for each learning difference image. If a person determines that no anomaly has occurred, the degree of anomaly may be set to "not applicable" and the type of anomaly may be set to "not applicable."

[0045] The supervised model 204 is a machine learning model that, when an inspection differential image is input, outputs the degree of abnormality of the predetermined line 30 that appears in the inspection differential image. In detail, when an inspection differential image is input, the supervised model 204 outputs the degree of change of the predetermined line 30 that appears in the inspection differential image between 0% and 100%, and when the inspection differential image has a portion where the degree of change is a predetermined value or more (e.g., 70% or more), outputs the degree of abnormality for that portion as a numerical value of 1, 2, or 3. Therefore, in this embodiment, the determination unit 11C determines that an abnormality has occurred in the predetermined line 30 when the output of the degree of change of the supervised model 204 is 70% or more, and determines that no abnormality has occurred in the predetermined line 30 when the output is less than 70%.

[0046] The learning unit 11D performs machine learning of the supervised model 204. For example, the learning unit 11D performs supervised learning of the supervised model 204 using the above-mentioned training data.

[0047] As described above, the training data input to the supervised model 204 is a set of a learning differential image and an anomaly label. In this embodiment, as the training data, abnormality data indicating a case where an abnormality occurs on the track and abnormality-free data indicating a case where no abnormality occurs on the track are used.

[0048] For example, as data with abnormality, a pair of a learning difference image in which the difference between the learning reference orthoimage and the learning comparison orthoimage includes damage to the track such as cracks and missing parts, and an abnormality label of "degree of abnormality: 2, type of abnormality: crack" can be created. In addition, as data without abnormality, a pair of a learning difference image in which the difference does not include damage to the track but includes changes over time of objects around the track (e.g. changes in the arrangement, shape, or dimensions of weeds), and an abnormality label of "degree of abnormality: not applicable, type of abnormality: not applicable" can be created. The learning unit 11D performs supervised learning of the supervised model 204 based on the above data with abnormality and data without abnormality. As a result, the supervised model 204 can output the degree of change of "70% or more" at which the determination unit 11C determines that an abnormality has occurred only when there is a change that the user wants to detect in the inspection difference image. Therefore, by using the supervised model 204, for example, when the difference shown in the inspection difference image is only the change in the size of the weeds growing around the railroad tracks, the degree of change of "less than 70%" at which the determination unit 11C determines that no abnormality has occurred can be output. Note that the method of supervised learning here can be any known method as appropriate.

[0049] Here, the determination unit 11C inputs the divided inspection difference image to the supervised model 204. As an example, the determination unit 11C divides the inspection difference image into ranges (ranges R1 to R4) indicated by the four images constituting the comparison image on which the inspection difference image is based. Then, the determination unit 11C inputs the four divided inspection difference images to the supervised model 204 in a predetermined order.

[0050] When the determination unit 11C determines that an abnormality has occurred in the predetermined track 30, the output unit 11E outputs the position information of the predetermined track 30 where the abnormality has occurred based on the position information of the position where the comparison image that was the basis of the inspection differential image was taken. For example, it is assumed that the part where the determination unit 11C determines that an abnormality has occurred in the inspection differential image is shown in the image of range R1 among the images of four ranges of the comparison image that was the basis of the inspection differential image. In this case, the output unit 11E acquires the position information added to the image of range R1 from the comparison image stored in the image database 202. Then, the output unit 11E outputs the acquired position information to a predetermined external device.

[0051] Next, a flow of operations until the position information of the predetermined line 30 where an abnormality has occurred is output to a predetermined external device will be described with reference to the flowchart shown in FIG.

[0052] 5 is a first flowchart showing the flow of processing executed by the information processing device 10. The CPU 11 reads the information processing program 200 from the storage 14, expands it in the RAM 13, and executes it, thereby performing the processing. The processing is automatically and repeatedly executed, for example, at regular time intervals.

[0053] 5, the CPU 11 acquires a reference image and a comparative image from the image database 202. Then, the CPU 11 proceeds to step S11.

[0054] In step S11, the CPU 11 generates a reference orthoimage from the reference image acquired in step S10, and a comparison orthoimage from the comparison image, respectively. Then, the CPU 11 proceeds to step S12.

[0055] In step S12, the CPU 11 generates and acquires an inspection difference image, which is the difference between the reference orthoimage and the comparison orthoimage generated in step S11.Then, the CPU 11 proceeds to step S13.

[0056] In step S13, the CPU 11 divides the inspection difference image acquired in step S12 and inputs the divided images to the supervised model 204. Then, the CPU 11 proceeds to step S14.

[0057] In step S14, the CPU 11 determines whether or not an abnormality has occurred in the predetermined line 30 based on the degree of change in the predetermined line 30 appearing in the inspection difference image output from the supervised model 204. If the CPU 11 determines that an abnormality has occurred in the predetermined line 30 (step S14: YES), the CPU 11 proceeds to step S15. On the other hand, if the CPU 11 determines that no abnormality has occurred in the predetermined line 30 (step S14: NO), the CPU 11 ends the process shown in FIG.

[0058] In step S15, the CPU 11 outputs to a predetermined external device the position information of the predetermined track 30 where the abnormality occurred, based on the position information of the position where the comparison image that was the basis of the inspection differential image was captured. Then, the CPU 11 ends the process shown in FIG.

[0059] Next, a flow of additional learning of the supervised model 204 that outputs the degree of abnormality of the predetermined line 30 appearing in the inspection difference image will be described with reference to the flowchart shown in FIG.

[0060] Fig. 6 is a second flowchart showing the flow of processing executed by the information processing device 10. The CPU 11 reads the information processing program 200 from the storage 14, expands it in the RAM 13, and executes it, thereby performing the processing. The processing is automatically and repeatedly executed, for example, every time a certain period of time has elapsed. Note that steps S10 to S15 shown in Fig. 6 are the same as those in Fig. 5, and therefore description of the overlapping parts will be omitted or simplified.

[0061] 6, the CPU 11 determines whether or not an abnormality has occurred in the predetermined line 30 based on the degree of change in the predetermined line 30 appearing in the inspection difference image output from the supervised model 204. Here, if the CPU 11 determines that an abnormality has occurred in the predetermined line 30 (step S14: YES), the CPU 11 proceeds to step S15. On the other hand, if the CPU 11 determines that no abnormality has occurred in the predetermined line 30 (step S14: NO), the CPU 11 ends the process shown in FIG.

[0062] In step S15, the CPU 11 outputs position information of the predetermined line 30 where the abnormality occurred, based on the position information of the position where the comparison image that was the basis of the inspection differential image was captured. Then, the CPU 11 proceeds to step S20.

[0063] In step S20, the CPU 11 acquires the actual degree of abnormality. The "actual degree of abnormality" is a numerical parameter of 1, 2, or 3 that is assigned by a human being to evaluate the degree of abnormality of the predetermined line 30 that appears in the inspection differential image. As an example, when the process shown in FIG. 6 proceeds to step S20, a screen that prompts the user to input the actual degree of abnormality is displayed on the display unit 16. The CPU 11 acquires the numerical parameter of 1, 2, or 3 that is manually input on the screen as the actual degree of abnormality. Then, the CPU 11 proceeds to step S21.

[0064] In step S21, the CPU 11 additionally learns the supervised model 204. Then, the CPU 11 ends the process shown in FIG.

[0065] Here, in step S21, the CPU 11 additionally learns the supervised model 204 in the following manner.

[0066] For example, the CPU 11 performs supervised learning of the supervised model 204 using the actual degree of abnormality acquired in step S20 as training data linked to the inspection differential image acquired in step S12. In this case, as an example of training data, a pair of the inspection differential image acquired in step S12 and an abnormality label of "actual degree of abnormality: 3, type of abnormality: crack" can be created. Then, the CPU 11 performs supervised learning of the supervised model 204 using the created training data.

[0067] As another example, the CPU 11 may additionally learn the supervised model 204 by reinforcement learning. The reinforcement learning is to learn the supervised model 204 by providing a reward according to a value determined so that the smaller the difference between the degree of abnormality output by the supervised model 204 in step S13 and the actual degree of abnormality acquired in step S20, the larger the reward is. Note that the method of reinforcement learning here can be any known method as appropriate.

[0068] As described above, in the information processing device 10, the CPU 11, as a function of the acquisition unit 11A, acquires an inspection difference image that is a difference between a reference ortho-image and a comparison ortho-image. In addition, as a function of the determination unit 11C, the CPU 11 inputs the acquired inspection difference image to the supervised model 204 to determine whether or not an abnormality has occurred in the predetermined track 30. Then, when the CPU 11 determines that an abnormality has occurred in the predetermined track 30, it outputs, as a function of the output unit 11E, position information of the predetermined track 30 where the abnormality has occurred based on position information of the position where the comparison image that is the basis of the inspection difference image was taken. As a result, according to the information processing device 10, the user can be made to understand the position information of the predetermined track 30 where an abnormality has been determined to have occurred based on the ortho-image generated from the image in which the predetermined track 30 was taken.

[0069] As described above, in this embodiment, as the teacher data to be input to the supervised model 204, abnormality data indicating a case where an abnormality occurs on the track and abnormality-free data indicating a case where an abnormality does not occur on the track are used. The abnormality-free data is a set of a learning difference image in which the difference between the learning reference orthoimage and the learning comparison orthoimage does not include damage to the track and includes changes over time of objects present around the track, and an abnormality label including the degree of abnormality indicating a case where an abnormality does not occur. As a result, according to this embodiment, by using the supervised model 204, it is possible to determine that an abnormality has occurred on the predetermined track 30 only when there is a change that the user wants to detect in the inspection difference image.

[0070] Furthermore, in the information processing device 10, the CPU 11 acquires the actual degree of abnormality as a function of the acquisition unit 11A. Then, as a function of the learning unit 11D, the CPU 11 performs machine learning of the supervised model 204 using the acquired actual degree of abnormality as teacher data linked to the inspection differential image. As a result, the information processing device 10 can determine whether or not an abnormality has occurred in the predetermined line 30, based on the actual degree of abnormality of the predetermined line 30 that appears in the inspection differential image evaluated by a human expert.

[0071] Furthermore, in the information processing device 10, the CPU 11, as a function of the learning unit 11D, performs reinforcement learning on the supervised model 204 using a reward according to a value determined so that the smaller the difference between the acquired actual degree of abnormality and the degree of abnormality output by the supervised model 204, the larger the reward. As a result, according to the information processing device 10, the supervised model 204 learns an ideal behavior that maximizes the reward, thereby making it possible to reduce the difference between the actual degree of abnormality and the degree of abnormality output by the supervised model 204.

[0072] Furthermore, in the information processing device 10, the CPU 11, as a function of the acquisition unit 11A, acquires the actual degree of abnormality when it is determined that an abnormality has occurred on the predetermined line 30. As a result, according to the information processing device 10, it is possible to reduce the effort required of a person to input the actual degree of abnormality, compared to a case where the actual degree of abnormality is acquired when it is determined that no abnormality has occurred on the predetermined line 30.

[0073] Moreover, in the information processing device 10, the CPU 11, as a function of the acquisition unit 11A, acquires a reference image and a comparison image from the image database 202. Then, the CPU 11, as a function of the generation unit 11B, generates a reference ortho-image from the reference image and a comparison ortho-image from the comparison image. As a result, the information processing device 10 can generate a reference ortho-image and a comparison ortho-image used when determining whether or not an abnormality has occurred on the predetermined track 30.

[0074] Furthermore, in the information processing device 10, the CPU 11, as a function of the determination unit 11C, inputs the segmented inspection differential image to the supervised model 204. As a result, according to the information processing device 10, it is possible to quickly identify the position information of the predetermined track 30 where an abnormality has occurred, compared to the case where an unsegmented inspection differential image is input to the supervised model 204.

[0075] Second Embodiment Next, a second embodiment of the anomaly detection system 100 according to the present invention will be described while omitting or simplifying parts that overlap with the above embodiment.

[0076] Fig. 7 is a second diagram showing a schematic configuration of the anomaly detection system 100. As shown in Fig. 7, the anomaly detection system 100 has an information processing device 10, an aircraft 20, an image capturing device 40, and a positioning device 60. The information processing device 10, the aircraft 20, and the image capturing device 40 are capable of communicating with each other via a communication means N.

[0077] 7, the photographing device 40 includes a camera 45 and a communication I / F 47. The photographing device 40 is, for example, a mobile terminal such as a smartphone, a mobile phone, or a tablet terminal.

[0078] The camera 45 is an imaging device that captures an image of a railroad track as an object. In this embodiment, a visible light camera capable of capturing visible light images is used as the camera 45. However, the present invention is not limited to this example. For example, an infrared camera capable of capturing infrared images may be used as the camera 45.

[0079] The communication I / F 47 is an interface for communicating with other devices such as the information processing device 10.

[0080] The positioning device 60 is a Global Navigation Satellite System (GNSS) receiver for receiving signals transmitted from positioning satellites. In this embodiment, a Real-Time Kinematics (RTK) module is applied as the positioning device 60. The positioning device 60 is detachably attached to the image capture device 40, and is paired with the image capture device 40 by Bluetooth (registered trademark), for example.

[0081] Next, an image captured by the image capturing device 40 will be described with reference to FIG. In the photographing device 40, a human photographer photographs the predetermined track 30. For example, the photographing mode of the photographing device 40 is activated by a predetermined operation performed by the photographer. Then, while the photographing mode is activated, the photographing device 40 periodically photographs the predetermined track 30 by the photographer moving around with the camera 45 pointed at the predetermined track 30.

[0082] For example, in FIG. 2, the photographer first aims the camera 45 at the predetermined track 30 at the bottom of the figure, and then goes back and forth a number of times in the range R1 to R4 to the left and right to capture a number of images of the predetermined track 30 with the photographing device 40. After that, the photographer aims the camera 45 at the predetermined track 30 at the top of the figure, and then goes back and forth a number of times in the range R1 to R4 to the left and right to capture a number of images of the predetermined track 30 with the photographing device 40. Note that the photographing ranges of two adjacent images photographed by the photographing device 40 partially overlap. Then, the photographing device 40 adds photographing date information indicating the same photographing date to the multiple images photographed above, tags the images as a specific group, and transmits the multiple images to the information processing device 10.

[0083] Next, a hardware configuration of the information processing device 10 according to the second embodiment will be described. The hardware configuration is the same as that of the first embodiment, as shown in FIG.

[0084] In the second embodiment, the image database 202 stores images transmitted from the image capture device 40 in addition to images transmitted from the flying object 20. As an example, the image database 202 stores images transmitted from the flying object 20 and images transmitted from the image capture device 40, sorted by capture date.

[0085] In the second embodiment, the supervised model 204 is a learning model that has been trained in advance so that, when an inspection differential image and inspection point cloud data described later are input, the supervised model 204 outputs the degree of abnormality of the predetermined line 30 that appears in at least one of the inspection differential image and the inspection point cloud data. Details of the inspection point cloud data will be described later.

[0086] Next, a functional configuration of the information processing device 10 according to the second embodiment will be described. The functional configuration is the same as that of the first embodiment, as shown in Fig. 4. As described above, the functional configuration of the information processing device 10 according to the second embodiment will be described below while omitting or simplifying the overlapping parts with the above embodiment.

[0087] In the second embodiment, the acquisition unit 11A acquires, as reference images, from the image database 202, an image captured by the flying object 20 and an image captured by the photographing device 40, which have the same photographing date indicated by the photographing date information. Specifically, the acquisition unit 11A acquires, as reference images, an image of the predetermined track 30 captured by the flying object 20 at a first time point and an image of the predetermined track 30 captured by the photographing device 40 at the first time point from the image database 202. Hereinafter, of the reference images, the image captured by the flying object 20 may be referred to as the "first reference image," and the image captured by the photographing device 40 may be referred to as the "second reference image."

[0088] Moreover, the acquisition unit 11A acquires, as comparison images, from the image database 202, an image captured by the flying object 20 and an image captured by the photographing device 40 that have the same photographing date indicated by the photographing date information. Specifically, the acquisition unit 11A acquires, as comparison images, an image of the predetermined track 30 captured by the flying object 20 at a second time point that is a time point after the first time point, and an image of the predetermined track 30 captured by the photographing device 40 at the second time point, from the image database 202. Hereinafter, of the comparison images, the image captured by the flying object 20 may be referred to as a "first comparison image," and the image captured by the photographing device 40 may be referred to as a "second comparison image."

[0089] In the second embodiment, the generating unit 11B generates a reference orthoimage from the first reference image acquired by the acquiring unit 11A, and generates a comparison orthoimage from the first comparison image.

[0090] Moreover, the generating unit 11B generates reference point cloud data from the second reference image acquired by the acquiring unit 11A, and generates comparison point cloud data from the second comparison image. The reference point cloud data and the comparison point cloud data are point cloud data including three-dimensional coordinate values ​​(X, Y, Z) and color information (R, G, B) of the predetermined track 30. The generating unit 11B generates the reference point cloud data and the comparison point cloud data by performing known image processing such as SfM processing and MVS (Multi-View Stereo) processing on the second reference image and the second comparison image, respectively.

[0091] In the second embodiment, the acquisition unit 11A acquires an inspection difference image, which is a difference between a reference ortho-image and a comparison ortho-image generated by the generation unit 11B. The acquisition unit 11A also acquires inspection point cloud data, which is a difference between the reference point cloud data and the comparison point cloud data generated by the generation unit 11B. The acquisition unit 11A acquires, for example, an inspection point cloud data, which is a difference between the reference point cloud data and the comparison point cloud data generated by the generation unit 11B.<URL: https: / / www.rccm.co.jp / product / mesher / SpaceClaim / post_10.html> The reference point cloud data and the comparison point cloud data are subjected to known image processing such as those disclosed in the above publications to obtain inspection point cloud data.

[0092] In the second embodiment, the determination unit 11C inputs the inspection differential image and the inspection point cloud data acquired by the acquisition unit 11A to the supervised model 204, and determines whether or not an abnormality has occurred in the predetermined line 30. Here, the supervised model 204 outputs a result using at least one of the input inspection differential image and the inspection point cloud data.

[0093] As in the first embodiment, when the determination unit 11C determines that an abnormality has occurred in the predetermined line 30, the output unit 11E outputs position information of the predetermined line 30 where the abnormality has occurred. For example, in the second embodiment, the output unit 11E outputs, to a predetermined external device, the three-dimensional coordinate value of a portion where the determination unit 11C has determined that an abnormality has occurred in the inspection point cloud data, as the position information of the predetermined line 30 where the abnormality has occurred.

[0094] Next, a flow of operations until the position information of the predetermined line 30 where an abnormality has occurred is output to a predetermined external device will be described with reference to the flowchart shown in FIG.

[0095] 8 is a third flowchart showing the flow of processing executed by the information processing device 10. The CPU 11 reads the information processing program 200 from the storage 14, deploys it in the RAM 13, and executes it, thereby performing the processing. The processing is automatically and repeatedly executed, for example, at regular time intervals.

[0096] 8, CPU 11 acquires a reference image and a comparative image from image database 202. Here, CPU 11 acquires a first reference image and a second reference image as the reference images, and acquires a first comparative image and a second comparative image as the comparative images. Then, CPU 11 proceeds to step S31.

[0097] In step S31, the CPU 11 generates a reference orthoimage from the first reference image acquired in step S30, and a comparison orthoimage from the first comparison image, respectively. Then, the CPU 11 proceeds to step S32.

[0098] In step S32, the CPU 11 generates and acquires an inspection difference image, which is the difference between the reference orthoimage and the comparison orthoimage generated in step S31. Then, the CPU 11 proceeds to step S33.

[0099] In step S33, the CPU 11 generates reference point cloud data from the second reference image acquired in step S30, and generates comparison point cloud data from the second comparison image, respectively.Then, the CPU 11 proceeds to step S34.

[0100] In step S34, the CPU 11 generates and acquires inspection point cloud data which is the difference between the reference point cloud data and the comparison point cloud data generated in step S33. Then, the CPU 11 proceeds to step S35.

[0101] In step S35, the CPU 11 inputs the inspection difference image acquired in step S32 and the inspection point cloud data acquired in step S34 to the supervised model 204. Then, the CPU 11 proceeds to step S36.

[0102] In step S36, the CPU 11 determines whether or not an abnormality has occurred on the predetermined line 30 based on the degree of change in the predetermined line 30 appearing in at least one of the inspection differential image and the inspection point cloud data output from the supervised model 204. Here, if the CPU 11 determines that an abnormality has occurred on the predetermined line 30 (step S36: YES), the CPU 11 proceeds to step S37. On the other hand, if the CPU 11 determines that no abnormality has occurred on the predetermined line 30 (step S36: NO), the CPU 11 ends the process shown in FIG.

[0103] In step S37, the CPU 11 outputs the position information of the predetermined line 30 where the abnormality has occurred to a predetermined external device. Then, the CPU 11 ends the process shown in FIG.

[0104] (others) Although the embodiment has been described above, the technical scope of the present invention is not limited to the scope described in the above embodiment. Various modifications or improvements can be made to the above embodiment without departing from the gist of the invention, and such modifications or improvements are also included in the technical scope of the present invention.

[0105] Furthermore, the above embodiments do not limit the invention according to the claims, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention. The above embodiments include inventions at various stages, and various inventions are extracted by combining the multiple components disclosed. Even if some components are deleted from all the components shown in the embodiments, as long as the effect is obtained, the configuration from which these some components are deleted can be extracted as an invention.

[0106] For example, in the above embodiment, the information processing device 10 and the flying object 20 are configured as separate entities. However, this is not limited to this example. For example, the information processing device 10 may be built into the flying object 20.

[0107] In the above embodiment, the information processing device 10 and the flying object 20 are capable of communicating with each other via the communication means N. However, this is not limited to this example. For example, the information processing device 10 and the flying object 20 do not need to be capable of communicating with each other via the communication means N. In this case, for example, the flying object 20 stores the captured image captured by the camera 25 in a storage device such as a memory card. Then, the CPU 11 of the information processing device 10 acquires the captured image captured by the camera 25 from the storage device via the input / output I / F provided in the information processing device 10.

[0108] In the above embodiment, the object is described as a railroad track, but is not limited thereto. For example, the object may be a road, a pipe, an overpass, a soundproof wall, a wire rope supporting a bridge, a plant, a windmill blade, a solar panel, an exterior wall of a building, a roof of a building, a tunnel, an embankment, a tree, a slope of a snowy mountain, etc.

[0109] In the above embodiment, the CPU 11 inputs the divided inspection difference image to the supervised model 204, but this is not limiting. For example, the CPU 11 may input the acquired inspection difference image to the supervised model 204 as it is.

[0110] In the above embodiment, the object shown in the training difference image of the teacher data input to the supervised model 204 is the same type of object as the object shown in the inspection difference image, specifically, a railroad, but this is not limited to this. For example, the object shown in the training difference image may be a different type of object from the object shown in the inspection difference image. Specifically, the object shown in the training difference image may be a road or an overpass, and the object shown in the inspection difference image may be a railroad.

[0111] In the above embodiment, the CPU 11 determines whether an abnormality has occurred on the predetermined track 30 based on an image of the predetermined track 30 photographed by the flying object 20 from above, but this is not limited thereto. For example, the CPU 11 may determine whether an abnormality has occurred on the predetermined track 30 based on images of the predetermined track 30 photographed by the flying object 20 from below and both left and right sides in addition to the image of the predetermined track 30 photographed from above. In addition, when based on the images photographed from the below and both left and right sides, the inspection difference image input to the supervised model 204 may not be a difference image showing the difference between orthoimages. For example, when determining whether an abnormality has occurred on the predetermined track 30 based on an image of the predetermined track 30 photographed from the left side (hereinafter, "left side image"), the inspection difference image may be a difference image showing the difference between a left side image photographed at a first time point and a left side image photographed at a second time point that is a time point after the first time point.

[0112] In the above embodiment, the CPU 11 inputs an inspection difference image, which is a difference between a reference ortho-image and a comparison ortho-image, to the supervised model 204, but this is not limiting. For example, the inspection difference image may be a difference between one reference image and one comparison image, and the CPU 11 may input the inspection difference image to the supervised model 204. In other words, the inspection difference image does not have to be an image that indicates a difference between ortho-images. In this case, the CPU 11 may divide the inspection difference image into several regions and input the divided inspection difference images to the supervised model 204 in a predetermined order.

[0113] In the above embodiment, a specific example of the line as the target object is described with reference to Fig. 2, but the structure of the line is not limited to this. Hereinafter, another specific example of the line will be described with reference to Fig. 9.

[0114] FIG. 9 is a second explanatory diagram for explaining a specific example of a track that is the target object. In FIG. 9, a "predetermined track 300" is shown as the track. As shown in FIG. 9, the predetermined track 300 includes a rail 320 on which a railway vehicle runs, and a sleeper 340 that is provided under the rail 320 and vertically supports the rail 320. Although not shown in FIG. 9, paving stones called "ballast" are laid all around the predetermined track 300. Similarly, although not shown in FIG. 9, there are also weeds growing around the predetermined track 300.

[0115] Here, in FIG. 9, an arrow D indicates the traveling direction of the flying object 20. That is, the flying object 20 photographs the predetermined track 300 while moving from the left side to the right side in the figure. FIG. 9 shows an example in which the flying object 20 photographs four ranges of the predetermined track 300, ranges R5, R6, R7, and R8, from above while moving. The images of the four ranges of the predetermined track 300 photographed by the flying object 20 are numbered consecutively (e.g., image of range R5 is 5, image of range R6 is 6, image of range R7 is 7, and image of range R8 is 8). Also, as shown in FIG. 9, the four ranges partially overlap each other between adjacent ranges.

[0116] Moreover, location information indicating the respective photographing positions is added to the four images of the range R5 to R8 photographed by the flying object 20. Furthermore, photographing date information indicating the same photographing date is added to the four images. Then, the four images are tagged as a predetermined group and transmitted to the information processing device 10.

[0117] In the above embodiment, the photographing device 40 has the camera 45 and the communication I / F 47, but the configuration of the photographing device 40 is not limited to this. For example, the photographing device 40 may have a LIDAR (light detection and ranging) in addition to these configurations. If the photographing device 40 has a LIDAR, the photographing device 40 also measures the track using the LIDAR while photographing with the camera 45, and measures point cloud data of the track for a measurable range. Then, when the point cloud data is measured, the photographing device 40 transmits the point cloud data to the information processing device 10 together with the image photographed by the camera 45. In this case, the information processing device 10 generates reference point cloud data and comparison point cloud data based on the image and point cloud data transmitted from the photographing device 40.

[0118] In addition, in the above embodiment, the processor refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPU: Central Processing Unit, etc.) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.).

[0119] In addition, the operations of the processor in the above embodiments may be performed not only by one processor, but also by multiple processors located at physically separate locations working together. The order of the operations of the processor is not limited to the order described in the above embodiments, and may be changed as appropriate.

[0120] In the above embodiment, various processes executed by the CPU 11 by reading software (programs) may be executed by various processors other than the CPU. In this case, examples of the processor include a PLD (Programmable Logic Device) such as an FPGA (Field-Programmable Gate Array) whose circuit configuration can be changed after manufacture, and a dedicated electric circuit such as an ASIC (Application Specific Integrated Circuit) which is a processor having a circuit configuration designed exclusively for executing a specific process. In addition, the various processes may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same or different types (for example, a plurality of FPGAs, a combination of a CPU and an FPGA, etc.). In addition, the hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0121] In the above embodiment, the information processing program 200 is stored (installed) in advance in the storage 14, but the present invention is not limited to this. The information processing program 200 may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The information processing program 200 may be downloaded from an external device via a communication means N.

[0122] In the above embodiment, the processing in the information processing device 10 is realized by a software configuration using a computer by executing a program, but the present invention is not limited to this. For example, the processing in the information processing device 10 may be realized by a hardware configuration or a combination of a hardware configuration and a software configuration.

[0123] In addition, the configurations of the information processing device 10 and the flying object 20 described in the above embodiment are merely examples, and it goes without saying that unnecessary parts may be deleted or new parts may be added within the scope of the present invention.

[0124] Furthermore, the processing flow in the information processing device 10 described in the above embodiment is also one example, and it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged within the scope of the present invention.

[0125] Furthermore, the present disclosure is not limited to the above, and it goes without saying that various modifications can be made without departing from the spirit and scope of the present disclosure. [Explanation of symbols]

[0126] 10. Information processing device 11 CPU (Computer) 11A Acquisition Department 11B Generation part 11C Judgment section 11D Learning Department 11E Output section 200 Information Processing Program 204 Supervised Model

Claims

1. An acquisition unit that acquires an inspection difference image that is a difference between a reference ortho-image generated from a reference image in which an object is photographed and a comparison ortho-image generated from a comparison image in which the object is photographed at a timing different from that of the reference image; a determination unit that inputs the inspection difference image acquired by the acquisition unit into a supervised model that has been trained in advance using as training data a set of a learning difference image, which is a difference between a learning reference ortho-image other than the reference ortho-image and a learning comparison ortho-image other than the comparison ortho-image, and a label indicating an abnormality of the object that appears in the learning difference image; and determines whether the abnormality has occurred in the object. an output unit that outputs position information of the object where the abnormality has occurred based on position information of a position where the comparison image that is the basis of the inspection difference image was captured; An information processing device comprising:

2. the label includes a degree of the abnormality; The acquisition unit further acquires an actual degree of the abnormality, a learning unit that performs machine learning of the supervised model using the actual degree of the abnormality further acquired by the acquisition unit as training data linked to the inspection difference image, The information processing device according to claim 1 .

3. the label includes a degree of the abnormality; the supervised model, when receiving the inspection difference image, outputs a degree of abnormality of the object appearing in the inspection difference image; The acquisition unit further acquires an actual degree of the abnormality, a learning unit that performs reinforcement learning of the supervised model using a reward according to a value that is set so that the smaller the difference between the actual degree of abnormality further acquired by the acquisition unit and the degree of abnormality output by the supervised model, the larger the reward. The information processing device according to claim 1 .

4. The acquisition unit further acquires an actual degree of the abnormality when the determination unit determines that the abnormality has occurred in the object. The information processing device according to claim 2 .

5. The acquisition unit acquires the reference image and the comparison image, A generation unit that generates the reference ortho-image from the reference image and the comparison ortho-image from the comparison image, The information processing device according to claim 1 .

6. The determination unit inputs the divided inspection difference image to the supervised model. The information processing device according to claim 1 .

7. Acquire an inspection difference image which is a difference between a reference ortho-image generated from a reference image in which an object is photographed and a comparison ortho-image generated from a comparison image in which the object is photographed at a timing different from that of the reference image; inputting the acquired inspection difference image into a supervised model that has been trained in advance using as training data a set of a learning difference image, which is a difference between a learning reference ortho-image other than the reference ortho-image and a learning comparison ortho-image other than the comparison ortho-image, and a label indicating an abnormality of the object that appears in the learning difference image; and determining whether or not the abnormality has occurred in the object; outputting position information of the object where the abnormality has occurred based on position information of a position where the comparison image that is the basis of the inspection difference image was captured; An information processing method in which processing is performed by a computer.

8. Acquire an inspection difference image which is a difference between a reference ortho-image generated from a reference image in which an object is photographed and a comparison ortho-image generated from a comparison image in which the object is photographed at a timing different from that of the reference image; inputting the acquired inspection difference image into a supervised model that has been trained in advance using as training data a set of a learning difference image, which is a difference between a learning reference ortho-image other than the reference ortho-image and a learning comparison ortho-image other than the comparison ortho-image, and a label indicating an abnormality of the object that appears in the learning difference image; and determining whether or not the abnormality has occurred in the object; outputting position information of the object where the abnormality has occurred based on position information of a position where the comparison image that is the basis of the inspection difference image was captured; An information processing program for causing a computer to execute processing.

Images