Detection device, detection method, and detection program

Abstract

To enable inspection of abnormal parts as a priority for inspection targets. [Solution] The detection device includes an acquisition unit and a detection unit. The acquisition unit acquires an extracted image in which an inspection object is extracted from a predetermined image based on depth information estimated from the predetermined image. The detection unit detects abnormalities in the inspection object based on segments included in the extracted image among segments obtained by segmenting the predetermined image.

Description

[Technical Field]

[0001] The present invention relates to a detection device, a detection method, and a detection program. [Background technology]

[0002] Services have been provided to detect defects and damage to structures from images captured by drones, etc. However, the captured images often contain various information other than the structure being inspected (for example, objects other than the inspection target or the background), and there is a need to detect defects and damage by focusing only on the inspection target.

[0003] Patent Document 1 discloses a technology for object detection using monocular depth estimation. However, this technology has room for improvement in terms of accurately extracting an inspection target from an image and detecting abnormalities in the extracted inspection target. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2022-142789 Summary of the Invention [Problem to be solved by the invention]

[0005] The detection device of the present application comprises an acquisition unit that acquires an extracted image in which an inspection object is extracted from a specified image based on depth information estimated from the specified image, and a detection unit that detects abnormal areas in the inspection object based on segments included in the extracted image among segments obtained by segmentation of the specified image. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram illustrating the problem of the present invention. [Figure 2]FIG. 2 is a diagram illustrating an overview of the proposed technology. [Figure 3] FIG. 3 is a diagram illustrating an example of the configuration of an information processing system according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of the configuration of a detection device according to an embodiment. [Figure 5] FIG. 5 is a diagram illustrating a learning method according to the embodiment. [Figure 6] FIG. 6 is a diagram showing a method for determining the range of depth values. [Figure 7] FIG. 7 is a diagram illustrating an image synthesis method according to the embodiment. [Figure 8] FIG. 8 is a diagram (1) showing the procedure of information processing executed by the detection device. [Figure 9] FIG. 9 is a diagram (2-1) showing the procedure of information processing executed by the detection device. [Figure 10] FIG. 10 is a diagram (2-2) showing the procedure of information processing executed by the detection device. [Figure 11] FIG. 11 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the detection device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0007] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0008] One or more embodiments (including examples, modifications, and application examples) described below can be implemented independently. However, at least a portion of the embodiments described below may be implemented in appropriate combination with at least a portion of another embodiment. These embodiments may include novel features that are different from each other. Therefore, these embodiments may contribute to solving different purposes or problems and may produce different effects from each other.

[0009] Furthermore, in the following embodiments, a "base station BS" is exemplified as an inspection target, but the inspection target to which the proposed technology of the present application can be applied is not limited to a base station BS. In other words, the proposed technology of the present application can be applied to missions and services in which various structures are inspected.

[0010] In addition, the images exemplified in the following embodiments are assumed to be captured images obtained by photographing from a position where the inspection target is located in front. However, in the images used in the proposed technology of the present application, the inspection target does not necessarily need to be located in front.

[0011] Furthermore, the technology proposed in this application differs from conventional technologies in that it is not simply a combination of a semantic segmentation model and a monocular depth estimation model, but rather utilizes unique learning and depth information to solve the above-mentioned problems.

[0012] <Embodiment> 1. Introduction Images taken by drones through aerial photography (taking pictures of base stations BS from a drone) are used to inspect base stations BS for abnormalities (for example, holes or rusty areas). For example, one possible method is to use an AI model to analyze the captured images to detect abnormalities.

[0013] However, captured images often contain various information other than the base station BS being inspected (for example, a steel tower in the background that is unrelated to the base station BS), and a general-purpose AI model may recognize things other than the inspection target as base stations BS, and may detect abnormalities that exist outside the inspection target. This makes it impossible to carry out accurate and efficient inspections. Therefore, there is a need to appropriately exclude things other than the inspection target and prioritize inspection of abnormalities in the base station BS.

[0014] The proposed technology of the present application (hereinafter referred to as "proposed technology") has been made in consideration of the above-mentioned problems, and aims to accurately extract an inspection target from an image and detect abnormalities in the extracted inspection target. Prior to explaining the proposed technology, the above-mentioned problems will first be explained in more detail using Figures 1 and 2.

[0015] [2. Description of the assignment] Figure 1 is a diagram illustrating the problem of the present application. Figure 1(a) shows an image IM1 captured from the side of a base station BS by a drone. The image on the right shows the state before the AI ​​model is applied, and the image on the left shows the state after the AI ​​model is applied. The captured image IM1 includes the base station BS being inspected in the foreground, but an unrelated steel tower (circled area) that is not being inspected is captured in the background. When this captured image IM1 is applied to a semantic segmentation model, an area AR of an abnormality is detected not only in the base station BS, but also in the steel tower in the background that is not being inspected.

[0016] The same can be said for Figure 1(b). Figure 1(b) shows image IM2, which was taken from above by a drone, with the image on the right showing the state before the AI ​​model was applied, and the image on the left showing the state after the AI ​​model was applied. Image IM2 includes the base station BS, which is the subject of inspection, in the foreground, but an unrelated steel tower (circled area) that is not the subject of inspection appears in the background. When this type of image IM2 is applied to a semantic segmentation model, an area AR of an abnormality is detected not only in the base station BS, but also in the steel tower in the background that is not the subject of inspection.

[0017] In this way, if abnormalities are detected in areas other than the inspection target, for example, workers will have to distinguish between the inspection target and the non-inspection target, which will make the work more complicated, and there is also a risk that they will mistakenly identify the abnormality. Therefore, there is a need to appropriately exclude areas other than the inspection target and give priority to inspecting abnormalities for base stations BS.

[0018] [3. Overview of proposed technology] Based on the explanation so far, an outline of the proposed technology will be explained. Figure 2 is a diagram showing an outline of the proposed technology. Figure 2 shows a scene in which an abnormality is detected in a base station BS included in a captured image IM1.

[0019] In addition, the proposed technology may use two AI models, as shown in Figure 2. Specifically, the proposed method may use model M1, which is a semantic segmentation model, and model M2, which is a monocular depth estimation model.

[0020] Model M1 may be realized by deep learning by associating predefined labels or categories (e.g., anomaly areas or base station BS areas) with pixels in an image. As shown in Figure 2, model M1 is used to detect anomalies. However, model M1 may mistakenly recognize a steel tower in the background that is not the inspection target as the inspection target base station BS, and may detect anomaly areas AR even in the steel tower. Therefore, model M2 is used to exclude objects other than the inspection target and extract the inspection target base station BS.

[0021] In the proposed technology, the model M1 detects the abnormal area AR in the captured image IM n and the captured image IM n Based on the depth information DP estimated based on n The extracted image ex_IM from which the base station BS area is extracted n As will be described later, the extracted image ex_IM nTo obtain the range RG, a range RG of depth values ​​determined during learning of the model M1, which is a range RG of depth values ​​that is higher than the possibility of an abnormality being present for the base station BS, may be used.

[0022] As a result, the extracted image ex_IM is generated as shown in Figure 2. n In the example of Figure 1, the area AR of the abnormal part is detected only for the base station BS, and the area AR of the abnormal part is not detected for the steel tower in the background. Therefore, in the proposed technology, n The area AR (segment) of the abnormality included in the image can be detected as the abnormality owned by the base station BS. Therefore, the proposed technology makes it possible to extract the inspection object from the image with high accuracy and detect the abnormality in the extracted inspection object.

[0023] In addition, the proposed technology will utilize AI to improve communication quality and increase operational efficiency, and will serve as an innovative technological foundation for the telecommunications industry, thereby contributing to the achievement of Goal 9 of the Sustainable Development Goals (SDGs), "Build resilient infrastructure, promote inclusive and sustainable industrialization, and promote inclusive and sustainable industrialization."

[0024] [4. System Configuration] A system in which the proposed technology described in Fig. 2 is realized will be described. Fig. 3 is a diagram showing an example of the configuration of an information processing system 1 according to an embodiment. As shown in Fig. 3, the information processing system 1 may include a user device 10, a mobile device 20, and a detection device 100. Note that the information processing system 1 may include a plurality of user devices 10, a plurality of mobile devices 20, and a plurality of detection devices 100.

[0025] The user device 10 is an example of an information processing terminal used by a user (e.g., a worker performing inspection work). The user device 10 may be a smartphone, a wearable device, a tablet terminal, a notebook PC (Personal Computer), a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), or the like. For example, an application AP may be installed in the user device 10 to realize transmission and reception of information between the user device 10 and the detection device 100. The application AP may be a general-purpose application such as a web browser, or may be a dedicated application newly implemented in accordance with the proposed technology.

[0026] The user device 10 may display the detection result by the detection device 100. In the example of FIG. 2, the user device 10 displays an extracted image IM in which the abnormal portion is reflected. 11 may be displayed.

[0027] The mobile device 20 is a device body equipped with a photographing function, and may be, for example, a drone. The mobile device 20 also has a camera 21 as the photographing function. For example, a worker may move the mobile device 20 to an appropriate photographing position and operate the camera 21 at that position to photograph the base station BS. The mobile device 20 and the camera 21 may be operated by the user device 10 or by a dedicated controller.

[0028] The detection device 100 is a central device that executes information processing related to the proposed technology and may be implemented as a cloud server device. The detection device 100, the mobile device 20, and the user device 10 are connected to each other via a network N so as to be able to communicate with each other via wired or wireless communication.

[0029] One approach of information processing according to the embodiment is to detect the area of ​​the abnormality by limiting it to an area estimated to be a base station BS using the output result of model M1 (semantic segmentation model) for an image (an example of a predetermined image) in which the abnormality is unknown and the output result of model M2 (monocular depth estimation model) for an image in which the abnormality is unknown. Another approach of information processing according to the embodiment is to generate model M1 that estimates the area of ​​the abnormality by limiting it to an area estimated to be a base station BS by using depth information estimated by model M2 as learning data for model M1.

[0030] 5. Configuration of the detection device The detection device 100 according to the embodiment will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the configuration of the detection device 100 according to the embodiment. As shown in Fig. 4, the detection device 100 includes a communication unit 110, a storage unit 120, and a control unit 130.

[0031] <Communication Unit 110> The communication unit 110 is realized by, for example, a network interface card (NIC), etc. For example, the communication unit 110 transmits and receives information to and from the user device 10 and the mobile device 20.

[0032] <Storage section 120> The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 120 may store, for example, data and programs related to the information processing according to the embodiment. As shown in FIG. 4, the storage unit 120 may include a teacher image data storage unit 121, a model data storage unit 122, and a detection result storage unit 123.

[0033] <Control unit 130> The control unit 130 is realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like using RAM as a work area to execute various programs (for example, the information processing program according to the embodiment) stored in a storage device inside the detection device 100. The control unit 130 is also realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0034] As shown in Fig. 4, control unit 130 has a learning unit 131, a determination unit 132, a segmentation unit 133, a generation unit 134, an acquisition unit 135, a detection unit 136, and an image generation unit, and realizes or executes the functions and actions of information processing described below. Note that the internal configuration of control unit 130 is not limited to the configuration shown in Fig. 4, and may have other configurations as long as they perform the information processing described below. Furthermore, the connection relationship between each processing unit included in control unit 130 is not limited to the connection relationship shown in Fig. 4, and may be other connection relationships.

[0035] <Study Section 131> The learning unit 131 (first learning unit) takes an image of an inspection target having an abnormality, in which the area AR of the abnormality is annotated in pixel units, as a teacher image L_IM n The learning unit 131 learns a model M1 (segmentation model) so as to estimate the area AR of the abnormal part for the input image inIM as follows: That is, the learning unit 131 learns a teacher image L_IM which is an image of an inspection target having an abnormal part, and in which the inside of the area AR of the abnormal part is annotated in pixel units. n By having the model M1 learn the features of the above, a model M1 is generated that outputs an area AR of an abnormal part for the input image inIM.

[0036] Here, the learning method by the learning unit 131 will be described with reference to Fig. 5. Fig. 5 is a diagram showing the learning method according to the embodiment. Fig. 5 shows an original image oIM captured by the base station BS. n(Color image with RGB information) and the original image oIM as the correct label n Annotation information AN n and the training image L_IM n The scene where model M1 is trained is shown. n is an image of a base station BS having an abnormality (an example of an image of an inspection target), and may include other objects besides the inspection target.

[0037] For example, annotation information AN n is the original image oIM n The annotation information AN may be information in which each pixel in the area indicating the base station BS is labeled with category information (Base Station) indicating the base station BS. n is the original image oIM n The information may be information in which category information (Abnormality) indicating the abnormality is labeled for each pixel in the region indicating the abnormality among the pixels of the training image L_IM. n is a set of multiple original images (oIM) taken by the base station BS from various positions and directions. n may be prepared for each

[0038] The learning unit 131 (first learning unit) learns the teacher image L_IM n may be input to a model M (e.g., a pre-trained semantic segmentation model), and the model M1 may be trained to estimate an area AR of an abnormality in an area estimated to be a base station BS. For example, the training unit 131 may train the model M1 to estimate a base station area indicating the base station BS for the input image inIM, and to estimate an area AR of an abnormality within the base station area. That is, when the input image inIM is input, the training unit 131 may generate a model M1 that segments an area AR of an abnormality that is estimated to be included in the base station BS, and outputs the segmentation result.

[0039] As shown in Fig. 5, model M1 may mistakenly recognize a steel tower in the background of the input image inIM as the base station BS, which is the target of inspection, and may detect an abnormal area AR not only in the base station BS but also in the steel tower. Therefore, model M2 is used to determine the area that is estimated to be the base station BS.

[0040] <Determining Unit 132> Returning to FIG. 4, the determination unit 132 determines the depth image dp_oIM n Based on a comparison of the image of the annotated inspection target and the image of the target object, the determination unit 132 determines a depth range RG for estimating an area AR of an abnormal area in the predetermined image IM. For example, the determination unit 132 may determine the minimum depth value min as the distance from the imaging point to the area AR of the abnormal area AR that is the shortest distance from the imaging point. As a result, for example, the depth range RG may be defined as a range up to infinity that includes the minimum value min as a threshold.

[0041] Here, the determination method by the determination unit 132 will be described with reference to FIG. 6. FIG. 6 is a diagram showing a determination method for determining the depth value range RG. In FIG. 6, a teacher image L_IM1 (teacher image L_IM n The original image oIM1 and the training image L_IM2 (an example of training image L_IM n The original image oIM2 and the training image L_IM3 (an example of training image L_IM n 10 shows a scene in which a range RG of depth values ​​including the minimum value min as a threshold is determined based on each of the original images oIM3 used as an example of the depth range RG.

[0042] According to the example of FIG. 6(a), the determining unit 132 determines whether the depth image dp_oIM1 (depth image dp_oIM nThe depth image dp_oIM1 (an example of the depth image dp_oIM1) is compared with the teacher image L_IM1 including the annotation information AN1, and the determination unit 132 identifies the anomaly region AR11 located closest to the imaging point among the anomaly region AR as the correct label. Because the depth image dp_oIM1 includes a depth value for each pixel, the determination unit 132 identifies the anomaly region AR11 located at the shortest distance from the imaging point based on the depth values ​​for each corresponding pixel between the depth image dp_oIM1 and the teacher image L_IM1 and the anomaly region AR as the correct label. The determination unit 132 may then calculate the minimum value min based on the depth values ​​for each pixel included in the anomaly region AR11. For example, the determination unit 132 may calculate the minimum value min by statistically processing the depth values ​​for each pixel included in the anomaly region AR11. For example, if the statistical value of the depth value obtained as a result of the statistical processing is "180", the determining unit 132 may calculate the minimum value min="180" as shown in FIG. 6(a).

[0043] According to the example of FIG. 6(b), the determining unit 132 determines whether the depth image dp_oIM2 (depth image dp_oIM n The depth image dp_oIM2 (an example of the depth image dp_oIM2) is compared with the teacher image L_IM2 containing the annotation information AN2, and the determination unit 132 identifies the region AR21 of the abnormal region located closest to the imaging point among the regions AR of the abnormal region as the correct answer label. Because the depth image dp_oIM2 includes a depth value for each pixel, the determination unit 132 identifies the region AR21 of the abnormal region located at the shortest distance from the imaging point based on the depth values ​​of each corresponding pixel between the depth image dp_oIM2 and the teacher image L_IM2 and the region AR of the abnormal region as the correct answer label. The determination unit 132 may then calculate the minimum value min based on the depth values ​​of each pixel included in the region AR21 of the abnormal region. For example, the determination unit 132 may calculate the minimum value min by statistically processing the depth values ​​of each pixel included in the region AR21 of the abnormal region. For example, if the statistical value of the depth value is "80" as a result of the statistical processing, the determining unit 132 may calculate the minimum value min="80" as shown in FIG. 6(b).

[0044] According to the example of FIG. 6(c), the determining unit 132 determines whether the depth image dp_oIM3 (depth image dp_IM n The depth image dp_oIM3 (an example of the depth image dp_oIM3) is compared with the teacher image L_IM3 containing the annotation information AN3, and the determination unit 132 identifies the anomaly region AR31 located closest to the imaging point among the anomaly region AR labeled as the correct answer. Because the depth image dp_oIM3 includes a depth value for each pixel, the determination unit 132 identifies the anomaly region AR31 located at the shortest distance from the imaging point based on the depth values ​​for each corresponding pixel between the depth image dp_oIM3 and the teacher image L_IM3 and the anomaly region AR labeled as the correct answer. The determination unit 132 may then calculate the minimum value min based on the depth values ​​for each pixel included in the anomaly region AR31. For example, the determination unit 132 may calculate the minimum value min by statistically processing the depth values ​​for each pixel included in the anomaly region AR31. For example, if the statistical processing results in a depth value of "150", the determining unit 132 may calculate the minimum value min="150" as shown in FIG. 6(c).

[0045] In this state, the determination unit 132 may determine an optimal minimum value min from among the minimum values ​​min "180," "80," and "150" based on predetermined conditions. For example, the determination unit 132 may display all of the minimum values ​​min "180," "80," and "150" on the user device 10 and allow the user to select a desired minimum value min from these. Then, the determination unit 132 may determine the minimum value min selected by the user as the optimal minimum value min.

[0046] Here, the smaller the minimum value min, the shallower the depth (the shortest distance from the imaging point). Therefore, for example, if the user places importance on recall, the user may select the minimum value min "80," which is considered to have the lowest risk of missing an abnormality. On the other hand, for example, if the user places importance on another evaluation index (for example, f-value), the determination unit 132 may select the optimal one from the minimum value min "180," the minimum value min "80," and the minimum value min "150."

[0047] The depth image dp_oIM n may be generated by the generating unit 134, which will be described later. For example, the generating unit 134 generates the depth image dp_oIM based on the depth value for each pixel, which is the output result when the original image oIM is input to the model M2. n can be generated.

[0048] <Segmentation section 133> 4, the segmentation unit 133 (first segmentation unit) performs segmentation on the predetermined image IM based on the predetermined image IM and the trained model M1. For example, the segmentation unit 133 inputs the predetermined image IM to the trained model M1, thereby classifying an area AR of an abnormal part from the predetermined image IM.

[0049] <Generation unit 134> The generation unit 134 (first generation unit) generates a depth image dp_IM corresponding to the predetermined image IM based on the depth value for each pixel, which is the output result when the predetermined image IM is input to the model M2.

[0050] <Acquisition part 135> The acquisition unit 135 acquires an extracted image ex_IM in which the inspection object is extracted from the predetermined image IM based on the depth information estimated based on the predetermined image IM. n For example, the acquisition unit 135 acquires the segmented predetermined image, the IM, and the depth image dp_oIM. n Based on the comparison with the extracted image ex_IM n Get.

[0051] <Detection unit 136> The detection unit 136 detects an extracted image ex_IM from among the segments obtained by segmenting the predetermined image IM. n For example, the detection unit 136 detects an abnormal portion from the inspection target based on the segment included in the extracted image ex_IM n may be detected as an abnormal portion of the inspection object.

[0052] <Image generation unit 137> The image generation unit 137 generates an image of the inspection object and a depth image dp_oIM corresponding to the image of the inspection object. n For example, the image generating unit 137 dynamically generates an image of the inspection target having an abnormality based on an image of a base station BS having no abnormality and a depth image dp_oIM generated based on depth information DP estimated from the image of the base station BS. n Based on the above, a partial image pt_oIM estimated to include the base station BS from the image of the base station BS is obtained. n Then, the image generating unit 137 extracts the extracted partial image pt_oIM. n By virtually synthesizing the abnormal part, an image of the base station BS having the abnormal part is generated.

[0053] Here, the image synthesis method by the image generation unit 137 will be described with reference to Fig. 7. Fig. 7 is a diagram showing the image synthesis method according to the embodiment. The original image oIM shown in Fig. 7 n ' is an image of the base station BS that does not have an abnormality, and may be one of the original images oIM (color image having RGB information) taken by the base station BS. FIG. 7 shows such an original image oIM n ´, the original image oIM with the abnormal part n A scene is shown in which the two are combined.

[0054] First, the image generation unit 137 generates an original image oIM n ´ is input to model M2, and the output is the depth value for each pixel. nIn this state, the image generating unit 137 generates a depth image dp_oIM n ´, the base station BS is assumed to be in the foreground, and the depth image dp_oIM n 7 shows an example in which the image generation unit 137 extracts a pixel range px_AR as pixels having depth values ​​indicating the position of the foreground.

[0055] First, the image generation unit 137 converts the pixel range px_AR into the original image oIM n ´, and the partial image pt_oIM corresponding to the pixel range px_AR n Then, the image generating unit 137 may extract the partial image pt_oIM. n For example, the image generating unit 137 may generate a partial image pt_oIM by virtually synthesizing an abnormal portion. n A virtual abnormality location v_AR that simulates the abnormality location may be superimposed on the image ´.

[0056] The learning unit 131 generates a partial image pt_oIM on which the virtual abnormality v_AR is superimposed. n ´ Teacher Image L_IM n It may be used as.

[0057] Until now, there have been few images of inspection targets with abnormalities, and there are only a sufficient number of training images L_IM required for learning model M1. n It is difficult to collect training images L_IM n However, the image synthesis by the image generation unit 137 allows the teacher image L_IM to be generated easily and quickly without manual intervention. n This allows for more efficient generation of training data, which in turn improves the performance of model M1.

[0058] [6. Example of operation of the detection device] Fig. 8 is a diagram (1) showing the procedure of information processing executed by the detection device 100. Fig. 8 shows a scene in which, as information processing according to the embodiment, an area AR of an abnormal area is detected limited to an area estimated to be a base station BS using the output result of a model M1 for an original image oIM1 (an example of a predetermined image in which an abnormal area is unknown) and the output result of a model M2 for the original image oIM1.

[0059] The original image oIM1 is a captured image obtained by photographing with the camera 21, and is assumed to be the latest captured image uploaded from the mobile device 20 to the detection device 100. The model M1 shown in Fig. 8 is learned by the learning unit 131 (first learning unit) according to the method described in Fig. 5. Furthermore, it is assumed that the depth value range RG has also been determined by the determination unit 132.

[0060] First, the segmentation unit 133 (first segmentation unit) inputs the original image oIM1 into the model M1, thereby executing segmentation of the region AR of the abnormal part in the original image oIM1 (step S81).

[0061] The model M1 segments the region AR of the abnormal area by inferring the region AR of the abnormal area (step S82). Then, the model M1 outputs the original image oIM1 in which the region AR of the abnormal area has been segmented (step S83).

[0062] Meanwhile, the generation unit 134 (first generation unit) inputs the original image oIM1 into the model M2 to estimate the depth information DP of the original image oIM1 (step S84). For example, the generation unit 134 estimates a depth value for each pixel included in the original image oIM1.

[0063] The model M2 estimates the depth value for each pixel by inferring the depth information DP (step S85), and then outputs the depth information DP including the depth value (step S86).

[0064] The generation unit 134 generates a depth image dp_oIM1 based on the depth values ​​included in the depth information DP (step S87). Note that the processes of steps S81 to S83 and the processes of steps S84 to S87 may be executed in parallel.

[0065] In this state, the acquisition unit 135 acquires an extracted image ex_IM1 based on a comparison between the segmented original image oIM1 and the depth image dp_oIM1 (step S88). For example, the acquisition unit 135 may identify a pixel region px_AR corresponding to the depth value range RG based on the depth information DP (depth value) included in the depth image dp_oIM1 and the depth value range RG. The pixel region px_AR is a region from which the foreground portion of the depth image dp_oIM1 is mainly extracted. As shown in FIG. 8, the pixel region px_AR includes the base station BS in the foreground but does not include the steel tower in the background.

[0066] The acquisition unit 135 may then acquire an extracted image ex_IM1 from the segmented original image oIM1, in which the region AR of the abnormal area has been extracted in association with the pixel region px_AR. Acquiring the extracted image ex_IM1 includes a process of leaving the segmentation of the portion extracted in the pixel region px_AR (the region AR of the abnormal area) for the segmented original image oIM1, and excluding the segmentation (the region AR of the abnormal area) existing outside the pixel region px_AR from the original image oIM1. On the other hand, acquiring the extracted image ex_IM1 may also be a process of cutting out the portion extracted in the pixel region px_AR from the segmented original image oIM1.

[0067] As another process, the acquisition unit 135 may cut out the portion extracted in the pixel region px_AR from the segmented original image oIM1, and then reflect the segmentation within the pixel region px_AR (the region AR of the abnormal area) in the cut-out extracted image ex_IM1.

[0068] When the extracted image ex_IM1 is acquired in this way, the detection unit 136 detects the area AR of the abnormal part included as a segment in the extracted image ex_IM1 as the abnormal part belonging to the base station BS (step S89).

[0069] According to the information processing shown in Figure 8, by comparing the original image oIM1 in which the area AR of the abnormal area is extracted by inputting the original image oIM1 into model M1 with the depth image dp_oIM1 generated based on inputting the original image oIM1 into model M2 (comparison based on the depth value range RG), it is possible to obtain an image in which the area AR of the abnormal area is reflected, limited to the area estimated to be the base station BS.Therefore, it is possible to obtain an image in which the area AR of the abnormal area is reflected, with priority given to the base station BS, thereby satisfying the need to inspect abnormal areas with priority given to the base station BS.

[0070] 7. Operational Example of the Detection Device (Modification) The detection device 100 may be implemented in a form different from the embodiment described in Fig. 8. Therefore, Fig. 9 and Fig. 10 describe a modified example of the detection device 100. In the description of the modified example, the learning unit 131 corresponds to a second learning unit, the segmentation unit 133 corresponds to a second segmentation unit, and the generation unit 134 corresponds to a second generation unit.

[0071] (Learning phase) Fig. 9 is a diagram (2-1) showing the procedure of information processing executed by the detection device 100. As a modified example of the information processing according to the embodiment, Fig. 9 shows a scene in which a model M1' is generated that estimates an area AR of an abnormal part by using depth information DP estimated by a model M2 as learning data. Also, Fig. 9 shows a scene in which an original image oIM n An example of this is the original image oIM3.

[0072] 9 shows a scene in which a model M1' is trained based on a training image L_IM3 consisting of an original image oIM3 (a color image having RGB information) captured by a base station BS and annotation information AN3 assigned to the original image oIM3 as a correct label. The original image oIM3 is an image of a base station BS having an abnormality (an example of an image of an inspection target), and may also include other objects besides the inspection target.

[0073] For example, the annotation information AN3 may be information in which, among the pixels of the original image oIM3, each pixel in the area indicating the base station BS is labeled with category information (Base Station) indicating the base station BS. n 9 shows a scene in which one teacher image L_IM3 is used for learning, but it may be information in which category information (Abnormality) indicating the abnormality is labeled for each pixel in the region indicating the abnormality among the pixels of the image L_IM3. n Multiple corresponding training images L_IM n may be used for learning.

[0074] In this state, the generation unit 134 (second generation unit) may input the original image oIM3 to the model M2 to estimate the depth information DP of the original image oIM3 (step S91). For example, the generation unit 134 may estimate a depth value for each pixel included in the original image oIM3.

[0075] The model M2 estimates the depth value for each pixel by inferring the depth information DP (step S92), and then outputs the depth information DP including the depth value (step S93).

[0076] The generation unit 134 generates a depth image dp_oIM3 based on the depth values ​​included in the depth information DP (step S94).

[0077] Here, the learning unit 131 (second learning unit) may generate an image that links (combines) the annotated original image oIM3 (teacher image L_IM3) and the depth image dp_oIM3 as a teacher image L_IM3', as shown in Figure 9 (step S95).

[0078] Although not shown in FIG. 9, the determining unit 132 may determine a range RG of depth values ​​between steps S91 and S95.

[0079] The learning unit 131 may input the teacher image L_IM3' into a model M (e.g., a pre-trained semantic segmentation model), input the teacher image L_IM3' into the model M (e.g., a pre-trained semantic segmentation model) so as to estimate the area of ​​the abnormality for the part of the input image that is estimated to include the inspection object, and train the model M1' to estimate the area AR of the abnormality for the area that is estimated to be a base station BS (step S96).

[0080] For example, the learning unit 131 may learn a model M1' to estimate a base station area indicating a base station BS for an input image inIM and estimate an area AR of an abnormality within the base station area, similar to the learning of the model M1 described in Fig. 5. That is, when the input image inIM is input, the learning unit 131 may generate a model M1' that segments an area AR of an abnormality that is estimated to be included in the base station BS and outputs the segmentation result. However, as described above, this method alone may result in the generation of a model M1' that erroneously recognizes a steel tower in the background other than the inspection target as the inspection target base station BS.

[0081] Therefore, the learning unit 131 performs learning that takes into account not only the shape but also the depth information DP by combining the depth image dp_oIM3, for which the depth value range RG has been determined, with the annotated original image oIM3 (teacher image L_IM3). For example, the learning unit 131 may train the model M1' to determine that the correct abnormality exists in the pixel area px_AR (the area where the foreground part is mainly extracted) corresponding to the depth value range RG and in the base station area indicated by the annotation information AN3. As a result, the learning unit 131 can generate a model M1' with improved accuracy so as not to erroneously recognize anything other than the inspection target as the base station BS to be inspected.

[0082] In this learning phase, the learning unit 131 also generates the partial image pt_oIM synthesized by the image generating unit 137. n ´ Teacher Image L_IM n For example, the learning unit 131 may use the partial image pt_oIM3′ synthesized by the image generating unit 137 as the teacher image L_IM3.

[0083] (Inference phase) Fig. 10 is a diagram (2-2) showing the procedure of information processing executed by the detection device 100. As a modified example of the information processing according to the embodiment, Fig. 10 shows a scene in which an area AR of an abnormality is detected by limiting the area to an area estimated to be a base station BS using the output result of a model M1' for an original image oIM1 (an example of a predetermined image in which an abnormality is unknown).

[0084] The original image oIM1 is a captured image obtained by photographing with the camera 21, and is assumed to be the latest captured image uploaded from the mobile device 20 to the detection device 100. The model M1′ shown in FIG. 10 is learned by the learning unit 131 (second learning unit) according to the method described in FIG. 9.

[0085] First, the segmentation unit 133 (second segmentation unit) inputs the original image oIM1 to a model M1', thereby executing segmentation of an area AR of an abnormal area in the original image oIM1 (step S1001). Specifically, as shown in Fig. 10, the segmentation unit 133 inputs a combined image C_oIM1 obtained by combining the original image oIM1 and the depth image dp_oIM1 to the model M1', thereby executing segmentation of an area AR of an abnormal area in the original image oIM1.

[0086] The reason why the combined image C_oIM1 is input to the model M1' is that, as explained in Figure 9, the model M1' uses the annotated original image oIM n and the depth image dp_oIM n The combined training image L_IM n This is because the model M1' has been learned using the original image oIM1. Note that, as a pre-stage process before the segmentation unit 133 inputs the data to the model M1', the generation unit 134 may generate a depth image dp_oIM1 based on the depth information DP estimated based on the original image oIM1 and the model M2.

[0087] 10, the model M1' segments the region AR of the abnormal area by inferring the region AR of the abnormal area (step S1002). Then, the model M1' outputs the original image oIM1 in which the region AR of the abnormal area has been segmented (step S1003).

[0088] In this state, the acquisition unit 135 acquires an extracted image ex_IM1 based on the output original image oIM1 (step S1004). According to the example in Fig. 10, the original image oIM1 output by the model M1' can be said to be an extracted image ex_IM1 in which the region AR of the abnormal part is extracted in association with the pixel region px_AR (the region in which the foreground part is mainly extracted) corresponding to the depth value range RG.

[0089] When the extracted image ex_IM1 is acquired in this way, the detection unit 136 detects the area AR of the abnormal part included as a segment in the extracted image ex_IM1 as the abnormal part belonging to the base station BS (step S1005).

[0090] 7, when model M1 is used in the inference phase, the acquisition unit 135 acquires the extracted image ex_IM1 by further combining model M2 in the inference phase. On the other hand, when model M1' is used, the acquisition unit 135 can acquire the extracted image ex_IM1 in the inference phase without using model M2. The former method is different from the former method in that the acquisition unit 135 can acquire the extracted image ex_IM1 in the learning phase without using model M2. n The latter method has the advantage of reducing the cost of learning because it does not require the use of depth images dp_oIM in the inference phase. n The advantage of this method is that it does not require the generation of a model M1, and the inference speed is faster than that of the former method. As such, since each method has its own advantages, for example, a worker can use model M1 or model M1' depending on the situation.

[0091] [8. Hardware Configuration] The detection device 100 according to the embodiment may be realized, for example, by a computer 1000 configured as shown in Fig. 11. Fig. 11 is a hardware configuration diagram showing an example of a computer that realizes the functions of the detection device 100 according to the embodiment. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, an HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

[0092] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 starts up, programs that depend on the hardware of the computer 1000, and the like.

[0093] The HDD 1400 stores programs executed by the CPU 1100, data used by these programs, etc. The communication interface 1500 receives data from other devices via a predetermined communication network and sends the data to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the predetermined communication network.

[0094] The CPU 1100 controls an output device such as a display and an input device such as a keyboard via the input / output interface 1600. The CPU 1100 acquires data from the input device via the input / output interface 1600. The CPU 1100 also outputs generated data to the output device via the input / output interface 1600.

[0095] Media interface 1700 reads a program or data stored in recording medium 1800 and provides it to CPU 1100 via RAM 1200. CPU 1100 loads the program or data from recording medium 1800 onto RAM 1200 via media interface 1700 and executes the loaded program. Recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0096] For example, when the computer 1000 functions as the detection device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes programs loaded onto the RAM 1200, thereby realizing the functions of the control unit 130. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from another device via a predetermined communication network.

[0097] [9. Other] Furthermore, among the processes described in each of the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0098] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0099] Furthermore, the above-described embodiments can be combined as appropriate within the scope of not causing any contradiction in the processing content.

[0100] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be implemented in other forms that include the aspects described in the "present invention" section and that have been modified and improved in various ways based on the knowledge of those skilled in the art. [Explanation of symbols]

[0101] 1. Information Processing Systems 10 User equipment 20 Mobile Devices 21 Camera 100 Detection device 130 Control Unit 131 Learning Section (1st Learning Section, 2nd Learning Section) 132 Decision Section 133 Segmentation section (first segmentation section, second segmentation section) 134 Generation unit (first generation unit, second generation unit) 135 Acquisition Department 136 Detector 137 Image Generation Unit

Claims

1. A first segmentation unit that generates segments indicating candidate areas of the abnormality in a specified image including an inspection target, based on a trained model that has been trained to estimate the area of ​​the abnormality using a teacher image in which the area of ​​the abnormality in the original image is annotated on a pixel-by-pixel basis, and a specified image including an inspection target; a determination unit that determines a depth value range for detecting the region of the abnormality based on a depth value estimated for the teacher image corresponding to the original image including the inspection object; an acquisition unit that acquires, as an extracted image, an image portion of the predetermined image that corresponds to the determined range of depth values; a detection unit that detects, from among the segments generated by the first segmentation unit, a segment included in the extracted image as the abnormal portion of the inspection object; A detection device comprising:

2. The determination unit determines the range of depth values ​​at the stage when the learning is performed. The detection device according to claim 1 .

3. the determination unit determines, as a minimum depth value, the distance in a region of the abnormality that is located at a shortest distance from an imaging point; the range of depth values ​​includes the minimum value as a threshold; The detection device according to claim 1 .

4. A detection method performed by a detection device, comprising: a first segmentation step of generating segments indicating candidate regions of the abnormality in a predetermined image including an inspection target, based on a trained model trained to estimate the region of the abnormality using a teacher image in which the region of the abnormality in the original image is annotated on a pixel-by-pixel basis, and the predetermined image including an inspection target; a determining step of determining a depth value range for detecting the region of the abnormality based on a depth value estimated for the teacher image corresponding to the original image including the inspection object; an acquisition step of acquiring an image portion of the predetermined image corresponding to the determined range of depth values ​​as an extracted image; a detection step of detecting, from among the segments generated by the first segmentation step, a segment included in the extracted image as the abnormal portion of the inspection object; A detection method comprising:

5. A first segmentation procedure for generating segments indicating candidate areas of the abnormality in a specified image including an inspection target, based on a trained model that has been trained to estimate the area of ​​the abnormality using a teacher image in which the area of ​​the abnormality in the original image is annotated on a pixel-by-pixel basis, and the specified image including an inspection target; a determination step of determining a depth value range for detecting the region of the abnormality based on a depth value estimated for the teacher image corresponding to the original image including the inspection object; an acquisition step of acquiring, as an extracted image, an image portion of the predetermined image that corresponds to the determined range of depth values; a detection step of detecting, from among the segments generated by the first segmentation step, a segment included in the extracted image as the abnormal portion of the inspection object; A detection program that causes a computer to run the following.

Images