Program, detection device, and detection method

Abstract

To provide a technique for efficiently detecting a detection target from a processing target image without requiring a model construction cost of a machine learning model and expert knowledge.SOLUTION: An aspect of the present disclosure relates to a program that causes one or more computers to execute acquiring an image, dividing the image into a plurality of regions, and detecting a region that satisfies a filtering condition.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to a program, a detection device, and a detection method. [Background technology]

[0002] With the recent advances in information and communication technology, tasks that were previously performed manually are now being replaced by computers (known as digital transformation (DX)). For example, manual inspection tasks are now being replaced by computers by using computers to perform object detection and anomaly detection on images of the target object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-004276 [Patent Document 2] Japanese Patent Publication No. 2020-010805 Summary of the Invention [Problem to be solved by the invention]

[0004] Deep learning, which has recently attracted attention, is known to be effective for such object detection and anomaly detection. However, generating such machine learning models requires a large amount of training data, which can increase the cost of building the model. Furthermore, building such machine learning models can require advanced expertise.

[0005] An object of the present disclosure is to provide a technology for efficiently detecting a detection target from an image to be processed without requiring specialized knowledge or the cost of building a machine learning model. [Means for solving the problem]

[0006] One aspect of the present disclosure relates to a program that causes one or more computers to acquire an image, divide the image into a plurality of regions, and detect regions that satisfy a filtering condition. [Effects of the Invention]

[0007] According to the present disclosure, it is possible to provide a technology for efficiently detecting a detection target from an image to be processed, without requiring the cost of building a machine learning model or specialized knowledge. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram illustrating an anomaly detection use case according to one embodiment of the present disclosure. [Figure 2] FIG. 2 is a schematic diagram illustrating a detection device according to one embodiment of the present disclosure. [Figure 3] FIG. 3 is a block diagram illustrating a hardware configuration of a detection device according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is a block diagram illustrating a functional configuration of a detection device according to an embodiment of the present disclosure. [Figure 5] FIG. 5 is a schematic diagram illustrating a detection process according to one embodiment of the present disclosure. [Figure 6] FIG. 6 is a diagram illustrating filtering based on descriptions of region content according to one embodiment of the present disclosure. [Figure 7] FIG. 7 is a diagram illustrating filtering based on edge strength according to one embodiment of the present disclosure. [Figure 8] FIG. 8 is a diagram illustrating a user interface screen according to one embodiment of the present disclosure. [Figure 9] FIG. 9 is a flowchart illustrating a detection process according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0010] In the following embodiments, a detection device is disclosed that detects a detection target (for example, a defect, an abnormality, etc.) related to an object or the like from an image obtained by capturing the object or the like.

[0011] [Summary of the Disclosure] A detection device 100 according to an embodiment described below uses an image of an object to detect detection targets such as defects or abnormalities in the object. For example, as shown in Fig. 1, the object may be a packaged product that can be heated in a microwave oven, and the detection device 100 may detect detection targets such as foreign matter (for example, dust, hair, etc.) attached to the package or damage to the package from a product image captured of the exterior of the packaged product.

[0012] Specifically, as shown in Fig. 2, when the detection device 100 acquires an image of an object, it divides the acquired image into multiple regions (e.g., segments, layers, clusters, etc.). The detection device 100 then filters the multiple regions according to filtering conditions set by the user, and outputs the regions that satisfy the filtering conditions as detection results. By dividing the image into multiple regions and checking the filtering conditions for each region, it is possible to identify the detection target using simple filtering conditions.

[0013] For example, the filtering conditions may relate to the edge strength or depth value of each region. Specifically, they may relate to whether the edge strength or depth value of the region is equal to or greater than a predetermined threshold. The detection device 100 may determine regions that satisfy the filtering conditions set by the user as detection candidates for defects or abnormalities, and determine regions that do not satisfy the filtering conditions as normal. Here, the edge strength of a region in the present invention represents the strength of a portion of an image acquired by the detection device 100 where color or brightness changes suddenly (i.e., an edge, border, or boundary in the image, in the vertical or horizontal direction, or both vertical and horizontal directions). For example, it is calculated by calculating the difference between the values ​​of adjacent pixels. Furthermore, the depth value of a region in the present invention is a quantified version of the depth in the image acquired by the detection device 100 and refers to processing performed on the image to obtain three-dimensional depth estimation. These edge strengths and depth values ​​are difficult to obtain directly using a machine learning model. Therefore, separately obtaining and utilizing them enables judgment from a perspective other than machine learning.

[0014] As another example, the filtering conditions may be defined as descriptions of the image content of each region. For example, to detect foreign objects in the packaged product shown in FIG. 1 , such filtering conditions may be text descriptions in natural language, such as "a foreign object different from the package design is captured on the surface" or "a hair-like foreign object is captured." For such text descriptions, the detection device 100 may perform filtering using a multimodal generative model that receives an image and the description as input and outputs a determination result as to whether the input image matches the description. Multimodal refers to the integration of data in different formats to perform advanced information processing.

[0015] In this way, the detection device 100 can detect areas that satisfy the filtering conditions set by the user, such as detection targets such as defects and abnormalities, from an image without generating a machine learning model itself.

[0016] Here, the detection device 100 may be realized by a computing device such as a server, a personal computer (PC), a smartphone, or a tablet, and may have a hardware configuration such as that shown in Fig. 3. That is, the detection device 100 includes a drive device 101, a storage device 102, a memory device 103, a processor 104, a user interface (UI) device 105, and a communication device 106, which are interconnected via a bus B.

[0017] The programs or instructions that realize the various functions and processes of the detection device 100 may be stored in a removable storage medium such as a CD-ROM (Compact Disk-Read Only Memory) or flash memory. When the storage medium is set in the drive device 101, the programs or instructions are installed from the storage medium to the storage device 102 or memory device 103 via the drive device 101. However, the programs or instructions do not necessarily have to be installed from the storage medium, and may be downloaded from any external device via a network or the like.

[0018] The storage device 102 is realized by a hard disk drive or the like, and stores installed programs or instructions as well as files, data, etc. used to execute the programs or instructions.

[0019] The memory device 103 is realized by a random access memory, a static memory, or the like, and when a program or instruction is activated, reads and stores the program, instruction, data, or the like from the storage device 102. The storage device 102, the memory device 103, and the removable storage medium may be collectively referred to as a non-transitory storage medium.

[0020] The processor 104 may be realized by one or more CPUs (Central Processing Units), GPUs (Graphics Processing Units), processing circuitry, etc., which may be composed of one or more processor cores, and performs various functions and processes of the detection device 100 in accordance with programs, instructions, data such as parameters required to execute the programs or instructions, etc. stored in the memory device 103.

[0021] The user interface (UI) device 105 may be composed of input devices such as a keyboard, mouse, camera, microphone, etc., output devices such as a display, speaker, headset, printer, etc., and input / output devices such as a touch panel, and realizes an interface between a user and the detection device 100. For example, a user operates the detection device 100 by operating a GUI (Graphical User Interface) displayed on a display or touch panel using a keyboard, mouse, etc.

[0022] The communication device 106 is realized by various communication circuits that execute wired and / or wireless communication processing with external devices, the Internet, a LAN (Local Area Network), a cellular network, or other communication networks.

[0023] However, the above-described hardware configuration is merely an example, and the detection device 100 according to the present disclosure may be realized by any other suitable hardware configuration.

[0024] [Detection device] Next, a detection device 100 according to an embodiment of the present disclosure will be described. Fig. 4 is a block diagram showing a functional configuration of the detection device 100 according to an embodiment of the present disclosure. As shown in Fig. 4, the detection device 100 includes an acquisition unit 110, a region division unit 120, and a filtering unit 130. For example, one or more functional units of the acquisition unit 110, the region division unit 120, and the filtering unit 130 may be realized by one or more processors 104 executing one or more programs or instructions.

[0025] The acquisition unit 110 acquires an image. For example, the acquisition unit 110 acquires a processing target image from a camera or a user device communicatively connected to the detection device 100, and provides the acquired processing target image to the region division unit 120. Here, the processing target image may be, for example, a captured image of an object including a detection target such as a defect or an abnormality.

[0026] Specifically, a camera for capturing images of products is installed on a product production line or the like, and images of each product passing through the production line are acquired by the camera and transmitted to the detection device 100. Alternatively, an image selected by a user from images stored in a user terminal may be transmitted to the detection device 100. Note that the images according to the present disclosure are not necessarily limited to still images, and may be moving images or videos including moving images and audio.

[0027] The region dividing unit 120 divides the image into a plurality of regions. Specifically, the region dividing unit 120 may divide the image into a plurality of regions (e.g., segments, layers, clusters, etc.) according to any known segmentation method such as Segment Anything. For example, as shown in FIG. 5 , the region dividing unit 120 divides the image according to any known segmentation method and obtains a region division result consisting of a plurality of region images. Then, the region dividing unit 120 provides the region division result to the filtering unit 130.

[0028] By dividing the image to be processed into multiple regions in this way and checking whether the filtering conditions described below are satisfied for each region, it is possible to simplify the filtering conditions for identifying the detection target.

[0029] The filtering unit 130 detects regions that satisfy the filtering conditions. Specifically, upon acquiring the region division results, the filtering unit 130 filters each region according to one or more filtering conditions specified by the user, and identifies the region to be detected.

[0030] Here, the filtering conditions may be specified by a description of the content of the region. The description of the content of each region image may be, for example, a text description written in natural language, such as "there is a hole on the surface, and the content is visible," "the content is the same color as the surface," "the outline of the hole is an ochre color with a certain width," and / or "there is a black crack around the hole," as shown in FIG.

[0031] In this case, the filtering unit 130 may input the target image, each region image, and text description into a multimodal model capable of processing images and text, such as CLIP, and obtain the degree of match between the image and the description from the image classification model. Alternatively, the filtering unit 130 may input prompt data, along with the target image, each region image, and text description, into a multimodal generative model, such as GPT 4v, and obtain the degree of match between the image and the description from the generative model. Here, the prompt data may be, for example, a text description such as, "Based on the image, mask image, and text, please calculate the degree of match between the mask image and the text on a scale of 0 to 1.0."

[0032] In this case, the filtering condition may be that the degree of match is equal to or greater than a threshold. If the acquired degree of match is equal to or greater than the threshold, the filtering unit 130 may determine that the region image satisfies the filtering condition and check the next filtering condition as a detection target candidate. On the other hand, if the acquired degree of match is less than the threshold, the filtering unit 130 determines that the region image does not satisfy the filtering condition and excludes the region image from the detection target candidates.

[0033] 6, for example, assume that the threshold for the degree of match between a region image and a description is set to 0.8. In this case, the degree of match for region image R2 is acquired as 0.9, and the filtering unit 130 determines region image R2 as a detection target candidate and checks the next filtering condition. On the other hand, the degree of match for region image R3 is acquired as 0.2, and the filtering unit 130 excludes region image R3 from the detection target candidates.

[0034] The filtering condition may also be related to the edge strength of the region. For example, the filtering condition may be that the average value of the edge strength is equal to or greater than a threshold. The filtering unit 130 calculates the average value of the edge strength in each region image, and if the calculated average value of the edge strength is equal to or greater than a threshold, the filtering unit 130 may determine that the region image satisfies the filtering condition and determine the region image as a detection target candidate. On the other hand, if the calculated average value of the edge strength is less than the threshold, the filtering unit 130 determines that the region image does not satisfy the filtering condition and excludes the region image from the detection target candidates.

[0035] For example, as shown in Fig. 7, it is assumed that the threshold value for the average value of edge strength is set to 100. If the average value of edge strength of region image R1 is calculated as 30, the filtering unit 130 excludes region image R1 from the detection target candidates. On the other hand, if the average values ​​of edge strength of region images R2 and R3 are calculated as 160 and 130, respectively, the filtering unit 130 determines region images R2 and R3 as detection target candidates and determines the next filtering condition.

[0036] The filtering condition may also be related to the depth value of the region. For example, the filtering condition may be that the average value of the depth values ​​is equal to or greater than a threshold. The filtering unit 130 calculates the average value of the depth values ​​for each region image. If the calculated average value of the depth values ​​is equal to or greater than the threshold, the filtering unit 130 may determine that the region image satisfies the filtering condition and check the next filtering condition as a detection target candidate. On the other hand, if the calculated average value of the depth values ​​is less than the threshold, the filtering unit 130 determines that the region image does not satisfy the filtering condition and excludes the region image from the detection target candidates.

[0037] In this way, the filtering unit 130 outputs, as a detection result, a region image that satisfies all of the filtering conditions, such as edge strength, depth value, and region content description, as shown in FIG. 5, for example.

[0038] The filtering unit 130 may provide the user device with a user interface screen for setting filtering conditions. For example, the user interface screen may be as shown in Fig. 8, in which an edge intensity map and a depth value map can be specified as filtering conditions, and thresholds for the edge intensity and depth value can be input. Also, a text editor in which a description of the region content can be input may be displayed, and a threshold for the degree of match between the region image and the description can be input.

[0039] In this way, the filtering unit 130 outputs the detection result for the selected image according to the filtering conditions input on the user interface screen.

[0040] The detection device 100 described above can detect areas that satisfy the filtering conditions set by the user, such as detection targets such as defects and abnormalities, from an image without generating a machine learning model itself.

[0041] [Detection process] Next, a detection process according to an embodiment of the present disclosure will be described. The detection process can be realized by the detection device 100, more specifically, by the processor 104 of the detection device 100 executing a program stored in the memory device 103. Fig. 9 is a flowchart showing the detection process according to an embodiment of the present disclosure.

[0042] 9, in step S101, the detection device 100 acquires an image. Specifically, the detection device 100 acquires an image of an object to be detected or the like.

[0043] In step S102, the detection device 100 divides the image into a plurality of regions. Specifically, the detection device 100 divides the image acquired in step S101 using any segmentation method, and acquires a region division result including a plurality of regions (e.g., segments, layers, clusters, etc.).

[0044] In step S103, the detection device 100 detects a region that satisfies the filtering conditions. Specifically, the detection device 100 performs filtering on each region image obtained as a result of region segmentation in step S102, according to one or more filtering conditions.

[0045] The filtering conditions are typically set by the user of the detection device 100 and may be, for example, a threshold value for the edge strength of each region image, a depth threshold value, or a degree of match with an input text-based description. However, the filtering conditions according to the present disclosure are not necessarily limited to these and may be any other condition for effectively detecting the detection target. Other conditions may include, for example, inputting a two-dimensional map from any image, such as an image with HSV color information, a thermographic image, or an infrared image, and using the two-dimensional map as the filtering condition. Furthermore, shape information (such as image feature point information or geometric information) may also be used as the filtering condition. Using two-dimensional maps or shape information as conditions enables highly accurate inspection and detection without the need for large amounts of training data or machine learning models.

[0046] Then, the detection device 100 determines whether each region image resulting from the region division satisfies the filtering conditions, and outputs the region image that satisfies all or a certain number or more of the filtering conditions set by the user as the detection result.

[0047] According to the above-described detection process, it is possible to detect areas that satisfy the filtering conditions set by the user, i.e., detection targets such as defects and abnormalities, from an image without generating a machine learning model on your own.

[0048] Although the examples of the present disclosure have been described in detail above, the present disclosure is not limited to the specific embodiments described above, and various modifications and variations are possible within the scope of the gist of the present disclosure as set forth in the claims. [Explanation of symbols]

[0049] 100 Detection device 110 Acquisition Department 120 Area division part 130 Filtering section

Claims

1. Acquiring an image; Dividing the image into a plurality of regions; Detecting regions that satisfy a filtering condition; A program that causes one or more computers to execute the following.

2. The program of claim 1 , wherein the filtering condition is defined by a description of the content of the region.

3. The program of claim 2 , wherein the detecting utilizes a generative model to output a detection result for the description of the region.

4. The program according to claim 1 , wherein the filtering condition relates to an edge strength of the region.

5. The computer-readable medium according to claim 1 , wherein the filtering condition relates to a depth value of the region.

6. The program according to claim 1 , wherein the detecting step provides a user interface screen for setting the filtering condition to a user device.

7. an acquisition unit that acquires an image; a region dividing unit that divides the image into a plurality of regions; a filtering unit that detects an area that satisfies a filtering condition; A detection device comprising:

8. Acquiring an image; Dividing the image into a plurality of regions; Detecting regions that satisfy a filtering condition; The detection method is executed by one or more computers.

Images