Inspection system, inspection method, and inspection device

Abstract

We collect training data related to the inspection of objects whose effects due to environmental changes have been reduced. [Solution] An inspection system comprising a terminal device that displays content including virtual objects and an inspection device that can communicate with the terminal device, wherein the terminal device 20 includes a display processing unit that superimposes content so as to follow real objects on an captured image, a judgment object selection unit, a region calculation unit that calculates an object judgment region, and a judgment result transmission unit that transmits judgment result information including a judgment result regarding the presence or absence of a judgment target, an captured image and a judgment region to the inspection device, and the inspection device includes a judgment result receiving unit that receives the judgment result information, a partial image extraction unit that extracts a partial image including the judgment region in the captured image based on the captured image and the judgment region included in the judgment result information, and a learning processing unit that trains a judgment model that detects a judgment target within the judgment region using the extracted partial image and the judgment result included in the judgment result information as training data.

Description

Technical Field

[0001] Embodiments of the present invention relate to a technique for inspecting an object.

Background Art

[0002] There is known a technique for determining the quality of an object to be determined on an imaging image of a manufactured product using artificial intelligence. As this type of technique, for a product image obtained by photographing a product, an inspection target part image is generated by cutting out an image of the inspection target part based on a setting file indicating the position and range of the inspection target part of the product, and the generated inspection target part image is stored in an image storage unit. A pass / fail determination process is performed by applying artificial intelligence to the inspection target part image of the pass / fail determination target object shown to be a pass / fail determination target by production instruction information describing the product specifications. Among the inspection target part images stored in the image storage unit, the inspection target part images related to products designated as learning target objects that are not pass / fail determination targets by the production instruction information are accumulated in the image storage unit as learning data for a learning model applied to artificial intelligence. A machine learning system is known (see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] One problem is to collect learning data related to the inspection of an object with reduced influence due to environmental changes.

[0005] The embodiments of the present invention have been made to solve the above-mentioned problems and aim to provide a technology that can collect learning data related to the inspection of objects with reduced influence from environmental changes. [Means for solving the problem]

[0006] To solve the above-mentioned problems, an embodiment of the present invention provides an inspection system comprising a terminal device that displays content including at least one virtual object, and an inspection device capable of communicating with the terminal device, wherein the terminal device includes a display processing unit that superimposes and displays the content so as to follow a real object on an captured image, a selection unit that selects a judgment object which is a virtual object corresponding to a judgment target in the real object, a region calculation unit that calculates a judgment region which is the area of ​​the selected judgment object on the captured image, and a judgment information transmission unit that transmits judgment result information to the inspection device, which includes a judgment result by the user of the terminal device regarding the presence or absence of the judgment target, the captured image, and the calculated judgment region, wherein the inspection device includes a judgment result receiving unit that receives the judgment result information, a partial image extraction unit that extracts a partial image including the judgment region in the captured image based on the captured image and the judgment region included in the received judgment result information, and a learning processing unit that trains a judgment model which is a machine learning model that detects the judgment target within the judgment region, using the extracted partial image and the judgment result included in the judgment result information as training data.

[0007] Furthermore, an embodiment of the present invention is an inspection device that can communicate with a terminal device that displays content including at least one virtual object superimposed on a real object on an captured image so as to follow it, and comprises: a determination result receiving unit that receives determination result information including a determination result by the user of the terminal device regarding the presence or absence of a determination target in the real object, the captured image, and a determination region which is the area of ​​a determination object which is a virtual object corresponding to the determination target on the captured image; a partial image extraction unit that extracts a partial image including the determination region in the captured image based on the captured image and the determination region included in the received determination result information; and a learning processing unit that trains a determination model which is a machine learning model that detects the determination target within the determination region, using the extracted partial image and the determination result included in the determination result information as training data. [Brief explanation of the drawing]

[0008] [Figure 1] This is a schematic diagram showing the configuration of the inspection system according to the first embodiment. [Figure 2] This is a block diagram showing the hardware configuration of a terminal device according to the first embodiment. [Figure 3] This is a block diagram showing the hardware configuration of the inspection device according to the first embodiment. [Figure 4] This is a block diagram showing the functional configuration of a terminal device according to the first embodiment. [Figure 5] This is a block diagram showing the functional configuration of the inspection device according to the first embodiment. [Figure 6] This is a schematic diagram showing content superimposed on a real object according to the first embodiment. [Figure 7] This is a flowchart showing the operation of the determination process according to the first embodiment. [Figure 8] This is a schematic diagram showing the determination region according to the first embodiment. [Figure 9] This is a diagram showing the determination information according to the first embodiment. [Figure 10] This figure shows the determination result information according to the first embodiment. [Figure 11] It is a flowchart showing the operation of the learning process according to the first embodiment. [Figure 12] It is a diagram showing learning information according to the first embodiment. [Figure 13] It is a flowchart showing the operation of the learning process according to the second embodiment. [Figure 14] It is a diagram showing a model selection screen. [Figure 15] It is a flowchart showing the operation of the learning process according to the third embodiment. [Figure 16] It is a flowchart showing the operation of the determination process according to the fourth embodiment. [Figure 17] It is a schematic diagram showing a partial image according to the fourth embodiment. [Figure 18] It is a schematic diagram showing the separation distance according to the fourth embodiment.

Modes for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0010] <First Embodiment> (Hardware Configuration) The hardware configuration of the content display system according to the first embodiment will be described. FIG. 1 is a schematic diagram showing the configuration of the inspection system according to the present embodiment. FIGS. 2 and 3 are block diagrams showing the hardware configurations of the terminal device and the inspection device, respectively.

[0011] As shown in FIG. 1, the inspection system 1 according to the present embodiment includes a terminal device 10 and an inspection device 20 that are connected to each other so as to be able to transmit and receive data. In the present embodiment, the terminal device 10 is a terminal device used by a user who performs work related to the inspection of a manufactured product, and is configured as a portable terminal device, specifically, a tablet-type terminal device. The inspection device 20 is a device that makes a determination on a determination target in a manufactured product, and is a device having a higher data processing ability than the terminal device 10. In the present embodiment, the inspection device 20 is configured as a PC (Personal Computer).

[0012] Note that the terminal device 10 may be configured as a smartphone or a head-mounted display, or may be configured as a device capable of performing display on a display or projection by a projector. Further, various functions executed by the inspection device 20 described later may be executed by the terminal device 10, or may be executed by a server device connected to the terminal device 10 via the Internet.

[0013] An actual object is an object to be inspected by a user, and is an actual object having at least one determination target that requires determination of the presence or absence and evaluation of the position. In the present embodiment, the determination target is a welding spot that is a spot-welded part, and it is determined whether or not the welding spot exists at a position where it should originally exist. Here, the welding spot is determined to be "normal" when it is at the original position defined in the design of the actual object, and is determined to be "abnormal" when it is not at the original position.

[0014] The terminal device 10 connects to a network via a wireless LAN (Local Area Network) installed at the work site where the actual object is placed, and communicates with the inspection device 20 via the network. Note that the communication between the terminal device 10 and the inspection device 20 may be performed via a wired connection using a USB (Univer Serial Bus) cable or the like.

[0015] As shown in Figure 2, the terminal device 10 includes, as hardware, a CPU (Central Processing Unit) 11, RAM (Random Access Memory) 12, a storage device 13, a touch panel display 14, a network interface 15, a camera 16, and a depth sensor 17. The CPU 11 and RAM 12 work together to execute various functions, and the storage device 13 stores various data used in the processing performed by these functions. The touch panel display 14 has a display and a touch sensor and is an input / output device located on the front side of the terminal device 10. The network interface 15 communicates wirelessly with the inspection device 20.

[0016] Camera 16 is an imaging device mounted on the back of the terminal device 10 that captures two-dimensional images and videos using visible light. Depth sensor 17 is also mounted on the back of the terminal device 10 and measures the distance from the terminal device 10 to surrounding objects by emitting a laser and receiving the reflected light using a photodetector. In this embodiment, depth sensor 17 uses a Time of Flight (ToF) method, which converts the delay time between the emitted light pulse and the received light pulse into distance, to measure the distance from the terminal device 10 to surrounding objects and create a three-dimensional point cloud. The imaging direction by camera 16 and the measurement direction by depth sensor 17 are assumed to be the same.

[0017] As shown in Figure 3, the inspection device 20 includes, as hardware, a CPU 21, RAM 22, storage device 23, input / output interface 24, and network interface 25. The CPU 21 and RAM 22 work together to perform various functions, and the storage device 23 stores various data used in the processing performed by these functions. The input / output interface 24 performs data input and output with input devices such as keyboards and output devices such as displays connected to the inspection device 20. The network interface 25 performs wired and wireless communication with other devices, including the terminal device 10.

[0018] (Functional configuration of terminal devices) The functional configuration of the terminal device according to the first embodiment will now be described. Figure 4 is a block diagram showing the functional configuration of the terminal device according to this embodiment.

[0019] As shown in Figure 4, the terminal device 10 includes, as functions, a display processing unit 101, an imaging processing unit 102, a selection unit 103, a region calculation unit 104, a judgment information transmission unit 105, a judgment result receiving unit 106, and a judgment result registration unit 107.

[0020] The display processing unit 101 overlays content onto the video captured by the camera 16 and displays it on the touch panel display 14. Here, the content is 3D data that mimics the real object, and examples of this type of data include CAD (Computer Aided Design) data, which is the design information of the real object. The 3D data as content includes a virtual object (hereinafter referred to as the judgment object) that corresponds to the real object to be judged. When the content is rotated and scaled in the virtual space so that the contour of the content and the contour of the real object on the touch panel display 14 substantially coincide, the display processing unit 101 continues to overlay the content onto the real object in order to maintain this contour agreement.

[0021] In this embodiment, content tracking of a real object is performed using VisionLib, an SDK (Software Development Kit) for object tracking. However, any known technique, such as those disclosed in Patent Document 2, may be used for superimposing and tracking content onto a real object. Examples of such techniques include calculating the position and orientation of the terminal device 10 using known methods such as SLAM (Simultaneous Localization and Mapping), SfM (Structure from Motion), and VSLAM (Visual SLAM), and superimposing and tracking content onto a real object based on markers attached to the real object or feature points of the real object.

[0022] The imaging processing unit 102 causes the camera 16 to capture an image based on the user's shooting instruction via the touch panel display 14. The selection unit 103 selects a determination object included in the content superimposed on the actual object on the captured image captured by the imaging processing unit 102 by the display processing unit 101, based on the user's instruction via the touch panel display 14.

[0023] The region calculation unit 104 calculates a determination region, which is a rectangular region on the captured image corresponding to the region surrounding the selected determination object displayed on the touch panel display 14. In this embodiment, the determination region is represented by the "determination object coordinate X" and "determination object coordinate Y," which are the x,y coordinate positions of the pixel at the upper left corner of the determination region; the "determination object width," which is the number of pixels in the x direction of the rectangle with the "determination object coordinate X" and "determination object coordinate Y" as the origin; and the "determination object height," which is the number of pixels in the y direction of the rectangle with the "determination object coordinate X" and "determination object coordinate Y" as the origin.

[0024] The judgment information transmission unit 105 transmits judgment information to the inspection device 20, which includes at least the captured image and information indicating the judgment area in the captured image, and also transmits judgment result information to the inspection device 20, which includes the judgment result made by the user in the terminal device 10. The judgment information and judgment result information will be described in detail later.

[0025] The judgment result receiving unit 106 receives the judgment result from the inspection device 20 based on the judgment information transmitted by the judgment information transmission unit 105. The judgment result registration unit 107 stores the judgment result received by the judgment result receiving unit 106 in the storage device 13.

[0026] (Functional configuration of the inspection device) The functional configuration of the inspection device according to the first embodiment will now be described. Figure 5 is a block diagram showing the functional configuration of the inspection device according to this embodiment.

[0027] As shown in Figure 5, the inspection device 20 includes, functionally, a judgment information receiving unit 201, a partial image extraction unit 202, at least one judgment model 203, a judgment processing unit 204, a judgment result transmission unit 205, a judgment result receiving unit 206, and a learning processing unit 207.

[0028] The judgment information receiving unit 201 receives judgment information transmitted by the judgment information transmitting unit 105 of the terminal device 10. The partial image extraction unit 202 extracts the area to be judged in the captured image as a partial image based on the judgment information received by the judgment information receiving unit 201.

[0029] The judgment model 203 is a machine learning model trained through supervised learning using a partial image containing most of the target for judgment as training data, so that it can detect the welding point to be judged from the input image. The judgment processing unit 204 determines the target for judgment as "normal" if the judgment model 203 detects the target for judgment from the partial image extracted by the partial image extraction unit 202, and as "abnormal" otherwise.

[0030] The judgment result transmission unit 205 transmits the judgment result from the judgment processing unit 204 to the terminal device 10. The judgment result receiving unit 206 receives the judgment result transmitted by the terminal device 10. The learning processing unit 207 performs additional learning on the judgment model 203 based on the judgment result received by the judgment result receiving unit 206.

[0031] (Content overlay display) The superimposed display of content according to the first embodiment will now be described. Figure 6 is a schematic diagram showing content superimposed on a real object according to this embodiment.

[0032] As shown in Figure 6, the display processing unit 101 of the terminal device 10 overlays content CN onto the real object RO captured by the camera 16 and displays it on the touch panel display 14. The real object RO has at least one determination target JRO_1, JRO_2, and the content CN, which is 3D data corresponding to this real object RO, has determination objects JVO_1, JVO_2, respectively, which correspond to determination targets JRO_1, JRO_2.

[0033] As described above, the content CN is generated based on CAD data, which is the design information of the actual object RO. Therefore, the objects to be judged, JRO_1 and JRO_2, are considered normal if they are located in the same position as the judgment objects JVO_1 and JVO_2, and abnormal otherwise. The display processing unit 101 displays the judgment objects JVO_1 and JVO_2 on the touch panel display 14 for the user to select. The display processing unit 101 also displays a user interface on the touch panel display 14 for the user to select whether the judgment object selected by the user is "normal" or "abnormal".

[0034] (Decision process) The determination process according to the first embodiment will now be described. Figure 7 is a flowchart showing the operation of the determination process according to this embodiment. Figure 8 is a schematic diagram showing the determination area according to this embodiment. Figures 9 and 10 show the determination information and determination result information according to this embodiment, respectively. It should be assumed that the content superposition process described above has been performed in advance prior to the operation shown in Figure 7. In Figure 7, the processing operation of the terminal device is shown to the left of the dotted line, and the processing operation of the inspection device is shown to the right of the dotted line.

[0035] As shown in Figure 7, first, the imaging processing unit 102 causes the camera 16 to capture an image of a real object based on the user's instructions (S101). After capturing, the selection unit 103 selects a determination object for the content superimposed on the captured image by the display processing unit 101 based on the user's instructions (S102). In this embodiment, for the sake of simplicity, it is assumed that only one determination object is selected.

[0036] Next, the region calculation unit 104 calculates the determination region of the determination object selected by the selection unit 103 (S103). As shown in Figure 8, the determination region JA is the region surrounding the selected determination object of the content superimposed on the actual object in the captured image IMG, which has a predetermined image width W0 and image height H0. When calculating the determination region JA, the region calculation unit 104 calculates the origin position P1 of the determination region in the captured image IMG (determination object coordinates X, determination object coordinates Y), the determination object width W1, and the determination object height H1.

[0037] Next, the judgment information transmission unit 105 transmits judgment information to the inspection device 20 (S104). As shown in Figure 9, the judgment information is a combination of "judgment object ID", "category", "imported image", "image width", "image height", "judgment object coordinate X", "judgment object coordinate Y", "judgment object width", and "judgment object height".

[0038] "Judgment Object ID" is an identifier that uniquely identifies the judgment object selected by the selection unit 103. "Category" indicates the type of judgment object selected by the selection unit 103. Note that judgment objects in content are pre-assigned to a type. "Captured Image" indicates the address path of the captured image included in the judgment information. "Image Width" and "Image Height" indicate the image width and image height of the captured image. "Judgment Object Coordinate X", "Judgment Object Coordinate Y", "Judgment Object Width", and "Judgment Object Height" indicate the judgment area on the captured image.

[0039] Next, the judgment information receiving unit 201 of the inspection device 20 receives the judgment information transmitted by the terminal device 10 (S105), and the partial image extraction unit 202 extracts the image within the judgment region of the captured image included in the received judgment information as a partial image (S106). Here, the partial image extraction unit 202 extracts the partial image from the captured image based on the "judgment object coordinate X", "judgment object coordinate Y", "judgment object width", and "judgment object height" in the judgment information.

[0040] After extracting the partial image, the determination processing unit 204 performs detection of the object to be determined using the determination model 203 (S107). Here, the determination model 203 performs detection of the type of component indicated in the "category" of the determination information within the partial image extracted by the partial image extraction unit 202. Next, the determination processing unit 204 determines whether or not the object to be determined has been detected from the partial image using the determination model 203 (S108).

[0041] If a target for judgment is detected (S108, YES), the judgment processing unit 204 determines whether the confidence level is equal to or greater than a predetermined threshold, which is the confidence threshold (S109). Here, the judgment processing unit 204 uses at least one of the evaluation metrics of a machine learning model, namely recall, precision, or F-score, or a metric calculated based on these, as the confidence level. In this embodiment, if partial images judged as "normal" are used as training data for the judgment model 203, recall may be used as the confidence level. Furthermore, as will be described in detail later, if both partial images judged as "normal" and partial images judged as "abnormal" are used as training data, at least one of precision or F-score may be used as the confidence level.

[0042] If the confidence level is equal to or greater than the confidence threshold (S109, YES), the determination processing unit 204 determines that the target of determination corresponding to the determination object indicated by the "determination object ID" is "normal" (S110), and the determination result transmission unit 205 transmits the determination result information, including this determination result, to the terminal device 10 (S112). As shown in Figure 10, the determination result information is obtained by further associating the determination result with the determination information received by the determination information receiving unit 201.

[0043] Next, the judgment result receiving unit 106 of the terminal device 10 receives the judgment result information transmitted by the inspection device 20 (S113), the judgment result registration unit 107 registers the received judgment result information (S114), and the judgment process is completed.

[0044] If no object to be judged is detected in step S108 (S108, NO), or if the confidence level is not equal to or greater than the confidence threshold in step S109 (S109, NO), the judgment processing unit 204 determines that the object to be judged corresponding to the judgment object indicated by the "judgment object ID" is "abnormal" (S111), and the judgment result transmission unit 205 transmits judgment result information including this judgment result to the terminal device 10 (S112).

[0045] Thus, the determination process can determine whether the object to be determined, corresponding to the determination object selected by the user, is "normal" or "abnormal" based on the determination model 203.

[0046] (Learning process) The operation of the learning process according to the first embodiment will now be described. Figure 11 is a flowchart showing the operation of the learning process according to this embodiment. Figure 12 is a diagram showing the learning information according to this embodiment. It should be assumed that the content overlay processing described above has been performed in advance prior to the operation shown in Figure 11. In Figure 11, the processing operation of the terminal device is shown to the left of the dotted line, and the processing operation of the inspection device is shown to the right of the dotted line.

[0047] As shown in Figure 11, in the terminal device 10, when the user selects a determination result for the determination target corresponding to the determination object (S201), the imaging processing unit 102 determines whether the determination result selected by the user is "normal" or not (S202).

[0048] If the judgment result is "normal" (S202, YES), the imaging processing unit 102 causes the camera 16 to capture an image of the actual object (S203), the region calculation unit 104 calculates the judgment region of the judgment object judged by the user in the content superimposed on the captured image (S204), and the judgment information transmission unit 105 transmits the judgment result information, including the user's judgment result, to the inspection device 20 (S205).

[0049] Next, the judgment result receiving unit 206 of the inspection device 20 receives the judgment result information transmitted by the terminal device 10 (S206), the partial image extraction unit 202 extracts a partial image based on the judgment result information (S207), and the learning processing unit 207 stores the learning information including the extracted partial image in the storage device 23 (S208).

[0050] As shown in Figure 12, the training information is a combination of "Judgment Object ID," "Category," "Partial Image," and "Judgment Result." The "Judgment Object ID" is a unique identifier that identifies the judgment object selected by the user. The "Category" indicates the type of judgment object selected by the user. The "Partial Image" indicates the address path of the partial image included in the training information. The "Judgment Result" indicates the judgment result for the target to be judged, corresponding to the judgment object selected by the user.

[0051] After storing the learning information, the learning processing unit 207 determines whether the amount of learning information stored in the storage device 23 is equal to or greater than a preset learning threshold (S209).

[0052] If the amount of training information is equal to or greater than the training threshold (S209, YES), the training processing unit 207 performs additional training on the judgment model 203 using the training information stored in the memory device 23 (S210), replaces the existing judgment model 203 with the newly trained judgment model 203 (S211), and the training process is completed. Alternatively, the training processing unit 207 may generate a new judgment model instead of performing additional training. When the training processing unit 207 trains or generates the judgment model 203, the "category" and "judgment result" from the training information are labeled to the partial images.

[0053] On the other hand, if the amount of training information is not equal to or greater than the training threshold (S209, NO), the training process is terminated.

[0054] Furthermore, if the judgment result in step S202 is not "normal" (S202, NO), the learning process is terminated.

[0055] Thus, the learning process collects training information, including partial images within the judgment area of ​​the object to be judged in the content, as training data, and trains a judgment model based on images in which the area occupied by the object to be judged is large. This reduces the elements that cause noise in machine learning compared to training a judgment model based on captured images without limiting the area, so that training data that can flexibly respond to changes in the environment related to the inspection work can be collected. Here, changes in the environment related to the inspection work include changes in the imaging position relative to the actual object, and the amount and angle of incidence of light illuminating the actual object.

[0056] <Second Embodiment> A second embodiment of the inspection system will now be described. The operation of the inspection device in the learning process in the second embodiment differs from that of the first embodiment. The operation of the inspection device in the learning process, which differs from that of the first embodiment, will be described below. Figure 13 is a flowchart showing the operation of the learning process according to this embodiment. Figure 14 is a diagram showing the model selection screen. Note that only the operation of the inspection device is shown in Figure 13, and operations similar to those in the learning process of the first embodiment are denoted by the same reference numerals as in Figure 11.

[0057] As shown in Figure 13, when the learning processing unit 207 stores the learning information, including the extracted partial images, in the storage device 23 (S208), it divides a portion of the stored learning information into evaluation information based on predetermined criteria (S301). Preferably, the learning processing unit 207 divides the learning information so that the total number of evaluation information is less than the total number of learning information, for example, by dividing the learning information into evaluation information with a probability of 10%.

[0058] After classifying the learning information, the learning processing unit 207 determines whether the number of learning information items stored in the storage device 23 is equal to or greater than a preset learning threshold (S302).

[0059] If the number of training information is equal to or greater than the training threshold (S302, YES), the training processing unit 207 performs additional training on the judgment model 203 using the training information stored in the memory device 23 (S303), and saves the additionally trained judgment model 203 as a judgment model candidate (S304). The training processing unit 207 may also generate a new judgment model as a judgment model candidate.

[0060] After saving the candidate models for judgment, the learning processing unit 207 determines whether the number of evaluation information is equal to or greater than a preset evaluation threshold (S305).

[0061] If the number of evaluation information items is equal to or greater than the evaluation threshold (S305, YES), the learning processing unit 207 uses each of the candidate judgment models to perform detection of the target to be judged on the partial image included in the evaluation information (S306), and determines for each of the candidate judgment models whether the detection rate of the target to be judged is equal to or greater than a preset detection threshold (S307).

[0062] If there is at least one candidate model for judgment whose detection rate is equal to or greater than the detection threshold (S307, YES), the learning processing unit 207 transmits candidate model information for judgment regarding the candidate model whose detection rate is equal to or greater than the detection threshold to the terminal device 10 (S308). The candidate model information for judgment associates the "model name," which is the name of the candidate model for judgment, the "creation date and time" of the candidate model for judgment, and the "evaluation," which indicates the detection rate of the candidate model for judgment. Upon receiving the candidate model information for judgment, the terminal device 10 displays the candidate model for judgment on the touch panel display 14 so that it can be selected, as shown in Figure 14.

[0063] When the user selects a candidate model for classification, the learning processing unit 207 replaces the existing classification model 203 with the selected candidate model, updates the classification model 203 (309), and the learning process is terminated.

[0064] Furthermore, if there are no candidate models for judgment with a detection rate equal to or greater than the detection threshold in step S307 (S307, NO), if the number of evaluation information is not equal to or greater than the evaluation threshold in step S305 (S305, NO), or if the number of training information is not equal to or greater than the training threshold in step S302 (S302, NO), the training process is terminated.

[0065] Thus, according to the learning process of this embodiment, the user can select a judgment model according to the situation. For example, in a situation where the environment related to the inspection has changed significantly, the user can select the judgment model with the most recent creation date, and in a situation where the environment has not changed much, the user can select a judgment model with a high evaluation.

[0066] <Third Embodiment> A third embodiment of the inspection system will now be described. The operation of the learning process in the third embodiment of the inspection system differs from that of the first embodiment of the inspection system. The operation of the learning process that differs from that of the first embodiment will be described below. Figure 15 is a flowchart of the operation of the learning process according to this embodiment. Note that in Figure 15, operations similar to those of the learning process in the first embodiment are denoted by the same reference numerals as in Figure 11.

[0067] As shown in Figure 15, the learning process according to this embodiment omits the operation of step S202 in the terminal device 10. This means that regardless of whether the judgment result selected by the user is "normal" or "abnormal", the judgment result information corresponding to this judgment result is transmitted to the inspection device 20, which is different from the learning process according to the first embodiment.

[0068] When the inspection device 20 receives this judgment result information (S206), and a partial image is extracted from this judgment result information (S207), the learning processing unit 207 determines whether the judgment result indicated by the judgment result information is "normal" or not (S401).

[0069] If the determination result is "normal" (S401, YES), the learning processing unit 207 stores the learning information in the storage device 23 in the same manner as the operation of step S208 of the learning process according to the first embodiment (S402).

[0070] On the other hand, if the judgment result is not "normal" (S401, NO), the learning processing unit 207 sets the category to "unknown" (S403) and stores the learning information for which the judgment result is "abnormal" in the storage device 23 (S402).

[0071] Thus, according to the learning process of this embodiment, both partial images from cases where the judgment result is "normal" and partial images from cases where the judgment result is "abnormal" can be used to train the judgment model.

[0072] <Fourth Embodiment> A description of the inspection system according to the fourth embodiment will be provided. The operation of the inspection device in the determination process of the inspection system according to the fourth embodiment differs from that of the inspection system according to the first embodiment. The operation of the inspection device in the determination process, which differs from that of the first embodiment, will be described below. Figure 16 is a flowchart showing the operation of the determination process according to this embodiment. Figures 17 and 18 are schematic diagrams showing a partial image and separation distance according to this embodiment, respectively. Note that only the operation of the inspection device is shown in Figure 16, and operations similar to the learning process according to the first embodiment are denoted by the same reference numerals as in Figure 7.

[0073] As shown in Figure 16, when the determination information transmitted from the terminal device 10 is received (S105), the partial image extraction unit 202 extracts an image within the extended region from the captured image as a partial image based on the determination information (S501). Here, the extended region is a region that includes the determination region and has a width W2 and height H2 that are greater than the width W3 and height H3 of the determination region, as shown in Figure 17. The extended region is set so that its center coincides with the center of the determination region, that is, the determination region is located in the center of the extended region. The determination processing unit 204 performs detection of the object to be determined using the determination model 203 on such a partial image (S502).

[0074] If a target for judgment is detected (S108, YES) and the confidence level is above a threshold (S109, YES), the judgment processing unit 204 calculates the separation distance between the detected target for judgment and the judgment object (S503) and determines whether the separation distance is above a preset distance threshold (504). As shown in Figure 18, this separation distance is the distance between the center position C3 of the partial image and the center position C4 of the detection area, which is the area surrounding the detected target for judgment. As described above, since the partial image is an image within an extended area where the judgment area is positioned in the center, the center position C3 is effectively the center position of the judgment area surrounding the judgment object.

[0075] If the separation distance is greater than or equal to the distance threshold (S504, YES), the determination processing unit 204 determines that the determination target corresponding to the determination object indicated by the "determination object ID" is "normal" (S505), and the determination result transmission unit 205 transmits the determination result information, including this determination result, to the terminal device 10 (S112).

[0076] On the other hand, if the separation distance is not greater than or equal to the distance threshold (S504, NO), if the confidence level in step S109 is not greater than or equal to the confidence threshold (S109, NO), or if the object to be judged is not detected in step S108 (S108, NO), the judgment processing unit 204 determines that the object to be judged corresponding to the judgment object indicated by the "judgment object ID" is "abnormal" (S506), and the judgment result transmission unit 205 transmits the judgment result information, including this judgment result, to the terminal device 10 (S112).

[0077] Thus, according to the determination process of this embodiment, even if the object to be determined is detected outside the determination area, it can be determined to be "normal" if the distance from the determination area is relatively short.

[0078] While embodiments of the invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications are permitted without departing from the spirit of the invention. These embodiments and their variations are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as described in the claims. [Explanation of symbols]

[0079] 1. Inspection System 10 Terminal devices 20 Inspection equipment 101 Display Processing Unit 103 Selection Section 104 Area calculation section 106 Judgment Information Transmission Unit 201 Receiving unit for judgment information 202 Partial image extraction section 203 Judgment Model 204 Determination Processing Unit 206 Judgment Result Receiving Unit 207 Learning Processing Unit

Claims

1. An inspection system comprising a terminal device that displays content including at least one virtual object, and an inspection device that can communicate with the terminal device, The aforementioned terminal device is A display processing unit that overlays and displays the content so as to follow the actual object on the captured image, A selection unit that selects a determination object which is a virtual object corresponding to the object to be determined in the actual object, A region calculation unit calculates a determination region which is the region of the selected determination object on the captured image, The system includes a determination information transmission unit that transmits determination result information to the inspection device, which includes the determination result by the user of the terminal device regarding the presence or absence of the object to be determined, the captured image, and the calculated determination region. The inspection device, A determination result receiving unit that receives the aforementioned determination result information, A partial image extraction unit extracts a partial image including the determination region in the captured image based on the captured image and the determination region included in the received determination result information. An inspection system comprising a learning processing unit that trains a judgment model, which is a machine learning model that detects the object to be judged within the judgment area, using the extracted partial image and the judgment result included in the judgment result information as training data.

2. The inspection system according to claim 1, characterized in that the learning processing unit divides the received judgment result information into learning or evaluation, trains the judgment model based on the judgment result information divided for learning, and evaluates the judgment result of the judgment model based on the judgment result information divided for evaluation.

3. The judgment information transmission unit further transmits judgment information, including the captured image and the calculated judgment region, to the inspection device. The inspection device, A determination information receiving unit that receives the transmitted determination information, The system further comprises a determination processing unit that determines the presence or absence of a determination target within the determination area using the determination model based on the determination information, The partial image extraction unit extracts a partial image from the captured image that includes the determination region, based on the captured image and the determination region included in the received determination information. The inspection system according to claim 1 or 2, characterized in that the determination processing unit determines the presence or absence of the object to be determined based on a partial image based on the determination information.

4. The partial image extraction unit extracts a partial image within an extended region that is larger in width and height than the determination region in the captured image, based on the captured image and the determination region included in the received determination information. The inspection system according to claim 3, characterized in that the determination processing unit determines that the object to be determined is present in the partial image when the object to be determined is detected in the partial image and the distance between the detected object to be determined and the determination region in the partial image is less than or equal to a predetermined threshold.

5. The inspection system according to claim 4, characterized in that the determination processing unit sets the distance between the center position of the detection area, which is the area surrounding the detected object to be determined, and the center position of the determination area.

6. The inspection system according to claim 1, characterized in that the learning processing unit performs additional training on the judgment model using the extracted partial image and the judgment result included in the judgment result information as training data.

7. The inspection system according to claim 1, characterized in that the learning processing unit generates the judgment model using the extracted partial image and the judgment result included in the judgment result information as training data.

8. An inspection method performed by a terminal device that displays content including at least one virtual object, and an inspection device that can communicate with the terminal device, The aforementioned terminal device The content is superimposed and displayed so as to follow the actual object on the captured image. Select a determination object which is a virtual object corresponding to the object to be determined in the aforementioned actual object, The determination region, which is the area of ​​the selected determination object on the captured image, is calculated. The determination result information, including the determination result by the user of the terminal device regarding the presence or absence of a determination target corresponding to the determination object, the captured image, and the calculated determination region, is transmitted to the inspection device. The aforementioned inspection device Upon receiving the aforementioned determination result information, Based on the captured image and the determination region included in the received determination result information, a partial image including the determination region in the captured image is extracted. An inspection method for training a judgment model, which is a machine learning model that detects the object to be judged within the judgment region, using the extracted partial image and the judgment result included in the judgment result information as training data.

9. An inspection device that can communicate with a terminal device that displays content including at least one virtual object superimposed on a real object on an captured image so as to follow it, A determination result receiving unit that receives determination result information including the determination result by the user of the terminal device regarding the presence or absence of a determination target in the actual object, the captured image, and the determination region which is the area of ​​the determination object, which is a virtual object corresponding to the determination target on the captured image. A partial image extraction unit extracts a partial image including the determination region in the captured image based on the captured image and the determination region included in the received determination result information. A learning processing unit trains a judgment model, which is a machine learning model that detects the object to be judged within the judgment region, using the extracted partial image and the judgment result included in the judgment result information as training data. An inspection device equipped with the following features.

Images