Information processing device, information processing method, and program

Abstract

Provided are an information processing device, an information processing method, and a program, whereby visibility of a detection object in an image can be improved, and display that can suppress oversight can be realized. An information processing device comprising a processor, wherein the processor acquires an image, executes region extraction processing for extracting a region of a detection object from the image, generates position information of the region from region information of the extracted region, switches between a first display mode for causing the region information to be displayed by a mode for visually requesting information for communicating the position of the region on a display screen on the basis of the position information, and a second display mode for causing the region information to be displayed by a different mode than the first display mode, in accordance with at least one size from among the size of the extracted region and a display size of the region displayed on a display screen, and causes the region extraction processing result to be displayed.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program, and particularly relates to an information processing technology applied to processing for displaying a processing result of image processing. [Background Art]

[0002] In non-destructive inspection of industrial products using X-ray transmission images and the like, defects such as air bubbles, foreign matter, and cracks occurring in products are detected by visual observation of images obtained by transmission-photographing products to be inspected, and non-defective / defective products are determined.

[0003] The image processing apparatus described in Patent Document 1 comprises: a radiation image acquisition unit that acquires a radiation image obtained by imaging an object to be inspected irradiated with radiation; a reference image storage unit that stores a reference image which is a radiation image of a normal object to be inspected captured under the same imaging conditions as the radiation image acquired by the radiation image acquisition unit; a difference value detection unit that detects a difference value of pixel values between corresponding pixels of the radiation image acquired by the radiation image acquisition unit and the reference image stored in the reference image storage unit; and a display control unit that causes a display unit to display a difference region between the radiation image and the reference image based on the detection result of the difference value detection unit, such that the positive or negative sign of the difference value in the difference region can be discriminated.

[0004] The defect inspection apparatus described in Patent Document 2 comprises: an image acquisition means that acquires a light-receiving image generated based on reflected light or transmitted light from an object to be inspected obtained by irradiating the object with light or radiation; an image processing means that calculates the position and features of a defect candidate of the object to be inspected from the light-receiving image; a storage means that stores the calculation result of the position and features of the defect candidate obtained by the image processing means, and a diagnosis result indicating whether the defect candidate is a defect corresponding to the calculation result; and a simulation means that analyzes the process of occurrence and growth of defects from the calculation result by the image processing means and the diagnosis result stored in the storage means, and simulates the growth prediction of the defect candidate. [Prior Art Documents] [Patent Documents]

[0005] [Patent Document 1] International Publication No. 2016 / 174926 [Patent Document 2] International Publication No. 2017 / 130550 [Overview of the project] [Problems that the invention aims to solve]

[0006] Defects visible in X-ray transmission images are often difficult to distinguish from the background visually due to their weak signal intensity, ambiguous boundaries, and minute size. In recent years, methods have been developed using artificial intelligence (AI) to extract regions of specific objects from images. It is expected that by utilizing such region-extraction type AI, it will be possible to extract minute and weak defects along with their size and shape characteristics.

[0007] However, if the information obtained through region extraction AI processing is simply displayed on a monitor, inspectors may overlook defects if the defect size is small or if the defect is displayed too small on the monitor.

[0008] This disclosure is made in view of these circumstances and aims to provide an information processing device, an information processing method, and a program that can improve the visibility of detection targets extracted from images and realize a display that can suppress oversights. [Means for solving the problem]

[0009] An information processing device according to one aspect of the present disclosure is an information processing device equipped with a processor, the processor acquires an image, performs a region extraction process to extract a region to be detected from the image, generates region position information from the region information of the extracted region, and displays the region extraction processing result by switching between a first display mode, which displays information indicating the position of the region on the display screen in a visually appealing manner based on the position information, according to at least one of the size of the extracted region and the display size of the region displayed on the display screen, and a second display mode, which displays the region information in a manner different from the first display mode.

[0010] According to this embodiment, the mode of displaying the processing results is switched according to at least one of the sizes of the extracted region and the display size of the region displayed on the display screen, and information indicating the position or region of the detected target in the image is displayed visually. acknowledge It is possible to display the information in an easy-to-understand manner. This helps to reduce the chances of overlooking the target of detection.

[0011] In other embodiments of the present disclosure, the processor may be configured to perform region extraction processing using a segmentation model that performs image segmentation.

[0012] In other embodiments of the present disclosure, the information processing device may be a learning model trained using machine learning to extract regions to be detected from an input image.

[0013] In other embodiments of the present disclosure, the position information may include information indicating the position of the centroid or the center of the circumscribing rectangle of the extracted region.

[0014] In other aspects of the information processing apparatus relating to the present disclosure, the first display mode may include a configuration that displays a rectangular or circular frame as information indicating the location of the extracted region.

[0015] In other embodiments of the present disclosure, the second display mode may include a segmentation mask display that fills in the extracted region.

[0016] In other embodiments of the present disclosure, the processor may be configured to display the extracted region in a first display mode when the size of the extracted region is smaller than a first reference size, and to display the extracted region in a second display mode when the size of the extracted region is larger than the first reference size.

[0017] In other embodiments of the present disclosure, the processor may be configured to display an area on a display screen in a first display mode when the display size is smaller than a second reference size, and to display an area in a second display mode when the display size is larger than the second reference size.

[0018] In other embodiments of the present disclosure, the information processing device may be configured to receive instructions for enlargement and reduction, change the display magnification of the display screen according to the received instructions, and switch between a first display mode and a second display mode according to the display magnification.

[0019] An information processing device according to other embodiments of this disclosure may further include an input device for receiving instructions for enlargement and reduction.

[0020] An information processing apparatus according to other aspects of this disclosure may further include a display device for displaying processing results.

[0021] In information processing devices relating to other aspects of this disclosure, the image may be an X-ray transmission image.

[0022] In information processing devices according to other embodiments of the present disclosure, the image may be an X-ray transmission image of a cast metal part, a forged metal part, or a welded metal part.

[0023] In an information processing apparatus according to another aspect of the present disclosure, the detection target may be a defect.

[0024] In the information processing apparatus according to another aspect of the present disclosure, the defect includes air bubbles, porosity, FMLD (Foreign material less dense) and FMMD (Foreign material more dense), and may be configured to include at least one of them.

[0025] An information processing method according to another aspect of the present disclosure is an information processing method executed by an information processing apparatus, comprising: acquiring an image; executing region extraction processing for extracting a region of a detection target from the image; generating position information of the region from region information of the extracted region; and switching between a first display mode, in which information notifying the position of the region on a display screen is displayed in a visually appealing manner based on the position information in accordance with at least one of a size of the extracted region and a display size of the region displayed on the display screen, and a second display mode, in which the region information is displayed in a mode different from the first display mode, to display a processing result of the region extraction.

[0026] An information processing method according to another aspect of the present disclosure is an information processing method executed by an information processing apparatus, comprising: acquiring region information of a detection target region in an image and position information of the region; and switching between a first display mode, in which information notifying the position of the region on a display screen is displayed in a visually appealing manner based on the position information in accordance with at least one of a size of the region and a display size of the region displayed on the display screen, and a second display mode, in which the region information is displayed in a mode different from the first display mode.

[0027] A program relating to another aspect of this disclosure provides a computer with the following functions: a function to acquire an image; a function to perform region extraction processing to extract a region to be detected from the image; a function to generate region location information from region information of the extracted region; and a function to display the region extraction processing result by switching between a first display mode, which displays information indicating the location of the region on the display screen in a visually appealing manner based on the location information, according to at least one of the sizes of the extracted region and the display size of the region displayed on the display screen, and a second display mode, which displays the region information in a manner different from the first display mode. [Effects of the Invention]

[0028] According to this disclosure, the region and / or location of the object to be detected within the image can be displayed in a manner that is easy to see, thereby reducing the likelihood of overlooking the object to be detected. [Brief explanation of the drawing]

[0029] [Figure 1] Figure 1 is a schematic diagram showing an example of the configuration of an image processing system according to an embodiment. [Figure 2] Figure 2 schematically shows an example of how an inspection image obtained after processing by an information processing device is displayed. [Figure 3] Figure 3 shows an example of an inspection image in which a minute defect was detected. [Figure 4] Figure 4 shows an example of an inspection image in which multiple defects were detected. [Figure 5] Figure 5 shows an example of a magnified view of a portion of the examination image from Figure 3. [Figure 6] Figure 6 is a block diagram showing an example of the hardware configuration of an information processing device according to the embodiment. [Figure 7] Figure 7 is a flowchart showing an example of operation 1 in the information processing device according to the embodiment. [Figure 8] Figure 8 is a flowchart showing example 2 of the operation of the information processing device according to the embodiment. [Figure 9]Figure 9 is a flowchart showing example 3 of the operation of the information processing device according to the embodiment. [Figure 10] Figure 10 is a flowchart showing example 4 of the operation of the information processing device according to the embodiment. [Figure 11] Figure 11 is a flowchart showing example 5 of the operation of the information processing device according to the embodiment. [Figure 12] Figure 12 is a block diagram showing an example of the configuration of the imaging system. [Modes for carrying out the invention]

[0030] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. In this specification, identical components are denoted by the same reference numerals, and redundant descriptions are omitted where appropriate.

[0031] Figure 1 is a functional block diagram schematically showing the functional configuration of an information processing device 10 according to an embodiment of this disclosure. Here, an example of an information processing device 10 applied to an industrial X-ray inspection system for non-destructive inspection of an object is shown. The information processing device 10 is a device that performs region extraction processing to extract defective areas from an image IM of an industrial product to be inspected, and displays the processing results on a display device 34.

[0032] The information processing device 10 can be realized by a combination of computer hardware and software. Software is synonymous with a program. The computer that functions as the information processing device 10 may be a workstation, a personal computer, a tablet terminal, or a server.

[0033] The information processing device 10 includes an image acquisition unit 12, a region extraction unit 14, a position information generation unit 16, a rectangular frame generation unit 17, a size determination unit 18, a display mode selection unit 20, and a display control unit 22. The information processing device 10 can also be connected to an input device 32 and a display device 34. "Connection" is not limited to wired connections; it may also be a wireless connection.

[0034] The image acquisition unit 12 receives an image IM to be processed and acquires the image IM. The image IM is, for example, an X-ray transmission image obtained by irradiating a metal part, which is the object to be inspected, with X-rays. The image acquisition unit 12 may be configured to include a communication interface for receiving image IMs from an external device such as an imaging device or an image management server, or it may be configured to include a media interface for reading image IMs from removable media such as a memory card. The image acquisition unit 12 may also be configured to include an image acquisition program for automatically acquiring image IMs from an external device. The image IM acquired via the image acquisition unit 12 is sent to the region extraction unit 14.

[0035] The region extraction unit 14 is an AI processing unit that performs region extraction processing on the image IM using the segmentation model SM to extract defect regions to be detected from the image IM. The segmentation model SM is a learning model trained using machine learning to perform the task of image segmentation, and for the input image IM, it classifies whether the entire image (all pixels) within the image IM is a defect region on a pixel-by-pixel basis and divides the image IM into regions.

[0036] Defects in cast or forged metal parts, or welded metal parts, include, for example, bubbles, porosity, FMLD (Foreign Material Less Dense), and FM. M At least one of the following is included: D (Foreign material more dense). FMLD is the inclusion of foreign material defects (low density) that appear as black in X-ray transmission images. FMMD is the inclusion of foreign material defects (high density) that appear as white in X-ray transmission images.

[0037] The segmentation model SM may be a two-class classification detection model that determines whether each pixel is a defective region or not, or it may be a multi-class classification detection model that determines what type of defect each pixel has. The segmentation model SM is constructed using, for example, a convolutional neural network (CNN) that has convolutional layers. A fully convolutional network (FCN), a type of CNN, is one of the suitable models for the task of image segmentation. As the segmentation model SM in this example, for example, a neural network with a network structure called "U-net," which is a type of FCN, can be applied.

[0038] The segmentation model SM optimizes its parameters through machine learning using a training dataset that includes numerous training data sets where training images are associated with ground truth data for those images. Here, ground truth data refers to data that indicates the regions of defects present in the image, and may be, for example, a mask image with the regions of defects filled in.

[0039] The segmentation model SM generates a score for the received image IM that indicates the likelihood of classification for each pixel within the image IM, i.e., the likelihood of defects. The region extraction unit 14 includes a segmentation mask generation unit 15 that generates a segmentation mask based on the score generated by the segmentation model SM.

[0040] A segmentation mask is a mask image that fills in the defect regions within an image IM, representing the shape of the extracted defects on a pixel-by-pixel basis. The segmentation mask generation unit 15 generates a segmentation mask by using a threshold to binarize the pixel values ​​based on the defect likelihood score for each pixel obtained by the segmentation model SM, and by labeling clusters of pixels (connected regions) where defects appear to be linked as regions of the same defect. It is possible that multiple defect regions may be extracted from a single image IM. In that case, a segmentation mask is generated for each of the extracted defect regions.

[0041] In Figure 1, the segmentation model SM and the segmentation mask generation unit 15 are shown separately, but the segmentation mask generation unit 15 may be incorporated into the segmentation model SM. The segmentation mask or defect label data obtained by image segmentation in the region extraction unit 14 can become region information of defects extracted from the image IM.

[0042] The position information generation unit 16 generates position information indicating the location (detection position) of a defective region based on the defect region information obtained by the processing of the region extraction unit 14. If multiple defective regions are extracted from the image IM, position information is generated for each of the multiple defective regions. The position information may be, for example, image coordinates indicating the position of the centroid of the defective region, or image coordinates indicating the position of the center of the circumscribing rectangle of the defective region. Generating position information from region information can also be understood as converting region information into position information.

[0043] The rectangular frame generation unit 17 generates a rectangular frame surrounding the detected defect location indicated by the location information generated by the location information generation unit 16. For example, the rectangular frame generation unit 17 generates a rectangular frame centered on the centroid coordinates of the extracted defect region. Note that "rectangle" includes squares. This rectangular frame is displayed on the display screen of the display device 34 as information that visually indicates the detected defect location. From the viewpoint of suppressing overlooking detected defects, it is desirable that the rectangular frame be a rectangular size that is easily visible to the naked eye on the display screen. The size of the rectangular frame may be a predetermined fixed size.

[0044] In the information processing device 10, the size determination unit 18 determines the size of the defect, and depending on the determination result, the display mode is switched between a mode in which a rectangular frame is displayed and a mode in which a segmentation mask is displayed without displaying the rectangular frame.

[0045] The position information generation unit 16 may generate position information for all defective regions extracted by the region extraction unit 14, or it may generate position information based on the determination result of the size determination unit 18 when it is necessary to display a rectangular frame. Similarly, the rectangular frame generation unit 17 may generate a rectangular frame for all defective regions extracted by the region extraction unit 14, or it may generate a rectangular frame based on the determination result of the size determination unit 18 when it is necessary to display a rectangular frame.

[0046] The size of the defect determined by the size determination unit 18 may include a detected size representing the size of the defect region extracted by the region extraction unit 14, and a displayed size representing the size of the defect region displayed on the display screen of the display device 34. The size determination unit 18 includes a detected size determination unit 24 and a displayed size determination unit 25.

[0047] The detection size determination unit 24 determines the size (detection size) of the defective region extracted by the region extraction unit 14. The detection size determination unit 24 can determine the area of ​​the defective region by, for example, counting the number of pixels in each extracted defective region. The detection size may be expressed as the pixel count value of the defective region, or as the area obtained by multiplying the count value by the area of ​​one pixel.

[0048] The detection size determination unit 24 compares the detected size with the first reference size Th1 and provides the comparison result to the display mode selection unit 20. If the detected size is smaller than the first reference size Th1, it is determined that there is a high possibility of overlooking something.

[0049] The display size determination unit 25 determines the size (display size) of the defect area displayed on the display screen of the display device 34. The actual visibility of the defect area on the display screen may differ depending on the display conditions, such as the specifications of the display device 34, including the screen size and resolution of the display device 34, and the display magnification specified by the input device 32. The display size determination unit 25 may acquire information regarding display conditions, including specifications (display condition information), from the display device 34, or it may accept information input from the input device 32.

[0050] The display size determination unit 25 evaluates the display size of the defect area on the display screen based on the defect area information and display conditions, and provides the display mode selection unit 20 with the result of comparing the display size with the second reference size Th2. If the display size is smaller than the second reference size Th2, it is determined that there is a high possibility of overlooking the defect.

[0051] The display mode selection unit 20 performs a process to select a display mode for the region extraction processing result by the region extraction unit 14, based on the determination result of the size determination unit 18. That is, the display mode selection unit 20 performs a process to switch between a first display mode that displays a rectangular frame and a second display mode that displays a segmentation mask, according to the determination result of the size determination unit 18. Selecting a display mode can also be understood as determining a display mode. The display mode is changed when a different display mode is selected.

[0052] The first display mode is sometimes called the rectangular frame method, and the second display mode is sometimes called the coloring method. The coloring method may also be called the fill method or the segmentation mask method. Note that the rectangular frame method is not limited to the method of displaying a rectangular frame without displaying the segmentation mask, but may also include the method of displaying a rectangular frame along with the segmentation mask. On the other hand, the coloring method is a display method that displays the segmentation mask without displaying the rectangular frame. Switching between the rectangular frame method and the coloring method includes the concept of switching whether or not to display the rectangular frame.

[0053] The display control unit 22 generates display data necessary for displaying the processing result according to the selection result by the display mode selection unit 20, and performs display control on the display device 34. If the resolution (recording resolution) of the image IM is greater than the screen resolution (monitor resolution) of the display device 34, that is, if the resolution of the image IM is higher than the resolution of the display device 34, the data of the image IM is displayed in a reduced format when the image IM is displayed on the display device 34. Then, if necessary, it accepts instructions for enlargement or reduction, and the display magnification is changed according to the instruction to perform enlargement or reduction.

[0054] The display control unit 22 includes a display magnification control unit 28. The display magnification control unit 28 performs display enlargement or reduction processing according to instructions received via the input device 32. For example, if the reference display magnification of the display device 34 is 100%, the display magnification control unit 28 can change the display magnification within a range of 10% to 500% according to the specification from the input device 32. Note that the configuration is not limited to one in which the display magnification can be changed; a configuration in which the display magnification is fixed is also possible. In that case, the display magnification control unit 28 and processing units such as the display size determination unit 25 may be omitted. In another configuration, regardless of whether or not there is a display magnification change function, the detection size determination unit 24 may be omitted, and the display size determination unit 25 may determine the defect size on the display screen.

[0055] The input device 32 is comprised of, for example, a keyboard, mouse, multi-touch panel, or other pointing device, or an audio input device, or an appropriate combination thereof. The display device 34 is comprised of, for example, a liquid crystal display, an organic electro-luminescence (OEL) display, or a projector, or an appropriate combination thereof. The input device 32 and the display device 34 may be integrated, such as a touch panel. The input device 32 and the display device 34 may also be the input device and display device of a terminal device connected to the information processing device 10 via a communication line.

[0056] [Example of display of examination images] Figures 2 to 5 schematically show examples of display of inspection images obtained after processing by the information processing device 10. The inspection image IMG1 shown in Figure 2 shows a metal part 50, which is the object to be inspected. The metal part 50 is formed, for example, by casting or forging, and has a first part portion 51 with a relatively thin wall thickness and a second part portion 52 with a thicker wall thickness than the first part portion. In the inspection image IMG1, the area outside the metal part 50 is the area of ​​the background 54.

[0057] In the inspection image IMG1, a defect is detected in the second component portion 52 of the metal part 50, and a segmentation mask showing the region information of the detected defect region DA1 is displayed. If the extracted defect region DA1 is larger than the first reference size, and the segmentation mask is displayed at a size that is sufficiently visible on the display screen of the display device 34, a segmentation mask with the defect region DA1 filled in is displayed, as shown in Figure 2.

[0058] Furthermore, in order to improve the visibility of the segmentation mask on the display screen, it is preferable that the color used for filling is a chromatic color that contributes to visual differentiation from the surrounding non-defective areas (areas other than defects). In addition, at least one of the hue, lightness, and saturation of the segmentation mask may be different depending on whether a defect is detected in the first part portion 51 or in the second part portion 52. The segmentation mask may be displayed as a blinking display (intermittent display).

[0059] Figure 3 shows an example of the display of inspection image IMG2 in which a minute defect has been detected. If the detected defect region DA2 is smaller than the first reference size Th1, and even if its segmentation mask is displayed, it becomes too small to be easily seen on the display screen of the display device 34, then a rectangular frame RF2 is displayed to visually indicate the detected location of the defect region DA2, as shown in Figure 3. For example, if the display size on the display screen becomes a few pixels or less, the rectangular frame RF2 is displayed. The display of the rectangular frame RF2 may also be a blinking display.

[0060] Figure 4 shows an example of the display of inspection image IMG3 in which multiple defects were detected. In inspection image IMG3 shown in Figure 4, segmentation mask display and rectangular frame display are mixed, corresponding to the size of the detected defects. For the defect region DA1 extracted in inspection image IMG3, a segmentation mask is displayed as in Figure 2, and for the minute defect regions DA2 and DA3, which are likely to be overlooked, rectangular frames RF2 and RF3 are displayed as in Figure 3. Note that the display of rectangular frames RF2 and RF3 may be set to blink, and the display of the segmentation mask may be set to always be displayed (non-blinking).

[0061] Figure 5 shows an example of the display of inspection image IMG4, which is an enlarged view of a portion of inspection image IMG2 in Figure 3. When the display size of the defect region DA2 becomes larger than the second reference size Th2 due to the enlargement, the display switches from the rectangular frame RF2 in Figure 3 to a coloring method as shown in Figure 5, and a segmentation mask indicating the region information of the defect region DA2 is displayed.

[0062] Although not shown in the diagram, if a command to reduce the display magnification is received from the display state in Figure 2 or Figure 5, and the display size of the defect areas DA1 and DA2 becomes smaller than the second reference size, the display switches from the coloring method to the rectangular frame method. When the display magnification is changed and enlargement or reduction is performed, the size of the rectangular frame displayed on the display screen may be fixed.

[0063] Figures 2 to 5 illustrate inspection images of metal parts 50 formed by casting or forging, but the same procedure applies when detecting welding defects from X-ray transmission images of welded metal parts.

[0064] 《Example of the configuration of the information processing device 10》 Figure 6 is a block diagram showing an example of the hardware configuration of an information processing device 10 according to an embodiment. The information processing device 10 includes a processor 102, a computer-readable medium 104 which is a non-temporary tangible object, a communication interface 106, and an input / output interface 108.

[0065] The processor 102 includes a CPU (Central Processing Unit). The processor 102 may also include a GPU (Graphics Processing Unit). The processor 102 is connected to a computer-readable medium 104, a communication interface 106, and an input / output interface 108 via a bus 110.

[0066] The input device 32 and the display device 34 are connected to the bus 110 via the input / output interface 108.

[0067] The computer-readable medium 104 includes memory, which is the main memory, and storage, which is the auxiliary memory. The computer-readable medium 104 may be, for example, semiconductor memory, a hard disk drive (HDD), or a solid state drive (SSD), or a combination of these.

[0068] The computer-readable medium 104 stores various programs and data, including a region extraction program 114, a position information generation program 116, a rectangular frame generation program 117, a size determination program 118, a display mode selection program 120, and a display control program 122. The term "program" includes the concept of a program module. The region extraction program 114 includes a segmentation model SM and a segmentation mask generation program 115.

[0069] The region extraction program 114 is a program that enables the processor 102 to function as a region extraction unit 14. Similarly, the position information generation program 116, the rectangular frame generation program 117, the size determination program 118, the display mode selection program 120, and the display control program 122 are programs that enable the processor 102 to function as the position information generation unit 16, the rectangular frame generation unit 17, the size determination unit 18, the display mode selection unit 20, and the display control unit 22, respectively.

[0070] Example 1 of an information processing method executed by the information processing device 10 Figure 7 is a flowchart illustrating example 1 of the operation of the information processing device 10 according to the embodiment. The flowchart in Figure 7 can be applied, for example, to a device configuration where the display magnification is fixed or when the display magnification is set to 100%.

[0071] In step S11, the processor 102 acquires the image to be processed. In step S12, the processor 102 performs a region extraction process on the acquired image, which extracts defective regions using the segmentation model SM.

[0072] Next, in step S13, the processor 102 generates location information from the defect region information obtained by the region extraction process.

[0073] In step S14, the processor 102 determines whether the size of the defect (detected size) obtained from the region information is smaller than the first reference size Th1. If the result of the determination in step S14 is Yes, that is, if the detected size is smaller than the first reference size Th1, the processor 102 proceeds to step S16.

[0074] In step S16, the processor 102 adopts a rectangular frame method and displays a rectangular frame indicating the location of the defect (detection location) based on the position information.

[0075] On the other hand, if the result of step S14 is a No determination, that is, if the detected size is greater than the first reference size Th1 or if the detected size is equal to the first reference size Th1, the processor 102 proceeds to step S18. Note that in step S14, if the detected size is equal to the first reference size Th1, it is also possible to proceed to step S16 instead of step S18.

[0076] In step S18, the processor 102 employs a coloring method and displays the segmentation mask without displaying the rectangular frame.

[0077] If multiple defective regions are detected within the image, steps S14 to S18 are performed for each defective region.

[0078] After step S16 or step S18, the processor 102 terminates the flowchart shown in Figure 7.

[0079] Example 2 of an information processing method executed by the information processing device 10. Figure 8 is a flowchart illustrating example 2 of the operation of the information processing device 10 according to the embodiment. In the flowchart of Figure 8, steps common to Figure 7 are given the same step numbers, and redundant explanations are omitted. The flowchart of Figure 8 may be used instead of the flowchart of Figure 7. The differences between Figure 8 and Figure 7 will be explained below.

[0080] In the flowchart of Figure 8, the process in step S13 of Figure 7 is performed as step S15 after the Yes determination in step S14. That is, after step S12 in Figure 8, the processor 102 proceeds to step S14 and determines the detection size. If the determination result in step S14 is a Yes determination, the processor 102 proceeds to step S15.

[0081] In step S15, the processor 102 generates position information from the defect region information obtained by the region extraction process. After step S15, the processor 102 proceeds to step S16 and displays a rectangular frame.

[0082] On the other hand, if the result of step S14 is a No judgment, the processor 102 proceeds to step S18 and displays the segmentation mask.

[0083] After step S16 or step S18, the processor 102 terminates the flowchart shown in Figure 8.

[0084] 《Example 3 of an information processing method executed by the information processing device 10》 Figure 9 is a flowchart showing example 3 of the operation of the information processing device 10 according to the embodiment. figure The flowchart in Figure 9 can be applied, for example, to a configuration that does not include the detection size determination unit 24. In the flowchart of Figure 9, steps common to Figure 7 are given the same step numbers, and redundant explanations are omitted. The differences between Figure 9 and Figure 7 will be explained below.

[0085] In the flowchart of Figure 9, steps S22 to S28 are included instead of steps S14 to S18 in Figure 7.

[0086] After step S13, the processor 102 proceeds to step S22. In step S22, the processor 102 acquires the display conditions. The specifications of the display device 34 among the display conditions may be stored in advance in the computer-readable medium 104.

[0087] After step S22, in step S24, the processor 102 calculates the display size of the defect based on the extracted defect region information and display conditions, and determines whether the display size is smaller than the second reference size Th2.

[0088] If the result of the determination in step S24 is a Yes determination, that is, if the display size is smaller than the second reference size Th2, the processor 102 proceeds to step S26. Step S26 is the same process as step S16 in Figure 7.

[0089] On the other hand, if the result of the determination in step S24 is No, that is, if the display size is larger than the second reference size Th2 or if the display size is equal to the second reference size Th2, the processor 102 proceeds to step S28. Step S28 is the same process as step S18 in Figure 7. Note that in step S24, if the display size is equal to the second reference size Th2, it is also possible to proceed to step S26 instead of step S28.

[0090] If multiple defective regions are detected within the image, steps S24 to S28 are performed for each defective region.

[0091] After step S26 or step S28, the processor 102 terminates the flowchart in Figure 9. Note that, similar to Figure 8, the processing in step S13 may also be performed after the Yes determination in step S24 (between step S24 and step S26).

[0092] 《Example 4 of an information processing method executed by the information processing device 10》 Figure 10 is a flowchart of example 4 of the operation of the information processing device 10 according to the embodiment. The flowchart in Figure 10 is an example of controlling the display mode by utilizing both the detection size determination unit 24 and the display size determination unit 25. In the flowchart of Figure 10, 7th grade Steps common to both Figure 10 and Figure 9 are given the same step number, and redundant explanations are omitted. The differences between Figure 10 and Figure 7 are explained below.

[0093] In the flowchart of Figure 10, steps S22, S24, and S28 are included instead of step S18 in Figure 7, and step S26 is included instead of step S16 in Figure 7. That is, if the result of the determination in step S14 is a No determination, the processor 102 proceeds to step S22 and obtains the display condition.

[0094] Next, if the result of step S24 is a Yes, the processor 102 proceeds to step S26 and displays the rectangular frame. On the other hand, if the result of step S24 is a No, the processor 102 proceeds to step S28 and displays the segmentation mask.

[0095] If multiple defective regions are detected within the image, steps S14 to S28 are performed for each defective region.

[0096] After step S26 or step S28, the processor 102 terminates the flowchart in Figure 10. Note that, similar to Figure 8, the processing in step S13 may also be performed after the Yes determination in step S14 and after the Yes determination in step S24.

[0097] 《Example 5 of an information processing method executed by the information processing device 10》 Figure 11 is a flowchart of example 5 of the operation of the information processing device 10 according to the embodiment. The flowchart in Figure 11 is an example of control that switches between the rectangular frame method and the coloring method in conjunction with the zoom in or out operation. The flowchart in Figure 11 is executed after any of the flowcharts in Figures 7 to 10 have been executed.

[0098] In step S31, the processor 102 receives instructions regarding the display. The user can input instructions from the input device 32, such as instructions to change the display magnification or instructions to end the display.

[0099] In step S32, the processor 102 determines whether or not it has received an instruction to change the display magnification. If the user performs an operation to enlarge or reduce the display, and the result of the determination in step S32 is Yes, the processor 102 proceeds to step S34. Steps S34, S36, and S38 are equivalent to steps S24, S26, and S28 described in Figure 9.

[0100] In step S34, the processor 102 determines the display size of the defect on the display screen using the specified display magnification. If the display size is smaller than the second reference size Th2 and the result of the determination in step S34 is Yes, the processor 102 proceeds to step S36 and displays a rectangular frame. On the other hand, if the result of the determination in step S34 is No, that is, if the display size is larger than the second reference size Th2 or if the display size is equal to the second reference size Th2, the processor 102 proceeds to step S38 and displays a segmentation mask.

[0101] After step S36 or step S38, the processor 102 proceeds to step S39. If multiple defective regions are detected in the image, the processing in steps S34 to S38 is performed for each defective region. Also, if the determination result in step S32 is No, the processor 102 proceeds to step S39.

[0102] In step S39, the processor 102 determines whether to terminate the display. If the result of the determination in step S39 is No, the processor 102 returns to step S31. On the other hand, if the result of the determination in step S39 is Yes, the processor 102 terminates the flowchart in Figure 11.

[0103] Although not shown in Figure 11, if the processor 102 receives an instruction other than changing the display magnification or ending the display, it may execute the processing corresponding to the received instruction.

[0104] Examples of display methods other than rectangular frames In this embodiment, an example was described in which a rectangular frame is displayed as information indicating the location of a defect. However, the method of visually displaying information indicating the detection location is not limited to a rectangular frame. For example, instead of a rectangular frame, a circular frame, another polygonal frame, or a closed curve may be used. Furthermore, the information indicating the detection location is not limited to being displayed as a surrounding frame. The frame lines may be displayed as dashed lines, or, for example, bracket marks indicating the four corners of the rectangle or arrow marks may be displayed.

[0105] Examples of display methods other than segmentation masks In this embodiment, an example of displaying region information including the shape characteristics of a defect was described as a segmentation mask display; however, the method of displaying region information is not limited to a segmentation mask. For example, an outline (border) of the defect region may be generated based on the region information, and the outline of the defect region may be displayed instead of, or in addition to, a segmentation mask.

[0106] Image type The images to be processed are not limited to X-ray transmission images; they may also be images generated by receiving reflected light from visible light and / or infrared light using an image sensor, or images taken using a scanning electron microscope (SEM). Furthermore, the images are not limited to two-dimensional images; they may also be three-dimensional images, such as 3D CT (Computed Tomography) images, which are reconstructed three-dimensionally from a large number of continuously obtained two-dimensional slice images. When dealing with three-dimensional images, the concept of pixels in two-dimensional images can be understood by replacing them with voxels, and the concept of area in two-dimensional images can be understood by replacing it with the volume of a spatial area in three-dimensional images.

[0107] 《Example of a shooting system configuration》 Next, an example of an imaging system 500 for capturing images of an object under inspection (OBJ) will be described. Figure 12 is a schematic block diagram showing an example configuration of the imaging system 500. The imaging system 500 is for capturing images of an object under inspection (OBJ) placed in an imaging room 514, and includes an imaging control unit 502, an imaging operation unit 504, an image recording unit 506, a camera 508, and radiation sources 510 and 512.

[0108] The shooting control unit 502 includes a CPU that controls the operation of each part of the shooting system 500. The shooting control unit 502 receives operation input from the operator (photographer) via the shooting operation unit 504 and transmits control signals corresponding to this operation input to each part of the shooting system 500 to control the operation of each part.

[0109] The imaging control unit 504 includes an input device that receives operation input from the operator. The operator can input information about the object under inspection (OBJ), input instructions for imaging conditions and instructions to execute imaging for the camera 508, input instructions for radiation irradiation conditions for the radiation sources 510 and 512, and input instructions to record the images obtained by imaging in the image recording unit 506. Imaging conditions include, for example, exposure time, focal length, aperture, imaging angle, and imaging location. Radiation irradiation conditions include irradiation start time, irradiation duration, irradiation angle, and irradiation intensity.

[0110] The image recording unit 506 records image data (received image) of the object under inspection (OBJ) captured by the camera 508. The image recording unit 506 records information for identifying the object under inspection (OBJ) in association with the image data.

[0111] The camera 508 and radiation sources 510 and 512 are located inside the imaging room 514. The radiation sources 510 and 512 are, for example, X-ray sources, and the partitions and entrances between the imaging room 514 and the outside are protected from X-rays by X-ray shielding materials (e.g., lead or concrete). However, when imaging is performed by irradiating the object under examination (OBJ) with visible light, it is not necessary to use the protected imaging room 514.

[0112] The radiation sources 510 and 512 irradiate the object under examination OBJ, which is placed in the imaging room 514, with radiation according to instructions from the imaging control unit 502.

[0113] Camera 508, in accordance with instructions from the imaging control unit 502 to perform imaging, receives radiation from the radiation source 510 that is irradiated onto the object under inspection OBJ and reflected by the object under inspection OBJ, or radiation from the radiation source 512 that is irradiated onto the object under inspection OBJ and passes through the object under inspection OBJ, and images the object under inspection OBJ. The object under inspection OBJ is held in the imaging chamber 514 by a holding member (not shown) (e.g., a manipulator, a mounting table, or a movable mounting table), and the distance and angle of the object under inspection OBJ relative to the camera 508, radiation sources 510 and 512 are adjustable. The operator can control the relative positions of the object under inspection OBJ, camera 508, radiation sources 510 and 512 via the imaging control unit 502, and can image a desired area of ​​the object under inspection OBJ.

[0114] The radiation sources 510 and 512 cease irradiating the object OBJ under inspection in synchronization with the completion of the imaging by the camera 508.

[0115] In the example shown in Figure 12, the camera 508 is located inside the imaging room 514, but the camera 508 may be located outside the imaging room 514 as long as it can capture images of the object under examination (OBJ) inside the imaging room 514. Also, in the example shown in Figure 12, there is one camera 508 and two radiation sources 510 and 512, but the number of cameras and radiation sources is not limited to these. For example, there may be multiple cameras and radiation sources, or just one of each. The imaging control unit 502, imaging operation unit 504, and image recording unit 506 can be implemented using a combination of computer hardware and software.

[0116] The information processing device 10 may be connected to the imaging system 500 in a communicative manner, or the information processing device 10 may be configured to function as the imaging control unit 502, imaging operation unit 504, and image recording unit 506 of the imaging system 500.

[0117] Regarding programs that operate computers: It is possible to record a program that enables a computer to implement some or all of the processing functions of the information processing device 10 on a computer-readable medium, such as an optical disk, magnetic disk, or semiconductor memory, which is a tangible, non-temporary information storage medium, and to provide the program through this information storage medium.

[0118] Alternatively, instead of providing programs by storing them on such non-temporary computer-readable media, it is also possible to provide program signals as a download service using telecommunication lines such as the Internet.

[0119] Some or all of the processing functions of the information processing device 10 may be implemented by cloud computing, or they may be provided as a SaaS (Software as a Service) service.

[0120] Regarding the hardware configuration of each processing unit: The hardware structure of the processing unit that executes various processes in the information processing device 10, such as the image acquisition unit 12, region extraction unit 14, segmentation mask generation unit 15, position information generation unit 16, rectangular frame generation unit 17, size determination unit 18, detection size determination unit 24, display size determination unit 25, display mode selection unit 20, display control unit 22, and display magnification control unit 28, is, for example, various processors as shown below.

[0121] Various types of processors include CPUs, which are general-purpose processors that execute programs and function as various processing units; GPUs, which are processors specialized for image processing; Programmable Logic Devices (PLDs), such as FPGAs (Field Programmable Gate Arrays), which are processors whose circuit configuration can be changed after manufacturing; and Dedicated Electrical Circuits, such as ASICs (Application Specific Integrated Circuits), which are processors with circuit configurations specifically designed to perform particular processing.

[0122] A single processing unit may be composed of one of these various processors, or it may be composed of two or more processors of the same or different type. For example, a single processing unit may be composed of multiple FPGAs, or a combination of a CPU and an FPGA, or a combination of a CPU and a GPU. Alternatively, multiple processing units may be composed of a single processor. Examples of composing multiple processing units with a single processor include, firstly, a configuration where one or more CPUs and software are combined to form a single processor, and this processor functions as multiple processing units, as is typical of computers such as clients and servers. Secondly, a configuration where a processor is used that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip, as is typical of System-on-a-Chip (SoC) systems. Thus, various processing units are configured, in terms of hardware structure, using one or more of the above-mentioned various processors.

[0123] Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit composed of circuit elements such as semiconductor devices.

[0124] Advantages of this embodiment The information processing device 10 according to this embodiment has the following advantages.

[0125] [1] Segmentation hmm Region extraction AI, such as Model SM, can extract target regions from images with high accuracy, enabling the detection of minute and weak defects and the understanding of defect shapes.

[0126] [2] The information processing device 10 controls the display method (display format) for the defects that have occurred, according to at least one of the following sizes: the detection size, which is determined from the defect region information extracted by the region extraction process, and the display size on the defect display screen, which is determined based on the detection size and the display conditions. For minute and weak defects that are easily overlooked, a rectangular frame that is easy to see is displayed, which suppresses oversights and has the effect of enabling comprehensive visual evaluation of defects.

[0127] [3] The information processing device 10 can automatically detect (extract) defect signals from X-ray transmission images obtained by imaging various minute and weak defects that occur during the molding of metal parts by casting or forging, and during welding, using AI-based image processing, and present the defect information to the inspector in a manner that is easy to see.

[0128] [4] The technology for controlling the display of defect detection results realized by the information processing device 10 can be applied to various fields, not just defect detection.

[0129] Other application examples In the embodiments described above, an example of an information processing device 10 applied to a defect inspection device for industrial products was described, but the scope of this disclosure is not limited to this example. The technology of this disclosure can be broadly applied to devices that detect a specific object from an image and display information indicating its region and / or location, such as a device that detects printing defects from printed materials and displays the detection results, or a device that detects lesions from medical images and displays the detection results.

[0130] 《Example 1》 The processing functions of the information processing device 10 may be implemented using multiple information processing devices. For example, a computer system may be adopted in which a first information processing device and a second information processing device are connected via a communication line. The first information processing device may be equipped with the processing functions of an image acquisition unit 12, a region extraction unit 14, and a location information generation unit 16, while the second information processing device may be equipped with the processing functions of a rectangular frame generation unit 17, a size determination unit 18, a display mode selection unit 20, and a display control unit 22. The communication line may be a local area network or a wide area network.

[0131] In this case, the first information processing device generates region information and location information of the defect to be detected from the image to be processed. The second information processing device acquires the region information and location information of the defect generated by the first information processing device and switches the display mode according to the determination result of the size determination unit 18. The display data obtained after processing by the second information processing device can be transmitted to another terminal device via a communication line, and the processing result can be displayed on the display device of the terminal device.

[0132] 《Modified Example 2》 In the above embodiment, an example of performing region extraction using the segmentation model SM was described. However, the image processing method for automatically extracting the target region from an image is not limited to this example; various region extraction methods can be applied.

[0133] Variation 3 In the above embodiment, an example of processing a still image as an image was described, but the shooting device may also shoot a video, and the information processing device 10 may extract some frames from the shot video and perform similar processing.

[0134] "others" This disclosure is not limited to the embodiments described above, and various modifications are possible without departing from the spirit of the technical idea of ​​this disclosure. [Explanation of Symbols]

[0135] 10 Information Processing Devices 12 Image acquisition unit 14 Region extraction part 15. Segmentation mask generation unit 16 Location information generation section 17 Rectangular frame generation section 18. Size determination section 20 Display Mode Selection Section 22 Display Control Unit 24 Detection size determination unit 25 Display size determination unit 28 Display Magnification Control Unit 32 Input devices 34 Display device 50 metal parts 51 First part 52 Second part 54 Background 102 processors 104 Computer-readable media 106 Communication Interface 108 Input / Output Interfaces 110 Bus 114 Region Extraction Program 115 Segmentation Mask Generation Program 116 Location Information Generation Program 117 Rectangular Frame Generation Program 118 Size Determination Program 120 Display Mode Selection Program 122 Display Control Program 500 shooting systems 502 Imaging Control Unit 504 Shooting Control Unit 506 Image Recording Unit 508 Camera 510 Radiation source 512 Radiation source 514 Photography Room SM Segmentation Model DA1, DA2, DA3 Defect Regions RF2, RF3 Rectangular Frame IM image IMG1, IMG2, IMG3, IMG4: Examination images OBJ (Object under inspection) S11-S39 Steps in Information Processing Methods

Claims

1. An information processing device equipped with a processor, The aforementioned processor, Get the image, A region extraction process is performed to extract the region to be detected from the aforementioned image. From the region information of the extracted region, the position information of the region is generated. The system is configured to display the results of the region extraction by switching between a first display mode, which displays information indicating the location of the region on the display screen in a visually appealing manner based on the position information, according to the size of the extracted region, and a second display mode, which displays the region information representing the shape of the region in pixel units in a manner different from the first display mode. A visual representation of information indicating the location of the said area includes displaying a rectangular frame, a circular frame, a polygonal frame, a closed curve, bracket marks indicating the four corners of a rectangle, or arrow marks surrounding the said area. The second display mode includes displaying a segmentation mask that fills in the extracted region or displaying the outline of the extracted region, The aforementioned processor, If the size of the extracted region is smaller than the first reference size, it is displayed in the first display mode. If the size of the extracted region is larger than the first reference size, it is displayed in the second display mode. Information processing device.

2. The first display mode is a mode in which information indicating the location of the area is displayed in a predetermined size. The information processing apparatus according to claim 1.

3. The first display mode is a mode in which information indicating the location of the area is displayed in a predetermined fixed size. The information processing apparatus according to claim 1.

4. The aforementioned processor, If the size of the extracted region is smaller than the first reference size, the region information is displayed, and information indicating the location of the region is displayed in the first display mode. The information processing apparatus according to claim 1.

5. An information processing device comprising a processor, The aforementioned processor, Get the image, A region extraction process is performed to extract the region to be detected from the aforementioned image. From the region information of the extracted region, the position information of the region is generated. The system is configured to display the results of the region extraction process by switching between a first display mode, which displays information indicating the location of the region on the display screen in a visually appealing manner based on the location information, according to the display size of the region displayed on the display screen, and a second display mode, which displays the region information in a manner different from the first display mode. The aforementioned processor, If the display size of the area displayed on the display screen is smaller than the second reference size, the display is made in the first display mode. If the display size is larger than the second reference size, the display is shown in the second display mode. Information processing device.

6. The aforementioned processor, We accept instructions to enlarge and reduce the display. In accordance with the instructions received, the display magnification of the display screen is changed. Depending on the display magnification, the first display mode and the second display mode are switched. The information processing apparatus according to claim 5.

7. The system further includes an input device that receives input for instructions regarding the enlarged display and the reduced display. The information processing apparatus according to claim 6.

8. The first display mode includes displaying a rectangular or circular frame as information indicating the location of the extracted region, The information processing apparatus according to claim 5.

9. The second display mode includes a segmentation mask display that fills in the extracted region, The information processing apparatus according to claim 5.

10. The second display mode includes displaying the outline of the extracted region, The information processing apparatus according to claim 5.

11. The aforementioned processor, The region extraction process is performed using a segmentation model that performs image segmentation. The information processing apparatus according to claim 1 or 5.

12. The aforementioned segmentation model is A learning model trained using machine learning to extract the target region from the input image. The information processing apparatus according to claim 11.

13. The position information includes information indicating the position of the centroid or the center of the circumscribing rectangle of the extracted region. The information processing apparatus according to claim 1 or 5.

14. The system further includes a display device for displaying the processing results. The information processing apparatus according to claim 1 or 5.

15. The aforementioned image is an X-ray transmission image. The information processing apparatus according to claim 1 or 5.

16. The aforementioned image is an X-ray transmission image of a cast metal part, a forged metal part, or a welded metal part. The information processing apparatus according to claim 1 or 5.

17. The object being detected is a defect. The information processing apparatus according to claim 1 or 5.

18. The aforementioned defect is, It includes at least one of the following: bubbles, porosity, FMLD (Foreign material less dense), and FMMD (Foreign material more dense). The information processing apparatus according to claim 17.

19. The first display mode includes flashing information indicating the location of the area, The information processing apparatus according to claim 1 or 5.

20. An information processing method performed by an information processing device, Acquiring an image and, This involves performing a region extraction process to extract the region to be detected from the aforementioned image, To generate location information of the region from the region information of the extracted region, The process of extracting a region is performed by switching between a first display mode, which displays information indicating the location of the region on the display screen in a visually appealing manner based on the location information, according to the size of the extracted region, and a second display mode, which displays the region information representing the shape of the region in pixel units in a manner different from the first display mode, to display the results of the region extraction. A visual representation of information indicating the location of the said area includes displaying a rectangular frame, a circular frame, a polygonal frame, a closed curve, bracket marks indicating the four corners of a rectangle, or arrow marks surrounding the said area. The second display mode includes displaying a segmentation mask that fills in the extracted region or displaying the outline of the extracted region, If the size of the extracted region is smaller than the first reference size, it will be displayed in the first display mode. This includes displaying the extracted region in the second display mode if the size of the extracted region is larger than the first reference size. Information processing methods.

21. An information processing method performed by an information processing device, Acquiring an image and, This involves performing a region extraction process to extract the region to be detected from the aforementioned image, To generate location information of the region from the region information of the extracted region, The process includes switching between a first display mode, in which information indicating the location of the area is displayed on the display screen in a visually appealing manner based on the location information, and a second display mode, in which the area information is displayed in a manner different from the first display mode, to display the results of the area extraction process, depending on the display size of the area displayed on the display screen. If the display size of the area displayed on the display screen is smaller than the second reference size, the display is made in the first display mode. This includes displaying the image in the second display mode when the aforementioned display size is larger than the second reference size. Information processing methods.

22. An information processing method performed by an information processing device, To obtain region information and positional information of the region to be detected in the image, Depending on the size of the area, the system switches between a first display mode in which information indicating the location of the area is displayed on the display screen in a visually appealing manner based on the position information, and a second display mode in which the area information representing the shape of the area in pixel units is displayed in a manner different from the first display mode. Includes, A visual representation of information indicating the location of the said area includes displaying a rectangular frame, a circular frame, a polygonal frame, a closed curve, bracket marks indicating the four corners of a rectangle, or arrow marks surrounding the said area. The second display mode includes displaying a segmentation mask that fills in the region or displaying the outline of the region, If the size of the area is smaller than the first reference size, it will be displayed in the first display mode. This includes displaying the area in the second display mode if the size of the area is larger than the first reference size. Information processing methods.

23. An information processing method performed by an information processing device, To obtain region information and positional information of the region to be detected in the image, Depending on the display size of the area displayed on the display screen, a first display mode is used to display information indicating the location of the area on the display screen in a visually appealing manner based on the location information, and a second display mode is used to display the area information in a manner different from the first display mode. Includes, If the display size of the area displayed on the display screen is smaller than the second reference size, the display is made in the first display mode. This includes displaying the image in the second display mode when the aforementioned display size is larger than the second reference size. Information processing methods.

24. A program that causes a computer to execute the information processing method described in any one of claims 20 to 23.

25. A non-temporary and computer-readable recording medium on which the program described in claim 24 is recorded.

Images