Image processing device, image processing method, and program

Abstract

Provided is an image processing device capable of detecting an object to be detected that remains on a bottom surface of a transparent container or on a medium surface inside the transparent container. A plurality of images are captured by rotating a transparent container and show an object to be detected remaining on a bottom surface of the transparent container or an object to be detected remaining on a surface, inside the transparent container, on which a medium is in contact with another medium, the plurality of images being compared to determine, from among candidate areas of the object to be detected that are shown in the images, a candidate area that moves in a moving direction according to the rotation. The presence or absence of the object to be detected is determined by using: a first determination result obtained by using image information on the candidate area moving in the moving direction according to the rotation and a first learning model to determine whether the candidate area is the object to be detected; and a second determination result obtained by using information indicating a time-series change of the candidate area moving in the moving direction according to the rotation and a second learning model to determine whether the candidate area is the object to be detected.

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, program Regarding. [Background technology]

[0002] Techniques for detecting the presence of a detection target, such as a foreign object contained in a transparent container, have been disclosed. For example, Patent Document 1 narrows down an inspection area from an image using image processing, and distinguishes between air bubbles and foreign objects within the area using the shape of the air bubbles. In Patent Document 1, when determining whether a foreign object is present in a liquid sealed in a translucent container such as a syringe, the container is vibrated or rotated to cause the foreign object to float, and the presence or absence of the foreign object in the liquid is determined by distinguishing between the floating foreign object and the air bubbles generated by the vibration or rotation.

[0003] An example of a method for detecting foreign matter in liquid is disclosed in Patent Document 2. In the method for detecting foreign matter in liquid of Patent Document 2, foreign matter floating in a container is detected based on the difference in brightness between air bubbles that appear with high brightness and foreign matter that appears with low brightness by adjusting the relative light intensities of a transmission light source and a reflection light source.

[0004] Patent Document 3 discloses a technology for detecting foreign objects suspended by rotation using feature quantities obtained from differential images. In Patent Document 3, two sets of images are selected at a predetermined time interval to generate a differential image. In addition, Patent Document 3 performs a differential process on the differential image to extract edge pixels, and then groups the extracted edges to extract individual floating differential images. In addition, Patent Document 3 also obtains feature quantities such as the distribution and shape of differential values ​​from the individual grouped floating differential images, and distinguishes between foreign objects and air bubbles from the feature quantities to detect foreign objects in liquid.

[0005] Patent Document 4 discloses a technique relating to a method for inspecting the appearance of a transparent film, which inspects the appearance of the transparent film for defects such as foreign matter and air bubbles by performing computer image processing on an image of the transparent film having low reflectance. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-44688 [Patent Document 2] Japanese Patent Application Publication No. 11-125604 [Patent Document 3] Japanese Patent Application Laid-Open No. 2004-354100 [Patent Document 4] Japanese Patent Application Laid-Open No. 2009-264915 Summary of the Invention [Problem to be solved by the invention]

[0007] When a detection target, such as a foreign object mixed in the medium inside a transparent container as described above, remains on a curved portion such as the bottom surface of the transparent container or on the surface of the medium inside the transparent container (e.g., the liquid surface), there is a problem that it is difficult to detect the tiny detection target due to the influence of light reflection caused by the curvature of the shape of the bottom surface or the surface of the medium of the transparent container.

[0008] Therefore, the present invention provides an image processing device, an image processing method, and program The purpose is to provide. [Means for solving the problem]

[0009] According to a first aspect of the present invention, an image processing device compares multiple images of a detection target inside a transparent container taken by rotating the transparent container, and determines candidate areas of the detection target that appear in the images that move in a direction corresponding to the rotation.The image processing device determines whether the candidate area is the detection target using a first judgment result obtained by using image information of the candidate area that moves in a direction corresponding to the rotation and a first learning model, and a second judgment result obtained by using information indicating the time series changes in the candidate area that moves in a direction corresponding to the rotation and a second learning model to determine whether the candidate area is the detection target.

[0010] According to a second aspect of the present invention, an image processing method is an image showing a detection target inside a transparent container, and compares multiple images taken by rotating the transparent container to determine, among candidate areas for the detection target shown in the images, those that move in a movement direction corresponding to the rotation.The method determines whether or not the detection target is present using a first judgment result obtained by using image information of the candidate area that moves in a movement direction corresponding to the rotation and a first learning model, and a second judgment result obtained by using information indicating the time series changes in the candidate area that moves in a movement direction corresponding to the rotation and a second learning model to determine whether or not the candidate area is the detection target.

[0011] According to a third aspect of the present invention, the storage medium stores a program that causes a computer of an image processing device to execute the following processes: a process of comparing multiple images of a detection target inside a transparent container, the images being images of the detection target inside the transparent container taken by rotating the transparent container, and determining, among candidate areas of the detection target shown in the images, candidate areas that move in a movement direction corresponding to the rotation; and a process of determining the presence or absence of the detection target using a first judgment result that determines whether or not the candidate area is the detection target using image information of the candidate area that moves in a movement direction corresponding to the rotation and a first learning model; and a second judgment result that determines whether or not the candidate area is the detection target using information indicating the time series changes in the candidate area that moves in a movement direction corresponding to the rotation and a second learning model. [Effects of the Invention]

[0012] According to the present invention, it is possible to more accurately detect a detection target that is staying on a curved portion inside a transparent container or on a medium surface inside a transparent container. [Brief explanation of the drawings]

[0013] [Figure 1] 1 shows a first perspective view of a foreign object detection device according to an embodiment of the present invention; [Figure 2] FIG. 2 shows a second perspective view of a foreign object detection device according to an embodiment of the present invention. [Figure 3] FIG. 2 shows a third perspective view of a foreign object detection device according to an embodiment of the present invention. [Figure 4] 1 is a first diagram showing an example of the appearance of a foreign object in a transparent container according to an embodiment of the present invention. FIG. [Figure 5] FIG. 10 is a second diagram showing an example of the appearance of foreign matter in a transparent container according to one embodiment of the present invention. [Figure 6] 1 is a diagram illustrating a hardware configuration of an image processing apparatus according to an embodiment of the present invention. [Figure 7] FIG. 1 is a first diagram showing functional blocks of an image processing apparatus according to an embodiment of the present invention. [Figure 8] FIG. 2 is a second diagram showing functional blocks of the image processing device according to the embodiment of the present invention. [Figure 9] FIG. 10 is a third diagram showing functional blocks of the image processing device according to the embodiment of the present invention. [Figure 10] FIG. 2 is a first diagram showing a processing flow of the image processing device according to the embodiment of the present invention. [Figure 11] FIG. 10 is a second diagram showing the processing flow of the image processing device according to the embodiment of the present invention. [Figure 12] FIG. 1 is a first diagram illustrating an overview of processing performed by an image processing apparatus according to an embodiment of the present invention. [Figure 13] FIG. 10 is a second diagram illustrating the outline of processing performed by the image processing device according to the embodiment of the present invention. [Figure 14] FIG. 10 is a third diagram illustrating an outline of processing performed by the image processing device according to the embodiment of the present invention. [Figure 15] FIG. 4 is a fourth diagram illustrating an outline of processing performed by the image processing device according to the embodiment of the present invention. [Figure 16] FIG. 10 shows a fourth perspective view of a foreign object detection device according to an embodiment of the present invention. [Figure 17] FIG. 4 is a fourth diagram showing functional blocks of the image processing device according to the embodiment of the present invention. [Figure 18] FIG. 10 is a diagram showing a processing flow of an image processing apparatus according to a second embodiment of the present invention. [Figure 19]FIG. 10 is a diagram showing an outline of processing performed by an image processing apparatus according to a second embodiment of the present invention. [Figure 20] FIG. 1 is a diagram showing a minimum configuration of an image processing device according to the present invention. [Figure 21] FIG. 1 is a diagram showing a processing flow in an image processing apparatus with a minimum configuration according to the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. [First embodiment] [Configuration Description] First, an embodiment of the invention will be described in detail with reference to the drawings.

[0015] Fig. 1 shows a first perspective view of a foreign object detection device 100 including an image processing device 1 of the present invention. As shown in Fig. 1, the foreign object detection device 100 is composed of the image processing device 1, a transparent container 2 to be inspected, a rotation device 4 that rotates the transparent container 2, and a reflected light 5 and a transmitted light 6 that illuminate the liquid 3 inside the transparent container 2. The transparent container 2 contains the liquid 3.

[0016] In the first embodiment, the transparent container 2 has a cylindrical shape, and the medium contained in the transparent container is a liquid 3. A foreign object to be detected resides on the bottom surface of the transparent container 2, and the image processing device 1 determines the presence or absence of the foreign object based on an image including the foreign object residing on the bottom surface of the transparent container 2. The foreign object to be detected resides on the surface of the liquid 3 in the transparent container 2, and the image processing device 1 may determine the presence or absence of the foreign object based on an image including the foreign object residing on the surface of the liquid 3. The bottom surface of the transparent container 2 is an example of a curved portion of the wall surface of the transparent container 2. The surface of the liquid 3 in the transparent container 2 is one aspect of the surface where the medium (liquid 3) contained in the transparent container 2 comes into contact with another medium (air) within the transparent container 2. The image processing device 1 particularly determines the detection object in an image that shows the detection object residing on the curved portion of the wall surface of the transparent container 2 or the surface where the medium contained in the transparent container 2 comes into contact with another medium within the transparent container. The medium contained in the transparent container 2 is not limited to the liquid 3. Furthermore, the curved portion of the wall surface of the transparent container 2 is not limited to the bottom surface. Furthermore, the surface where the liquid 3 contained in the transparent container 2 comes into contact with other media within the transparent container 2 is not limited to the liquid surface of the liquid 3. The image processing device 1 may determine the type of detection target (foreign matter, air bubbles, etc.). The image processing device 1 may detect a detection target remaining in a medium other than the liquid contained in the transparent container 2.

[0017] In other words, the image processing device 1 may detect the detection target from an image that shows the detection target remaining on the bottom surface of the transparent container 2 or the detection target remaining on the surface where the medium contained in the transparent container 2 comes into contact with another medium within the transparent container 2. The image processing device 1 may detect the detection target only from an image that shows the detection target remaining on the bottom surface of the transparent container 2, or may detect the detection target only from an image that shows the detection target remaining on the surface of the liquid 3 in the transparent container 2. The image processing device 1 may also detect the detection target or an image that shows the detection target floating in the liquid 3 in the transparent container 2.

[0018] The rotation device 4 holds the cylindrical transparent container 2 by its bottom while holding it vertically, and rotates it around the center of the bottom (z-axis) as the rotation axis, thereby rotating the transparent container 2. The detection target suspended in the liquid 3 contained in the transparent container 2 rotates around the rotation axis as the transparent container 2 rotates. In this embodiment, the image processing device 1 includes a photographing device 10. The photographing device 10 is installed in a position where it can photograph the detection target, and where the photographing direction is fixed to the photographing axis (x-axis) perpendicular to the z-axis. While the rotation device 4 rotates the transparent container 2, the image processing device 1 controls the photographing device 10 to photograph the detection target and continuously acquire multiple images with a field of view that captures the detection target. While the photographing device 10 is photographing, the relationship between the z-axis and the x-axis is maintained. As a result, the image processing device 1 acquires multiple images including the detection target, whose position changes as the rotation device 4 rotates. In the example of FIG. 1, the pair of image processing device 1 including image capture device 10 and reflective illuminator 5 and transmitted illuminator 6 are located on opposite sides of an imaginary plane including the rotation axis (z-axis).

[0019] FIG. 2 shows a second perspective view of a foreign object detection device 100 including an image processing device 1 of the present invention. While FIG. 1 shows an example in which the foreign object detection device 100 is equipped with a reflected illuminator 5 and a transmitted illuminator 6, the foreign object detection device 100 does not necessarily have to be equipped with a transmitted illuminator 6. In the example of FIG. 2, the image processing device 1 equipped with the photographing device 10 and the reflected illuminator 5 are each positioned on one side of an imaginary plane including the axis of rotation (z-axis). When the reflected illuminator 5 illuminates the liquid 3 in this state, the detection target, such as a foreign object, often appears in the image with a higher brightness than its surroundings due to the reflection of light. In this case, the image processing device 1 determines the detection target, such as a foreign object, from the area that appears brighter than its surroundings.

[0020] FIG. 3 shows a third perspective view of a foreign object detection device 100 including an image processing device 1 of the present invention. While FIG. 1 illustrates an example in which the foreign object detection device 100 includes a reflected illuminator 5 and a transmitted illuminator 6, the foreign object detection device 100 does not necessarily need to include the reflected illuminator 5. In the example of FIG. 3, the image processing device 1 including the image capture device 10 and the reflected illuminator 5 are located on opposite sides of an imaginary plane including the rotation axis (z-axis). When the transmitted illuminator 6 illuminates the liquid 3 in this state, light is blocked by the detection target, such as a foreign object. In many cases, the detection target appears in the image generated by the image capture device 10, which captures the liquid from the opposite side of the imaginary plane from the transmitted illuminator 6, with a lower brightness than its surroundings. In this case, the image processing device 1 identifies the detection target, such as a foreign object, from the area that appears lower in brightness than its surroundings.

[0021] FIG. 4 is a first diagram showing an example of the appearance of a foreign object in a transparent container. 4 shows first and second examples of a state in which a foreign object is present in the liquid 3 when an image of the transparent container 2 is captured by the image capturing device 10 of the image processing device 1. When the detection target is near the bottom surface 41 of the transparent container 2, the image capturing device 10 generates an image 4a capturing the detection target on the bottom surface 41. When the detection target is near the bottom surface 41 of the transparent container 2, the image capturing device 10 generates an image 4b ​​capturing the detection target on the liquid surface 42.

[0022] FIG. 5 is a second diagram showing an example of the appearance of foreign matter in a transparent container. 5 shows a third example of a state in which a foreign object is present in the liquid 3 when the transparent container 2 is photographed by the photographing device 10 of the image processing device 1. When a detection target is floating in the liquid 3 in the transparent container 2, the photographing device 10 may generate an image 5a capturing the floating detection target. The image processing device 1 may detect the detection target based on the image 5a.

[0023] Here, when the detection target is floating in the liquid 3, as in the third example (Image 5a) where a foreign object is present in the liquid 3, controlling the lighting environment makes it easy to maintain a constant brightness value in the image area other than the detection target, as shown in Image 5a, making it easy to reveal the detection target area. In addition, in the third example (Image 5a), the floating detection target gradually falls in the direction of gravity, making it easy to detect the detection target by detecting its movement. In contrast, as shown in the first example (Image 4a) and the second example (Image 4b), it is necessary to accurately detect the detection target that is staying near the liquid surface or the bottom of the transparent container 2. In the first example (Image 4a) and the second example (Image 4b), the curved parts of the wall of the transparent container 2, such as the liquid surface and the bottom area, are captured in the image, making the brightness value of the areas in the image other than the detection target constant, making it difficult to reveal the detection target area. Furthermore, because the detection target is staying near the bottom or the liquid surface, it cannot be detected by its falling motion. Therefore, the image processing device 1 has both a function of revealing the area of ​​the detection target even near the liquid surface or bottom, and a function of detecting the remaining detection target by the rotational movement of the container.

[0024] FIG. 6 is a hardware configuration diagram of the image processing device according to this embodiment. As shown in this diagram, the image processing device 1 is a computer equipped with various hardware components such as a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, an HDD (Hard Disk Drive) 104, a communication module 105, and a database 106. The image processing device 1 also includes an imaging device 10.

[0025] FIG. 7 is a first diagram showing functional blocks of the image processing device. 7, the image processing device 1 executes a pre-stored image processing program, thereby enabling the image processing device 1 to perform the functions of an imaging unit 301, a histogram generation unit 302, a histogram storage unit 303, an extraction unit 304, a difference detection unit 305, a clustering unit 306, a tracking determination unit 307, and a classification unit 308.

[0026] The photographing unit 301 controls the photographing device 10 to acquire an image generated by the photographing device 10 . The histogram generating unit 302 generates a reference histogram indicating the frequency of each pixel value of each pixel in a plurality of images (1, 2, . . . N frames) that are created in advance by capturing images of a transparent container while it is being rotated. The histogram storage unit 303 stores the reference histogram generated in advance by the histogram generation unit 302 . The extraction unit 304 compares the frequency of occurrence (histogram) of the brightness of each pixel in one selected image from multiple images (1, 2,...N frames) obtained by sequentially photographing the transparent container 2 while rotating it with the rotation device 4, based on a comparison with the reference histogram stored in the histogram memory unit 303, and extracts candidate areas for detection targets such as foreign objects and bubbles in the selected image. The difference detection unit 305 sequentially compares pixel values ​​of candidate areas for detection extracted from a plurality of images taken before and after, and performs processing to remove candidate areas that have not moved from the detection targets. The clustering unit 306 clusters groups of adjacent pixels among the pixels remaining as candidate regions as one candidate region. The tracking determination unit 307 identifies, from among the clustered candidate regions (detection windows), related regions before and after the movement in response to the rotation of the transparent container, in each pixel of a plurality of images taken at different times. The classification unit 308 identifies the detection target using a first determination result obtained by determining whether or not the candidate area is the detection target using image information of the candidate area moving in the movement direction according to the rotation and a first learning model, and a second determination result obtained by determining whether or not the candidate area is the detection target using information indicating time-series changes in the candidate area moving in the movement direction according to the rotation and a second learning model. Identifying the detection target is one aspect of the process of determining the presence or absence of the detection target.

[0027] FIG. 8 is a second diagram showing the functional blocks of the image processing device. 8 shows detailed functional blocks of the tracking determination unit 307. The tracking determination unit 307 performs the functions of a weighting matrix generation unit 401, a new trajectory weighting unit 402, a position weighting unit 403, a size change weighting unit 404, a rotation direction weighting unit 405, a connection determination unit 406, an unused area storage unit 407, an existing trajectory storage unit 408, and a trajectory length determination unit 409.

[0028] The weighting matrix generation unit 401 generates a weighting matrix M for calculating the relationship between each tracking result of the candidate areas for detection targets up to the N-1th frame image and each candidate area for detection targets in the Nth frame image, based on the weighting information obtained from the new trajectory weighting unit 402, the position weighting unit 403, the size change weighting unit 404, and the rotation direction weighting unit 405.

[0029] The new trajectory weighting unit 402, position weighting unit 403, size change weighting unit 404, and rotation direction weighting unit 405 each determine weighting information indicating whether the correlation between each tracking result of the candidate area for detection target up to the N-1th frame image from a different perspective and each candidate area for detection target in the Nth frame image is strong or weak. The weighting units will be described in detail later.

[0030] The link determination unit 406 uses a weighting matrix M and an algorithm for solving assignment problems, such as the Hungarian algorithm, to assign each candidate area of ​​the detection target in the Nth frame image to each tracking result of the candidate area of ​​the detection target in the N-1th frame images.

[0031] The unused area storage unit 407 stores candidate areas that have not been assigned to the tracking results by the connection determination unit 406 . The existing trajectory storage unit 408 stores, for each frame, each candidate area assigned to the tracking result by the connection determination unit 406 and the identification information of the trajectory in association with each other. The trajectory length determination unit 409 determines, from the trajectory information stored in the existing trajectory storage unit 408, a trajectory that has a connection length equal to or greater than a threshold value as a trajectory to be detected.

[0032] FIG. 9 is a third diagram showing functional blocks of the image processing device. The functional blocks shown in FIG. 9 are detailed functional blocks of the identification unit 308. The classification unit 308 performs the functions of a pseudo-texture memory unit 501, a synthesis target image generation unit 502, a tracking time-series data memory unit 503, an actual image memory unit 504, a synthesis target image memory unit 505, a classifier 506, a learning unit 507, and a classification result memory unit 508.

[0033] The pseudo texture storage unit 501 stores pseudo textures. The synthesis target image generating unit 502 generates a synthesis target image that simulates the detected object by cutting out the pseudo texture into a random shape and superimposing it at a random position and angle on the image. The tracking time series data storage unit 503 stores time series data of the tracking results obtained when tracking the detection target. The actual image storage unit 504 stores a plurality of images of actual foreign matter, air bubbles, and other detection targets, which have different shapes and sizes. The compositing target image storage unit 505 stores the compositing target image generated by the compositing target image generation unit 502 . The classifier 506 classifies detection targets such as foreign matter, bubbles, etc., using the learning model generated by the learning unit 507. Specifically, the classifier 506 outputs a three-value classification result including erroneous detection of foreign matter, bubbles, and other regions. The learning unit 507 generates a learning model for identifying the detection target using the determination result of the tracking determination unit 307. The classification result storage unit 508 stores the determination result of the classifier 506 .

[0034] First, the image processing device 1 generates a first learning model and a second learning model in advance. The first learning model is a learning model for identifying whether a candidate area of ​​the detection target moving in a movement direction corresponding to the rotation of the transparent container 2 is a detection target from images (1, 2, . . ., N frames) generated by repeatedly capturing images as the transparent container 2 rotates. The second learning model is a learning model for identifying whether a candidate area of ​​the detection target moving in a movement direction corresponding to the rotation of the transparent container 2 is a detection target, based on information on each of the images (1, 2, . . ., N frames) showing time-series changes in the candidate area moving in the movement direction corresponding to the rotation. The image processing device 1 identifies the detection target using these two learning models, the first learning model and the second learning model.

[0035] Specifically, the learning unit 507 performs a first machine learning process using actual images of foreign objects or bubbles that are actual detection targets, which are prepared as learning data, and pseudo images of these detection targets as correct answer data. The actual images are recorded in the actual image storage unit 504. The pseudo images are recorded in the synthesis target image storage unit 505. As an example, the actual images and pseudo images are image information whose center is the center of the area of ​​the detection target.

[0036] The learning unit 507 performs machine learning processing using the real images and pseudo images stored in these storage units. The pseudo images are pseudo images of detection targets, such as foreign objects and bubbles, generated by the synthesis target image generation unit 502 using textures (such as image color samples) stored in the pseudo texture storage unit 501. These pseudo images are created by randomly cropping the texture and superimposing it at random positions and angles on the image. For example, the learning unit 507 generates a first learning model, using a first machine learning process, that includes information on parameters such as weights of a neural network that receives an input image and outputs a result indicating that the detection target indicated by the real image or pseudo image is present in the input image, if the image shows the detection target. When the classifier 506 inputs an image into the neural network generated using the first learning model, it outputs information indicating that the detection target is present in the input image. Alternatively, the first learning model may be information constituting a neural network that identifies the detection target. In this way, the learning unit 507 generates a first learning model that can robustly detect even unknown foreign objects and other detection targets, using an actual image that shows the actual appearance of the detection target and a pseudo image created by combining textures.

[0037] The learning unit 507 also generates tracking time-series data as learning data based on multiple image frames capturing the detection target actually contained in the rotating transparent container 2, and uses the data to perform a second machine learning process. The tracking time-series data is recorded in the tracking time-series data storage unit 503. The tracking time-series data may be information such as a pixel identifier, position (image coordinates), and number of pixels indicating the detection target area in previous and next images in a chronological order, as well as information such as a movement amount and movement vector based on the change in the position of the area in previous and next image frames. For example, when tracking time-series data for multiple frames of images captured in chronological order is input through the second machine learning process, the learning unit 507 generates a second learning model for outputting a result indicating that each trajectory determined to be a candidate area of ​​the detection target in each frame of image is the trajectory of the detection target. The second learning model is learning result information including parameter information such as weights for a neural network. When the classifier 506 inputs tracking time series data relating to images of multiple frames into a neural network generated using the second learning model, if each trajectory determined to be a candidate area for the detection target in the image of each frame is the trajectory of the detection target, it outputs information indicating that it is the detection target.

[0038] The classifier 506 then determines whether the image contains a detection target using a first learning model and a second learning model, which are used to determine whether an image contains a detection target based on different information. Alternatively, the classifier 506 may identify an identifier of the detection target from the image using the first learning model and the second learning model, which are used to determine whether an image contains a detection target based on different information. Since the first learning model and the second learning model are used to determine the detection target, the detection target can be identified with higher accuracy. By using the first learning model and the second learning model to determine the detection target contained in the image, the classifier 506 can perform a three-value classification determination, including whether the detection target is a foreign object, a bubble, or a false detection of other background. Note that the image processing device 1 may use the first learning model and the second learning model to distinguish and identify multiple detection targets as foreign objects.

[0039] FIG. 10 is a first diagram showing the processing flow of the image processing apparatus according to this embodiment. Next, the process by which the image processing device 1 generates a reference histogram will be described. First, the photographing direction and angle of view of the photographing device 10 are fixed. As an example, the photographing device 10 photographs an image including the vicinity of the bottom surface of the transparent container 2. In this state, the rotation device 4 rotates the transparent container 2 around the Z axis. The rotational motion of the transparent container 2 is constant. As a result, the same part of the transparent container 2 appears at a regular interval in the image generated by the photographing device 10. The photographing unit 301 of the image processing device 1 controls the photographing device 10 to continue photographing. Note that when the reference histogram is generated, it is assumed that the transparent container 2 does not contain any detection targets (foreign matter, air bubbles, etc.).

[0040] The photographing unit 301 sequentially acquires images (1, 2,...,N frames) generated by the photographing device 10 (step S101). The photographing device 10 may generate several tens of images while the transparent container 2 makes one rotation. The histogram generating unit 302 acquires multiple images (1, 2,...,N frames). The histogram generating unit 302 sets a rectangular histogram generation range a including one reference pixel in the image and its surrounding pixels for each of the multiple images by shifting the reference pixel by one. The histogram generation range a may be, for example, a range of 5 pixels vertically and 5 pixels horizontally.

[0041] The histogram generation unit 302 generates a reference histogram for the histogram generation range a based on each pixel in each histogram generation range a at the same intra-image position in multiple consecutive images acquired in time series from time 0 to time t (step S102). The histogram generation unit 302 generates reference histograms for the number of pixels in the image. For example, the reference histogram has a vertical axis representing pixel values ​​of one of 255 levels of brightness or color information such as RGB, and a horizontal axis representing frequency of occurrence. The image processing device 1 generates the reference histogram using images generated while the transparent container 2 rotates multiple times. The histogram generation unit 302 records the generated reference histogram in the histogram storage unit 303. The above-described process corresponds to the histogram generation unit 302 creating a reference histogram h_i∈Ω in which each pixel i∈Ω is stored from time 0 to time t, where Ω represents the image area captured by the imaging unit 301. The histogram storage unit 303 stores the reference histogram h_iεΩ for each pixel obtained by the histogram generation unit 302 .

[0042] FIG. 11 is a third diagram showing the processing flow of the image processing device according to this embodiment. FIG. 12 is a first diagram showing an outline of processing performed by the image processing device according to this embodiment. Next, a process performed by the image processing device 1 when actually identifying a detection target will be described. As in the case of generating the reference histogram, the rotation device 4 rotates the transparent container 2. In this state, the photographing unit 301 of the image processing device 1 controls the photographing device 10 to continue photographing. The photographing unit 301 acquires one image (one frame) generated by the photographing device 10 (step S201). The extraction unit 304 acquires a reference histogram generated for one pixel from the reference histograms stored in the histogram storage unit 303 (step S202). The extraction unit 304 uses the reference histogram stored in the histogram storage unit 303 to identify a pixel in the image newly obtained from the photographing unit 301 at a position that matches the pixel for which the reference histogram was generated (step S203). The extraction unit 304 compares the pixel value of the identified pixel with the reference histogram and determines whether the pixel is in a candidate region for detection or in another region.

[0043] If the acquired image contains a detection target such as a foreign object, the pixel values ​​of the pixels representing the detection target will be similar to the pixel values ​​with low frequency in the reference histogram generated based on an image that does not contain the detection target. On the other hand, the pixel values ​​of pixels not representing the detection target will be similar to the pixel values ​​with high frequency in the reference histogram. Therefore, the extraction unit 304 compares the pixel values ​​of pixels identified from the image in which the detection target is to be detected with the reference histogram to determine whether the identified pixels can be considered as candidate regions for the detection target. Specifically, the extraction unit 304 determines whether the output frequency of the pixel values ​​of the pixels identified from the image is lower than the frequency threshold θ set in the reference histogram (step S204). If the pixel value of the pixel identified from the image is lower than the frequency threshold θ set in the reference histogram, the extraction unit 304 determines that the pixel is a candidate for the detection target (step S205). On the other hand, if the output frequency of the pixel value of a pixel identified from the image is higher than the frequency threshold θ set in the reference histogram, the extraction unit 304 determines that the pixel is not a candidate for the detection target because it is close to an image containing the detection target (step S206).

[0044] In the processing by the extraction unit 304 described above, the formula for determining whether each pixel i∈Ω of a newly captured image is a candidate region is given by the following formula, using the reference histogram h_i stored in the histogram storage unit 303 and the pixel value b_i of pixel i of one image newly captured by the imaging unit 301: When h_i(b_i)>θ, δ_i=1 When h_i(b_i) < θ, δ_i=0 Here, when δ_i=1, the pixel is in the candidate area, and when δ_i=0, the pixel is in an area other than the candidate area. When the extraction unit 304 determines that h_i(b_i)<θ using the pixel value bi of pixel i and the threshold θ, it outputs δ_i=0. δ_i=0 indicates that pixel i is a candidate for detection.

[0045] The threshold θ is a threshold for the cumulative amount of the histogram when distinguishing between candidate regions for detection targets and other regions. Here, we will further describe how to determine the threshold θ. Taking advantage of the fact that the transparent container 2 rotates in a fixed direction, the cosine similarity between the container's rotation direction vector and the optical flow of each pixel is calculated. The extraction unit 304 sets the threshold θ so that the threshold increases when the cosine similarity increases and decreases when the cosine similarity decreases. In other words, the extraction unit 304 is more likely to recognize a candidate region for detection targets when the movement vector for each pixel matches the rotation direction of the container, and less likely to recognize it when it does not. This utilizes the fact that detection targets within the transparent container 2, such as foreign objects and bubbles, tend to have optical flow detected in the movement direction more easily than other regions other than the detection target.

[0046] The extraction unit 304 similarly determines whether or not all pixels in one acquired image are candidate regions for detection targets (step S207).The extraction unit 304 also similarly determines whether or not all pixels in each image sequentially generated by the image capture device 10 are candidate regions for detection targets (step S208).

[0047] The difference detection unit 305 filters the group of candidate regions for detection targets obtained by the extraction unit 304 and extracts only pixels in candidate regions for which the luminance values ​​change between previous and next frames (step S209). Because the transparent container 2 is rotating, foreign objects and air bubbles move within the transparent container 2 along with the container. Therefore, when calculating the difference between previous and next frames for regions of foreign objects and air bubbles, a large difference in luminance values ​​is likely to occur. On the other hand, for pixels in regions other than the detection target, the amount of change in luminance values ​​is small even when calculating the difference between previous and next frames. Therefore, the difference detection unit 305 identifies candidate regions for which the luminance values ​​change between frames by calculating the difference between previous and next frames. Note that the extraction process of the candidate region for detection targets by the extraction unit 304 and the filtering process by the difference detection unit 305 can be performed in the same order to achieve the same effect, and they may also be performed in parallel.

[0048] The clustering unit 306 searches for connected pixels of eight neighboring pixels surrounding the pixels of the candidate area to be detected detected in the image processed by the extraction unit 304 and the difference detection unit 305, and if the connected pixels are determined to be the candidate area to be detected, it integrates the candidate areas by clustering and identifies them as a single candidate area (step S210).

[0049] The tracking determination unit 307 sequentially tracks the candidate region identified by the clustering unit 306 between the previous and next frames, and when a sufficiently long trajectory is obtained, determines that it is the trajectory of a detection target such as a foreign object or air bubble.

[0050] The tracking determination unit 307 associates each candidate area c_i∈C in the current frame with the tracking results t_j∈T up to the current frame, and updates the tracking results. Here, C represents the set of candidate areas newly detected in the current frame, and T represents the set of tracking results between the current frames. To update the tracking results, a weighting matrix M is generated, and candidate areas of the same trajectory of the detected object between frames are associated using this weighting matrix M and the Hungarian algorithm.

[0051] Here, the weighting matrix generation unit 401 of the tracking determination unit 307 generates a weighting matrix M based on each weighting information, namely, weighting information α obtained from the new trajectory weighting unit 402, weighting information β obtained from the position weighting unit 403, weighting information γ obtained from the size change weighting unit 404, and weighting information δ obtained from the rotation direction weighting unit 405.

[0052] The new trajectory weighting unit 402 uses the weight information α to strengthen the association between the candidate area in the last image of the consecutive images and the candidate area in the new image following the last image, the longer the continuity of the candidate areas identified as associated areas in multiple consecutive past images. In other words, the new trajectory weighting unit 402 uses the weight information α to update the weights included in the weight matrix M generated by the weight matrix generation unit 401 so that the longer the trajectory length indicated by the tracking result, the higher the priority for connecting the candidate areas in the new image.

[0053] FIG. 13 is a first diagram illustrating an outline of processing performed by the image processing device according to the first embodiment. The position weighting unit 403 uses the weight information β to strengthen the correlation between candidate regions located farther from the center of rotation of the transparent container 2 in the images and the tracking results up to the N-1th frame, even if the candidate regions in the Nth frame are farther apart. The farther an object is located from the center of rotation due to the rotation of the transparent container 2, the longer the distance between its appearance in the previous and next images. Therefore, even if the object is far from the center of rotation and its appearance in the previous and next images is far apart, it is desirable to determine that its trajectory is connected. Therefore, the position weighting unit 403 uses the weight information β to update the weights included in the weight matrix M generated by the weight matrix generation unit 401 so that candidate regions located farther from the center of rotation of the transparent container 2 can be preferentially linked to the tracking results.

[0054] FIG. 14 is a second diagram illustrating the outline of processing performed by the image processing device according to the first embodiment. The size change weighting unit 404 uses the weight information γ to weaken the correlation between the candidate regions in the previous and next images as the shape of a candidate region containing multiple pixels changes significantly. When the shape of a candidate region changes significantly, the likelihood of the candidate regions in the previous and next images becoming less related increases. Therefore, the size change weighting unit 404 uses the weight information γ to update the weights to weaken the correlation between the candidate region in the tracking result in the N-1th frame and the candidate region in the Nth frame when the difference between the number of pixels in the candidate region in the tracking result in the N-1th frame and the number of pixels in the candidate region in the Nth frame is greater than a predetermined threshold. In Figure 14(a), the shape of the candidate region changes little, so the correlation between the previous and next images is high. On the other hand, in Figure 14(b), the total change in the candidate region is large, so the correlation between the previous and next images is low.

[0055] FIG. 15 is a third diagram illustrating the outline of processing performed by the image processing device according to the first embodiment. The rotation direction weighting unit 405 uses the weight information δ to weaken the correlation between candidate regions that move against the direction of rotation of the transparent container 2 in each of the previous and next images. That is, the rotation direction weighting unit 405 uses the weight information δ to calculate the cosine similarity between the centroid vector from the image position of the candidate region in the N-1th frame to the image position of each candidate region in the Nth frame and the vector in the rotation direction of the container, and updates the weight to exclude regions with low similarity. In Figure 15(a), the direction of the rotation direction vector (solid line) and the direction of movement vector (dotted line) of the centroid of the candidate region to be detected are similar, so it is highly likely to be a candidate region to be detected. On the other hand, in Figure 15(b), the direction of the rotation direction vector (solid line) and the direction of movement vector (dotted line) of the centroid of the candidate region to be detected are not similar, so it is unlikely to be a candidate region to be detected.

[0056] The weighting matrix generation unit 401 generates a weighting matrix M for each tracking result up to the N-1th frame and each candidate area in the Nth frame. An element m_ij∈M of the weighting matrix M for the candidate area c_i∈C appearing in the Nth frame and the tracking result t_j∈T up to the N-1th frame can be expressed by equation (1). In equation (1), "*" indicates multiplication.

[0057] m_ij = (λ_1*α+λ_2*β+λ_3*γ+λ_4*δ)D_ij ···(1)

[0058] Here, D_ij represents the L2 norm of the centroid coordinates of the latest candidate region that constitutes cluster c_i and tracking result t_j. Weight information α, β, γ, and δ represent weights generated by the new trajectory weighting unit 402, position weighting unit 403, size change weighting unit 404, and rotation direction weighting unit 405, and λ_1, λ_2, λ_3, and λ_4 represent weights that adjust them.

[0059] The link determination unit 406 of the tracking determination unit 307 then uses the weight matrix M and the Hungarian algorithm to assign candidate areas for the Nth frame to the tracking results for the N-1th frame (step S211). This classifies each candidate area in the Nth frame into one that is not included in the tracking trajectory and one that is included in the tracking trajectory. The processing by the tracking determination unit 307 is one aspect of processing that identifies candidate areas in previous and next images captured in chronological order among multiple images as related areas using weight information (weight matrix M) indicating the strength of continuity between the candidate areas in those images. The link determination unit 406 records pixel information for candidate areas not included in the tracking trajectory in the image of the Nth frame in the unused area storage unit 407. For example, the link determination unit 406 associates the identifier of the image of the Nth frame with the identifier of the candidate area not included in the tracking trajectory and image information constituting the candidate area, and records them in the unused area storage unit 407. The connection determination unit 406 records pixel information of a candidate area included in the trajectory being tracked in the image of the Nth frame in the existing trajectory storage unit 408. For example, the connection determination unit 406 associates an identifier of the image of the Nth frame with an identifier of a candidate area included in the trajectory being tracked, image information constituting the candidate area, and the like, and records them in the existing trajectory storage unit 408.

[0060] The trajectory length determination unit 409 identifies candidate regions from the existing trajectory storage unit 408 that have a concatenation length equal to or greater than a predetermined threshold as candidate regions for detection of foreign matter, air bubbles, etc. (step S212). For example, the trajectory length determination unit 409 calculates the distance from the position of the first candidate region indicating the trajectory (e.g., the first frame image) to the position of the candidate region in the last image (e.g., the Nth frame image) based on the identifiers of the images of N frames recorded in the existing trajectory storage unit 408, the identifiers of the candidate regions included in the trajectory being tracked, and image information (such as pixel positions) that constitute the candidate regions. If the distance is equal to or greater than the threshold, the trajectory length determination unit 409 identifies the candidate regions indicating the trajectory detected in each of the images from the first frame to the Nth frame as candidate regions for detection of foreign matter, air bubbles, etc.

[0061] The identification unit 308 acquires N frames of images containing information on one or more candidate regions of the detection target identified by the tracking determination unit 307 through the above process. The information on the candidate regions includes, for example, information indicating pixels in the images. The identification unit 308 performs the following process to determine whether the target object is present in the transparent container 2 using the candidate regions of the detection target identified by the tracking determination unit 307 in each of the N frames of images. In this process, the identification unit 308 generates tracking time-series data using the positions (image coordinates) of the candidate regions of the detection target in the N frames of images. Alternatively, the tracking time-series data may be data generated based on previous and subsequent images among the N frames of images generated by the link determination unit 406 of the tracking determination unit 307 during the link determination process. As described above, the tracking time-series data may include, for example, information such as the identifiers, positions (image coordinates), and number of pixels of pixels indicating the target object in the previous and subsequent images in chronological order, the amount of movement based on the change in the position of the region in the previous and subsequent image frames, and the movement vector. One piece of tracking time-series data may be generated for N frames of images, or tracking time-series data may be generated for each N frames of images.

[0062] The classifier 506 of the identification unit 308 generates square patch images each including a candidate region centered on the candidate region in each image of N frames. Based on each image of N frames and the tracking time-series data, the classifier 506 selects, from among the patch images corresponding to one trajectory in each image of N frames, the patch image with the smallest candidate region, the patch image with the largest candidate region, and the patch image whose candidate region range is closest to the average among the patch images of that trajectory. The classifier 506 inputs the selected patch images into a first learning model. As a result, the classifier 506 outputs a first determination result for each input patch image indicating whether the candidate region in the patch image is a detection target (step S213). The first determination result may be information indicating whether the patch image corresponding to one or more candidate regions included in the image is a detection target, such as a foreign object, a bubble, or another background region. The first determination result may also be information indicating whether the patch image corresponding to one or more candidate regions included in the image matches the correct detection target used as learning data. The classifier 506 similarly generates patch images indicating other trajectories in each image of the N frames based on the tracking time-series data, inputs each patch image selected in each trajectory into the first learning model, and similarly outputs a first determination result indicating whether the candidate area in those patch images is a detection target. If the classifier 506 outputs the first determination result indicating that the patch image is a detection target, the classifier 506 records information indicating that the transparent container 2 contains a foreign object that is a detection target in the identification result storage unit 508. The classifier 506 may record the patch image determined to be a detection target in the identification result storage unit 508 by linking it to the identifier of the transparent container 2.

[0063] Furthermore, the classifier 506 of the identification unit 308 selects tracking time series data corresponding to one trajectory in each image of the N frames. The classifier 506 inputs the selected tracking time series data to the second learning model. As a result, the classifier 506 outputs a second determination result indicating whether or not the input tracking time series data is a detection target (step S214). The classifier 506 inputs tracking time series data of other trajectories in each image of the N frames to the second learning model, and similarly outputs a second determination result indicating whether or not the trajectories are a detection target. If the classifier 506 outputs the tracking time series data as a detection target in the second determination result, it records information indicating that the transparent container 2 contains a foreign object that is a detection target in the identification result storage unit 508.

[0064] According to the above-described process, foreign objects that are stuck in positions that are difficult to see due to the influence of light refraction, such as the liquid surface or bottom surface, are rotated together with the transparent container 2, and by determining whether the movement characteristics of the detection object, such as foreign objects that move due to the rotation, are characteristics of movement that follow the rotation, it is possible to more accurately detect detection objects, such as foreign objects that are stuck near the liquid surface or bottom surface.

[0065] Second Embodiment

[0066] FIG. 16 shows a fourth perspective view of the foreign object detection device 100. As shown in FIG. 16, the foreign matter detection device 100 is composed of an image processing device 1, a transparent container 2 to be inspected, a rotation device 4 that rotates the transparent container 2, a transmitted light 6 that illuminates the liquid 3 inside the transparent container 2, and a rocking device 7 that rocks the transparent container 2. The transparent container 2 contains the liquid 3.

[0067] In the second embodiment, the transparent container 2 has a cylindrical shape, and the medium contained in the transparent container 2 is a liquid 3. A foreign object to be detected resides on the bottom surface of the liquid 3, and the image processing device 1 determines the presence or absence of the foreign object based on an image including the foreign object remaining on the bottom surface of the liquid 3. The foreign object to be detected resides on the surface of the liquid 3, and the image processing device 1 may determine the presence or absence of the foreign object based on an image including the foreign object remaining on the surface of the liquid 3. The image processing device 1 may also determine the type of the detection object (foreign object, air bubble, etc.). The image processing device 1 may detect a detection object remaining in a medium other than the liquid contained in the transparent container 2. In other words, the image processing device 1 may detect the detection object from an image that shows the detection object remaining on the bottom surface of the transparent container 2 or the detection object remaining on a surface where the medium contained in the transparent container 2 comes into contact with another medium within the transparent container 2. The image processing device 1 may detect the detection target from an image that shows the detection target remaining on the bottom surface of the transparent container 2, or may detect the detection target from an image that shows the detection target remaining on the top surface of the transparent container 2.

[0068] The rotation device 4 holds the cylindrical transparent container 2 by its bottom while holding it vertically, and rotates it around the center of the bottom (z-axis) as the rotation axis, thereby rotating the transparent container 2. The detection target, which may be on the surface, bottom, or suspended in the liquid 3 contained in the transparent container 2, rotates around the rotation axis as the transparent container 2 rotates. In this embodiment, the image processing device 1 includes a photographing device 10. The photographing device 10 is installed in a position where it can photograph the detection target, and where the photographing direction is fixed to the photographing axis (x-axis) perpendicular to the z-axis. While the rotation device 4 rotates the transparent container 2, the image processing device 1 controls the photographing device 10 to photograph the detection target and continuously acquire multiple images with a field of view that captures the detection target. The relationship between the z-axis and the x-axis is maintained while the photographing device 10 is capturing images. As a result, the image processing device 1 acquires multiple images including the detection target, whose position changes as the rotation device 4 rotates. In the second embodiment, the reflected illumination 5 may be used instead of the transmitted illumination 6, or both may be used.

[0069] The structural difference between the foreign object detection device 100 of this embodiment and that of the first embodiment is that the foreign object detection device 100 of the second embodiment is equipped with a rocking device 7 that rocks the transparent container 2 before it is rotated by the rotation device 4. In the second embodiment, the rocking device 7 grips the upper part of the transparent container 2, for example, and rocks the transparent container 2 from side to side like a pendulum. This moves the position of the detection target, such as foreign objects remaining in the liquid inside the rocking device 7, on the liquid surface, or at the bottom of the liquid. The image processing device 1 of the second embodiment detects the detection target using multiple images of the transparent container 2 being rotated by the rotation device 4 before it is rocked by the rocking device 7, and multiple images of the transparent container 2 being rotated by the rotation device 4 after it has been rocked by the rocking device 7.

[0070] FIG. 17 is a second diagram showing the functional blocks of the image processing device. 17, the image processing device 1 executes a pre-stored image processing program, thereby enabling the image processing device 1 to perform the functions of a photographing unit 701, an image storage unit 702, a pattern searching unit 703, an extraction unit 704, a clustering unit 706, a tracking determination unit 707, and an identification unit 708.

[0071] The photographing unit 701 controls the photographing device 10 to acquire an image generated by the photographing device 10 . The image storage unit 702 stores the images acquired by the photographing unit 701 from the photographing device 10 . The pattern search unit 703 performs processing to match patterns that change due to the influence of light reflections and the like in the image between the image before and after the swing. The extraction unit 704 extracts candidate regions for the detection target using the images before and after the shaking, which have matching patterns. The clustering unit 706 clusters groups of adjacent pixels among the pixels remaining as candidate regions into one candidate region. The tracking determination unit 707 identifies, from among the clustered candidate regions (detection windows), related regions before and after the movement in response to the rotation of the transparent container, for each pixel of a plurality of images taken at different times. The classification unit 708 identifies the detection target using a first determination result obtained by determining whether or not the candidate area is the detection target using image information of the candidate area moving in the movement direction according to the rotation and a first learning model, and a second determination result obtained by determining whether or not the candidate area is the detection target using information indicating time-series changes in the candidate area moving in the movement direction according to the rotation and a second learning model. Identifying the detection target is one aspect of the process of determining the presence or absence of the detection target.

[0072] In the second embodiment, the functions of the clustering unit 706, the tracking determination unit 707, and the identification unit 708 are the same as those in the first embodiment.

[0073] FIG. 18 is a diagram showing a processing flow of the image processing apparatus according to the second embodiment. FIG. 19 is a diagram showing an outline of processing performed by the image processing apparatus according to the second embodiment. Next, a description will be given of the processing of the second embodiment when the image processing device 1 actually identifies the detection target. Note that, similar to the processing described in the first embodiment, it is assumed that the first learning model and the second learning model have been generated.

[0074] First, the rotation device 4 rotates the transparent container 2 around the z-axis. In this state, the photographing unit 701 of the image processing device 1 controls the photographing device 10 to continue photographing. The photographing unit 701 sequentially acquires images (frames) generated by the photographing device 10 (step S301). The photographing unit 701 sequentially records each acquired image in the image storage unit 702 (step S302). Each of these recorded images is an image before shaking.

[0075] Next, under the control of the image processing device 1, the rocking device 7 swings the transparent container 2 in a panning direction (left and right) with the upper part as a fulcrum (step S303). This causes the transparent container 2 to swing in the panning direction (left and right) when viewed from the camera 10. Alternatively, an administrator may manually swing the transparent container 2 in the panning direction. Then, the rotation device 4 again rotates the transparent container 2 around the z-axis (step S304). The camera unit 701 of the image processing device 1 controls the camera 10 to continue shooting. The camera unit 701 sequentially acquires images (frames) generated by the camera 10 (step S305). Each of these images is an image after swinging.

[0076] The pattern search unit 703 of the image processing device 1 compares pixel areas in each image before shaking (FIG. 19(a)) and each image after shaking (FIG. 19(b)) where the brightness is above a predetermined threshold due to the influence of light, and if they match, determines that the positions as seen from the photographing device 10 due to the rotation of the transparent container 2 are the same rotation positions (step S306). Note that the pattern search unit 703 compares pixel areas in each image before shaking and each image after shaking where the brightness is above a predetermined threshold due to the influence of light, and may determine that the positions are the same if the areas match to a certain extent. The pattern search unit 703 identifies the matching image before shaking and image after shaking as origin images for extraction of candidate areas for detection targets (step S307).

[0077] The extraction unit 704 compares the image before shaking that the pattern search unit 703 determined to be the origin image with the image after shaking, and extracts areas with different pixel values ​​as candidate areas for detection (FIG. 19(c)) by calculating the pixel differences between the images (step S308). This allows the candidate areas for detection to be distinguished from other background areas. The processing by the extraction unit 704 is one aspect of processing in which areas with similar light reflection are identified based on the pixel differences between a first image of the transparent container 2 captured by the image capture device 10 and a second image of the transparent container 2 captured by the image capture device 10 after shaking the transparent container 2, and a candidate area for detection with changes in pixel values ​​is extracted based on the difference between the first image and the second image in which the areas with similar light reflection match.

[0078] The clustering unit 706 searches for connected pixels of eight neighboring pixels surrounding the pixels of the candidate area to be detected detected in the image processed by the extraction unit 704, and if the connected pixels are determined to be the candidate area to be detected, it integrates the candidate areas by clustering and identifies them as a single candidate area (step S309).

[0079] The tracking determination unit 707 sequentially tracks the candidate region identified by the clustering unit 706 between the previous and next frames, and when a sufficiently long trajectory is obtained, determines that it is the trajectory of a detection target such as a foreign object or air bubble.

[0080] The tracking determination unit 707 associates each candidate area c_i∈C in the current frame with the tracking results t_j∈T up to the current frame, and updates the tracking results. Here, C represents the set of candidate areas newly detected in the current frame, and T represents the set of tracking results between the current frames. To update the tracking results, a weighting matrix M is generated, and candidate areas of the same trajectory of the detected object between frames are associated using this weighting matrix M and the Hungarian algorithm.

[0081] Here, the weighting matrix generation unit 401 of the tracking determination unit 707 generates a weighting matrix M, similar to the first embodiment, based on each weighting information, namely, weighting information α obtained from the new trajectory weighting unit 402, weighting information β obtained from the position weighting unit 403, weighting information γ obtained from the size change weighting unit 404, and weighting information δ obtained from the rotation direction weighting unit 405.

[0082] The weighting matrix generation unit 401 generates a weighting matrix M for each tracking result up to the N-1th frame and each candidate area in the Nth frame. An element m_ij∈M of the weighting matrix M for the candidate area c_i∈C appearing in the Nth frame and the tracking result t_j∈T up to the N-1th frame can be expressed by equation (2). In equation (1), "*" indicates multiplication.

[0083] m_ij = (λ_1*α+λ_2*β+λ_3*γ+λ_4*δ)D_ij ···(2)

[0084] Here, D_ij represents the L2 norm of the centroid coordinates of the latest candidate region that constitutes cluster c_i and tracking result t_j. Weight information α, β, γ, and δ represent weights generated by the new trajectory weighting unit 402, position weighting unit 403, size change weighting unit 404, and rotation direction weighting unit 405, and λ_1, λ_2, λ_3, and λ_4 represent weights that adjust them.

[0085] The connection determination unit 406 of the tracking determination unit 707 then uses the weight matrix M and the Hungarian algorithm to assign candidate areas for the Nth frame to the tracking results for the N-1th frame (step S310). As a result, the candidate areas included in the Nth frame are classified into candidate areas not included in the tracking trajectory and candidate areas included in the tracking trajectory. The connection determination unit 406 records pixel information for candidate areas not included in the tracking trajectory in the image of the Nth frame in the unused area storage unit 407. For example, the connection determination unit 406 associates the identifier of the image of the Nth frame with the identifier of a candidate area not included in the tracking trajectory and image information constituting the candidate area, and records them in the unused area storage unit 407. The connection determination unit 406 records pixel information for candidate areas included in the tracking trajectory in the image of the Nth frame in the existing trajectory storage unit 408. For example, the link determination unit 406 links the identifier of the image of the Nth frame with the identifier of the candidate area included in the trajectory being tracked and image information that constitutes the candidate area, and records them in the existing trajectory storage unit 408 .

[0086] The trajectory length determination unit 409 identifies candidate regions from the existing trajectory storage unit 408 that have a concatenation length equal to or greater than a predetermined threshold as candidate regions for detection of foreign matter, air bubbles, etc. (step S311). For example, the trajectory length determination unit 409 calculates the distance from the position of the first candidate region indicating the trajectory (e.g., the first frame image) to the position of the candidate region in the last image (e.g., the Nth frame image) based on the identifiers of the images of N frames recorded in the existing trajectory storage unit 408, the identifiers of the candidate regions included in the trajectory being tracked, and image information (such as pixel positions) that constitute the candidate regions. If the distance is equal to or greater than the threshold, the trajectory length determination unit 409 identifies the candidate regions indicating the trajectory detected in each of the images from the first frame to the Nth frame as candidate regions for detection of foreign matter, air bubbles, etc.

[0087] The identification unit 708 acquires N frames of images containing information on one or more candidate regions for the detection target identified by the tracking determination unit 707 through the above process. The information on the candidate regions includes, for example, information indicating pixels in the images. The identification unit 708 performs the following process to determine whether the target object is present in the transparent container 2 using the candidate regions for the detection target identified by the tracking determination unit 707 in each of the N frames of images. In this process, the identification unit 708 generates tracking time-series data using the positions (image coordinates) of the candidate regions for the detection target in the N frames of images. Alternatively, the tracking time-series data may be data generated based on previous and subsequent images among the N frames of images generated by the link determination unit 406 of the tracking determination unit 707 during the link determination process. As described above, the tracking time-series data may include, for example, information such as the identifiers, positions (image coordinates), and number of pixels of pixels indicating the target object in the previous and subsequent images in chronological order, the amount of movement based on the change in the position of the region in the previous and subsequent image frames, and the movement vector. One piece of tracking time-series data may be generated for N frames of images, or tracking time-series data may be generated for each N frames of images.

[0088] The classifier 506 of the identification unit 708 generates square patch images each including a candidate region centered on the candidate region in each image of N frames. Based on each image of N frames and the tracking time-series data, the classifier 506 selects, from among the patch images corresponding to one trajectory in each image of N frames, the patch image with the smallest candidate region, the patch image with the largest candidate region, and the patch image whose candidate region range is closest to the average among the patch images of that trajectory. The classifier 506 inputs the selected patch images into a first learning model. As a result, the classifier 506 outputs a first determination result for each input patch image indicating whether the candidate region in the patch image is a detection target (step S312). The first determination result may be information indicating whether the patch image corresponding to one or more candidate regions included in the image is a detection target, such as a foreign object, a bubble, or another background region. The first determination result may also be information indicating whether the patch image corresponding to one or more candidate regions included in the image matches the correct detection target used as learning data. The classifier 506 similarly generates patch images indicating other trajectories in each image of the N frames based on the tracking time-series data, inputs each patch image selected in each trajectory into the first learning model, and similarly outputs a first determination result indicating whether the candidate area in those patch images is a detection target. If the classifier 506 outputs the first determination result indicating that the patch image is a detection target, the classifier 506 records information indicating that the transparent container 2 contains a foreign object that is a detection target in the identification result storage unit 508. The classifier 506 may record the patch image determined to be a detection target in the identification result storage unit 508 by linking it to the identifier of the transparent container 2.

[0089] Furthermore, the classifier 506 of the identification unit 708 selects tracking time series data corresponding to one trajectory in each image of the N frames. The classifier 506 inputs the selected tracking time series data to the second learning model. As a result, the classifier 506 outputs a second determination result indicating whether or not the input tracking time series data is a detection target (step S313). The classifier 506 inputs tracking time series data of other trajectories in each image of the N frames to the second learning model, and similarly outputs a second determination result indicating whether or not the trajectories are a detection target. If the classifier 506 outputs the second determination result indicating that the tracking time series data is a detection target, it records information indicating that the transparent container 2 contains a foreign object that is a detection target in the identification result storage unit 508.

[0090] According to the above-described process, foreign objects that are stuck in positions that are difficult to see due to the influence of light refraction, such as on the liquid surface or bottom surface, are rotated together with the transparent container 2, and by determining whether the movement characteristics of the detection object, such as a foreign object that moves due to the rotation, are characteristics of movement that follow the rotation, it is possible to more accurately detect detection objects, such as foreign objects that are stuck near the liquid surface or bottom surface.

[0091] FIG. 20 is a diagram showing the minimum configuration of an image processing device. FIG. 21 is a diagram showing a processing flow in an image processing device with a minimum configuration. The image processing device 1 includes at least a first determination means and a second determination means. The first determination means compares multiple images that show the detection target inside the transparent container 2 and are taken by rotating the transparent container 2, and determines, from among the candidate areas for the detection target shown in the images, the candidate areas that move in the direction of movement corresponding to the rotation (step S2101). The second determination means determines whether or not the candidate area is the detection target using a first determination result obtained by determining whether or not the candidate area is the detection target using image information of the candidate area moving in the movement direction according to the rotation and a first learning model, and a second determination result obtained by determining whether or not the candidate area is the detection target using information indicating the time series changes in the candidate area moving in the movement direction according to the rotation and a second learning model (step S2102).

[0092] The image processing device 1 described above has an internal computer system. The steps of each of the above-described processes are stored in the form of a program on a computer-readable storage medium, and the computer reads and executes this program to perform the above-described processes. Here, the computer-readable storage medium refers to a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, etc. Alternatively, the computer program may be distributed to a computer via a communication line, and the computer that receives the program may execute the program.

[0093] The program may also be a program for realizing some of the functions described above, or may be a so-called differential file (differential program) that can realize the functions described above in combination with a program already recorded in the computer system. [Industrial Applicability]

[0094] The present invention can be applied to applications such as automatically and quickly inspecting whether foreign matter has been mixed into liquids such as medical drugs, as opposed to manual inspections, and can also be used to inspect whether foreign matter has been mixed into beverages during production.

[0095] Note that part or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0096] (Appendix 1) an image showing a detection target inside a transparent container, the image being taken while the transparent container is rotated; comparing a plurality of images taken while the transparent container is rotated; and determining, among candidate areas of the detection target shown in the images, the candidate area that moves in a movement direction corresponding to the rotation; The presence or absence of the detection target is determined using a first determination result obtained by determining whether or not the candidate area is the detection target using image information of the candidate area moving in the movement direction according to the rotation and a first learning model, and a second determination result obtained by determining whether or not the candidate area is the detection target using information indicating time-series changes in the candidate area moving in the movement direction according to the rotation and a second learning model. Image processing device.

[0097] (Appendix 2) an extraction means for comparing a reference histogram generated based on pixel values ​​of each pixel of the plurality of images previously created by fixing the image capturing device and rotating the transparent container with the pixel values ​​of each pixel of the newly acquired image, and extracting a candidate region of the detection target in the newly acquired image; 2. The image processing device according to claim 1, comprising:

[0098] (Appendix 3) an extraction means for fixing the image capturing device, identifying areas with similar light reflection based on the difference between each pixel of a first image of the transparent container captured by the capturing device and a second image of the transparent container captured by the capturing device after shaking the transparent container, and extracting the candidate area of ​​the detection target where the pixel value has changed based on the difference between the plateau and the second image where the area with similar light reflection matches; 3. The image processing device according to claim 2, comprising:

[0099] (Appendix 4) a tracking determination means for determining a related area in the plurality of pixels according to a movement of the candidate area based on a rotation of the transparent container; The tracking determination means identifies the candidate areas in the previous and next images as the related areas using weight information indicating the strength of continuity of the candidate areas in the previous and next images taken in time series among the plurality of images and the candidate areas in the previous and next images. 4. An image processing device according to any one of claims 1 to 3.

[0100] (Appendix 5) The weight information is information that, when the continuity of the candidate regions identified as the relevant regions in a plurality of consecutive past images is long, strengthens the association between the candidate region in the last image of the consecutive past images and the candidate region in a new image following the last image. 5. The image processing device according to claim 4.

[0101] (Appendix 6) The weight information is information that strengthens the association between a candidate region located at a position farther from the center of rotation of the transparent container shown in the image, even if the candidate regions in the previous and next images are far apart. 6. The image processing device according to claim 4 or 5.

[0102] (Appendix 7) The weight information is information that weakens the association of the candidate regions in the preceding and following images based on a large change in the shape of the candidate region that includes a plurality of pixels. 7. An image processing device according to any one of claims 4 to 6.

[0103] (Appendix 8) The weight information is information that weakens the association of the candidate region in each of the front and rear images that move against the direction of the rotation of the transparent container. 8. An image processing device according to any one of claims 4 to 7.

[0104] (Appendix 9) The image shows a detection target that is staying on a curved portion of the wall surface of the transparent container, or a detection target that is staying on a surface where a medium contained in the transparent container comes into contact with another medium inside the transparent container. 9. An image processing device according to any one of claims 1 to 8.

[0105] (Appendix 10) an image showing a detection target inside a transparent container, the image being taken while the transparent container is rotated; comparing a plurality of images taken while the transparent container is rotated; and determining, among candidate areas of the detection target shown in the images, the candidate area that moves in a movement direction corresponding to the rotation; The presence or absence of the detection target is determined using a first determination result obtained by determining whether or not the candidate area is the detection target using image information of the candidate area moving in the movement direction according to the rotation and a first learning model, and a second determination result obtained by determining whether or not the candidate area is the detection target using information indicating time-series changes in the candidate area moving in the movement direction according to the rotation and a second learning model. Image processing methods.

[0106] (Appendix 11) The computer of the image processing device A process of comparing a plurality of images of the detection target inside the transparent container taken while rotating the transparent container, and determining a candidate area of ​​the detection target shown in the image that moves in a movement direction corresponding to the rotation; a process of determining whether or not the detection target is present using a first determination result obtained by determining whether or not the candidate area is the detection target using image information of the candidate area moving in a movement direction according to the rotation and a first learning model, and a second determination result obtained by determining whether or not the candidate area is the detection target using information indicating time-series changes in the candidate area moving in a movement direction according to the rotation and a second learning model; A storage medium that stores a program that executes the above. [Explanation of symbols]

[0107] 1. Image processing device 2...Transparent container 4. Rotating device 5. Reflected lighting 6. Transmitted illumination 7. Oscillating device 301,701···Photography Department 302 Histogram generation unit 303 Histogram memory 304,704...Extraction part 305,705...Difference detection section 306,706···Clustering section 307,707 Tracking decision section 308,708···Identification unit 702 Image storage unit 703 Pattern search section 401...Weight matrix generation unit 402...New trajectory weighting unit 403 Position weighting unit 404...Size change weighting section 405 Rotational direction weighting section 406...Connection determination section 407...Unused area storage section 408 Existing trajectory memory section 409...Trajectory length determination section 501: Pseudo texture memory unit 502: Synthesis target image generation unit 503 Tracking time series data storage unit 504 Actual image storage unit 505: Synthesis target image storage unit 506...Classifier 507···Learning Department 508: Classification result storage unit

Claims

1. an image showing a detection target inside a transparent container, the image being taken while the transparent container is rotated; comparing a plurality of images taken while the transparent container is rotated; and determining, among candidate areas of the detection target shown in the images, the candidate area that moves in a movement direction corresponding to the rotation; The presence or absence of the detection target is determined using a first determination result obtained by determining whether or not the candidate area is the detection target using image information of the candidate area moving in the movement direction according to the rotation and a first learning model, and a second determination result obtained by determining whether or not the candidate area is the detection target using information indicating time-series changes in the candidate area moving in the movement direction according to the rotation and a second learning model. Image processing device.

2. an extraction means for comparing a reference histogram generated based on pixel values ​​of each pixel of the plurality of images previously created by fixing the image capturing device and rotating the transparent container with the pixel values ​​of each pixel of the newly acquired image, and extracting a candidate region of the detection target in the newly acquired image; The image processing device according to claim 1 , comprising:

3. an extraction means for fixing the image capturing device, identifying areas with similar light reflection based on the difference between each pixel of a first image of the transparent container captured by the capturing device and a second image of the transparent container captured by the capturing device after shaking the transparent container, and extracting the candidate area of ​​the detection target where the pixel value has changed based on the difference between the first image and the second image where the areas with similar light reflection match; The image processing device according to claim 2 , comprising:

4. a tracking determination means for identifying a related area in the plurality of images according to a movement of the candidate area based on a rotation of the transparent container; The tracking determination means identifies the candidate areas in the previous and next images as the related areas using weight information indicating the strength of continuity of the candidate areas in the previous and next images taken in time series among the plurality of images and the candidate areas in the previous and next images. The image processing device according to any one of claims 1 to 3.

5. The weight information is information that, when the continuity of candidate regions identified as the relevant regions in a plurality of consecutive past images is long, strengthens the association between the candidate region in the last image of the consecutive past images and the candidate region in a new image following the last image. The image processing device according to claim 4 .

6. The weight information is information that strengthens the association between a candidate region located at a position farther from the center of rotation of the transparent container shown in the image, even if the candidate regions in the previous and next images are far apart.

6. The image processing device according to claim 4 or claim 5.

7. The weight information is information that weakens the association of the candidate regions in the preceding and following images based on a large change in the shape of the candidate region that includes a plurality of pixels. The image processing device according to any one of claims 4 to 6.

8. The weight information is information that weakens the association of the candidate region in each of the front and rear images that move against the direction of the rotation of the transparent container. The image processing device according to any one of claims 4 to 7.

9. The image is an image of a detection target that is staying on a curved portion of the wall surface of the transparent container, or a detection target that is staying on a surface where a medium contained in the transparent container comes into contact with another medium inside the transparent container. The image processing device according to any one of claims 1 to 8.

10. an image showing a detection target inside a transparent container, the image being taken while the transparent container is rotated; comparing a plurality of images taken while the transparent container is rotated; and determining, among candidate areas of the detection target shown in the images, the candidate area that moves in a movement direction corresponding to the rotation; The presence or absence of the detection target is determined using a first determination result obtained by determining whether or not the candidate area is the detection target using image information of the candidate area moving in the movement direction according to the rotation and a first learning model, and a second determination result obtained by determining whether or not the candidate area is the detection target using information indicating time-series changes in the candidate area moving in the movement direction according to the rotation and a second learning model. Image processing methods.

11. The computer of the image processing device A process of comparing a plurality of images of the detection target inside the transparent container taken while rotating the transparent container, and determining a candidate area of ​​the detection target shown in the image that moves in a movement direction corresponding to the rotation; a process of determining whether or not the detection target is present using a first determination result obtained by determining whether or not the candidate area is the detection target using image information of the candidate area moving in a movement direction according to the rotation and a first learning model, and a second determination result obtained by determining whether or not the candidate area is the detection target using information indicating time-series changes in the candidate area moving in a movement direction according to the rotation and a second learning model; A program that executes the following.

Images