Image processing device and image processing method

Abstract

To provide an image processing device and an image processing method.SOLUTION: In a target operating system 11 comprising a camera 12, an image processing device 10, and an operating apparatus 13, the image processing device 10 has a memory 17, a communication unit 16, and a control unit 18. The memory stores a template image group including a plurality of images of detection objects 14. The communication unit acquires an entire image including a collation object from the camera 12. The control unit acquires a representative image based on a part of the template image group. The control unit determines the similarity between at least part of the entire image and the representative image.SELECTED DRAWING: Figure 1

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to Patent Application No. 2021-139428, filed in Japan on August 27, 2021, the entire disclosure of which is incorporated herein by reference. [Technical Field]

[0002] The present disclosure relates to an image processing device and an image processing method. [Background technology]

[0003] It is necessary to recognize the position of a detection target within an arbitrary area from an image. For example, in a factory where detection targets are manufactured, when a robot is required to operate the detection targets placed on a tray, it is necessary to recognize the position of the detection target.

[0004] Template matching is known, in which an image of a detection target is used as a template image to detect the position of the detection target within an image of an arbitrary region. The appearance of a detection target placed in an arbitrary region may vary from a fixed template image due to various disturbance factors, such as changes in sunlight illuminance, the position of a camera capturing an image of the arbitrary region, and manual handling of the detection target. Therefore, template matching that is robust against disturbance factors has been proposed (see Non-Patent Documents 1 to 3). [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Masataka Saito and Manabu Hashimoto, "Robust Image Matching against Illumination Changes Using Stable Pixels Based on Gray Co-occurrence Analysis," Information Processing Society of Japan Technical Report, Vol. 2012-CVIM-182, No. 17, 2012 [Non-patent document 2] Masataka Saito and Manabu Hashimoto, "Robust Image Matching Based on Disturbance Pixel Estimation Using Statistical Intensity Fluctuation Analysis," Vision Technology Practical Use Workshop (ViEW2013), IS2-d8 [Non-patent document 3] Shinohara, Nobuyuki, and Hashimoto, Manabu, "Updated Template Matching Based on Statistical Disturbance Pixel Estimation," Journal of the Japan Society for Precision Engineering, Vol. 83, No. 12, pp. 11178-1183, 2017 Summary of the Invention

[0006] An image processing device according to a first aspect comprises: a memory for storing a template image group including a plurality of images of the detection target; an acquisition unit that acquires an entire image including a matching target; The image processing device further includes a control unit that acquires a representative image based on a portion of the template image group and determines whether at least a portion of the entire image is similar to the representative image.

[0007] An image processing device according to a second aspect comprises: a memory for storing a template image group including a plurality of images of the detection target; an acquisition unit that acquires an entire image including a matching target; a control unit that acquires a representative image based on a part of the template image group and determines the similarity between at least a part of the entire image and the representative image, The control unit calculates a contribution rate of a portion of the entire image used in the similarity determination based on statistics of at least a portion of the template image group, and uses the contribution rate for the similarity determination.

[0008] An image processing method according to a third aspect comprises: storing a template image group including a plurality of images of the detection target; Acquiring an entire image including a matching target; obtaining a representative image based on a portion of the set of template images; and determining whether at least a portion of the entire image is similar to the representative image. [Brief explanation of the drawings]

[0009] [Figure 1]FIG. 1 is a diagram showing an example of the configuration of an object manipulation system including an image processing device according to the first embodiment. [Figure 2] 1. FIG. 4 is a diagram for explaining a similarity determination between a representative image and a partial region of the entire image, which is performed by the control unit in FIG. 1 to detect the position of the detection target in the entire image. [Figure 3] 1. FIG. 4 is a diagram for explaining a method for creating a statistical model for the control unit in FIG. 1 to calculate the degree of contribution. [Figure 4] 1. FIG. 4 is a diagram for explaining a method for creating another statistical model for the control unit in FIG. 1 to calculate the degree of contribution. [Figure 5] 1. FIG. 6 is a diagram for explaining a method for creating yet another statistical model for the control unit in FIG. 1 to calculate the degree of contribution. [Figure 6] FIG. 4 is a diagram for explaining a method of calculating a contribution rate based on the statistical model of FIG. 3. [Figure 7] 4 is a flowchart for explaining a location detection process executed by a control unit in the first embodiment. [Figure 8] 10 is a first flowchart illustrating a location detection process executed by a control unit in a second embodiment. [Figure 9] 10 is a first flowchart illustrating a location detection process executed by a control unit in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of an image processing device to which the present disclosure is applied will be described with reference to the drawings.

[0011] 1 shows an example of the configuration of an object manipulation system 11 including an image processing device 10 according to the first embodiment. The object manipulation system 11 may include a camera 12, the image processing device 10, and a manipulation device 13.

[0012] The object manipulation system 11 performs a specific operation on a detection target 14, such as a manufactured product, in a factory, for example. The detection target 14 is placed on a tray 15, for example. The camera 12 may capture an image of the tray 15 on which the detection target 14 is placed. The camera 12 may generate images at a predetermined frame rate, such as 30 fps, in other words, periodically. The image processing device 10 may detect the position of the detection target 14 using the image captured by the camera 12. The manipulation device 13 may perform an operation on the detection target 14 at the position detected by the image processing device 10. The manipulation device 13 may be, for example, a robot, and may grip the detection target 14 as an operation.

[0013] The image processing device 10 includes a communication unit (acquisition unit) 16, a memory 17, and a control unit 18. The image processing device 10 may further include an input unit 19.

[0014] The communication unit 16 acquires the entire image from the camera 12. The entire image is an image in which the position of the detection target 14 is detected by the image processing device 10. Specifically, the entire image is an image in which a subject to be matched is captured. The matched subject is an object to be matched with the detection target of the representative image, as will be described later. The communication unit 16 may be, for example, a communication interface that communicates with an external device to exchange information and commands. The communication unit 16 may provide the operation device 13 with the position of the detection target 14 in the entire image.

[0015] The memory 17 includes any storage device, such as a random access memory (RAM) and a read only memory (ROM), etc. The memory 17 may store various programs that cause the control unit 18 to function and various information that the control unit 20 uses.

[0016] The memory 17 stores a template image group. The template image group includes a plurality of detection target images. The plurality of detection target images may be acquired, for example, in chronological order as described below. The plurality of detection target images may include, for example, a plurality of detection target images generated from a single image by a padding process. The plurality of detection target images may also include, for example, a plurality of detection target images acquired under different environments. The present embodiment will be described taking as an example a template image group including a plurality of detection target images acquired in chronological order. The detection target image is an image of the detection target 14.

[0017] The plurality of detection target images may also include a plurality of processed images obtained by performing image processing on each of a portion of the plurality of detection target images. The image processing may be performed by the control unit 18. The image processing may be, for example, a brightness adjustment process that applies, to the entirety of each of the plurality of detection target images, a change in pixel value between a plurality of detection target images that were acquired before and after the detection target image in the template image group. Alternatively, the image processing may be, for example, a process that adds a disturbance that has suddenly occurred to a plurality of detection target images that are acquired before and after the portion of the plurality of detection target images.

[0018] The input unit 19 may include one or more interfaces that detect user operation inputs, and may include, for example, physical keys, capacitive keys, and a touch screen that is integrated with a display device.

[0019] The control unit 18 includes one or more processors and memories. The processor may include a general-purpose processor that loads a specific program to execute a specific function, and a dedicated processor specialized for a specific process. The dedicated processor may include an application-specific integrated circuit (ASIC). The processor may include a programmable logic device (PLD). The PLD may include a field-programmable gate array (FPGA). The control unit 18 may be either a system-on-a-chip (SoC) or a system in a package (SiP) in which one or more processors work together.

[0020] The control unit 18 generates a representative image based on a partial image group extracted from the template image group. The extraction of a partial image group from the template image group and the generation of a representative image will be described later using a detailed example. For example, the control unit 18 may generate a representative image each time an entire image is acquired.

[0021] As shown in FIG. 2, the control unit 18 performs a similarity determination between the representative image ri, which has a size smaller than the entire image wi, and the region in a state where the representative image ri is overlaid on at least a portion of the entire image wi. By performing the similarity determination, it is possible to determine whether the matching target in the entire image wi is the detection target. In this embodiment, an example is described in which a representative image ri, which is smaller than the entire image wi, is overlaid on a portion of the entire image wi. The similarity determination will be described later using a detailed example. Furthermore, the control unit 18 can detect the location of the detection target, for example, in addition to determining whether the matching target is the detection target. Specifically, the control unit 18 displaces the portion of the entire image wi overlaid on the representative image ri and performs a similarity determination at each position to detect the location of the detection target 14 in the entire image wi. If the control unit 18 detects an image of the detection target 14 in the entire image wi, it stores the image in the memory 17 as a detection target image to be included in the template image group.

[0022] Next, extraction of a portion of images from the template image group in this embodiment will be described. The control unit 18 may, for example, determine a first time point, which is the start time of a time range regarding the acquisition time of the detection target images to be extracted from the template image group. The first time point may be determined by any method. For example, the first time point may be determined based on a chronological change in the appearance effect occurring in the images included in the template image group. When the input unit 19 detects an operation input specifying the first time point, the time point corresponding to the operation input may be determined as the first time point. Alternatively, the first time point may be determined based on analysis of the template image group by the control unit 18.

[0023] The control unit 18 determines the first time point based on, for example, a time-series change in pixel values ​​of each image included in the template image group. The pixel value of each image may be, for example, the pixel value of a specific pixel constituting the image, the average pixel value of all images, or the average pixel value of a specific range. More specifically, the control unit 18 calculates, for example, a period that is at least twice the period corresponding to the lowest frequency component among the time-series changes in pixel values, and determines the first time point to be the time point obtained by tracing back the calculated period from the current time point.

[0024] The control unit 18 may determine, as a first image group, an image group including a plurality of images acquired from a first time point in the past to the most recent time point along a time series among the images included in the template image group. The control unit 18 may determine a sampling interval for extracting further images from the first image group. The sampling interval may be determined by any method. For example, the sampling interval may be determined based on a change in the appearance of the images included in the template image group along a time series. When the input unit 19 detects an operation input specifying a sampling interval, the interval corresponding to the operation input may be determined as the sampling interval. Alternatively, the sampling interval may be determined based on an analysis of the template image group by the control unit 18.

[0025] The control unit 18 determines the sampling interval based on, for example, a time-series change in pixel values ​​of each image included in the template image group. As with the determination at the first time point, the pixel value of each image may be the pixel value of a specific pixel constituting the image, the average value of pixel values ​​of all images, the average value of pixel values ​​in a specific range, etc. More specifically, the control unit 18 determines, for example, a period that is at least twice the period corresponding to the highest frequency component in the time-series change in pixel values ​​as the sampling interval.

[0026] For example, if there is no change over time in the pixel values ​​of each image included in the template image group, the control unit 18 may determine the sampling interval to be any interval equal to or greater than the acquisition period of the detection target images. Note that if the absolute value of the difference between chronologically consecutive pixel values ​​is less than a first threshold, it may be considered that there is no change over time in the pixel values ​​of each image included in the template image group. Furthermore, for example, if there is no change over time in the pixel values ​​of each image included in the template image group, the control unit 18 may determine the first time point so that the number of detection target images detected from the first image group at the sampling interval described above exceeds a predetermined sampling number.

[0027] The control unit 18 may determine, from the first image group, an image group including a plurality of images acquired at sampling intervals from the first time point as a second image group. The control unit 18 may further extract images from the second image group as described below.

[0028] In order to extract the image, the control unit 18 may determine at least one disturbance factor for the detection target image whose position has been detected from the entire image wi within a predetermined time range going back from the present and whose position has been included in the template image group. The detection target image whose position has been detected within a predetermined time range going back from the present and whose position has been included in the template image group may be a detection target image included in each of the entire images wi of a plurality of frames. The disturbance factors in the plurality of detection target images corresponding to the plurality of entire images wi may be the same or similar.

[0029] Disturbance factors are factors that affect the appearance, such as changes over time in the direction and intensity of lighting due to sunlight, changes in the color received due to changes in the position of the camera 12, whether or not a label is attached to the detection target 14 or its position, and changes in illuminance due to the passage of a worker or robot between the lighting and the detection target 14.

[0030] The disturbance factor may be determined by any method. For example, when the input unit 19 detects an operation input specifying a situation when an appearance effect occurs in a newly acquired entire image wi, the control unit 18 determines a factor corresponding to the situation as the disturbance factor. When the control unit 18 detects an operation input specifying multiple situations, the control unit 18 may determine multiple disturbance factors. Alternatively, a single or multiple disturbance factors may be determined based on the analysis of the second image group by the control unit 18.

[0031] For example, the control unit 18 may determine the disturbance factor by generating a comparison image that reflects the disturbance factor. The control unit 18 may generate the comparison image using detection target images within a predetermined time range going back from the present. For example, if there is a single detection target image, the comparison image is the detection target image itself. For example, if there are multiple detection target images, the comparison image may be generated based on the multiple detection target images. A comparison image based on multiple detection target images may be generated by, for example, averaging the multiple detection target images.

[0032] The control unit 18 may extract an image group of the detection target images corresponding to the determined disturbance factor from the second image group. When multiple disturbance factors are determined, the control unit 18 may extract multiple image groups of the detection target images corresponding to each disturbance factor.

[0033] In a configuration in which a disturbance factor is determined based on an operation input detected by the input unit 19, on the premise that a situation that affects appearance is associated with each detection target image included in the template image group, the control unit 18 may extract, from the second image group, a group of detection target images associated with a situation corresponding to the disturbance factor. In a configuration in which a disturbance factor is determined by analyzing the second image group, the control unit 18 compares each detection target image included in the second image group with a comparison image, and extracts a group of detection target images corresponding to the comparison image.

[0034] For example, the control unit 18 compares the pixel values ​​of each pixel constituting the comparison image with each pixel constituting each detection target image included in the second image group to extract a group of detection target images determined to be similar to the comparison image. Furthermore, for example, the control unit 18 performs clustering on the comparison image and the group of images included in the second image group to extract a group of detection target images that belong to the same cluster as the comparison image. Furthermore, for example, the control unit 18 performs disturbance determination on all detection target images included in the second image group to extract a group of detection target images that are included in the comparison image and include pixels determined to be disturbances.

[0035] The control unit 18 may generate the representative images ri based on a second image group extracted from the template image group, or based on an image group corresponding to a disturbance factor further extracted from the second image group. The control unit 18 may generate the representative images ri by various methods.

[0036] For example, the control unit 18 generates the representative image ri by setting a statistical quantity based on the pixel values ​​of pixels at corresponding addresses between the images included in the second image group as the pixel value of the pixel. The statistical quantity may be, for example, the average value, the mode value, the median value, or the like.

[0037] Note that the pixels at corresponding addresses may be only pixels located at the same address between the images, as shown in Fig. 3. Alternatively, the pixels at corresponding addresses may be multiple pixels included in a 3x3 pixel area, for example, centered on the same address, as shown in Fig. 4. Alternatively, the pixels at corresponding addresses may be multiple pixels in a section including the same address among sections obtained by dividing each image into multiple grids, as shown in Fig. 5. Furthermore, the statistics may take into account multiple detection target images including the above-mentioned multiple processed images.

[0038] Alternatively, for example, the control unit 18 may generate a representative image ri by performing a disturbance determination on each pixel in each image included in an image group corresponding to a single disturbance factor. Note that the single disturbance factor is considered to affect the pixel values ​​of all images for which the disturbance determination is performed. Therefore, the single disturbance factor is not considered a disturbance in the image for which the disturbance determination is performed. The disturbance determination performed here is a determination of whether or not a sudden disturbance has occurred in each image included in an image group corresponding to the determined disturbance factor. For example, the control unit 18 creates a histogram of pixel values ​​of pixels at corresponding addresses in each image and determines pixel values ​​whose frequency is equal to or greater than a threshold as non-disturbance. Alternatively, the control unit 18 approximates pixel values ​​of pixels at corresponding addresses in each image to a Gaussian distribution and determines pixel values ​​whose probability density is equal to or greater than a threshold as non-disturbance. Alternatively, the control unit 18 determines disturbance by cluster classification of pixel values ​​of pixels at corresponding addresses in each image using a statistical model such as K-Means or a Gaussian mixture distribution. The control unit 18 generates a representative image ri by setting a statistical quantity based on the pixel values ​​determined to be non-disturbance as the pixel values ​​of the pixels. The statistical quantity may be, for example, the average value, the mode value, or the median value.

[0039] Furthermore, for example, the control unit 18 generates the representative image ri by selecting a single detection target image based on the disturbance factor from among a group of images corresponding to a single disturbance factor. For example, the control unit 18 selects as the representative image ri the detection target image having the smallest difference in pixel value from a statistical quantity based on pixel values ​​of pixels at corresponding addresses.

[0040] Furthermore, for example, when extracting multiple image groups corresponding to multiple disturbance factors, the control unit 18 determines the largest disturbance factor in the comparison images among the multiple disturbance factors. The largest disturbance factor may be determined arbitrarily. For example, the largest disturbance factor may be the disturbance factor with the largest number of images included in the corresponding image group, the disturbance factor with the largest number of corresponding pixels, or the disturbance factor with the most recent occurrence. Based on the image group corresponding to the determined largest disturbance factor, the control unit 18 generates a representative image ri by calculating statistics based on pixel values ​​determined to be non-disturbing, or by selecting a single detection target image from the image group.

[0041] Furthermore, for example, when extracting multiple image groups corresponding to multiple disturbance factors, the control unit 18 generates multiple primary representative images for each of the multiple image groups. The primary representative images may be generated in the same manner as the representative images ri generated based on the image groups corresponding to a single disturbance factor described above. The control unit 18 may generate the representative images ri by replacing the multiple detection target images with multiple primary representative images in the method for generating representative images ri based on the multiple detection target images described above. The control unit 18 may generate the representative images ri by, for example, setting statistics based on pixel values ​​of pixels at the same address in each of the multiple primary representative images as the pixel values.

[0042] Furthermore, for example, the control unit 18 calculates a plurality of statistics based on pixel values ​​of pixels at corresponding addresses in each of a plurality of image groups corresponding to a plurality of disturbance factors. The control unit 18 generates a representative image ri by determining the pixel value of each pixel based on the plurality of statistics. For example, when a portion of a plurality of statistics corresponding to pixels at corresponding addresses in each of a plurality of image groups is common, the control unit 18 may determine the common statistical value as the pixel value of the pixel. Having a portion of a plurality of statistics in common may mean not only that the statistical values ​​match, but also that the statistical values ​​fall within a predetermined range. The statistical value may be, for example, the average value, mode, median, etc.

[0043] As described above, the control unit 18 performs the similarity determination by comparing the generated representative image ri with a partial region of the entire image wi. The control unit 18 may perform the similarity determination by comparing the pixel values ​​of at least a portion of all pixels constituting the generated representative image ri with the pixel values ​​of pixels at the same addresses as the pixels in the partial region of the entire image wi.

[0044] The control unit 18 may compare all pixels constituting the generated representative image ri with all pixels in a partial region of the entire image wi.

[0045] Alternatively, the control unit 18 may select pixels to be used for comparison with the entire image wi from among the multiple pixels constituting the generated representative image ri. For example, the control unit 18 may determine feature points such as edges and corners in the representative image ri as pixels to be used for comparison.

[0046] Furthermore, the control unit 18 may perform the similarity determination using a contribution degree calculated for each of a plurality of pixels constituting the representative image ri or for each of a plurality of pixels constituting a partial region of the entire image wi. The contribution degree is an index indicating the degree to which a corresponding pixel contributes to the similarity determination. The control unit 18 may calculate the contribution degree based on statistics of at least a portion of the template image group. At least a portion of the template images used to calculate the contribution degree may be the image group used to generate the representative image ri. Alternatively, at least a portion of the template images used to calculate the contribution degree may be at least a portion of the image group corresponding to a disturbance factor other than the single disturbance factor selected from among the plurality of disturbance factors determined for the entire image wi.

[0047] To calculate the contribution, the control unit 18 creates a statistical model, such as a histogram or Gaussian distribution of pixel values ​​of at least one pixel corresponding to each address in at least a portion of the detection target images in the template image group. As shown in FIG. 3, the at least one pixel corresponding to the address may be only the pixel located at the address. Alternatively, as shown in FIG. 4, the at least one pixel corresponding to the address may be multiple pixels included in a 3×3 pixel area centered around the address. Alternatively, as shown in FIG. 5, the at least one pixel corresponding to the address may be multiple pixels within a section that includes the address in a grid obtained by dividing the detection target image into multiple sections. The statistical model may also take into account multiple detection target images, including the above-described multiple processed images.

[0048] The control unit 18 reads out the frequency corresponding to the pixel value of the pixel at each address in the partial region of the entire image wi corresponding to the pixel at each address used for comparison as determined above in the statistical model of the pixel. As shown in Figure 6, the control unit 18 calculates the contribution of the pixel in question according to the frequency. The contribution may be calculated so as to increase according to the frequency.

[0049] The control unit 18 may select pixels to be used in the similarity determination from among multiple pixels constituting the representative image ri or multiple pixels constituting a portion of the entire image wi, corresponding to the pixels used for comparison, based on the contribution degree. For example, the control unit 18 may use pixels whose contribution degree is equal to or greater than a threshold for the similarity determination. Alternatively, the control unit 18 may not perform similarity determination based on similarity for pixels whose contribution degree is equal to or less than a threshold. Specifically, for example, if a pixel used for comparison of the representative image ri has a pixel value with a low frequency in the corresponding statistical model or a pixel value outside a certain range centered on the most frequent pixel value, the control unit 18 may not perform similarity determination. Alternatively, the control unit 18 may apply weighting according to the contribution degree in the similarity determination. For example, the control unit 18 may multiply the difference between pixels at the same address in the representative image ri and a portion of the entire image wi by a weighting that changes according to the contribution degree.

[0050] The control unit 18 may calculate the similarity by summing up the differences between pixels constituting the generated representative image ri and pixels in a partial region of the entire image wi, which have the same address. The control unit 18 may determine that the partial region of the entire image wi, where the similarity is equal to or less than a threshold, is the location of the detection target 14.

[0051] Alternatively, the control unit 18 may not need to perform analogy judgment based on similarity if certain conditions are met using the created statistical model of pixels to calculate the contribution. In this case, a partial region of the entire image wi may be determined to be dissimilar to the representative image. The specific conditions include when, among all the pixels used for comparison in the partial region, there are many pixels with low-frequency pixel values ​​in the corresponding statistical model, or when there are many pixels with pixel values ​​outside a certain range centered on the most frequent pixel value.

[0052] Regarding pixel values, for example, if the frequency of the pixel value of the pixel in a statistical model corresponding to each pixel is equal to or less than a threshold, the pixel value may be determined to be a low-frequency pixel value. Furthermore, for example, in a statistical model approximated by a Gaussian model, if the frequency of the pixel value of the pixel is located outside a certain range centered on the pixel value of the most frequent value, the pixel value may be determined to be a pixel value outside the certain range centered on the pixel value of the most frequent value. When the statistical model is approximated by a Gaussian mixture model, the certain range may be set to, for example, a range of ±6σ centered on the mode of each Gaussian distribution.

[0053] "A large number of pixels with low-frequency pixel values ​​among all pixels" may mean that the number of pixels with low-frequency pixel values ​​is equal to or less than a threshold value. Alternatively, "A large number of pixels with low-frequency pixel values ​​among all pixels" may mean that the proportion of pixels with low-frequency pixel values ​​to all pixels used for comparison is equal to or less than a threshold value.

[0054] As described above, when the control unit 18 detects the location of the detection target 14 in the entire image wi, the control unit 18 includes an image portion of a partial area of ​​the entire image wi that corresponds to the location as a detection target image in the template image group and stores the detection target image in the memory 17. When the control unit 18 determines a disturbance factor based on an operation input detected by the input unit 19 and detects the location of the detection target 14 in the entire image wi, the control unit 18 may associate the detection target image that corresponds to the location with a situation that affects the appearance and include the detection target image in the template image group.

[0055] Next, the process of detecting the presence position executed by the control unit 18 in the first embodiment will be described with reference to the flowchart of Fig. 7. The process of detecting the presence position starts by acquiring one frame of the entire image wi.

[0056] In step S100, the control unit 18 temporarily specifies a partial area of ​​the acquired whole image wi that is the same size as the detection target image included in the template image group as a region for similarity determination. After specifying the area, the process proceeds to step S101.

[0057] In step S101, the control unit 18 determines a first time point and a sampling period based on an analysis of each detection target image included in the detection or template image group in the input unit 19. After the determination, the process proceeds to step S102.

[0058] In step S102, the control unit 18 extracts a second group of images based on the first time point and sampling interval determined in step S101. After extraction, the process proceeds to step S103.

[0059] In step S103, the control unit 18 determines the disturbance factor based on an input specifying the situation to the input unit 19 or an analysis of the second image group. After the determination, the process proceeds to step S104.

[0060] In step S104, the control unit 18 generates a representative image ri based on the image group corresponding to the disturbance factor determined in step S103, which is extracted from the second image group extracted in step S102. After generating the representative image ri, the process proceeds to step S105.

[0061] In step S105, the control unit 18 selects pixels in the representative image ri generated in step S104 to be used for comparison with the partial region identified in step S100. After selecting the pixels, the process proceeds to step S106.

[0062] In step S106, the control unit 18 extracts a portion of the template images. After the extraction, the process proceeds to step S107.

[0063] In step S107, the control unit 18 creates a statistical model for each pixel of the plurality of detection target images extracted in step S106. After creating the statistical model, the process proceeds to step S108.

[0064] In step S108, the control unit 18 uses the statistical model created in step S107 to determine whether or not the number of pixels with low frequency pixel values ​​is large. If the number of low frequency pixels is not large, the process proceeds to step S109. If the number of low frequency pixels is large, the process proceeds to step S114.

[0065] In step S109, the control unit 18 calculates the contribution by reading out the frequencies corresponding to the pixel values ​​of the pixels that make up the partial area identified in step S100 from the statistical model created in step S107. After calculating the contribution, the process proceeds to step S110.

[0066] In step S110, the control unit 18 calculates the similarity between the partial region identified in step S100 and the representative image ri based on the contribution calculated in step S109. After calculating the similarity, the process proceeds to step S111.

[0067] In step S111, the control unit 18 determines whether the similarity calculated in step S110 is equal to or less than a threshold. If it is equal to or less than the threshold, the process proceeds to step S112. If it is not equal to or less than the threshold, the process proceeds to step S114.

[0068] In step S112, the control unit 18 determines that the partial area identified in step S100 is the location of the detection target 14. After the determination, the process proceeds to step S113.

[0069] In step S113, the control unit 18 stores the image of the partial area identified in step S100 as a detection target image in the memory 17 so as to be included in the template image group. After storing, the process proceeds to step S114.

[0070] In step S114, the control unit 18 determines whether the entire region of the entire image wi has been specified as a partial region. If the entire region has not been specified, the process proceeds to step S115. If the entire region has been specified, the detection process ends.

[0071] In step S115, the control unit 18 displaces the partial area identified in step S100 while maintaining the same size. After the displacement, the process returns to step S100.

[0072] The image processing device 10 of the first embodiment configured as described above acquires a representative image ri based on a portion of a group of template images and performs a similarity determination between at least a portion of the entire image wi and the representative image ri. With this configuration, the image processing device 10 can generate the representative image ri including a detection target image that includes a frequently occurring disturbance. Therefore, even if the entire image wi is affected by a disturbance, the image processing device 10 can include the influence of the frequently occurring disturbance in the representative image ri, and can determine whether a subject appearing in the entire image corresponds to the detection target 14. As a result, the image processing device 10 can continuously and robustly detect the detection target 14 using a relatively small number of learning images, in other words, a group of images, even if disturbances occur frequently.

[0073] Furthermore, the image processing device 10 of the first embodiment stores the detection target image of the detection target 14 detected within the entire image wi in the memory 17 so that it is included in the template image group. With this configuration, the image processing device 10 can reflect, in the representative image ri, the tendency of disturbances that are likely to affect the entire image wi when later detecting the location of the detection target 14. Therefore, the image processing device 10 can improve the robustness of detection when later detecting the location of the detection target 14.

[0074] Furthermore, the image processing device 10 of the first embodiment generates a representative image ri based on a second image group including a plurality of images acquired at sampling intervals from a first image group including a plurality of images acquired chronologically from a first point in time among detection target images included in a template image group. With this configuration, the image processing device 10 determines the first point in time based on the longest occurrence period of disturbances that affect the capture of the entire image wi, thereby allowing disturbances that may be affecting the entire image wi to be reflected in the representative image ri. Furthermore, the image processing device 10 determines the sampling interval based on the shortest occurrence period of disturbances that affect the capture of the entire image wi, thereby generating an appropriate representative image ri using only some of the detection target images included in the template image group, rather than using all of them. Therefore, the image processing device 10 can reduce the processing load for generating the representative image ri, thereby reducing the load associated with detecting the location of the detection target 14.

[0075] Furthermore, the image processing device 10 of the first embodiment determines at least one disturbance factor that has the same or similar appearance effect on the detection target 14 included in the multiple whole images wi, extracts a group of images corresponding to the disturbance factor from the second image group, and generates a representative image ri based on the group of images corresponding to the disturbance factor. With this configuration, the image processing device 10 can reflect disturbances that are likely to affect the acquired representative image ri in the representative image ri. Therefore, the image processing device 10 can further improve the robustness of detection of the detection target 14.

[0076] Furthermore, the image processing device 10 of the first embodiment determines a disturbance factor based on a detection target image in the template image group associated with the situation when an operation input for a situation that affects the appearance of the detection target 14 occurs, and when detecting the location of the detection target 14 in the entire image wi, associates the situation with an image of the location and includes it in the template image group. With this configuration, the image processing device 10 can clearly reflect the occurring disturbance factor in the representative image ri. Therefore, the image processing device 10 can further improve the robustness of detection of the detection target 14.

[0077] Furthermore, the image processing device 10 of the first embodiment selects pixels to be used for comparison with the entire image wi from among the multiple pixels constituting the representative image ri. With this configuration, the image processing device 10 can reduce the processing load of similarity determination compared to a configuration in which all pixels constituting the representative image ri are used to compare with a partial region of the entire image wi.

[0078] Furthermore, the image processing device 10 of the first embodiment generates a representative image ri by determining, as the pixel value of a pixel, a statistical quantity based on the pixel values ​​of pixels at the same address in each of a group of images corresponding to a disturbance factor. With this configuration, the image processing device 10 can eliminate the influence of pixel values ​​that may be considered disturbances in the group of images corresponding to the disturbance factor, in other words, pixel values ​​that occur infrequently in the group of images. Therefore, the image processing device 10 can further improve the robustness of detection of the detection target 14.

[0079] Furthermore, the image processing device 10 of the first embodiment calculates the contribution of a portion of the entire image wi used for similarity determination based on statistics of at least a portion of the detection target image in the template image group, and uses the calculated contribution for similarity determination. With this configuration, the image processing device 10 can reduce the possibility that the similarity determination result of a pixel with a low contribution is determined to be highly similar. Therefore, when a disturbance that does not occur in the template image group occurs in the entire image wi, the image processing device 10 can eliminate the influence of the disturbance in the similarity determination. Therefore, the image processing device 10 can further improve the robustness of detection of the detection target 14.

[0080] Furthermore, in the image processing device 10 of the first embodiment, at least a part of the template images used to calculate the contribution ratio are the images used to generate the representative image ri. With this configuration, when a disturbance that is not reflected in the representative image ri occurs in the entire image wi, the image processing device 10 can eliminate the influence of the disturbance in the similarity determination. Therefore, the image processing device 10 can further improve the robustness of detection of the detection target 14.

[0081] Furthermore, in the image processing device 10 of the first embodiment, at least a portion of the template images used to calculate the contribution rate are at least a portion of the images corresponding to disturbance factors other than the single disturbance factor selected from among the multiple disturbance factors determined for the entire image wi. With this configuration, when a disturbance factor that is not reflected in the representative image ri but may have occurred occurs in the entire image wi, the image processing device 10 can eliminate the influence of the disturbance factor that is not reflected in the representative image ri in the similarity determination. Therefore, the image processing device 10 can further improve the robustness of detection of the detection target 14.

[0082] In the first embodiment, the representative image ri is generated based on the image group corresponding to the disturbance factor extracted from the second image group, but this is not limiting. The representative image ri may be generated from the second image group without specifying the image group corresponding to the disturbance factor.

[0083] Next, an image processing device according to a second embodiment of the present disclosure will be described. The second embodiment differs from the first embodiment in the method of extracting an image group from a template image group in the control unit 18. The second embodiment will be described below, focusing on the differences from the first embodiment. Note that parts having the same configuration as those in the first embodiment will be assigned the same reference numerals.

[0084] 1, the image processing device 10 according to the second embodiment includes a communication unit 16, a memory 17, and a control unit 18, similar to the first embodiment. Also, similar to the first embodiment, the image processing device 10 may include an input unit 19. The configurations and functions of the communication unit 16, memory 17, and input unit 19 in the second embodiment are the same as those in the first embodiment. The configuration of the control unit 18 in the second embodiment is the same as that in the first embodiment.

[0085] In the second embodiment, unlike the first embodiment, the control unit 18 does not extract a second image group from the template image group, but directly extracts an image group corresponding to the disturbance factor from the template image group. As in the first embodiment, the detection target image whose presence position is detected within a predetermined time range going back from the present and is included in the template image group may be the detection target image in at least one frame of the entire image wi, or may be multiple detection target images in each of multiple frames of the entire image wi. The disturbance factors in the multiple detection target images may be the same or similar.

[0086] In the second embodiment, unlike the first embodiment, the control unit 18 may generate the representative image ri based on a group of images corresponding to the disturbance factor extracted from the group of template images. The method of generating the representative image ri based on the group of images corresponding to the disturbance factor may be similar to that of the first embodiment.

[0087] In the second embodiment, the control unit 18, as in the first embodiment, performs a similarity determination by comparing the generated representative image ri with a partial region of the entire image wi. Therefore, the control unit 18 may compare all pixels constituting the generated representative image ri with all pixels in the partial region of the entire image wi. Alternatively, the control unit 18 may select pixels to be used for comparison with the entire image wi from among the multiple pixels constituting the generated representative image ri. Furthermore, the control unit 18 may perform a similarity determination using a contribution degree calculated for each pixel constituting the partial region of the entire image wi.

[0088] Next, the process of detecting the presence position executed by the control unit 18 in the second embodiment will be described with reference to the flowchart of Fig. 8. The process of detecting the presence position starts by acquiring one frame of the entire image wi.

[0089] In step S200, the control unit 18 determines the disturbance factor based on an input specifying the situation to the input unit 19 or an analysis of the detection target image whose position has been detected from the entire image wi and included in the template image group within a predetermined time range going back from the present. After the determination, the process proceeds to step S201.

[0090] In step S201, the control unit 18 extracts, from the template images, an image group corresponding to the disturbance factor determined in step S200. After extraction, the process proceeds to step S202.

[0091] In steps S202 to S213, the control unit 18 performs processing similar to steps S104 to S115 in the detection processing of the first embodiment.

[0092] The image processing device 10 of the second embodiment configured as described above also generates a representative image ri based on a portion of images extracted from a group of template images, and detects the location of the detection target 14 in the entire image wi by determining the similarity between the portion of the entire image wi and the representative image ri. Therefore, even if the frequency of disturbances is high, the image processing device 10 can continuously and robustly detect the detection target 14 using a relatively small number of learning images, in other words, a group of images.

[0093] Next, an image processing device according to a third embodiment of the present disclosure will be described. The third embodiment differs from the first embodiment in the method of extracting an image group from a template image group in the control unit 18. The third embodiment will be described below, focusing on the differences from the first embodiment. Note that parts having the same configuration as those in the first embodiment will be assigned the same reference numerals.

[0094] 1, the image processing device 10 according to the third embodiment includes a communication unit 16, a memory 17, and a control unit 18, similar to the first embodiment. Also, similar to the first embodiment, the image processing device 10 may include an input unit 19. The configurations and functions of the communication unit 16, memory 17, and input unit 19 in the third embodiment are the same as those in the first embodiment. The configuration of the control unit 18 in the third embodiment is the same as that in the first embodiment.

[0095] In the third embodiment, unlike the first embodiment, the control unit 18 is not limited to extracting the second image group and the image group corresponding to the disturbance factor from the template image group, but extracts any image group. The control unit 18 may generate a representative image ri based on the extracted image group. The method for generating the representative image ri based on the extracted image group may be similar to the method for generating the representative image ri based on the second image group in the first embodiment.

[0096] In the third embodiment, the control unit 18, as in the first embodiment, performs a similarity determination by comparing the generated representative image ri with a partial region of the entire image wi. Therefore, the control unit 18 may compare all pixels constituting the generated representative image ri with all pixels in the partial region of the entire image wi. Alternatively, the control unit 18 may select pixels to be used for comparison with the entire image wi from among the multiple pixels constituting the generated representative image ri. Note that in the third embodiment, the control unit 18 calculates a contribution degree calculated for each pixel constituting the partial region of the entire image wi based on statistics of at least a portion of the template image group, and performs a similarity determination using the contribution degree.

[0097] Next, the process of detecting a location of an object to be detected by the control unit 18 in the third embodiment will be described with reference to the flowchart of Fig. 9. The process of detecting a location of an object to be detected starts by acquiring one frame of an entire image wi.

[0098] In step S300, the control unit 18 extracts arbitrarily determined images from the template images. After extraction, the process proceeds to step S301.

[0099] In steps S301 to S312, the control unit 18 performs processing similar to steps S104 to S115 in the detection processing of the first embodiment.

[0100] The image processing device 10 of the third embodiment configured as described above generates a representative image ri based on a portion of images extracted from a group of template images, and when detecting the location of the target object 14 in the entire image wi by determining the similarity between a portion of the entire image wi and the representative image ri, calculates the contribution of the portion of the entire image wi used in the similarity determination based on statistics of at least a portion of the template images, and uses the contribution for the similarity determination. With this configuration, if a disturbance not reflected in the representative image ri occurs in the entire image wi, the image processing device 10 can eliminate the influence of the disturbance in the similarity determination. Therefore, the image processing device 10 can further improve the robustness of detection of the target object 14.

[0101] The above has described an embodiment of the image processing device 10, but embodiments of the present disclosure can also be embodied as a method or program for implementing the device, as well as a storage medium on which a program is recorded (for example, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a CD-RW, a magnetic tape, a hard disk, or a memory card, etc.).

[0102] Furthermore, the implementation form of the program is not limited to application programs such as object code compiled by a compiler or program code executed by an interpreter, but may also be in the form of a program module incorporated into an operating system. Furthermore, the program may or may not be configured so that all processing is performed solely by the CPU on the control board. The program may also be configured so that part or all of it is executed by another processing unit mounted on an expansion board or expansion unit added to the board as needed.

[0103] The drawings illustrating the embodiments of the present disclosure are schematic, and the dimensional ratios and the like in the drawings do not necessarily correspond to the actual ones.

[0104] Although the embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art could make various modifications or alterations based on the present disclosure. Therefore, it should be noted that these modifications or alterations are included in the scope of the present disclosure. For example, the functions included in each component can be rearranged so as not to cause logical inconsistencies, and multiple components can be combined or divided into one.

[0105] All of the features described in this disclosure and / or all steps of all of the disclosed methods or processes may be combined in any combination except combinations in which these features are mutually exclusive. Furthermore, each feature described in this disclosure may be replaced by an alternative feature serving the same, equivalent, or similar purpose, unless expressly denied. Thus, unless expressly denied, each disclosed feature is only one example of a generic series of identical or equivalent features.

[0106] Furthermore, embodiments of the present disclosure are not limited to the specific configurations of any of the above-described embodiments, but rather extend to any novel feature or combination thereof described herein, or any novel method or process step or combination thereof described herein.

[0107] In this disclosure, descriptions such as "first" and "second" are identifiers for distinguishing the configuration. In this disclosure, the configurations distinguished by descriptions such as "first" and "second" can have their numbers exchanged. For example, the first point in time can exchange the identifiers "first" and "second" with the second point in time. The exchange of identifiers is performed simultaneously. The configurations remain distinguished even after the exchange of identifiers. Identifiers may be deleted. A configuration from which an identifier has been deleted is distinguished by a symbol. The descriptions of identifiers such as "first" and "second" in this disclosure should not be used solely to interpret the order of the configurations or to justify the existence of an identifier with a smaller number. [Explanation of symbols]

[0108] 10 Image processing device 11 Target operating system 12 Camera 13 Operating equipment 14 Detection target 15 trays 16 Communications Department 17. Memory 18 Control Unit 19 Input section ri Representative image wi full image

Claims

1. an image processing device comprising: a control unit that acquires a representative image having pixels with pixel values ​​determined based on the most frequent pixel values ​​of pixels located at corresponding addresses in a group of template images including a plurality of images to be detected; acquires an entire image that includes the object to be matched and is different from the plurality of images; and determines the similarity between at least a portion of the entire image and the representative image.

2. 2. The image processing device according to claim 1, When the detection target is detected in the entire image in the similarity determination, the control unit newly adds the detected image of the detection target to the group of template images. Image processing device.

3. 3. The image processing device according to claim 2, The image of the detection target that will become a new part of the template image group is a partial area of ​​the entire image. Image processing device.

4. 2. The image processing device according to claim 1, The representative image has a size smaller than the entire image. Image processing device.

5. 2. The image processing device according to claim 1, The representative image is acquired based on a part of the template image group. Image processing device.

6. 2. The image processing device according to claim 1, When the entire image is acquired, the representative image is acquired. Image processing device.

7. determining a pixel value of each pixel based on a mode value of pixel values ​​of pixels located at corresponding addresses of a plurality of images in a template image group including the plurality of images to be detected, and acquiring a representative image; acquiring a whole image that includes a matching target and is different from the plurality of images; and determining a similarity between at least a part of the entire image and the representative image. Image processing methods.

8. A camera and 2. The image processing device according to claim 1, wherein the image processing device detects a position of a detection target using an image captured by the camera; an operation device that operates the detection target at the position detected by the image processing device; Target operating system.

Images