Image processing device, image processing method, and operation device controller

Abstract

This image processing device comprises a communication unit and a control unit. The communication unit acquires a first image. The control unit determines similarity between at least a part of the first image and a second image. The second image is generated on the basis of a training image group. The training image group includes a plurality of detection target images different from the first image. The control unit can re-generate the second image. The control unit regenerates the second image on the basis of the training image group, which includes more unused images than used images that were used to generate the second image.

Description

Image processing device, image processing method, and operation device controller CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to Japanese Patent Application No. 2023-097071, filed on June 13, 2023, the entire disclosure of which is incorporated herein by reference.

[0002] The present disclosure relates to an image processing device, an image processing method, and an operation device controller.

[0003] Conventionally, an object recognition method using template matching has been known (see Patent Document 1).

[0004] Japanese Patent Application Laid-Open No. 2018-151748

[0005] An image processing device according to a first aspect includes an acquisition unit that acquires a first image, a second image generated based on a training image group that includes a plurality of detection target images different from the first image, and a control unit that performs a similarity judgment with at least a portion of the first image, wherein the control unit is capable of regenerating the second image and regenerates the second image based on a training image group that includes a plurality of unused images that are greater than the used images used to generate the second image.

[0006] An image processing method according to a second aspect includes acquiring a first image; determining the similarity between a second image generated based on a training image group including a plurality of detection target images different from the first image and at least a portion of the first image; and regenerating the second image based on a training image group including a plurality of unused images that are greater than the used images used to generate the second image.

[0007] FIG. 1 is a diagram illustrating an example of the configuration of an object manipulation system including an image processing device according to an embodiment. FIG. 2 is a diagram illustrating a similarity determination between a representative image and a partial region of the entire image, performed by the control unit of FIG. 1 in order to detect the location of the detection target in the entire image. FIG. 3 is a diagram illustrating a method for creating a statistical model for determining the pixel value of each pixel of the representative image. FIG. 4 is a diagram illustrating a method for creating another statistical model for determining the pixel value of each pixel of the representative image. FIG. 5 is a diagram illustrating a method for creating yet another statistical model for determining the pixel value of each pixel of the representative image. FIG. 6 is a statistical model of each pixel used in creating the representative image in order to determine matching pixels to be excluded from the calculation of similarity. FIG. 7 is a flowchart illustrating a location detection process executed by the control unit of FIG. 1. FIG. 8 is a flowchart illustrating a representative image update subroutine executed by the control unit of FIG. 1.

[0008] Hereinafter, an embodiment of an image processing device to which the present disclosure is applied will be described with reference to the drawings.

[0009] 1 shows an example of the configuration of an object manipulation system 11 including an image processing device 10 according to an embodiment. The object manipulation system 11 may include a camera 12, the image processing device 10, a manipulation device 13, and a manipulation device controller 20.

[0010] The object manipulation system 11 performs a specific operation on a detection target 14, such as a manufactured product, in a factory, for example. The detection target 14 is placed on a tray 15, for example. The camera 12 may capture an image of the tray 15 on which the detection target 14 is placed. The camera 12 may generate images periodically, for example, at a predetermined frame rate such as 30 fps. The image processing device 10 may detect the position of the detection target 14 using the image captured by the camera 12. The manipulation device 13 may perform a predetermined task on the detection target 14 at the position detected by the image processing device 10. The manipulation device 13 may be, for example, a robot, and may grasp multiple detection targets 14 as part of its task. The manipulation device controller 20 includes a control unit (second control unit) 21. The control unit 21 acquires a similarity determination (described below) in the image processing device 10. The control unit 21 can control the manipulation device 13 based on the similarity determination. If the operation device 13 is a robot, the operation device controller 20 is, for example, a robot controller.

[0011] The image processing device 10 includes a communication unit (acquisition unit) 16 and a control unit 18. The image processing device 10 may further include a memory 17 and an input unit 19.

[0012] The communication unit 16 acquires a first image from the camera 12. The first image is also referred to as a full image. The full image is an image in which the location of the detection target 14 is detected by the image processing device 10. Specifically, the full image is an image that includes a subject to be matched. The full image also includes a background. The matching target is an object to be matched with the detection target 14 in the second image, as described below. The second image is also referred to as a template image or a representative image. Note that, for example, when the camera 12 captures a specific location, the matching target may not be included in the full image. However, it does not matter if the matching target is not included in the full image. In this case, the image processing device 10 determines that the detection target 14 is not included in the full image. In other words, the image processing device 10 can determine whether the detection target 14 is included in the full image. The communication unit 16 may be, for example, a communication interface that communicates information and commands with an external device. The communication unit 16 may provide the operating device 13 with the location of the detection target 14 in the full image.

[0013] The memory 17 includes any storage device, such as a random access memory (RAM) or a read-only memory (ROM), etc. The memory 17 may store various programs that cause the control unit 18 to function and various information that the control unit 18 uses.

[0014] The memory 17 stores a template generation image group. The template generation image group includes multiple detection target images. The template generation image group may include all detection target images stored in the memory 17. In this case, the template generation image group is also referred to as a total image group. Detection target images stored in the memory 17 may be deleted or added as appropriate. The number of detection target images in the total image group may increase or decrease due to the deletion or addition of detection target images. Unlike the first image, the detection target image is an image of the detection target 14. The multiple detection target images may be acquired in chronological order, as described below. The multiple detection target images may include, for example, at least one original detection target image acquired from the camera 12 and at least one augmented detection target image generated by an augmentation process from the original detection target image. The multiple detection target images may also include, for example, multiple detection target images acquired under different environments. This embodiment will be described using an example of a template generation image group including multiple detection target images acquired in chronological order.

[0015] The plurality of detection target images may also include a plurality of processed images obtained by performing image processing on each of a portion of the plurality of detection target images. The image processing may be performed by the control unit 18. The image processing may be, for example, a brightness adjustment process that applies, to the entirety of each of the plurality of detection target images, a change in pixel value between a plurality of detection target images acquired before and after the detection target image in the template generation image group. Alternatively, the image processing may be, for example, a process that adds a disturbance that has suddenly occurred to a plurality of detection target images acquired before and after the portion of the plurality of detection target images.

[0016] The template generation image group may include a training image group. The training image group includes a plurality of detection target images that are a portion of all of the detection target images included in the template generation image group. As will be described later, the detection target images included in the training image group are used to generate or regenerate a representative image. When generating or regenerating a representative image, the training image group may be selected and extracted as training images from the template generation image group each time.

[0017] The input unit 19 may include one or more interfaces that detect user operation inputs, such as physical keys, capacitive keys, and a touch screen that is integrated with a display device.

[0018] The control unit 18 includes one or more processors and memories. The processor may include a general-purpose processor that loads a specific program to execute a specific function, and a dedicated processor specialized for a specific process. The dedicated processor may include an application-specific integrated circuit (ASIC). The processor may include a programmable logic device (PLD). The PLD may include a field-programmable gate array (FPGA). The control unit 18 may be either a system-on-a-chip (SoC) or a system in a package (SiP) in which one or more processors work together.

[0019] As shown in FIG. 2 , the control unit 18 performs a similarity determination between the second image ri (described later) and at least a partial region of the first image wi, with the second image ri being superimposed on the partial region. By performing the similarity determination, the control unit 18 may determine whether the matching target in the first image wi is the detection target 14, or whether the detection target 14 is captured in the first image wi. Note that in this embodiment, an example will be described in which a second image (representative image) ri, which is smaller than the first image (entire image) wi, is superimposed on a partial region of the first image (entire image) wi. The similarity determination will be described later using a detailed example.

[0020] Furthermore, the control unit 18 may detect the location of the detection target 14 in addition to determining whether the matching target is the detection target 14. Specifically, the control unit 18 may detect the location of the detection target 14 in the entire image wi by displacing the representative image ri relatively to the entire image wi, thereby displacing an overlapping region that is a part of the entire image wi that overlaps with the representative image ri, and performing a similarity determination at each position.

[0021] If the control unit 18 determines that the overlapping region is similar to the representative image ri, it may acquire the partial region as an unused detection target image. Furthermore, the control unit 18 may include the acquired unused detection target image in the training image group. Specifically, the control unit 18 stores a partial region in the entire image wi that is determined to be similar to the representative image ri in the memory 17 so that it is included in the training image group as an unused detection target image, which will be described later. Note that an unused detection target image is also referred to as an unused image, and in particular, an unused image that has not been used to generate or regenerate the representative image ri is also referred to as a first unused image.

[0022] The control unit 18 regenerates the representative image ri based on the training image group. The representative image ri has a size equal to or smaller than the entire image wi. The control unit 18 performs the regeneration by replacing the representative image ri stored in the working memory of the control unit 18 or the memory 17 with an image newly created based on the training image group. Note that the regeneration of the representative image ri is also referred to as updating the representative image ri.

[0023] The control unit 18 may store a newly generated image based on the training image group as the initial representative image ri when the representative image ri is not stored in the work memory or the memory 17. Before the representative image ri is stored, in other words, before the initial representative image ri is generated, the training image group may include a plurality of images that are actually captured of the detection target 14 as initial detection target images.

[0024] The number of detection target images to be included in the training image group may be determined arbitrarily. For example, the number of detection target images to be included in the training image group may be the same as, greater than, or smaller than a first number, which will be described later.

[0025] The control unit 18 is capable of regenerating the representative image ri. The control unit 18 may be capable of regenerating the representative image ri two or more times. The control unit 18 may regenerate the representative image ri when the number of unused images included in the training image group reaches a first number. The first number may be determined according to the size and generation method of the representative image ri. For example, in a configuration in which the representative image ri is created based on statistical values ​​of pixel values ​​of multiple detection target images described below, the first number may be determined to be the lowest value at which a significant statistical value can be calculated. Note that the timing of regenerating the representative image ri may be set arbitrarily. For example, the control unit 18 may regenerate the representative image ri every predetermined elapsed time, or may regenerate the representative image ri when the accuracy of similarity determination drops below a predetermined threshold.

[0026] An unused image is a detection target image that has never been used to generate or regenerate the representative image ri. The control unit 18 may recognize an unused image as a used detection target image after using it to regenerate the representative image ri. In other words, a used detection target image is a detection target image that was used to regenerate the representative image ri before the last time. Note that a used detection target image is also referred to as a used image, and in particular, a used image used to generate or regenerate the representative image ri is also referred to as a first used image.

[0027] The training image set includes a greater number of first unused images than the first used images. In other words, the control unit 18 uses more first unused images than the first used images to update the representative image ri. The training image set may include a plurality of unused images not used to update the previous representative image ri when updating the representative image ri. Furthermore, when updating the new representative image ri, the proportion of the plurality of unused images in the training image set may be greater than the proportion of used images used to update the previous second image in the training image set. The training image set may include the first used image. Alternatively, the training image set may not include the first used image.

[0028] Note that even when a first used image is used to regenerate the representative image ri, the training image group does not have to include a first used image that has been used four or more times in generating and regenerating the representative image ri. Also, even when a first used image is used to regenerate the representative image ri, the training image group may include more first used images that have been used n-1 times in generating and regenerating the representative image ri than first used images that have been used n times, where n is any natural number. Specifically, for example, the training image group may include more first used images that have been used two times in generating and regenerating the representative image ri than first used images that have been used three times in generating and regenerating the representative image ri.

[0029] When including the first unused image in the training image group, the control unit 18 may delete the first used images in the training image group used to update the immediately preceding representative image ri from the training image group in chronological order. In other words, the control unit 18 may replace the oldest first used image included in the training image group with a new first unused image.

[0030] Alternatively, the control unit 18 may change the ratio of the number of first unused images to the total number of detection target images in the training image group for each regeneration. In other words, the control unit 18 may change the number of detection target images included in the training image group for each regeneration. Specifically, the control unit 18 may use a number of unused images in regenerating the second image that is different from the number of unused images used in the previous regeneration. In a configuration in which the ratio is changed, the control unit 18 may perform the above-mentioned replacement when the number of detection target images included in the training image group is the number of detection target images to be used when updating the next representative image ri, and when a new first unused image is to be included in the training image group.

[0031] The number of detection target images included in the training image group may be different when the representative image ri is generated and when it is regenerated. The number of detection target images included in the training image group may be the same as or different from the first number.

[0032] The control unit 18 may generate the representative image ri using various methods using the training image group. For example, the control unit 18 may create a statistical model such as a histogram or a Gaussian mixture distribution of pixel values ​​based on pixel values ​​of pixels at corresponding addresses between each detection target image included in the training image group. The control unit 18 may determine a pixel value for each pixel based on the statistical model created for each pixel. The control unit 18 may generate the representative image ri by determining pixel values ​​at addresses of multiple pixels.

[0033] Specifically, the control unit 18 may determine the pixel value of each pixel in the representative image ri as the pixel value of the pixel in the representative image ri. Furthermore, the control unit 18 may determine a range for classifying pixel values ​​of the pixel in the statistical model into multiple classes in order to perform disturbance determination for a portion of the entire image wi in the similarity determination described below. The multiple classes may include a no-disturbance class, a known-disturbance class, and an unknown-disturbance class. As shown in FIG. 6 , the control unit 18 may recognize a peak including the most frequent value as the no-disturbance class in the statistical model. In a statistical model including multiple peaks, the control unit 18 may recognize peaks other than the no-disturbance class as the known-disturbance class. The control unit 18 may recognize a range of pixel values ​​other than the no-disturbance class and the known-disturbance class as the unknown-disturbance class. For example, in a configuration in which the statistical model is approximated to a Gaussian mixture model, the pixel value range defining the no-disturbance class and the known-disturbance class may be set to a range of ±6σ centered on the mode of each Gaussian distribution. The ranges of pixel values ​​that define the no-disturbance class and the known-disturbance class can be set arbitrarily within a range that satisfies a desired accuracy.

[0034] Note that the pixels at corresponding addresses may be only pixels located at the same address between the images, as shown in Fig. 3. Alternatively, the pixels at corresponding addresses may be multiple pixels included in a 3 x 3 pixel area, for example, centered on the same address, as shown in Fig. 4. Alternatively, the pixels at corresponding addresses may be multiple pixels in a section including the same address among sections obtained by dividing each image into multiple grids, as shown in Fig. 5. Furthermore, the statistics may take into account multiple detection target images including the above-mentioned multiple processed images.

[0035] The control unit 18 performs a similarity determination by comparing the generated representative image ri with a partial region of the entire image wi. For example, the control unit 18 may perform the similarity determination by comparing the pixel values ​​of at least a portion of all pixels constituting the representative image ri with the pixel values ​​of pixels in the entire image wi that overlap with the representative image ri (pixels at the same address) when the generated representative image ri is superimposed on a portion of the entire image wi. The control unit 18 may then determine whether or not there is a region in the entire image wi that is similar to the representative image ri by displacing the overlapping region between the representative image ri and the entire image wi and comparing the pixels of the representative image ri with the pixels in the entire image wi that overlap with the representative image ri for each displacement. Note that when performing a similarity determination between the entire image wi and the representative image ri, the control unit 18 does not need to actually superimpose the entire image wi and the representative image ri; it is sufficient to calculate the positions of the pixels of the entire image wi and the representative image ri as if they were superimposed.

[0036] The control unit 18 may compare all the pixels constituting the generated representative image ri with all the pixels in a partial region of the entire image wi.

[0037] Alternatively, the control unit 18 may select, as matching pixels, pixels to be used for comparison with the entire image wi from among the multiple pixels constituting the generated representative image ri. For example, the control unit 18 may determine multiple characteristic pixels in the representative image ri as matching pixels. The control unit 18 may select pixels corresponding to feature points such as edges and corners as characteristic pixels. Alternatively, the control unit 18 may select multiple characteristic pixels based on, for example, a spatial co-occurrence histogram calculated for pixel values ​​of multiple pixels arbitrarily selected from the representative image ri. A combination of pixel values ​​that appears frequently in the spatial co-occurrence histogram indicates that the combination appears frequently in the representative image ri but is not characteristic. Therefore, the control unit 18 may select, as characteristic pixels, multiple pixels corresponding to combinations whose frequency in the spatial co-occurrence histogram exceeds zero and is equal to or less than a characteristic threshold.

[0038] The control unit 18 may select the matching pixels from all pixels or from a plurality of characteristic pixels. A configuration in which the matching pixels are selected from a plurality of characteristic pixels will be described below as an example. For example, the control unit 18 may select a plurality of stable pixels (hereinafter, also referred to as "stable pixels") as matching pixels from the plurality of characteristic pixels. In other words, the control unit 18 may use the stable pixels as matching pixels for comparison.

[0039] The control unit 18 may select stable pixels using at least one detection target image in the training image group. Specifically, the control unit 18 may extract stable pixels with small variations in pixel values ​​by, for example, comparing the pixel values ​​of pixels at the same address in the multiple detection target images with the pixel values ​​of pixels at the same address in the representative image ri, and further by analogy judgment. More specifically, the control unit 18 may select stable pixels based on an inter-image co-occurrence histogram calculated for the multiple detection target images and the representative image ri. A combination of pixel values ​​with a high frequency in the inter-image co-occurrence histogram indicates that the pixel values ​​are stable over time, in other words, that they are less susceptible to sudden disturbances. The control unit 18 may calculate a stability rate for each pixel of the representative image ri based on multiple inter-image co-occurrence histograms calculated for each of the multiple detection target images. The control unit 18 may select characteristic pixels whose stability rate is equal to or greater than a stability threshold as stable pixels.

[0040] The number of detection target images used to extract stable pixels may be less than the number of detection target images used to generate or update the representative image ri containing the stable pixels. Stable pixels may be reselected at least once for each similarity determination between the first image and the second image. The period for reselecting stable pixels may be the same as or different from the period for regenerating the representative image ri. Reselecting stable pixels means changing the detection target images used to select stable pixels. For reselecting stable pixels, detection target images that were not used in the previous selection or reselection of stable pixels may be used. Note that unused detection target images are also referred to as unused images. In particular, unused images that have not been used in the selection or reselection of stable pixels are also referred to as second unused images. Note that the second unused images may refer to the same as the first unused images described above. The detection target images used to select stable pixels may be selected from the updated training image group in order of most recently acquired time.

[0041] The proportion of the second unused images in the training image set for selecting or reselecting stable pixels may be the same as the proportion of the first unused images in the training image set for updating the representative image ri. Alternatively, the proportion of the second unused images in the training image set for selecting or reselecting stable pixels may be different from the proportion of the first unused images in the training image set for updating the representative image ri. More specifically, the proportion of the second unused images in the training image set for selecting or reselecting stable pixels may be lower than the proportion of the first unused images in the training image set for updating the representative image ri. Alternatively, the proportion of the second unused images for selecting or reselecting stable pixels may be higher than the proportion of the first unused images for regenerating the representative image ri.

[0042] The control unit 18 may extract stable pixels directly from the representative image ri. For example, the control unit 18 may calculate the product of the normalized frequency of the spatial co-occurrence histogram calculated as described above and the inverse of the normalized frequency of the inter-image co-occurrence histogram calculated as described above. The control unit 18 may extract, as stable pixels, pixel combinations for which the product is equal to or less than a threshold.

[0043] The control unit 18 may store the addresses of the determined group of match pixels in the memory 17. The group of match pixels is a plurality of match pixels that are newly selected for the representative image ri stored in the memory 17 based on the updated group of training images. The control unit 18 may update the addresses of the group of match pixels by replacing the addresses of the group of match pixels stored in the memory 17 with the addresses of the newly determined group of match pixels. The control unit 18 may store the statistical model for the match pixels in the memory 17 together with the group of addresses of the match pixels.

[0044] In the similarity determination, the control unit 18 may calculate the similarity by summing the differences in pixel values ​​between pixels constituting the generated representative image ri and pixels in a partial region of the entire image wi, which have the same address. In a configuration in which matching pixels are selected, the control unit 18 may calculate the similarity by focusing on multiple matching pixels in the representative image ri. The control unit 18 may determine that a partial region of the entire image wi, where the similarity is below a threshold, is an image portion containing an image similar to the detection target 14. The control unit 18 may determine the position of that image portion as the location of the detection target 14. Note that when summing the differences in pixel values, a lower similarity indicates a higher degree of match. On the other hand, a similarity that increases with a higher degree of match may be used, such as a configuration in which the similarity is determined using the cosine similarity of pixel values. In such a configuration, a partial region of the entire image wi, where the similarity is above a threshold, may be determined as an image portion containing an image similar to the detection target 14.

[0045] The control unit 18 may exclude some of the match pixels from the calculation of the similarity. The excluded pixels may be pixels whose pixel values ​​in a partial region of the entire image wi that have the same address as the match pixel are classified into the known disturbance class or the unknown disturbance class in the pixel statistical model, as shown in Fig. 6. The control unit 18 may perform a similarity determination using the remaining match pixels that have been excluded.

[0046] Alternatively, the control unit 18 may not need to perform analogy judgment based on similarity if a specific condition is met using a statistical model of pixels created for disturbance judgment. In this case, a partial region of the entire image wi may be determined to be dissimilar to the representative image ri. The specific condition is when, among all pixels used for comparison in the partial region, there are a large number of pixels having pixel values ​​classified into the known disturbance class and the unknown disturbance class in the corresponding statistical model, or when there are a large number of pixels having pixel values ​​outside a certain range centered on the most frequent pixel value. A large number of pixels may mean that the number exceeds a predetermined percentage of all pixels.

[0047] "A large number of pixels that satisfy the above condition among all pixels" may mean that the number of pixels whose pixel values ​​satisfy the above condition is equal to or greater than a threshold value, or "A large number of pixels that satisfy the above condition among all pixels" may mean that the ratio of pixels that satisfy the above condition to all pixels used for comparison is equal to or greater than a threshold value.

[0048] As described above, when detecting the location of the detection target 14 in the entire image wi, the control unit 18 may include an image portion of a partial area of ​​the entire image wi corresponding to the location as an unused image in the training image group and store it in the memory 17. Furthermore, when the similarity falls within a first range having a limit value on the matching side, the control unit 18 may include the image portion as an unused image in the training image group. The limit value on the matching side is an upper limit value in a configuration in which a higher similarity indicates a higher degree of matching, and a lower limit value in a configuration in which a lower similarity indicates a higher degree of matching.

[0049] Next, the process of detecting the location of an object to be detected by the control unit 18 in this embodiment will be described with reference to the flowchart of Fig. 7. The process of detecting the location of an object to be detected starts by acquiring one frame of the entire image wi.

[0050] In step S100, the control unit 18 reads out the statistical model of the matching pixels in the representative image ri from the memory 17. After reading out, the process proceeds to step S101.

[0051] In step S101, the control unit 18 extracts a partial area of ​​the acquired whole image wi, the partial area having the same size as the detection target image included in the training image group, as an area for similarity determination. After extraction, the process proceeds to step S102.

[0052] In step S102, the control unit 18 compares the pixel values ​​at the same addresses as the match pixels in the partial region extracted in the most recent step S101 with the statistical model of the match pixels. After the comparison, the process proceeds to step S103.

[0053] In step S103, the control unit 18 determines whether there are many pixels recognized as disturbances in the partial region based on the comparison in the most recent step S102. If there are not many pixels recognized as disturbances, the process proceeds to step S104. If there are many pixels recognized as disturbances, the process proceeds to step S111.

[0054] In step S104, the control unit 18 reads the pixel value of the matching pixel at the same address as the pixel determined to be non-disturbance by the comparison in the most recent step S102 from the memory 17. After reading, the process proceeds to step S105.

[0055] In step S105, the control unit 18 calculates the similarity between the representative image ri and the partial region based on the pixel values ​​of the matching pixels read out in step S104 and the pixels of the partial region at the same addresses. After the calculation, the process proceeds to step S106.

[0056] In step S106, the control unit 18 determines whether the similarity is equal to or less than a threshold value. If it is equal to or less than the threshold value, the process proceeds to step S107. If it is not equal to or less than the threshold value, the process proceeds to step S111.

[0057] In step S107, the control unit 18 determines that the partial area extracted in the immediately preceding step S101 is the location of the detection target 14. After the determination, the process proceeds to step S108.

[0058] In step S108, the control unit 18 determines whether the similarity calculated in the most recent step S105 is within a first range. If it is within the first range, the process proceeds to step S109. If it is not within the first range, the process proceeds to step S111.

[0059] In step S109, the control unit 18 stores the partial region extracted in the most recent step S101 as an unused image in the memory 17 so as to be included in the training image group. After storage, the process proceeds to step S110.

[0060] In step S110, the control unit 18 determines whether the number of unused images included in the training image set has reached the first number. If so, the process proceeds to step S200. If not, the process proceeds to step S111.

[0061] In step S200, the control unit 18 updates the representative image ri as described below. After updating, the process proceeds to step S111.

[0062] In step S111, the control unit 18 determines whether the entire region of the entire image wi has been extracted as a partial region. If the entire region has not been extracted, the process proceeds to step S112. If the entire region has been extracted, the detection process ends.

[0063] In step S112, the control unit 18 displaces the partial area extracted in step S101 while maintaining the same size. After the displacement, the process returns to step S101.

[0064] Next, the representative image update subroutine S200 executed in step S200 will be described with reference to the flowchart of FIG.

[0065] In step S201, the control unit 18 reads out the training image group from the memory 17. After reading out, the process proceeds to step S202.

[0066] In step S202, the control unit 18 creates a statistical model for each pixel based on the plurality of detection target images included in the training image group read out in the most recent step S201. After creation, the process proceeds to step S203.

[0067] In step S203, the control unit 18 generates a representative image ri based on the statistical model created in the most recent step S202. After generation, the process proceeds to step S204.

[0068] In step S204, the control unit 18 determines matching pixels in the representative image ri generated in the most recent step S203. The control unit 18 may use the training image group and the representative image ri to determine the matching pixels. After the determination, the process proceeds to step S205.

[0069] In step S205, the control unit 18 replaces the statistical model created in the most recent step S202, the representative image ri generated in the most recent step S203, and the matching pixels determined in the most recent step S204 with the statistical model, the representative image ri, and the matching pixels already stored in the memory 17. After the replacement, the process proceeds to step S206.

[0070] In step S206, the control unit 18 converts the status of all the detection target images included in step S201 from unused to used. After the conversion, the representative image update subroutine S200 ends, and the process returns to the presence position detection process.

[0071] The image processing device 10 of this embodiment, configured as described above, includes a communication unit (acquisition unit) 16 that acquires a first image wi, a training image set including multiple target images different from the first image wi, and a similarity determination between at least a portion of the first image wi and a second image ri. The second image wi is then regenerated based on a training image set including a larger number of unused images than the unused images used to generate the second image ri. With this configuration, the image processing device 10 uses more recently acquired target images when updating the second image ri, which is used to determine the similarity between the first image wi and a portion of the first image wi. This allows the image processing device 10 to create a second image ri that better reflects the effects of disturbances. Therefore, the image processing device 10 can improve the robustness of detecting the target 14, even when disturbances occur.

[0072] Furthermore, in the image processing device 10 of this embodiment, the training image group includes used images. With this configuration, the image processing device 10 can use detection target images under various circumstances to generate the second image ri. Therefore, the image processing device 10 can reduce the possibility of missing the detection target image from the first image wi.

[0073] Furthermore, in the image processing device 10 of this embodiment, the training image group does not include used images. With this configuration, the image processing device 10 can reflect the influence of a recent disturbance in the second image ri. Therefore, the image processing device 10 can further improve the robustness of detecting the detection target 14 from the first image wi under conditions where a disturbance is occurring.

[0074] Furthermore, in the image processing device 10 of this embodiment, if the similarity for similarity determination falls within a first range with a limit value on the matching side, a portion of the first image wi is included in the training image group as an unused image. If there are many detection target images that are highly similar to the second image ri, the possibility of missing an image due to a disturbance to the detection target image increases. In response to such an event, the image processing device 10 having the above-described configuration can reduce the possibility of missing a detection by ensuring diversity in the detection target images used to update the second image ri.

[0075] Furthermore, when the image processing device 10 of this embodiment includes unused images in the training image group, it deletes from the training image group the images of the detection target 14 that were used in updating the most recent second image ri, in order of oldest to newest, from the training image group. This configuration allows the image processing device 10 to gradually reduce the influence of past disturbances on the second image ri, thereby reducing the possibility of missed detections, while gradually increasing the influence of recent disturbances on the second image ri. Therefore, the image processing device 10 can improve the robustness of detecting the detection target 14 while reducing the possibility of missed detections.

[0076] Furthermore, the image processing device 10 of this embodiment regenerates the second image ri based on a training image group containing a different number of images than when the second image ri was generated. Before the start of actual imaging, it is necessary to reflect in the second image ri the influence of detection target images under various conditions, while after the start of imaging, it is necessary to reflect in the second image ri the influence of variable disturbances that may occur. In response to such requirements, the image processing device 10 having the above-described configuration can appropriately address each purpose before and after the start of imaging because the number of images of the detection target 14 included in the training image group is different.

[0077] Furthermore, the image processing device 10 of this embodiment uses a different number of unused images when regenerating the second image ri than the number of unused images used in the previous regeneration. The time interval during which any type of disturbance occurs may vary depending on the type of disturbance. In response to such an event, the image processing device 10 having the above-described configuration can generate an appropriate representative image ri according to the length of the disturbance by changing the number of unused images used. As a result, the image processing device 10 can further improve the robustness of detecting the detection target 14.

[0078] Furthermore, the image processing device 10 of this embodiment performs similarity determination by comparing the second image ri with the first image wi, and uses multiple characteristic pixels of the second image ri as comparison targets with the first image wi. A configuration in which all pixels of the second image ri are compared with a portion of the second image ri imposes a high computational load and requires a relatively long time to perform detection. On the other hand, the image processing device 10 having the above configuration reduces the computational load while suppressing a decrease in detection accuracy by comparing characteristic pixels, thereby shortening the detection time.

[0079] Furthermore, the image processing device 10 of this embodiment performs similarity determination by comparing the second image ri with the first image wi, and uses multiple stable pixels in the second image ri as comparison targets, and the multiple stable pixels are selected based on the similarity determination between the second image ri and at least one detection target image in the training image group. Characteristic pixels in the second image ri may become characteristic due to noise, etc. Using such pixels for comparison may reduce the accuracy of detection. In response to such events, the image processing device 10 having the above-described configuration can reduce the influence of noise and suppress a decrease in detection accuracy.

[0080] Furthermore, in the image processing device 10 of this embodiment, the number of detection target images used to select stable pixels is smaller than the number of detection target images used to generate or regenerate the second image ri. The second image ri is assumed to have the minimum number necessary to reflect the general trend of the situation. On the other hand, as the number of detection target images to be reflected increases, disturbances in the most recent detection target image may be considered noise, potentially causing pixels that are actually stable to be determined as unstable. In response to such events, the image processing device 10 having the above-described configuration can appropriately generate the second image ri and determine its stability.

[0081] Furthermore, in the image processing device 10 of this embodiment, the period for reselecting stable pixels is different from the period for regenerating the second image ri. When a disturbance occurs in the matching target and the disturbance is reflected in the second image ri, the pixel where the disturbance occurs is useful for calculating the similarity. The occurrence time of the disturbance varies and differs from the update period of the second image ri. In response to such an event, the image processing device 10 having the above-described configuration can accommodate a reselection period for stability determination that differs from the period appropriate for regenerating the second image ri, by differentiating the periods for reselection and regeneration.

[0082] In the image processing device 10 of this embodiment, the period for reselecting stable pixels is the same as the period for regenerating the second image ri. With this configuration, the image processing device 10 can reduce the processing load by matching the periods for both.

[0083] Furthermore, in the image processing device 10 of this embodiment, the proportion of unused images used to reselect stable pixels differs from the proportion of unused images used to regenerate the second image ri. The appropriate proportion of unused images for updating the second image ri may differ from the appropriate proportion of unused images for stability determination. In response to such an event, the image processing device 10 having the above-described configuration can reflect the general trend in the second image ri while more closely reflecting the influence of the most recent detection target image for stability determination.

[0084] Furthermore, in the image processing device 10 of this embodiment, the proportion of unused images used to reselect stable pixels is higher than the proportion of unused images used to regenerate the second image ri, and the amount of data in the training image group is small. With this configuration, the image processing device 10 can extract stable pixels taking into account disturbances and brightness changes that appear in the most recently acquired first image wi. Therefore, the image processing device 10 can select stable pixels for each lot in situations where disturbances occur repeatedly over a short period of time, such as when there is variation between lots (when one lot has fewer pixels than the number required to regenerate the second image ri). As a result, the image processing device 10 can improve its robustness.

[0085] In one embodiment, (1) an image processing device includes: an acquisition unit that acquires a first image; a second image generated based on a training image group including a plurality of detection target images different from the first image; and a first control unit that performs a similarity determination with at least a portion of the first image, wherein the first control unit is capable of regenerating the second image and regenerates the second image based on a training image group including a plurality of unused images that are greater than the number of used images used to generate the second image.

[0086] (2) In the image processing device of (1), the first control unit is capable of regenerating the second image two or more times, and when the second image is regenerated, the training image group includes a plurality of unused images that have not been used in the past regeneration of the second image, and a proportion of the plurality of unused images in the training image group is greater than a proportion of used images that have been used in the past regeneration of the second image in the training image group.

[0087] (3) In the image processing device of (1) or (2) above, the training image group is selected each time the second image is generated or regenerated from among a plurality of detection target images stored in memory.

[0088] (4) In the image processing device according to any one of (1) to (3), the training image group includes the used image.

[0089] (5) In the image processing device according to any one of (1) to (3), the training image group does not include the used image.

[0090] (6) In any of the image processing devices described in (1) to (5), when the first control unit determines that at least a portion of the first image is similar to the second image, the first control unit acquires the portion as the unused image of the detection target image.

[0091] (7) In the image processing device according to any one of (1) to (6) above, the first control unit regenerates the second image when the number of the plurality of unused images reaches a first number.

[0092] (8) In any of the image processing devices described in (1) to (7), the first control unit regenerates the second image based on the training image group having a different number of images than when the second image was generated.

[0093] (9) In any of the image processing devices described in (1) to (8) above, when regenerating the second image, the first control unit uses a number of the plurality of unused images that is different from the number of the plurality of unused images used in the previous regeneration.

[0094] (10) In any of the image processing devices (1) to (9) above, the first control unit performs the similarity judgment by comparing the second image with the first image, and uses a plurality of stable pixels in the second image as comparison targets, and the plurality of stable pixels are selected based on the similarity judgment between the second image and at least one detection target image in the training image group.

[0095] (11) In the image processing device of (10), the number of the plurality of detection target images used to select the stable pixels is smaller than the number of the detection target images used to generate or regenerate the second image.

[0096] (12) In the image processing device of (10) or (11) above, the stable pixel can be reselected and is selected based on at least one unused image that has not been used in the past selection or reselection of the stable pixel.

[0097] (13) In the image processing device according to any one of (10) to (12), the proportion of unused images used for reselecting the stable pixels is different from the proportion of unused images used for regenerating the second image.

[0098] In one embodiment (14), an image processing method includes: acquiring a first image; determining a similarity between a second image generated based on a training image group including a plurality of detection target images different from the first image and at least a portion of the first image; and regenerating the second image based on a training image group including a plurality of unused images that are greater than the used images used to generate the second image.

[0099] In one embodiment (15), the operation device controller includes a control unit that acquires the similarity determination in any of the image processing devices (1) to (13) above, and is capable of controlling the operation device based on the similarity determination.

[0100] The above has described an embodiment of the image processing device 10, but embodiments of the present disclosure can also be embodied as a method or program for implementing the device, as well as a storage medium on which a program is recorded (for example, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a CD-RW, a magnetic tape, a hard disk, or a memory card, etc.).

[0101] Furthermore, the implementation form of the program is not limited to application programs such as object code compiled by a compiler or program code executed by an interpreter, but may also be in the form of a program module incorporated into an operating system. Furthermore, the program may or may not be configured so that all processing is performed solely by the CPU on the control board. The program may also be configured so that part or all of it is executed by another processing unit mounted on an expansion board or expansion unit added to the board as needed.

[0102] The drawings illustrating the embodiments of the present disclosure are schematic, and the dimensional ratios and the like in the drawings do not necessarily correspond to the actual ones.

[0103] Although the embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art could make various modifications or alterations based on the present disclosure. Therefore, it should be noted that these modifications or alterations are included in the scope of the present disclosure. For example, the functions included in each component can be rearranged so as not to cause logical inconsistencies, and multiple components can be combined or divided into one.

[0104] For example, in this embodiment, the training image set includes a greater number of unused images than the number of used images, but is not limited to such a configuration. The number of used images included in the training image set may be the same as or greater than the number of unused images.

[0105] All of the features described in this disclosure and / or all steps of all of the disclosed methods or processes may be combined in any combination except combinations in which these features are mutually exclusive. Furthermore, each feature described in this disclosure may be replaced by an alternative feature serving the same, equivalent, or similar purpose, unless expressly denied. Thus, unless expressly denied, each disclosed feature is only one example of a generic series of identical or equivalent features.

[0106] Furthermore, embodiments of the present disclosure are not limited to the specific configurations of any of the above-described embodiments, but rather extend to any novel feature or combination thereof described herein, or any novel method or process step or combination thereof described herein.

[0107] In this disclosure, descriptions such as "first" are identifiers for distinguishing the configuration.

[0108] REFERENCE SIGNS LIST 10 Image processing device 11 Object operation system 12 Camera 13 Operation device 14 Detection object 15 Tray 16 Communication unit 17 Memory 18 Control unit 19 Input unit 20 Operation device controller 21 Control unit ri Representative image wi Whole image

Claims

1. An acquisition unit that acquires the first image, The system comprises a second image generated based on a training image set including multiple detection target images different from the first image, and a first control unit that performs a similarity determination between the second image and at least a part of the first image. The first control unit is, The second image can be regenerated, The second image is regenerated based on a training image set that includes more unused images than the used images used to generate the second image. Image processing device.

2. In the image processing apparatus according to claim 1, The first control unit is capable of regenerating the second image two or more times, When the second image is regenerated, the training image set includes multiple unused images that have not been used to regenerate the second image in the past, and the proportion of these multiple unused images in the training image set is greater than the proportion of used images in the training image set that have been used to regenerate the second image in the past. Image processing device.

3. In the image processing apparatus according to claim 1 or 2, The aforementioned training image set is selected each time from a plurality of detection target images stored in memory when generating or regenerating the second image. Image processing device.

4. In the image processing apparatus according to claim 1 or 2, The training image set includes the used images. Image processing device.

5. In the image processing apparatus according to claim 1 or 2, The aforementioned training image set does not include the previously used images. Image processing device.

6. In the image processing apparatus according to claim 1 or 2, If the first control unit determines that at least a portion of the first image is similar to the second image, it acquires the portion as the unused image of the detected image. Image processing device.

7. In the image processing apparatus according to claim 1 or 2, The first control unit regenerates the second image when the number of unused images reaches a first number. Image processing device.

8. In the image processing apparatus according to claim 1 or 2, The first control unit regenerates the second image based on the training image group, which has a different number of images than when the second image was generated. Image processing device.

9. In the image processing apparatus according to claim 1 or 2, The first control unit, when regenerating the second image, uses a different number of the plurality of unused images than the number used in the previous regeneration. Image processing device.

10. In the image processing apparatus according to claim 1 or 2, The first control unit performs the similarity determination by comparing the second image with the first image, and uses a plurality of stable pixels from the second image as comparison targets. The plurality of stable pixels are selected based on a similarity determination between the second image and at least one target image in the training image group. Image processing device.

11. In the image processing apparatus according to claim 10, The number of the multiple detection target images used to select the stable pixels is less than the number of detection target images used to generate or regenerate the second image. Image processing device.

12. In the image processing apparatus according to claim 10, The aforementioned stable pixels are It is reselectable, Selected based on at least one unused image that was not used during the previous selection or re-selection of the aforementioned stable pixels. Image processing device.

13. In the image processing apparatus according to claim 10, The proportion of unused images used for the re-selection of the stable pixels is different from the proportion of unused images used for the regeneration of the second image. Image processing device.

14. To obtain the first image, The process involves determining the similarity between a second image, generated based on a training image set that includes multiple detection target images different from the first image, and at least a portion of the first image. This includes regenerating the second image based on a training image set that includes more unused images than the used images used to generate the second image. Image processing methods.

15. An operating device controller comprising a second control unit capable of obtaining the similarity determination in the image processing apparatus according to claim 1 or 2, and controlling the operating device based on the similarity determination.