Image processing device and method

Abstract

The present disclosure pertains to an image processing device and a method that make it possible to inhibit an increase in the processing time of clustering of an image. By carrying out clustering of sparse pixels included in an image and supplementing the sparse information obtained through said clustering via image filtering using an image signal as a guide, a dense clustering result is derived. The sparse information is, for instance, a model coefficient obtained in the clustering, or a clustering result. The present disclosure can be applied to, for instance, an image processing device, an image processing method, and the like.

Description

[Technical field]

[0001] The present disclosure relates to an image processing device and method, and more particularly to an image processing device and method capable of suppressing an increase in processing time for image clustering. [Background technology]

[0002] Conventionally, image clustering has been used in various image processing (see, for example, Patent Document 1). For example, Patent Document 1 discloses a method of performing clustering on an image, performing pixel interpolation using the class data, and restoring thinned pixels. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 5-328185 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional clustering method, all pixels of the image to be processed are clustered, which may increase the processing time.

[0005] The present disclosure has been made in consideration of such circumstances, and makes it possible to suppress an increase in the processing time for image clustering. [Means for solving the problem]

[0006] An image processing device according to one aspect of the present technology is an image processing device including a clustering unit that performs clustering of sparse pixels included in an image, and a complementation processing unit that derives a dense clustering result by complementing the sparse information obtained by the clustering by the clustering unit through image filtering guided by an image signal.

[0007] An image processing method according to one aspect of the present technology is an image processing method that derives a dense clustering result by clustering sparse pixels contained in an image and complementing the sparse information obtained by the clustering with image filtering guided by an image signal.

[0008] An image processing device according to another aspect of the present technology is an image processing device that includes a clustering unit that performs local clustering, which is clustering of dense pixels included in a local region of an image, using information obtained in wide-area clustering, which is clustering of sparse pixels included in a wide area of ​​the image.

[0009] An image processing method according to another aspect of the present technology is an image processing method that uses information obtained in wide-area clustering, which is clustering of sparse pixels contained in a wide area of ​​an image, to perform local clustering, which is clustering of dense pixels contained in a local area of ​​the image.

[0010] In an image processing device and method according to one aspect of the present technology, sparse pixels contained in an image are clustered, and the sparse information obtained by the clustering is supplemented by image filtering guided by the image signal to derive a dense clustering result.

[0011] In an image processing device and method according to another aspect of the present technology, information obtained in wide-area clustering, which is clustering of sparse pixels contained in a wide area of ​​an image, is used to perform local clustering, which is clustering of dense pixels contained in a local area of ​​the image. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram showing an example of a main configuration of an image processing device; [Diagram 2] FIG. 11 is a diagram illustrating an example of image filtering. [Diagram 3] FIG. 13 is a diagram illustrating an example of sparse model coefficients. [Figure 4] FIG. 13 is a diagram illustrating an example of a guide. [Diagram 5] FIG. 13 is a diagram illustrating an example of dense model coefficients. [Figure 6] FIG. 11 is a diagram illustrating an example of a clustering result. [Figure 7] 13 is a flowchart illustrating an example of the flow of a clustering process. [Figure 8] 1 is a block diagram showing an example of a main configuration of an image processing device; [Figure 9] FIG. 1 is a diagram illustrating an example of a farm field. [Figure 10] 13 is a flowchart illustrating an example of the flow of a clustering process. [Figure 11] 1 is a block diagram showing an example of a main configuration of an image processing device; [Figure 12] FIG. 11 is a diagram illustrating an example of stitching information. [Figure 13] 13 is a flowchart illustrating an example of the flow of a clustering process. [Figure 14] 1 is a block diagram showing an example of a main configuration of an image processing device; [Figure 15] 13 is a flowchart illustrating an example of the flow of a clustering process. [Figure 16] FIG. 1 is a diagram illustrating an example of an overview of image clustering. [Figure 17] 1 is a block diagram showing an example of a main configuration of an image processing device; [Figure 18] 13 is a flowchart illustrating an example of the flow of a clustering process. [Figure 19] 1 is a block diagram showing an example of a main configuration of an image processing device; [Figure 20] 13 is a flowchart illustrating an example of the flow of a clustering process. [Figure 21] 1 is a block diagram showing an example of a main configuration of an image processing device; [Figure 22] FIG. 11 is a diagram illustrating an example of a comparison of clustering results. [Figure 23]13 is a flowchart illustrating an example of the flow of a clustering process. [Figure 24] 1 is a block diagram showing an example of a main configuration of an image processing device; [Diagram 25] 13 is a flowchart illustrating an example of the flow of a clustering process. [Figure 26] 1 is a block diagram showing an example of a main configuration of an image processing device; [Figure 27] 13 is a flowchart illustrating an example of the flow of a clustering process. [Figure 28] FIG. 1 is a diagram illustrating an example of how a CT image is generated. [Figure 29] 1A to 1C are diagrams illustrating an example of how a CT image showing an example of a global region and a local region is generated. [Diagram 30] 1 is a block diagram showing an example of a main configuration of an image processing device; [Diagram 31] 13 is a flowchart illustrating an example of the flow of a clustering process. [Diagram 32] FIG. 1 is a block diagram showing an example of a main configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described in the following order. 1. First embodiment (sparse clustering and image filtering) 2. Second embodiment (global clustering and sparse local clustering) 3. Third embodiment (global clustering and dense local clustering) 4. Fourth embodiment (clustering in vegetation area analysis) 5. Fifth embodiment (clustering of CT images) 6. Notes

[0014] <1. First embodiment> <Image clustering> Conventionally, image clustering has been used in various image processing applications. For example, Patent Document 1 discloses a method of performing clustering on an image, performing pixel interpolation using the class data, and restoring thinned pixels.

[0015] In addition, for example, image clustering is used when a farm field is imaged multiple times from above while moving around using a so-called drone or airplane, and the captured images are used to analyze vegetation (classifying vegetation and soil, etc.).

[0016] However, in the conventional clustering method, all pixels of the image to be processed are clustered, which may increase the processing time.

[0017] <Sparse clustering and image filtering> Therefore, sparse pixels included in an image are clustered, and the sparse information obtained by the clustering is complemented by image filtering guided by an image signal to derive a dense clustering result. The information used for this image filtering may be, for example, a learning model coefficient or a clustering result. This "complementation" by image filtering means not only the interpolation of information (filling in missing data) but also the appropriate optimization according to the image structure. In other words, an optimized dense clustering result is obtained by this image filtering.

[0018] For example, in the case of images of a farm field, since the images are captured in an outdoor environment, the lighting environment changes greatly during the image capture work, and there is a possibility that unevenness in the signal distribution within the same subject (multiple pixels of the same subject have different signal characteristics) may occur due to cast shadows, shading, etc. By performing clustering as described above, it is possible to quickly obtain clustering results using surrounding image structure information even in such cases. In other words, by applying this technology, regularization according to the geometric structure of the guide image can be reflected in the clustering results, so that even for images with large changes in the lighting environment outdoors or images with uneven signal distribution within the same subject due to cast shadows and shading, it is possible to obtain results classified into classes for each subject.

[0019] <Image processing device> Fig. 1 is a block diagram showing an example of the configuration of an image processing device to which the present technology is applied. The image processing device 100 shown in Fig. 1 is a device that performs image clustering. The image processing device 100 receives a captured image 20 as an input, performs image clustering on the captured image 20, and outputs a clustering result 30.

[0020] The captured image 20 may be, for example, a stitched image formed by pasting together a plurality of captured images (P1 to Pn). It may also be a moving image made up of a plurality of frame images. It may also be a file (a group of captured images) in which a plurality of captured images are compiled into one, or it may be a single captured image. Of course, it may also be an image other than a captured image (for example, a CG image, etc.). The captured image 20 may also be an image in the wavelength range of visible light (RGB), or may be an image of the wavelength range of invisible light such as near-infrared light. It may also be both of these.

[0021] Note that Fig. 1 shows the main processing units, data flows, etc., and is not limited to all that is shown in Fig. 1. In other words, in this image processing device 100, there may be processing units that are not shown as blocks in Fig. 1, and there may be processes and data flows that are not shown as arrows, etc. in Fig. 1.

[0022] As shown in FIG. 1, the image processing device 100 includes a sampling pixel selection unit 111, a clustering unit 112, and a complementation processing unit 113.

[0023] The sampling pixel selection unit 111 performs processing related to the selection of sampling pixels which are target pixels for clustering. For example, the sampling pixel selection unit 111 acquires a captured image 20. The sampling pixel selection unit 111 also selects some pixels of the captured image 20 as sampling pixels. At that time, the sampling pixel selection unit 111 selects the sampling pixels so that they are sparsely spaced.

[0024] The "sparse state" refers to the state of a pixel group (or information corresponding to the pixel group) consisting of some pixels of a captured image, and indicates the state of a pixel group (or information corresponding to the pixel group) consisting of at least fewer pixels than the "dense state" described later. For example, a pixel group (or information corresponding to the pixel group) consisting of pixels that are not adjacent to each other may be in a "sparse state". In other words, in the case of sampling pixels, sampling pixels in which only pixels that are not adjacent to each other in the captured image 20 are selected may be sampling pixels in a sparse state (also called sparse sampling pixels). In addition, a pixel group (or information corresponding to the pixel group) selected from a specified image at a ratio (number) smaller than a specified threshold may be in a "sparse state". In other words, in the case of sampling pixels, sampling pixels selected at a ratio (number) smaller than a specified threshold with respect to the number of pixels in the captured image 20 may be sparse sampling pixels.

[0025] The sampled pixel selection unit 111 supplies the selected sparsely sampled pixels to a clustering unit 112 .

[0026] The clustering unit 112 performs processing related to clustering. For example, the clustering unit 112 acquires sparsely sampled pixels supplied from the sampled pixel selection unit 111. The clustering unit 112 processes the acquired sparsely sampled pixels and performs clustering. Any method of this clustering may be used. For example, the GMM method, the k-means method, or the like may be applied. The clustering unit 112 supplies sparse information acquired by this clustering to the complementation processing unit 113.

[0027] The sparse information is information corresponding to each sampled pixel (i.e., in a sparse state) obtained by clustering the sparsely sampled pixels. For example, it may be a learning model coefficient, a clustering result, or both.

[0028] The complement processing unit 113 performs processing related to complementing sparse information. For example, the complement processing unit 113 acquires sparse information (learning model coefficients, clustering results, etc.) supplied from the clustering unit 112. The complement processing unit 113 also acquires the captured image 20.

[0029] This captured image 20 may be the same as the captured image supplied to the sampling pixel selection unit 111 (i.e., the captured image to be clustered), or may be a captured image different from the captured image to be clustered, but at approximately the same time and in approximately the same range as the captured image to be clustered. For example, it may be another captured image obtained by another capture at approximately the same time and with approximately the same angle of view as the capture for obtaining the captured image to be clustered. For example, a captured image 20 in the wavelength range of visible light (RGB) may be supplied to the sampling pixel selection unit 111, and a captured image 20 obtained by imaging the wavelength range of invisible light such as near-infrared light may be supplied to the complementation processing unit 113.

[0030] The complementation processing unit 113 performs image filtering (complementation processing) on ​​the sparse information acquired from the clustering unit 112 using the image signal (acquired captured image 20) as a guide, and derives a dense clustering result.

[0031] The term "dense state" refers to the state of a pixel group (or information corresponding to the pixel group) that is composed of some or all of the pixels of a captured image, and indicates the state of a pixel group (or information corresponding to the pixel group) that is composed of at least more pixels than the above-mentioned "dense state". For example, a pixel group (or information corresponding to the pixel group) that includes pixels that are adjacent to each other may be considered to be in a "dense state". In other words, in the case of a clustering result, the clustering result of sampled pixels that include pixels that are adjacent to each other in the captured image 20 may be considered to be in a dense state (also referred to as a dense clustering result). In addition, a pixel group (or information corresponding to the pixel group) selected from a specified image at a ratio (number) equal to or greater than a predetermined threshold may be considered to be in a "dense state". In other words, in the case of a clustering result, the clustering result of sampled pixels selected at a ratio (number) equal to or greater than a predetermined threshold to the number of pixels of the captured image 20 may be considered to be a dense clustering result.

[0032] For example, the complementation processing unit 113 receives the likelihood (likelihood image) of each pixel for each class, sequentially applies image filtering using the original image as a guide to complement the likelihood image, and re-determines the class from the filtered likelihood image to obtain a dense clustering result. Image filtering allows regularization according to the geometric structure of the guide image to be reflected in the clustering result, so that the complementation processing unit 113 can obtain a result in which the objects are classified into classes, even for images in which the lighting environment changes greatly outdoors or images in which the signal distribution within the same object is uneven due to cast shadows or shading. For example, it is possible to suppress the occurrence of a phenomenon in which parts of the same color of the same object are classified into different classes due to differences in brightness when they are partly shaded.

[0033] The complementation processing unit 113 outputs the clustering result 30 (dense clustering result) obtained by the complementation processing to the outside of the image processing device 100 as the image processing result by the image processing device 100.

[0034] <Image filtering> Any method can be used for this image filtering (complementation processing). For image filtering, by using fast edge-preserving filtering such as Fast Global Smother filtering, Domain Transform filtering, Fast Bilateral Solver filtering, or Domain Transform Solver filtering, it is possible to obtain clustering results that are faster than prediction using all pixels and are robust against noise and disturbances.

[0035] For example, the complementation processing unit 113 may perform energy minimization of the clustering result using GrabCut described in Jianbo Li, et.al, "KM_GrabCut: a fast interactive image segmentation algorithm", ICGIP2014 (also referred to as Non-Patent Document 1), perform global optimization using Cost-Volume Filtering described in C. Rhemann, et.al, "Fast Cost-Volume Filtering for Visual Correspondence and Beyond", CVPR2011 (also referred to as Non-Patent Document 2), and may perform information densification using an FGS filter described in D. Min, et.al, "Fast Global Image Smoothing Based on Weighted Least Squares", IEEE TIP2014 (also referred to as Non-Patent Document 3).

[0036] The FGWLS (Fast Global Weighted Least Squares Filter) described in Non-Patent Document 3 is a process that decomposes the WLS (Weighted Least Squares Filter) described in Z Farbman, et al., "Edge-Preserving Decompositions for Multi-Scale Tone and Detail Manipulation," Proceedings of ACM SIGGRAPH 2008. (also referred to as Non-Patent Document 4) into a one-dimensional recursive filter and repeatedly applies it in the x and y axis directions to obtain a global optimal solution in a constant time operation. This process expands and densifies sparse data according to image structures such as textures and edges (according to the adjacent relationships between pixels obtained based on the structure).

[0037] 2A, assume that image filtering as described above is performed on pixels in a first color area 131 shown in a diagonal line pattern and pixels in a second color area 132 shown in a mesh pattern, using image 130 consisting of a gray and white swirl pattern as a guide. First color area 131 is located in a gray area of ​​image 130. Second color area 132 is located in a white area of ​​image 130.

[0038] By repeatedly performing linear recursive operations on adjacent pixels in the x and y directions, the first color region 131 expands in the gray region of the image 130 as shown in FIG. 2B, FIG. 2C, and FIG. 2D. Similarly, the second color region 132 expands in the white region of the image 130. Then, in the state shown in FIG. 2D, the region on the image 130 is filled with the first color region 131 and the second color region 132. That is, the first color region 131 and the second color region 132, which were sparse in FIG. 2A (a small part of the region on the image 130), become dense in FIG. 2D (filling the entire region on the image 130).

[0039] In this way, by performing image filtering, sparse data can be complemented and densified according to the structure of the image used as a guide. Therefore, the image processing device 100 can obtain a more accurate clustering result. As described above, the "complementation" by this filtering means not only the interpolation of information (filling in missing data) but also the optimization according to the image structure as appropriate. In other words, an optimized dense clustering result can be obtained by this image filtering. Therefore, the image processing device 100 can obtain a more accurate clustering result.

[0040] In addition to the above examples, image filtering may be performed using rule-based filtering described in, for example, Eduardo SL Gastal and Manuel M Oliveira, "Domain transform for edge-aware image and video processing", In ACM Transactions on Graphics (TOG), volume 30, page 69. ACM, 2011. (also referred to as Non-Patent Document 5), Jonathan T Barron and Ben Poole, "The Fast Bilateral Solver", In European Conference on Computer Vision (ECCV), pages 617-632. Springer International Publishing, 2016. (also referred to as Non-Patent Document 6), Akash Bapat, Jan-Michael Frahm, "The Domain Transform Solver", The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 6014-6023. (also referred to as Non-Patent Document 7), etc.Also, Hang Su, Varun Jampani, Deqing Sun, Orazio Gallo, Erik Learned-Miller, Jan Kautz, "Pixel-Adaptive Convolutional Neural Networks", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. (also referred to as non-patent document 8), Yu-Kai Huang, Tsung-Han Wu, Yueh-Cheng Liu, Winston H. Hsu, "Indoor Depth Completion with Boundary Consistency and Self-Attention", (ICCV), 2019. (also referred to as Non-Patent Document 9), Jie Tang, Fei-Peng Tian, ​​Wei Feng, Jian Li, Ping Tan, "Learning Guided Convolutional Network for Depth Completion", arXiv preprint arXiv:1908.01238, 2019. (also referred to as Non-Patent Document 10) or the like may be applied.

[0041] The clustering unit 112 performs clustering as described above, and supplies sparse information (model coefficients, clustering results, etc.) to the complementation processing unit 113.

[0042] Fig. 3 is a diagram showing an example of a result of visualizing a part of the model coefficients. For example, assume that sparse model coefficients 141 as shown in Fig. 3A are supplied from the clustering unit 112 to the complementation processing unit 113. Model coefficients 142 in Fig. 3B are an enlarged version of a part of the model coefficients 141 in Fig. 3A. A group of gray points shown in the model coefficients 142 indicate the model coefficients of the pixels at those positions. In this way, the model coefficients 141 are composed of sparse information (model coefficients of some pixels).

[0043] Fig. 3C is a diagram showing a schematic structure of such sparse model coefficients 141. In Fig. 3C, gray squares indicate pixels where model coefficients exist. As shown in this example, the model coefficients 141 are composed of model coefficients 144 for one pixel provided for each region 143 of a predetermined size. For example, if the region 143 is 4x4 pixels, the amount of data of the model coefficients 141 is 1 / 16 of that in the dense case (model coefficients for all pixels).

[0044] For such sparse model coefficients 141, the complement processing unit 113 performs image filtering using an image signal as a guide. Fig. 4 is a diagram showing an example of a portion of an image used as the guide. For example, the complement processing unit 113 performs image filtering for the sparse model coefficients 141 using an image 151 (A in Fig. 4) included in the captured image 20 as a guide. An image 152 shown in B in Fig. 4 is an enlarged portion of the image 151.

[0045] Fig. 5 is a diagram showing an example of a result of visualizing a part of the model coefficients obtained by the image filtering. For example, a model coefficient 161 as shown in Fig. 5A is obtained by image filtering by the complementation processing unit 113. A model coefficient 162 shown in Fig. 5B is an enlarged part of the model coefficient 161. As is clear from a comparison with the model coefficient 142 (Fig. 3B), the model coefficient 162 (i.e., the model coefficient 161) is in a dense state.

[0046] C of Fig. 5 is a diagram showing a model structure of such model coefficients 161. In C of Fig. 5, gray squares indicate pixels where model coefficients exist. That is, in this example, the model coefficients 161 are composed of model coefficients of all pixels. For example, if the area 163 is 4x4 pixels, each area 163 has model coefficients 164 for 16 pixels. Therefore, the amount of data of the model coefficients 161 (A of Fig. 5) is 16 times the amount of data of the model coefficients 141 (A of Fig. 3).

[0047] Clustering result 171 shown in Fig. 6A is an example of a clustering result derived using such dense model coefficients 161. Clustering result 172 shown in Fig. 6B is an enlarged portion of clustering result 171. In this way, by performing image filtering, a dense clustering result can be obtained from sparse model coefficients.

[0048] For example, in the case of the structural examples of C in FIG. 3 and C in FIG. 5, the processing time of the clustering to obtain the sparse model coefficients 141 varies depending on the method used for clustering. For example, even in the case of a simple k-means method, the order of calculation amount O is O=(Nk) when the number of data is N and the number of iterations is treated as a constant k, which is about 1 / 16 of the clustering processing time to obtain the dense model coefficients 161. When the processing time of the image filtering is taken into consideration, the total processing time is about 1 / 3 to 1 / 4 of the time required to obtain the dense model coefficients 161 by clustering. That is, by applying the sparse clustering and image filtering as described above, the image processing device 100 can obtain the dense clustering result at a higher speed. That is, the increase in the processing time can be suppressed.

[0049] <Flow of clustering process> An example of the flow of the clustering process executed by the image processing device 1000 will be described with reference to the flowchart of Fig. 7. When the clustering process starts, the sampling pixel selection unit 111 acquires the captured image 20 in step S101.

[0050] In step S102, the sampling pixel selection unit 111 selects and determines sparse sampling pixels from the captured image acquired in step S101.

[0051] In step S103, the clustering unit 112 performs clustering on the sparsely sampled pixels determined in step S102.

[0052] In step S104, the complementation processing unit 113 acquires the captured image 20, and using the captured image 20 as a guide, performs image filtering on the sparse information (learning model coefficients and clustering results) obtained by the processing of step S103, and complements the sparse information to derive a dense clustering result.

[0053] In step S105, the complementation processing unit 113 outputs the dense clustering result obtained by the process of step S104 as the clustering result 30. When the process of step S105 ends, the clustering process ends.

[0054] By performing each process as described above, the image processing device 100 can suppress an increase in the processing time for image clustering.

[0055] <Use of field information> For example, when performing vegetation analysis (classification of vegetation and soil, etc.) on a farm field, there is a method of performing clustering on a stitched image formed by stitching together a plurality of captured images of the farm field captured from the air. In such a case, it is not necessary to cluster areas other than the farm field among the areas included in the stitched image. However, it is generally difficult to perform image capture within the range of the farm field and control the image not to capture images outside the farm field, and the stitched image formed by stitching together captured images also includes areas outside the farm field. Therefore, if clustering is performed on the entire stitched image, clustering will also be performed on areas outside the farm field, which may unnecessarily increase processing time due to unnecessary processing.

[0056] Therefore, only pixels within the field are selected as sampling pixels (i.e., pixels in areas outside the field are not selected as sampling pixels). Field information (field boundary information) is information about the field, for example, information indicating the range of the field that is the target area for image clustering. Therefore, such field information is used to identify the field area contained in the captured image, and sampling pixels are selected only within the identified field. In this way, it is possible to prevent unnecessary increases in clustering and unnecessary increases in processing time.

[0057] <Image processing device> Fig. 8 is a block diagram showing an example of the main configuration of the image processing device 100 in this case. The captured image 20 is a stitched image formed by stitching together a plurality of captured images captured from above the farm field to be subjected to clustering processing. As shown in Fig. 8, the image processing device 100 in this case has a farm field area storage unit 201 in addition to the configuration shown in Fig. 1.

[0058] The field area storage unit 201 has a storage medium and stores information indicating the area of ​​the field to be processed (field area) in the storage medium (the storage area of ​​the storage medium). This information indicating the field area may be any information. For example, it may be information indicating the field area using coordinate information based on a GPS (Global Positioning System) or the like (also called GPS coordinate information), information indicating which pixels of the captured image 20 are within the field area, or other information.

[0059] For example, in response to a request from the sampling pixel selection unit 111, the field area storage unit 201 supplies information indicating the area of ​​the field that is stored in (the storage area of) its own storage medium to the sampling pixel selection unit 111 as field information.

[0060] The sampling pixel selection unit 111 acquires the field information and, based on the field information, identifies the field area included in the captured image 20. For example, in the case of field information indicating the field area using GPS coordinate information, the sampling pixel selection unit 111 identifies corresponding pixels within the field area of ​​the captured image 20 by checking the field information against GPS coordinate information indicating the imaging range of the captured image 20, which is included in the metadata of the captured image, etc.

[0061] For example, a field area 211, which is a part of a captured image as shown in A of Fig. 9, is to be processed. The field area storage unit 201 stores information indicating this field area 211, and supplies this field information to the sampling pixel selection unit 111. Based on the field information, the sampling pixel selection unit 111 selects sampling pixels within this field area 211, as shown in B of Fig. 9, and omits the selection of sampling pixels outside the field area 211.

[0062] In this case, the method of selecting the sampled pixels is the same as in Fig. 1. That is, the sampled pixel selection unit 111 selects sparse sampled pixels within the field area 211 indicated by the field information, and supplies them to the clustering unit 112.

[0063] In this way, the sampling pixels to be processed by the clustering unit 112 are composed only of pixels within the field area. In other words, it is possible to prevent the clustering unit 112 and the complementation processing unit 113 from processing pixels outside the field area. Therefore, the image processing device 100 can suppress an unnecessary increase in clustering and an unnecessary increase in processing time.

[0064] <Flow of clustering process> An example of the flow of the clustering process in this case will be described with reference to the flowchart in Fig. 10. When the clustering process is started, in step S121, the sampling pixel selection unit 111 acquires the captured image 20. In addition, the sampling pixel selection unit 111 acquires farm field information from the farm field area storage unit 201.

[0065] In step S122, the sampling pixel selection unit 111 selects and determines sparse sampling pixels from the field region included in the captured image acquired in step S121 based on the field information.

[0066] The processes in steps S123 to S125 are executed in the same manner as the processes in steps S103 to S105 (FIG. 7). When the process in step S125 ends, the clustering process ends.

[0067] By performing each process as described above, the image processing device 100 can suppress an increase in the processing time for image clustering.

[0068] <Use of stitching information> For example, when a plurality of captured images of a part of a farm field are stitched together to generate a stitched image including the entire farm field as described above, the captured images generally include overlapping areas. In other words, it is generally difficult to control the capture so that the captured images do not overlap each other.

[0069] If the selection of sampling pixels is performed independently for each captured image, the pixels in the area where the captured images overlap may be selected as the sampling pixels for each of the captured images. In other words, the pixels at the same positions in the captured images may be selected as the sampling pixels. If multiple sampling pixels at the same positions occur in this way, clustering will be performed multiple times for one position. Therefore, there is a possibility that the processing time will increase unnecessarily due to such redundant processing.

[0070] A stitched image is generated by selecting one of the captured images in the overlapping area of ​​the captured images and stitching the captured images together in a state where the captured images do not overlap. That is, a stitching area is set in each captured image so that the captured images do not overlap with other captured images, and the stitching areas of the captured images are pasted together to generate a stitched image.

[0071] Furthermore, if the captured image includes an area outside the clustering target area (e.g., outside the farm field area), pixels in such an area may be selected as sampled pixels. In such a case, clustering may be performed on pixels in an area that does not require clustering, and the unnecessary processing may unnecessarily increase the processing time.

[0072] The stitching region described above can be set so as not to include such unnecessary regions. Therefore, by stitching the stitching regions of the captured images together, a stitched image that does not include regions that are not subject to clustering processing can be generated.

[0073] Therefore, only pixels in such stitching regions are selected as sampled pixels. In other words, in a region where multiple captured images overlap, sampled pixels are selected only in one of the captured images. Also, sampled pixels are selected so as not to include pixels in regions that are not the subject of clustering.

[0074] The stitching information includes information indicating such a stitching region of each captured image. That is, the stitching information includes information regarding the overlap of captured images and the region to be processed by clustering. Therefore, the stitching information is used to specify the stitching region, and sampling pixels are selected only within the specified stitching region. In this way, it is possible to suppress an increase in redundant clustering and unnecessary clustering, and to suppress an increase in unnecessary processing time.

[0075] <Image processing device> Fig. 11 is a block diagram showing an example of the main configuration of the image processing device 100 in this case. The captured image 20 is a stitched image formed by stitching together a plurality of captured images captured from above the farm field to be subjected to clustering processing. As shown in Fig. 11, the image processing device 100 in this case has a stitching information storage unit 231 in addition to the configuration shown in Fig. 1.

[0076] The stitching information storage unit 231 has a storage medium and stores stitching information including information indicating the stitching region of each captured image in (the storage region of) the storage medium. The information indicating the stitching region may be any information. For example, it may be information indicating the stitching region using GPS coordinate information, or information indicating the stitching region using coordinate information in the captured image.

[0077] The stitching information storage unit 231 supplies the stitching information stored in (the storage area of) its own storage medium to the sampling pixel selection unit 111 in response to a request from the sampling pixel selection unit 111, for example.

[0078] The sampling pixel selection unit 111 acquires the stitching information and specifies a stitching region of each captured image based on the stitching information. For example, as shown in A of Fig. 12, when selecting sampling pixels in a captured image 241 used to generate a stitched image 240, the sampling pixel selection unit 111 specifies a stitching region such as a diagonally shaded region shown in B of Fig. 12 based on the stitching information (taking into account overlap with the surrounding captured images 242 and 243), and selects sampling pixels in the stitching region.

[0079] 12B, the area where the captured image 241 and the captured image 242 overlap each other is set as a stitching joining area of ​​the captured image 242, so sampling pixels are selected in the processing for the captured image 242. Similarly, the area where the captured image 241 and the captured image 243 overlap each other is set as a stitching joining area of ​​the captured image 243, so sampling pixels are selected in the processing for the captured image 243.

[0080] Also, as shown in A of FIG. 12, when selecting sampling pixels in the captured image 244 used to generate the stitched image 240, the sampling pixel selection unit 111 identifies a stitching stitching area such as the shaded area shown in C of FIG. 12 based on the stitching information (taking into account the area to be clustered), and selects sampling pixels in that stitching stitching area.

[0081] 12C, the area of ​​the captured image 244 outside the stitched image 240 is set to be outside the stitching region. In other words, the area of ​​the captured image 244 within the stitched image 240 is set to be the stitching region.

[0082] In this case, the method of selecting the sampled pixels is the same as in Fig. 1. That is, the sampled pixel selection unit 111 selects sparse sampled pixels within the stitching region indicated by the stitching information, and supplies them to the clustering unit 112.

[0083] In this way, the image processing device 100 can prevent clustering from being performed multiple times for one position and from being performed in an unnecessary region. In other words, the image processing device 100 can prevent an increase in redundant clustering or unnecessary clustering, and can prevent an unnecessary increase in processing time.

[0084] <Flow of clustering process> An example of the flow of the clustering process in this case will be described with reference to the flowchart in Fig. 13. When the clustering process starts, in step S141, the sampling pixel selection unit 111 acquires the captured image 20. In addition, the sampling pixel selection unit 111 acquires stitching information from the stitching information storage unit 231.

[0085] In step S142, the sampling pixel selection unit 111 selects and determines sparse sampling pixels from the stitched region of the captured image acquired in step S141 based on the stitching information.

[0086] The processes in steps S143 to S145 are executed in the same manner as the processes in steps S103 to S105 (FIG. 7). When the process in step S145 ends, the clustering process ends.

[0087] By performing each process as described above, the image processing device 100 can suppress an increase in the processing time for image clustering.

[0088] <Using flat area information> In general, corners and edges of a captured image are areas where pixels of different classes contact each other, and it is difficult to determine which adjacent pixel to propagate color from. In other words, flat areas have higher clustering accuracy than corners and edges.

[0089] Therefore, in order to cluster the pixels in the flat region, sampling pixels are selected in the flat region. In other words, flat regions in the captured image are identified using flat region information, which is information about the flat region, and sampling pixels are selected in the flat region. In this way, more accurate clustering results can be obtained.

[0090] <Image processing device> 14 is a block diagram showing an example of the main configuration of the image processing device 100 in this case. As shown in Fig. 14, the image processing device 100 in this case has a flat region detection unit 261 in addition to the configuration shown in Fig. 1.

[0091] The flat region detection section 261 performs processing related to detection of a flat region. For example, the flat region detection section 261 acquires the captured image 20.

[0092] The captured image 20 may be the same as the captured image supplied to the sampling pixel selection unit 111 (i.e., the captured image to be clustered) or the captured image supplied to the complement processing unit 113 (i.e., the captured image to be used as a guide), or may be a captured image different from the captured image to be clustered or the captured image to be used as a guide, but of approximately the same time and range as the captured image to be clustered or the captured image to be used as a guide. For example, it may be another captured image obtained by another capture at approximately the same time and with approximately the same angle of view as the capture for obtaining the captured image to be clustered or the captured image to be used as a guide. For example, the captured image 20 in the wavelength range of visible light (RGB) may be supplied to the sampling pixel selection unit 111 or the complement processing unit 113, and the captured image 20 obtained by imaging the wavelength range of invisible light such as near-infrared light may be supplied to the flat region detection unit 261.

[0093] The flat region detection section 261 detects flat regions in the captured image and supplies the sampling pixel selection section 111 with flat region information indicating the detected flat regions.

[0094] The sampling pixel selection unit 111 acquires the flat region information and, based on the flat region information, selects sampling pixels in flat regions included in the captured image 20. In this case, the method of selecting sampling pixels is the same as in Fig. 1. That is, the sampling pixel selection unit 111 selects sparse sampling pixels in the flat region and supplies them to the clustering unit 112.

[0095] In this way, the image processing device 100 can obtain more accurate clustering results.

[0096] <Flow of clustering process> An example of the flow of the clustering process in this case will be described with reference to the flowchart of Fig. 15. When the clustering process starts, the sampling pixel selection section 111 acquires the captured image 20 in step S161.

[0097] In step S162, the flat region detection section 261 acquires the captured image 20 and detects flat regions in the captured image 20.

[0098] In step S163, the sampling pixel selection unit 111 selects and determines sparse sampling pixels from the flat region detected in step S162 of the captured image acquired in step S161.

[0099] The processes in steps S164 to S166 are executed in the same manner as the processes in steps S103 to S105 (FIG. 7). When the process in step S166 ends, the clustering process ends.

[0100] By executing each process as described above, the image processing device 100 can obtain a more accurate clustering result.

[0101] <Use of multiple information> In the above, the image processing device 100 has been described as selecting sampling pixels using any one of auxiliary information, stitching information, or flat region information, but the present invention is not limited to this. For example, the image processing device 100 may select sampling pixels using at least two or more of the field information, stitching information, and flat region information. In this way, it is possible to obtain the effects of using each piece of information. Of course, the image processing device 100 may select sampling pixels using information other than the above-mentioned information in addition to any one or more of these pieces of information.

[0102] <2. Second embodiment> <Global clustering and sparse local clustering> In image clustering, for example, information obtained in wide-area clustering, which is clustering of sparse pixels in a wide area (also called a global area), may be used to perform local clustering, which is clustering of pixels in a local area (also called a local area).

[0103] For example, assume that clustering is performed on a stitched image 270 (an image of the entire field) obtained by stitching together a plurality of captured images 271 (stitched areas) of a field, as shown on the right side of Figure 16, and the vegetation of the field is analyzed.

[0104] In such clustering, first, the entire field (the entire stitched image 270) is set as a wide area, and wide-area clustering is performed on the wide area (i.e., the entire stitched image 270) as pre-learning. For example, sparse wide-area sampled pixels 272 (white circles in the figure) are selected from the entire stitched image 270 (whole wide area) as wide-area sampled pixels that are sampled pixels to be subjected to wide-area clustering. Then, clustering (i.e., wide-area clustering) is performed on the wide-area sampled pixels 272.

[0105] Next, each captured image 271 (frame image) is treated as a local region, and local clustering is performed for each captured image 271 as additional learning using information obtained by wide-area clustering (e.g., a learning model, clustering results, etc.). For example, when the captured image 271A is to be processed, local sampling pixels are selected from the captured image 271A as local sampling pixels that are sampling pixels to be subjected to local clustering. Then, clustering (i.e., local clustering) is performed on the local sampling pixels.

[0106] Note that local sampling pixels may also be selected from captured images surrounding the captured image 271A to be processed (for example, the captured image 271B processed immediately before the captured image 271A, the captured image 271C processed immediately after the captured image 271A, etc.). This additional learning may be performed using information obtained by additional learning of the captured image immediately before (that is, information obtained by local clustering of the captured image 271B (for example, a learning model, clustering results, etc.)) (that is, sequential learning may be performed).

[0107] By using the information obtained in wide-area clustering in this way, a model that has been estimated once can be used, so that a stable model (one that is less affected by initial value fluctuations) can be obtained quickly in local clustering. Also, by targeting sparsely sampled pixels in wide-area clustering, clustering results can be obtained quickly. Therefore, it is possible to suppress an increase in processing time while suppressing a decrease in the robustness of image clustering.

[0108] In such a clustering method, the present technology described in the first embodiment is applied. For example, in the above-mentioned local clustering, sparse locally sampled pixels are clustered, and sparse information obtained by the clustering (for example, a learning model, a clustering result, etc.) is complemented by image filtering using an image signal as a guide, thereby deriving a dense clustering result. In this way, in the local clustering, it is possible to suppress an increase in processing time as described in the first embodiment.

[0109] <Image processing device> FIG. 17 is a block diagram showing an example of the main configuration of an image processing device in this case.

[0110] 17 is a device that performs image clustering, similar to the image processing device 100. That is, the image processing device 300 receives captured images 20 as input, performs image clustering on the captured images 20, and outputs the clustering result 30.

[0111] As in the first embodiment, the captured image 20 may be, for example, a stitched image formed by pasting together a plurality of captured images (P1 to Pn). It may also be a moving image made up of a plurality of frame images. Furthermore, it may be a file (a group of captured images) in which a plurality of captured images are compiled into one, or it may be a single captured image. Of course, it may also be an image other than a captured image (for example, a CG image, etc.). The captured image 20 may also be an image in the wavelength range of visible light (RGB), or may be an image of the wavelength range of invisible light such as near-infrared light. It may also be both of these.

[0112] In the following description, the captured image 20 will be described as a stitched image 270 corresponding to the entire farm field obtained by stitching together captured images 271 capturing parts of the farm field as in the example of Fig. 16. Also, the wide area (global region) will be described as the entire stitched image 270, and the local area will be described as each captured image 271 (one frame's worth of captured images).

[0113] Note that Fig. 17 shows the main processing units, data flows, etc., and is not limited to all that is shown in Fig. 17. In other words, in this image processing device 300, there may be processing units that are not shown as blocks in Fig. 17, and there may be processes and data flows that are not shown as arrows, etc. in Fig. 17.

[0114] As shown in FIG. 17, the image processing device 300 includes a pre-learning unit 311, an additional learning unit 312, and a coefficient storage unit 313.

[0115] The pre-learning unit 311 performs image clustering (wide-area clustering) on ​​a wide area (for example, the entire captured image 20) as pre-learning. At that time, the pre-learning unit 311 performs wide-area clustering on sparse pixels. The pre-learning unit 311 has a sampling pixel selection unit 321 and a clustering unit 322.

[0116] The sampling pixel selection unit 321 performs processing related to the selection of wide area sampling pixels which are target pixels for wide area clustering. For example, the sampling pixel selection unit 321 acquires a captured image 20. In addition, the sampling pixel selection unit 321 selects wide area sampling pixels from the captured image 20 so that the pixels are sparsely spaced.

[0117] The sampled pixel selection unit 321 supplies the selected sparse wide-area sampled pixels to a clustering unit 322 .

[0118] The clustering unit 322 performs processing related to wide-area clustering. For example, the clustering unit 322 acquires sparse wide-area sampled pixels supplied from the sampled pixel selection unit 321. The clustering unit 322 processes the acquired sparse wide-area sampled pixels and performs wide-area clustering (pre-learning). Any method can be used for this wide-area clustering. For example, a Gaussian Mixture Model (GMM) or a k-means method may be applied to this pre-learning.

[0119] The clustering unit 322 supplies the information obtained by this pre-learning (wide-area clustering), such as the pre-learning model coefficients and the wide-area clustering results, to the coefficient storage unit 313.

[0120] Furthermore, the additional learning unit 312 performs image clustering (local clustering) for local regions (e.g., each stitched captured image) using information obtained by wide-area clustering as an initial value as additional learning performed using information obtained by pre-learning as an initial value. Similar to the image processing device 100, the additional learning unit 312 performs clustering of sparsely sampled pixels and image filtering of the sparse information obtained by the clustering using the captured image 20 as a guide, thereby deriving a dense clustering result.

[0121] Similar to the image processing device 100 (FIG. 1), the additional learning unit 312 includes a sampling pixel selection unit 111, a clustering unit 112, and a complementation processing unit 113.

[0122] The sampling pixel selection unit 111 performs processing related to the selection of sparse sampling pixels, similar to the case of Fig. 1. For example, the sampling pixel selection unit 111 acquires a captured image 20. At that time, the entire stitching image may be supplied to the sampling pixel selection unit 111, or each captured image (frame image) constituting the stitching image may be supplied to the sampling pixel selection unit 111 one by one.

[0123] The sampling pixel selection unit 111 selects sparse sampling pixels (local sampling pixels) from each captured image (local region). At that time, the sampling pixel selection unit 111 may also select captured images (local regions) surrounding the captured image to be processed, such as the captured image (local region) that was the previous captured image to be processed or the captured image (local region) that is the next captured image to be processed, as targets for selecting local sampling pixels. In other words, the sampling pixel selection unit 111 may select sparse local sampling pixels from the local region to be processed or the local regions surrounding the local region.

[0124] The sampled pixel selection unit 111 supplies the selected local sampled pixels to a clustering unit 112 .

[0125] 1, the clustering unit 112 performs local clustering of the sparse locally sampled pixels, and supplies the obtained sparse information (e.g., model coefficients of additional learning and wide-area clustering results, etc.) to the complementation processing unit 113. However, in this case, the clustering unit 112 obtains information obtained by pre-learning (wide-area clustering) stored in the coefficient storage unit 313, such as model coefficients of pre-learning and wide-area clustering results, and performs local clustering using the information obtained by pre-learning (model coefficients of pre-learning and wide-area clustering results, etc.) as initial values.

[0126] That is, the clustering unit 112 acquires sparse local sampling pixels supplied from the sampling pixel selection unit 111. The clustering unit 112 also acquires sparse information (e.g., model coefficients of pre-learning, wide-area clustering results, etc.) obtained by pre-learning (wide-area clustering) stored in the coefficient storage unit 313. The clustering unit 112 processes the acquired sparse local sampling, sets the information obtained by the pre-learning (model coefficients of pre-learning, wide-area clustering results, etc.) as initial values, and performs local clustering as additional learning. The clustering unit 112 supplies the sparse information obtained by this additional learning (local clustering) (e.g., model coefficients of additional learning, local clustering results, etc.) to the complementation processing unit 113.

[0127] The clustering unit 112 may further perform local clustering (current local clustering) on ​​the local region to be processed this time by using information obtained in the local clustering (previous local clustering) on ​​the local region to be processed immediately before. In other words, the clustering unit 112 may perform sequential learning using the previous learning model, learning result, etc. as additional learning.

[0128] In this case, the clustering unit 112 stores information obtained by the sequential learning (e.g., model coefficients of the sequential learning and local clustering results, etc.) in the coefficient storage unit 313. That is, the clustering unit 112 acquires information obtained by the previous sequential learning in addition to information obtained by pre-learning from the coefficient storage unit 313, and performs local clustering (sequential learning). Then, the clustering unit 112 supplies the information obtained by the sequential learning (e.g., model coefficients of the sequential learning and local clustering results, etc.) to the complementation processing unit 113 and also to the coefficient storage unit 313 for storage. The information stored in the coefficient storage unit 313 is used for the next sequential learning (local clustering for the local region to be processed next).

[0129] This sequential learning makes it possible to rapidly derive a clustering result in a local area that reflects the wide-area clustering result and the clustering result in the adjacent local area. This makes it possible to suppress an increase in processing time while suppressing a decrease in the robustness of the image clustering.

[0130] In other words, if the above-described sequential learning is not performed as the additional learning, the supply of information obtained by the additional learning (such as the model coefficients of the additional learning and the local clustering results) to the coefficient memory unit 313 (i.e., arrow 341 in FIG. 17) can be omitted.

[0131] The complement processing unit 113 performs processing related to complementing sparse information, similar to the case of Fig. 1. For example, the complement processing unit 113 acquires sparse information (model coefficients of additional learning and clustering results) supplied from the clustering unit 112. Furthermore, the complement processing unit 113 performs image filtering (complementation processing) on ​​the sparse information using the image signal as a guide, and derives a dense clustering result as a local clustering result. The complement processing unit 113 outputs the clustering result 30 (dense clustering result) obtained by the complement processing to the outside of the image processing device 100 as the image processing result by the image processing device 100.

[0132] The coefficient storage unit 313 acquires information obtained by pre-learning (model coefficients of pre-learning and wide-area clustering results) supplied from the pre-learning unit 311 (clustering unit 322) and stores it in its own storage medium (storage area). When the additional learning unit 312 performs sequential learning, the coefficient storage unit 313 acquires information obtained by sequential learning (model coefficients of sequential learning and wide-area clustering results) supplied from the additional learning unit 312 (clustering unit 112) and stores it in its own storage medium (storage area). Furthermore, based on a request from the clustering unit 112, for example, the coefficient storage unit 313 supplies the clustering unit 112 with the information obtained by pre-learning and the information obtained by sequential learning that are stored in its own storage medium (storage area).

[0133] With such a configuration, the image processing device 300 can utilize a model that has been estimated once by using information obtained in wide-area clustering, and can quickly obtain a stable model (with less influence of initial value fluctuation) in local clustering. Also, the image processing device 100 can quickly obtain a clustering result for sparsely sampled pixels in wide-area clustering. Furthermore, the image processing device 100 performs local sampling of sparse locally sampled pixels, and performs image filtering using an image as a guide on the sparse information obtained in the local sampling, thereby quickly deriving a dense clustering result. Therefore, the image processing device 300 can suppress an increase in processing time while suppressing a decrease in robustness of image clustering.

[0134] <Flow of clustering process> An example of the flow of the clustering process in this case will be described with reference to the flowchart in Fig. 18. When the clustering process is started, in step S201, the sampling pixel selection unit 321 of the pre-learning unit 311 acquires the captured image 20 of the stitching image (for example, the stitching image 270) as a global image that is an image of a global region (wide area).

[0135] In step S202, the sampling pixel selection unit 321 selects and determines sparse wide-area sampling pixels from the global image acquired in step S201.

[0136] In step S203, the clustering unit 322 performs wide-area clustering on the sparse wide-area sampled pixels determined in step S202 as pre-learning.

[0137] In step S204, the coefficient storage unit 313 stores information obtained by the pre-learning performed in step S203 (for example, the model coefficients of the pre-learning and the wide-area clustering results).

[0138] In step S205, the sampling pixel selection unit 111 of the additional learning unit 312 acquires a local image to be processed from among multiple local images (images of local regions (local areas)) included in the global image acquired in step S201. The sampling pixel selection unit 111 also selects and determines sparse local sampling pixels from the local image to be processed.

[0139] In step S206, the clustering unit 112 performs local clustering on the sparse locally sampled pixels determined in step S205 as additional learning. At that time, the clustering unit 112 performs the sequential learning using information obtained by pre-learning stored in the coefficient storage unit 313 and information obtained by the previous additional learning (sequential learning).

[0140] In step S207, the coefficient storage unit 313 stores information (for example, model coefficients of the additional learning and local clustering results) obtained by the additional learning (sequential learning) performed in step S206.

[0141] In step S208, the complementation processing unit 113 acquires the captured image 20, and using the captured image 20 as a guide, performs image filtering on the sparse information (model coefficients of additional learning and clustering results) obtained by the processing of step S206, and complements the sparse information to derive a dense clustering result.

[0142] In step S209, the additional learning unit 312 determines whether additional learning has been performed on all local images. If it is determined that an unprocessed local image exists, the process returns to step S205, and the subsequent processes are executed on the next local image as the processing target. That is, the processes of steps S205 to S209 are executed for each local image. If it is determined in step S209 that all local images have been processed, the process proceeds to step S210.

[0143] In step S210, the complementation processing unit 113 outputs the thus optimized clustering result 30. When the process of step S210 ends, the clustering process ends.

[0144] By performing each process as described above, the image processing device 300 can suppress an increase in processing time while suppressing a decrease in the robustness of image clustering.

[0145] In addition, when sequential learning is not performed as the additional learning, the process of step S207 can be omitted. In addition, in step S206, the clustering unit 112 performs additional learning using information obtained by pre-learning stored in the coefficient storage unit 313.

[0146] <Wide-area sampling pixel reference> Note that the selection of local sampled pixels may be performed taking into consideration the selection result of the wide area sampled pixels. For example, local sampled pixels may be selected from pixels other than the wide area sampled pixels. In other words, the wide area sampled pixels may be excluded from candidates for local sampled pixels.

[0147] Furthermore, when the additional learning unit 312 (clustering unit 112) performs sequential learning as additional learning in which current local clustering is performed using information obtained by previous local clustering, the sampling pixel selection unit 111 may select current local sampling pixels by further taking into consideration the selection result of the previous local sampling pixels. For example, the current local sampling pixels may be selected from pixels other than the previous local sampling pixels. In other words, the previous local sampling pixels may be excluded from candidates for the current local sampling pixels.

[0148] In this way, in the additional learning, by performing local clustering while excluding wide-area sampled pixels, it is possible to suppress an increase in clustering redundancy and further suppress a decrease in robustness of image clustering. Also, in the sequential learning, by performing local clustering this time while excluding the previous locally sampled pixels, it is possible to suppress an increase in clustering redundancy and further suppress a decrease in robustness of image clustering. Therefore, it is possible to suppress an increase in processing time while suppressing a decrease in robustness of image clustering.

[0149] <Image processing device> 19 is a block diagram showing an example of the main configuration of an image processing device 300 in this case. As shown in Fig. 19, the image processing device 300 in this case has a sampled pixel storage unit 351 in addition to the configuration in the example of Fig. 17.

[0150] In this case, the sampled pixel selection unit 321 of the pre-learning unit 311 supplies the selected wide-area sampled pixels to the clustering unit 322 and also to the sampled pixel storage unit 351 .

[0151] The sampled pixel storage unit 351 has a storage medium and performs processing related to storage of sampled pixels. For example, the sampled pixel storage unit 351 acquires wide-area sampled pixels supplied from (the sampled pixel selection unit 321 of) the pre-learning unit 311, and stores them in (the storage area of) its own storage medium.

[0152] Furthermore, the sampled pixel storage unit 351 supplies the wide area sampled pixels stored in (the storage area of) its own storage medium to the sampled pixel selection unit 111, for example, based on a request from the sampled pixel selection unit 111.

[0153] In this case, the sampling pixel selection unit 111 acquires the wide area sampling pixels stored in the sampling pixel storage unit 351. The sampling pixel selection unit 111 selects sparse local sampling pixels from pixels other than the wide area sampling pixels in the local region (frame image) to be processed, and supplies them to the clustering unit 112. In this manner, the clustering unit 112 can suppress an increase in redundancy in clustering, and further suppress a decrease in robustness of image clustering.

[0154] When the additional learning unit 312 performs sequential learning, the sampled pixel selection unit 111 of the additional learning unit 312 supplies the selected local sampled pixels to the clustering unit 112 and also to the sampled pixel storage unit 351 .

[0155] In this case, the sampled pixel storage unit 351 acquires the local sampled pixels supplied from (the sampled pixel selection unit 111 of) the additional learning unit 312 and stores them in (the storage area of) its own storage medium. Also, based on a request from the sampled pixel selection unit 111, for example, the sampled pixel storage unit 351 supplies the wide area sampled pixels and the previous local sampled pixels stored in (the storage area of) its own storage medium to the sampled pixel selection unit 111.

[0156] Then, the sampling pixel selection unit 111 acquires the wide area sampled pixels and the previous local sampled pixels from the sampling pixel storage unit 351. The sampling pixel selection unit 111 selects sparse local sampled pixels from among pixels other than the wide area sampled pixels and the previous local sampled pixels in the local region (frame image) to be processed, and supplies them to the clustering unit 112. In this manner, the clustering unit 112 can suppress an increase in redundancy in clustering and further suppress a decrease in robustness of image clustering.

[0157] In other words, when the above-mentioned sequential learning is not performed as additional learning, the supply of the local sampled pixels to the sampled pixel storage unit 351 (that is, the arrow 361 in FIG. 19) can be omitted.

[0158] <Flow of clustering process> An example of the flow of the clustering process executed by the image processing device 300 in this case will be described with reference to the flowchart in Fig. 20. When the clustering process is started, the processes of steps S251 and S252 are executed in the same manner as the processes of steps S201 and S202 (Fig. 18).

[0159] In step S253, the sampled pixel storage unit 351 stores the sparse wide area sampled pixels determined in step S252.

[0160] When the process of step S253 ends, the processes of steps S254 and S255 are executed in the same manner as the processes of steps S203 and S204 (FIG. 18).

[0161] In step S256, the sampling pixel selection unit 111 of the additional learning unit 312 acquires a local image to be processed from the group of local images included in the global image acquired in step S251. The sampling pixel selection unit 111 also selects and determines sparse local sampling pixels from pixels other than the wide-area sampling pixels and the previous local sampling pixels of the local image to be processed.

[0162] In step S257, the sampled pixel storage unit 351 stores the sparse locally sampled pixels (current locally sampled pixels) determined in step S256.

[0163] When step S257 is completed, the processes of steps S258 to S260 are executed in the same manner as the processes of steps S206 to S208 (FIG. 18).

[0164] In step S261, the additional learning unit 312 determines whether additional learning has been performed on all local images. If it is determined that an unprocessed local image exists, the process returns to step S256, and the subsequent processes are executed on the next local image as the processing target. That is, the processes of steps S256 to S261 are executed for each local image. If it is determined in step S261 that all local images have been processed, the process proceeds to step S262.

[0165] In step S262, the complementation processing unit 113 outputs the thus optimized clustering result 30. When the process of step S262 ends, the clustering process ends.

[0166] By performing each process as described above, the image processing device 300 can suppress an increase in processing time while suppressing a decrease in the robustness of image clustering.

[0167] If sequential learning is not performed as the additional learning, the processes of steps S255 and S259 can be omitted. In step S256, the sampled pixel selection unit 111 selects sampled pixels using the wide area sampled pixels stored in the sampled pixel storage unit 351. In step S258, the clustering unit 112 performs additional learning using information obtained by pre-learning stored in the coefficient storage unit 313.

[0168] <Other configurations> In addition, in the image processing device 300 in FIG. 17, the pre-learning unit 311 may be configured as another device. That is, the image processing device 300 may have an additional learning unit 312 and a coefficient storage unit 313. In this case, the coefficient storage unit 313 acquires and stores sparse information (pre-learning model coefficients or clustering results, etc.) obtained in (the pre-learning unit 311 of) another device. In addition, the additional learning unit 312 performs local clustering of sparse locally sampled pixels using the sparse information obtained in (the pre-learning unit 311 of) another device stored in the coefficient storage unit 313.

[0169] 17, the pre-learning unit 311 and the coefficient storage unit 313 may be configured as other devices. That is, the image processing device 300 may have an additional learning unit 312. In this case, the additional learning unit 312 performs local clustering of sparse locally sampled pixels using sparse information obtained in (the pre-learning unit 311 of) another device that is stored in (the coefficient storage unit 313 of) the other device.

[0170] In either case, similarly to the case of FIG. 17, the image processing device 300 can suppress an increase in processing time while suppressing a decrease in robustness of image clustering.

[0171] In addition, in the image processing device 300 of FIG. 19, the pre-learning unit 311 may be configured as another device. That is, the image processing device 300 may have an additional learning unit 312, a coefficient storage unit 313, and a sampled pixel storage unit 351. In this case, the coefficient storage unit 313 acquires and stores sparse information (pre-learned model coefficients or clustering results, etc.) obtained in the other device (pre-learning unit 311). In addition, the sampled pixel storage unit 351 acquires and stores sparse wide-area sampled pixels selected in the other device (pre-learning unit 311). Furthermore, the additional learning unit 312 selects sparse local sampled pixels based on the sparse wide-area sampled pixels selected in the other device (pre-learning unit 311) stored in the sampled pixel storage unit 351, and performs local clustering of the selected sparse local sampled pixels using the sparse information obtained in the other device (pre-learning unit 311) stored in the coefficient storage unit 313.

[0172] In addition, in the image processing device 300 of FIG. 19, the pre-learning unit 311 and the coefficient storage unit 313 may be configured as other devices. That is, the image processing device 300 may have an additional learning unit 312 and a sampled pixel storage unit 351. In this case, the sampled pixel storage unit 351 acquires and stores wide area sampled pixels selected in the other device (the pre-learning unit 311). In addition, the additional learning unit 312 selects sparse local sampled pixels based on the sparse wide area sampled pixels selected in the other device (the pre-learning unit 311) stored in the sampled pixel storage unit 351, and performs local clustering of the selected sparse local sampled pixels using sparse information obtained in the other device (the pre-learning unit 311) stored in the other device (the coefficient storage unit 313).

[0173] Furthermore, in the image processing device 300 of FIG. 19, the pre-learning unit 311 and the sampled pixel storage unit 351 may be configured as other devices. That is, the image processing device 300 may have an additional learning unit 312 and a coefficient storage unit 313. In this case, the coefficient storage unit 313 acquires and stores information (pre-learning model coefficients or clustering results, etc.) obtained in the other device (pre-learning unit 311). In addition, the additional learning unit 312 selects sparse local sampled pixels based on sparse wide-area sampled pixels selected in the other device (pre-learning unit 311) stored in the other device (sampled pixel storage unit 351), and performs local clustering of the selected sparse local sampled pixels using sparse information obtained in the other device (pre-learning unit 311) stored in the coefficient storage unit 313.

[0174] 19, the pre-learning unit 311, the coefficient storage unit 313, and the sampled pixel storage unit 351 may be configured as other devices. That is, the image processing device 300 may have an additional learning unit 312. In this case, the additional learning unit 312 selects sparse local sampled pixels based on sparse wide-area sampled pixels selected in another device (the pre-learning unit 311) and stored in the other device (the sampled pixel storage unit 351), and performs local clustering of the selected sparse local sampled pixels using sparse information obtained in the other device (the pre-learning unit 311) and stored in the other device (the coefficient storage unit 313).

[0175] In either case, similarly to the case of FIG. 19, the image processing device 300 can suppress an increase in processing time while suppressing a decrease in the robustness of image clustering.

[0176] Of course, in either of these cases, the additional learning section 312 can perform the above-mentioned sequential learning as additional learning, similarly to the cases of FIG. 17 and FIG.

[0177] Furthermore, the image processing device 300 may select local sampling pixels using at least one of the field information, stitching information, and flat area information described in the first embodiment. In this way, the effects of using each piece of information in additional learning can be obtained. Of course, the image processing device 300 may select sampling pixels using information other than those described above in addition to one or more of these pieces of information.

[0178] In the present embodiment, the captured image 20 is a stitched image, but the captured image 20 may be a moving image made up of a plurality of frame images, a file (a group of captured images) in which a plurality of captured images are compiled, or a single captured image. Of course, the captured image 20 may be an image other than a captured image (e.g., a CG image, etc.). The captured image 20 may be an image in the wavelength range of visible light (RGB), or may be an image of the wavelength range of invisible light such as near-infrared light. It may also be both of them.

[0179] Furthermore, the wide area (global area) does not have to be the entire captured image 20, and the local area (local area) does not have to be one frame of the captured image. The local area only needs to be an area within the wide area that is narrower than the wide area. To this extent, the wide area and the local area may each be any area within the captured image 20.

[0180] <3. Third embodiment> <Global clustering and dense local clustering> As described above in the second embodiment, in the clustering of an image, for example, local clustering may be performed using sparse information obtained in the wide-area clustering of sparsely sampled pixels. Then, the local clustering may be performed on densely sampled pixels. That is, instead of performing local clustering of sparsely sampled pixels and deriving a dense clustering result by image filtering guided by an image signal on the obtained sparse information as in the second embodiment, local clustering may be performed on densely sampled pixels.

[0181] In this case, as in the second embodiment, a model estimated once can be used in wide-area clustering, so that a stable model (with less influence of initial value fluctuation) can be obtained quickly in local clustering. Also, by targeting sparsely sampled pixels in wide-area clustering, clustering results can be obtained quickly. Therefore, it is possible to suppress an increase in processing time while suppressing a decrease in robustness of image clustering.

[0182] <Image processing device> Fig. 21 is a block diagram showing an example of the main configuration of an image processing device in this case. The image processing device 400 shown in Fig. 21 is a device that performs image clustering, similar to the image processing device 300. That is, the image processing device 400 receives a captured image 20 as an input, performs image clustering on the captured image 20, and outputs the clustering result 30.

[0183] As in the second embodiment, the captured image 20 may be, for example, a stitched image formed by pasting together a plurality of captured images (P1 to Pn). It may also be a moving image made up of a plurality of frame images. Furthermore, it may be a file (a group of captured images) in which a plurality of captured images are compiled into one, or it may be a single captured image. Of course, it may also be an image other than a captured image (for example, a CG image, etc.). The captured image 20 may also be an image in the wavelength range of visible light (RGB), or may be an image of the wavelength range of invisible light such as near-infrared light. It may also be both of these.

[0184] In the following description, the captured image 20 will be described as a stitched image 270 corresponding to the entire farm field obtained by stitching together captured images 271 capturing parts of the farm field as in the example of Fig. 16. Also, the wide area (global region) will be described as the entire stitched image 270, and the local area will be described as each captured image 271 (one frame's worth of captured images).

[0185] Note that Fig. 21 shows the main processing units, data flows, etc., and is not limited to all that is shown in Fig. 21. In other words, in this image processing device 400, there may be processing units that are not shown as blocks in Fig. 21, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 21.

[0186] As shown in FIG. 21, the image processing device 400 has a pre-learning unit 311, an additional learning unit 312, and a coefficient storage unit 313, similar to the image processing device 300 (FIG. 17).

[0187] The pre-learning unit 311, like the image processing device 300 (FIG. 17), has a sampling pixel selection unit 321 and a clustering unit 322, performs wide-area clustering of sparse wide-area sampling pixels as pre-learning, and supplies information obtained by the pre-learning to the coefficient storage unit 313. The information obtained by this pre-learning is information corresponding to each sampling pixel (i.e., in a sparse state) obtained by wide-area clustering. For example, it may be a model coefficient of the pre-learning, a clustering result, or both.

[0188] The coefficient storage unit 313 has a configuration similar to that of the image processing device 300 (FIG. 17), and stores sparse information (e.g., pre-learning model coefficients and wide-area clustering results) supplied from the pre-learning unit 311. In addition, the coefficient storage unit 313 supplies the stored sparse information to the additional learning unit 312 (the clustering unit 412) in response to a request from the additional learning unit 312 (the clustering unit 412), for example.

[0189] The additional learning unit 312 performs additional learning using sparse information obtained by pre-learning (e.g., pre-learned model coefficients, wide-area clustering results, etc.) as initial values, similarly to the case of the image processing device 300 (FIG. 17). However, in this case, the additional learning unit 312 performs local clustering of densely locally sampled pixels as the additional learning, and derives a dense clustering result.

[0190] Any method can be used for this local clustering. For example, the Structure-constrained Gaussian Mixture Model (SC-GMM) can be applied to this additional learning. In SC-GMM, optimization that takes image structure information into account is introduced to clustering in color space. For example, the texture or edge structure is used to find the adjacent relationship between pixels, and class classification is performed based on the adjacent relationship. In this way, more accurate clustering can be performed.

[0191] As shown in FIG. 21, the additional learning unit 312 in this case has a sampling pixel selection unit 411, a clustering unit 412, and an optimization unit 413.

[0192] The sampling pixel selection unit 411 performs processing related to the selection of local sampling pixels. For example, the sampling pixel selection unit 411 acquires the captured image 20. At that time, the entire stitching image may be supplied to the sampling pixel selection unit 411, or each captured image (frame image) constituting the stitching image may be supplied to the sampling pixel selection unit 411 one by one.

[0193] Furthermore, the sampling pixel selection unit 411 selects some or all of the pixels of each captured image (local region) as local sampling pixels. At this time, the sampling pixel selection unit 411 selects the local sampling pixels so that they are dense. Note that the sampling pixel selection unit 411 may also select captured images (local regions) surrounding the captured image to be processed, such as the captured image (local region) that was the previous captured image to be processed or the captured image (local region) that is the next captured image to be processed, as targets for selecting local sampling pixels. In other words, the sampling pixel selection unit 411 may select dense local sampling pixels from the local region to be processed or the local regions surrounding it.

[0194] The sampled pixel selection unit 411 supplies the selected dense local sampled pixels to a clustering unit 412 .

[0195] The clustering unit 412 performs processing related to local clustering. For example, the clustering unit 412 acquires dense local sampled pixels supplied from the sampled pixel selection unit 411. In addition, the clustering unit 412 acquires sparse information (for example, model coefficients of pre-learning, wide-area clustering results, etc.) obtained by pre-learning (wide-area clustering) stored in the coefficient storage unit 313.

[0196] The clustering unit 412 performs local clustering of dense local sampling using sparse information obtained by the pre-learning as an initial value. The clustering unit 412 supplies information obtained by this additional learning (local clustering of densely local sampled pixels) to the optimization unit 413. The information obtained by this additional learning is information corresponding to each sampled pixel (i.e., dense state) obtained by local clustering. For example, it may be a model coefficient of the additional learning, a clustering result, or both.

[0197] The clustering unit 412 may further perform local clustering (current local clustering) on ​​the local region to be processed this time by using information obtained in local clustering (previous local clustering) on ​​the local region to be processed immediately before. In other words, the clustering unit 412 may perform sequential learning using the previous learning model, clustering results, etc. as additional learning.

[0198] In this case, the clustering unit 412 stores dense information obtained by the sequential learning (model coefficients of the sequential learning, local clustering results, etc.) in the coefficient storage unit 313. The clustering unit 412 also acquires dense information obtained by the previous sequential learning in addition to sparse information obtained by pre-learning from the coefficient storage unit 313, and performs local clustering (sequential learning). The clustering unit 412 then supplies the information obtained by the sequential learning (model coefficients of the sequential learning, local clustering results, etc.) to the optimization unit 413 and also to the coefficient storage unit 313 for storage. The information stored in the coefficient storage unit 313 is used for the next sequential learning (local clustering for the local region to be processed next).

[0199] This sequential learning makes it possible to rapidly derive a clustering result in a local area that reflects the wide-area clustering result and the clustering result in the adjacent local area. This makes it possible to suppress an increase in processing time while suppressing a decrease in the robustness of the image clustering.

[0200] In other words, if the above-described sequential learning is not performed as the additional learning, the supply of information obtained by the additional learning (such as the model coefficients of the additional learning and the local clustering results) to the coefficient memory unit 313 (i.e., arrow 421 in FIG. 21) can be omitted.

[0201] The optimization unit 413 performs processing related to optimization of the clustering result. For example, the optimization unit 413 acquires information obtained by additional learning (model coefficients of additional learning, local clustering results, etc.) supplied from the clustering unit 412. In addition, the optimization unit 413 acquires the captured image 20.

[0202] This captured image 20 may be the same as the captured image 20 (i.e., the captured image to be clustered) supplied to the sampling pixel selection unit 321 or the sampling pixel selection unit 411, or may be a captured image different from the captured image to be clustered, but at approximately the same time and in approximately the same range as the captured image to be clustered. For example, it may be another captured image obtained by another capture at approximately the same time and with approximately the same angle of view as the capture for obtaining the captured image to be clustered. For example, the captured image 20 in the wavelength range of visible light (RGB) may be supplied to the sampling pixel selection unit 321 or the sampling pixel selection unit 411, and the captured image 20 obtained by imaging the wavelength range of invisible light such as near-infrared light may be supplied to the optimization unit 413.

[0203] The optimization unit 413 optimizes dense information obtained by additional learning using the captured image 20, and derives an optimized dense clustering result. For example, the optimization unit 413 obtains adjacent relations between pixels in consideration of image structure information (texture and edge structure) of the captured image 20, and optimizes model coefficients and clustering results based on the adjacent relations.

[0204] The optimization unit 413 outputs the clustering result 30 obtained by this process (that is, the clustering result on which the optimization process has been performed) to the outside of the image processing device 400 as the image processing result by the image processing device 400.

[0205] The image processing device 400, having such a configuration, can perform local clustering by utilizing a model estimated once in wide-area clustering. Therefore, the image processing device 400 can quickly obtain a stable model (less affected by initial value fluctuation) in local clustering. Furthermore, by having such a configuration, the image processing device 400 can quickly obtain clustering results even in wide-area clustering by targeting sparsely sampled pixels. Therefore, the image processing device 400 can suppress an increase in processing time while suppressing a decrease in robustness of image clustering.

[0206] Clustering result 431 shown in A of Fig. 22 shows an example of a clustering result derived by the image processing device 400. Clustering result 432 shown in B of Fig. 22 shows an example of a clustering result derived by the image processing device 300. In other words, almost the same clustering result can be obtained by any image processing device. In other words, the image processing device 400 can suppress an increase in processing time while suppressing a decrease in robustness of image clustering, similar to the case of the image processing device 300.

[0207] <Flow of clustering process> An example of the flow of the clustering process executed by the image processing device 400 will be described with reference to the flowchart in Fig. 23. When the clustering process is started, each process from step S301 to step S304 is executed in the same manner as each process from step S201 to step S204 (Fig. 18).

[0208] In step S305, the sampling pixel selection unit 411 of the additional learning unit 312 acquires a local image to be processed from the local images included in the global image acquired in step S301. The sampling pixel selection unit 411 also selects and determines dense local sampling pixels from the local image to be processed.

[0209] In step S306, the clustering unit 412 performs local clustering on the densely local sampled pixels determined in step S305 as additional learning. At that time, the clustering unit 412 performs the sequential learning using information obtained by pre-learning stored in the coefficient storage unit 313 and information obtained by the previous additional learning (sequential learning).

[0210] In step S307, the coefficient storage unit 313 stores information (for example, model coefficients of the additional learning and local clustering results) obtained by the additional learning (sequential learning) performed in step S306.

[0211] In step S308, the optimization unit 413 optimizes information (for example, model coefficients of the additional learning and local clustering results) obtained by the additional learning (sequential learning) performed in step S306, and derives the optimized clustering results.

[0212] In step S309, the additional learning unit 312 determines whether additional learning has been performed on all local images. If it is determined that an unprocessed local image exists, the process returns to step S305, and the subsequent processes are executed on the next local image as the processing target. That is, the processes of steps S305 to S309 are executed for each local image. If it is determined in step S309 that all local images have been processed, the process proceeds to step S310.

[0213] In step S310, optimization unit 413 outputs the thus optimized clustering result 30. When the process of step S310 ends, the clustering process ends.

[0214] By performing each process as described above, the image processing device 400 can suppress an increase in processing time while suppressing a decrease in the robustness of image clustering.

[0215] In addition, when sequential learning is not performed as the additional learning, the process of step S307 can be omitted. In addition, in step S306, the clustering unit 412 performs additional learning using information obtained by pre-learning stored in the coefficient storage unit 313.

[0216] <Wide-area sampling pixel reference> As in the case of the image processing device 300 described in the second embodiment, the selection of local sampled pixels may be performed taking into consideration the selection result of the wide area sampled pixels. For example, the local sampled pixels may be selected from pixels other than the wide area sampled pixels. In other words, the wide area sampled pixels may be excluded from candidates for local sampled pixels.

[0217] Furthermore, when the additional learning unit 312 (clustering unit 412) performs sequential learning as additional learning in which current local clustering is performed using information obtained by previous local clustering, the sampling pixel selection unit 411 may select current local sampling pixels further taking into consideration the selection result of the previous local sampling pixels. For example, the current local sampling pixels may be selected from pixels other than the previous local sampling pixels. In other words, the previous local sampling pixels may be excluded from candidates for the current local sampling pixels.

[0218] In this way, in the additional learning, by performing local clustering while excluding wide-area sampled pixels, it is possible to suppress an increase in clustering redundancy and further suppress a decrease in robustness of image clustering. Also, in the sequential learning, by performing local clustering this time while excluding the previous locally sampled pixels, it is possible to suppress an increase in clustering redundancy and further suppress a decrease in robustness of image clustering. Therefore, it is possible to suppress an increase in processing time while suppressing a decrease in robustness of image clustering.

[0219] <Image processing device> Fig. 24 is a block diagram showing an example of the main configuration of the image processing device 400 in this case. As shown in Fig. 24, the image processing device 400 in this case has a sampled pixel storage unit 351 in addition to the configuration in the example of Fig. 21, similar to the case of the image processing device 300 in Fig. 19.

[0220] In this case, the sampled pixel selection unit 321 of the pre-learning unit 311 supplies the selected wide-area sampled pixels to the clustering unit 322 and also to the sampled pixel storage unit 351 .

[0221] The sampled pixel storage unit 351 has a storage medium and performs processing related to storage of sampled pixels, similar to the case of Fig. 19. For example, the sampled pixel storage unit 351 acquires wide-area sampled pixels supplied from (the sampled pixel selection unit 321 of) the pre-learning unit 311, and stores them in (the storage area of) its own storage medium.

[0222] Furthermore, the sampled pixel storage unit 351 supplies the wide area sampled pixels stored in (the storage area of) its own storage medium to the sampled pixel selection unit 411, for example, based on a request from the sampled pixel selection unit 411.

[0223] In this case, the sampling pixel selection unit 411 acquires the wide area sampling pixels stored in the sampling pixel storage unit 351. The sampling pixel selection unit 411 selects dense local sampling pixels from among pixels other than the wide area sampling pixels in the local region (frame image) to be processed, and supplies them to the clustering unit 412. In this manner, the clustering unit 412 can suppress an increase in redundancy in clustering, and further suppress a decrease in robustness of image clustering.

[0224] When the additional learning unit 312 performs sequential learning, the sampled pixel selection unit 411 of the additional learning unit 312 supplies the selected local sampled pixels to the clustering unit 412 and also to the sampled pixel storage unit 351 .

[0225] In this case, the sampled pixel storage unit 351 acquires the local sampled pixels supplied from (the sampled pixel selection unit 411 of) the additional learning unit 312 and stores them in (the storage area of) its own storage medium. Also, based on a request from the sampled pixel selection unit 411, for example, the sampled pixel storage unit 351 supplies the wide area sampled pixels and the previous local sampled pixels stored in (the storage area of) its own storage medium to the sampled pixel selection unit 411.

[0226] Then, the sampling pixel selection unit 411 acquires the wide area sampled pixels and the previous local sampled pixels from the sampling pixel storage unit 351. The sampling pixel selection unit 411 selects dense local sampled pixels from among pixels other than the wide area sampled pixels and the previous local sampled pixels in the local region (frame image) to be processed, and supplies them to the clustering unit 412. In this manner, the clustering unit 412 can suppress an increase in redundancy in clustering and further suppress a decrease in robustness of image clustering.

[0227] In other words, when the above-mentioned sequential learning is not performed as additional learning, the supply of the local sampled pixels to the sampled pixel storage unit 351 (that is, the arrow 441 in FIG. 24) can be omitted.

[0228] <Flow of clustering process> An example of the flow of the clustering process executed by the image processing device 400 in this case will be described with reference to the flowchart in Fig. 25. When the clustering process is started, the processes of steps S351 and S352 are executed in the same manner as the processes of steps S301 and S302 (Fig. 23).

[0229] In step S353, the sampled pixel storage unit 351 stores the sparse wide-area sampled pixels determined in step S352.

[0230] When the process of step S353 ends, the processes of steps S354 and S355 are executed in the same manner as the processes of steps S303 and S304 (FIG. 23).

[0231] In step S356, the sampling pixel selection unit 411 of the additional learning unit 312 acquires a local image to be processed from the group of local images included in the global image acquired in step S351. The sampling pixel selection unit 411 also selects and determines dense local sampling pixels from pixels other than the wide-area sampling pixels and the previous local sampling pixels of the local image to be processed.

[0232] In step S357, the sampled pixel storage unit 351 stores the densely locally sampled pixels (currently locally sampled pixels) determined in step S356.

[0233] When step S357 is completed, the processes of steps S358 to S360 are executed in the same manner as the processes of steps S306 to S308 (FIG. 23).

[0234] In step S361, the additional learning unit 312 determines whether additional learning has been performed on all local images. If it is determined that an unprocessed local image exists, the process returns to step S356, and the subsequent processes are executed on the next local image as the processing target. That is, the processes of steps S356 to S361 are executed for each local image. If it is determined in step S361 that all local images have been processed, the process proceeds to step S362.

[0235] In step S362, optimization unit 413 outputs the thus optimized clustering result 30. When the process of step S362 ends, the clustering process ends.

[0236] By performing each process as described above, the image processing device 400 can suppress an increase in processing time while suppressing a decrease in the robustness of image clustering.

[0237] If sequential learning is not performed as the additional learning, the processes of steps S355 and S359 can be omitted. In step S356, the sampled pixel selection unit 411 selects sampled pixels using the wide area sampled pixels stored in the sampled pixel storage unit 351. In step S358, the clustering unit 412 performs additional learning using information obtained by prior learning stored in the coefficient storage unit 313.

[0238] <Other configurations> In addition, in the image processing device 400 of FIG. 21, the pre-learning unit 311 may be configured as another device. That is, the image processing device 400 may have an additional learning unit 312 and a coefficient storage unit 313. In this case, the coefficient storage unit 313 acquires and stores sparse information (pre-learning model coefficients or clustering results, etc.) obtained in (the pre-learning unit 311 of) another device. In addition, the additional learning unit 312 performs local clustering of densely local sampled pixels using the sparse information obtained in (the pre-learning unit 311 of) another device stored in the coefficient storage unit 313.

[0239] 21, the pre-learning unit 311 and the coefficient storage unit 313 may be configured as other devices. That is, the image processing device 400 may have an additional learning unit 312. In this case, the additional learning unit 312 performs local clustering of dense locally sampled pixels using sparse information obtained in (the pre-learning unit 311 of) another device that is stored in (the coefficient storage unit 313 of) the other device.

[0240] In either case, similarly to the case of FIG. 21, the image processing device 400 can suppress an increase in processing time while suppressing a decrease in the robustness of image clustering.

[0241] Also, in the image processing device 400 of FIG. 24, the pre-learning unit 311 may be configured as another device. That is, the image processing device 400 may have an additional learning unit 312, a coefficient storage unit 313, and a sampled pixel storage unit 351. In this case, the coefficient storage unit 313 acquires and stores sparse information (pre-learned model coefficients or clustering results, etc.) acquired in the other device (pre-learning unit 311). Also, the sampled pixel storage unit 351 acquires and stores sparse wide-area sampled pixels selected in the other device (pre-learning unit 311). Furthermore, the additional learning unit 312 selects dense local sampled pixels based on the sparse wide-area sampled pixels selected in the other device (pre-learning unit 311) stored in the sampled pixel storage unit 351, and performs local clustering of the selected dense local sampled pixels using the sparse information acquired in the other device (pre-learning unit 311) stored in the coefficient storage unit 313.

[0242] In addition, in the image processing device 400 of FIG. 24, the pre-learning unit 311 and the coefficient storage unit 313 may be configured as other devices. That is, the image processing device 400 may have an additional learning unit 312 and a sampled pixel storage unit 351. In this case, the sampled pixel storage unit 351 acquires and stores wide area sampled pixels selected in the other device (the pre-learning unit 311). In addition, the additional learning unit 312 selects dense local sampled pixels based on the sparse wide area sampled pixels selected in the other device (the pre-learning unit 311) stored in the sampled pixel storage unit 351, and performs local clustering of the selected dense local sampled pixels using sparse information obtained in the other device (the pre-learning unit 311) stored in the other device (the coefficient storage unit 313).

[0243] Furthermore, in the image processing device 400 of FIG. 24, the pre-learning unit 311 and the sampled pixel storage unit 351 may be configured as other devices. That is, the image processing device 400 may have an additional learning unit 312 and a coefficient storage unit 313. In this case, the coefficient storage unit 313 acquires and stores information (pre-learning model coefficients or clustering results, etc.) obtained in the other device (pre-learning unit 311). In addition, the additional learning unit 312 selects dense local sampled pixels based on sparse wide-area sampled pixels selected in the other device (pre-learning unit 311) stored in the other device (sampled pixel storage unit 351), and performs local clustering of the selected dense local sampled pixels using sparse information obtained in the other device (pre-learning unit 311) stored in the coefficient storage unit 313.

[0244] 24, the pre-learning unit 311, the coefficient storage unit 313, and the sampled pixel storage unit 351 may be configured as other devices. That is, the image processing device 400 may have an additional learning unit 312. In this case, the additional learning unit 312 selects dense local sampled pixels based on sparse wide area sampled pixels selected in another device (the pre-learning unit 311) and stored in the other device (the sampled pixel storage unit 351), and performs local clustering of the selected dense local sampled pixels using sparse information obtained in the other device (the pre-learning unit 311) and stored in the other device (the coefficient storage unit 313).

[0245] In either case, similarly to the case of FIG. 24, the image processing device 400 can suppress an increase in processing time while suppressing a decrease in the robustness of image clustering.

[0246] Of course, in either case, the additional learning section 312 can perform the above-mentioned sequential learning as additional learning, similarly to the cases of FIG. 21 and FIG.

[0247] Furthermore, the image processing device 400 may select local sampling pixels using at least one of the farm field information, stitching information, and flat area information described in the first embodiment. In this way, the effects of using each piece of information in additional learning can be obtained. Of course, the image processing device 400 may select sampling pixels using information other than those described above in addition to one or more of these pieces of information.

[0248] In the present embodiment, the captured image 20 is a stitched image, but the captured image 20 may be a moving image made up of a plurality of frame images, a file (a group of captured images) in which a plurality of captured images are compiled, or a single captured image. Of course, the captured image 20 may be an image other than a captured image (e.g., a CG image, etc.). The captured image 20 may be an image in the wavelength range of visible light (RGB), or may be an image of the wavelength range of invisible light such as near-infrared light. It may also be both of them.

[0249] Furthermore, the wide area (global area) does not have to be the entire captured image 20, and the local area (local area) does not have to be one frame of the captured image. The local area only needs to be an area within the wide area that is narrower than the wide area. To this extent, the wide area and the local area may each be any area within the captured image 20.

[0250] <4. Fourth embodiment> <Application to vegetation area analysis> The image processing device described above in the first to third embodiments (the image processing device 100, the image processing device 300, or the image processing device 400) can be used for analyzing vegetation areas, for example.

[0251] <Image processing device> 26 is a diagram showing an example of an embodiment of an image processing device to which the present technology is applied. The image processing device 500 is a device that performs vegetation region analysis, and receives as input a captured image 20 of, for example, a farm field, performs vegetation region analysis of the captured image 20 using image clustering, and outputs vegetation region information 520 that is the analysis result.

[0252] As in the above-described embodiments, the captured image 20 may be, for example, a stitched image formed by pasting together a plurality of captured images (P1 to Pn). It may also be a moving image made up of a plurality of frame images. Furthermore, it may be a file (a group of captured images) in which a plurality of captured images are compiled into one, or it may be a single captured image. The captured image 20 may be an image in the wavelength range of visible light (RGB), or may be an image of the wavelength range of invisible light such as near-infrared light. It may also be both of these.

[0253] Note that Fig. 26 shows the main processing units, data flows, etc., and is not limited to all that is shown in Fig. 26. In other words, in this image processing device 500, there may be processing units not shown as blocks in Fig. 26, and there may be processing and data flows not shown as arrows, etc. in Fig. 26.

[0254] 26, the image processing device 500 has a clustering unit 511 and a vegetation region determination unit 512. The clustering unit 511 performs clustering on the captured image 20 to derive a dense clustering result. The above-mentioned image processing device can be applied to this clustering unit 511. That is, the clustering unit 511 has a configuration similar to any one of the above-mentioned image processing devices, and performs similar processing (clustering) to derive a clustering result from the captured image 20. The clustering unit 511 supplies the clustering result to the vegetation region determination unit 512.

[0255] The vegetation region determination unit 512 performs processing related to the determination of a vegetation region. For example, the vegetation region determination unit 512 acquires the clustering result supplied from the clustering unit 511. The vegetation region determination unit 512 also acquires the captured image 20. The vegetation region determination unit 512 determines a vegetation region using this information, and outputs vegetation region information 520, which is the analysis result. In this manner, the image processing device 500 can more quickly generate an analysis result of a vegetation region in which a reduction in robustness is suppressed.

[0256] <Flow of clustering process> An example of the flow of the clustering process in this case will be described with reference to the flowchart of Fig. 27. When the clustering process starts, the clustering section 511 acquires the captured image 20 in step S501.

[0257] In step S502, the clustering unit 511 performs a clustering process to obtain a dense clustering result. The above-mentioned clustering process can be applied to this clustering process. That is, the clustering unit 511 performs the clustering process in a similar flow to any one of the above-mentioned flowcharts to derive a dense clustering result.

[0258] In step S503, the vegetation region determining unit 512 determines a vegetation region based on the clustering result obtained in step S502, and obtains vegetation region information 520.

[0259] In step S504, the vegetation region determining unit 512 outputs the vegetation region information 520 obtained by the process of step S503. When the process of step S504 ends, the clustering process ends.

[0260] By executing each process as described above, the image processing device 500 can obtain a more accurate clustering result. Therefore, the image processing device 500 can generate the vegetation area information 520 with reduced robustness at a higher speed.

[0261] <5. Fifth embodiment> <Application to medical devices> The present technology described in the first to third embodiments is not limited to the analysis of vegetation regions as described above, but can be applied to any technology in any field. For example, the present technology can be used in medical equipment.

[0262] For example, a CT (Computed Tomography) inspection device irradiates X-rays onto a human body while rotating it, collects the intensity of the X-rays that penetrate the body with a detector, and analyzes and calculates the obtained data with a computer to create various images. For example, as shown in A of FIG. 28, a CT inspection device can obtain a tomographic image of an arbitrary position and direction, such as an XY plane, a YZ plane, or an XZ plane, by irradiating X-rays onto a patient 601. For example, a plurality of CT images 611 can be obtained, such as CT images 611-1 to 611-5 shown in B of FIG. 28. The present technology may be applied to clustering of a plurality of CT images 611 obtained by such a CT inspection.

[0263] In this case, for example, as shown in A of Fig. 29, one CT image 651 (CT Slice) may be treated as a wide area (global area), and a predetermined part 652 of the CT image 651, such as a block, may be treated as a local area (local area), and the clustering may be performed by applying the above-mentioned present technology. That is, in this case, both the wide area and the local area are treated as two-dimensional planes, and clustering is performed on each CT image one by one. In this case, processing can be performed in the same manner as in the case of the captured image of the farm field described above.

[0264] For example, when applying the method described in the third embodiment, wide-area clustering (pre-learning) is performed on sparse wide-area sampled pixels selected from the entire CT image 651, and the obtained sparse information (pre-learned model coefficients, clustering results, etc.) is used as initial values ​​to perform local clustering (additional learning) on ​​dense local sampled pixels in each block, and dense clustering results are derived.

[0265] Furthermore, for example, when applying the method described in the second embodiment, wide-area clustering (pre-learning) is performed on sparse wide-area sampled pixels selected from the entire CT image 651, and the obtained sparse information (pre-learned model coefficients, clustering results, etc.) is used as initial values ​​to perform local clustering (additional learning) on ​​sparse local sampled pixels in each block, and the obtained sparse information (additional learning model coefficients, clustering results, etc.) is complemented by filtering using a two-dimensional image as a guide, to derive dense clustering results.

[0266] In this case, filtering is a two-dimensional process that propagates the colors of neighboring pixels on a two-dimensional plane (i.e., on the same CT image). For example, the following formula (1) is used to propagate the colors of neighboring pixels x j to processing pixel x i We derive W i,j is a weighting coefficient, which is derived as shown in the following equation (2).

[0267] TIFF0007658369000001.tif15107...(1) TIFF0007658369000002.tif22151...(2)

[0268] Furthermore, for example, when applying the method described in the first embodiment, clustering is performed on sparsely sampled pixels selected from the entire CT image 651, and the obtained sparse information (learning model coefficients, clustering results, etc.) is complemented by filtering using the two-dimensional image as a guide to derive dense clustering results.

[0269] In this case, filtering is a two-dimensional process that propagates the colors of neighboring pixels on a two-dimensional plane (i.e., on the same CT image). For example, by using the above formula (1), the surrounding pixels x j to processing pixel x i We derive W i,jis a weighting coefficient, which is derived as shown in equation (2) above.

[0270] 29B, a CT image 651 (CT Slice) may be a local region, a CT volume 653 (CT Volume) that is a three-dimensional region made up of a plurality of CT images 651 may be a wide region (global region), and clustering may be performed by applying the above-described present technology. That is, in this case, the wide region is a collection of two-dimensional planes (three-dimensional region), the local region is a two-dimensional plane, and clustering of the CT volume is performed collectively.

[0271] For example, when applying the method described in the third embodiment, wide-area clustering (pre-learning) is performed on sparse wide-area sampled pixels selected from the CT volume 653 (all CT images 651), and the obtained sparse information (pre-learned model coefficients, clustering results, etc.) is used as initial values ​​to perform local clustering (additional learning) on ​​dense local sampled pixels in each CT image 651, and dense clustering results are derived.

[0272] Furthermore, for example, when applying the method described in the second embodiment, wide-area clustering (pre-learning) is performed on sparse wide-area sampled pixels selected from the CT volume 653 (all CT images 651), and the obtained sparse information (model coefficients and clustering results of the pre-learning, etc.) is used as initial values ​​to perform local clustering (additional learning) on ​​sparse local sampled pixels in each CT image 651, and the obtained sparse information (model coefficients and clustering results of the additional learning, etc.) is complemented by filtering using a two-dimensional image as a guide, to derive dense clustering results.

[0273] In this case, filtering is a three-dimensional process that propagates the colors of adjacent pixels in a three-dimensional space. In other words, in this case, not only the colors of adjacent pixels on the same CT image are propagated, but also the colors of adjacent pixels on adjacent CT images can be propagated. For example, by using the above formula (1), the surrounding pixels xj to processing pixel x i In this case, the weighting factor W i,j is derived as shown in the following equation (3).

[0274] TIFF0007658369000003.tif20150...(3)

[0275] Furthermore, for example, when applying the method described in the first embodiment, clustering is performed on sparsely sampled pixels selected from the CT volume 653 (all CT images 651), and the obtained sparse information (learning model coefficients, clustering results, etc.) is complemented by filtering using a two-dimensional image as a guide to derive dense clustering results.

[0276] In this case, the filtering is performed by the above-mentioned three-dimensional processing. For example, by using the above-mentioned formula (1), the surrounding pixels x j to processing pixel x i We derive W i,j is a weighting coefficient, which is derived as shown in equation (3) above.

[0277] 29C, the CT volume 653 may be a global region, and the voxels 654, which are three-dimensional regions of a predetermined size obtained by dividing the CT volume 653, may be local regions, and the clustering may be performed by applying the present technology described above. In other words, in this case, both the global region and the local region are three-dimensional regions, and the clustering of the CT volume is performed collectively.

[0278] For example, when applying the method described in the third embodiment, wide-area clustering (pre-learning) is performed on sparse wide-area sampled pixels selected from the CT volume 653 (all CT images 651), and the obtained sparse information (pre-learned model coefficients, clustering results, etc.) is used as initial values ​​to perform local clustering (additional learning) on ​​dense local sampled pixels in each voxel 654, and a dense clustering result is derived.

[0279] Furthermore, for example, when applying the method described in the second embodiment, wide-area clustering (pre-learning) is performed on sparse wide-area sampled pixels selected from the CT volume 653 (all CT images 651), and the obtained sparse information (pre-learned model coefficients, clustering results, etc.) is used as initial values ​​to perform local clustering (additional learning) on ​​sparse local sampled pixels in each voxel 654. The obtained sparse information (additional learning model coefficients, clustering results, etc.) is complemented by filtering using 3D data as a guide, and dense clustering results are derived.

[0280] In this case, filtering is a three-dimensional process that propagates the color of adjacent pixels in three-dimensional space. For example, by using the above formula (1), the color of a neighboring pixel x j to processing pixel x i In this case, the weighting factor W i,j is derived as shown in equation (3) above.

[0281] Furthermore, for example, when applying the method described in the first embodiment, clustering is performed on sparsely sampled pixels selected from the CT volume 653 (all CT images 651), and the obtained sparse information (learning model coefficients, clustering results, etc.) is complemented by filtering using 3D data as a guide to derive dense clustering results.

[0282] In this case, the filtering is performed by the above-mentioned three-dimensional processing. For example, by using the above-mentioned formula (1), the surrounding pixels x j to processing pixel x i We derive W i,j is a weighting coefficient, which is derived as shown in equation (3) above.

[0283] In the case of CT images constituting a CT volume, the correlation between the image structures of the images is generally high, so that even in the filtering of three-dimensional processing, more accurate clustering results can be obtained, as in the case of two-dimensional processing. Therefore, even when applying this technology to the medical equipment described above, it is possible to suppress an increase in processing time while suppressing a decrease in the robustness of image clustering.

[0284] <Image processing device> An example of the main configuration of an image processing device in this case is shown in Fig. 30. The image processing device 700 shown in Fig. 30 is a device that performs clustering of CT images (CT volumes), receives a captured image 710 which is a CT image (CT volume), performs clustering on the captured image 710, and outputs a clustered CT image 720 as the clustering result.

[0285] Note that Fig. 30 shows the main processing units, data flows, etc., and is not limited to all that is shown in Fig. 30. In other words, in this image processing device 700, there may be processing units that are not shown as blocks in Fig. 30, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 30.

[0286] As shown in Fig. 30, the image processing device 700 has a clustering unit 711 and an analysis unit 712. The clustering unit 711 performs clustering on the captured image 710 to derive a dense clustering result. The above-mentioned image processing device can be applied to this clustering unit 711. That is, the clustering unit 711 has a configuration similar to any one of the above-mentioned image processing devices, and derives a clustering result from the captured image 710 by performing a similar process (clustering). The clustering unit 711 supplies the clustering result to the analysis unit 712.

[0287] The analysis unit 712 performs processing related to image analysis based on the clustering result. For example, the analysis unit 712 acquires the clustering result supplied from the clustering unit 711. The analysis unit 712 also acquires the captured image 710. The analysis unit 712 analyzes and images the structure of the human body, which is the subject in the captured image 710, based on the clustering result. The analysis unit 712 outputs the generated CT image 720 as the analysis result. In this way, the image processing device 700 can generate the CT image 720 with reduced robustness at a higher speed.

[0288] <Flow of clustering process> An example of the flow of the clustering process in this case will be described with reference to the flowchart in Fig. 31. When the clustering process starts, the clustering section 711 acquires the captured image 710 in step S701.

[0289] In step S702, the clustering unit 711 performs clustering processing to obtain a dense clustering result. The above-mentioned clustering processing can be applied to this clustering processing. That is, the clustering unit 711 performs clustering processing in a similar flow to any one of the above-mentioned flowcharts to derive a dense clustering result.

[0290] In step S703, the analysis unit 712 analyzes the image based on the clustering result obtained in step S702.

[0291] In step S704, the analysis unit 712 outputs the CT image 720 as the analysis result obtained by the process of step S703. When the process of step S704 ends, the clustering process ends.

[0292] By executing each process as described above, the image processing device 700 can obtain a more accurate clustering result. Therefore, the image processing device 700 can generate the CT image 720 with reduced robustness at a higher speed.

[0293] <6. Notes> <Computer> The above-mentioned series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed in a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, etc., capable of executing various functions by installing various programs.

[0294] FIG. 32 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes by a program.

[0295] In a computer 900 shown in FIG. 32, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0296] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.

[0297] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 912 includes, for example, a display, a speaker, an output terminal, etc. The storage unit 913 includes, for example, a hard disk, a RAM disk, a non-volatile memory, etc. The communication unit 914 includes, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0298] In the computer configured as above, the series of processes described above are performed by the CPU 901, for example, loading a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing the program. The RAM 903 also stores data and the like necessary for the CPU 901 to execute various processes as appropriate.

[0299] The program executed by the computer can be applied by recording it on a removable medium 921 such as a package medium, for example. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by mounting the removable medium 921 on the drive 915.

[0300] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0301] In addition, this program can also be installed in advance in the ROM 902 or the storage unit 913.

[0302] <Target of application of this technology> Furthermore, although an image processing device that performs image clustering has been described above as an application example of the present technology, the present technology can be applied to any configuration.

[0303] For example, the present technology can be applied to various electronic devices, such as transmitters and receivers (e.g., television sets and mobile phones) in satellite broadcasting, wired broadcasting such as cable TV, distribution over the Internet, and distribution to terminals via cellular communication, or devices (e.g., hard disk recorders and cameras) that record images on media such as optical disks, magnetic disks, and flash memories, and play images from these storage media.

[0304] Furthermore, for example, the present technology can also be implemented as part of an apparatus, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module (e.g., a video module) using multiple processors, a unit (e.g., a video unit) using multiple modules, or a set in which other functions are added to a unit (e.g., a video set).

[0305] In addition, for example, the present technology can also be applied to a network system composed of multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and cooperate with each other to perform processing via a network. For example, the present technology may be implemented in a cloud service that provides services related to images (video images) to any terminal such as a computer, an AV (Audio Visual) device, a portable information processing terminal, or an IoT (Internet of Things) device.

[0306] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same case. Therefore, multiple devices housed in separate cases and connected via a network, and a single device in which multiple modules are housed in a single case, are both systems.

[0307] <Fields and applications where this technology can be applied> Systems, devices, processing units, etc. to which the present technology is applied can be used in any field, such as transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, and nature monitoring. In addition, the applications are also arbitrary.

[0308] <Other> The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0309] For example, the configuration described above as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or processing unit). Furthermore, as long as the configuration and operation of the system as a whole are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0310] Also, for example, the above-mentioned program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and can obtain the necessary information.

[0311] Also, for example, each step of one flowchart may be executed by one device, or may be shared and executed by multiple devices. Furthermore, when one step includes multiple processes, the multiple processes may be executed by one device, or may be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as one step.

[0312] Also, for example, the processing of the steps of a program executed by a computer may be executed in chronological order according to the order described in this specification, or may be executed in parallel, or individually at a required timing such as when a call is made. In other words, the processing of each step may be executed in an order different from the order described above, so long as no contradiction occurs. Furthermore, the processing of the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0313] Also, for example, a plurality of techniques related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any of a plurality of the present technologies can be implemented in combination. For example, a part or all of the present technology described in any of the embodiments can be implemented in combination with a part or all of the present technology described in other embodiments. Also, a part or all of any of the above-mentioned present technologies can be implemented in combination with other technologies not described above.

[0314] The present technology can also be configured as follows. (1) a clustering unit that performs clustering of sparse pixels included in an image; a complementation processing unit that complements sparse information obtained by the clustering by the clustering unit through image filtering using an image signal as a guide, thereby deriving a dense clustering result; An image processing device comprising: (2) The sparse information is a model coefficient or a clustering result obtained in the clustering. An image processing device as described in (1). (3) further comprising a sampling pixel selection unit for selecting sparse sampling pixels from the image; The clustering unit performs the clustering on the sparsely sampled pixels selected by the sampled pixel selection unit. An image processing device according to (1) or (2). (4) The sampling pixel selection unit selects the sampling pixels from a portion of the image included in the processing target area based on information about the processing target area. An image processing device as described in (3). (5) The image is a stitched image formed by pasting together a plurality of images; The sampling pixel selection unit selects the sampling pixels based on stitching information, which is information about overlaps between the multiple images in the stitched image. An image processing device according to (3) or (4). (6) The sampling pixel selection unit selects the sampling pixels from the flat region of the image based on information about the flat region. An image processing device according to any one of (3) to (5). (7) The clustering unit performs, as the clustering, local clustering, which is clustering of sparse pixels included in a local region of the image, using sparse information obtained by wide-area clustering, which is clustering of sparse pixels included in a wide area of ​​the image; The complementation processing unit complements sparse information obtained by the local clustering with the image filtering to derive a dense clustering result for the local region. An image processing device according to any one of (1) to (6). (8) The sparse information obtained by the wide-area clustering is a model coefficient or a clustering result. An image processing device according to (7). (9) The clustering unit further performs the local clustering on the local region of the processing target by using sparse information obtained in the local clustering on the local region of the previous processing target. An image processing device according to (7) or (8). (10) A sampling pixel selection unit that selects sparse sampling pixels from the local region, The clustering unit performs the local clustering on the sparsely sampled pixels selected by the sampled pixel selection unit. An image processing device according to any one of (7) to (9). (11) The sampling pixel selection unit selects the sampling pixels from pixels in the local region excluding pixels that have been subjected to the wide area clustering. The image processing device according to (10). (12) A wide-area clustering unit that performs the wide-area clustering, The clustering unit performs the local clustering using information obtained in the wide-area clustering performed by the wide-area clustering unit. An image processing device according to any one of (7) to (11). (13) Clustering sparse pixels in the image is performed. The sparse information obtained by the clustering is complemented by image filtering guided by the image signal to derive a dense clustering result. Image processing methods.

[0315] (14) A clustering unit that performs local clustering, which is clustering of dense pixels included in a local region of the image, using information obtained in wide-area clustering, which is clustering of sparse pixels included in a wide area of ​​the image. An image processing device comprising: (15) Using information obtained in wide-area clustering, which is clustering of sparse pixels included in a wide area of ​​an image, local clustering, which is clustering of dense pixels included in a local area of ​​the image, is performed. Image processing methods. [Explanation of symbols]

[0316] 100 image processing device, 111 sampling pixel selection unit, 112 clustering unit, 113 complementation processing unit, 201 field area storage unit, 231 stitching information storage unit, 261 flat area storage unit, 300 image processing device, 311 pre-learning unit, 312 additional learning unit, 313 coefficient storage unit, 321 sampling pixel selection unit, 322 clustering unit, 351 sampling pixel storage unit, 400 image processing device, 411 sampling pixel selection unit, 412 clustering unit, 413 optimization unit, 500 image processing device, 511 clustering unit, 512 vegetation area determination unit, 700 image processing device, 711 clustering unit, 712 analysis unit, 900 computer

Claims

1. A clustering unit that performs clustering of sparse pixels included in an image; a complementation processing unit that complements sparse information obtained by the clustering by the clustering unit through image filtering using an image signal as a guide, thereby deriving a dense clustering result; An image processing device comprising:

2. The sparse information is a model coefficient or a clustering result obtained in the clustering. The image processing device according to claim 1 .

3. A sampling pixel selection unit that selects sparse sampling pixels from the image, The clustering unit performs the clustering on the sparsely sampled pixels selected by the sampled pixel selection unit. The image processing device according to claim 1 .

4. The sampling pixel selection unit selects the sampling pixels from a portion of the image included in the processing target area based on information about the processing target area. The image processing device according to claim 3 .

5. The image is a stitched image formed by pasting together a plurality of images; The sampling pixel selection unit selects the sampling pixels based on stitching information, which is information about overlaps between the plurality of images in the stitched image. The image processing device according to claim 3 .

6. The sampling pixel selection unit selects the sampling pixels from the flat region of the image based on information about the flat region. The image processing device according to claim 3 .

7. the clustering unit performs, as the clustering, local clustering, which is clustering of sparse pixels included in a local region of the image, using sparse information obtained by wide-area clustering, which is clustering of sparse pixels included in a wide area of ​​the image; The complementation processing unit complements sparse information obtained by the local clustering with the image filtering to derive a dense clustering result for the local region. The image processing device according to claim 1 .

8. The sparse information obtained by the global clustering is the model coefficient or the clustering result. The image processing device according to claim 7.

9. The clustering unit further performs the local clustering on the local region to be processed by using sparse information obtained in the local clustering on the previous local region to be processed. The image processing device according to claim 7.

10. A sampling pixel selection unit that selects sparse sampling pixels from the local region, The clustering unit performs the local clustering on the sparsely sampled pixels selected by the sampled pixel selection unit. The image processing device according to claim 7.

11. The sampling pixel selection unit selects the sampling pixels from pixels in the local region excluding the pixels that have been subjected to the wide area clustering. The image processing device according to claim 10.

12. a wide-area clustering unit that performs the wide-area clustering, The clustering unit performs the local clustering using information obtained in the wide-area clustering performed by the wide-area clustering unit. The image processing device according to claim 7.

13. Clustering of sparse pixels in the image is performed, The sparse information obtained by the clustering is complemented by image filtering guided by the image signal to derive a dense clustering result. Image processing methods.

14. A clustering unit that performs local clustering, which is clustering of dense pixels included in a local region of an image, using information obtained in wide-area clustering, which is clustering of sparse pixels included in a wide area of ​​the image. An image processing device comprising:

15. Using information obtained in wide-area clustering, which is clustering of sparse pixels contained in a wide area of ​​an image, local clustering, which is clustering of dense pixels contained in a local area of ​​the image, is performed. Image processing methods.

Images