Image processing device, image processing method, and program

Abstract

The invention discloses an image processing apparatus, an image processing method, and a program. The image processing apparatus includes at least one processor. The processor performs a segmentation process for segmenting the image into a plurality of partial images, and segmenting the image at an edge portion of each of the partial images so as to have an overlapping portion overlapping with another adjacent partial image; performing calculation processing on each of the plurality of local images, the calculation processing calculating a numerical value used for category classification for each pixel, i.e., a numerical value corresponding to a probability that the pixel belongs to a specific category; an integration process is performed in which numerical values calculated for each of the plurality of partial images are integrated into the size of the image before the segmentation process.

Description

Technical Field

[0001] The technology of the invention relates to an image processing device, an image processing method, and a program. Background Art

[0002] As a technology related to image processing, the following technology is known. In Japanese Patent Gazette No. 2020-16928, an image processing device is known, which comprises: a likelihood value calculation unit, which calculates a likelihood value representing the possibility that an area contained in an object image is an area representing a structure; a separation degree calculation unit, which calculates a separation degree showing the ease of separation between an area representing a structure and an area other than the structure based on a statistic of the likelihood value; a conversion unit, which converts the object image when the separation degree is less than a first threshold value to generate a new object image as a calculation object of the likelihood value; and a synthesis unit, which generates a synthesized likelihood value by synthesizing the likelihood values ​​calculated for each object image.

[0003] Japanese Patent Gazette No. 2021-149473 describes an image processing device comprising: a segmentation unit, which divides a high-quality image and a low-quality image into multiple regions; a learning unit, which enables multiple learning models to learn low-quality images and high-quality image pairs for each segmented region; and a restoration unit, which uses the same method as the segmentation unit to segment the low-quality image of the restoration object, and for each segmented region, restores the low-quality image of the restoration object based on the high-quality image input to the corresponding learning model and output from each learning model.

[0004] Japanese Patent Gazette No. 2020-160997 describes an image processing device comprising: an acquisition unit, which acquires a learning image; a repair image generation unit, which generates a plurality of learning repair images based on the learning image; a label assignment unit, which assigns labels to the plurality of learning repair images according to preset label assignment conditions; and a suspected image generation unit, which takes the learning repair image and the label as input and learns to generate a neural network that restores the learning repair image. Summary of the invention

[0005] Technical issues to be solved by the invention

[0006] Partitioning, which is known as one of the image recognition techniques, includes, for example, the processing of classifying all images that constitute the image. Partitioning can be achieved using CNN (Convolutional Neural Network). There are cases where the size of the image input to CNN is too large and computing resources such as memory are insufficient. Therefore, a method of dividing the image into multiple partial images, partitioning each partial image, and integrating the results can be considered.

[0007] On the other hand, in the CNN-based partitioning, there is a tendency that the classification accuracy near the edge of the image is lower than that in the central part of the image. The reason is that the pixels around the object pixel are also considered for category classification. In order to avoid this problem, it is considered to perform image segmentation in a manner that the local images each have a portion (hereinafter referred to as the overlapping portion) that overlaps with other adjacent local images at the edge, and for the overlapping portion of each local image, the classification label (for example, "0" or "1") obtained for the multiple local images forming the overlapping portion is used to perform category classification. That is, the classification labels obtained for the multiple local images are integrated into the original image size. According to this method, for the overlapping portion, only the degree of selective use of any one of the classification labels obtained for the multiple local images forming the overlapping portion can be processed, and the effect of improving the accuracy of category classification near the edge of the image is limited. As a result, in the label map obtained by integrating the results of the partitioning of each local image, a low-precision area appears near the boundary between the local images, resulting in inaccurate and unnatural results. That is, according to the method of integrating the classification labels obtained for a plurality of partial images into the original image size, it is sometimes difficult to obtain an accurate and natural partitioning result due to the lack of information in the classification labels.

[0008] The technology of the invention has been made in view of the above-mentioned situation, and an object of the invention is to improve the accuracy of category classification in image processing for category classification including processing of a plurality of partial images obtained by dividing an image.

[0009] Means for solving technical problems

[0010] The image processing device involved in the technology of the invention has at least one processor. The processor performs segmentation processing, which is processing of segmenting an image into a plurality of partial images, and the image is segmented in a manner that an overlapping portion overlaps with other adjacent partial images at the edge of each partial image; performs calculation processing on each of the plurality of partial images, wherein the calculation processing calculates a numerical value used for category classification for each pixel, that is, a numerical value corresponding to the probability that the pixel belongs to a specific category; and performs integration processing of integrating the numerical values ​​calculated for each of the plurality of partial images into the size of the image before the segmentation processing.

[0011] The above-mentioned numerical value used for category classification may also be the probability or logit number that the pixel belongs to a specific category.

[0012] The processor may also perform arithmetic processing using the numerical values ​​in the overlapping parts of the plurality of partial images forming a common overlapping part as the operation object in the integrated processing. The processor may also perform a processing of calculating a statistical value of the numerical value of the operation object. The processor may also perform a calculation corresponding to the pixel position on the numerical value of the operation object. The processor may also perform a processing of calculating a weighted average using a weight coefficient corresponding to the pixel position on the numerical value of the operation object.

[0013] The processor may classify the image into a category based on a combined value obtained by combining the numerical values ​​in the combining process. The processor may classify the image into a category using an Argmax function or a threshold value based on the combined value.

[0014] In the image processing method involved in the technology of the invention, at least one processor of the image processing device executes the following steps: performing segmentation processing, wherein the segmentation processing is a processing of dividing an image into multiple local images, and dividing the image in a manner of having overlapping portions overlapping with other adjacent local images at the edge portions of each local image; performing calculation processing on the multiple local images respectively, wherein the calculation processing calculates a numerical value used for category classification for each pixel, that is, a numerical value indicating the probability that the pixel belongs to a specific category; performing calculation processing on the multiple local images respectively, wherein the calculation processing calculates a numerical value used for category classification for each pixel, that is, a numerical value corresponding to the probability that the pixel belongs to a specific category; and performing integration processing to integrate the numerical values ​​calculated for the multiple local images into the size of the image before the segmentation processing.

[0015] The program involved in the technology of the invention is used to cause at least one processor of an image processing device to execute the following steps: performing a segmentation process, wherein the segmentation process is a process of dividing an image into multiple local images, and dividing the image in a manner that has overlapping portions that overlap with other adjacent local images at the edge portions of each local image; performing calculation processing on the multiple local images respectively, wherein the calculation processing calculates a numerical value used for category classification for each pixel, that is, a numerical value indicating the probability that the pixel belongs to a specific category; and performing integration processing to integrate the numerical values ​​calculated for the multiple local images into the size of the image before the segmentation process.

[0016] Effects of the Invention

[0017] According to the technology of the present invention, it is possible to improve the accuracy of category classification in image processing for category classification including processing of a plurality of partial images obtained by dividing an image. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a diagram showing an example of the hardware configuration of an image processing device according to an embodiment of the inventive technique.

[0019] Figure 2 This is a functional block diagram showing an example of a functional configuration of an image processing device according to an embodiment of the inventive technique.

[0020] Figure 3A This is a diagram showing an example of a method of segmentation processing according to an embodiment of the inventive technique.

[0021] Figure 3B This is a diagram showing an example of a method of segmentation processing according to an embodiment of the inventive technique.

[0022] Figure 4 This is a diagram showing an example of a method of segmentation processing according to an embodiment of the inventive technique.

[0023] Figure 5 This is a diagram showing the concept of integrated processing according to an embodiment of the inventive technique.

[0024] Figure 6 This is a flowchart showing an example of the flow of image processing according to an embodiment of the inventive technique. DETAILED DESCRIPTION

[0025] Hereinafter, an example of an embodiment of the technology of the invention will be described with reference to the drawings. In addition, in each of the drawings, the same reference numerals are given to the same or equivalent components and parts, and repeated descriptions are omitted.

[0026] Figure 1 1 is a diagram showing an example of the hardware structure of an image processing device 10 involved in an embodiment of the technology of the invention. The image processing device 10 has a function of classifying objects contained in an image to be processed (hereinafter referred to as a processing object image) by category using partitions. The processing object image is not particularly limited, and for example, it may be a radiographic image obtained in a non-destructive inspection of an industrial product. The radiographic image may, for example, include an image of a defect (scar) generated inside an object (such as a metal product). The image processing device 10 may also detect or recognize the defect (scar) contained in the radiographic image by, for example, category classification.

[0027] The image processing device 10 includes a CPU (Central Processing Unit) 101, a RAM (Random Access Memory) 102, a non-volatile memory 103, an input device 104, a display 105, and a network interface 106. These hardware are connected to a bus 107. The display 105 is, for example, a liquid crystal display or an LED (Light Emitting Diode) display. The input device 104 includes, for example, a keyboard and a mouse, and may also include a proximity input device such as a touch panel display, a voice input device such as a microphone, etc. The network interface 106 is an interface for connecting the image processing device 10 to a network.

[0028] The nonvolatile memory 103 is a nonvolatile storage medium such as a hard disk and a flash memory. The nonvolatile memory 103 stores an image processing program 110 and a category classification model 120. The RAM 102 is a working memory used by the CPU 101 to execute processing. The CPU 101 loads the image processing program 110 stored in the nonvolatile memory 103 to the RAM 102 and executes processing according to the image processing program 110. The CPU 101 is an example of a "processor" of the invented technology.

[0029] The category classification model 120 is a CNN-based mathematical model that is constructed in a way that predicts classification labels on a pixel-by-pixel basis for the processing object image. The category classification model 120 can also apply the well-known encoder-decoder model (ED-CNN: Encoder-Decoder Convolutional Neural Network), for example. The encoder-decoder model is a model composed of an encoder that extracts features from an image through a convolutional layer and a decoder that outputs a probability map based on the extracted features. The probability map is the result of deriving the probability that the pixel belongs to a certain category for each pixel in the processing object image. The category classification model 120 is constructed by machine learning using multiple images with correct labels as supervisory data.

[0030] Figure 2 1 is a functional block diagram showing an example of the functional configuration of the image processing device 10. The image processing device 10 functions as an image acquisition unit 11, a segmentation unit 12, a calculation unit 13, an integration unit 14, and a classification unit 15 when the CPU 101 executes an image processing program 110.

[0031] The image acquisition unit 11 acquires a processing target image. The processing target image may be downloaded from an image server (not shown) and stored in the nonvolatile memory 103, for example.

[0032] The segmentation processing unit 12 performs segmentation processing to segment the processing target image acquired by the image acquisition unit 11 into a plurality of partial images. Figure 3A FIG. 1 is a diagram showing an example of a segmentation process. Figure 3A As shown, the segmentation processing unit 12 divides the processing object image 200 into a plurality of local images 210. The shape of each local image 210 may be, for example, a square or a rectangle. When processing (calculation processing described later) using the category classification model 120 is performed for each local image 210, the size of each local image 210 is set to a size that does not require insufficient computing resources including the CPU 101 and the RAM 102. The size of each local image 210 may be, for example, 256 pixels × 256 pixels. For example, in the case where the processing object image 200 is a radiographic image including an image of a defect (scar) formed inside an object (such as a metal product), the size of the defect (scar) that is the object of category classification is, for example, about 1 to 10 pixels. That is, each local image 210 may include images of a plurality of defects (scars).

[0033] The segmentation processing unit 12 segments the processing target image 200 so that each of the partial images 210 has an overlapping portion 211 that overlaps with another adjacent partial image 210 at the edge portion. Figure 3B 2 shows a partial image 210E and the adjacent partial images 210A, 210B, 210C, 210D, 210F, 210G, 210H, and 210I. For example, the upper edge of the partial image 210E has an overlapping portion 211T that overlaps with the partial image 210B. The right edge of the partial image 210E has an overlapping portion 211R that overlaps with the partial image 210F. The lower edge of the partial image 210E has an overlapping portion 211B that overlaps with the partial image 210H. The left edge of the partial image 210E has an overlapping portion 211L that overlaps with the partial image 210D. The upper right corner of the partial image 210E has an overlapping portion 211TR that overlaps with the partial images 210B, 210C, and 210F. The lower right corner of the partial image 210E has an overlapping portion 211BR overlapping with the partial images 210F, 210I, and 210H. The lower left corner of the partial image 210E has an overlapping portion 211BL overlapping with the partial images 210H, 210G, and 210D. The upper left corner of the partial image 210E has an overlapping portion 211TL overlapping with the partial images 210D, 210A, and 210B.

[0034] In addition, Figure 3A and Figure 3B The example in FIG. 1 shows a case where a plurality of partial images 210 are arranged in a matrix to perform segmentation processing. Figure 4As shown, the segmentation process may be performed so that a plurality of partial images 210 form a honeycomb arrangement. That is, the distance between the centers of the partial images 210 may be constant and the partial images 210 may be arranged so as to fill the space.

[0035] The calculation processing unit 13 performs calculation processing on multiple local images 210 respectively. The calculation processing uses the category classification model 120 to calculate a numerical value for category classification for each pixel, that is, a numerical value corresponding to the probability that the pixel belongs to a specific category. The above-mentioned "numerical value" is a numerical value used to estimate the classification label that should be assigned to the pixel. For example, it can be the probability φ that the pixel belongs to a specific category, or it can be the logit f(φ) of the probability φ. The logit f(φ) of the probability φ is expressed by the following formula (1). Below, the case where the numerical value calculated by the calculation processing unit 13 is the probability φ that the pixel belongs to a specific category is explained as an example. That is, the calculation processing unit 13 uses the category classification model 120 to generate probability maps for multiple local images 210 respectively.

[0036] [Mathematical formula 1]

[0037]

[0038] The integration processing unit 14 performs integration processing of integrating the probabilities φ (probability maps) calculated for each pixel of the plurality of partial images 210 into the size of the processing target image 200 before the segmentation processing. Figure 5 : is a diagram showing the concept of the integration processing performed by the integration processing unit 14. As far as the integration processing unit 14 is concerned, the portion other than the overlapping portion 211 of the local image 210 directly outputs the probability φ calculated for each pixel of the plurality of local images 210 as the integrated value of the pixel. On the other hand, the integration processing unit 14 performs an operation processing on the probability φ in the overlapping portion 211 of each of the plurality of local images 210 forming the common overlapping portion 211 as the operation object. Specifically, the integration processing unit 14 outputs a value f(φ6, φ8) as the integrated value of one pixel in the overlapping portion 211, and outputs a value f(φ7, φ9) as the integrated value of another pixel in the overlapping portion 211, the values ​​f(φ6, φ7) being obtained by Figure 5 The values ​​f(φ7, φ9) are obtained by performing calculations on the probabilities φ6 and φ8 in the overlapped portions 211 of the two partial images 210 shown in FIG. 1 and 2 as operation objects, and the values ​​f(φ7, φ9) are obtained by performing calculations on the probabilities φ7 and φ9 as operation objects. In addition, the partial images 210 forming the common overlapped portion 211 have four corners (see FIG. 1 ). Figure 3B ), and perform calculation processing on the four probabilities φ calculated for the four partial images 210 respectively as calculation objects.

[0039] The integration processing unit 14 may also perform a process of calculating a statistical value for the probability φ of the operation object in the above-mentioned operation processing performed on the overlapping unit 211. The statistical value may be, for example, an average value, a median value, a maximum value, or a minimum value. For example, when the statistical value is an average value, Figure 5 In the example shown, the integration processing unit 14 outputs the average value of the probability φ6 and the probability φ8 as the integrated value of one pixel in the overlapping portion 211, and outputs the average value of the probability φ7 and the probability φ9 as the integrated value of another pixel in the overlapping portion 211. On the other hand, when the statistical value is the maximum value, Figure 5 In the example shown, the integration processing unit 14 outputs the maximum value between probability φ6 and probability φ8 as the integrated value of a pixel in the overlapping portion 211, and outputs the maximum value between probability φ7 and probability φ9 as the integrated value of another pixel in the overlapping portion 211.

[0040] The integration processing unit 14 may also perform operations corresponding to the pixel positions on the probability φ of the operation object in the above-mentioned operation processing performed on the overlapping unit 211. Specifically, the integration processing unit 14 may also perform weighted average processing using weight coefficients corresponding to the pixel positions to calculate the probability φ of the operation object. In this case, it is preferable to use a weight coefficient that is smaller as the pixel position is closer to the edge, so that the contribution rate of the probability φ calculated for the pixels close to the edge of each local image 210 is relatively reduced. That is, in Figure 5 In the example shown, when the weight coefficients added to the probabilities φ6, φ7, φ8, and φ9 in the overlapping portion 211 are respectively set to w6, w7, w8, and w9, the following equations (2) to (5) hold. The weighted averages X(φ6, φ8) and X(φ7, φ9) are expressed by the following equations (6) and (7).

[0041] w6>w7…(2)

[0042] w8<w9…(3)

[0043] w6+w8=1…(4)

[0044] w7+w9=1…(5)

[0045] X(φ6,φ8)=w6·φ6+w8·φ8…(6)

[0046] X(φ7,φ9)=w7·φ9+w7·φ9…(7)

[0047] It is preferable to set the weight coefficient in such a way that the difference between adjacent pixels of the integrated value obtained as a result of the integration process is within a specified range. In other words, it is preferable to set the weight coefficient in such a way that the integrated value does not become a value that is significantly different between adjacent pixels. Such a weight coefficient can be expressed, for example, by the following equations (8) to (13).

[0048] [Mathematical formula 2]

[0049] f(P)=Σw p value(p)···(8)

[0050] st∑w p =1···(9)

[0051]

[0052] In formula (8), f(P) is a processing function for the integration processing of a set P of a plurality of partial images p forming the common overlapping portion 211. p is the weight coefficient assigned to a certain pixel of the local image p. value(p) is the processing function for calculating the probability φ of the local image p. Formula (9) indicates that the sum of the weight coefficients is 1. In formula (10), size is the size of the local image p. pos(p) is a function that outputs the pixel position of the local image p in a rectangular coordinate system. Formula (10) indicates that "0" is assigned as the weight coefficient to the pixels located at the outermost periphery of the local image p. Formula (11) indicates that the weight coefficient changes continuously according to the distance from the edge of the pixel position.

[0053] Formula (12) indicates that the weight coefficient assigned to a certain pixel changes linearly according to the distance from the edge of the pixel. Formula (13) indicates that the weight coefficient assigned to a certain pixel changes exponentially according to the distance from the edge of the pixel.

[0054] [Mathematical formula 3]

[0055]

[0056] [Formula 4]

[0057]

[0058] In addition, the operation corresponding to the pixel position is not limited to the weighted average using the weight coefficient corresponding to the pixel position. The integration processing unit 14 may output the probability φ calculated for the pixel farthest from the edge among the probabilities φ of the operation objects as the integrated value. Figure 5In the example shown, the integration processing unit 14 may output the probability φ6 calculated for the pixel far from the edge from among the probabilities φ6 and φ8 as the integrated value of a pixel in the overlapping portion 211, and output the probability φ7 and the probability φ9 calculated for the pixel far from the edge from among the probabilities φ9 as the integrated value of another pixel in the overlapping portion 211.

[0059] The classification processing unit 15 classifies the processing object image 200 into categories for each pixel based on the integrated value of the probability φ obtained through the integrated processing. That is, the classification processing unit 15 estimates the category to which each pixel of the processing object image 200 belongs based on the integrated value of the probability φ, and assigns a classification label corresponding to the estimated category to the pixel. The classification processing unit 15 may also perform category classification using, for example, the Argmax function. In addition, the classification processing unit 15 may also perform category classification using, for example, a threshold value. Furthermore, the classification processing unit 15 may also be installed in an image processing device different from the image processing device 10. In this case, the image processing device 10 does not need to perform classification processing.

[0060] Figure 6 1 is a flowchart showing an example of the flow of image processing performed by the CPU 101 when the image processing program 110 is executed. The image processing program 110 is executed when the user instructs to start the processing by operating the input device 104, for example.

[0061] In step S1, the image acquisition unit 11 acquires the processing target image 200. In step S2, the segmentation processing unit 12 performs segmentation processing to segment the processing target image 200 acquired in step S1 into a plurality of partial images 210. The segmentation processing unit 12 segments the processing target image 200 in such a manner that the edge of each partial image 210 has an overlapping portion 211 that overlaps with another adjacent partial image 210.

[0062] In step S3, the calculation processing unit 13 performs calculation processing on each of the plurality of partial images 210, and the calculation processing uses the category classification model 120 to calculate, for each pixel, a numerical value used for category classification, that is, a numerical value corresponding to the probability that the pixel belongs to a specific category. The above-mentioned "numerical value" may be, for example, the probability φ that the pixel belongs to a specific category, or its logit number f(φ).

[0063] In step S4, the integration processing unit 14 integrates the probabilities φ (probability maps) calculated for the plurality of partial images 210 into the size of the processing target image 200 before the segmentation processing. The integration processing unit 14 outputs the probability φ calculated for each pixel of the plurality of partial images 210 as the integrated value of the pixel in the portion other than the overlapping portion 211 of the partial image 210. On the other hand, the integration processing unit 14 performs the calculation processing on the probability φ in the overlapping portion 211 of each of the plurality of partial images 210 forming the common overlapping portion 211.

[0064] In step S5, the classification processing unit 15 classifies the processing target image into categories for each pixel based on the integrated value obtained by the integration processing in step S4. That is, the classification processing unit 15 estimates the category to which each pixel of the processing target image 200 belongs based on the integrated value of the probability φ, and assigns a classification label corresponding to the estimated category to the pixel. The classification processing unit 15 may also perform category classification using, for example, the Argmax function. In addition, the classification processing unit 15 may also perform category classification using, for example, a threshold value.

[0065] As described above, the image processing device 10 according to the embodiment of the inventive technology performs segmentation processing, which is processing for segmenting the processing target image 200 into a plurality of partial images 210, and segmenting the processing target image 200 in a manner that the edge portion of each partial image 210 has an overlapping portion 211 that overlaps with other adjacent partial images. The image processing device 10 performs calculation processing on each of the plurality of partial images 210, and the calculation processing calculates a numerical value used for class classification, that is, a numerical value corresponding to the probability that the pixel belongs to a specific class, for each pixel. The image processing device 10 performs integration processing to integrate the numerical values ​​calculated for each of the plurality of partial images 210 into the size of the processing target image 200 before the segmentation processing. In the integration processing, the image processing device 10 performs calculation processing on the numerical values ​​in the overlapping portions of each of the plurality of partial images 210 that form a common overlapping portion as calculation objects. The image processing device 10 classifies the processing target image 200 into a class based on the integrated value obtained by integrating the numerical values ​​in the integration processing.

[0066] Partitioning, which is known as one of the image recognition technologies, can be implemented using CNN. When the size of the image input to CNN is too large, sometimes computing resources such as memory are insufficient. Therefore, a method is considered to divide the image into multiple local images, partition each local image, and integrate the results.

[0067] On the other hand, in the CNN-based partitioning, there is a tendency that the classification accuracy near the edge of the image is lower than the classification accuracy in the central part of the image. This is because the pixels around the target pixel are also considered for category classification. In order to avoid this problem, it is considered to perform image segmentation in a manner that each local image has an overlapping portion that overlaps with other adjacent local images at the edge, and for the overlapping portion of each local image, the classification label (for example, "0" or "1") obtained for the multiple local images forming the overlapping portion is used to perform category classification. That is, the classification labels obtained for the multiple local images are integrated into the original image size. According to this method, for the overlapping portion, only the degree of selective use of any one of the classification labels obtained for the multiple local images forming the overlapping portion can be processed, and the effect of improving the accuracy of the category classification near the edge of the image is limited. As a result, in the label map obtained by integrating the results of the partitioning of each local image, a low-precision area appears near the boundary between the local images, resulting in inaccurate and unnatural results. That is, according to the method of integrating the classification labels obtained for the multiple local images into the original image size, it is sometimes difficult to obtain accurate and natural partitioning results due to the lack of information in the classification labels.

[0068] According to the image processing device 10 involved in the embodiment of the technology of the invention, the numerical values ​​such as probability φ calculated for multiple local images 210 are used as the object of integrated processing. As a result, compared with the case of integrating the classification labels, it is possible to perform advanced calculation processing in the integrated processing, and the accuracy of category classification in the overlapping part 211 can be improved. In addition, as described above, in the CNN-based partitioning, there is a tendency that the classification accuracy near the edge of the image is lower than the classification accuracy in the central part of the image. According to the image processing device 10, in the calculation processing in the integrated processing, since the calculation corresponding to the pixel position is performed, it is possible to deal with the problem of reduced accuracy of category classification near the edge of the image.

[0069] In each of the above-mentioned embodiments, for example, as a hardware structure of a processing unit (processing unit) that performs various processes, such as the image acquisition unit 11, the segmentation processing unit 12, the calculation processing unit 13, the integration processing unit 14, and the classification processing unit 15, various processors (processors) as shown below can be used. Among the above-mentioned various processors, as described above, in addition to general-purpose processors that execute software (programs) and function as various processing units, namely CPUs and GPUs, processors that can change the circuit structure after manufacturing, such as FPGAs, programmable logic devices (PLDs), ASICs (Application Specific Integrated Circuits), and other processors that have circuit structures specially designed to perform specific processes, namely dedicated circuits, etc.

[0070] A processing unit may be composed of one of these various processors, or a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs, a combination of a CPU and an FPGA). In addition, multiple processing units may be composed of one processor.

[0071] As an example of a processor constituting multiple processing units, first, there is a form such as computers such as clients and servers, which use a combination of one or more CPUs and software to constitute a processor and use the processor as multiple processing units. Secondly, there is a form such as a system on chip (System On Chip: SoC) that uses a processor that implements the functions of the entire system including multiple processing units using one IC (Integrated Circuit) chip. In this way, various processing units are constituted using one or more of the above-mentioned various processors as a hardware structure.

[0072] Furthermore, as the hardware configuration of these various processors, more specifically, a circuit (circuitry) formed by combining circuit elements such as semiconductor elements can be used.

[0073] In addition, in the above-mentioned embodiment, the image processing program 110 is described as being pre-stored (installed) in the non-volatile memory 103, but the present invention is not limited thereto. The image processing program 110 may also be provided in a manner recorded in a recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory. In addition, the image processing program 110 may also be provided in a manner of being downloaded from an external device via a network. That is, the program (i.e., program product) described in this embodiment may be provided by a recording medium or may be transmitted from an external computer.

[0074] Regarding the above-mentioned embodiment, the following supplementary notes are also disclosed.

[0075] (Note 1)

[0076] An image processing device comprises at least one processor, wherein:

[0077] The processor

[0078] Performing a segmentation process, wherein the segmentation process is a process of segmenting an image into a plurality of partial images, wherein the image is segmented in a manner such that an overlapping portion overlaps with another adjacent partial image at an edge portion of each of the partial images;

[0079] Performing calculation processing on the plurality of partial images respectively, wherein the calculation processing calculates, for each pixel, a numerical value used for category classification, that is, a numerical value corresponding to a probability that the pixel belongs to a specific category;

[0080] An integration process is performed to integrate the numerical values ​​respectively calculated for the plurality of partial images into a size of the image before the division process.

[0081] (Note 2)

[0082] An image processing device according to Supplement 1, wherein:

[0083] The value is the probability or logit of the pixel belonging to a specific category.

[0084] (Note 3)

[0085] The image processing device described in Supplement 1 or Supplement 2, wherein:

[0086] The processor performs, in the integration process, a calculation process using the numerical value in the overlapping portion of each of the plurality of partial images forming a common overlapping portion as a calculation target.

[0087] (Note 4)

[0088] An image processing device according to Supplement 3, wherein:

[0089] The processor performs processing for calculating a statistical value of the numerical value of the operation object.

[0090] (Note 5)

[0091] An image processing device according to Supplement 3, wherein:

[0092] The processor performs an operation corresponding to a pixel position on the numerical value of the operation object.

[0093] (Note 6)

[0094] An image processing device according to Supplement 5, wherein:

[0095] The processor performs a process of calculating a weighted average using a weight coefficient corresponding to a pixel position for the numerical value of the operation object.

[0096] (Note 7)

[0097] An image processing device according to any one of Supplementary Notes 1 to 6, wherein:

[0098] The processor classifies the image into the category based on an integrated value obtained by integrating the numerical values ​​in the integration process.

[0099] (Note 8)

[0100] An image processing device according to Supplement 7, wherein:

[0101] The processor performs the category classification based on the integrated value using an Argmax function or a threshold.

[0102] (Note 9)

[0103] An image processing method, wherein at least one processor of an image processing device performs the following steps:

[0104] Performing a segmentation process, wherein the segmentation process is a process of segmenting an image into a plurality of partial images, wherein the image is segmented in a manner such that an overlapping portion overlaps with another adjacent partial image at an edge portion of each of the partial images;

[0105] Performing calculation processing on the plurality of partial images respectively, wherein the calculation processing calculates, for each pixel, a numerical value used for category classification, that is, a numerical value indicating a probability that the pixel belongs to a specific category;

[0106] performing calculation processing on each of the plurality of partial images, the calculation processing calculating, for each pixel, a numerical value used for category classification, that is, a numerical value corresponding to a probability that the pixel belongs to a specific category; and

[0107] An integration process is performed to integrate the numerical values ​​respectively calculated for the plurality of partial images into a size of the image before the division process.

[0108] (Note 10)

[0109] A program, wherein the program is used to cause at least one processor of an image processing device to execute the following steps:

[0110] Performing a segmentation process, wherein the segmentation process is a process of segmenting an image into a plurality of partial images, wherein the image is segmented in a manner such that an overlapping portion overlaps with another adjacent partial image at an edge portion of each of the partial images;

[0111] performing calculation processing on each of the plurality of partial images, the calculation processing calculating, for each pixel, a numerical value used for category classification, that is, a numerical value indicating a probability that the pixel belongs to a specific category; and

[0112] An integration process is performed to integrate the numerical values ​​respectively calculated for the plurality of partial images into a size of the image before the division process.

[0113] In addition, the disclosure of Japanese Patent Application No. 2022-153069 filed on September 26, 2022 is incorporated herein by reference in its entirety. In addition, all documents, patent applications, and technical standards described in this specification are incorporated herein by reference to the same extent as if each document, patent application, and technical standard were specifically and individually described as being incorporated by reference.

Claims

1. An image processing device having at least one processor, characterized in that: The processor Performing a segmentation process, wherein the segmentation process is a process of segmenting an image into a plurality of partial images, wherein the image is segmented in a manner such that an overlapping portion overlaps with another adjacent partial image at an edge portion of each of the partial images; Performing calculation processing on the plurality of partial images respectively, wherein the calculation processing calculates, for each pixel, a numerical value used for category classification, that is, a numerical value corresponding to a probability that the pixel belongs to a specific category; as well as An integration process is performed to integrate the numerical values ​​respectively calculated for the plurality of partial images into a size of the image before the division process.

2. The image processing device according to claim 1, wherein: The value is the probability or logit of the pixel belonging to a specific category.

3. The image processing device according to claim 2, wherein: The processor performs, in the integration process, a calculation process using the numerical value in the overlapping portion of each of the plurality of partial images forming a common overlapping portion as a calculation target.

4. The image processing device according to claim 3, wherein: The processor performs processing for calculating a statistical value of the numerical value of the operation object.

5. The image processing device according to claim 3, wherein: The processor performs an operation corresponding to a pixel position on the numerical value of the operation object.

6. The image processing device according to claim 5, wherein: The processor performs a process of calculating a weighted average using a weight coefficient corresponding to a pixel position for the numerical value of the operation object.

7. The image processing device according to any one of claims 1 to 6, wherein: The processor classifies the image into the category based on an integrated value obtained by integrating the numerical values ​​in the integration process.

8. The image processing device according to claim 7, wherein: The processor performs the category classification based on the integrated value using an Argmax function or a threshold.

9. An image processing method, characterized in that: At least one processor of the image processing device executes the following steps: Performing a segmentation process, wherein the segmentation process is a process of segmenting an image into a plurality of partial images, wherein the image is segmented in a manner such that an overlapping portion overlaps with another adjacent partial image at an edge portion of each of the partial images; Performing calculation processing on the plurality of partial images respectively, wherein the calculation processing calculates, for each pixel, a numerical value used for category classification, that is, a numerical value indicating a probability that the pixel belongs to a specific category; Performing calculation processing on the plurality of partial images respectively, wherein the calculation processing calculates, for each pixel, a numerical value used for category classification, that is, a numerical value corresponding to a probability that the pixel belongs to a specific category; as well as An integration process is performed to integrate the numerical values ​​respectively calculated for the plurality of partial images into a size of the image before the division process.

10. A program, characterized in that It is used to enable at least one processor of the image processing device to execute the following steps: Performing a segmentation process, wherein the segmentation process is a process of segmenting an image into a plurality of partial images, wherein the image is segmented in a manner such that an overlapping portion overlaps with another adjacent partial image at an edge portion of each of the partial images; Performing calculation processing on the plurality of partial images respectively, wherein the calculation processing calculates, for each pixel, a numerical value used for category classification, that is, a numerical value indicating a probability that the pixel belongs to a specific category; as well as An integration process is performed to integrate the numerical values ​​respectively calculated for the plurality of partial images into a size of the image before the division process.

Images