Image classification device and image classification method

Abstract

This allows for a reduction in the number of parameters used to classify the objects being classified in an image. [Solution] The image classification device is configured to include: an image division unit 1 that divides an image containing the object to be classified into multiple parts and outputs divided images, which are the multiple divided images; a conversion unit 2 that converts each divided image output from the image division unit 1 into a one-dimensional vector having multiple vector components; a feature vector calculation unit 3 that repeatedly selects a set of two or more one-dimensional vectors from the multiple one-dimensional vectors converted by the conversion unit 2, and calculates a feature vector related to the selected set based on the multiple vector components of the two or more one-dimensional vectors included in the set; and a classification unit 4 that classifies the object to be classified based on the multiple feature vectors calculated by the feature vector calculation unit 3.

Description

[Technical Field]

[0001] This disclosure relates to an image classification device and an image classification method. [Background technology]

[0002] There is an image classification device that classifies the objects to be classified that are depicted in an image. As an example of such an image classification device, Non-Patent Document 1 discloses an image classification device that divides an image containing the object to be classified into multiple parts, calculates the feature quantities of each divided image, and classifies the object based on the feature quantities of the multiple divided images. The image classification device disclosed in Non-Patent Document 1 prepares numerous parameters (hereinafter referred to as "feature calculation parameters") to be used for the feature quantities of the divided images, and calculates the feature quantities of the divided images using these feature calculation parameters. Examples of feature calculation parameters include parameters related to weights and parameters related to bias. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Ilya Tolstikhin, et al., “MLP-Mixer: An all-MLP Architecture for Vision,” [online], May 4, 2021, arXiv, [Retrieved August 5, 2024], Internet<URL:https: / / doi.org / 10.48550 / arXiv.2105.01601> [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] The image classification device disclosed in Non-Patent Document 1 had the problem of requiring a large number of feature calculation parameters to calculate the feature quantities of segmented images. Therefore, the image classification device needed to implement hardware such as a large amount of memory to store the feature calculation parameters.

[0005] This disclosure was made to solve the above-mentioned problems and aims to provide an image classification device that can reduce the number of parameters used to classify the objects to be classified in an image. [Means for solving the problem]

[0006] The image classification device according to this disclosure includes: an image division unit that divides an image containing the object to be classified into multiple parts and outputs divided images which are the multiple divided images; a conversion unit that converts each divided image output from the image division unit into a one-dimensional vector having multiple vector components; a feature vector calculation unit that repeatedly selects a set of two or more one-dimensional vectors from the multiple one-dimensional vectors converted by the conversion unit, and calculates a feature vector related to the selected set based on the multiple vector components of the two or more one-dimensional vectors included in the set; and a classification unit that classifies the object to be classified based on the multiple feature vectors calculated by the feature vector calculation unit. [Effects of the Invention]

[0007] According to this disclosure, the number of parameters used to classify the objects to be classified in an image can be reduced. [Brief explanation of the drawing]

[0008] [Figure 1] This is a configuration diagram showing an image classification device according to Embodiment 1. [Figure 2] This is a hardware configuration diagram showing the hardware of the image classification device according to Embodiment 1. [Figure 3]This is a hardware configuration diagram of a computer when an image classification device is implemented using software or firmware. [Figure 4] This is a flowchart showing the image classification method, which is the processing procedure of an image classification device. [Figure 5] This is an explanatory diagram showing an example of an image Tg containing the object to be classified. [Figure 6] This is an explanatory diagram showing an example of a segmented image SPn (n=1,···,N), which is the image after segmentation by the image segmentation unit 1. [Figure 7] This is an explanatory diagram showing an example of a fully connected layer. [Figure 8] This is an explanatory diagram illustrating an example where the vector components {y1 8, y2 8, y3 8, y1 9, y2 9, y3 9} of the one-dimensional vectors h8 and h9 included in set SET8 are arranged in one row and one column. [Figure 9] This is an explanatory diagram showing the process of calculating the feature vector fvn by the feature vector calculation unit 3. [Figure 10] This is an explanatory diagram showing the process of calculating the feature vector fvn by the feature vector calculation unit 3. [Figure 11] This is a configuration diagram showing an image classification device according to Embodiment 2. [Figure 12] This is a hardware configuration diagram showing the hardware of the image classification device according to Embodiment 2. [Figure 13] This is an explanatory diagram showing an example of the activation function fact(·). [Modes for carrying out the invention]

[0009] To provide a more detailed explanation of this disclosure, the forms for implementing this disclosure will be described below with reference to the attached drawings.

[0010] Embodiment 1. Figure 1 is a configuration diagram showing an image classification device according to Embodiment 1. Figure 2 is a hardware configuration diagram showing the hardware of the image classification device according to Embodiment 1. The image classification device shown in Figure 1 comprises an image segmentation unit 1, a transformation unit 2, a feature vector calculation unit 3, and a classification unit 4.

[0011] The image segmentation unit 1 is implemented, for example, by the image segmentation circuit 21 shown in Figure 2. Image segmentation unit 1 acquires images containing the object to be classified. The image splitting unit 1 divides the image containing the object to be classified into multiple parts. Specifically, the image division unit 1 divides the image containing the object to be classified into multiple parts such that each divided image includes a portion of an adjacent divided image. Alternatively, the image division unit 1 divides the image containing the object to be classified into multiple parts such that each divided image does not include a portion of an adjacent divided image. The image splitting unit 1 outputs the split images, which are multiple images after splitting, to the conversion unit 2.

[0012] The conversion unit 2 is implemented, for example, by the conversion circuit 22 shown in Figure 2. The conversion unit 2 includes a conversion processing unit 2a and a normalization processing unit 2b. The conversion unit 2 obtains multiple divided images from the image division unit 1. The conversion unit 2 converts each segmented image into a one-dimensional vector having multiple vector components. The conversion unit 2 outputs multiple one-dimensional vectors to the feature vector calculation unit 3.

[0013] The conversion processing unit 2a acquires multiple divided images from the image division unit 1. The conversion processing unit 2a converts each segmented image into a one-dimensional vector. The transformation processing unit 2a outputs each one-dimensional vector to the normalization processing unit 2b. The normalization processing unit 2b obtains each one-dimensional vector from the transformation processing unit 2a. The normalization processing unit 2b performs normalization on each one-dimensional vector. The normalization processing unit 2b outputs the one-dimensional vector after each normalization process to the feature vector calculation unit 3.

[0014] The feature vector calculation unit 3 is realized by, for example, the feature vector calculation circuit 23 shown in FIG. 2. The feature vector calculation unit 3 includes a vector component arrangement unit 11, a vector component multiplication unit 12, and a feature vector calculation processing unit 13. The feature vector calculation unit 3 acquires a plurality of one-dimensional vectors from the conversion unit 2. The feature vector calculation unit 3 repeatedly selects a set of two or more one-dimensional vectors from among the plurality of one-dimensional vectors. For each selected set, the feature vector calculation unit 3 calculates a feature vector related to the selected set based on a plurality of vector components included in the two or more one-dimensional vectors included in the set. The feature vector calculation unit 3 outputs the feature vectors related to the plurality of sets to the classification unit 4.

[0015] The vector component arrangement unit 11 acquires a plurality of one-dimensional vectors from the conversion unit 2. The vector component arrangement unit 11 repeatedly selects a set of two or more one-dimensional vectors from among the plurality of one-dimensional vectors. The vector component arrangement unit 11 arranges the plurality of vector components included in the two or more one-dimensional vectors included in the selected set in a row and arranges the plurality of vector components included in the two or more one-dimensional vectors in a column. The vector component arrangement unit 11 outputs the one-dimensional vector having the plurality of vector components arranged in a row and the one-dimensional vector having the plurality of vector components arranged in a column to the vector component multiplication unit 12 and the feature vector calculation processing unit 13, respectively.

[0016] The vector component multiplication unit 12 includes a 90-degree rotation unit 12a, a first multiplication processing unit 12b, an addition result arrangement unit 12c, a second multiplication processing unit 12d, and a multiplication result addition unit 12e. The vector component multiplication unit 12 acquires the one-dimensional vector having the plurality of vector components arranged in a row and the one-dimensional vector having the plurality of vector components arranged in a column from the vector component arrangement unit 11. The vector component multiplication unit 12 multiplies each vector component arranged in a row by each vector component arranged in a column. The vector component multiplication unit 12 outputs the multiplication result of each vector component to the feature vector calculation processing unit 13.

[0017] The 90-degree rotation unit 12a acquires from the vector component arrangement unit 11 a one-dimensional vector having each vector component arranged in a row and a one-dimensional vector having each vector component arranged in a column. The 90-degree rotation unit 12a rotates the one-dimensional vector having each vector component arranged in a row by 90 degrees and rotates the one-dimensional vector having each vector component arranged in a column by 90 degrees.

[0018] The first multiplication processing unit 12b acquires from the 90-degree rotation unit 12a each rotated one-dimensional vector. The first multiplication processing unit 12b multiplies each vector component arranged in a row of the rotated one-dimensional vector by each vector component arranged in a column of the rotated one-dimensional vector. The first multiplication processing unit 12b generates a one-dimensional vector by arranging the multiplication results of each vector component in a column. The first multiplication processing unit 12b outputs the generated one-dimensional vector to the addition result arrangement unit 12c.

[0019] The addition result arrangement unit 12c adds the one-dimensional vector output from the first multiplication processing unit 12b and the one-dimensional vector output from the vector component arrangement unit 11. The addition result arrangement unit 12c compresses the added one-dimensional vector so that the number of vector components of the added one-dimensional vector matches the number of vector components of two or more one-dimensional vectors included in the set selected by the vector component arrangement unit 11. The addition result placement unit 12c adds each vector component of the compressed one-dimensional vector to each vector component of the two or more one-dimensional vectors included in the selected set. The addition result placement unit 12c places the sum of each vector component in one row and the sum of each vector component in one column. The addition result placement unit 12c outputs a one-dimensional vector having multiple vector components arranged in one row and a one-dimensional vector having multiple vector components arranged in one column to the second multiplication processing unit 12d.

[0020] The second multiplication processing unit 12d obtains a one-dimensional vector having multiple vector components arranged in one row and a one-dimensional vector having multiple vector components arranged in one column from the addition result placement unit 12c. The second multiplication processing unit 12d multiplies each vector component arranged in a row with each vector component arranged in a column. The second multiplication processing unit 12d generates a one-dimensional vector by arranging the multiplication results of each vector component in a single column. The second multiplication processing unit 12d outputs the generated one-dimensional vector to the multiplication result addition unit 12e.

[0021] The multiplication result addition unit 12e adds the one-dimensional vector output from the first multiplication processing unit 12b and the one-dimensional vector output from the addition result placement unit 12c. The one-dimensional vector output from the addition result placement unit 12c is a one-dimensional vector having multiple vector components arranged in a single row. The multiplication result addition unit 12e outputs the one-dimensional vector after addition to the feature vector calculation processing unit 13.

[0022] The feature vector calculation processing unit 13 obtains the one-dimensional vector after addition from the multiplication result addition unit 12e. The feature vector calculation processing unit 13 compresses the added one-dimensional vector so that the number of vector components in the added one-dimensional vector matches the number of vector components in two or more one-dimensional vectors included in the set selected by the vector component arrangement unit 11. The feature vector calculation processing unit 13 uses the compressed one-dimensional vector to calculate the feature vectors related to the set selected by the vector component arrangement unit 11. The feature vector calculation processing unit 13 outputs feature vectors relating to multiple pairs to the classification unit 4.

[0023] The classification unit 4 is implemented, for example, by the classification circuit 24 shown in Figure 2. The classification unit 4 includes an average value calculation unit 14a and a classification processing unit 14b. The classification unit 4 obtains feature vectors relating to multiple pairs from the feature vector calculation unit 3. The classification unit 4 classifies the objects to be classified based on feature vectors relating to multiple sets. The classification unit 4 displays the classification result of the items to be classified on a display device (not shown), for example.

[0024] The average value calculation unit 14a obtains feature vectors relating to multiple pairs from the feature vector calculation processing unit 13. The average value calculation unit 14a calculates the average value of each vector component contained in multiple feature vectors. The average value calculation unit 14a outputs a feature vector having the average value of each vector component to the classification processing unit 14b. The classification processing unit 14b obtains the average value of the feature vector, which has the average value of each of its vector components, from the average value calculation unit 14a. The classification processing unit 14b classifies the objects to be classified based on the average value of each vector component.

[0025] In Figure 1, the image classification device is assumed to be implemented by dedicated hardware as shown in Figure 2, with each of its components—the image segmentation unit 1, the transformation unit 2, the feature vector calculation unit 3, and the classification unit 4—being implemented by dedicated hardware. Specifically, the image classification device is assumed to be implemented by an image segmentation circuit 21, a transformation circuit 22, a feature vector calculation circuit 23, and a classification circuit 24. Each of the image segmentation circuit 21, conversion circuit 22, feature vector calculation circuit 23, and classification circuit 24 can be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0026] The components of an image classification device are not limited to those implemented by dedicated hardware; the image classification device may also be implemented by software, firmware, or a combination of software and firmware. Software or firmware is stored as a program in the computer's memory. A computer refers to the hardware that executes programs, and includes, for example, a CPU (Central Processing Unit), GPU (Graphics Processing Unit), central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, processor, or DSP (Digital Signal Processor).

[0027] Figure 3 is a hardware configuration diagram of a computer when an image classification device is implemented using software or firmware. When the image classification device is realized by software, firmware, etc., a program for causing a computer to execute the respective processing procedures in the image segmentation unit 1, the conversion unit 2, the feature vector calculation unit 3, and the classification unit 4 is stored in the memory 31. Then, the processor 32 of the computer executes the program stored in the memory 31.

[0028] Also, FIG. 2 shows an example in which each component of the image classification device is realized by dedicated hardware, and FIG. 3 shows an example in which the image classification device is realized by software, firmware, etc. However, this is only an example, and some components in the image classification device may be realized by dedicated hardware and the remaining components may be realized by software, firmware, etc.

[0029] Next, the operation of the image classification device shown in FIG. 1 will be described. FIG. 4 is a flowchart showing an image classification method which is a processing procedure of the image classification device. The image segmentation unit 1 acquires, as an image Tg in which the classification target is reflected, for example, an image as shown in FIG. 5. FIG. 5 is an explanatory diagram showing an example of an image Tg in which the classification target is reflected. In FIG. 5, an example in which the classification target is a passenger car is shown. The image segmentation unit 1 divides the image Tg in which the classification target is reflected into a plurality of parts (step ST1 in FIG. 4). The image segmentation unit 1 outputs the divided images SP, which are the plurality of divided images, to the conversion unit 2.

[0030] Hereinafter, the division processing of the image Tg by the image segmentation unit 1 will be specifically described. FIG. 6 is an explanatory diagram showing an example of the divided image SP n (n = 1, ···, N). In the example of FIG. 6, N = 7. When the image segmentation unit 1 divides the image Tg into N (N is an integer greater than or equal to 2) when the number of pixels in the horizontal direction of the image Tg in which the classification target is reflected is P, each of the divided images SP nThe image Tg may be divided equally such that the number of pixels for (n=1,···,N) is P / N. Below are the divided images SP when the image Tg is divided equally. n Equal division image ED n It is called that. However, in the image classification device shown in Figure 1, in order to avoid losing the amount of spatial information contained in the image Tg, each segmented image SP n However, the image division unit 1 overlaps and divides image Tg so that it includes a portion of an adjacent equally divided image. For example, if the divided image is SP1, the adjacent equally divided image is SP2, and if the divided image is SP2, the adjacent equally divided images are SP1 and SP3. For example, if the divided image is SP6, the adjacent equally divided images are SP5 and SP7, and if the divided image is SP7, the adjacent equally divided image is SP6.

[0031] When overlapping and dividing an image Tg, for example, the leftmost divided image SP1 includes the leftmost image EDL2, which is part of the second equally divided image ED2 from the left. If the horizontal number of pixels in image EDL2 is M (where M is an integer greater than or equal to 1), then the number of pixels in the leftmost divided image SP1 is P / N + M. For example, the second segmented image SP2 from the left includes the rightmost image EDR1, which is part of the leftmost equally segmented image ED1, and the leftmost image EDL3, which is part of the third equally segmented image ED3 from the left. If the number of horizontal pixels in images EDR1 and EDR3 is M, then the number of pixels in the second segmented image SP2 from the left is P / N + M + M. For example, the third segmented image from the left, SP3, includes the rightmost image EDR2, which is part of the second equally segmented image ED2 from the left, and the leftmost image EDL4, which is part of the fourth equally segmented image ED4 from the left. If the number of horizontal pixels in images EDR2 and EDR4 is M, then the number of pixels in the third segmented image SP3 from the left is P / N + M + M. For example, the rightmost split image SP N This is the second equally divided image from the right, ED. N-1The rightmost image EDR, which is a part of the images N-1 is included. EDR N-1 If the number of horizontal pixels of EDR N is M, the number of pixels of the rightmost divided image SP

[0032] The conversion unit 2 acquires N divided images SP1 to SP N from the image division unit 1. The conversion unit 2 converts the divided image SP n (n = 1, ···, N) into a one-dimensional vector a n having K vector components (step ST2 in FIG. 4). K is an integer of 2 or more. The conversion unit 2 outputs the one-dimensional vector a n (n = 1, ···, N) to the feature vector calculation unit 3.

[0033] Hereinafter, the conversion process of the one-dimensional vector a n by the conversion unit 2 will be specifically described. The conversion processing unit 2a acquires N divided images SP1 to SP N from the image division unit 1. The conversion processing unit 2a includes, for example, a fully connected layer as shown in FIG. 7. The conversion processing unit 2a uses the fully connected layer to convert the divided image SP ? n into a one-dimensional vector a n . The conversion processing unit 2a outputs the one-dimensional vector a n to the normalization processing unit 2b.

[0034] FIG. 7 is an explanatory diagram showing an example of a fully connected layer. The input of the fully connected layer shown in FIG. 7 corresponds to the pixel value x n of each pixel included in the divided image SP n , and the output of the fully connected layer shown in FIG. 7 corresponds to the vector component y n of the one-dimensional vector a k n . k = 1, ···, K. A weight coefficient w is set between each input and each output of a fully connected layer. The weight coefficient w is a parameter related to the weights. In the example in Figure 7, the segmented image SP n The number of pixels is 4, and the one-dimensional vector a n The vector component y that has k n The number K is 2. In the example in Figure 7, the one-dimensional vector a n =(y1 n ,y n 2) can be expressed as shown in the following equations (1) and (2). y1 n =x1 n w 11 +x2 n w 12 +x3 n w 13 +x4 n w 14 +b1(1) y2 n =x1 n w 21 +x2 n w 22 +x3 n w 23 +x4 n w 24 +b2(2) In equations (1) and (2), b1 and b2 are parameters related to bias, respectively. The image classification device disclosed in Non-Patent Document 1 has a processing unit that calculates the feature quantities of segmented images, and in order to perform calculations such as CNN (Convolutional Neural Network) or MLP (Multilayer Perceptron), it provides weight parameters and bias parameters as feature calculation parameters. In the image classification device shown in Figure 1, the first multiplication processing unit 12b and the second multiplication processing unit 12d, which are processing units described later for calculating the feature quantities of segmented images, do not use parameters related to weights or parameters related to bias. Therefore, the image classification device shown in Figure 1 can reduce the number of parameters used to classify the objects to be classified in the image compared to the image classification device disclosed in Non-Patent Document 1.

[0035] The normalization processing unit 2b receives a one-dimensional vector a from the transformation processing unit 2a. n Get (n=1,···,N). The normalization processing unit 2b processes a one-dimensional vector a as shown in the following equations (3) to (5). n Perform normalization on the data. The normalization processing unit 2b outputs the normalized one-dimensional vector h n The result is output to the feature vector calculation unit 3.

[0036] TIFF2026057637000002.tif65166 In equation (5), f[] is the normalization function, and g and b are constants.

[0037] The feature vector calculation unit 3 receives the normalized one-dimensional vector h from the normalization processing unit 2b of the transformation unit 2. n Get (n=1,···,N). For example, the internal memory of the feature vector calculation unit 3 contains a one-dimensional vector h 0 The one-dimensional vector h is stored there. 0 This is, for example, a one-dimensional vector representing a plain background that does not contain vector components related to the object to be classified. The feature vector calculation unit 3 generates (N+1) one-dimensional vectors h 0 ~h N From among them, a set of two or more one-dimensional vectors n Select N items. The feature vector calculation unit 3 calculates the selected set SET n Each set nBased on the multiple vector components of two or more one-dimensional vectors contained within, select a set SET n Feature vector fv related to this feature n Calculate (Step ST3 in Figure 4). The feature vector calculation unit 3 calculates N sets SET1 to SET N Feature vectors fv1~fv N Output to classification unit 4.

[0038] The following is the feature vector fv calculated by the feature vector calculation unit 3. n The calculation process will be explained in detail. Figure 8 shows the one-dimensional vector h included in set SET8. 8 ,h 9 The vector components {y1 8 ,y2 8 ,y3 8 ,y1 9 ,y2 9 ,y3 9 This is an explanatory diagram showing an example where the characters} are arranged in one row and one column. The vector component arrangement unit 11 of the feature vector calculation unit 3 receives the normalized one-dimensional vector h from the transformation unit 2. n Obtain (n=1,···,N). In the example in Figure 8, N=9. The vector component arrangement section 11 consists of (N+1) one-dimensional vectors h 0 ~h N From among them, a set of two or more one-dimensional vectors n Select N items.

[0039] group SET n The number of two or more one-dimensional vectors contained in a set is J (where J is an integer greater than or equal to 2), and the set is SET. n Two or more one-dimensional vectors h are included. n The interval is S (where S is an integer greater than or equal to 1), and J and S are predetermined. For the sake of simplicity, we will assume that N=9, J=2, and S=1. In this case, the one-dimensional vector included in set SET1 is {h 0 ,h 1}, the one-dimensional vectors included in set SET2 are {h 1 , h 2}, the one-dimensional vectors included in set SET3 are {h 2 , h 3}, the one-dimensional vectors included in set SET4 are {h 3 , h 4}, the one-dimensional vectors included in set SET5 are {h 4 , h 5}. Also, the one-dimensional vectors included in set SET6 are {h 5 , h 6}, the one-dimensional vectors included in set SET7 are {h 6 , h 7}, the one-dimensional vectors included in set SET8 are {h 7 h 8}, the one-dimensional vectors included in set SET9 are {h 8 , h 9}.

[0040] The vector component arrangement unit 11 arranges a plurality of vector components included in two or more one-dimensional vectors included in set SET n (n = 1, ···, N) in a row. Also, the vector component arrangement unit 11 arranges a plurality of vector components included in two or more one-dimensional vectors included in set SET n in a column. The vector component arrangement unit 11 outputs a one-dimensional vector O lin,n having a plurality of vector components arranged in a row and a one-dimensional vector O colu,n having a plurality of vector components arranged in a column to the vector component multiplication unit 12.

[0041] In FIG. 8, an example in which two or more one-dimensional vectors included in set SET9 are {h 8 , h 9} is shown. If the one-dimensional vector is {h 8 [[ID=6�]], h 9}, as shown in FIG. 8, the K = 3 vector components {y1 8}, {y2 8 , y28 ,y3 8} and a one-dimensional vector h 9 The vector components {y1} have K=3 9 ,y2 9 ,y3 9 The} and are arranged on a single line. In this case, the one-dimensional vector O lin,9 As {y1 8 ,y2 8 ,y3 8 ,y1 9 ,y2 9 ,y3 9 The result is output to the vector component multiplication unit 12. Also, one-dimensional vector h 8 The vector components {y1} have K=3 8 ,y2 8 ,y3 8} and a one-dimensional vector h 9 The vector components {y1} have K=3 9 ,y2 9 ,y3 9 The} and are arranged in a single column. In this case, the one-dimensional vector O colu,9 As {y1 8 ,y2 8 ,y3 8 ,y1 9 ,y2 9 ,y3 9 The result is output to the vector component multiplication unit 12.

[0042] Here, a one-dimensional vector h 8 The vector components {y1 8 ,y2 8 ,y3 8} and a one-dimensional vector h 9 The vector component {y1 9 ,y2 9 ,y3 9} and are arranged in one row, forming a one-dimensional vector O lin,9 ga {y1 8 ,y2 8 ,y3 8 ,y1 9 ,y2 9 ,y3 9 This shows an example where}. However, this is just one example, for example, a one-dimensional vector O lin,9ga {y3 9 ,y2 9 ,y1 9 ,y3 8 ,y2 8 ,y1 8} is also acceptable. Also, here, a one-dimensional vector h 8 The vector components {y1 8 ,y2 8 ,y3 8} and a one-dimensional vector h 9 The vector components {y1 9 ,y2 9 ,y3 9} and are arranged in a row, forming a one-dimensional vector O colu,9 ga {y1 8 ,y2 8 ,y3 8 ,y1 9 ,y2 9 ,y3 9 This shows an example where}. However, this is just one example, for example, a one-dimensional vector O colu,9 ga {y3 9 ,y2 9 ,y1 9 ,y3 8 ,y2 8 ,y1 8} is also acceptable.

[0043] Figure 9 shows the feature vector fv calculated by the feature vector calculation unit 3. n This is an explanatory diagram showing the calculation process. The 90-degree rotation unit 12a of the vector component multiplication unit 12 rotates the one-dimensional vector O from the vector component arrangement unit 11. lin,n And a one-dimensional vector O colu,n To obtain the following. The 90-degree rotation part 12a is a one-dimensional vector O lin,n and a one-dimensional vector O colu,n Rotate them by 90 degrees. Figure 9 shows the one-dimensional vector O, which is rotated by the 90-degree section 12a. lin,9 and one-dimensional vector O colu,9 This shows each of them rotated 90 degrees. In the example in Figure 9, the one-dimensional vector O after a 90-degree rotation is shown. lin,9 is {y39 ,y2 9 ,y1 9 ,y3 8 ,y2 8 ,y1 8} is the one-dimensional vector O after a 90-degree rotation. lin,9 O is a one-dimensional vector colu,9 It will be treated as such. In the example in Figure 9, the one-dimensional vector O after a 90-degree rotation is shown. colu,9 is, {y1 8 ,y2 8 ,y3 8 ,y1 9 ,y2 9 ,y3 9} is the one-dimensional vector O after a 90-degree rotation. colu,9 O is a one-dimensional vector lin,9 It will be treated as such. The 90-degree rotation section 12a is a one-dimensional vector O after 90-degree rotation. lin,n and the one-dimensional vector O after a 90-degree rotation colu,n The output is sent to the first multiplication processing unit 12b.

[0044] The first multiplication processing unit 12b receives a one-dimensional vector O after a 90-degree rotation from the 90-degree rotation unit 12a. lin,n and the one-dimensional vector O after a 90-degree rotation colu,n To obtain the following. The first multiplication processing unit 12b generates a one-dimensional vector O after a 90-degree rotation. lin,n The respective vector components of and the one-dimensional vector O after a 90-degree rotation colu,n Multiply each of its respective vector components. In the example in Figure 9, the shaded area represents the one-dimensional vector O after a 90-degree rotation. lin,9 The vector component {y3 9 ,y2 9 ,y1 9 ,y3 8 ,y2 8 ,y1 8} and the one-dimensional vector O after a 90-degree rotation. colu,9 The vector components {y1 8 ,y2 8 ,y3 8 ,y1 9 ,y2 9,y3 9 This is the area where the result of multiplication with} is stored. The white area is the one-dimensional vector O after a 90-degree rotation. lin,9 The vector component {y3 9 ,y2 9 ,y1 9 ,y3 8 ,y2 8 ,y1 8} and the one-dimensional vector O after a 90-degree rotation. colu,9 The vector components {y1 8 ,y2 8 ,y3 8 ,y1 9 ,y2 9 ,y3 9 This is an area where the multiplication result is not stored because multiplication with} does not occur.

[0045] In the example shown in Figure 9, the following multiplication is performed. · y3 9 ×y3 9 Multiplication of y2 9 ×y3 9 Multiplication of y1 9 ×y3 9 Multiplication of y3 8 ×y3 9 Multiplication of y2 8 ×y3 9 Multiplication of y1 8 ×y3 9 multiplication · y2 9 ×y2 9 Multiplication of y1 9 ×y2 9 Multiplication of y3 8 ×y2 9 Multiplication of y2 8 ×y2 9 Multiplication of y1 8 ×y2 9 multiplication · y1 9 ×y1 9 Multiplication of y3 8 ×y1 9 Multiplication of y2 8 ×y1 9 Multiplication of y1 8 ×y1 9 multiplication · y3 8 ×y38 Multiplication of y2 8 ×y3 8 Multiplication of y1 8 ×y3 9 multiplication · y2 8 ×y2 8 Multiplication of y1 8 ×y2 8 multiplication · y1 8 ×y1 8 multiplication

[0046] The first multiplication processing unit 12b generates a one-dimensional vector O after a 90-degree rotation. lin,n The respective vector components of and the one-dimensional vector O after a 90-degree rotation colu,n By arranging the results of multiplying each of the vector components that it possesses in a single line, we obtain a one-dimensional vector O nonlin,n Generates. The first multiplication processing unit 12b processes a one-dimensional vector O nonlin,n The result is output to the addition result placement unit 12c. In the above multiplication example, the first multiplication processing unit 12b arranges the multiplication results in a single line, thereby generating a one-dimensional vector O as shown in equation (6) below. nonlin,n This is generated. O nonlin,9 ={y3 9 ×y3 9 , y2 9 ×y3 9 , y1 9 ×y3 9 , y3 8 ×y3 9 , y2 8 ×y3 9 , y1 8 ×y3 9 , y2 9 ×y2 9 , y1 9 ×y2 9 , y3 8 ×y2 9 , y2 8 ×y2 9 , y1 8 ×y2 9 , y1 9 ×y1 9 , y3 8×y1 9 , y2 8 ×y1 9 , y1 8 ×y1 9 , y3 8 ×y3 8 , y2 8 ×y3 8 , y1 8 ×y3 9 , y2 8 ×y2 8 , y1 8 ×y2 8 , y1 8 ×y1 8 (6)

[0047] The addition result placement unit 12c receives a one-dimensional vector O from the first multiplication processing unit 12b. nonlin,n Obtain a one-dimensional vector O from the vector component arrangement unit 11. lin,n Obtain it. The addition result placement unit 12c is a one-dimensional vector O, as shown in equation (7) below. nonlin,n and a one-dimensional vector O lin,n By adding and, we obtain a one-dimensional vector O total,n Generates. O total,n =O lin,n +O nonlin,n (7) In the example in Figure 9, O total,9 This can be expressed as shown in equation (8) below. O total,9 ={y1 8 ,y2 8 ,y3 8 ,y1 9 ,y2 9 ,y3 9 ,y3 9 ×y3 9 , y2 9 ×y3 9 , y1 9 ×y3 9 , y3 8 ×y3 9 , y2 8 ×y3 9 , y1 8 ×y3 9 , y2 9 ×y29 , y1 9 ×y2 9 , y3 8 ×y2 9 , y2 8 ×y2 9 , y1 8 ×y2 9 , y1 9 ×y1 9 , y3 8 ×y1 9 , y2 8 ×y1 9 , y1 8 ×y1 9 , y3 8 ×y3 8 , y2 8 ×y3 8 , y1 8 ×y3 9 , y2 8 ×y2 8 , y1 8 ×y2 8 , y1 8 ×y1 8} (8)

[0048] The addition result placement unit 12c is a one-dimensional vector O total,n The number of vector components that it has is, n The number of vector components of two or more one-dimensional vectors contained in (n=1,···,N) is matched by the number of one-dimensional vectors O total,n Compress it. One-dimensional vector O total,n The compression process itself is a well-known technique, so a detailed explanation will be omitted, but for example, in a fully connected layer as shown in Figure 7, each input is a one-dimensional vector O total,n Given the vector components of the vector, the compressed one-dimensional vector O can be obtained from the outputs of each fully connected layer. total,n The vector components of are obtained. In this case, the number of inputs to the fully connected layer is a one-dimensional vector O total,n The number of vector components is the same as that of the fully connected layer, and the number of outputs of the SET n This is the same as the number of vector components that two or more one-dimensional vectors contained within it have. The addition result placement unit 12c is the compressed one-dimensional vector O total,n Each of the vector components that it possesses, and the set SET n The vector components of two or more one-dimensional vectors contained within are added together. Compressed one-dimensional vector O total,n Each of the vector components that it possesses is, for example, {c1 n ,c2 n ,c3 n ,c4 n ,c5 n ,c6 n} and the set SET n Each of the vector components of two or more one-dimensional vectors contained in {d1 n d2 n d3 n d4 n d5 n d6 n If so, the sum of each vector component is Add n This is expressed as shown in equation (9) below. Add n ={a1 n a2 n a3 n a4 n a5 n a6 n} ={c1 n +d1 n ,c2 n +d2 n ,c3 n +d3 n ,c4 n +d4 n ,c5 n +d5 n ,c6 n +d6 n (9)

[0049] The addition result placement unit 12c displays the addition result of each vector component. n Arrange them on one line, and add the sum of the respective vector components. n Arrange them in a single row. The addition result placement unit 12c is a one-dimensional vector O having multiple vector components arranged in a single row. lin,n ' and a one-dimensional vector O having multiple vector components arranged in a single column colu,n ' and are output to the second multiplication processing unit 12d. Figure 10 shows the feature vector fv calculated by the feature vector calculation unit 3. n This is an explanatory diagram showing the calculation process. In the example in Figure 10, the sum of each vector component is shown. n {a1 n a2 n a3 n a4 n a5 n a6 n The} are arranged in one row, and a one-dimensional vector O lin,n 'as {a1 n a2 n a3 n a4 n a5 n a6 n} is output to the second multiplication processing unit 12d. Also, the sum of the respective vector components Add n {a1 n a2 n a3 n a4 n a5 n a6 n The} are arranged in a single column, and a one-dimensional vector O colu,n 'as {a1 n a2 n a3 n a4 n a5 n a6 n} is output to the second multiplication processing unit 12d.

[0050] The second multiplication processing unit 12d receives a one-dimensional vector O from the addition result placement unit 12c. lin,n ' and a one-dimensional vector O colu,n 'Get it. The second multiplication processing unit 12d processes a one-dimensional vector O lin,n The respective vector components of ' and the one-dimensional vector O colu,nMultiply each of the vector components that ' possesses. In the example in Figure 10, the shaded area is a one-dimensional vector O lin,n The vector component {a1 n a2 n a3 n a4 n a5 n a6 n} and a one-dimensional vector O colu,n Each of the vector components {a1 n a2 n a3 n a4 n a5 n a6 n This is the region where the result of multiplication with} is stored. The white region is a one-dimensional vector O lin,n The vector component {a1 n a2 n a3 n a4 n a5 n a6 n} and a one-dimensional vector O colu,n Each of the vector components {a1 n a2 n a3 n a4 n a5 n a6 n This is an area where the multiplication result is not stored because multiplication with} does not occur.

[0051] In the example shown in Figure 10, the following multiplication is performed. ·a1 n ×a6 n Multiplication of a2 n ×a6 n Multiplication of a3 n ×a6 n Multiplication of a4 n ×a6 n Multiplication of a5 n ×a6 n Multiplication of a6 n ×a6 n multiplication ·a2 n ×a5 n Multiplication of a3 n ×a5n Multiplication of a4 n ×a5 n Multiplication of a5 n ×a5 n Multiplication of a6 n ×a5 n multiplication a3 n ×a4 n Multiplication of a4 n ×a4 n Multiplication of a5 n ×a4 n Multiplication of a6 n ×a4 n multiplication a4 n ×a3 n Multiplication of a5 n ×a3 n Multiplication of a6 n ×a3 n multiplication a5 n ×a2 n Multiplication of a6 n ×a2 n multiplication ·a6 n ×a1 n multiplication

[0052] The second multiplication processing unit 12d processes a one-dimensional vector O lin,n The respective vector components of ' and the one-dimensional vector O colu,n By arranging the results of multiplying each of the vector components that ' has into a single line, we obtain a one-dimensional vector O nonlin,n Generates '. The second multiplication processing unit 12d processes a one-dimensional vector O nonlin,n The result of the multiplication is output to the addition unit 12e. In the above multiplication example, the second multiplication processing unit 12d arranges the multiplication results in a single line, thereby generating a one-dimensional vector O as shown in equation (10) below. nonlin,n ' is generated. O nonlin,n ' ={a1 n ×a6 n a2 n ×a6 n a3 n ×a6 n a4n ×a6 n a5 n ×a6 n a6 n ×a6 n a2 n ×a5 n a3 n ×a5 n a4 n ×a5 n a5 n ×a5 n a6 n ×a5 n a3 n ×a4 n a4 n ×a4 n a5 n ×a4 n a6 n ×a4 n a4 n ×a3 n a5 n ×a3 n a6 n ×a3 n a5 n ×a2 n a6 n ×a2 n a6 n ×a1 n (10)

[0053] The multiplication result addition unit 12e receives a one-dimensional vector O from the second multiplication processing unit 12d. nonlin,n ' is obtained, and from the addition result placement unit 12c, a one-dimensional vector O lin,n Get '. The multiplication result addition unit 12e is a one-dimensional vector O, as shown in equation (11) below. nonlin,n ' and a one-dimensional vector O lin,n By adding ' and , a one-dimensional vector O total,n Generates '. O total,n '=O lin,n '+O nonlin,n (11) In the example in Figure 10, O total,n ' can be expressed as shown in equation (12) below. O total,n ' ={a1n a2 n a3 n a4 n a5 n a6 n a1 n ×a6 n a2 n ×a6 n a3 n ×a6 n a4 n ×a6 n a5 n ×a6 n a6 n ×a6 n a2 n ×a5 n a3 n ×a5 n a4 n ×a5 n a5 n ×a5 n a6 n ×a5 n a3 n ×a4 n a4 n ×a4 n a5 n ×a4 n a6 n ×a4 n a4 n ×a3 n a5 n ×a3 n a6 n ×a3 n a5 n ×a2 n a6 n ×a2 n a6 n ×a1 n} (12) The multiplication result addition unit 12e is a one-dimensional vector O total,n The output is sent to the feature vector calculation processing unit 13.

[0054] The feature vector calculation processing unit 13 calculates a one-dimensional vector O from the multiplication result addition unit 12e. total,n Get '. The feature vector calculation processing unit 13 calculates a one-dimensional vector O total,nThe number of vector components that ' has is, n The number of vector components of two or more one-dimensional vectors contained in (n=1,···,N) is matched by the number of one-dimensional vectors O total,n Compress '. One-dimensional vector O by the feature vector calculation processing unit 13 total,n The compression process of ' is performed by the addition result placement unit 12c to create a one-dimensional vector O total,n This is similar to the compression process. The feature vector calculation processing unit 13 calculates the compressed one-dimensional vector O total,n Each of the vector components that ' possesses, and the set SET n The vector components of two or more one-dimensional vectors contained within are added together.

[0055] Compressed one-dimensional vector O total,n Each of the vector components that ' has is, for example, {cc1 n cc2 n cc3 n cc4 n cc5 n cc6 n} and the set SET n Each of the vector components of two or more one-dimensional vectors contained in {dd1 n dd2 n dd3 n dd4 n dd5 n dd6 n If so, the sum of each vector component is Add n ' becomes as shown in equation (13) below. Add n ' ={aa1 n ,aa2 n ,aa3 n ,aa4 n ,aa5 n ,aa6 n} ={cc1 n +dd1 n cc2 n +dd2 n cc3 n +dd3n cc4 n +dd4 n cc5 n +dd5 n cc6 n +dd6 n (13) The feature vector calculation processing unit 13 performs a set SET n Feature vector fv related to this feature n The sum of the respective vector components is added. n A one-dimensional vector O having ' total,n Output " to classification unit 4.

[0056] The classification unit 4 receives the set SET from the feature vector calculation unit 3. n Feature vector fv related to (n=1,···,N) n Obtain it. The classification unit 4 consists of N feature vectors fv1~fv N Based on this, classify the items to be classified (Step ST4 in Figure 4).

[0057] Specifically, the mean calculation unit 14a of the classification unit 4 calculates the average value of N feature vectors fv1 to fv N Calculate the average value of each vector component contained within. The average value calculation unit 14a calculates a feature vector fv that has the average value of each vector component. ave The output is sent to the classification processing unit 14b. The average value of each vector component is aa1 1 ~aa1 N The average value of aa2 1 ~aa2 N The average value of aa3 1 ~aa3 N The average value of aa4 1 ~aa4 N The average value of aa5 1 ~aa5 N The average value and aa6 1 ~aa6 N This is the average value. The classification processing unit 14b processes the feature vector fv ave ={aa1 1 ~aa1 N The average value of aa21 ~aa2 N The average value of aa3 1 ~aa3 N The average value of aa4 1 ~aa4 N The average value of aa5 1 ~aa5 N The average value of aa6 1 ~aa6 N Based on the average value, the objects to be classified in image Tg are classified.

[0058] Feature vector fv ave The classification process of the object to be classified based on this is a well-known technique, so a detailed explanation will be omitted, but for example, for a trained model, the feature vector fv ave By providing the necessary information, the learning model can obtain classification results for the items to be classified. These classification results, for example, show the probability that an item belongs to each of several categories. Examples of these categories include passenger cars, light vehicles, buses, trucks, motorcycles, or bicycles. During training, the learning model uses, for example, a feature vector fv ave When a pair of the feature vector fv and the classification result is given as training data, ave This is a supervised model that has learned the corresponding classification results. The learning model, during inference, uses the feature vector fv ave Given the parameters, the system outputs the probability that each item belongs to its respective category as a classification result. The classification unit 4 displays the classification result of the items to be classified on a display device (not shown), for example.

[0059] In the above embodiment 1, the image classification device is configured to include: an image division unit 1 that divides an image containing the object to be classified into multiple parts and outputs divided images, which are the multiple divided images; a transformation unit 2 that converts each divided image output from the image division unit 1 into a one-dimensional vector having multiple vector components; a feature vector calculation unit 3 that repeatedly selects a set of two or more one-dimensional vectors from the multiple one-dimensional vectors converted by the transformation unit 2, and calculates a feature vector related to the selected set based on the multiple vector components of the two or more one-dimensional vectors included in the set; and a classification unit 4 that classifies the object to be classified based on the multiple feature vectors calculated by the feature vector calculation unit 3. Therefore, the image classification device can reduce the number of parameters used to classify the object to be classified in the image.

[0060] In Embodiment 1, the image division unit 1 is configured as an image classification device that divides an image containing the object to be classified into multiple parts such that each divided image includes a portion of an adjacent divided image. Therefore, the image classification device can avoid losing the amount of spatial information contained in the image containing the object to be classified, and as a result, the classification accuracy of the object to be classified in the image can be improved.

[0061] In Embodiment 1, the image classification device is configured such that the feature vector calculation unit 3 includes a vector component arrangement unit 11 that arranges multiple vector components of two or more one-dimensional vectors included in the selected set in one row and multiple vector components of two or more one-dimensional vectors in one column, a vector component multiplication unit 12 that multiplies each vector component arranged in one row by the vector component arrangement unit 11 with each vector component arranged in one column by the vector component arrangement unit 11, and a feature vector calculation processing unit 13 that calculates a feature vector relating to the selected set using the multiple vector components of two or more one-dimensional vectors included in the selected set and the multiple multiplication results from the vector component multiplication unit 12. Therefore, the image classification device can calculate a feature vector relating to the selected set.

[0062] In Embodiment 1, the image classification device is configured such that the classification unit 4 calculates the average value of each vector component of the multiple feature vectors calculated by the feature vector calculation unit 3, and classifies the object to be classified based on the average value. Therefore, the image classification device can reduce the number of parameters used to classify the objects to be classified in the image, and can also improve the classification accuracy of the objects to be classified in the image. Alternatively, instead of the classification unit 4 calculating the average value of each vector component and classifying the items based on the average value, the classification unit 4 may, for example, calculate the median value of each vector component and classify the items based on the median value.

[0063] In the image classification device shown in Figure 1, the vector component multiplication unit 12 comprises a 90-degree rotation unit 12a, a first multiplication processing unit 12b, an addition result placement unit 12c, a second multiplication processing unit 12d, and a multiplication result addition unit 12e. However, this is merely one example, and the vector component multiplication unit 12 may also comprise a 90-degree rotation unit 12a, a first multiplication processing unit 12b, and an addition result placement unit 12c. In this case, the feature vector calculation processing unit 13 calculates the one-dimensional vector O output from the multiplication result addition unit 12e. total,n Compress the resulting one-dimensional vector O total,n Each of the vector components that it possesses, and the set SET n The vector components of two or more one-dimensional vectors contained within are added together. Then, the feature vector calculation processing unit 13 calculates the set SET n Feature vector fv related to this feature n As such, a one-dimensional vector O having the sum of its respective vector components. total,n Output to classification unit 4.

[0064] Embodiment 2. Embodiment 2 describes an image classification device that includes a position information addition unit 5 that adds position information indicating the position of the segmented image related to each feature vector to each feature vector calculated by the feature vector calculation unit 3.

[0065] Figure 11 is a configuration diagram showing an image classification device according to Embodiment 2. In Figure 11, the same reference numerals as in Figure 1 indicate the same or corresponding parts, so a detailed explanation is omitted. Figure 12 is a hardware configuration diagram showing the hardware of the image classification device according to Embodiment 2. In Figure 12, the same reference numerals as in Figure 2 indicate the same or corresponding parts, so a detailed explanation is omitted. The image classification device shown in Figure 11 comprises an image splitting unit 1, a transformation unit 2, a feature vector calculation unit 3, a position information addition unit 5, and a classification unit 6.

[0066] The location information addition unit 5 is implemented, for example, by the location information addition circuit 25 shown in Figure 12. The location information addition unit 5 obtains feature vectors relating to multiple sets from the feature vector calculation unit 3. The position information addition unit 5 adds position information to each feature vector, indicating the position of the segmented image related to that feature vector. The location information addition unit 5 outputs the feature vector after location information addition to the classification unit 6.

[0067] The classification unit 6 is implemented, for example, by the classification circuit 26 shown in Figure 12. The classification unit 6 includes an average value calculation unit 15a and a classification processing unit 15b. The classification unit 6 obtains multiple feature vectors with added location information from the location information addition unit 5. The classification unit 6 classifies the objects to be classified based on multiple feature vectors to which location information has been added. The classification unit 6 displays the classification result of the items to be classified on a display device (not shown), for example.

[0068] The average value calculation unit 15a obtains multiple feature vectors with added location information from the location information addition unit 5. The average value calculation unit 15a calculates the average value of each vector component contained in the multiple feature vectors after position information has been added. The average value calculation unit 15a outputs a feature vector having the average value of each vector component to the classification processing unit 15b. The classification processing unit 15b obtains the average value of the feature vector, which has the average value of each of its vector components, from the average value calculation unit 15a. The classification processing unit 15b classifies the objects to be classified based on the average value of each vector component.

[0069] In Figure 11, the image classification device is assumed to be implemented by dedicated hardware as shown in Figure 12, with each of its components—the image segmentation unit 1, the conversion unit 2, the feature vector calculation unit 3, the location information addition unit 5, and the classification unit 6—being implemented by dedicated hardware. Specifically, the image classification device is assumed to be implemented by an image segmentation circuit 21, a conversion circuit 22, a feature vector calculation circuit 23, a location information addition circuit 25, and a classification circuit 26. Each of the image segmentation circuit 21, conversion circuit 22, feature vector calculation circuit 23, position information addition circuit 25, and classification circuit 26 can be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof.

[0070] The components of an image classification device are not limited to those implemented by dedicated hardware; the image classification device may also be implemented by software, firmware, or a combination of software and firmware. When an image classification device is implemented using software or firmware, a program is stored in the memory 31 shown in Figure 3 that instructs a computer to execute the respective processing procedures in the image segmentation unit 1, the transformation unit 2, the feature vector calculation unit 3, the position information addition unit 5, and the classification unit 6. Then, the processor 32 shown in Figure 3 executes the program stored in the memory 31.

[0071] Furthermore, Figure 12 shows an example in which each component of the image classification device is implemented by dedicated hardware, and Figure 3 shows an example in which the image classification device is implemented by software or firmware, etc. However, this is only one example, and some components of the image classification device may be implemented by dedicated hardware, while the remaining components may be implemented by software or firmware, etc.

[0072] Next, the operation of the image classification device shown in Figure 11 will be explained. However, it is the same as the image classification device shown in Figure 1, except for the location information addition unit 5 and the classification unit 6. Therefore, only the operation of the location information addition unit 5 and the classification unit 6 will be explained here.

[0073] The position information addition unit 5 receives the set SET from the feature vector calculation unit 3. n Feature vector fv related to (n=1,···,N) n Obtain it. The location information addition unit 5 uses the feature vector fv n For this, the feature vector fv n Related segmented image SP n Location information indicating the position (pos) n Add it. For example, split image SP n Location information indicating the position (pos) n is a split image SP n This is positional information indicating the relative position between the position of [element name] and the position of the segmented image SP1. For example, split image SP n-1 Location information indicating the position (pos) n-1 is a split image SP n-1 This is positional information indicating the relative position between the position of [element name] and the position of the segmented image SP1. For example, position information pos1, which indicates the position of segmented image SP1, is position information that indicates the relative position between the position of segmented image SP1 and the position of segmented image SP1.

[0074] The position information addition unit 5 uses the feature vector fv as shown in equation (14) below. n In contrast, location information pos n Add it. fv n '=pos n ×fv n (14) The location information addition unit 5 adds the feature vector fv n Output ' to classification unit 6.

[0075] The classification unit 6 receives the feature vector fv after adding location information from the location information addition unit 5. n Get (n=1,···,N). The classification unit 6 consists of N feature vectors fv1'~fv N Classify the items to be classified based on '.

[0076] Specifically, the mean calculation unit 15a of the classification unit 6 calculates the N feature vectors fv1'~fv N Calculate the average value of each vector component contained within '. The average value calculation unit 15a calculates a feature vector fv that has the average value of each vector component. ave The output is sent to the classification processing unit 14b. The classification processing unit 15b processes the feature vector fv ave Based on this, classify the items to be classified in image Tg.

[0077] Feature vector fv ave The classification process of the target to be classified based on ' is a well-known technique, so a detailed explanation will be omitted, but for example, for a trained model, the feature vector fv ave By providing the input, the learning model can be used to obtain the classification result for the target object. During training, the learning model uses, for example, a feature vector fv ave When a set of location information and classification result is given as training data, the feature vector fv ave This is a supervised model that has learned classification results corresponding to ' and location information.' The learning model, during inference, uses the feature vector fv ave 'and location information pos n Given the given parameters, the system outputs the probability that each item belongs to its respective category as a classification result. The classification unit 6 displays the classification result of the items to be classified on a display device (not shown), for example.

[0078] In the above embodiment 2, the image classification device shown in Figure 11 is configured such that a position information addition unit 5 adds position information indicating the position of the segmented image related to each feature vector calculated by the feature vector calculation unit 3, and the classification unit 6 classifies the object to be classified based on the multiple feature vectors to which position information has been added by the position information addition unit 5. Therefore, the image classification device shown in Figure 11 can classify the object to be classified with higher accuracy than the image classification device shown in Figure 1.

[0079] In the image classification device shown in Figure 11, the position information addition unit 5 adds the feature vector fv n For this, the feature vector fv n Related segmented image SP n Location information indicating the position (pos) n It has been added. The position information addition unit 5 generates the feature vector fv n against location information pos n Before adding the following, you may perform the following preprocessing steps.

[0080] The position information addition unit 5 receives the set SET from the feature vector calculation unit 3. n Feature vector fv related to (n=1,···,N) n Obtain it. The location information addition unit 5 uses the feature vector fv n An uncompressed process is performed to increase the number of vector components in the feature vector fv. n This is the reverse process of compression. The position information addition unit 5 processes the feature vector fv after uncompression. n Substitute this into the activation function fact(·) shown in Figure 13. Figure 13 is an explanatory diagram showing an example of the activation function fact(·). In Figure 13, the horizontal axis (labeled "Input" in the figure) represents the feature vector fv that is substituted into the activation function fact(·).n The vector components of the function, represented by the vertical axis (labeled "Output" in the figure), are the vector components output from the activation function fact(·). The activation function fact(·) is used to emphasize only certain features, and the feature vector fv n This function increases one or more of the multiple vector components of a given vector, while setting the remaining components to zero.

[0081] The position information addition unit 5 displays the activated feature vector fv n This obtains the output vector of the activation function fact(·). The position information addition unit 5 displays the activated feature vector fv n A dropout process is performed, which randomly sets the value of one of the vector components of the vector to 0. The position information addition unit 5 outputs the feature vector fv after dropout processing. n The number of vector components that " has is, n One-dimensional vector h contained in (n=1,···,N) n The feature vector fv n Compress ". The position information addition unit 5 outputs the compressed feature vector fv n In contrast to location information pos n Add it.

[0082] Furthermore, this disclosure allows for free combination of each embodiment, modification of any component in each embodiment, or omission of any component in each embodiment. [Explanation of symbols]

[0083] 1 Image splitting unit, 2 Transformation unit, 2a Transformation processing unit, 2b Normalization processing unit, 3 Feature vector calculation unit, 4 Classification unit, 5 Position information addition unit, 6 Classification unit, 11 Vector component placement unit, 12 Vector component multiplication unit, 12a 90-degree rotation unit, 12b First multiplication processing unit, 12c Addition result placement unit, 12d Second multiplication processing unit, 12e Multiplication result addition unit, 13 Feature vector calculation processing unit, 14a Average value calculation unit, 14b Classification processing unit, 15a Average value calculation unit, 15b Classification processing unit, 21 Image splitting circuit, 22 Transformation circuit, 23 Feature vector calculation circuit, 24 Classification circuit, 25 Position information addition circuit, 26 Classification circuit, 31 Memory, 32 Processor.

Claims

1. An image splitting unit divides an image containing the object to be classified into multiple parts and outputs the resulting split images, A transformation unit that converts each divided image output from the image division unit into a one-dimensional vector having multiple vector components, A feature vector calculation unit repeatedly selects pairs of two or more one-dimensional vectors from among a plurality of one-dimensional vectors converted by the conversion unit, and for each selected pair, calculates a feature vector related to the selected pair based on the plurality of vector components of the two or more one-dimensional vectors contained in the pair. Based on the multiple feature vectors calculated by the feature vector calculation unit, a classification unit classifies the object to be classified. An image classification device equipped with [unclear] features.

2. The aforementioned image division unit is The image containing the object to be classified is divided into multiple parts such that each divided image includes a portion of an adjacent divided image. The image classification apparatus according to claim 1, characterized in that it is a feature of the present invention.

3. The conversion unit is Normalization is performed on each one-dimensional vector, and the normalized one-dimensional vector is output to the feature vector calculation unit. The image classification apparatus according to claim 1, characterized in that it is a feature of the present invention.

4. The feature vector calculation unit, A vector component arrangement unit that arranges multiple vector components of two or more one-dimensional vectors included in the selected set in one row, and arranges multiple vector components of the two or more one-dimensional vectors in one column, A vector component multiplication unit multiplies each vector component arranged in one row by the vector component arrangement unit and each vector component arranged in one column by the vector component arrangement unit, A feature vector calculation processing unit calculates a feature vector relating to the selected set using a plurality of vector components of two or more one-dimensional vectors included in the selected set and a plurality of multiplication results from the vector component multiplication unit. The image classification device according to claim 1, characterized by comprising the following features.

5. The aforementioned classification unit is The feature vector calculation unit calculates the average value of each vector component of the multiple feature vectors it calculates, and classifies the target to be classified based on the average value. The image classification apparatus according to claim 1, characterized in that it is a feature of the present invention.

6. The feature vector calculation unit adds position information indicating the position of the segmented image related to each feature vector to each feature vector calculated by the feature vector calculation unit, The aforementioned classification unit is The classification target is classified based on a plurality of feature vectors to which location information has been added by the location information addition unit. The image classification apparatus according to any one of claims 1 to 5, characterized by the following:

7. The image splitting unit divides the image containing the object to be classified into multiple parts, and outputs the split images, which are the multiple images after the division. The conversion unit converts each divided image output from the image division unit into a one-dimensional vector having multiple vector components. The feature vector calculation unit repeatedly selects pairs of two or more one-dimensional vectors from among the multiple one-dimensional vectors converted by the transformation unit, and for each selected pair, calculates a feature vector related to the selected pair based on the multiple vector components of the two or more one-dimensional vectors included in the pair. The classification unit classifies the objects to be classified based on the multiple feature vectors calculated by the feature vector calculation unit. Image classification method.

Images