Feature Quantity Extraction Device, Feature Quantity Extraction Method, Recognition Device, Recognition Method, and Storage Medium

Abstract

The recognition device (100) of the present invention includes: an acquisition unit (11) that acquires a captured image obtained by capturing a prescribed object; a determination unit (12) that determines linear regions extending in a plurality of mutually different directions from a region including the prescribed object in the captured image acquired by the acquisition unit (11); and an extraction unit (13) that extracts a feature amount based on pixel values from the linear regions determined by the determination unit (12).

Description

Technical Field

[0001] The present invention relates to a feature quantity extraction device, a feature quantity extraction method, an identification device, an identification method, and a storage medium. Background Art

[0002] Conventionally, the following processing has been performed: A part related to a diagnosis object such as a black spot on a human skin is photographed by a camera, and the photographed image is used to assist the diagnosis of the diagnosis object part. For example, in the technique disclosed in Non-Patent Document 1, for such diagnostic assistance, a whole photographed image including a part other than the diagnosis object part in the skin and a cropped image obtained by cropping the diagnosis object part from the whole photographed image are acquired, and a plurality of feature quantities such as color histograms are extracted from the acquired whole photographed image and cropped image, and each of the plurality of extracted feature quantities is input to the corresponding one of a plurality of recognizers, and the outputs of the plurality of recognizers thus obtained are averaged, thereby calculating a disease risk score indicating whether the diagnosis object part is a disease.

[0003] Prior Art Documents

[0004] Non-Patent Documents

[0005] Non-Patent Document 1: N.C. Codella, Q.B. Nguyen, S. Pankanti, D. Gutman, B. Helba, A. Halpern, J.R. Smith, "Deep learning ensembles for melanoma recognition in dermoscopy images", IBM Journal of Research and Development, vol. 61, no. 4 / 5, 2017 Summary of the Invention

[0006] Problems to be Solved by the Invention

[0007] In the technology disclosed in Non-Patent Document 1, since feature amounts are extracted from each of a whole captured image and a cropped image, that is, from 2D images, the load of the process for such extraction becomes excessively large. In order to suppress the load of such a process, it is considered to determine a partial region of a diagnostic object portion in a captured image and extract a feature amount from the determined partial region. However, if such a process is performed, it may not be possible to appropriately extract a feature amount for the diagnostic object portion, and furthermore, it may not be possible to appropriately perform identification of a disease using a disease risk score. The above problems apply not only to the case of extracting a feature amount of a diagnostic object portion of a person's skin, but also to the case of extracting a feature amount of another appropriate object including at least one of color and luminance that are different from each other, for example, a feature amount of a diagnostic object portion of a person's cervical part.

[0008] The present invention has been made to solve the above problems, and an object thereof is to provide a feature amount extraction device, a feature amount extraction method, an identification device, an identification method, and a program that can appropriately extract a feature amount while suppressing the load for extracting the feature amount.

[0009] Means for Solving the Problem

[0010] To achieve the above object, the feature amount extraction device of the present invention includes:

[0011] An acquisition unit that acquires a captured image obtained by capturing a prescribed object;

[0012] A determination unit that determines linear regions extending in a plurality of different directions from a region including the prescribed object in the captured image acquired by the acquisition unit;

[0013] An extraction unit that extracts a feature amount based on pixel values from the linear regions determined by the determination unit.

[0014] Advantageous Effects of the Invention

[0015] According to the present invention, it is possible to appropriately extract a feature amount while suppressing the load for extracting the feature amount. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a diagram showing a functional configuration of an identification device according to a first embodiment.

[0017] Figure 2 is a diagram for explaining a linear region determined by a determination unit according to the first embodiment.

[0018] Figure 3 is a diagram for explaining an example of acquiring pixel values along a curve.

[0019] Figure 4This is a diagram illustrating an example of the obtained one-dimensional data.

[0020] Figure 5 This is a flowchart of the learning process of the first embodiment.

[0021] Figure 6 This is a flowchart of the recognition process of the first embodiment.

[0022] Figure 7 This is a diagram illustrating the structural outline of the one-dimensional CNN of the second embodiment.

[0023] Figure 8 This is a flowchart of the learning process of the second embodiment.

[0024] Figure 9 This is a flowchart of the recognition process of the second embodiment.

[0025] Figure 10 This is a diagram showing the functional structure of the recognition device of the third embodiment.

[0026] Figure 11 This is a flowchart of the optimal parameter selection process of the third embodiment.

[0027] Figure 12 This is a flowchart of the active score calculation process of the third embodiment. Detailed Embodiments

[0028] Hereinafter, the recognition device and the like according to the embodiments of the present invention will be described with reference to the drawings. In addition, the same or corresponding parts in the drawings are denoted by the same reference numerals.

[0029] (First Embodiment)

[0030] The recognition device 100 according to the first embodiment of the present invention is a device for recognizing a specified object based on a captured image obtained by photographing the specified object. For example, the recognition device 100 recognizes whether the diagnostic object part (e.g., a tumor, a black spot, a freckle, a bump, a blood vessel, a skin disease part such as atopic eczema or drug eruption, a rash caused by collagen disease, a part suspected of having a skin disease) included in a person's skin, which is the diagnostic object as the specified object, is benign or malignant based on the captured image obtained by photographing the person's skin. The recognition by such a recognition device 100 is performed to assist a doctor's diagnosis. In addition, in the recognition based on the image, the recognition device 100 does not directly use two-dimensional image data, but obtains one-dimensional data from the image data and uses the obtained one-dimensional data to recognize the specified object in the image.

[0031] Specifically, the recognition device 100 obtains one-dimensional data from the image data (learning image data) included in a pre-prepared image data set for learning, uses the feature amounts extracted from the one-dimensional data to make a support vector machine (SVM) learn (machine learning), and uses the learned SVM to perform image recognition. In addition, since the recognition device 100 extracts the feature amounts for making the SVM learn, it is also a feature amount extraction device.

[0032] A correct answer label corresponding to the use of image recognition is assigned to the learning image data. Regarding the recognition device 100 of the first embodiment, as the above-described diagnostic target part, based on the image of the black spot taken, it is recognized whether the black spot is benign or malignant. Therefore, the learning image data is the image data of the black spot whose benign or malignant nature has been determined, and in each learning image data, whether the black spot shown in the image is benign or malignant is assigned as the correct answer label.

[0033] In addition, here, it is assumed that the recognition device 100 performs the recognition of the benign / malignant nature of the black spot and is described, but the recognition target of the recognition device 100 is not limited to skin diseases such as black spots. For example, regarding cervical cancer, oral cancer (tongue cancer, gum cancer, floor of the mouth cancer), etc., these diagnostic target parts can also be used as the object, and the disease name of the object can be recognized based on the image obtained by photographing the object. In this case, the learning image data is the image data obtained by photographing the diagnostic target part whose disease name has been determined, and in each learning image data, the disease name of the diagnostic target part shown in the image is assigned as the correct answer label.

[0034] Furthermore, the recognition target is not limited to the diagnostic target part of the human body. Other appropriate objects can also be considered. For example, for the deterioration state of concrete, the image of the concrete is used for recognition. In this case, the learning image data is the image data obtained by photographing the concrete whose deterioration has been determined, and in each learning image data, the information on whether the concrete shown in the image has deteriorated is assigned as the correct answer label. In addition, the fact that the recognition target is not limited to the diagnostic target part of the human body also applies to other embodiments and modification examples described later.

[0035] As Figure 1 shown, the recognition device 100 of the first embodiment includes a control unit 10, a storage unit 20, an image input unit 31, an output unit 32, a communication unit 33, and an operation input unit 34 as functional structures.

[0036] The control unit 10 is composed of a CPU (Central Processing Unit) and the like, and realizes the functions of the following respective units (acquisition unit 11, determination unit 12, extraction unit 13, setting unit 14, recognition unit 15) by executing a program stored in the storage unit 20.

[0037] The storage unit 20 is composed of a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and stores a program executed by the CPU of the control unit 10 and necessary data. The storage unit 20 also stores image data (learning image data) for learning by the recognition unit 15. However, the recognition device 100 may also acquire a part or all of the learning image data from the communication unit 33.

[0038] The image input unit 31 is a device for inputting learning image data and image data of an image of an object to be recognized (unknown) taken by a user. The image input unit 31 is, for example, a camera, and inputs the captured image data to the control unit 10. The image used by the recognition device 100 when recognizing a specified object is an image including a plurality of regions in which at least one of color and brightness is different from each other. Therefore, the image input unit 31 inputs an image including a plurality of regions in which at least one of color and brightness is different from each other to the control unit 10. Among the plurality of regions in which at least one of color and brightness is different from each other, for example, a region including an edge portion of the image and other regions other than this.

[0039] In addition, as the image input unit 31, if the control unit 10 can acquire image data, it is not limited to a camera, and any device can be used. For example, in the case where the storage unit 20 stores image data in advance and the control unit 10 reads out the image data from the storage unit 20 and thereby acquires the image data, the storage unit 20 also serves as the image input unit 31. In addition, in the case where the control unit 10 acquires image data from an external server or the like via the communication unit 33, the communication unit 33 also serves as the image input unit 31.

[0040] The output unit 32 is a device for the control unit 10 to output an image recognition result and the like. For example, the output unit 32 is a liquid crystal display or an organic EL (Electro-Luminescence) display. However, the recognition device 100 may include these displays as the output unit 32, or may include an output unit 32 as an interface for connecting to an external display. When the recognition device 100 includes an output unit 32 as an interface, the image recognition result and the like are displayed on an external display connected via the output unit 32.

[0041] The communication unit 33 is a device (such as a network interface) for transmitting and receiving data to and from other external devices (for example, a server of a database storing image data, etc.). The control unit 10 can acquire image data via the communication unit 33.

[0042] The operation input unit 34 is a device that accepts operation inputs from the user to the recognition device 100, such as a keyboard, a mouse, a touch panel, etc. The recognition device 100 receives instructions from the user via the operation input unit 34.

[0043] Next, the functions of the control unit 10 will be described. The control unit 10 implements the functions of the acquisition unit 11, the determination unit 12, the extraction unit 13, the setting unit 14, and the recognition unit 15.

[0044] The acquisition unit 11 acquires the image captured by the image input unit 31. As described above, the image used when the recognition device 100 recognizes a specified object is an image including a plurality of regions where at least one of color and brightness is different from each other. The acquisition unit 11 acquires a captured image obtained by capturing a specified object (such as a diagnostic object part of the skin like a black spot) including such regions. In addition, the acquisition unit 11 does not need to acquire the captured image in real time. For example, it can acquire a captured image captured in the past and stored in the storage unit 20, or can receive and acquire a captured image captured in the past and saved in other servers, etc. via the communication unit 33. The acquisition unit 11 functions as an acquisition unit for acquiring the captured image.

[0045] The determination unit 12 determines linear regions extending in a plurality of mutually different directions from the recognition region including a specified object (such as a diagnostic object part of the skin) in the captured image acquired by the acquisition unit 11. The lines extending in a plurality of mutually different directions refer to a plurality of lines extending in mutually different directions (for example, a plurality of lines radially extending from near the center of the specified object) and lines in which the directions of local regions on the lines face a plurality of directions (circles, ellipses, spirals, etc.). Here, the "local region on the line" is a part (a minute line segment) of various positions of the line. For example, as Figure 2 shown, the determination unit 12 determines the linear region through which the spiral curve 43 passes within the recognition region 42 including the diagnostic object part of the skin (the part suspected of being a diseased part), that is, the object region 41. The object region 41 includes a plurality of regions 41a, 41b where at least one of color and brightness is different from each other. The determination unit 12 functions as a determination unit for determining such a linear region.

[0046] The extraction unit 13 extracts a feature amount based on pixel values from the linear region determined by the determination unit 12. More specifically, first, as Figure 3As shown, the extraction unit 13 obtains one-dimensional data formed by sequentially obtaining pixel values (values of each of R, G, and B) along a spiral curve 43 from one end to the other end of the region determined by the determination unit 12. Here, the pixel value is represented by the value of each of the R (Red), G (Green), and B (Blue) components in the RGB color space, and the obtained one-dimensional data becomes one-dimensional data of three-channel amounts (each of Red, Green, and Blue) as shown in Figure 4 In addition, in Figure 4 it is recorded as "distance" on the horizontal axis, but the "distance" in Figure 4 refers to "the distance when observing along the spiral from one end point of the spiral". In addition, in this one-dimensional data, additional information such as time information indicating when the image was taken, patient ID information indicating which patient the captured image is of, and doctor ID information indicating the doctor of the patient being photographed can also be added.

[0047] Then, the extraction unit 13 calculates prescribed statistical values (for example, average, variance, maximum value, minimum value, median, slope of the regression line (a line obtained by approximating one-dimensional data with a linear function and obtained by the least squares method, etc.), contribution rate of the regression line (indicating the degree to which the regression line fits the one-dimensional data of each of the RGB components used in this calculation, also called the coefficient of determination), etc.) for the one-dimensional data of each of the obtained R component, G component, and B component. In addition, the extraction unit 13 also processes the differences (R component - G component, R component - B component, G component - B component) and ratios (R component / G component, R component / B component, G component / B component) between the RGB components in the same way as one-dimensional data of each of them, and calculates the prescribed statistical values (average, variance, etc.) as described above.

[0048] And the extraction unit 13 extracts, for example, the variances of the one-dimensional data of the R component and the G component respectively, the slopes and contribution rates of the regression lines of the one-dimensional data of each of the RGB components, the variance of the absolute value of the differences (differences of the RGB components (R component - G component, R component - B component, G component - B component)) between the one-dimensional data of each of the RGB components, the slopes and contribution rates of the regression lines of R component / G component and R component / B component among the ratios (ratios of the RGB components (R component / G component, R component / B component, G component / B component)) between the one-dimensional data of each of the RGB components, and the sum of the differences between the maximum value and the minimum value in the one-dimensional data of each of the RGB components (difference between the maximum value of the RGB component and the minimum value of the RGB component), a total of 16 values as feature quantities. That is, in this example, the feature quantity is represented by a 16-dimensional vector. The extraction unit 13 functions as an extraction unit for extracting feature quantities.

[0049] In addition, here, the extraction unit 13 acquires each pixel value as the value of each of the R component, G component, and B component in the RGB color space. However, the color space representing the pixel value is not limited to the RGB color space. For example, the YUV color space, Lab color space, etc. may also be used. Since these color spaces each represent one pixel value with three components, the R component, Y component, and L component can be represented as the first component in each color space, the G component, U component, and a component can be represented as the second component in each color space, and the B component, V component, and b component can be represented as the third component in each color space.

[0050] Moreover, the extraction unit 13 may also acquire the pixel values of the region on the line determined by the determination unit 12 as the one-dimensional data of each of the first component, second component, and third component in any of the above color spaces, and extract at least one of the variances of the one-dimensional data of the first component and the second component respectively, the slopes and contribution rates of the regression lines of the one-dimensional data of the first component, second component, and third component respectively, the variance of the absolute values of the differences between the one-dimensional data, the slopes and contribution rates of the regression lines of the ratios of the one-dimensional data to each other, i.e., the first component / second component and the first component / third component, and the sum of the differences between the maximum value and the minimum value of the one-dimensional data, a total of 16 values, as feature quantities.

[0051] The setting unit 14 randomly sets the parameters for defining a spiral curve passing through the region determined by the determination unit 12. This parameter is, for example, the type of spiral (Archimedean spiral, parabolic spiral, etc., which is the Archimedean spiral in Figure 2 ), the number of turns of the spiral (which is 3 turns in Figure 2 ), the orientation (which is clockwise from the inside to the outside in Figure 2 ), the starting angle (which is the direction perpendicular by 90 degrees with respect to the rectangle of the recognition region 42 in Figure 2 ), etc. The setting unit 14 functions as a setting unit for randomly setting the parameters of the spiral defining the spiral region.

[0052] The recognition unit 15 uses the feature quantities extracted by the extraction unit 13 to recognize a specified object (such as the part to be diagnosed for the skin). Specifically, the recognition unit 15 is an SVM obtained by performing machine learning using the feature quantities extracted by the extraction unit 13 from the learning image data. Here, since the recognition device 100 is a device for recognizing whether a skin dark spot is benign or malignant, the recognition unit 15 has an SVM for recognizing whether a dark spot is benign or malignant. Then, based on the output value obtained by inputting the feature quantity as input data into this SVM, the recognition unit 15 recognizes whether this dark spot is benign or malignant. That is, if the output value of the SVM is a value indicating benign, the recognition result of the recognition unit 15 is benign, and if the output value of the SVM is a value indicating malignant, the recognition result of the recognition unit 15 is malignant.

[0053] In addition, in the present embodiment, a case where the recognition device 100 recognizes the benign / malignant nature of a dark spot has been described. However, the content recognized by the recognition device 100 for the dark spot is not limited to the benign / malignant nature, and the object to be recognized is not limited to the dark spot. The recognition device 100 may also recognize the disease names of skin diseases (such as pigmented moles (dark spots), melanoma, seborrheic keratosis, etc.). In this case, the recognition unit 15 includes an SVM that performs one-to-many recognition based on the number of recognized classes. For example, when the recognition device 100 recognizes three diseases, namely pigmented moles (dark spots), melanoma, and seborrheic keratosis, the recognition unit 15 includes three SVMs: an SVM for recognizing pigmented moles (dark spots), an SVM for recognizing melanoma, and an SVM for recognizing seborrheic keratosis. Then, the feature quantity is input into these three SVMs, and the disease name corresponding to the SVM that obtains the output with the highest probability becomes the recognition result of the recognition unit 15.

[0054] More generally, when there are n classes, such as A, B, C, …, as the objects recognized by the recognition device 100, the recognition unit 15 includes n SVMs for recognizing “A or other than A”, “B or other than B”, “C or other than C”, …. Then, the feature quantity is input into these n SVMs, and the class (A, B, C, …) corresponding to the SVM that obtains the output with the highest probability becomes the recognition result of the recognition unit 15. The recognition unit 15 functions as a recognition unit for recognizing a specified object (such as a diagnostic object part of the skin like a dark spot).

[0055] Above, the functional structure of the recognition device 100 has been described. Next, refer to Figure 5 The learning process of the recognition device 100 will be described. When the user instructs the recognition device 100 to start the learning process via the operation input unit 34, the learning process starts.

[0056] First, the acquisition unit 11 acquires learning image data from the storage unit 20 (step S101). Step S101 is also referred to as the acquisition step. Next, the determination unit 12 extracts an object region 41 that includes a specified object (such as a diagnostic object part of the skin like a dark spot) from the learning image data acquired by the acquisition unit 11 (step S102).

[0057] Regarding the extraction of the object region 41 in step S102, it can be automatically performed using a deep neural network (DNN) such as FCN (Fully Convolutional Networks) that has learned the lesion region (here, the black spot), or can be semi-automatically performed using techniques such as interactive graph cutting, or can be manually performed by the user specifying the object region 41 through the operation input unit 34, etc.

[0058] Then, the determination unit 12 calculates the rectangular recognition region 42 that encloses the extracted object region 41 (step S103). For example, the determination unit 12 sets Figure 2 the rectangle shown, with the maximum diameter of any two points inside the extracted object region 41 as the major axis, and determines the minor axis in such a way that it is inscribed in the object region 41 in the direction orthogonal to the major axis, and calculates the recognition region 42.

[0059] Next, the setting unit 14 sets random values as the parameters of the spiral (type of spiral, number of turns of the spiral, orientation, starting angle) that define the inside of the recognition region 42 calculated by the determination unit 12 (step S104). For example, as the type of spiral, it is randomly set from Archimedean spiral, parabolic spiral, etc. Also, as the number of turns of the spiral, for example, it is randomly set from a number around 2 to 6. In addition, the number of turns can be set to any number, and can also be set to a number greater than 6 above. Also, the number of turns does not have to be an integer, and can also be set to 3.5 turns, 3.25 turns, etc. Also, as the orientation of the spiral, it is randomly set to clockwise or counterclockwise. In addition, the starting angle of the spiral can be randomly set to any angle, but for example, certain candidates such as 0 degrees, 90 degrees, 180 degrees, 270 degrees can be determined and randomly set from among them.

[0060] Then, the determination unit 12 uses the parameters of the spiral set by the setting unit 14, as Figure 2As shown, a spiral is defined such that the center of the spiral is substantially coincident with the center of the recognition region 42 and is inscribed in the recognition region 42 (step S105). Step S105 is a step of determining linear regions extending in a plurality of different directions from the recognition region 42, and is thus also referred to as a determination step. Further, the center of the spiral may be exactly coincident with the center of the recognition region 42 or may be offset. Additionally, when offsetting the center of the spiral, the amount and direction of the offset of the center of the spiral from the center of the recognition region 42 may be set randomly. Also, the amount and direction of the offset between the center of the recognition region 42 and the center of the spiral may be added to the "parameters defining the spiral" set in step S104. If the offset amount and direction are added to the "parameters defining the spiral", more diverse learning under the spiral is performed, and thus it is possible to improve the accuracy of the recognition unit 15.

[0061] Next, as Figure 3 shown, the extraction unit 13 obtains pixel values along the spiral defined by the determination unit 12, and obtains Figure 4 the one-dimensional data of each of RGB as shown (step S106). Further, regarding the acquisition of pixel values, it may be acquired in a form in which a new pixel value is appended to the one-dimensional data whenever the pixel changes along the spiral, or it may be acquired in a form in which the pixel value at that position is appended to the one-dimensional data whenever a predetermined distance is advanced along the spiral. Additionally, as the color space, a color space other than the RGB color space (YUV color space, Lab color space, etc.) may be used as described above. When using a color space other than the RGB color space, the extraction unit 13 performs a color space conversion when obtaining the one-dimensional data. For example, when converting the RGB color space to the YUV color space, the extraction unit 13 obtains the one-dimensional data for each of the Y component, U component, and V component.

[0062] Further, the extraction unit 13 may set the size (the number of acquired pixel values) of the one-dimensional data acquired in step S106 to a fixed length. In the case of setting it to a fixed length, the extraction unit 13 performs linear interpolation on the one-dimensional data to make the size consistent with the specified length.

[0063] Then, the extraction unit 13 calculates the average, variance, etc. for the acquired one-dimensional data, or calculates the average, variance, etc. for the differences and ratios between the components, thereby extracting feature amounts (such as the above-mentioned 16-dimensional vector), and stores them together with the correct answer label assigned to the learning image data in the storage unit 20 (step S107). Step S107 is also referred to as an extraction step.

[0064] Next, the control unit 10 determines whether the processes from step S104 to step S107 have been repeated a specified number of times (e.g., 10 times) (step S108). If the specified number of times has not been repeated (step S108; No), the process returns to step S104. By repeating the specified number of times, the variation in the parameters of the randomly set spiral in step S104 increases, and thus the feature amounts based on the one-dimensional data along the spirals of various shapes can be extracted, so that the robustness of the learning of the SVM of the recognition unit 15 can be improved. However, the robustness of the learning of the SVM of the recognition unit 15 can also be improved by increasing the number of learning image data. Therefore, when a very large number (e.g., 10,000 or more) of learning image data can be prepared, the repetition of the specified number of times may not be performed in step S108 (the process of step S108 can be skipped, and the process can immediately proceed to step S109).

[0065] If the specified number of times has been repeated (step S108; Yes), the control unit 10 determines whether the processes from step S103 to step S108 have been performed for all the object regions in the learning image data (step S109). This is to extract the learning feature amounts for all the object regions when there are multiple object regions in one learning image data. If the processes from step S103 to step S108 have not been performed for all the object regions (step S109; No), the process returns to step S103.

[0066] If the processes from step S103 to step S108 have been performed for all the object regions (step S109; Yes), the control unit 10 determines whether the feature amount extraction has been performed for all the learning image data (step S110). If the feature amount extraction has not been performed for all the learning image data yet (step S110; No), the process returns to step S101.

[0067] If the feature amount extraction has been performed for all the learning image data (step S110; Yes), the SVM of the recognition unit 15 is made to learn using the feature amounts and the correct answer labels stored in the storage unit 20 in step S107 (step S111), and the learning process ends. In addition, Figure 5 The flowchart shown is merely an example, and the processing order and processing contents may be changed in order to improve the learning efficiency and recognition performance.

[0068] The above describes the learning process. Next, with reference to Figure 6 The recognition process of performing image recognition using the recognition unit 15 (SVM) learned through the above learning process is described. When the user instructs the recognition device 100 to start the recognition process via the operation input unit 34, the recognition process starts.

[0069] First, the acquisition unit 11 acquires the image to be recognized (recognition image) from the image input unit 31 or the like (step S201). Step S201 is also referred to as the acquisition step. For example, the image input unit 31 captures the image that the user wants to recognize, and thereby, the acquisition unit 11 acquires the recognition image.

[0070] Next, the determination unit 12 extracts the object region 41 that includes a specified object (such as a diagnostic object part of the skin like a black spot) from the recognition image acquired by the acquisition unit 11 (step S202). The extraction of the object region 41 in step S202 can be performed in any of the automatic, semi-automatic, and manual manners, similarly to the extraction in the above-mentioned step S102.

[0071] Next, the determination unit 12 calculates the rectangular recognition region 42 that encloses the extracted object region 41 (step S203). For example, the determination unit 12 sets Figure 2 the rectangle shown, with the maximum diameter of any two points inside the extracted object region 41 as the major axis, and determines the minor axis in such a way that it is inscribed in the object region 41 in the direction orthogonal to the major axis, to calculate the recognition region 42.

[0072] Then, the determination unit 12 sets typical values (for example, Figure 2 the type of spiral shown: Archimedean spiral, number of turns: 3, orientation: clockwise, start angle: perpendicular 90-degree direction) as the parameters (type of spiral, number of turns of the spiral, orientation, start angle) that define the spiral inside the calculated recognition region 42. Using the set parameters, with the center of the recognition region 42 as the center of the spiral, a spiral inscribed in the recognition region 42 is defined (step S204). Step S204 is a step of determining linear regions extending in a plurality of different directions from the recognition region 42, and thus is also referred to as the determination step. In addition, in step S204, the determination unit 12 may randomly set the parameters instead of setting typical values as the parameters that define the spiral.

[0073] Then, the extraction unit 13 acquires the pixel values along the spiral defined by the determination unit 12 as Figure 3 shown, and acquires the 1D data for each of RGB as Figure 4 shown (step S205). Step S205 is also referred to as the 1D data acquisition step. Additionally, in step S205, the extraction unit 13 functions as a 1D data acquisition unit. Moreover, regarding the acquisition unit of the pixel values and the color space used at this time, they are consistent with the content performed in the learning process ( Figure 5 , step S106). Additionally, when the size of the 1D data is set to a fixed length in the learning process, the extraction unit 13 performs linear interpolation on the 1D data here to make the size consistent with the specified length that is the same as that in the learning process.

[0074] Then, the extraction unit 13 calculates the average, variance, etc. for the obtained one-dimensional data, or calculates the average, variance, etc. for the differences and ratios between components, thereby extracting feature quantities (such as the above-mentioned 16-dimensional vector) (step S206).

[0075] Then, the recognition unit 15 inputs the feature quantities extracted in step S206 to the SVM that has completed the learning process through Figure 5 , thereby performing the recognition of a specified object (black spot) included in the object region extracted in step S202 (step S207). Step S207 is also referred to as the recognition step. Then, the control unit 10 outputs the result recognized by the recognition unit 15 to the output unit 32, that is, outputs to the output unit 32 whether the black spot is a benign black spot or a malignant black spot (step S208), and ends the recognition process.

[0076] As described above, the recognition process has been described. As described above, the recognition device 100 of the first embodiment uses one-dimensional data when extracting feature quantities from an image, so compared with the case of directly processing two-dimensional image data, the processing load can be reduced. That is, the recognition device 100 can perform image recognition at high speed only through the recognition device 100 without using external computer resources. Therefore, the recognition device 100 can also be used when cloud processing (computer resources such as servers on the Internet) cannot be used, when multiple images need to be recognized (such as image screening, etc.), when real-time recognition is required, etc.

[0077] In addition, in the recognition device 100, by changing the parameters of the line (helix) when obtaining one-dimensional data, compared with the case of directly learning a two-dimensional image, more learning data (one-dimensional data) can be obtained from one image, so the total number of image data prepared for learning can be suppressed to be less.

[0078] In addition, in the above-mentioned first embodiment, linear recognition is performed by SVM, but in the learning process ( Figure 5 ) step S111 and the recognition process ( Figure 6 ) step S207, non-linear recognition can also be performed by using an SVM, etc., where the SVM uses the kernel method and corresponds to non-linear recognition. In addition, thresholds can be set for each dimension of the feature quantity, and according to whether the element of this dimension is above the threshold or less than the threshold, the feature quantity can be represented by a binary vector, and recognition can be performed through the non-linear score of the operation result based on this binary vector.

[0079] (Second Embodiment)

[0080] The recognition device 100 of the first embodiment uses predefined feature quantities and performs recognition through SVM. However, if a DNN such as a Convolutional Neural Network (CNN) is applied to the above-mentioned one-dimensional data, recognition can be performed without previously defining feature quantities (such as the calculation method of each element of the vector constituting the feature quantity). The recognition device 101 of such a second embodiment will be described.

[0081] The functional structure of the recognition device 101 of the second embodiment is the same as that of Figure 1 the recognition device 100 of the first embodiment shown, but the recognition unit 15 is a CNN instead of an SVM, and the extraction unit 13 makes the CNN perform machine learning by updating the weights inside the CNN of the recognition unit 15. As a result, the extraction unit 13 can extract the output value of the CNN and the feature maps inside the CNN as feature quantities.

[0082] Here, the CNN used in the recognition device 101 of the second embodiment will be described. A CNN is a neural network that mimics the functions of nerve cells in the human visual cortex and generally directly processes two-dimensional image data for image recognition and the like. However, as Figure 7 shown, the CNN of the second embodiment is a one-dimensional convolutional neural network that takes one-dimensional data of RGB (3 channels) as input data and directly processes it as one-dimensional data for N-class recognition.

[0083] As Figure 7 shown, the CNN of the second embodiment directly performs convolutional processing (filter scanning) and pooling processing (window scanning) on the input layer 111 of the input RGB one-dimensional data in one dimension, and gradually calculates smaller-sized feature maps 112, 113, 114, 115, 116, 117, and finally obtains an N-dimensional output vector from the output layer 118. In the example shown in Figure 7 for the input one-dimensional data, by scanning the convolutional processing filters 121, 123, 124, 125 and the pooling processing windows 122, 126 in the horizontal direction with a stride of 2, smaller-sized feature maps are gradually calculated to obtain the final output. In addition, "scanning with a stride of 2" means skipping one pixel or the elements constituting the feature map for scanning.

[0084] Weight coefficients are assigned to each element of filters 121, 123, 124, and 125, and scanned horizontally on the input 1D data or feature map with the same number of channels. Thus, the inner product results of scalar values are output at each focus point of the input 1D data or feature map, thereby obtaining a new feature map. Then, by applying multiple (n) filters, feature maps of that number (n channels) are obtained. Additionally, every time scanning is performed with a stride of 2, the size in the horizontal direction of the feature map becomes 1 / 2 of the size. As a result, as going to the subsequent stage, more global feature extraction is performed (the filter size relatively expands with respect to the feature map size).

[0085] In Figure 7 In the example shown, the final layer (feature map 117) of the intermediate layer of the CNN and the output layer 118 are connected by the fully-connected connection 127 to which weight coefficients are assigned, and weighted addition operations are performed in the same manner as in a normal neural network. Since the final layer of the intermediate layer of the CNN is connected to the output layer 118 by the fully-connected connection 127, it is also called the fully-connected layer. In this example, since N-class recognition is performed, the output layer 118 has N elements (or units), and the size of the estimated recognition probability is represented according to the magnitude of the values of the elements.

[0086] In the CNN, it is possible to obtain the weight coefficients assigned to each connection of the fully-connected connection 127 and the weight coefficients of the above-mentioned filters 121, 123, 124, 125 using the prepared learning data. Specifically, first, the learning data is input as an input image to the input layer 111, and forward propagation is performed to obtain an output result from the output layer 118. Then, the extraction unit 13 obtains the difference (error) between the output result and the correct answer (the correct answer label given to the input learning data), and using the error backpropagation method, updates the weight coefficients in the direction of reducing the error. This operation is repeated while reducing the learning rate (the update amount of the weight coefficients in the error backpropagation method), thereby converging the values of the weight coefficients.

[0087] After learning the respective weight coefficients of the CNN using the learning data, by performing forward propagation with unknown image data as the input image data, an inference value for recognition based on the input image, that is, an output result, is obtained in the output layer 118. This output result becomes the recognition result of the recognition unit 15.

[0088] Above, an outline of the 1D CNN included in the recognition unit 15 of the recognition device 101 has been described. In addition, Figure 7 The CNN shown is merely an example of a 1D CNN. If it is a CNN (1D CNN) that inputs 1D data to the input layer and directly processes it as 1D data, the recognition device 101 can use any 1D CNN.

[0089] Next, referring toFigure 8 The learning process performed on the recognition device 101 will be described. When the user instructs the recognition device 101 to start the learning process via the operation input unit 34, the learning process is started.

[0090] First, the extraction unit 13 initializes all the weight parameters of the CNN of the recognition unit 15 (step S301). The subsequent steps S302 to S307 are the same as the processing of steps S101 to S106 in the learning process of the first embodiment ( Figure 5 ), so the description thereof is omitted.

[0091] Then, the extraction unit 13 resamples the one-dimensional data of the pixel values (RGB, YUV, Lab, etc. according to the color space used) obtained in step S307 by linear interpolation so that the size is consistent with the size of the input layer of the CNN of the recognition unit 15 (in Figure 7 the example shown is 224) (step S308).

[0092] Next, the extraction unit 13 inputs the resampled one-dimensional data into the CNN of the recognition unit 15 (step S309) to obtain the output value of the CNN. Then, the extraction unit 13 calculates an error based on the output value of the CNN and the correct answer label assigned to the learning image data (step S310) and stores it in the storage unit 20.

[0093] Then, the control unit 10 determines whether the processing of steps S304 to S310 has been performed on all the object regions in the learning image data (step S311). This is because, when there are multiple object regions in one learning image data, all the object regions are input into the CNN for learning. If the processing of steps S304 to S310 has not been performed on all the object regions (step S311; No), the process returns to step S304.

[0094] If the processing of steps S304 to S310 has been performed on all the object regions (step S311; Yes), the control unit 10 determines whether the processing of steps S302 to S311 has been repeated a specified number of times (also referred to as the batch size, for example, 50) for the specified number of learning image data (step S312). If the specified number of times has not been repeated for the specified number of learning image data (step S312; No), the process returns to step S302. The repeated processing of this batch size is called mini-batch.

[0095] If the number of times of repeating the specified number of learning image data (step S312; Yes), the extraction unit 13 updates the weight parameters of the CNN of the recognition unit 15 based on the average value of the errors saved in the storage unit 20 in step S310 so far (step S313). Since it can be considered that the feature amount of the learning image data is extracted inside the CNN by step S313, step S313 is also called the extraction step. Then, the control unit 10 determines whether the processing from step S302 to step S313 has been repeated a specified number of times (for example, the number of times obtained by multiplying the value obtained by dividing the number of learning image data by the batch size by a specified number (also called the number of epochs, for example, 100)) (step S314).

[0096] If the processing from step S302 to step S313 has not been repeated a specified number of times (step S314; No), the process returns to step S302. If the specified number of times has been repeated (step S314; Yes), the learning process ends. In addition, in the learning process of the second embodiment ( Figure 8 ), compared with the learning process of the first embodiment ( Figure 5 ), the determination content and the position of the repetition process are slightly different. This is because, when learning the CNN, if learning is repeatedly performed on the same learning image data within the same mini-batch, overfitting may sometimes occur. However, Figure 8 The flowchart shown is just an example, and in order to improve the learning efficiency and recognition performance, the processing order and processing content can also be changed. For example, in the learning process of the second embodiment ( Figure 8 ), similar to step S108 of the learning process of the first embodiment ( Figure 5 ), the processing from step S305 to step S310 can be repeated a specified number of times. This is because, in step S305, since the parameters of the helix are randomly changed each time, even if the same learning data is repeatedly used, overfitting will not occur, and it may contribute to the improvement of recognition performance.

[0097] The above describes the learning process. Next, with reference to Figure 9 , the recognition process of performing image recognition using the recognition unit 15 (CNN) learned through the above learning process will be described. When the user instructs the recognition device 101 to start the recognition process via the operation input unit 34, the recognition process starts.

[0098] Steps S401 to S405 are the same as the recognition process of the first embodiment ( Figure 6) The processing of steps S201 to S205 is the same, so the description is omitted. Then, the extraction unit 13 resamples the one-dimensional data of the pixel values (RGB, YUV, Lab, etc. according to the color space used) obtained in step S405 by linear interpolation so that the size is the same as the size of the input layer of the CNN of the recognition unit 15 (step S406).

[0099] Next, the extraction unit 13 inputs the resampled one-dimensional data into the CNN of the recognition unit 15 (step S407). Then, the recognition unit 15 obtains the output value of the CNN as the recognition result (step S408). Step S408 is also referred to as the recognition step. Then, the control unit 10 outputs the result recognized by the recognition unit 15 to the output unit 32 (step S409), and the recognition process ends.

[0100] As described above, the recognition process has been described. As described above, the recognition device 101 of the second embodiment only inputs one-dimensional data into the CNN, and the CNN automatically extracts feature amounts. Therefore, different from the recognition device 100, it is not necessary to pre-define feature amounts (calculation methods of each element of the vector constituting the feature amount, etc.).

[0101] In addition, in the recognition device 101, the data input to the CNN is one-dimensional data. Therefore, compared with a CNN that usually processes two-dimensional data, the processing load can be reduced. That is, the recognition device 101 can perform image recognition at high speed only by the recognition device 101 without using external computer resources. Therefore, like the recognition device 100, the recognition device 101 can also be used when cloud processing (computer resources such as servers on the Internet) cannot be used, when multiple images need to be recognized (such as image screening, etc.), when real-time recognition is required, etc.

[0102] In addition, like the recognition device 100, the recognition device 101 can obtain more learning data (one-dimensional data) from one image by changing the parameters of the line (helix) when obtaining one-dimensional data compared with the case of directly learning two-dimensional images. Therefore, the total number of image data prepared for learning can be suppressed to be less.

[0103] In addition, in a normal two-dimensional CNN, in order to make up for the shortage of learning data, transfer learning based on the results obtained by learning using a large general image recognition dataset (such as ImageNet) is mostly used. However, as Figure 7 shown, the CNN of the recognition device 101 is composed of one-dimensional feature maps, filters, windows, etc. Therefore, the number of internal parameters of the CNN is small, and a certain degree of performance can be presented even without transfer learning. That is, the number of learning image data and the learning time can be suppressed.

[0104] In particular, in the image recognition of malignant tumors, it is important to evaluate the distribution of local brightness and color discontinuity. On the other hand, the positional relationship between such local features as considered in a 2D CNN is not very important. Therefore, compared with a normal 2D CNN, the CNN of the recognition device 101 can extract necessary and sufficient information with a smaller number of parameters.

[0105] In addition, although the recognition unit 15 of the recognition device 101 uses a CNN as a DNN, a DNN such as an RNN (Recurrent Neural Network) that can input data of variable length can be used instead of the CNN. In this case, the resampling in step S308 of the learning process ( Figure 8 ) and the resampling in step S406 of the recognition process ( Figure 9 ) may not be performed.

[0106] In addition, in the case of using a DNN such as a CNN to extract feature quantities, the learning process can be performed in the same manner as the above-mentioned learning process ( Figure 8 ). The feature quantity extraction process can also be performed in the same manner as the above-mentioned recognition process ( Figure 9 ) except for steps S408 and S409. In the feature quantity extraction process, instead of step S408, the values of the output layer 118 of the CNN and the feature maps of the intermediate layers of the CNN (preferably the feature map immediately preceding the output layer (the feature map 117 in Figure 7 )) can be extracted as feature quantities.

[0107] Alternatively, the values of the output layer 118 of the CNN can be set as a feature quantity vector, and the feature quantities of the image can be extracted by a method of distance learning using this feature quantity vector. In this case, the feature quantity becomes a feature quantity applicable not only to recognition but also to similarity retrieval.

[0108] (Third Embodiment)

[0109] In the above-described embodiment, regarding the parameters of the helix, they are randomly determined in the learning process and set to typical values in the recognition process. However, it is considered that the parameters of the helix set in this way are in a state where effective parameters and less effective parameters for recognition are mixed. Therefore, a third embodiment for searching for effective parameters for the helix will be described.

[0110] In addition, the parameters of the helix are not parameters determined in the learning process (for example, the feature quantities extracted by the extraction unit 13, the internal weights of the CNN, etc.), and thus are called hyperparameters. In addition, as methods for searching for effective hyperparameters (hyperparameter optimization methods), grid search, random search, Bayesian optimization, etc. are known. In the third embodiment, random search among them is used.

[0111] As Figure 10 shown, the functional structure of the recognition device 102 of the third embodiment is obtained by adding the functions of the activity score acquisition unit 16 and the evaluation unit 17 to the functional structure of the recognition device 100 of the first embodiment or the recognition device 101 of the second embodiment. In addition, as the recognition unit 15, SVM can be used, or CNN can be used.

[0112] In addition, a test data set (a set of test data) including a plurality of test data (a set of image data for evaluation and correct answer labels) for evaluating the recognition accuracy of the recognition unit 15 learned through the learning process is given to the recognition device 102 of the third embodiment. The recognition device 102 uses this test data set to search for parameters effective for recognition from several spiral parameters. Therefore, as this test data set, it is preferable to give a test data set corresponding to the object recognized by the recognition device 102. For example, when the diagnostic object part of the skin is recognized by the recognition device 102, as the test data set, a set of a group of image data of skin diseases and the correct answer labels of this image data is used.

[0113] The activity score acquisition unit 16 includes a two-dimensional CNN into which two-dimensional image data composed of vertical and horizontal directions is input, and uses this two-dimensional CNN to obtain an activation map representing the degree of influence, which represents the magnitude of the influence of each pixel of the input image data on recognition. Then, on the obtained activation map, based on the values (degrees of influence) of the pixels obtained along the curve of the spiral, the activity score of the parameters of the spiral used when generating this spiral is obtained. The activity score of the parameters of the spiral is a value representing the magnitude of the influence of the pixels on the spiral obtained according to the parameters of this spiral on recognition, and is calculated through the activity score calculation process described later.

[0114] The evaluation unit 17 calculates an evaluation score, which is the evaluation value of the parameters of the selected spiral. The evaluation score is a value obtained by evaluating the fitness of the parameters of this spiral in the recognition by the recognition unit 15 based on the recognition accuracy of the recognition unit 15 learned based on the parameters of this spiral (and the activity score obtained by the activity score acquisition unit 16). Therefore, it is considered that if the parameters of the spiral with a high evaluation score are used, the recognition accuracy of the recognition unit 15 becomes high. The evaluation unit 17 functions as an evaluation unit.

[0115] The learning process performed by the recognition device 102 is the same as the learning processes of the first embodiment and the second embodiment ( Figure 5 , Figure 8 ), so the description is omitted.

[0116] In addition, the recognition process performed by the recognition device 102, except for the process of defining the spiral ( Figure 6 step S204,Figure 9 In addition to step S404), it is the same process as the recognition process of the first embodiment and the second embodiment ( Figure 6 , Figure 9 ). In the process of defining the helix in the third embodiment, the determination unit 12 defines the helix using the parameters of the helix selected by the optimal parameter selection process described below.

[0117] Therefore, when the user instructs the recognition device 102 to start the recognition process via the operation input unit 34, the recognition process is started. However, the optimal parameter selection process is started immediately before the process of defining the helix of the recognition process, and the process after the process of defining the helix of the recognition process is executed after the optimal parameter selection process ends.

[0118] In addition, the optimal parameter selection process can also be executed in parallel with the start of the recognition process. Further, before starting the recognition process, for example, if the user instructs the recognition device 102 to start the optimal parameter selection process, the optimal parameter selection process can be executed in advance. In this way, in the process of defining the helix of the recognition process, the determination unit 12 can define the helix using the parameters of the helix selected by the optimal parameter selection process executed in advance, so there is no need to execute the optimal parameter selection process during (or in parallel with) the recognition process.

[0119] Refer to Figure 11 The optimal parameter selection process performed by the recognition device 102 will be described. First, the control unit 10 randomly selects the parameters of the helix (type of helix, number of turns, orientation, angle) (step S501). This process is the same as the process of setting the helix parameters in step S104 of the learning process ( Figure 5 ) of the first embodiment.

[0120] Next, the activity score acquisition unit 16 calculates the activity score Sm of the parameters of the helix selected in step S501 (step S502). The details of this process (activity score calculation process) will be described later.

[0121] Next, the control unit 10 determines whether the activity score Sm calculated in step S502 is less than the threshold value (step S503). If it is less than the threshold value (step S503; yes), the process returns to step S501.

[0122] If it exceeds the threshold value (step S503; no), the control unit 10 performs a learning process (step S504). This learning process is the same as the learning processes of the first embodiment and the second embodiment ( Figure 5 , Figure 8 ). However, in the process of randomly setting the parameters of the helix ( Figure 5 step S104, Figure 8In step S305), the setting unit 14 does not set the parameters of the helix completely randomly, but stays at a level where it slightly changes around the value of the parameters of the helix selected in step S501.

[0123] Specifically, among the parameters of the helix selected in step S501, the type of the helix is not changed. In addition, the other parameters (the number of turns, direction, start angle of the helix) are set within a certain value range centered on the parameters selected in step S501. The parameters selected in step S501 (parameters for optimization) are used to search for the optimal parameters by making various changes to all the parameters of the helix. In contrast, the parameters set in step S104 (or step S305) of the learning process are parameters for simulating an increase in the learning data corresponding to the parameters selected in step S501. Therefore, it is preferable to set parameters that do not change the type of the helix but change the passing location in the recognition area 42.

[0124] Next, the evaluation unit 17 evaluates the parameters of the helix (parameters for evaluation) selected in step S501 based on the recognition accuracy of the recognition unit 15 learned in step S504 (step S505). Specifically, for each test data included in the test data set, the evaluation unit 17 obtains 1D data from the evaluation linear area, which is an area on the helix determined using the above-mentioned parameters of the helix (parameters for evaluation). Then, based on the obtained 1D data, the evaluation unit 17 causes the recognition unit 15 to recognize the diagnostic target part in the test data and calculates the recognition accuracy Sa (number of correct answers / number of test data). Then, the evaluation unit 17 uses the calculated recognition accuracy Sa as the evaluation score S. In addition, when there are multiple diagnostic target parts in the image data of one test data, the evaluation unit 17 may also recognize each of the diagnostic target parts and calculate (number of correct answers / sum of diagnostic target parts of all test data) as the recognition accuracy Sa.

[0125] In addition, the evaluation unit 17 may not only use the recognition accuracy Sa but also use the activity score Sm calculated in step S502 to obtain the evaluation score S by the following formula (1). Here, λ is an arbitrary number set to 0.0 < λ < 1.0 and is set to obtain a weighted average through the recognition accuracy Sa and the activity score Sm.

[0126] S = (1 - λ)Sm + λSa…(1)

[0127] Moreover, the evaluation unit 17 may not only calculate the recognition accuracy Sa and the activity score Sm but also calculate other evaluation values for evaluating the parameters of the helix. Instead of or together with the recognition accuracy Sa and the activity score Sm, the evaluation score S is obtained using the other evaluation values.

[0128] Then, the control unit 10 determines whether the steps S501 to S505 have been repeated a specified number of times (step S506). If the specified number of repetitions has not been reached (step S506; NO), the process returns to step S501. If the specified number of repetitions has been reached (step S506; YES), the control unit 10 selects the parameters of the helix (highest evaluation parameters) when the highest evaluation score S is obtained in step S505 as the optimal parameters (step S507), and ends the process. In step S507, the control unit 10 functions as a parameter selection unit.

[0129] In addition, in the above optimal parameter selection process ( Figure 11 ), when the activity score calculated in step S502 is less than the threshold value, the process immediately returns to step S501 to reselect the parameters of the helix. However, it is also considered that in the case where the activity score is low and it cannot be reliably said that the recognition accuracy Sa has decreased. To guard against such a situation, the process of step S503 can be skipped, and the learning process can be performed regardless of the value of the activity score, and the learning result can be evaluated.

[0130] Next, refer to Figure 12 to describe the activity score calculation process executed in step S502 of the optimal parameter selection process.

[0131] First, the control unit 10 uses the learning image data stored in the storage unit 20 to make the 2D CNN included in the activity score acquisition unit 16 learn (step S551). Specifically, in order to make the value output from the 2D CNN input with the learning image data close to the correct answer label given to the learning image data, the process of correcting the weights inside the 2D CNN is repeated a specified number of times (for example, the number of learning image data), thereby making the 2D CNN learn. Next, the control unit 10 initializes the variable i used to count the test data to 1 (step S552).

[0132] Then, the activity score acquisition unit 16 acquires the activation map Mi of the image data of the i-th test data (step S553). In addition, here, in the case of obtaining a CAM (Class Activation Mapping), it is assumed as the activation map, but other activation maps (for example, Grad-CAM (Gradient-weighted Class Activation Mapping), Guided Grad-CAM, etc.) can also be obtained. However, in any case, each pixel value of the activation map Mi is normalized to be 0 or more and 1 or less. That is, if the pixel has no effect on recognition at all, it is 0, and the greater the degree of influence, the closer the value is to 1.

[0133] Next, the activity score acquisition unit 16 uses the parameters of the helix selected in step S501 of the optimization parameter selection process ( Figure 11 ) to draw a helix on the obtained activation map Mi, obtains pixel values along the helix, and obtains the value Smi obtained by averaging the obtained pixel values (step S554).

[0134] Then, the control unit 10 increments the variable i by 1 (step S555), and determines whether the variable i is greater than the number of test data (step S556). If the variable i is less than or equal to the number of test data (step S556; NO), the process returns to step S553, and steps S553 to S555 are repeated until the variable i becomes greater than the number of test data.

[0135] Then, when the variable i becomes larger than the number of test data (step S556; YES), the activity score Sm is calculated and obtained by averaging the Smi of the previously obtained number of test data (step S557). In addition, the activity score calculation process is ended, and the process proceeds from step S503 of the optimal parameter selection process.

[0136] In addition, in step S551 described above, it is assumed that the 2D CNN is directly learned using the image data of the test data (in the normal 2D CNN learning process), and in step S553 described above, it is assumed that, for the generation of the activation map, one activation map (normal activation map) is obtained for one image data of the test data.

[0137] In this way, instead of directly using the image data of the test data, as in the learning process described in Figure 5 or Figure 8 , the rectangular region including the diagnostic object part is cut out from the image data for 2D CNN learning and for activation map generation. In this case, in step S551 described above, the value obtained by normalizing the size of each cut-out rectangular region to the size of the 2D CNN can be used to make the 2D CNN learn. In addition, in step S554 described above, one activation map can also be obtained for each of the cut-out rectangular regions. When the proportion of the area of the diagnostic object part in the entire image data is small, it is considered that the method of cutting out rectangular regions for 2D CNN learning or obtaining activation maps will improve the calculation accuracy of the activity score. In addition, the activity score can also be calculated separately in the case of directly using the image data and the case of cutting out rectangular regions, and the parameters of the helix can be evaluated based on their average values.

[0138] Through the optimal parameter selection process described above, in the recognition process of the third embodiment, the recognition is performed using the helix defined by the parameters of the helix with the highest evaluation score (highest evaluation parameter), so the recognition accuracy of the recognition device 102 can be improved.

[0139] As an evaluation score, by using the recognition accuracy Sa of the recognition unit 15 with respect to the test data set, it is possible to select the parameters of the helix with a high possibility of improving the recognition accuracy for recognition objects whose properties are close to the test data. In addition, although the calculation of the recognition accuracy Sa takes time, by using the activity score Sm as the evaluation score, it is possible to select the parameters of the helix considered effective for improving the recognition accuracy in a relatively short time.

[0140] In addition, in the above-described optimal parameter selection process, in the hyperparameter optimization method, random search was used to search for parameters, but other methods can also be used. For example, if grid search is used, in the process of step S501, instead of randomly selecting the parameters of the helix, all combinations of all parameters are selected one by one.

[0141] In addition, in the above-described optimal parameter selection process ( Figure 11 ), the evaluation unit 17 calculates both the activity score Sm and the recognition accuracy Sa, but it is not necessarily necessary to calculate both. The evaluation unit 17 can also calculate the value of either the activity score Sm or the recognition accuracy Sa in the optimal parameter selection process, and use the calculated value as the evaluation score S. When the activity score Sm is used as the evaluation score S, the step S504 of the optimal parameter selection process can be skipped, and in step S505, the evaluation unit 17 evaluates the parameters of the helix using the activity score Sm calculated in step S502 as the evaluation score S.

[0142] In addition, the above-described optimal parameter selection process has been described as a process for selecting the optimal parameters as the parameters of the helix during the recognition process. However, it can also be used when the recognition process of the second embodiment is used as the feature extraction process. In this case, the determination unit 12 uses the parameters of the helix selected in the optimal parameter selection process to define the helix in the process of defining the helix ( Figure 9 step S404).

[0143] (Modification example)

[0144] In the above-described embodiment, as the "lines extending in a plurality of mutually different directions" when obtaining one-dimensional data based on pixel values, a spiral curve is adopted. This is because a spiral can make a continuous linear region longer, so it can capture global features, and (corresponding to no breakpoints) can capture more local features. However, it is not necessary to limit the "lines extending in a plurality of mutually different directions" to a spiral curve.

[0145] For example, multiple straight lines radially extending from near the center of the recognition region, multiple straight lines connecting any (random) two points within the recognition region, concentric ellipses (including concentric circles) centered near the center of the recognition region, etc. may also be used. In the case of these lines, there are multiple lines, but preferably, instead of connecting the multiple lines to form a single long line as one-dimensional data, each line is processed as one-dimensional data of a different channel without connecting them (similar to RGB, etc.). This is because it is considered that since these lines are not continuous lines like a single helix, the method of treating only the continuous part as one-dimensional data corresponding to the quantity of one channel can well capture the features existing in each line.

[0146] Since the directions of the local regions on the lines such as the helix, the above-mentioned multiple straight lines radially extending, the random multiple straight lines, and the concentric ellipses are oriented in multiple directions, they are robust against the rotation of the recognition object and have the advantage of being less likely to miss the features existing in the recognition object. In addition, in dermoscopic diagnosis, there are cases where parallel structural features are utilized, but the above-mentioned various lines have multiple directions, thus having the advantage of being able to capture such structural features without missing them. In particular, the helix and the concentric ellipse have the advantage of being able to easily capture the features in the circumferential direction that match the contours of the disease region and other features in the dermoscopic image.

[0147] In the case of using the above-mentioned "lines extending in mutually different multiple directions" (not limited to spiral curves), for example, in the learning process ( Figure 5 ), in step S104 of the learning process, the setting unit 14 randomly sets parameters that define the above-mentioned "lines extending in mutually different multiple directions" (not limited to spiral curves). Then, in step S105, the determination unit 12 uses the parameters set by the setting unit 14 to define "lines extending in mutually different multiple directions" that extend within the recognition region 42 from near the center of the recognition region 42. Then, in step S106, the extraction unit 13 obtains pixel values along the "lines extending in mutually different multiple directions" defined by the determination unit 12. These processes are the same in steps S305 to S307 of the learning process ( Figure 8 ), steps S204 to S205 of the recognition process ( Figure 6 ), and steps S404 to S405 of the recognition process ( Figure 9 ).

[0148] In addition, in the above-mentioned optimal parameter selection process ( Figure 11) In this case, in step S501, the control unit 10 randomly selects the parameters of the spiral, but this is just an example. When using a line other than a spiral curve as the "lines extending in a plurality of different directions", in step S501, the control unit 10 may also randomly select the parameters defining such a line. In this case, in step S554 of the activity score calculation process ( Figure 12 ), the activity score acquisition unit 16 acquires pixel values along the "lines extending in a plurality of different directions" defined by the parameters selected in step S501, and obtains the value Smi obtained by averaging the acquired pixel values. Thus, in step S507, the control unit 10 can also select the parameters that result in the highest evaluation score S as the optimal parameters for parameters other than the spiral.

[0149] In the above-described embodiment, SVM and CNN are used as the models of the recognition unit 15, but a decision tree or the like may be used instead. In addition, in the third embodiment, CAM, Grad-CAM, and Guided Grad-CAM of CNN are illustrated as activation maps, but the activation map used by the activity score acquisition unit 16 is not limited to this. For example, the activity score acquisition unit 16 may acquire the activation map of RNN, or may acquire a Saliency Map as the activation map. When acquiring a Saliency Map as the activation map, each pixel value of the activation map Mi takes a value of 0 or more and 1 or less according to the probability of the line of sight of a person observing the image composed of each pixel value. That is, if the person's line of sight does not face this pixel at all, it is 0, and the higher the probability of the person's line of sight facing, the closer the value is to 1. In addition, when acquiring a Saliency Map as the activation map, when the specified object in the present invention is an object that is easily visible to a person's eyes, such as rust on a structure, a spoiled part of a fruit or vegetable, a commodity identified by an identification device, etc., the effects of the present invention can be obtained more effectively.

[0150] (Modification example of image data)

[0151] In addition, in the above-described embodiments and modifications, the image data acquired by the image input unit 31 is not limited to normal visible light images. As the light irradiated to the diagnostic target portion during imaging, white light (light formed by uniformly mixing lights of each wavelength of visible light), light in a wavelength region including visible light near purple and ultraviolet light (non-visible light) (for example, light having a wavelength of 320 nm to 440 nm, preferably 405 nm), infrared light including near-infrared light (for example, light having a wavelength of 750 nm to 1000 nm, preferably 945 nm), or any light formed by mixing these lights can be used. In addition, when the light receiving device of the image input unit 31 receives light from the diagnostic target portion (light reflected by the diagnostic target portion when the above light is irradiated and fluorescence generated in the diagnostic target portion due to the irradiated light), any one of these multiple lights (visible light, ultraviolet light, infrared light) or any combination of these multiple lights can also be received to acquire image data.

[0152] For example, in an ultraviolet image obtained by receiving ultraviolet light from a diagnostic target portion irradiated with ultraviolet light, skin spots can be observed. In addition, in an infrared image obtained by receiving infrared light from a diagnostic target portion irradiated with infrared light, the appearance of blood vessels can be observed. Therefore, a case where the recognition accuracy is improved by adding these images (ultraviolet image, infrared image, etc.) to a normal visible light image is also considered.

[0153] Specifically, in the ultraviolet image and the infrared image, in the same manner as the processing described in the above embodiments, a helix is defined, and 1D data is obtained by sequentially obtaining pixel values (for example, values representing the intensity of ultraviolet light and infrared light) along the curve of the helix. Then, the obtained 1D data is appended in the channel direction (processed as 1D data of different channels respectively in the same manner as RGB, etc.). Moreover, the learning process and the recognition process using the 1D data of the obtained multiple channels can be performed in the same manner as the learning process ( Figure 8 ) and the recognition process ( Figure 9 ) of the above-described CNN. In addition, the ultraviolet image and the infrared image can be used alone or in combination with other images (visible light image, infrared image, ultraviolet image, etc.).

[0154] In addition, each function of the recognition devices 100, 101, and 102 can also be implemented by a computer such as a general-purpose PC (Personal Computer). Specifically, in the above-described embodiment, the case where the programs for the learning process and the retrieval process performed on the recognition devices 100, 101, and 102 are pre-stored in the ROM of the storage unit 20 has been described. However, the programs can also be stored in a computer-readable recording medium such as a floppy disk, a CD-ROM (Compact Disc Read Only Memory), a DVD (Digital Versatile Disc), an MO (Magneto-Optical disc), a memory card, or a USB (Universal Serial Bus) memory and distributed. The programs are read into the computer and installed, thereby constituting a computer capable of implementing the above-described various functions.

[0155] As described above, the preferred embodiments of the present invention have been described. However, the present invention can be implemented in various embodiments and modifications without departing from the broad spirit and scope of the present invention. In addition, the above-described embodiments are for explaining the present invention and do not limit the scope of the present invention. That is, the scope of the present invention is not represented by the embodiments but by the technical solutions claimed for patent rights. Moreover, various modifications implemented within the scope of the technical solutions claimed for patent rights and within the meaning of equivalent inventions are considered to be within the scope of the present invention.

[0156] This application is based on Japanese Patent Application No. 2019-113990 filed on June 19, 2019, and Japanese Patent Application No. 2020-043778 filed on March 13, 2020. The entire descriptions, technical solutions claimed for patent rights, and drawings of Japanese Patent Application No. 2019-113990 and Japanese Patent Application No. 2020-043778 are incorporated herein by reference.

[0157] Industrial Applicability

[0158] The present invention can be applied to a feature quantity extraction device, a feature quantity extraction method, a recognition device, a recognition method, and a program that can appropriately extract feature quantities while suppressing the load for extracting the feature quantities.

[0159] Explanation of Reference Numerals

[0160] 10... Control unit, 11... Acquisition unit, 12... Determination unit, 13... Extraction unit, 14... Setting unit, 15... Identification unit, 16... Activity score acquisition unit, 17... Evaluation unit, 20... Storage unit, 31... Image input unit, 32... Output unit, 33... Communication unit, 34... Operation input unit, 41... Object area, 41a, 41b... Areas, 42... Identification area, 43... Curve, 100, 101, 102... Identification devices, 111... Input layer, 112, 113, 114, 115, 116, 117... Feature maps, 118... Output layer, 121, 123, 124, 125... Filters, 122, 126... Windows, 127... Fully connected connection.

Claims

1. A feature quantity extraction device, characterized in that, Comprising: An acquisition unit that acquires a captured image obtained by capturing a specified object; A determination unit that determines linear regions extending in a plurality of mutually different directions from a region including the specified object in the captured image acquired by the acquisition unit; An extraction unit that extracts a feature amount based on pixel values from the linear regions determined by the determination unit, The extraction unit acquires the pixel values of the determined linear regions as one-dimensional data, calculates a specified statistical value for the one-dimensional data, and extracts the feature amount, The extraction unit acquires the pixel values of the determined linear regions as one-dimensional data for each of a first component, a second component, and a third component in a specified color space, and extracts at least one of the variances of the one-dimensional data of the first component and the second component, the slopes and contribution rates of the regression lines of the one-dimensional data of the first component, the second component, and the third component, the variance of the absolute values of the differences between the one-dimensional data, the slopes and contribution rates of the regression lines of the ratios of the one-dimensional data to each other, i.e., the first component / the second component and the first component / the third component, and the sum of the differences between the maximum value and the minimum value of the one-dimensional data, a total of 16 values, as the feature amount.

2. The feature amount extraction device according to claim 1, wherein The linear region is a spiral region.

3. The feature amount extraction device according to claim 2, wherein The feature amount extraction device further comprises: a setting unit that randomly sets parameters for specifying the spiral of the spiral region.

4. The feature amount extraction device according to any one of claims 1 to 3, wherein The specified object is a diagnostic object part in a human skin that is the object of diagnosis.

5. The feature amount extraction device according to any one of claims 1 to 3, wherein The extraction unit acquires the pixel values of the determined linear regions as one-dimensional data, and inputs the one-dimensional data into a one-dimensional convolutional neural network to extract the feature amount.

6. The feature amount extraction device according to claim 5, wherein The feature amount extraction device further comprises an evaluation unit that calculates an evaluation value, which is a value obtained by evaluating the fitness of the parameters for determining the linear region, The determination unit selects, from the plurality of parameters, the parameter with the highest evaluation value calculated by the evaluation unit, i.e., the highest evaluation parameter, and uses the selected highest evaluation parameter to determine the linear region.

7. An identification device, characterized in that, Comprising: An acquisition unit that acquires a captured image obtained by capturing a specified object; A determination unit that determines linear regions extending in a plurality of mutually different directions from a region including the specified object in the captured image acquired by the acquisition unit; A one-dimensional data acquisition unit that acquires one-dimensional data based on pixel values from the linear regions determined by the determination unit; An identification unit that identifies the specified object based on the one-dimensional data acquired by the one-dimensional data acquisition unit, The recognition device further includes an evaluation unit that calculates an evaluation value, which is a value obtained by evaluating the fitness of parameters for determining the linear region. The determination unit selects, from among the multiple parameters, the parameter with the highest evaluation value calculated by the evaluation unit, i.e., the highest evaluation parameter, and uses the selected highest evaluation parameter to determine the linear region. The evaluation unit causes the determination unit to select evaluation parameters from among multiple different parameters, and uses the selected evaluation parameters to determine the linear region from a region containing a specified object in the evaluation image data. The evaluation unit causes the one-dimensional data acquisition unit to acquire the one-dimensional data from the evaluation linear region, which is the linear region determined by the determination unit using the evaluation parameters. The evaluation unit causes the recognition unit to recognize the specified object in the evaluation image data based on the one-dimensional data acquired by the one-dimensional data acquisition unit from the evaluation linear region. The evaluation unit calculates the recognition accuracy of the recognition unit based on the one-dimensional data acquired from the evaluation linear region, calculates the evaluation value based on the recognition accuracy, and calculates the recognition accuracy and the evaluation value for each of the multiple parameters.

8. The recognition device according to claim 7, wherein: The linear region is a spiral region.

9. The recognition device according to claim 8, wherein: The determination unit randomly sets parameters for defining the spiral of the spiral region, and uses the spiral defined by the randomly set parameters to determine the spiral region.

10. The recognition device according to claim 7, wherein: The evaluation unit uses the evaluation image data to obtain an activation map representing the degree of influence, which represents the magnitude of the influence of each pixel on recognition, and the evaluation unit calculates the evaluation value based on the degree of influence of each pixel obtained from the linear region on the activation map.

11. The recognition device according to any one of claims 7 to 10, wherein: The specified object is a diagnostic object part in the human skin that is the object of diagnosis.

12. The recognition device according to any one of claims 7 to 10, wherein: The recognition unit recognizes the specified object according to a model that has undergone machine learning in such a way that the input data based on the one-dimensional data is used as input and the recognition result of the specified object is used as output.

13. The recognition device according to claim 12, wherein: The recognition device further includes: an extraction unit that calculates specified statistical values for the one-dimensional data acquired by the one-dimensional data acquisition unit and extracts feature quantities. The recognition unit inputs the feature quantities into a support vector machine as the model and recognizes the specified object.

14. The recognition device according to claim 12, wherein: The recognition unit inputs the one-dimensional data obtained by the one-dimensional data acquisition unit into a one-dimensional convolutional neural network serving as the model to recognize the specified object.

15. A feature quantity extraction method, characterized in that Comprising: An acquisition step of acquiring a captured image obtained by capturing a specified object; A determination step of determining linear regions extending in a plurality of mutually different directions from a region including the specified object in the captured image acquired in the acquisition step; An extraction step of extracting a feature quantity based on pixel values from the linear regions determined in the determination step, In the extraction step, the pixel values of the determined linear regions are acquired as one-dimensional data, a specified statistical value is calculated for the one-dimensional data, and the feature quantity is extracted. In the extraction step, the pixel values of the determined linear regions are acquired as one-dimensional data for each of a first component, a second component, and a third component in a specified color space, and at least one of the variances of the one-dimensional data of the first component and the second component, the slopes and contribution rates of the regression lines of the one-dimensional data of the first component, the second component, and the third component, the variance of the absolute values of the differences between the one-dimensional data, the slopes and contribution rates of the regression lines of the ratios of the one-dimensional data to each other, i.e., the first component / the second component and the first component / the third component, and the sum of the differences between the maximum value and the minimum value of the one-dimensional data, a total of 16 values, is extracted as a feature quantity.

16. A recognition method, characterized in that Comprising: An acquisition step of acquiring a captured image obtained by capturing a specified object; A determination step of determining linear regions extending in a plurality of mutually different directions from a region including the specified object in the captured image acquired in the acquisition step; A one-dimensional data acquisition step of acquiring one-dimensional data based on pixel values from the linear regions determined in the determination step; A recognition step of recognizing the specified object based on the one-dimensional data acquired in the one-dimensional data acquisition step, The recognition method further includes an evaluation step of calculating an evaluation value, which is a value obtained by evaluating the fitness of parameters for determining the linear regions. In the determination step, the parameter with the highest calculated evaluation value, i.e., the highest evaluation parameter, is selected from the plurality of parameters, and the selected highest evaluation parameter is used to determine the linear regions. In the evaluation step, evaluation parameters are selected from a plurality of mutually different parameters, and using the selected evaluation parameters, the linear regions are determined from a region including the specified object in the evaluation image data. In the evaluation step, the one-dimensional data is acquired from the evaluation linear regions, and the evaluation linear regions are the linear regions determined using the evaluation parameters. In the evaluation step, based on the one-dimensional data acquired from the evaluation linear regions, the specified object in the evaluation image data is recognized. In the evaluation step, the recognition accuracy of the recognition step based on the one-dimensional data obtained from the evaluation linear region is calculated, and the evaluation value is calculated based on the recognition accuracy. The recognition accuracy and the evaluation value are calculated for each of the plurality of parameters.

17. A non-transitory storage medium storing a program, characterized in that, The program causes a computer to execute: An acquisition step of acquiring a captured image obtained by capturing a prescribed object; A determination step of determining, from a region including the prescribed object in the captured image acquired in the acquisition step, linear regions extending in a plurality of mutually different directions; An extraction step of extracting a feature amount based on pixel values from the linear regions determined in the determination step; In the extraction step, the pixel values of the determined linear region are acquired as one-dimensional data, a prescribed statistical value is calculated for the one-dimensional data, and the feature amount is extracted. In the extraction step, the pixel values of the determined linear region are acquired as one-dimensional data for each of a first component, a second component, and a third component in a prescribed color space, and at least one of the variances of the one-dimensional data of the first component and the second component, the slopes and contribution rates of the regression lines of the one-dimensional data of the first component, the second component, and the third component, the variance of the absolute values of the differences between the one-dimensional data, the slopes and contribution rates of the regression lines of the ratios of the one-dimensional data, i.e., the first component / the second component and the first component / the third component, and the sum of the differences between the maximum value and the minimum value of the one-dimensional data, a total of 16 values, is extracted as the feature amount.

18. A non-transitory storage medium storing a program, characterized in that, The program causes a computer to execute: An acquisition step of acquiring a captured image obtained by capturing a prescribed object; A determination step of determining, from a region including the prescribed object in the captured image acquired in the acquisition step, linear regions extending in a plurality of mutually different directions; A one-dimensional data acquisition step of acquiring one-dimensional data based on pixel values from the linear regions determined in the determination step; A recognition step of recognizing the prescribed object based on the one-dimensional data acquired in the one-dimensional data acquisition step; The program further causes the computer to execute an evaluation step of calculating an evaluation value, which is a value obtained by evaluating the fitness of a parameter for determining the linear region. In the determination step, the parameter with the highest calculated evaluation value, i.e., the highest evaluation parameter, is selected from the plurality of parameters, and the selected highest evaluation parameter is used to determine the linear region. In the evaluation step, an evaluation parameter is selected from a plurality of mutually different parameters, and the selected evaluation parameter is used to determine the linear region from a region including the prescribed object in the evaluation image data. In the evaluation step, the one-dimensional data is acquired from the evaluation linear region, which is the linear region determined using the evaluation parameter. In the evaluation step, the prescribed object in the evaluation image data is recognized based on the one-dimensional data acquired from the evaluation linear region. In the evaluation step, the recognition accuracy of the recognition step based on the one-dimensional data obtained from the evaluation linear region is calculated, and the evaluation value is calculated based on the recognition accuracy. The recognition accuracy and the evaluation value are calculated for each of the plurality of parameters.

Images