Image processing device, method, portable terminal device and recording medium

Abstract

The present invention provides an image processing device, an image processing method, a program, and a portable terminal device for more accurately extracting a user's attention area from an image with a small amount of operations. The above-mentioned problem is solved by an image processing device that divides an acquired image into multiple areas, selects at least one area from the multiple divided areas, analyzes the correlation between the selected area and areas other than the selected area, and outputs grouping candidates of areas belonging to the same group as the selected area based on the analyzed correlation.

Description

Technical Field

[0001] The present invention relates to an image processing device, an image processing method, a program and a portable terminal device, and in particular to a technology for extracting a focus area from an image. Background Art

[0002] The user desires to extract a desired region of interest from an image. To achieve this desire, Non-Patent Document 1 discloses a technique that uses AI (Artificial Intelligence) to assign meaningful labels to all pixels in an image for segmentation. Also, Non-Patent Document 2 discloses a technique that allows the user to specify a selection range by dragging and inputting in a single stroke.

[0003] Furthermore, Patent Document 1 discloses a technique for selecting a local area from an image for image processing such as enlargement and trimming.

[0004] Previous technical literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2005-078290

[0007] Non-patent literature

[0008] Non-Patent Literature 1: Semantic Image Segmentation Based on DeepLab in TensorFlow <URL:https: / / developers-jp.googleblog.com / 2018 / 04 / semantic-image-segmentation-with.html>

[0009] Non-Patent Document 2: Lasso Tool in Adobe Photoshop<URL:https: / / helpx.adobe.com / jp / photoshop / kb / 227192.html> Summary of the invention

[0010] Technical issues to be solved by the invention

[0011] However, the technology disclosed in Non-Patent Document 1 has a problem of not being able to handle data that was not used when AI was created. Also, the technology disclosed in Non-Patent Document 2 has a problem of requiring a large number of operations when selecting a fine area or when correcting an erroneous selection.

[0012] Furthermore, in the technology described in Patent Document 1, the local area to be output is limited to a rectangular area, and a non-rectangular area cannot be extracted.

[0013] The present invention has been made in view of such circumstances, and an object of the present invention is to provide an image processing device, an image processing method, a program, and a portable terminal device that can more accurately extract a user's attention area from an image with a small amount of operations.

[0014] Means for solving technical problems

[0015] In order to achieve the above-mentioned purpose, one method of an image processing device is an image processing device, which comprises: an image acquisition unit, which acquires an image; a region segmentation unit, which segments the acquired image into multiple regions; a selection unit, which selects at least one region among the multiple segmented regions; an analysis unit, which analyzes the correlation between the selected region and regions other than the selected region; and an output unit, which outputs grouping candidates of regions belonging to the same group as the selected region based on the analyzed correlation.

[0016] According to the present invention, the acquired image is segmented into multiple regions, at least one region among the segmented multiple regions is selected, the correlation between the selected region and regions other than the selected region is analyzed, and grouping candidates of regions belonging to the same group as the selected region are output based on the analyzed correlation, thereby being able to more accurately extract the user's focus area from the image as a grouping candidate with a small amount of operations.

[0017] The output unit preferably outputs a plurality of grouping candidates, thereby being able to more accurately extract the user's attention area.

[0018] The selection unit preferably selects at least one region based on a user input. Furthermore, the selection unit may select at least one region based on a learned machine learning model. Thus, the region of interest can be appropriately extracted.

[0019] Preferably, a processing unit is provided to perform correction processing on the grouping candidates. Thus, the corrected region of interest can be extracted.

[0020] The correction process is preferably a geometric transformation process, thereby making it possible to extract the region of interest that has been geometrically transformed.

[0021] The correction process is preferably a contour correction process, thereby making it possible to extract a region of interest whose contour has been corrected.

[0022] The region dividing unit preferably divides the image into a plurality of regions by clustering based on pixel information, thereby being able to appropriately divide the image into a plurality of regions.

[0023] The analyzing unit preferably analyzes the correlation between the selected region and the region adjacent to the selected region, thereby being able to output appropriate grouping candidates.

[0024] The analyzing unit preferably analyzes the similarity between the selected region and regions other than the selected region, thereby being able to output appropriate grouping candidates.

[0025] The analyzing unit preferably analyzes the correlation based on the learned machine learning model, thereby being able to output appropriate grouping candidates.

[0026] Preferably, a limiting unit is provided, the limiting unit limiting a range from the acquired image, and the area dividing unit dividing the image of the limited range into a plurality of areas. Thus, the region of interest can be extracted from a required range in the image, and the amount of calculation can be reduced.

[0027] In order to achieve the above-mentioned purpose, one method of a portable terminal device is a portable terminal device, which comprises: the image processing device described above; an imaging lens; an imaging element for capturing an optical image formed by the imaging lens; and a display unit, wherein an image acquisition unit acquires image data output from the imaging element as an image, and an output unit displays grouping candidates on the display unit.

[0028] According to the present invention, it is possible to more accurately extract a user's attention area from a captured image with a small amount of operations, and to display the extracted grouping candidates on a display unit.

[0029] In order to achieve the above-mentioned purpose, one mode of an image processing method is an image processing method, which comprises: an image acquisition process, acquiring an image; a region segmentation process, dividing the acquired image into multiple regions; a selection process, selecting at least one region among the multiple segmented regions; an analysis process, analyzing the correlation between the selected region and regions other than the selected region; and an output process, outputting grouping candidates of regions belonging to the same group as the selected region based on the analyzed correlation.

[0030] According to the present invention, it is possible to more accurately extract a user's attention area from an image with a small amount of operations. A program for causing a computer to execute the above-mentioned image processing method is also included in this embodiment.

[0031] Effects of the Invention

[0032] According to the present invention, a user's attention area can be extracted from an image more accurately with a small amount of operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a front perspective view of a tablet PC.

[0034] Figure 2 It is a rear perspective view of a tablet PC.

[0035] Figure 3 This is a block diagram showing the system configuration of a tablet PC.

[0036] Figure 4 This is a block diagram showing the system configuration of the region extraction unit.

[0037] Figure 5 is a flowchart showing an image processing method.

[0038] Figure 6 This is a diagram showing an example of an image within a range limited by a range limiting unit among images acquired by an image acquisition unit.

[0039] Figure 7 This is a diagram showing an example of a plurality of regions divided by a region dividing unit.

[0040] Figure 8 This is an example of the analysis result of the analysis department.

[0041] Fig. 9 This is a diagram showing an example of dividing a character into a plurality of areas.

[0042] Fig.10 This is a diagram showing an example in which each region constitutes one character.

[0043] Fig.11 This is a diagram showing an example in which each region constitutes one object.

[0044] Fig.12 This is a diagram showing an example in which non-adjacent regions constitute one object. DETAILED DESCRIPTION

[0045] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0046] <Deck Computer Terminal>

[0047] The image processing device according to the present embodiment can accurately extract the user's interest area from the image with a small amount of operation. For example, from an image of a logo or a sign, the area of ​​the logo or sign that the user is interested in can be extracted. Also, from an image of a face with face painting, the area of ​​the face painting pattern can be extracted.

[0048] Figure 1 1 is a front perspective view of a tablet terminal 10 (hereinafter referred to as tablet PC 10 . PC is an abbreviation of Personal Computer) which is an example of a portable terminal device to which the image processing device according to the present embodiment is applied. Figure 2 is a rear-view stereogram. Figure 1 and Figure 2 As shown, the tablet PC 10 includes a flat housing 12 having a rectangular outline. The housing 12 includes a touch panel display 14 , operation buttons 16 , a speaker 18 , a built-in camera 20 , and an external connection terminal 24 .

[0049] Figure 3 1 is a block diagram showing the system structure of the tablet PC 10. Figure 3 As shown, the tablet PC 10 includes a CPU (Central Processing Unit) 30 for controlling the overall operation of the tablet PC 10. The CPU 30 is connected to a main memory 34, a nonvolatile memory 36, a mobile communication unit 38, a wireless LAN (Local Area Network) communication unit 40, a short-range wireless communication unit 42, a wired communication unit 44, a touch panel display control unit 46, a key input unit 50, a voice processing unit 52, a camera processing unit 54, and an area extraction unit 60 via a system bus 32.

[0050] The CPU 30 reads an OS (Operating System) stored in the nonvolatile memory 36 , application programs running on the OS, standard data, etc., and expands and executes the operating program in the main memory 34 , thereby functioning as a control unit that controls the overall operation of the computer.

[0051] The main memory 34 is constituted by, for example, a RAM (Random Access Memory), and functions as a working memory of the CPU 30 .

[0052] The nonvolatile memory 36 is composed of, for example, an Electrically Erasable Programmable Read Only Memory (EEPROM) flash memory and stores an OS, application programs, and various standard data. The nonvolatile memory 36 functions as a storage unit of the tablet PC 10 and stores various data.

[0053] The mobile communication unit 38 transmits and receives data to and from the nearest wireless base station (not shown) via the antenna 38A using a 4th generation mobile communication system (4G: 4th Generation) or a 5th generation mobile communication system (5G: 5th Generation).

[0054] The wireless LAN communication unit 40 performs wireless LAN communication with a wireless LAN access point or an external device capable of wireless LAN communication via the antenna 40A in accordance with wireless LAN communication standards such as IEEE (Institute of Electrical and Electronics Engineers) 802.11a / b / g / n.

[0055] The short-distance wireless communication unit 42 communicates with other Bluetooth (registered trade name) standard devices within a Class 2 range (within a radius of approximately 10 m) via the antenna 42A.

[0056] The wired communication unit 44 communicates with an external device connected by a cable (not shown) via the external connection terminal 24 in accordance with a predetermined communication standard.

[0057] The touch panel display 14 (an example of a display unit) includes a display unit 14A such as a color LCD (Liquid Crystal Display) panel that displays images, etc., and a touch panel unit 14B such as a transparent electrode that is disposed on the front surface of the color LCD panel and receives touch input.

[0058] The touch panel unit 14B is, for example, an electrostatic capacitive touch panel, which is made of glass or plastic, and includes: a substrate body having light transmittance; position detection electrodes provided in a planar shape on the substrate body and having light transmittance; and an insulating layer provided on the position detection electrodes. The touch panel unit 14B is arranged on the front surface of the display unit 14A, and generates and outputs two-dimensional position coordinate information corresponding to the user's touch operation.

[0059] The touch panel display control unit 46 controls the touch panel display 14. The touch panel display control unit 46 associates the coordinates of an image displayed on the display unit 14A with the coordinates of a touch operation on the touch panel unit 14B.

[0060] In the following description, the display of the display portion 14A and the touch operation on the touch panel portion 14B are simply described as the display of the touch panel display 14 and the operation on the touch panel display 14 .

[0061] The key input unit 50 is composed of a drive circuit that drives the plurality of operation buttons 16 .

[0062] The audio processing unit 52 converts the digital audio data supplied via the system bus 32 into analog audio data. The analog audio signal is outputted from the speaker 18 .

[0063] The built-in camera 20 includes an imaging lens 20A and an imaging element 20B. The imaging lens 20A forms an optical image of an incident subject. The imaging element 20B captures the formed optical image and outputs an analog image signal.

[0064] The image processing unit 54 digitizes the analog image signal output from the built-in camera 20 (imaging element 20B), performs necessary signal processing on the digitized analog image signal, and outputs the digitized image signal.

[0065] The region extraction unit 60 is an image processing device that extracts a user's attention region from an image displayed on the touch-panel display 14 .

[0066] Figure 4 is a block diagram showing the system structure of the region extraction unit 60. Figure 4 As shown, the region extraction unit 60 includes an image acquisition unit 62 , a region segmentation unit 66 , a selection unit 68 , an analysis unit 70 , and an output unit 72 .

[0067] Touch panel display control unit 46 (refer to Figure 3 ) acquires, for example, a captured image (an example of image data output from an imaging element) captured by the built-in camera 20 and displays it on the touch panel display 14. The touch panel display control unit 46 may acquire an image read from the nonvolatile memory 36 and display it on the touch panel display 14, or may display an image acquired via the mobile communication unit 38, the wireless LAN communication unit 40, the short-range wireless communication unit 42, and the wired communication unit 44 on the touch panel display 14. The image acquisition unit 62 acquires images displayed on these touch panel displays 14. The images acquired by the image acquisition unit 62 may also be animations.

[0068] The image acquisition unit 62 includes a range limiting unit 64. The range limiting unit 64 limits the range of the image acquired by the image acquisition unit 62, for example, based on user input. The user inputs the range limiting unit 64 by performing a touch operation such as surrounding a desired range of the image using the touch panel display 14.

[0069] As preprocessing, the region segmentation unit 66 segments the image acquired by the image acquisition unit 62 into a plurality of regions. When the image range is limited by the range limiting unit 64, the region segmentation unit 66 divides the range limited by the range limiting unit 64 into a plurality of regions in the image acquired by the image acquisition unit 62. Thus, compared with the case where the entire image is divided into a plurality of regions, the calculation of extracting the region of interest can be reduced.

[0070] The region segmentation unit 66 performs clustering based on pixel information, for example, thereby segmenting the image into a plurality of regions. Thus, the image can be appropriately segmented into a plurality of regions. Furthermore, the region segmentation unit 66 can apply SLIC (Simple Linear Iterative Clustering) to segment into a plurality of regions, for example. SLIC is an algorithm that takes an image as input and segments it into superpixels as output, wherein the superpixel captures pixels that are close in distance and color as a group. Superpixels are pre-calculated as a preprocessing for image recognition, thereby reducing the amount of information in the image and making it easier to apply other image recognition algorithms.

[0071] The selection unit 68 selects at least one region from the divided regions. The selection unit 68 selects the region based on, for example, a user input. The user input to the selection unit 68 is performed by the user performing a touch operation such as tracking a desired region of the image or enclosing a desired region using the touch panel display 14. By allowing the user to select a region, the region of interest can be appropriately extracted.

[0072] Furthermore, the selection unit 68 may select the region based on the learned machine learning model. In this case, the segmentation result of the region segmentation unit 66 is input to the learned machine learning model, the user's usual attention region is estimated in the learned machine learning model, and the estimated attention region is set as the output of the selection unit 68. By selecting the region based on the learned machine learning model, the user's operations can be reduced and the attention region can be appropriately extracted.

[0073] The machine learning model that has completed learning can also add the area of ​​interest selected by the learning user as learning data.

[0074] The analysis unit 70 analyzes the association (identity or similarity) between the region selected by the selection unit 68 and the region other than the selected region. The analysis performed by the analysis unit 70 includes analyzing the similarity of each region based on the similarity of the pixel values ​​such as the color and brightness of the pixels included in each region. In addition, the maximum similarity means that the two are the same region. By analyzing the similarity, it is possible to output an appropriate grouping candidate.

[0075] Furthermore, the analysis performed by the analysis unit 70 is not limited to the analysis of similarity. For example, it includes the following analysis: when each area is combined, whether there is a meaningful relationship as an object, a character, or a character string. In this case, the analysis unit 70 appropriately uses the machine learning model that has been learned.

[0076] The output unit 72 outputs grouping candidates of regions belonging to the same group as the region selected by the selection unit 68, based on the correlation analyzed by the analysis unit 70. The output unit 72 displays, for example, the grouping candidate with the highest confidence on the touch panel display 14 via the touch panel display control unit 46. When there are a plurality of grouping candidates, the display of each different grouping candidate may be switched according to a user operation on the touch panel display 14.

[0077] Furthermore, the output unit 72 may sort the plurality of grouping candidates according to the degree of certainty and display the sorted grouping candidates in a list on the touch-panel display 14 .

[0078] The output unit 72 can apply the machine learning model that has been learned. In addition, the machine learning models that have been learned by the selection unit 68, the analysis unit 70, and the output unit 72 can also be learned simultaneously. In addition, the machine learning model that has been learned can be applied to different learning data sets for further additional learning. That is, the machine learning model that has been learned is not limited to the model that has been learned, but includes a model that can be updated later.

[0079] Furthermore, the output unit 72 includes a correction processing unit 74. The correction processing unit 74 performs correction processing on the grouping candidates. The correction processing is, for example, a geometric transformation processing such as distortion correction. Furthermore, the correction processing may be a contour correction processing. The output unit 72 outputs the grouping candidates that have been subjected to the correction processing by the correction processing unit 74. In this way, the output unit 72 can output the grouping candidates that have been subjected to the geometric transformation processing or the contour correction processing.

[0080] <Image processing method>

[0081] Figure 5 1 is a flowchart showing an image processing method based on the tablet PC 10. Figure 5 As shown, the image processing method includes an image acquisition process (step S1), a region segmentation process (step S2), a region selection process (step S3), a correlation analysis process (step S4) and an output process (step S5).

[0082] In step S1, the image acquisition unit 62 acquires an image displayed on the touch-panel display 14. Here, the range limiting unit 64 limits the range surrounded by the touch operation of the touch-panel display 14 by the user in the image displayed on the touch-panel display 14.

[0083] Figure 6 FIG. 6 is a diagram showing an example of a range defined by the range limiting unit 64 in an image acquired by the image acquisition unit 62. Figure 6 As shown, the region R1 is a region defined by the range defining section 64 .

[0084] In step S2, the area dividing unit 66 divides the image acquired by the image acquisition unit 62 into a plurality of areas. Here, the area dividing unit 66 divides the range defined by the range defining unit 64 in the image acquired by the image acquisition unit 62 into a plurality of areas.

[0085] Figure 7 FIG. 4 is a diagram showing an example of a plurality of regions divided by the region dividing unit 66. Figure 7 As shown, Figure 6 The region R1 shown is divided into a plurality of regions R11 to R19.

[0086] In step S3, the selection unit 68 selects at least one area from the divided multiple areas. Here, for example, in the touch panel display control unit 46, a content urging the user to select an area is displayed on the touch panel display 14. In contrast, the user surrounds the desired area by touching the touch panel display 14. The selection unit 68 selects the area surrounded by the user's touch operation from the multiple areas displayed on the touch panel display 14. For example, Figure 7 The region R13 is selected from among the plurality of regions R11 to R19 shown.

[0087] In step S4, the analysis unit 70 analyzes the correlation between the area selected in step S3 and the areas other than the selected area. For example, the analysis unit 70 analyzes the similarity of the pixel values ​​such as the color and brightness of the pixels included in each area for the selected area R13 and all areas R11, R12, R14 to R19 other than area R13. In addition, the correlation of the distance between the areas can be analyzed. Here, the distance can be the distance in pixel units or the distance when converted to actual size. It can be considered that the closer the distance between each other, the higher the correlation.

[0088] Furthermore, the analysis unit 70 may analyze the correlation between the selected region R13 and the region R14 adjacent to the region R13. Here, when it is determined that the correlation between the region R13 and the region R14 is high, the analysis unit 70 analyzes the correlation between the region R13 and the region R15 that is more adjacent to the region R14. In this way, when the adjacent region is a region with a high correlation, the analysis unit 70 may sequentially expand the adjacent regions and analyze the correlation. By analyzing the correlation of the adjacent regions, it is possible to output appropriate grouping candidates.

[0089] In step S5, the output unit 72 outputs grouping candidates of the region belonging to the same group as the region selected in step S3 based on the correlation analyzed in step S4. In the correction processing unit 74, geometric transformation processing or contour correction processing can be implemented. Here, the output unit 72 sorts the two grouping candidates according to the confidence level and displays them on the touch panel display 14.

[0090] Figure 8 This is an example of a grouping candidate based on the analysis result of the analysis unit 70. In the grouping candidate G1, the region R13 selected in step S3 is grouped into the region R22 which is the same as the regions R12, R14, and R15. Here, the region R11 is grouped into the region R21 alone, and the regions R16 to R19 are grouped into the region R23.

[0091] On the other hand, in the grouping candidate G2, the region R13 selected in step S3 is grouped into the same region R32 as the regions R12, R14 to R19. Also, the region R11 is grouped into the region R31 alone.

[0092] The output unit 72 causes the grouping candidates G1 and G2 to be displayed on the touch panel display 14, respectively. As the grouping candidates related to the region R13 selected in step S3, only the region R22 and the region R32 may be displayed.

[0093] Among the multiple grouping candidates displayed on the touch panel display 14, the user selects a desired grouping candidate by a touch operation on the touch panel display 14, thereby determining the selected grouping candidate as the user's area of interest. The output unit 72 displays the determined area of interest on the touch panel display 14 and stores it in the non-volatile memory 36, and ends the processing of this flowchart. The output unit 72 may store all the grouping candidates in the non-volatile memory 36.

[0094] In this way, by displaying multiple grouping candidates and allowing the user to make a selection, the user's area of interest can be extracted more accurately.

[0095] In Figure 8 the example shown, the analysis unit 70 analyzes the similarity of pixel values such as the color and brightness of pixels as the relevance, but the relevance analyzed by the analysis unit 70 is not limited to the similarity.

[0096] For example, when each region constitutes a part of a character, the relevance analyzed by the analysis unit 70 includes whether they are combined to form a single character.

[0097] Fig. 9 is a diagram showing an example of a character divided into multiple regions. In Fig. 9 the case shown, the analysis unit 70 analyzes the relevance between the "亻" in the region R41 and the "尹" in the region R42, and determines that they form a single character "伊". Accordingly, the output unit 72 groups the regions R41 and R42 into a single region R43 and outputs it.

[0098] Similarly, the analysis unit 70 analyzes the relevance between the "·" in the region R44 and the "|" in the region R45, and determines that they form a single character "i". Accordingly, the output unit 72 groups the regions R44 and R45 into a single region R46 and outputs it.

[0099] Also, when each region constitutes one or more characters, the relevance analyzed by the analysis unit 70 includes whether they are combined to correspond to any one of a word, an abbreviation, and a trade name.

[0100] Fig.10: is a diagram showing an example of each region constituting one character. Fig.10 In the case shown, the analysis unit 70 analyzes the relationship between "J" of region R51 and "R" of region R52, and determines that they constitute "JR" as (a part of) one business name. Therefore, the output unit 72 groups region R51 and region R52 into one region R53 and outputs it.

[0101] Similarly, the analysis unit 70 Figure 7 When the regions R12 to R19 shown in the figure are respectively recognized as regions composed of characters "F", "U", "J", "I", "F", "I", "L", and "M", the correlation between these regions is analyzed, and it can be determined that they are regions constituting the word "FUJI" and regions constituting the word "FILM". Therefore, the output unit 72 groups these regions into "FUJI" and "FILM" and outputs them.

[0102] In this case, the analysis unit 70 further analyzes the relationship between the area forming the word "FUJI" and the area forming the word "FILM", and can determine that they form the business name "FUJIFILM". In this case, the output unit 72 groups these areas into "FUJIFILM" and outputs them.

[0103] Furthermore, the relevance analyzed by the analyzing unit 70 includes the relevance of whether or not each region forms an object when they are respectively combined.

[0104] Fig.11 is a diagram showing an example of each region constituting one object. Fig.11 In the case shown, the analysis unit 70 analyzes the correlation between the metal part region R61 and region R62, the resin part region R63, the plastic part region R64, the glass part region R65 and region R66, and the rubber part region R67 and region R68, and can determine that the region R61 to region R68 have a correlation to constitute the automobile. Therefore, the output unit 72 groups the region R61 to region R68 into one region R70 and outputs it.

[0105] In the above description, the analysis unit 70 has analyzed the correlation between the selected region and the region adjacent to the selected region, but the region for which the correlation is analyzed is not limited to the adjacent region.

[0106] Fig.12 1 is a diagram showing an example of a range defined by the range defining unit 64, and is a diagram showing an example in which non-adjacent regions constitute one object. Fig.12 In the example shown, the image is divided into regions in a grid shape, and regions R81 to R88 having highly correlated pixel value similarities are not adjacent to each other but are dispersed.

[0107] Here, the region R81 is selected by the selection unit 68. The analysis unit 70 analyzes the relevance not only for the region adjacent to the selected region R81 but also for the nearby region. The nearby region is a region existing within a certain distance range.

[0108] As a result, the analysis unit 70 determines that the correlation between region R81 and regions R82 and R88 is high in the region near region R81. In this case, the analysis unit 70 analyzes the correlation between region R81 and regions closer to region R82 and region R88. As a result, the analysis unit 70 determines that the correlation between region R81 and regions R83 and R87 is high. In this way, when the nearby region is a highly correlated region, the analysis unit 70 sequentially expands the nearby region and analyzes the correlation, thereby Fig.12 As shown, the output unit 72 can group the regions R81 to R88 into one region R90 and output it.

[0109] Output unit 72 may set only regions R81 to R88 as region R90 or include regions between regions other than regions R81 to R88 as region R90. When only regions R81 to R88 are set as region R90, the regions between regions may be transparent regions.

[0110] As described above, according to the present embodiment, when a user wants to extract a region of interest from an image, the extraction can be performed more accurately with a small amount of operation.

[0111] <Others>

[0112] Here, a tablet terminal is described as an example of a portable terminal device to which the image processing device is applied, but the image processing device may also be applied to mobile communication terminals such as smartphones and mobile phones, notebook computers, and the like.

[0113] The above-mentioned image processing method may be configured as a program for causing a computer to realize each step, and may be configured as a non-transitory recording medium such as a CD-ROM (Compact Disk-Read Only Memory) storing the program.

[0114] In the embodiment described here, the hardware structure of the processing unit that performs various processes of the region extraction unit 60 is various processors as shown below. The various processors include: a general-purpose processor that executes software (program) and functions as various processing units, namely a CPU (Central Processing Unit), a processor that specializes in image processing, namely a GPU (Graphics Processing Unit), a processor that can change the circuit structure after manufacturing, such as a FPGA (Field Programmable Gate Array), a processor that has a circuit structure specially designed for executing specific processes, such as a programmable logic device (PLD), and an ASIC (Application Specific Integrated Circuit), which is a dedicated circuit.

[0115] A processing unit can be composed of one of these various processors, or it can be composed of two or more processors of the same type or different types (for example, a combination of multiple FPGAs or a CPU and FPGA, or a combination of a CPU and a GPU). In addition, multiple processing units can be composed of one processor. As an example of a plurality of processing units composed of one processor, first, there is the following method: represented by a computer such as a server or a client, a processor is composed of a combination of one or more CPUs and software, and the processor functions as multiple processing units. Second, there is the following method: represented by a system on chip (System on Chip: SoC) and the like, a processor that uses an IC (Integrated Circuit: integrated circuit) chip to implement the functions of the entire system including multiple processing units. In this way, as a hardware structure, various processing units are composed of one or more various processors.

[0116] Furthermore, more specifically, the hardware structure of these various processors is a circuit (Circuitry) in which circuit elements such as semiconductor elements are combined.

[0117] The technical scope of the present invention is not limited to the scope described in the above-mentioned embodiment. The configurations and the like in each embodiment can be appropriately combined between the embodiments within the scope not departing from the gist of the present invention.

[0118] Explanation of symbols

[0119] 10-tablet terminal (tablet PC), 12-frame, 14-touch panel display, 14A-display unit, 14B-touch panel unit, 16-operation button, 18-speaker, 20-built-in camera, 20A-imaging lens, 20B-imaging element, 24-external connection terminal, 32-system bus, 34-main memory, 36-nonvolatile memory, 38-mobile communication unit, 38A-antenna, 40-wireless LAN communication unit, 40A-antenna, 42-near field wireless communication unit, 42A-antenna, 44-wired communication unit, 46-touch panel display control unit, 50-key input Input unit, 52-speech processing unit, 54-camera processing unit, 60-region extraction unit, 62-image acquisition unit, 64-range limitation unit, 66-region segmentation unit, 68-selection unit, 70-analysis unit, 72-output unit, 74-correction processing unit, G1, G2-grouping candidates, R1-region, R11~R19-region, R21~R23-region, R31, R32-region, R41~R46-region, R51~R53-region, R61~R68-region, R70-region, R81~R88-region, R90-region, S1~S5-each step of the image processing method.

Claims

1. An image processing device comprising: An image acquisition unit, for acquiring images; A region segmentation unit, which segments the acquired image into a plurality of regions; A selection unit, selecting at least one area among the plurality of divided areas; an analyzing unit, which analyzes the correlation between the selected area and areas other than the selected area; and an output unit that outputs grouping candidates of regions that belong to the same group as the selected region based on the analyzed correlation, The selecting unit selects the at least one area according to a user input, The output unit outputs only the area including the selected area as the same group as the group candidate.

2. The image processing device according to claim 1, wherein: The output unit outputs a plurality of grouping candidates.

3. The image processing device according to claim 1 or 2, wherein: The selection unit selects the at least one region based on a machine learning model that outputs a learned at least one region among the input multiple regions when the divided multiple regions are input.

4. The image processing device according to claim 1 or 2, wherein: The image processing device includes a processing unit that performs correction processing on the grouping candidates.

5. The image processing device according to claim 4, wherein: The correction process is a geometric transformation process.

6. The image processing device according to claim 4, wherein: The correction process is a contour line modification process.

7. The image processing device according to claim 1 or 2, wherein: The region dividing unit divides the image into the plurality of regions by clustering the image based on pixel information.

8. The image processing device according to claim 1 or 2, wherein: The analyzing unit analyzes a correlation between the selected region and a region adjacent to the selected region.

9. The image processing device according to claim 1 or 2, wherein: The analyzing section analyzes a similarity between the selected region and a region other than the selected region.

10. The image processing device according to claim 1 or 2, wherein: The analysis unit analyzes the association based on a learned machine learning model that analyzes and outputs the similarity of each area when the selected area and areas other than the selected area are input, or a learned machine learning model that analyzes and outputs whether there is a meaningful relationship as objects, characters or character strings when the areas are combined when the selected area and areas other than the selected area are input.

11. The image processing device according to claim 10, wherein: When there are a plurality of grouping candidates, the output unit determines a grouping candidate from among the plurality of grouping candidates according to a user's selection, The analyzing unit analyzes the correlation according to the machine learning model, and the machine learning model performs additional learning using the correlation satisfied by the determined selected region of the grouping candidate and a region other than the selected region.

12. The image processing device according to claim 1 or 2, wherein: The image processing device includes a limiting unit that limits a range from the acquired image. The region dividing unit divides the image in the limited range into the plurality of regions.

13. A portable terminal device comprising: The image processing device according to any one of claims 1 to 12; Imaging lens; An imaging element for capturing an optical image formed by the imaging lens; and Display unit, The image acquisition section acquires image data output from the imaging element as an image, The output unit displays the grouping candidates on the display unit.

14. An image processing method, comprising: An image acquisition process, acquiring an image; A region segmentation step of segmenting the acquired image into a plurality of regions; A selection step of selecting at least one region among the divided regions; An analyzing step of analyzing the correlation between the selected region and regions other than the selected region; and an output step of outputting grouping candidates of regions belonging to the same group as the selected region based on the analyzed correlation, In the selection step, the at least one area is selected according to a user input, In the output step, only the regions including the selected regions as the same group are output as the grouping candidates.

15. The image processing method according to claim 14, further comprising: a determining step of determining a grouping candidate from among the plurality of grouping candidates according to a user's selection when there are a plurality of grouping candidates; In the analysis step, the correlation is analyzed based on a machine learning model. The image processing method further includes: an additional learning step of causing the machine learning model to perform additional learning using the correlation satisfied between the determined selected region of the grouping candidate and a region other than the selected region.

16. A recording medium which is non-transitory and computer-readable, wherein: When the instructions stored in the recording medium are read by a computer, the computer is caused to execute the image processing method according to claim 14 or 15.

Images