Image processing device, image processing method, and program

Abstract

This invention provides an image processing device, an image processing method, and a program that can more accurately match images to be compared. [Solution] The processor 100 acquires a first image and a second image, each containing multiple sub-image regions, associates the multiple sub-image regions in the first image with the multiple sub-image regions in the second image, and accepts user modifications to the association between the sub-image regions in the first image and the sub-image regions in the second image. Using the user-modified association as a constraint, the processor 100 re-associates the multiple sub-image regions in the first image other than those related to the constraint with the multiple sub-image regions in the second image other than those related to the constraint.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and a program.

Background Art

[0002] For comparison between medical images, an apparatus for performing alignment is known (for example, Patent Document 1). When a user designates several points of interest in two three-dimensional medical images, the medical image display apparatus described in Patent Document 1 determines coordinate conversion parameters based on the position information of the corresponding points and displays the corresponding images.

Prior Art Documents

Patent Documents

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, when the orientations and sizes of the images to be compared are different from each other, it is difficult to perform accurate association.

[0005] The present invention has been made in view of the above circumstances, and an object thereof is to perform more accurate association of the images to be compared. '

Means for Solving the Problems

[0006] To achieve the above objective, one aspect of the image processing apparatus according to the present invention includes a processor that acquires a first image and a second image, each containing a plurality of partial image regions, associates the plurality of partial image regions contained in the first image with the plurality of partial image regions contained in the second image, and accepts user modifications to the association between the plurality of partial image regions contained in the first image and the plurality of partial image regions contained in the second image. The processor uses the user-modified association as a constraint and re-associates the plurality of partial image regions in the first image other than the partial image regions related to the constraint with the plurality of partial image regions in the second image other than the partial image regions related to the constraint. [Effects of the Invention]

[0007] According to the present invention, it becomes possible to more accurately match images to be compared. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing an image processing apparatus according to an embodiment of the present invention. [Figure 2] This figure shows an example of the display of the detection screen of an image processing device. [Figure 3] This figure shows an example of a part of the detection screen. [Figure 4] This figure shows an example of the display screen for mapping in an image processing device. [Figure 5] This figure shows an example of a part of the display screen used for mapping. [Figure 6] This figure shows an example of a part of the display screen used for mapping. [Figure 7] This figure shows an example of the display screen for mapping in an image processing device. [Figure 8] This figure shows an example of the display screen for calculating the score of an image processing device. [Figure 9] This is a flowchart for image processing. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals.

[0010] The image processing apparatus 1 according to an embodiment of the present invention is a device that displays two images, each containing a plurality of partial image regions in which an object of interest appears, in a comparable manner. As shown in Figure 1, the image processing apparatus 1 comprises a processor 100 that controls the display of an image captured by a camera 10, a storage unit 110 that stores various data including the image, a display unit 120 that displays the image, and an operation unit 130 that receives user input.

[0011] The first and second images displayed by the image processing device 1 are images of the same subject captured at different times, and each image contains a group of partial image regions consisting of multiple partial image regions in which a subject of interest that changes over time appears. For example, the first and second images are medical images of human or animal skin, and each partial image region is a region in which a candidate skin lesion of the human or animal appears. Skin lesions include, for example, pigmented nevi (moles), malignant melanoma, etc. In the display example of the image processing device 1 in Figure 2, the first medical image 510 (first image) on the left is a current image of a person's back, and the second medical image 520 (second image) on the right is an image of the same person's back taken in the past. More specifically, the current first medical image 510 shows numerous candidate skin lesions of interest, and there are multiple partial image regions enclosed by rectangular frames surrounding each candidate lesion, and the first group of partial image regions 511 consisting of these partial image regions is included in the first medical image 510. Numerous candidate skin lesions of interest also appear in the past second medical image 520, and there are multiple partial image regions enclosed by rectangular frames surrounding each candidate lesion. The second partial image region group 521, consisting of these partial image regions, is included in the second medical image 520. By comparing the partial image regions of the first medical image 510 and the second medical image 520 in detail, it is possible to observe the changes in size, color, or shape of the candidate lesions over time.

[0012] Camera 10 is, for example, a digital camera, and transmits captured image data to the image processing device 1 via any communication means. It is not necessary to precisely match the shooting range and orientation of camera 10 for the first medical image 510 and the second medical image 520, but it is preferable that the first medical image 510 and the second medical image 520 are captured with approximately the same shooting range and orientation. This reduces the processing load, such as the matching of partial image regions.

[0013] The processor 100 of the image processing device 1 includes, for example, a CPU (Central Processing Unit) and its peripheral circuits, and performs various arithmetic operations. The processor 100 may consist of a single CPU or multiple CPUs. The processor 100 executes programs for various arithmetic operations, including image processing, stored in the storage unit 110. The processor 100 may also include volatile semiconductor memory such as RAM (Random Access Memory) that functions as the CPU's working memory. Furthermore, the processor 100 may also include arithmetic circuits such as a logical operation unit and a numerical operation unit.

[0014] The storage unit 110 includes, for example, a non-volatile semiconductor memory such as an EEPROM (Electrically Erasable and Programmable Read Only Memory) or flash memory. The storage unit 110 stores medical images captured by the camera 10, programs for various calculation processes including image processing performed by the processor 100, and various data used in the calculation processes of the processor 100.

[0015] The display unit 120 is an arbitrary display device that displays the medical image and the operation screen sent from the image processing apparatus 1, and for example, is a liquid crystal display or an organic EL (Electro-Luminescence) display. As shown in FIG. 2, the display unit 120 displays a detection screen 500 including two image frames arranged side by side on the left and right for displaying the first medical image 510 and the second medical image 520, and user interface elements (hereinafter referred to as UI elements) such as operation buttons and operation bars arranged around the first medical image 510 and the second medical image 520. The operation unit 130 includes a pointing device such as a mouse, a touch pad, or a touch panel, detects operations on the first medical image 510, the second medical image 520, and the UI elements displayed on the display unit 120, and sends an operation signal to the processor 100. The display unit 120 and the operation unit 130 may be an integrated touch panel display or the like.

[0016] The processor 100 functions as an image acquisition unit 101, a user interface 102, a target detection unit 103, a matching unit 104, a coordinate conversion unit 105, and a score calculation unit 106 by executing an image processing program stored in the storage unit 110. The image acquisition unit 101 acquires the medical image captured by the camera 10, stores it in the storage unit 110, and displays it on the display unit 120 via the user interface 102. The user interface 102 displays two image frames arranged side by side on the left and right for displaying the two first medical images 510 and the second medical image 520 acquired by the image acquisition unit 101, and UI elements such as operation buttons and operation bars arranged around the image frames. The user interface 102 acquires operation inputs by the operation unit 130 performed on the first medical image 510, the second medical image 520, and the UI elements, and outputs them to the matching unit 104 and the coordinate conversion unit 105.

[0017] As shown in Figure 2, the object detection unit 103 searches for and detects objects of interest in the first medical image 510 and the second medical image 520, respectively, and obtains the center coordinates of each object of interest. The method for detecting objects of interest can be any method, for example, a method using binarization and connected component analysis may be used. Alternatively, objects of interest may be detected by machine learning. In this case, the object detection unit 103 may obtain the size or color intensity value of the object of interest. As the size of the object of interest, for example, the diameter of the circle corresponding to the area of ​​the lesion candidate range in which the lesion candidate appears may be obtained, the diameter of the circumscribed circle of the lesion candidate range, or the maximum diameter of the lesion may be obtained. Also, as the color intensity, for example, the difference in brightness inside and outside the lesion candidate range may be obtained. The object detection unit 103 stores the ID number, which is the identification number of the detected object of interest, and the center coordinates in the storage unit 110. In the example shown in Figure 2, the objects of interest detected by the object detection unit 103 are candidate lesion areas on human skin. The object of interest detection unit 103 generates and displays a square frame surrounding each object of interest and assigns an ID number to each object of interest. The area enclosed by the square frame is a partial image area, and multiple partial image areas are displayed in the first medical image 510 and the second medical image 520 in a way that allows them to be identified by their ID numbers. The partial image areas included in the first group of partial image areas 511 of the first medical image 510 are enlarged and displayed vertically in the first partial image display area 512, and the partial image areas included in the second group of partial image areas 521 of the second medical image 520 are enlarged and displayed vertically in the second partial image display area 522. As shown in Figure 2 and Figure 3, which is an enlargement of the central part of Figure 2, checkboxes are provided next to the ID numbers of each partial image area in the first partial image display area 512 and the second partial image display area 522, and are used to select the partial image area to be used for correspondence. The partial image region selected by the checkbox is highlighted in the first medical image 510 and the second medical image 520. For example, the border color of the partial image region selected by the checkbox may be different from the border color of the partial image region not selected by the checkbox.

[0018] The UI elements displayed on the display unit 120 include operation buttons 531 for switching operations on the first medical image 510 or the second medical image 520 or partial image regions included therein, as shown in FIG. 3. The operation buttons 531 are used to switch the functions of the pointing device of the operation unit 130. For example, when the button 5311 is selected, the movement, enlargement or reduction of the first medical image 510 or the second medical image 520 can be executed by operating the pointing device. Also, when the button 5312 is selected, the target of interest can be selected by surrounding it with a free curve. Also, when the button 5313 is selected, the target of interest can be selected by surrounding it with a rectangle. Also, when the button 5314 is selected, the partial image region can be edited by changing the size of the frame surrounding the target of interest. Also, when the button 5315 is selected, a new partial image region can be created by drawing a frame. Also, when the button 5316 is selected, the correction mode for associating the partial image regions of the first medical image 510 and the second medical image 520, which will be described later, can be turned on.

[0019] Furthermore, the UI elements displayed on the display unit 120 include checkboxes 532 for performing operations to synchronize the first medical image 510 and the second medical image 520 or to switch the images displayed in the left and right image frames. Specifically, when the "Synchronized View" checkbox is checked, the first medical image 510 and the second medical image 520 displayed on the left and right are operated synchronously. For example, when the "Synchronized View" checkbox is checked and button 5311 is selected, the same movement, enlargement, or reduction can be performed simultaneously on the first medical image 510 and the second medical image 520. When the "Image Comparison" checkbox is checked, the image frames displaying the first medical image 510 and the second medical image 520 can be swapped between the left and right. More specifically, the second medical image 520 can be displayed in the left image frame while retaining the frame indicating the partial image region created for the first medical image 510. After the coordinate transformation process described later is performed, when displaying the second medical image 520 in the left image frame, the image of the second medical image 520 after the coordinate transformation can be displayed. This makes it possible to visually confirm the changes in the object of interest between the first medical image 510 and the second medical image 520.

[0020] In addition, a filter setting section 533 is displayed on the screen as another UI element. The filter setting section 533 is an index that specifies a range of at least one of the size and color density of the object of interest using the range sliders 5331 and 5332, and implements a filtering function that selects the object of interest within the set range. The lower limit of size or color density can be set using the left knob of the range sliders 5331 and 5332, and the upper limit of size or color density can be set using the right knob. If there is a knob at the left end or right end of the range sliders 5331 and 5332, filtering based on the lower or upper limit of size or color density may be disabled. For example, as shown in Figure 3, when the range of size of the candidate lesion area of ​​interest is set using the range slider 5331, the partial image area of ​​the candidate lesion that has a size within the set range may be selected from the first partial image area group 511 and the second partial image area group 521, and the checkbox may be checked. Furthermore, when the range of color intensity of the candidate lesion area of ​​interest is set using the range slider 5332, the partial image areas of the candidate lesion area having a color intensity within the set range are selected from the first partial image area group 511 and the second partial image area group 521, and the checkboxes are checked. Here, the size of the object of interest may be represented, for example, by the diameter of a perfect circle corresponding to the area of ​​the candidate lesion area, the diameter of the circumscribed circle of the candidate lesion area, or the maximum diameter of the candidate lesion area. Also, the color intensity may be represented, for example, by the difference in brightness inside and outside the candidate lesion area. The partial image areas selected by the checkboxes are highlighted in the first medical image 510 and the second medical image 520. Alternatively, the partial image areas selected by the checkboxes are used as targets for correspondence by the matching unit 104. If the "Synchronized View" checkbox 532 is checked and the first medical image 510 and the second medical image 520 are synchronized, the range slider of the filter setting unit 533 may be synchronized for the first partial image region group 511 and the second partial image region group 521, and the same range from the lower limit to the upper limit may be selected.

[0021] The matching unit 104 associates the partial image regions of the objects of interest selected by the user or by filter settings from the first medical image 510 and the second medical image 520, which have been detected by the object of interest detection unit 103. As shown in Figure 4, the matching unit 104 performs the matching when the matching button 541 is selected. The matching is performed using any algorithm. For example, the TPS-RPM (Thin plate spline robust point matching) algorithm is used. In the TPS-RPM algorithm, the objective function shown in equation (1) below is obtained. In equation (1), x i is the coordinate of the center point of the i-th object of interest in the first medical image 510, and y j is the coordinates of the center point of the j-th object of interest in the second medical image 520, N is the number of objects of interest in the first medical image 510, L is the number of objects of interest in the second medical image 520, f is the mapping function, and P is the correspondence matrix. The second term of equation (1) is the term used to adjust the bending energy and is expressed by equation (2).

[0022]

number

[0023]

number

[0024] The matching unit 104 finds the correspondence matrix P that minimizes the objective function E expressed by equation (1), and determines the corresponding objects of interest. Here, the correspondence matrix P is an (L+1) row (N+1) column matrix, with 1 added to the number of objects of interest to represent no correspondence. As a result of the correspondence by the matching unit 104, a list table 540 is displayed on the correspondence screen 600, showing enlarged and parallel matching result information 5441 combining partial image regions of the first medical image 510 and the second medical image 520, as shown in Figure 4. At this time, the ID numbers are reassigned, for example, in descending order.

[0025] The user can set constraints and make modifications to the results of the correspondence performed by the matching unit 104. The procedure for user operation will be explained in detail using Figure 5, which is an enlarged view of the list table 540 of the correspondence screen 600 shown in Figure 4. As shown in Figure 5, the correspondence screen 600 displays matching result information 5441 of the combination of the partial image region 543 of the first medical image 510 and the partial image region 544 of the second medical image 520, and a lock button 545 (fixing operation element) and an unlock button 546 (unlocking operation element) arranged next to them. The user can fix one of the combinations of partial image region 543 and partial image region 544 as a constraint condition when performing the next correspondence process. Figure 5 shows the state in which the combination is fixed by turning ON the lock button 545 for the combination with ID number 5. The matching unit 104 performs a re-processing of the correspondence between multiple partial image regions 543 other than the partial image region 543 related to the constraint conditions, and multiple partial image regions 544 other than the partial image region 544 related to the constraint conditions.

[0026] Furthermore, the user can modify each combination of partial image region 543 of the first partial image region group 511 and partial image region 544 of the second partial image region group 521 that the matching unit 104 has associated. Figure 5 shows the state in which the association has been released by selecting the release button 546 for the combination with ID number 1. After that, the user selects the button 5316 shown in Figure 4 and then selects a partial image region 544 from the second medical image 520 that is thought to correctly correspond to the partial image region 543 of the first medical image 510 whose association has been released. In the example in Figure 4, the partial image region 543 of the first medical image 510 with ID number 1 is associated with the partial image region 544 of the second medical image 520 with ID number 1, but it is thought that the partial image region 543 of the first medical image 510 with ID number 1 should correctly correspond to the partial image region 544 of the second medical image 520 with ID number 8. In this case, the user can select a partial image region with ID number 8 from the second medical image 520. At this time, a confirmation screen 547 as shown in Figure 6 is displayed, and the user can correct the mapping by selecting "Execute" on this confirmation screen 547. In this way, when the release operation is performed, a correction operation can be performed to newly associate the partial image region included in either the first partial image region group 511 or the second partial image region group 521 with the other partial image region of the first partial image region group 511 or the second partial image region group 521 that was involved in the release operation. Note that for combinations whose mapping has been corrected by the user, the lock button 545 may be turned ON to make this combination a constraint condition. As a result, the matching unit 104 can reprocess the mapping for multiple partial image regions 543 other than the partial image region 543 that was modified by the user, and multiple partial image regions 544 other than the partial image region 544 that was modified by the user.

[0027] Furthermore, there are cases where no correspondence is established because there is no corresponding partial image region in either the first partial image region group 511 of the first medical image 510 or the second partial image region group 521 of the second medical image 520. For example, if a candidate area of ​​skin lesion newly appears or disappears, no correspondence is established. In such cases, the absence of a correspondence can be used as a constraint. For example, as shown in Figure 7, there is no corresponding object of interest in the partial image region 543 with ID number 3 in the first medical image 510 in the second medical image 520. In this case, as shown in the list table 540 in Figure 7, the partial image region 544 with ID number 3 is left blank and the lock button 545 is turned ON. This fixes that there is no correspondence for the partial image region 543 with ID number 3, and the correspondence process can be performed for partial image regions 543 with ID numbers other than 3.

[0028] The coordinate transformation unit 105 corrects the image by performing a nonlinear coordinate transformation on either the first medical image 510 or the second medical image 520, as indicated by the correspondence matrix calculated by the matching unit 104. For example, the image corrected by performing a nonlinear coordinate transformation on the second medical image 520 may be displayed in the image frame on the right. Alternatively, by turning on the "Image Comparison" button on the display screen in Figure 4, the first medical image 510 and the second medical image 520 are swapped left to right, and the second medical image 520 is displayed on the left, and the second medical image 520 after the nonlinear coordinate transformation is displayed. In this case, since the second medical image 520 after the nonlinear coordinate transformation is displayed while retaining the frame of the partial image region of the first medical image 510, it becomes easier to visualize the changes over time in the corresponding candidate lesion region.

[0029] The matching unit 104 can bring the accuracy of the matching closer to 100% by repeatedly executing the matching process after the user has set constraints and modified the matching conditions. After the matching by the matching unit 104, the score calculation unit 106 compares the matched partial image region of the first medical image 510 with the partial image region of the second medical image 520 and calculates a score indicating the degree of change. The method of calculating the score can be any method, but in the case of candidate lesion regions of the skin, the calculation may be performed based on the rate of change of the color and size of the candidate lesion region. As shown in Figure 8, the score calculation is performed when the score calculation button 542 is selected. The score calculation result may be represented by a score data bar 550 or the like, as shown in Figure 8.

[0030] The operation of the image processing device 1 described above will now be explained in accordance with the flowchart in Figure 9. First, when the user selects an image file to be evaluated by the image processing device 1, the image acquisition unit 101 acquires the first medical image 510 and the second medical image 520 (step S101), and displays the first medical image 510 in the left image frame and the second medical image 520 in the right image frame of the detection screen 500 as shown in Figure 2. Next, the object of interest detection unit 103 detects objects of interest from the first medical image 510 and the second medical image 520, respectively (step S102). For example, if the first medical image 510 and the second medical image 520 are images of a person's back as shown in Figure 2, the unit detects candidate lesion areas on the skin. The object of interest detection unit 103 acquires the center coordinates of each of the detected objects of interest, assigns an identification number (ID number) to each object of interest, and stores the ID number, center coordinates, and the partial image area in which the object of interest appeared in the storage unit 110. Furthermore, the object detection unit 103 generates and displays a square frame surrounding each object of interest on the detection screen 500, and assigns an ID number to the partial image area of ​​each object of interest. The object detection unit 103 also enlarges the partial image areas included in the first partial image area group 511 of the first medical image 510 and displays them vertically in the first partial image display area 512, and enlarges the partial image areas included in the second partial image area group 521 of the second medical image 520 and displays them vertically in the second partial image display area 522.

[0031] The matching unit 104 selects a target of interest from among the targets of interest in the first medical image 510 and the second medical image 520 detected by the target of interest detection unit 103 to be used for matching (step S103). The selection of a target of interest is based on the user's selection of checkboxes provided next to the ID numbers of each partial image area in the first partial image display area 512 and the second partial image display area 522 shown in Figure 2. Alternatively, if the user operates the range sliders 5331 and 5332 of the filter setting unit 533 of the detection screen 500 to specify a range for at least one of the size and color density of the target of interest, the target of interest within the specified range is selected as the target for matching. A checkbox is placed next to the partial image area in which the selected target of interest appears.

[0032] The matching unit 104 associates the partial image regions of the objects of interest selected by the user or by filter settings from the objects of interest detected by the object of interest detection unit 103 in the first medical image 510 and the second medical image 520 (step S104). As shown in Figure 4, the matching unit 104 performs the association when the matching button 541 is selected. As a result of the association by the matching unit 104, a correspondence matrix is ​​calculated that shows the correspondence between the center coordinates of the objects of interest in the first medical image 510 and the center coordinates of the objects of interest in the second medical image 520 (step S105). Based on the correspondence matrix calculated in step S105, the coordinate transformation unit 105 transforms the coordinates of either the first medical image 510 or the second medical image 520 (step S106) and displays the transformed image in either the left or right image frame.

[0033] Multiple pairs of partial image regions of the first medical image 510 and the second medical image 520, which have been matched by the matching unit 104, are displayed in the list table 540 shown in Figure 4. The user checks the list table 540 of the first medical image 510, the second medical image 520, and the matched partial image regions to confirm whether there are any errors in the matching. If the user determines that all the matchings are correct (Step S107: Yes) and the user selects the score calculation button 542, the score calculation unit 106 calculates a score indicating the change over time for each partial image region (Step S108). The process then ends. If the object of interest is a candidate lesion region of the skin, the score may be calculated based on the rate of change in the color and size of the candidate lesion region.

[0034] If the correspondence is determined to be incorrect (Step S107: No), the user performs an operation to correct the correspondence (Step S109). Specifically, by selecting the release button 546, which is placed next to the corresponding partial image region 543 of the first medical image 510 and the partial image region 544 of the second medical image 520, as shown in Figure 5, the correspondence between the two partial image regions 543 is released. A new correspondence is generated by the user's selection operation on the first medical image 510 and the second medical image 520. When a new correspondence is made by the user's correction, a constraint condition is set to fix this correspondence (Step S110). Then, returning to Step S104, the matching unit 104 performs correspondence for the partial image regions 543 and 544 other than the partial image regions 543 and 544 related to the constraint condition (Step S104). Subsequently, the matching unit 104 again calculates a correspondence matrix showing the correspondence between the center coordinates of the object of interest in the first medical image 510 and the center coordinates of the object of interest in the second medical image 520 (step S105). The coordinate transformation unit 105 then transforms the coordinates of either the first medical image 510 or the second medical image 520 (step S106), and displays the transformed image in the left and right image frames. The process from steps S104 to S106 is repeated in this manner, and when the user determines that all the correspondences are correct (step S107: Yes), the score calculation button 542 is selected, and the score calculation unit 106 calculates a score showing the change over time for each partial image region (step S108), and the process ends.

[0035] As described above, in the image processing apparatus 1 according to this embodiment, the image acquisition unit 101 acquires a first medical image 510 including a first partial image region group 511 and a second medical image 520 including a second partial image region group 521, the object of interest detection unit 103 detects the object of interest from the first medical image 510 and the second medical image 520 respectively, and the matching unit associates the partial image regions in the first medical image 510 and the second medical image 520 in which the object of interest appears. The matching unit 104 accepts user modifications to the association of multiple pairs of partial image regions, and uses the user's modified association as a constraint condition to re-associate multiple partial image regions other than the partial image region related to the constraint condition of the first medical image 510 and multiple partial image regions other than the partial image region related to the constraint condition of the second medical image 520. This makes it possible to accurately compare the corresponding partial image regions.

[0036] Furthermore, the object detection unit 103 acquires the size or color density values ​​of each object of interest appearing in the partial image area, and when the user performs an operation to specify at least one of the size or color density ranges of the object of interest on the detection screen 500, the partial image area of ​​the object of interest corresponding to the size or color density range specified by the user is highlighted in the partial image area of ​​the first medical image 510 and the partial image area of ​​the second medical image 520, respectively. This makes it possible to easily select an object of interest.

[0037] Furthermore, the matching unit 104 displays on the display unit 120 a matching screen 600 which includes the first medical image 510 and the second medical image 520, on which multiple pairs of partial image regions have been associated, and a list 540 which displays enlarged and parallel displays of the multiple pairs of associated partial image regions included in the first partial image region group 511 and the partial image regions included in the second partial image region group 521. The matching unit 104 accepts user modifications to the association between the partial image regions included in the first partial image region group and the partial image regions included in the second partial image region group. The list 540 includes a release operation element that releases each of the multiple pairs of partial image region associations. When a release operation is performed on the release operation element, the matching unit 104 accepts a user operation to newly associate the other partial image region with the partial image region included in either the first partial image region group 511 or the second partial image region group 521 related to the release operation. This makes it possible to improve the accuracy of the partial image region association.

[0038] Although embodiments of the present invention have been described above, these embodiments are merely examples, and the scope of application of the present invention is not limited thereto. In other words, the embodiments of the present invention can be applied in various ways, and all embodiments fall within the scope of the present invention.

[0039] For example, in the embodiment, the matching unit 104 performs an association with the object of interest detected by the object of interest detection unit 103. However, before the matching unit 104 performs the association, several associations can be made by the user's selection, and the matching unit 104 can perform the association with the constraint of fixing these associations. This reduces the processing load on the matching unit 104 and improves the accuracy rate.

[0040] Furthermore, while the embodiment displays and allows comparison of two medical images, three or more medical images may be displayed. Alternatively, two images may be selected from the three or more medical images and associated with each other.

[0041] Furthermore, in this embodiment, the first and second images are defined as first medical images 510 and 520, which are images of human skin, and candidate lesion areas on the skin are detected and associated as objects of interest. However, the embodiment is not limited to this. The first and second images 520 may be other medical images, such as endoscopic images or tomographic images. Alternatively, images other than medical images may be used, such as images of the exterior walls of buildings. In that case, deterioration, damage, etc., on the exterior wall may be detected and associated as objects of interest.

[0042] Furthermore, in the above-described embodiment, the image processing program executed by the processor 100 is pre-stored in the non-volatile memory of the storage unit 110, but is not limited to this. The program for executing the above-described various processes may be implemented in an existing general-purpose computer or the like, thereby enabling it to function as the image processing apparatus according to the above-described embodiment.

[0043] The method of providing such programs is optional. For example, they may be distributed by storing them on a computer-readable storage medium (flexible disk, CD (Compact Disc)-ROM, DVD (Digital Versatile Disc)-ROM, MO (Magneto Optical Disc), memory card, USB memory, etc.), or they may be stored on network storage such as the internet and provided for download.

[0044] Furthermore, when the above-mentioned processing is performed through a division of labor between the OS (Operating System) and the application program, or through collaboration between the OS and the application program, only the application program may be stored on a recording medium or storage device. It is also possible to superimpose the program onto a carrier wave and distribute it over a network. For example, the above program may be posted on a bulletin board system (BBS) on a network and distributed over the network. This program can then be launched and executed under the control of the OS, just like other application programs, to perform the above-mentioned processing.

[0045] Furthermore, the processor 100 may consist of any single processor, such as a single processor, a multi-processor, or a multi-core processor, or it may be configured by combining any of these processors with processing circuits such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0046] This invention allows for various embodiments and modifications without departing from the broad spirit and scope of the invention. Furthermore, the embodiments described above are for illustrative purposes only and do not limit the scope of the invention. In other words, the scope of the invention is indicated not by the embodiments, but by the claims. Various modifications made within the scope of the claims and the equivalent scope of the meaning of the invention are considered to be within the scope of this invention. [Explanation of symbols]

[0047] 1…Image processing device, 100…Processor, 101…Image acquisition unit, 102…User interface, 104…Matching unit, 120…Display unit, 510…First medical image, 520…Second medical image

Claims

1. A first image and a second image are obtained, each containing multiple partial image regions, and a correspondence is made between the multiple partial image regions contained in the first image and the multiple partial image regions contained in the second image. The system accepts user modifications to the correspondence between the partial image region included in the first image and the partial image region included in the second image. Equipped with a processor, The processor, using the correspondence related to the user's modification as a constraint, re-establishes the correspondence between multiple partial image regions in the first image other than the partial image region relating to the constraint, and multiple partial image regions in the second image other than the partial image region relating to the constraint. Image processing device.

2. The multiple sub-image regions included in the first and second images are each image regions in which one object of interest appears. The processor detects the object of interest from the first image and the second image, and associates the detected object of interest with the partial image region in which it appears. The first image and the second image, which have been associated with multiple pairs of the aforementioned partial image regions, are displayed on the display unit. The image processing apparatus according to claim 1.

3. The processor calculates a correspondence matrix that associates the central coordinates of the object of interest included in the partial images of the first image and the second image with each other, and performs a correction on either the first image or the second image using the nonlinear coordinate transformation indicated by the correspondence matrix. The image processing apparatus according to claim 2.

4. If the partial image region corresponding to the partial image region contained in either the first image or the second image does not exist in the other, and therefore the correspondence is not established, the processor accepts a correction from the user indicating that no correspondence exists. The processor, under the constraint that there is no correspondence related to the user's modification, re-establishes correspondences for multiple partial image regions other than the partial image region related to the constraint in either the first image or the second image. The image processing apparatus according to claim 1.

5. The processor further accepts the user's selection of a plurality of partial image regions included in the first image and the second image, respectively, to be associated with the partial image region. The image processing apparatus according to claim 1.

6. The first and second images are medical images. The image processing apparatus according to any one of claims 1 to 5.

7. The first and second images are medical images of human or animal skin, and the first and second images include the partial image region in which a candidate lesion of the human or animal skin appears as the object of attention. The image processing apparatus according to claim 6.

8. The processor, We obtain a first image and a second image, each containing multiple sub-image regions. A correspondence is made between the multiple partial image regions included in the first image and the multiple partial image regions included in the second image. The system accepts user modifications to the correspondence between the partial image region included in the first image and the partial image region included in the second image. Using the correspondence related to the user's modification as a constraint, the correspondence between the multiple partial image regions in the first image other than the partial image region related to the constraint is re-established, and the multiple partial image regions in the second image other than the partial image region related to the constraint is re-established. Image processing methods.

9. Computers, Image acquisition unit that acquires a first image and a second image, each containing multiple partial image regions. A matching unit that associates a plurality of partial image regions included in the first image with a plurality of partial image regions included in the second image. A program for functioning as a user interface that accepts user modifications to the correspondence between the partial image region included in the first image and the partial image region included in the second image, The matching unit, using the correspondence related to the user's modification as a constraint, re-establishes the correspondence between multiple partial image regions other than the partial image region relating to the constraint in the first image and multiple partial image regions other than the partial image region relating to the constraint in the second image. program.

Images