Image display device, image display method, program, and storage medium

Abstract

To provide an image display device that has improved display of an image.SOLUTION: An image display device comprises: acquisition means that acquires a first image, first distance information that is distance information corresponding to the first image, a second image, and second distance information that is distance information corresponding to the second image; display means that displays the first image and the second image; detection means that detects a first measurement position that is a measurement position of the first image; and calculation means that calculates a corresponding position of the second image corresponding to the first measurement position based on the first distance information and the second distance information. The display means displays information related to the first measurement position superimposed on the corresponding position of the second image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image display device, an image display method, and a program.

Background Art

[0002] A technique has been proposed for measuring the size of a subject in an image by obtaining distance information from an imaging device to the subject at the time of image capture. In Patent Document 1, a method has been proposed for detecting a size measurement position specified by a user on an image and measuring the size between the specified positions. Patent Document 1 also describes superimposing and arranging side by side the size information measured in each image on the image.

[0003] Such an image display method can be used, for example, in a medical field to display an image of a skin disease part periodically photographed together with the size information of the disease part for checking the progress of the disease part. It can also be used for observing the progress of infrastructure inspections such as cracks and deformations in structures. Or it can also be used to confirm the difference in the size of substantially the same subject (for example, individual differences in animals and plants, individual differences in used goods and handicrafts in e-commerce).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the conventional method, the measurement results in each image are displayed on each image, and since it is necessary to compare each image to confirm the difference in size, the difference is difficult to understand. Further, when measuring substantially the same position on the subject, if the orientation and size of the subject in each image are different, it becomes difficult to specify substantially the same position on each image.

[0006] As mentioned above, there was room for improvement in image display in the past.

[0007] This invention has been made in view of the above-mentioned problems, and aims to provide an image display device with improved image display. [Means for solving the problem]

[0008] An image display device according to one embodiment of the present invention comprises: acquisition means for acquiring a first image, first distance information which is distance information corresponding to the first image, a second image, and second distance information which is distance information corresponding to the second image; display means for displaying the first image and the second image; detection means for detecting a first measurement position which is the measurement position of the first image; and calculation means for calculating a corresponding position in the second image corresponding to the first measurement position based on the first distance information and the second distance information, wherein the display means displays information regarding the first measurement position superimposed on the corresponding position in the second image. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide an image display device that improves the display of images. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 is a block diagram showing the functional configuration for executing an image display method according to one embodiment of the present invention. [Figure 2] This is a flowchart of the image display processing performed in the computer 100 of this embodiment. [Figure 3] This figure shows an example of the screen displayed on the output unit 14 after selecting the first image. [Figure 4] This figure shows an example of how to specify the measurement location when you want to measure an area. [Figure 5] This figure shows information regarding the measurement location superimposed on an enlarged image 302 of the first image 301. [Figure 6] Figure 6(A) shows an example of the screen displayed on the output unit 14 after the second image 601 is selected, and Figure 6(B) is a flowchart of the process for calculating the corresponding positions of the subjects in the first image 301 and the second image 601. [Figure 7] This diagram shows the images of the first image 301 and the second image 601 displayed in such a way that their on-screen sizes are approximately the same, based on the magnification ratio. [Figure 8] Figure 8(A) shows an example of the screen displayed on the output unit 14 when the position and orientation of the first image 301 and the second image 701 at the time of capture are slightly different, and Figure 8(B) is a flowchart of the process of scaling and deforming the second image based on distance information, taking into account the difference in orientation tilt. [Figure 9] This figure displays image 7012, which is an image obtained by transforming the second image 701 using the magnification calculated in S703. [Figure 10] Figure 10(A) shows an example of a screen displayed on the output unit 14 when the position and orientation at the time of capture of the first image 301 and the second image 801 are significantly different, and Figure 10(B) is a flowchart of the process for calculating the corresponding position based on distance information. [Figure 11] This diagram shows the information regarding the measurement position of the first image superimposed on the corresponding position 403 of the second image and displayed on the output unit 14. [Figure 12] Figure 12(A) shows an enlarged second image 602, and Figure 12(B) shows a new measurement position 4031 and an image 6021 with information about that measurement position 4031 superimposed. [Figure 13] This figure shows an example of a case where only the measurable range can be specified as the measurement location. [Figure 14] This figure shows an example of measuring an arbitrary object based on distance information. [Modes for carrying out the invention]

[0011] Hereinafter, embodiments for implementing the present invention will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0012] The following embodiments will describe an image display device, which is a device that executes an example of the image display method according to the present invention by a computer. The image display method according to the present invention uses, for example, a photographed image acquired from an imaging device such as a digital camera. The image display method according to the present invention acquires a plurality of distance information corresponding to the photographed image, and synthesizes and displays the dimension information of the subject shown in the photographed image on the photographed image appropriately deformed so as to be easy to compare. The program for realizing the image display method according to the present invention is executable on a desktop computer, a laptop computer, and a portable computer. Note that the distance information corresponding to a certain image is an example of the shape information indicating the shape of the subject shown in the certain image.

[0013] Referring to FIG. 1, the configuration of a computer as an example of an image display device that executes the image display method according to an embodiment of the present invention will be described. FIG. 1 is a block diagram showing a functional configuration for executing the image display method according to the present invention. The image display device that executes the image display method according to the present invention may have other components.

[0014] The computer 100 includes an input unit 10, a control unit 11, an arithmetic unit 12, a storage unit 13, and an output unit 14. The input unit 10 includes a user interface that acquires actions from a user, such as a mouse, a keyboard, and a joystick, and an interface (I / F) that acquires an image and distance information from a digital camera or another external device.

[0015] The control unit 11 controls each part of the entire computer 100, such as the control of the input unit 10, the arithmetic unit 12, the storage unit 13, and the output unit 14. The arithmetic unit 12 has an arithmetic processing device and executes an arithmetic processing program stored in the storage unit 13. The control unit 11 incorporates a CPU as a computer and controls each part of the entire computer 100 via a bus based on a computer program stored in a non-volatile memory. The CPU is an abbreviation for Central Processing Unit.

[0016] The storage unit 13 is composed of a primary storage device such as a memory and a secondary storage device such as a hard disk. The storage unit 13 stores various types of information such as image data, distance information, subject size information, and parameters input via the input unit 10. Further, the storage unit 13 stores the image synthesized and generated by the arithmetic unit 12.

[0017] The output unit 14 is, for example, a display and is an I / F for outputting a synthesized image.

[0018] The image display method according to the present invention may be executed on a PC or a portable computer realized by a touch panel display having the user interface functions of the input unit 10 and the output unit 14 in FIG. 1. At this time, the input unit 10 and the output unit 14 further have an interface for acquiring image and distance information from a digital camera or other external device not shown in the figure.

[0019] 《Image Display Method》 Regarding the image display processing executed in the computer 100 of the present embodiment, specific processing will be described using the flowchart of FIG. 2. The processing corresponding to the flowchart is realized by the control unit 11 reading out a corresponding processing program stored in, for example, a non-volatile memory in the storage unit 13 and expanding and executing it in a volatile memory in the control unit 11 so that each part operates.

[0020] In S201, the control unit 11 displays multiple images stored in the storage unit 13 on the output unit 14. The images stored in the storage unit 13 are images to which distance information has been added. The control unit 11 stores images acquired via, for example, an imaging unit and a distance measuring unit (not shown) in the storage unit 13. The control unit 11 also stores images acquired via the interface of the input unit 10, such as images captured by an external imaging device, in the storage unit 13. In this embodiment, an image of a skin disease area 105 on the patient's hand (see Figure 3) will be used for explanation.

[0021] In S202, the control unit 11 selects the image desired by the user (hereinafter referred to as the "first image") from among the displayed multiple images via the input unit 10.

[0022] Figure 3 is an example of the screen displayed in the output unit 14 after selecting the first image. In Figure 3, the images listed at the bottom are multiple images displayed in S201, and are images of the skin disease area 105 taken with different camera parameters, imaging position, and shooting time. From the displayed multiple images, the user selects the first image 301. The selected first image is displayed as a large enlarged image 302 in the middle to top of the screen. At this time, in order to clearly indicate that the currently selected image is present, the outer frame of the first image 301 may be made thicker than that of the other images, or the color of the frame may be changed.

[0023] In S203, the control unit 11 detects the measurement position desired by the user via the input unit 10 for the first image. Here, the measurement position is the position information on the image of two points or the line connecting those two points if the user wants to measure the length of the line between two points on a subject, or the position information on the image of a region if the user wants to measure the area of ​​a region. The user uses an input device such as a mouse or touch panel to specify the desired measurement position while looking at the displayed first image.

[0024] The method for specifying the measurement location can be changed as appropriate depending on the object the user wants to measure. For example, to measure the distance between two points on an object, the user can specify two points on the image. To find the length of a curve, the user can specify it using connected line segments that connect multiple points from the start to the end point. To measure an area, the user can specify the enclosed area by connecting the start and end points of the aforementioned curve.

[0025] Figure 4 shows an example of how to specify a measurement location when you want to measure an area. The measurement area 402 represents the measurement location. The square 401 represents a node for adjusting the outer shape of the measurement area 402. By adjusting the position of each node 401 on the enlarged image 302, the user adjusts the shape of the figure 402 enclosed by thin lines and specifies the position information of the area to be measured on the image. Alternatively, the enlarged image 302 may be divided into superpixels, and the user may specify the measurement location (measurement area) by specifying each superpixel.

[0026] Alternatively, technologies such as AI or neural networks may be used to detect specific subject areas, such as affected areas, and their location information may be used as the measurement location. Alternatively, the subject area may be detected with higher accuracy by using AI technology to perform area detection with measurement location information entered by the user, and the obtained location information may be used as the measurement location. AI is an abbreviation for Artificial Intelligence.

[0027] In S204, the control unit 11 displays information regarding the measurement position on the output unit 14 by superimposing it onto an enlarged image 302 of the first image 301, as shown in Figure 5. The information regarding the measurement position includes measurement results 501 such as area and length, information indicating the measurement location such as a shaded area 502, and scale lines 503 and auxiliary lines 504 that present this information to the user in an easy-to-understand manner. The area and length are calculated using existing methods based on distance information corresponding to the first image.

[0028] Here, distance information is not limited to the distance from the imaging device to the subject, but also includes information necessary for measuring the subject, such as the size in millimeters of each pixel in the image in physical space, or the focal length information at the time of shooting. The distance information may also be the distance to the object that has been recognized, or the distance values ​​corresponding to the map of the layers obtained by decomposing the image into layers for each distance.

[0029] In S205, the control unit 11 selects a user-desired image (hereinafter referred to as the "second image") from the displayed multiple images via the input unit 10, which is different from the first image. The second image may be one image or multiple images.

[0030] In S206, the control unit 11 calculates the position on the image corresponding to the measurement position in the first image (hereinafter referred to as the "corresponding position") in the second image. The control unit 11 performs this calculation based on the first image, the second image, and their distance information. The method for calculating the corresponding position is described below, but these methods are just examples, and other existing methods may be used, or the presented methods may be arbitrarily combined to perform the calculation.

[0031] Figure 6(A) is an example of the screen displayed on the output unit 14 after the second image 601 is selected. An enlarged image 602 of the second image 601 is displayed in the upper right corner of the screen.

[0032] In the first image 301 and the second image 601, if the subject (skin disease area 105) is located at approximately the same position within the images, and only the magnification differs, the corresponding position can be calculated using the process shown in the flowchart in Figure 6(B), for example.

[0033] First, in step S601 of Figure 6(B), the control unit 11 acquires the magnification ratios for both the first and second images. Next, in step S602, as shown in Figure 7, the control unit 11 scales the second image 601 to image 6012 based on the magnification ratios so that the size of the subject in the first image 301 and the second image 601 are approximately the same. Finally, in step S603, the control unit 11 sets the corresponding position 403 in the second image 6012 to be the same pixel coordinate position as the measurement position 402 in the first image. This method allows for the determination of corresponding positions with simple processing when the magnification ratios during shooting are different.

[0034] Next, we will explain the case where the position and orientation at the time of capture of the first image 301 and the second image 701 are slightly different, as shown in Figure 8(A). In this case, the control unit 11 scales and deforms the second image based on the distance information, taking into account the difference in orientation inclination, as shown in the flowchart in Figure 8(B), and then calculates the corresponding position on the image. Note that if the skin disease area 105 is in approximately the same position in the image and the shooting magnification is different, the corresponding position can be determined using the method described above, so we will assume that the shooting magnification is the same.

[0035] First, in step S701 of Figure 8(B), the control unit 11 acquires the distance distributions corresponding to the first image and the second image, respectively. Next, in step S702, the control unit 11 calculates the depth-direction slope of the distance distributions for each image. Then, in step S703, the control unit 11 uses these slopes to calculate the scaling factor at each position of the second image so that the second image approaches the first image.

[0036] Next, in S704, the control unit 11 deforms the second image 701 to image 7012 using the magnification calculated in S703, as shown in Figure 9. In S705, the control unit 11 compares the second image deformed in S704 with the first image, calculates the amount of positional shift between the two images, and corrects the second image. The control unit 11 calculates the amount of positional shift, for example, by matching feature points of the images. Finally, in S603, the control unit 11 sets the pixel coordinate position on the second image corrected in S705 to be the same as the measurement position 402 on the first image, and sets it to the corresponding position 403 on the second image. This method allows for the determination of the corresponding position with simple processing when the posture during shooting is different.

[0037] Let's explain the case where the position and orientation at the time of capture of the first image 301 and the second image 801 are significantly different, as shown in Figure 10(A). In this case, the control unit 11 calculates the corresponding position based on distance information, as shown in the flowchart in Figure 10(B).

[0038] In step S801 of Figure 10(B), the control unit 11 acquires distance information corresponding to the first image and the second image, respectively. Next, in step S802, the control unit 11 converts the distance distribution of the first and second images into a point cloud based on information such as focal length and sensor data. In step S803, the control unit 11 compares the two point clouds to estimate the differences in attitude and position between them and determines the correspondence between each point cloud. As a method for determining the correspondence between each point cloud, for example, a method that calculates the similarity between point clouds and estimates the difference in position and attitude, such as the ICP algorithm, can be used.

[0039] Finally, in S804, the control unit 11 obtains a point cloud of a second image corresponding to the measurement position, and sets the position coordinates on the second image corresponding to the point cloud as the corresponding position. This method allows for highly accurate determination of the corresponding position even when the position and orientation at the time of shooting differ significantly.

[0040] Even with the same subject, the shape and appearance may differ depending on the timing of the photograph, such as the degree of flexion of the arm joint or the healing status of skin diseased areas. When there are clearly different areas between the first and second images, the corresponding image and distance information may be excluded when determining the corresponding position. This improves the accuracy of calculating the corresponding position. The method for detecting the area may be specified in the same way as the method for detecting the measurement position.

[0041] In this case, in S207, the control unit 11 displays information regarding the measurement position of the first image superimposed on the corresponding position 403 of the second image on the output unit 14, as shown in Figure 11. As mentioned above, the information regarding the measurement position includes measurement results 501 such as area and length, information that identifies the measurement location such as the shaded area 502, and scale lines and auxiliary lines that present this information to the user in an easy-to-understand manner. By superimposing the measurement results of the first image onto the same subject in the second image, the user can easily understand the differences between similar subjects in the two images.

[0042] Furthermore, the first and second images may be arbitrarily distorted and displayed so that the size, posture, and on-screen position of the subjects in each image are similar. For example, if the size or posture of the subjects in the images differ, one or both images may be distorted and displayed based on the magnification and distance distribution, as in S602 and S704. If the positions differ, the amount of positional shift between the two images is calculated by comparing the images or distance distribution, and one or both images are displaced and displayed. As the apparent appearance of the first and second images becomes closer, users will be able to understand the differences between similar subjects more easily.

[0043] For similar reasons, if the size of the detection position is small relative to the size of the displayed image, the images near the detection position in the first image and the corresponding position in the second image may be enlarged and displayed on the output unit 14.

[0044] As explained above, by superimposing information about the measurement position in the first image onto the corresponding position in the second image, users can more easily understand the difference between the measurement targets in the first and second images.

[0045] Furthermore, in calculating the corresponding position of the second image, the distance information of the first and second images may be used to determine the differences in size, position, and orientation of the subject between the two images, and the corresponding position of the second image corresponding to the measurement position of the first image may be calculated based on this.

[0046] Alternatively, the accuracy in calculating the corresponding position in the second image can be determined, and this calculated accuracy may be superimposed on the second image and displayed.

[0047] In this embodiment, we have shown an example in which information regarding the measurement position of the first image is superimposed on the corresponding position of the second image. However, based on the results, the measurement position may be further specified on the second image and measurements may be performed. Figure 12 shows this.

[0048] Figure 12(A) shows an enlarged second image 602. Node 401 is displayed at the corresponding position 403. The user can specify the measurement position of the skin disease area 105 again by manipulating this node 401.

[0049] Figure 12(B) shows a new measurement position 4031 and an image 6021 in which information about that measurement position 4031 is superimposed. The area 5011 of the measurement position and the measurement region (shaded area) 5021 are displayed. In this way, by setting the previously determined corresponding position 403 as the initial position of the new measurement position, the user can more easily specify the measurement position 4031 in the second image.

[0050] Furthermore, in this embodiment, the measurement position of the first image was specified before calculating the corresponding position of the second image. However, the corresponding positions of the first and second images could be calculated first, allowing only the measurable range to be specified as the measurement position. This would prevent errors in detecting the corresponding position.

[0051] Figure 13 shows an example of a case where only the measurable range can be specified as the measurement location. In Figure 13, the first image 301 and the second image 801 are selected and enlarged. The grayed-out area 1101 in the enlarged image 302 of the first image 301 is an unmeasurable area because it is hidden by the shadow of the hand in the second image 801. Similarly, the grayed-out area 1102 in the enlarged image 802 of the second image 801 is an unmeasurable area because it is not visible in the first image 301. By displaying the grayed-out areas, the user can understand the unmeasurable areas at a glance, thus preventing detection errors in the corresponding location. In this embodiment, unmeasurable areas are displayed in gray, but other methods may be used as long as they are visible to the user.

[0052] Furthermore, after detecting the measurement position of the first image 301, images among the multiple images that are difficult to calculate the corresponding position of may be grayed out and displayed on the output unit 14. This allows the user to easily understand which of the multiple images are comparable.

[0053] In this embodiment, the area of ​​the skin diseased area 105 was measured, but the present invention may measure any object that can be measured based on distance information. For example, if the control unit 11 wants to measure the length of the skin diseased area 105 in the direction of the arm, it displays two nodes 1201 represented as squares in the enlarged image 302 of Figure 14. The user specifies the measurement position 402 (a straight line connecting node 1201 and node 1201) by manipulating the nodes 1201.

[0054] Information regarding the measurement position 402 in the first image 301 is superimposed on the enlarged image 302, and this information is further superimposed on the corresponding position 403 in the enlarged image 802 of the second image 801, allowing the user to easily understand the difference between the measured objects. Examples of information regarding the measurement position 402 in the first image 301 include the measurement result 501, and scale lines 503 and auxiliary lines 504 that present it to the user in an easy-to-understand manner.

[0055] Furthermore, similar to when measuring area, a new measurement position may be specified in the enlarged image 802, with the corresponding position 403 as the initial position. This makes it easier for the user to specify the measurement position in the second image 801.

[0056] Furthermore, in this embodiment, there was only one measurement position or corresponding position for each image, but two or more measurement positions may be specified to measure length, area, etc. For each corresponding position in the second image 801, information regarding the measurement position is displayed superimposed. In this case, if the information regarding the measurement position is displayed too densely on the output unit 14, hindering the visibility of the subject, the user may specify only the information regarding the measurement position via the input unit 10 and display it on the output unit 14.

[0057] Furthermore, if a single image contains multiple similar subjects, the user may specify the region of each subject via the input unit 10, and the control unit 11 may treat each specified region as one of the multiple images in this embodiment and perform the processing of the present invention.

[0058] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0059] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its essence. [Explanation of symbols]

[0060] 10 Input section 11 Control Unit 12 Arithmetic section 13 Storage section 14 Output section 100 Computers

Claims

1. An acquisition means for acquiring a first image, a second image, and first position information relating to a first position at which a measurement of a subject is made in the first image; a display means for displaying the second image on a display device; determining a second position corresponding to the first position in the second image based on distance information corresponding to the first image and the second image; The display means displays first measurement information obtained by measuring the subject at the first position in the first image in a superimposed manner at the second position in the second image.

1. An image display device comprising:

2. a designation means for accepting designation of the first measurement position by a user; 2. The image display device according to claim 1 .

3. The first measurement information is information about the length of a line connecting two points including the first position on the first image, or information about the area of ​​a region including the first position on the first image.

3. The image display device according to claim 1, wherein the first and second electrodes are arranged in a first direction.

4. The first measurement information is obtained by measuring a length or an area of ​​the subject based on the first image and first distance information in a depth direction corresponding to the first image.

4. The image display device according to claim 1, wherein the first and second electrodes are arranged in a first direction.

5. The first measurement information displayed by the display means includes a line connecting two points including the first position, or a frame indicating an area including the first position.

4. The image display device according to claim 3.

6. The display means adjusts the first measurement information so that the length or size of a predetermined subject in the first image and the second image is equivalent, and superimposes the first measurement information on the second image.

6. The image display device according to claim 1,

7. The display means displays at least one of the first image and the second image by enlarging or reducing the first image and the second image so that corresponding objects in the first image and the second image are of equal size.

7. The image display device according to claim 1, wherein the first and second electrodes are arranged in a first direction.

8. The display means further displays second measurement information obtained by measuring the subject in the second image in a superimposed manner.

8. The image display device according to claim 1, wherein the first and second electrodes are arranged in a first direction.

9. The designation means accepts designation of a position for measuring the subject in the second image within a predetermined range based on the first position.

9. The image display device according to claim 8.

10. The display means displays the first measurement information together with a scale comparable to the first measurement information in a superimposed manner.

8. The image display device according to claim 1, wherein the first and second electrodes are arranged in a first direction.

11. The first measurement information includes information regarding measurement accuracy when measuring the subject.

9. The image display device according to claim 1, wherein the first and second electrodes are arranged in a first direction.

12. The display means displays the first image side by side with the second image.

10. The image display device according to claim 1,

13. Obtaining first position information relating to a first image, a second image, and a first position at which a measurement of an object in the first image is made; displaying the second image on a display device; determining a second position in the second image corresponding to the first position based on distance information corresponding to the first image and the second image; The display means displays first measurement information obtained by measuring the subject at the first position in the first image in a superimposed manner at the second position in the second image.

13. An image display method comprising:

14. Obtaining first position information relating to a first image, a second image, and a first position at which a measurement of an object in the first image is made; displaying the second image on a display device; determining a second position in the second image corresponding to the first position based on distance information corresponding to the first image and the second image; The display means displays first measurement information obtained by measuring the subject at the first position in the first image in a superimposed manner at the second position in the second image. A program characterized by:

15. Obtaining first position information relating to a first image, a second image, and a first position at which a measurement of an object in the first image is made; displaying the second image on a display device; determining a second position in the second image corresponding to the first position based on distance information corresponding to the first image and the second image; The display means displays first measurement information obtained by measuring the subject at the first position in the first image in a superimposed manner at the second position in the second image. A storage medium storing a program characterized by:

Images