Interpolation method and interpolation device for high-saliency range

Abstract

The present invention is an interpolation device for determining a high-saliency range by interpolation, the device comprising: a video acquisition unit that acquires a video of an original image; a saliency calculation unit that calculates a saliency map by using the video as an input; a saliency binarization unit that calculates a binarized saliency map by using the saliency map as an input; a saliency edge acquisition unit that extracts a saliency edge by using the binarized saliency map as an input; a saliency edge excerpting unit that outputs an excerpted saliency edge by using the saliency edge as an input; and an interpolation-based removal unit that performs interpolation for a region having high saliency in the video by using the video, the binarized saliency map, and the excerpted saliency edge as inputs.

Description

Interpolation method and device for highly conspicuous areas

[0001] The present invention relates to a method and an apparatus for interpolating highly conspicuous areas, which are applied, for example, to object recognition and concealment of the areas.

[0002] In general, people with developmental disabilities, especially those with attention deficit disorder or hyperactivity disorder, tend to have low attention function, which is a filter function that selects specific information from input information and suppresses the processing of information that has not been selected, and are therefore prone to distraction.

[0003] In response to this, for example, Non-Patent Document 1 describes that the filtering function for visual information can be replaced by diminished reality (hereinafter abbreviated as DR (Diminished Reality)) technology, which hides real objects using augmented reality (hereinafter abbreviated as AR (Augmented Reality)) technology.

[0004] For example, if you are working in the office and are distracted by a colleague passing by the edge of your field of vision, it is possible that object recognition will be used to identify the colleague and make them invisible, preventing the distraction.

[0005] Yi Fei Cheng et al., “Towards Understanding Diminished Reality”, CHI, 2022.OpenCV: Saliency API”, https: / / docs.opencv.org / 4.x / d8 / d65 / group__saliency.html (retrieved July 30, 2020) “OpenCV: Morphological Transformations”, https: / / docs.opencv.org / 4.x / d9 / d61 / tutorial_py_morphological_ops.html (retrieved July 30, 2020) “OpenCV: Image Inpainting”, https: / / docs.opencv.org / 3.4 / df / d3d / tutorial_py_inpainting.html (retrieved July 30, 2020)

[0006] However, object recognition and occlusion by interpolation of the object's range require a large amount of calculation, which makes it difficult to perform in real time on a poor AR terminal.

[0007] The present invention has been made in view of the above circumstances, and aims to provide a method and device for interpolating a highly conspicuous range that enables calculations for object recognition and concealment by interpolation within that range to be performed in real time even on a poor AR terminal.

[0008] In order to achieve the above object, a first aspect of the present invention is an apparatus for determining a high-attractiveness range by interpolation, the interpolation apparatus comprising: an image acquisition unit that acquires an image of an original image; an attractiveness calculation unit that receives the image as input, calculates an attractiveness, and obtains an attractiveness map that is a map of attractiveness; an attractiveness binarization unit that binarizes the attractiveness map to calculate a binarized attractiveness map; an attractiveness edge acquisition unit that receives the binarized attractiveness map as input, and extracts attractiveness edges; an attractiveness edge extraction unit that receives the attractiveness edges as input, and outputs extracted attractiveness edges; and an interpolation obscuration unit that receives the image, the binarized attractiveness map, and the extracted attractiveness edges as input, and interpolates a high-attractiveness range in the image.

[0009] A second aspect of the present invention is the interpolation device of the first aspect, in which the attention calculation unit calculates the attention map using current frame information obtained from the video and the difference between the current frame information and previous frame information.

[0010] A third aspect of the present invention is the interpolation device of the first aspect, further comprising an image storage unit that stores an image for a predetermined period of time and outputs an attractiveness edge obtained based on the image to an attractiveness edge extraction unit, which further receives the attractiveness edge output from the image storage unit as input and outputs the extracted attractiveness edge.

[0011] A fourth aspect of the present invention is a method for determining a high-attractiveness range by interpolation, in which an image acquisition unit acquires an image of an original image, a processor takes the image as input and calculates an attractiveness map, takes the attractiveness map as input and calculates a binarized attractiveness map, takes the binarized attractiveness map as input and extracts attractiveness edges, takes the attractiveness edges as input and outputs extracted attractiveness edges, and interpolates a high-attractiveness range in the image using the image, the binarized attractiveness map, and the extracted attractiveness edges as input.

[0012] According to the interpolation method and apparatus of the present invention, calculations for object recognition and occlusion by interpolation of the area can be performed in real time even on a weak AR terminal.

[0013] Fig. 1 is a diagram showing the procedure of an interpolation method for a high-attraction range according to a first embodiment of the present invention. Fig. 2 is a block diagram showing an example of the configuration of an interpolation device according to the first embodiment of the present invention. Fig. 3 is a block diagram showing an example of the configuration of an interpolation device according to conventional technology. Fig. 4 is a diagram showing the procedure of an interpolation method for a high-attraction range according to a second embodiment of the present invention. Fig. 5 is a block diagram showing an example of the configuration of an interpolation device according to the second embodiment of the present invention. Fig. 6 is a simplified diagram showing the hardware configuration of a general computer.

[0014] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the size ratio between parts, etc., are not necessarily the same as those in reality. Furthermore, even when the same part is shown, the dimensions and ratios may be different depending on the drawing. In this specification and each drawing, elements similar to those explained in the previous drawings are given the same reference numerals, and detailed and redundant explanations will be omitted as appropriate.

[0015] (First embodiment) A first embodiment of the present invention will be described.

[0016] FIG. 1 is a diagram showing the procedure of an interpolation method for a highly conspicuous area according to a first embodiment of the present invention.

[0017] In the method for interpolating a high-attraction range according to the first embodiment of the present invention, as shown in FIG. 1 , (1) an original image is photographed, (2) a map of attractiveness b is calculated based on an image a of the photographed original image, (3) the map of attractiveness b is binarized c, (4) an edge d is extracted from the attractiveness b obtained by the binarization c, (5) the extracted edge e is output, and finally (6) an interpolation of a high-attraction range f is performed.

[0018] FIG. 2 is a block diagram showing an example of the configuration of an interpolation device according to a first embodiment that implements such an interpolation method.

[0019] That is, the interpolation device 10A for a high attractiveness range according to the first embodiment includes an image acquisition unit 12, an attractiveness calculation unit 14, an attractiveness binarization unit 16, an attractiveness edge acquisition unit 18, an attractiveness edge extraction unit 20, and an interpolation concealment unit 22. Note that the numbers in parentheses in the following description correspond to (1) to (6) in Figures 1.

[0020] The image acquisition unit 12 includes a camera, etc. The image acquisition unit 12 uses the camera, etc. to (1) capture an original image and acquire an image a of the original image.

[0021] The attention calculation unit 14 receives the image a acquired by the image acquisition unit 12 as an input, (2) calculates the attention b, and obtains an attention map b, which is a map of the attention. The attention can be calculated using current frame information such as RGB values ​​and luminance acquired from the image a, or the difference between the current frame information and the previous frame information. The specific calculation method is not particularly limited in this specification, and the method described in Non-Patent Document 2, for example, can be used.

[0022] The attractiveness binarization unit 16 receives the attractiveness map b calculated by the attractiveness calculation unit 14 as an input and (3) calculates a binarized attractiveness map c (hereinafter referred to as "binarized attractiveness map c"). The method for calculating the binarized attractiveness map c is not particularly limited in this specification, and the method described in Non-Patent Document 3, for example, can be used.

[0023] The attractiveness edge acquisition unit 18 receives the binarized attractiveness map c calculated by the attractiveness binarization unit 16 as an input and (4) extracts an attractiveness edge d (hereinafter referred to as an "attractiveness edge d"). The method for extracting the attractiveness edge d is not particularly limited in this specification, and the method described in Non-Patent Document 3, for example, can be used.

[0024] The attractiveness edge extracting unit 20 receives the attractiveness edge d extracted by the attractiveness edge acquiring unit 18 as input, and outputs (5) an extracted attractiveness edge e (hereinafter referred to as the "extracted attractiveness edge e").

[0025] The interpolation and concealment unit 22 receives as input the image a acquired by the image acquisition unit 12, the binarized attractiveness map c calculated by the attractiveness binarization unit 16, and the extracted attractiveness edge e output by the attractiveness edge extraction unit 20, and (6) interpolates a high attractiveness range f, which is a range with high attractiveness in the image a. The present specification does not particularly limit the method of this interpolation, and for example, the method described in Non-Patent Document 4 can be used.

[0026] The high-attraction range interpolation device 10A according to the first embodiment, configured as described above, does not require object recognition using the degree of attraction, and as shown in (4) and (5), interpolates using a small number of pixels selected from the periphery of the obscuration range, thereby reducing the amount of calculation and enabling even a weak AR terminal to perform high-attraction range interpolation in real time.

[0027] FIG. 3 is a block diagram showing an example of the configuration of a conventional interpolation device.

[0028] The interpolation device 60 according to the prior art includes an image acquisition unit 62, an object recognition unit 64, and an interpolation concealment unit 66. The image acquisition unit 62 includes a camera or the like. The image acquisition unit 62 acquires and outputs an image α using the camera or the like. The object recognition unit 64 receives the image α output from the image acquisition unit 62 as input, recognizes an object range β, and outputs the image. The interpolation concealment unit 66 receives the image α output from the image acquisition unit 62 and the recognized object range β output from the object recognition unit 64 as input, and interpolates the object range β in the image α.

[0029] As described above, the conventional interpolation device 60 does not perform attractiveness edge extraction, unlike the interpolation device 10A according to this embodiment. Because object recognition and elimination by interpolation of that range require a large amount of calculation, the conventional interpolation device 60 that does not perform attractiveness edge extraction may not be able to perform interpolation of high-attraction ranges in real time due to the large calculation load.

[0030] Second Embodiment Next, a second embodiment of the present invention will be described.

[0031] FIG. 4 is a diagram showing the procedure of an interpolation method for a highly conspicuous area according to the second embodiment of the present invention.

[0032] The interpolation method for a high-attractivity range according to the second embodiment of the present invention differs from the interpolation method according to the first embodiment in that (5) the high-attractivity range f is interpolated using (1) the extracted attractiveness edge e' of the low-attractivity range of the previous frame from the image a' of the previous frame original image, in addition to (2) the extracted attractiveness edge e of the high-attractivity range of the current frame from the image a of the current frame original image, (3) the extracted attractiveness edge e' of the high-attractivity range of the current frame from the image a of the current frame original image.

[0033] FIG. 5 is a block diagram showing an example of the configuration of an interpolation device according to the second embodiment for realizing this.

[0034] That is, the interpolation device 10B for a high attractiveness range according to the second embodiment adds an image storage unit 24 to the interpolation device 10A, and provides an attractiveness edge extraction unit 20' and an interpolation elimination unit 22' instead of the attractiveness edge extraction unit 20 and the interpolation elimination unit 22. The attractiveness edge extraction unit 20' has been improved so that it can handle the attractiveness edge d' provided from the image storage unit 24 in addition to the attractiveness edge d from the attractiveness edge acquisition unit 18, and the interpolation elimination unit 22' can handle the output from the attractiveness edge extraction unit 20'.

[0035] Therefore, the following description will focus on the components not included in the interpolation device 10A, namely, the attractiveness edge extraction unit 20', the interpolation concealment unit 22', and the video storage unit 24, but the same components as in Figure 2 are given the same reference numerals to avoid redundant explanation. The numbers in parentheses in the following description correspond to those in Figures 4 (1) to (5).

[0036] The video storage unit 24 (1) stores video a' spanning a predetermined period of time, e.g., video a' spanning a short period of time such as several frames, acquired by the video acquisition unit 12. The video storage unit 24 also outputs an attractiveness edge d' obtained based on the video a' to the attractiveness edge extraction unit 20'.

[0037] The attractiveness edge extraction unit 20' receives the attractiveness edge d' from the image storage unit 24 in addition to the attractiveness edge d from the attractiveness edge acquisition unit 18, and outputs (4) the extracted attractiveness edge e' (hereinafter referred to as the "extracted attractiveness edge e'").

[0038] The interpolation and concealment unit 22' receives as input (2) the image a of the current frame acquired by the image acquisition unit 12, the binarized attractiveness map c calculated by the attractiveness binarization unit 16, and (4) the extracted attractiveness edge e' output by the attractiveness edge extraction unit 20', and (5) interpolates a high attractiveness range f, which is a range with high attractiveness in the image a. The method of this interpolation is not particularly limited in this specification, and, as in the first embodiment, the method described in Non-Patent Document 4, for example, can be used.

[0039] The high-attractivity range interpolation device 10B of the second embodiment, with the above-described configuration, is capable of using an extracted range of a low-attractivity range in the previous frame in addition to an extracted edge of a high-attractivity range in the current frame during interpolation.

[0040] As described above, the high-attraction range interpolation device 10B according to the second embodiment also uses the degree of attraction to eliminate the need for object recognition, and performs interpolation using a small number of pixels selected from the periphery of the obscuration range, thereby reducing the amount of calculation and enabling even a weak AR terminal to perform interpolation of the high-attraction range in real time.

[0041] The interpolation devices 10A and 10B according to the first and second embodiments can both be realized by a computer such as a PC.

[0042] FIG. 6 is a simplified diagram showing the hardware configuration of a computer.

[0043] Computer 100 receives some kind of input from the outside, processes it, and outputs the results to the outside. Input device 120 is responsible for input, and output device 140 is responsible for output. CPU (Central Processing Unit) 110 controls the overall flow of data and processing, and performs calculations and other processes. In Figure 3, solid arrows represent the flow of data, and dashed arrows represent the flow of control.

[0044] In order for the computer 100 to perform processing, a program that describes the processing procedure is required. The program contains a series of instructions and data used by the instructions, all of which are temporarily stored in memory 130. The CPU 110 retrieves the instructions from memory 130 one by one, interprets them, and operates in accordance with those instructions.

[0045] Therefore, the image acquisition unit 12, the attractiveness calculation unit 14, the attractiveness binarization unit 16, the attractiveness edge acquisition unit 18, the attractiveness edge extraction units 20, 20', and the interpolation concealment units 22, 22' in the interpolation devices 10A, 10B according to the first and second embodiments are realized by the CPU 110 operating in accordance with the programs stored in the memory 130.

[0046] Although not shown in FIG. 6 , the computer 100 may have a built-in storage device such as a solid-state drive (SSD) or a hard disk drive (HDD). The video storage unit 24 of the interpolation device 10B according to the second embodiment may be implemented by such a built-in storage device. Furthermore, the video storage unit 24 is not limited to such a built-in storage device, and may also be implemented by an external storage medium such as a universal serial bus (USB) memory, or an area provided in a storage system such as a database server located in the cloud.

[0047] The present invention is not limited to the above-described embodiments, and can be embodied by modifying the components within the scope of the gist of the present invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0048] 10A, 10B Interpolation device 12 Image acquisition unit 14 Attractiveness calculation unit 16 Attractiveness binarization unit 18 Attractiveness edge acquisition unit 20, 20' Attractiveness edge extraction unit 22, 22' Interpolation and elimination unit 24 Image storage unit 60 Interpolation device according to prior art 62 Image acquisition unit 64 Object recognition unit 66 Interpolation and elimination unit 100 Computer 120 Input device 130 Memory 140 Output device a Image b Attractiveness map c Binarized attractiveness map d, d' Attractiveness edges e, e' Extracted attractiveness edges f Attractiveness range α Image β Range of object

Claims

1. An apparatus for determining a high attractiveness range by interpolation, comprising: an image acquisition unit that acquires an image of an original image; an attractiveness calculation unit that receives the image as input, calculates the attractiveness, and obtains an attractiveness map that is a map of the attractiveness; an attractiveness binarization unit that binarizes the attractiveness map and calculates a binarized attractiveness map; an attractiveness edge acquisition unit that receives the binarized attractiveness map as input, and extracts attractiveness edges; an attractiveness edge extraction unit that receives the attractiveness edges as input, and outputs extracted attractiveness edges; and an interpolation obscurance unit that receives the image, the binarized attractiveness map, and the extracted attractiveness edges as input, and interpolates a high attractiveness range in the image.

2. The interpolation device according to claim 1, wherein the attention calculation unit calculates the attention map using current frame information obtained from the video and the difference between the current frame information and previous frame information.

3. The interpolation device according to claim 1, further comprising an image storage unit that stores the image for a predetermined period of time and outputs an attractiveness edge obtained based on the image to the attractiveness edge extraction unit, wherein the attractiveness edge extraction unit further receives the attractiveness edge output from the image storage unit as an input and outputs the extracted attractiveness edge.

4. A method for determining a high attractiveness range by interpolation, wherein an image acquisition unit acquires an image of an original image, a processor takes the image as input, calculates the attractiveness, and calculates an attractiveness map that is a map of the attractiveness, binarizes the attractiveness map to calculate a binarized attractiveness map, takes the binarized attractiveness map as input, extracts attractiveness edges, takes the attractiveness edges as input, and outputs extracted attractiveness edges, and takes the image, the binarized attractiveness map, and the extracted attractiveness edges as input, and interpolates a high attractiveness range in the image.

Images