Video processing device, video processing method, and video processing program

Abstract

A video processing device according to an embodiment comprises: a first processing module that sets a first value in accordance with an operation of a user; and a second processing module that generates an image converted from the visual characteristics of a first organism to the visual characteristics of a second organism different from the first organism by using the first value and input image data including an image captured by dispersing light from a subject into wavelengths.

Description

Video processing device, video processing method, and video processing program

[0001] The embodiments relate to a video processing device, a video processing method, and a video processing program.

[0002] A technology described in Non-Patent Document 1 has been proposed as a method for generating RGB images or RGB videos based on hyperspectral images using color matching functions that correspond to human visual characteristics, thereby visualizing the visual world that we are accustomed to seeing.

[0003] However, Non-Patent Document 1 does not address the reproduction of the vision of organisms other than humans, and it is not possible to deepen understanding of the vision of other organisms or to intuitively and continuously understand the differences between the world seen by humans and the world seen by other organisms. Therefore, the technology of Non-Patent Document 1 has room for improvement.

[0004] Magnus Magnusson, et al. "Creating RGB images from hyperspectral images using a color matching function." IGARSS 2020-2020 IEEE International Geoscience and Remote Sensing Symposium. IEEE, 2020.

[0005] Generally, humans have three types of photoreceptors in bright light: L cones, M cones, and S cones. Humans visually perceive the outside world as a result of these three types of receptors capturing light. The sensitivity of human receptors to light wavelengths has a mountain-shaped characteristic, with peaks at 441 nm for S cones, 541 nm for M cones, and 566 nm for L cones. Color matching functions of the XYZ color system are known as functions that reflect the characteristics of these three types of receptors and are organized as easy-to-use engineering functions.

[0006] The color matching functions of the XYZ color system are composed of a function of the response of the L cone, a function of the response of the M cone, and a function of the response of the S cone to find the tristimulus values ​​"X", "Y", and "Z" for a certain spectral spectrum λ.

[0007] Meanwhile, biological and other research is also being conducted on the light sensitivity characteristics of each organism's vision at different frequencies. For example, it is known that honeybees have spectral sensitivity characteristics that allow them to receive light in the ultraviolet range, which is invisible to the human eye.

[0008] The present invention takes a hyperspectral image / video as input and continuously changes the image formed from an image of a color matching function that reflects human visual characteristics to an image that reflects the spectral sensitivity characteristics of another organism, thereby achieving a continuous change in the RGB image from the RGB image of the visual world seen by humans to an RGB image that simulates the visual world of another organism.

[0009] Furthermore, the present invention realizes a user interface that allows intuitive control of the degree to which an image resembles the field of view of a human or another living creature, thereby enabling intuitive field of view conversion operations.

[0010] The image processing device of the embodiment includes a first processing module that sets a first value in response to a user operation, and a second processing module that uses the first value and input image data that includes an image captured by dispersing light from a subject by wavelength to generate an image in which the visual characteristics of a first organism are converted into the visual characteristics of a second organism that is different from the first organism.

[0011] According to the embodiment, it is possible to provide an image that allows intuitive understanding of the visual perception of non-human organisms.

[0012] FIG. 1 is a block diagram showing an example of the configuration of an image processing device according to an embodiment. FIG. 2 is a block diagram showing an example of the configuration of an image processing device according to an embodiment. FIG. 3 is a diagram showing an example of a user interface. FIG. 4 is a diagram for explaining a hyperspectral image. FIG. 5 is a diagram showing an example of a standard white light spectrum. FIG. 6 is a diagram showing an example of a color matching function. FIG. 7 is a diagram showing an example of spectral sensitivity characteristics. FIG. 8 is a diagram for explaining temporal integration of an image. FIG. 9 is a flowchart showing an example of the operation of the image processing device according to an embodiment. FIG. 10 is a flowchart showing an example of the operation of the image processing device according to an embodiment.

[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Hereinafter, elements that are identical or similar to elements already described will be assigned the same or similar reference numerals, and duplicate descriptions will generally be omitted. For example, when there are multiple identical or similar elements, a common reference numeral may be used to describe each element without distinguishing between them, or a subnumber may be used in addition to the common reference numeral to describe each element distinctly.

[0014] (Embodiment) A video processing device, a video processing method, and a video processing program according to an embodiment will be described with reference to Figs.

[0015] (a) Configuration Fig. 1 is a block diagram showing an example of the configuration of a video processing device according to an embodiment. Fig. 1 shows the hardware configuration of a system including a video processing device 200 according to an embodiment.

[0016] The image processing device 200 according to the embodiment is a computer (arithmetic device) and includes a processor 21, a read-only memory (ROM) 22, a random access memory (RAM) 23, a storage 24, and an interface 25.

[0017] The processor 21 is a processing circuit capable of executing various programs (software, applications). The processor 21 controls the overall operation of the video processing device 200. The processor 21 includes a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). Note that multiple processors 21 may be provided within the video processing device 200.

[0018] The ROM 22 is a non-volatile semiconductor memory such as an EEPROM (registered trademark), and stores programs and control data for controlling the video processing device 200.

[0019] The RAM 23 is a volatile semiconductor memory such as a dynamic RAM (DRAM) or a static RAM (SRAM). The RAM 23 is used as a working area for the processor 21. The RAM 23 temporarily stores various data and parameters used by the processor 21.

[0020] The storage 24 is a non-volatile storage device such as a hard disk drive (HDD), a solid state drive (SSD), or a memory card. The storage 24 stores various types of information, data, and parameters. The storage 24 has a database configured by a collection of certain types of data. Programs and control data may be stored in the storage 24.

[0021] The storage 24 stores a program (video processing program) PRG relating to the video processing method executed by the video processing device 200 of this embodiment. The video processing method of this embodiment will be described later. The video processing program PRG includes multiple program codes corresponding to multiple processes of the video processing method of this embodiment. The video processing program PRG causes the processor 21 to execute various processes for video processing. The video processing program PRG may be stored in the ROM 22 or the RAM 23.

[0022] The interface 25 includes various connectors, various ports, a signal processing circuit, a communication module, etc. The interface 25 connects other devices 100 and 300 to the video processing device 200. The interface 25 is responsible for inputting and outputting information and data, various processes for information and data, and various controls for acquiring information and data.

[0023] The interface 25 can transfer signals, information, and data between the video processing device 200 and the other devices 100 and 300 by communication via a wireless or wired network. The interface 25 sends various data and signals from the other devices to the processor 21. The interface 25 sends various data and signals from the processor 21 to the other devices 100 and 300.

[0024] For example, the interface 25 is connected to the user interface device 100. The interface 25 receives input from a user of the video processing device 200 via the user interface device 100. This allows the video processing device 200 of this embodiment to receive an input signal (instruction) from the user in response to an operation of the user interface device 100.

[0025] For example, the interface 25 is connected to the display device 300. The interface 25 provides various information to the user of the video processing device 200 via the display device 300. This allows the video processing device 200 of the embodiment to display the processing results of the video processing device 200 on the display device 300.

[0026] The user interface device (also called an input device) 100 accepts user operations. The user interface device 100 generates a signal (control signal) in response to the user operation. The user interface device 100 sends the generated signal to the video processing device 200. The user interface device 100 includes an RGB display, a mouse, a keyboard, a touch panel, a microphone, and the like. The user interface device 100 may be included in the configuration of the video processing device 200.

[0027] The display device 300 displays the processing result output from the video processing device 200. The display device 300 is a display device such as an RGB display. The display device 300 may be included in the configuration of the video processing device 200.

[0028] The video processing device 200 may further be connected to other devices, such as external storage (not shown), via the interface 25. Various information, data, and signals are provided to the video processing device 200 from devices external to the video processing device 200 via the interface 25.

[0029] For example, the image processing device 200 of the embodiment may be connected to the hyperspectral camera 900 via the interface 25. This allows the image processing device 200 to communicate with the hyperspectral camera 900.

[0030] The hyperspectral camera 900 is a camera that captures an image by dispersing light from a subject into wavelengths. The hyperspectral camera 900 can acquire information on the wavelength of light from the subject in addition to an image having two-dimensional position information. The hyperspectral camera 900 can provide the image processing device 200 with a hyperspectral image (e.g., a still image) or a hyperspectral video (e.g., a video) captured by dispersing light from the subject into wavelengths.

[0031] For example, the image processing device 200 may be connected to a spectrometer (not shown) such as an optical spectrum analyzer via the interface 25. This allows the image processing device 200 to acquire information about the optical spectrum in the imaging environment of the hyperspectral camera 900.

[0032] The video processing device 200 of this embodiment executes various calculation processes for converting the field of view of a living thing shown in an image or video from the field of view of one living thing to the field of view of another living thing, based on the various data, signals, and parameters supplied, thereby generating an image / video converted from the field of view of one living thing to the field of view of another living thing.

[0033] The video processing device 200 of this embodiment provides a technology that uses hyperspectral images / videos to enable a user to intuitively and continuously visualize information (images / videos corresponding to the subject) obtained from the vision of organisms other than humans (referred to as other organisms) and information obtained from the intermediate vision between humans and other organisms, using the following various configurations and processes.

[0034] 2 is a schematic diagram showing an example of the configuration of a video processing device 200 according to an embodiment of the present invention, which illustrates the software configuration of the video processing device 200 according to an embodiment of the present invention.

[0035] The video processing device 200 of this embodiment is made up of a plurality of software configurations 201, 202, 203, and 204. The video processing device 200 of this embodiment includes a plurality of processing modules (functional blocks) 201 and 202 and a plurality of databases (storage areas) 203 and 204.

[0036] The processing module 201 is an operation user interface (operation UI) 201. The operation UI 201 controls the degree (proportion value) of visibility of other living things in an image according to the visibility of a certain living thing and the visibility of other living things. Hereinafter, the operation UI 201 will be referred to as a visibility degree operation UI 201.

[0037] The visibility degree operation UI 201 provides and displays a user interface for controlling the degree of visibility of other living creatures in an image to the user interface device 100. The visibility degree operation UI 201 receives a control signal (input signal) corresponding to an operation of the user interface from the user interface device 100. The control signal received by the visibility degree operation UI 201 is a parameter operation value.

[0038] The user interface provided by the user interface device 100 includes a configuration (operation unit) that can be operated by the user. For example, a slide bar is displayed on the screen of the user interface device 100 as the operation unit of the user interface.

[0039] FIG. 3 is a schematic diagram showing an example of a display of the operation section of the user interface displayed on the user interface device 100. As shown in FIG.

[0040] 3, the user interface device 100 displays a one-axis slide bar 111 on the screen 110 of the user interface device 100. The slide bar 111 is a user interface that is operated by the user.

[0041] The degree of change from one creature's view to another creature's view is controlled by the user sliding the knob on slide bar 111 .

[0042] 3, the degree of difference between the human field of view and the butterfly field of view in the image displayed on the display device 300 is controlled. For example, by setting the knob of the slide bar 111 to the human side, an image with a greater degree of human field of view is displayed. For example, by setting the knob of the slide bar 111 to the butterfly side, an image with a greater degree of butterfly field of view is displayed. For example, by setting the knob of the slide bar 111 to an intermediate position between the human field of view and the butterfly field of view, an image showing an intermediate field of view between the human field of view and the butterfly field of view (an intermediate representation of the subject in terms of field of view) is displayed.

[0043] A control signal corresponding to the operation of the slide bar 111 is supplied as a parameter operation value to the visibility degree operation UI 201. Based on the received parameter operation value, the visibility degree operation UI 201 determines a numerical value (α) to be used in a processing module 202 (described later). In this way, the degree of change in visibility from humans to other living creatures can be set by a single-axis parameter based on the operation of the slide bar 111.

[0044] 3 shows a slide bar 111 for changing the degree of visibility between a human and a butterfly, but other creatures may be used as long as they are different creatures. For example, the visibility degree operation UI 201 displays multiple slide bars 111 on the user interface device 100 as operation units of the user interface, depending on the types of other creatures that the user can select. The user selects and operates one slide bar from the multiple slide bars 111 that corresponds to the creature that the user wishes to observe.

[0045] By displaying a user interface such as that shown in Figure 3, the numerical value on one axis shown on the slide bar 111 can be manipulated while visualizing the degree to which human vision (e.g., field of view) is converted to the vision of another living organism, as a percentage such as a range from 0 to 1 (or a range from 0% to 100%).

[0046] The operation unit 111 of the user interface device 100 may be configured as hardware.

[0047] The processing module 202 is a conversion filter 202 that converts an image corresponding to the visual characteristics of one living thing in an image into an image corresponding to the visual characteristics of another living thing. The conversion filter 202 converts an image corresponding to the visual field of one living thing into an image corresponding to the visual field of another living thing. Hereinafter, the conversion filter 202 will be referred to as a living thing visual field conversion filter 202.

[0048] The biological visual field conversion filter 202 receives input image data 400 and optical spectrum data 500 from outside the image processing device 200 .

[0049] Input image data 400 is supplied to the image processing device 200 of the embodiment. The input image data 400 is a hyperspectral image or video. Hereinafter, the hyperspectral image or video will be referred to as a hyperspectral image (or HS image).

[0050] FIG. 4 is a diagram for explaining a hyperspectral image 400 that is input image data.

[0051] The example in FIG. 4A shows one frame of a still or moving hyperspectral image 400 .

[0052] As shown in FIG. 4A, a hyperspectral image 400 is represented by a plurality of pixels 411 arranged in a two-dimensional space, similar to a normal image.

[0053] However, the color characteristics of the subject observed by each pixel 411 are expressed by the intensity value (signal value) S(λ) for each wavelength of light from the subject, as shown in FIG. 4B.

[0054] When the observation results of one pixel 411 in a certain hyperspectral image 400 are graphed, they can be shown as a graph of intensity values ​​S(λ) for each wavelength (spectral component) as shown in FIG. 4(c).

[0055] The optical spectrum data 500 is supplied to the image processing device 200 of the embodiment. The optical spectrum data 500 is data of a standard white light spectrum.

[0056] FIG. 5 is a diagram for explaining an example of a standard white light spectrum 500 that is light spectrum data.

[0057] 5A, the standard white light spectrum 500 is expressed by the intensity value I(λ) for each wavelength spectral component, similar to the data of one pixel 411 in the hyperspectral image 400. When the standard white light spectrum is graphed, it can be shown as a graph of the intensity value I(λ) for each wavelength, as shown in FIG.

[0058] The characteristics (intensity value I(λ)) of the standard white light spectrum 500 become reference values ​​(calibration values, adjustment values) for the intensity value S(λ) of each pixel 411 of the hyperspectral image 400 when converting visual characteristics in the image.

[0059] The biological visibility conversion filter 202 receives data from the database 203 and data from the database 204 .

[0060] The database 203 includes a collection of data related to human color matching functions. The database 204 includes a collection of data related to the spectral sensitivity characteristics of organisms other than humans (other organisms). Hereinafter, the database 203 is also referred to as a color matching function database 203. Hereinafter, the database 204 is also referred to as a spectral sensitivity characteristics database 204.

[0061] FIG. 6 is a diagram for explaining an example of the color matching function.

[0062] Human color matching functions include three functions (graphs) that reflect the characteristics of vision to correspond to the three types of human photoreceptors under bright light: L cones, M cones, and S cones. In the color matching functions of the XYZ color system, humans have color matching functions consisting of a function of the following formula (f1) corresponding to L cones, a function of the following formula (f2) corresponding to M cones, and a function of the following formula (f3) corresponding to S cones.

[0063]

[0064]

[0065]

[0066] In the following, for convenience, formula (f1) will also be expressed as x(λ), formula (f2) as y(λ), and formula (f3) as z(λ).

[0067] The formulas (f1)(x(λ)), (f2)(y(λ)), and (f3)(z(λ)) are functions for finding the tristimulus values ​​XYZ, respectively.

[0068] As shown in FIG. 6A, data on the intensities of x(λ), y(λ), and z(λ) for each wavelength (spectral spectrum) is stored in the database 203.

[0069] As shown in FIG. 6(b), a graph of each function can be obtained by processing by the processor 21 based on FIG. 6(a).

[0070] For example, the color matching function database 203 is stored in the storage 24 .

[0071] FIG. 7 is a diagram for explaining an example of the spectral sensitivity characteristics of other living things.

[0072] If the vision of a certain other living organism includes four spectral sensitivity characteristics, the spectral sensitivity characteristics of the other organism include the characteristic (function) of the following equation (f4), the characteristic of equation (f5), the characteristic of equation (f6), and the characteristic of equation (f7).

[0073]

[0074]

[0075]

[0076]

[0077] In the following, for convenience, formula (f4) will also be expressed as a(λ), formula (f5) as b(λ), formula (f6) as c(λ), and formula (f7) as d(λ).

[0078] In another organism having three spectral sensitivity characteristics, as shown in (a) of Figure 7, data indicating the intensities of each of equations (f4)(a(λ)), (f5)(b(λ)), and (f6)(c(λ)) for each wavelength (spectral spectrum) are stored in database 204.

[0079] As shown in FIG. 7(b), a graph of each function is obtained by processing by the processor 21 based on FIG. 7(a).

[0080] Depending on the number of types of other living things that are the target of field of view conversion, the spectral sensitivity characteristics of multiple other living things are stored in database 204. Note that the number of visions (color vision) of other living things with spectral sensitivity characteristics may differ depending on the species of the other living things.

[0081] When the database 204 stores the spectral sensitivity characteristics of a plurality of other organisms, the spectral sensitivity characteristic of the other organism selected by the user is read from the database 204 in response to the user's operation of the user interface device 100 .

[0082] For example, the spectral sensitivity characteristic database 204 is stored in the storage 24 .

[0083] The living thing visibility conversion filter 202 also receives a value α from the visibility degree operation UI 201. The value α is a parameter that indicates the degree of visibility of other living things (or the degree of visibility of humans) according to a user request made by operating the slide bar 111. The value α is a value that can take a ratio between 0 and 1, for example. When the parameter operation value of the slide bar 111 is input as a percentage to the visibility degree operation UI 201, a process for adjusting the ratio is performed by taking the quotient by 100.

[0084] The biological visibility conversion filter 202 calculates various parameters according to the vision (visibility) of other living organisms based on the hyperspectral image 400, the standard white light spectrum 500, the color matching function, the spectral sensitivity characteristics, and the numerical value (visibility degree) α.

[0085] As a result, the biological field of view conversion filter 202 generates an image that corresponds to the field of view of the other living thing (e.g., an RGB image) or an image that corresponds to the field of view between a human and the other living thing (an image that reflects both the human field of view and the field of view of the other living thing).

[0086] The biological visual field conversion filter 202 sends the generated RGB image to the display device 300. The display device 300 displays the RGB image from the biological visual field conversion filter 202. The image displayed on the display device 300 is an image that has been subjected to spatial integration or temporal integration.

[0087] In this way, the image processing device 200 of this embodiment applies to the image processing device (computer) a biological visual field conversion filter (processing module) 202 that receives a hyperspectral image or hyperspectral video as input and generates an image / video that can be continuously changed from a color matching function that reflects human visual characteristics to the spectral sensitivity characteristics of other organisms. This allows the image processing device 200 of this embodiment to continuously change the RGB image from an RGB image of the visual world seen by humans to an RGB image that simulates the visual world of other organisms.

[0088] Furthermore, the image processing device 200 of this embodiment realizes a visibility degree operation UI 201 that allows intuitive control of the degree of proximity of an image to the visibility of a human or another living thing by a single-axis operation (parameter). This allows the image processing device 200 of this embodiment to perform intuitive operations related to the conversion of the visibility.

[0089] (b) Principles Various principles used in the video processing device of this embodiment will be described.

[0090] (b-1) View Conversion Algorithm of the Biological View Conversion Filter The algorithm of the biological view conversion filter 202 in the video processing device 200 of this embodiment will be described.

[0091] In this embodiment, the wavelength intensity S(λ) for each spectral component in each of the multiple pixels constituting the hyperspectral image (hyperspectral picture or hyperspectral video), the wavelength intensity I(λ) for each spectral component of the standard white light source or reference white point, three color matching functions x(λ), y(λ), z(λ) that reflect the visual field characteristics of humans, and multiple spectral sensitivity characteristics a(λ), b(λ), c(λ), d(λ), ... of other organisms are provided to the organism visual field conversion filter 202 (processor 21).

[0092] Regarding the wavelength intensity I(λ) of each spectral component of the standard white light source or the reference white point, for example, the image is captured so that a standard white plate is included in a part of the input hyperspectral image 400, and the wavelength characteristic S ｗ (λ) can be considered as the wavelength intensity I(λ) of the standard white light spectrum 500.

[0093] Regarding the spectral sensitivity characteristics of other organisms, there are organisms that have only one spectral sensitivity characteristic a(λ), organisms that have two spectral sensitivity characteristics a(λ) and b(λ), organisms that have three spectral sensitivity characteristics a(λ), b(λ), and c(λ), and organisms that have four or more spectral sensitivity characteristics a(λ), b(λ), c(λ), and d(λ).

[0094] In this embodiment, first, an algorithm for organism visual field conversion for other organisms having spectral sensitivity characteristics of three colors or less will be described.

[0095] In the algorithm of the organism visibility conversion filter 202 for other organisms having spectral sensitivity characteristics of three colors or less, first, the wavelength of peak sensitivity (also called peak wavelength) of the color matching function x(λ) (formula (f1)) corresponding to human L cones as shown in formula (f8), the wavelength of peak sensitivity of the color matching function y(λ) (formula (f2)) corresponding to human M cones as shown in formula (f9), and the wavelength of peak sensitivity of the color matching function z(λ) (formula (f3)) corresponding to human S cones as shown in formula (f10) are each calculated. In addition, the wavelength of peak sensitivity of the first spectral sensitivity characteristic a(λ) (formula (f4)) of the other organism as shown in formula (f11), the wavelength of peak sensitivity of the second spectral sensitivity characteristic b(λ) (formula (f5)) of the other organism as shown in formula (f12), and the wavelength of peak sensitivity of the third spectral sensitivity characteristic c(λ) (formula (f6)) of the other organism as shown in formula (f13) can each be determined.

[0096]

[0097]

[0098]

[0099]

[0100]

[0101]

[0102] In the following description, the peak wavelengths of the color matching functions in formulas (f8) to (f10) are expressed as x ｍａｘ , y ｍａｘ , z ｍａｘ The peak wavelengths of the spectral sensitivity characteristics in the formulas (f11) to (f13) are expressed as follows for convenience: ｍａｘ , b ｍａｘ , c ｍａｘ are also written as

[0103] Next, the wavelength x in formulas (f8) to (f10) ｍａｘ , y ｍａｘ , z ｍａｘ and the wavelength a ｍａｘ , b ｍａｘ , c ｍａｘ The pair including one of the sets x and y is determined as the first pair, and the two pairs with the smallest difference in peak wavelength are determined as the first pair. ｍａｘ , y ｍａｘ , z ｍａｘ and wavelength a ｍａｘ , b ｍａｘ , c ｍａｘ Among the sets including one of the peak wavelengths, excluding the peak wavelengths included in the first pair, the two pairs having the second smallest difference in peak wavelength are designated as the second pair, and the remaining two pairs are designated as the third pair.

[0104] In the following description, the first pair is x ｍａｘ and a ｍａｘ and the second pair is y ｍａｘ and b ｍａｘ and the third pair is z ｍａｘ and C ｍａｘ It is assumed that the

[0105] In addition, with regard to biological visual field conversion for other organisms with spectral sensitivity characteristics of two or less colors, if the spectral sensitivity of the other organism is one color, the spectral sensitivity characteristics b(λ) and c(λ) are set to zero (0). Also, if the spectral sensitivity of the other organism is two colors, the spectral sensitivity characteristic c(λ) is set to zero (0). In this way, by substituting the following descriptions, biological visual field conversion for other organisms with spectral sensitivity characteristics of two or less colors can be realized.

[0106] In the biological visual field conversion filter 202 of this embodiment, new tristimulus values ​​X'Y'Z' are calculated using a numerical value α ranging from 0 to 1 according to the following equations (f14), (f15), (f16) and (f17).

[0107]

[0108]

[0109]

[0110]

[0111] In the above formulas (f14) to (f17), when the value of α, which indicates the degree of field of view conversion, is 1, an image expressed in the CIE XYZ color space of the world seen by a person with trichromatic vision (the human visual world) is displayed. On the other hand, when the value of α is 0, an image expressed in the CIE XYZ color space of a world that simulates the visual world of other living things is displayed. Furthermore, when the value of α is continuously changed from 1 to 0 (or from 0 to 1), an intermediate image between an image corresponding to the human visual world and an image corresponding to the visual world of other living things according to the value of α is obtained.

[0112] As a result, the video processing device 200 of this embodiment can continuously obtain images that simulate the visual world of other living things from images of the visual world seen by humans.

[0113] A method for converting an image in the CIE XYZ color space, in which each pixel in one frame is expressed by three values ​​of X', Y', and Z', into an RGB image in the sRGB space used in various displays, etc., is carried out using the following equations (f18), (f19), (f20), and (f21).

[0114]

[0115]

[0116]

[0117]

[0118] In the algorithm of the biological visibility conversion filter 202 for other organisms with spectral sensitivity characteristics of three colors or less, the values ​​of X', Y', and Z' of each pixel of the hyperspectral image calculated by equations (f14) to (f17) are calculated by making the value of α continuously controllable from 1 to 0 (or 0 to 1) using a GUI such as a slide bar in the visibility degree operation UI 201.

[0119] Furthermore, the calculated X', Y', and Z' values ​​are converted into R, G, and B values ​​by equations (f18) to (f21).

[0120] As a result, the video processing device 200 of this embodiment realizes continuous change from RGB images / videos of the visual world seen by humans to RGB images / videos that simulate the visual world of other living things.

[0121] Next, an algorithm for a biological visibility conversion filter for other organisms having spectral sensitivity characteristics of four or more colors will be described.

[0122] First, for each of the human color matching functions x(λ), y(λ), z(λ) in the above formulas (f1) to (f3) and the spectral sensitivity characteristics a(λ), b(λ), c(λ), d(λ), ... of each organism in the above formulas (f4) to (f7), the wavelength of peak sensitivity x ｍａｘ , y ｍａｘ , z ｍａｘ , a ｍａｘ , b ｍａｘ, c ｍａｘ , d ｍａｘ , ... are required.

[0123] Next, wavelength x ｍａｘ , y ｍａｘ , z ｍａｘ and wavelength a ｍａｘ , b ｍａｘ , c ｍａｘ , d ｍａｘ , ..., the pair with the smallest difference in peak wavelength is determined as the first pair. ｍａｘ , y ｍａｘ , z ｍａｘ and wavelength a ｍａｘ , b ｍａｘ , c ｍａｘ , d ｍａｘ For pairs including one of the sets of

[0010] , ..., the pair having the second smallest difference in peak wavelength, excluding the peak wavelength included in the first pair, is determined as the second pair. Similarly, the pair having the third smallest difference in peak wavelength, excluding the peak wavelengths included in the first and second pairs, is determined as the third pair.

[0124] In the case of other organisms with spectral sensitivity characteristics of four or more colors, even after the third pair has been determined, there will still be remaining elements of spectral sensitivity characteristics a(λ), b(λ), c(λ), d(λ), ... that have not been selected as pairs.

[0125] For the peak wavelength of the remaining element, the peak wavelength x of the color matching function ｍａｘ , y ｍａｘ , z ｍａｘ The closest peak wavelength among these is selected as the remaining pair.

[0126] In the following, we consider the cases a(λ), b(λ), c(λ), and d(λ) of other organisms with four-color spectral sensitivity characteristics, and the first pair is x ｍａｘ and a ｍａｘ and the second pair is y ｍａｘ and b ｍａｘ and the third pair is z ｍａｘ and C ｍａｘ and the remaining pair is d ｍａｘ and y ｍａｘ It is assumed that the

[0127] In the case of other living things having spectral sensitivity characteristics of four or more colors, the living thing visibility conversion filter 202 also first generates an RGB image at a certain value α using the first to third pairs of elements.

[0128] The above-mentioned (Equation 14) to (Equation 17) are used as the calculation formulas used when generating RGB images, and if sRGB is used among the multiple types of RGB formats, the above-mentioned (Equation 18) to (Equation 21) can be used.

[0129] Next, one pair of elements from the first to third pairs is replaced with one of the remaining pairs. The replaced pair is determined by determining whether the remaining pair used for replacement has a peak wavelength x ｍａｘ , y ｍａｘ , z ｍａｘ The wavelength is determined according to which peak wavelength of the

[0130] In the hypothetical case of this example, the remaining pair is d ｍａｘ and y ｍａｘ Since it contains the same y ｍａｘ A second pair (b ｍａｘ , y ｍａｘ ) is replaced with the remaining pair.

[0131] An RGB image is generated using (Equation 14) to (Equation 17) after the pair replacement and a process of converting from XYZ representation to RGB representation (when converting to sRGB, (Equation 18) to (Equation 21)).

[0132] Assuming the case of another organism having four-color spectral sensitivity characteristics, the calculation formulas after replacing the remaining pairs in equations (f14) to (f17) are shown below in equations (f22), (f23), (f24), and (f25).

[0133]

[0134]

[0135]

[0136]

[0137] In the case of other organisms with spectral sensitivity characteristics of five or more colors, there will be multiple remaining pairs. However, by performing the same process as equations (f22) to (f25) to replace them one by one, an RGB image is generated in which the first remaining pair is replaced with one of the first to third pairs, an RGB image is generated in which the second remaining pair is replaced with one of the first to third pairs, and so on, and a different RGB image is created for each remaining pair.

[0138] The RGB images of the first to third pairs created based on the above process are referred to as "RGB ｂａｓｅ The RGB image replaced with the remaining pairs is called "RGB ｒ１ ", "RGB ｒ２ ", "RGB ｒ３ ", it is said.

[0139] In the process of calculation processing for generating RGB images, the number of pixels in the x-axis direction and the y-axis direction of each RGB image at the time of generation is the same.

[0140] In the case of other living organisms with the above-mentioned four-color spectral sensitivity characteristics, ｂａｓｅ " and "RGB ｒ１ " are generated.

[0141] In the case of other organisms having spectral sensitivity characteristics of four or more colors, the integration of each image is performed by spatial integration or temporal integration in the organism visibility conversion filter 202 .

[0142] (b-2) Image Integration There are two methods of image integration: spatial integration and temporal integration.

[0143] In spatial integration, RGB ｂａｓｅ and RGB ｒ１ , RGB ｒ２ , RGB ｒ３ , . . . based on a plurality of images, ａｖｅ " image is generated. ａｖｅ is RGB ｂａｓｅ and RGB ｒ１ , RGB ｒ２ , RGB ｒ３, .... is an image in which the average value of the pixels in the x and y coordinates of each image is calculated. ａｖｅ is the final display image.

[0144] Spatial integration does not require complex display processing. Since the displayed image is constant, spatial integration can realize an expression that places less strain on the user viewing the image. Spatial integration can also be expressed in printed materials and the like in addition to display on the display device 300.

[0145] However, spatial integration is RGB ｂａｓｅ and RGB ｒ１ , RGB ｒ２， RGB ｒ３， ...the features of each image are averaged.

[0146] Therefore, spatial integration may make it difficult to distinguish the distinctive features of each image from the differences in color.

[0147] Temporal integration is a presentation technique that presents images to the observer while preserving the distinctive features of each image.

[0148] FIG. 8 is a diagram showing a schematic diagram of an image displayed by temporal integration.

[0149] As shown in FIG. ｂａｓｅ , RGB ｒ１ , RGB ｒ２ Temporal integration of a plurality of images such as 301, 302, 303 can be realized by switching between the displayed images 301, 302, 303 in a time-division manner.

[0150] As a result, the video processing device 200 according to the embodiment can provide the user with an RGB image that reflects the characteristics of the visual field (color vision) of each living thing.

[0151] (c) Operation An example of operation of the video processing device of this embodiment will be described with reference to Figures 9 and 10. The example of operation of the video processing device 200 of this embodiment relates to the video processing method of this embodiment.

[0152] (c-1) Processing of the Visibility Degree Operation UI 201 FIG. 9 is a flowchart for explaining processing of the visibility degree operation UI 201 in response to operations on the user interface in an example of the operation of the video processing device of this embodiment.

[0153] As shown in FIG. 9, manipulation of the user interface is initiated when various processes are performed to change the display of the view of humans and other living beings.

[0154] <ST1> When the field of view is changed between the human field of view and the field of view of another living thing, the slide bar 111 in the screen 110 of FIG. 3 displayed on the user interface device 100 is operated in response to a user request.

[0155] In the video processing device 200, the processor 21 determines whether the value of the slide bar 111, which is a value input by the user, has changed. The visibility degree operation UI 201, which is a processing module, detects whether the value of the slide bar 111 has changed.

[0156] If the value of the slide bar 111 is not changed (NO in ST1), the processor 21 maintains the state of waiting for the user to operate the slide bar 111 (waiting for input).

[0157] <ST2> If the value on the slide bar 111 has been changed (YES in ST1), the processor 21 updates the UI display for the user.

[0158] In the processor 21, the visibility degree operation UI 201 updates the current value of the slide bar 111 and the GUI (Graphical User Interface) position of the slide bar 111 based on the operation of the slide bar 111 so as to provide feedback to the user on the content of the operation.

[0159] <ST3> After updating the UI display, the processor 21 sets the value α according to the value of the slide bar 111 (parameter operation value).

[0160] The visibility degree operation UI 201 calculates the value α based on the value of the operated slide bar 111. The visibility degree operation UI 201 sends the determined value α to the biological visibility conversion filter 202.

[0161] This causes the processor 21 to end the processing by the visibility degree operation UI 201 .

[0162] The processor 21 waits for processing by the visibility degree operation UI 201 until the slide bar 111 in the user interface device 100 is operated again. In response to the slide bar 111 being operated again, the processor 21 executes the processing of steps ST1, ST2, and ST3 described above in the visibility degree operation UI 201.

[0163] (c-2) Processing of the Biological Field of View Conversion Filter 202 The processing of the biological field of view conversion filter 202 in an example of the operation of the video processing device 200 of this embodiment will be described with reference to FIG.

[0164] FIG. 10 is a flowchart showing the processing of the biological visual field conversion filter 202 in an example of the operation of the video processing device 200 of this embodiment.

[0165] <ST10> In the image processing device 200 of this embodiment, the processor 21 receives a hyperspectral image (HS image) 400, which is input image data, and a standard white light spectrum 500, which is optical spectrum data. For example, the HS image 400 and the standard white light spectrum 500 are stored in the RAM 23 or the storage 24.

[0166] In the processor 21, the biological field of view conversion filter 202 reads one frame of the HS image 400. The biological field of view conversion filter 202 reads the standard white light spectrum 500.

[0167] <ST11> The processor 21 determines whether a new value α has been determined. The living organism visibility conversion filter 202 checks whether a new value α has been transmitted from the visibility degree operation UI 201.

[0168] <ST12> If a new value α has been determined (YES in ST11), the processor 21 acquires the new value α. The biological visibility conversion filter 202 receives the value α from the visibility degree operation UI 201. The biological visibility conversion filter 202 updates the value of α used in calculation processing within the filter 202 to the new value.

[0169] <ST13> The processor 21 acquires color matching functions for humans (formulas (f1) to (f3)) and spectral sensitivity characteristics of other organisms (formulas (f4) to (f7)). The processor 21 accesses the storage 24 to acquire data on the color matching functions and the spectral sensitivity characteristics.

[0170] The biological visibility conversion filter 202 receives the corresponding color matching function by referencing a database 203 in the storage 24. The biological visibility conversion filter 202 receives the corresponding spectral sensitivity characteristics of other organisms by referencing a database 204 in the storage 24.

[0171] <ST14> The processor 21 generates an RGB image including an image converted from the human field of view to the field of view of another living thing in accordance with the value α based on the algorithm for converting the field of view of the living thing.

[0172] The biological field of view conversion filter 202 generates an RGB image (a still image of an RGB image) or an RGB video (a moving image of an RGB image) based on data for one frame of a hyperspectral image 400 configured with intensity values ​​S(λ) for each wavelength spectral component, the spectrum I(λ) of the standard white light spectrum 500, the final value α stored in the module 202, human color matching functions x(λ), y(λ), z(λ) referenced from a database 203, and spectral sensitivity characteristics a(λ), b(λ), c(λ), d(λ), ... of other organisms referenced from a database 204. The algorithm for generating an RGB image or an RGB video in the biological field of view conversion filter 202 is based on the above-mentioned formulas (f14) to (f25).

[0173] If a new value α has not been determined (NO in ST11), the biological visual field conversion filter 202 executes the calculation process using the previous value α stored inside the module 202.

[0174] As a result, the biological field of view conversion filter 202 generates an RGB image (an RGB image converted from the human field of view to the field of view of another living thing) that includes an image corresponding to the human field of view and / or an image corresponding to the field of view of another living thing.

[0175] For example, when a living thing with a spectral sensitivity characteristic of three colors or less is selected as the object of field of view conversion, an RGB image is generated. For example, when a living thing with a spectral sensitivity characteristic of four colors or more is selected as the object of field of view conversion and representation by temporal integration is selected, an RGB image with display changes over time, as shown in FIG.

[0176] <ST15> The processor 21 displays the RGB image with the converted field of view to the user.

[0177] The living thing visual field conversion filter 202 sends the generated RGB image to the display device 300. The generated RGB image is displayed on the screen of the display device 300. In this way, an RGB image corresponding to an image corresponding to the human visual field, an RGB image including an image corresponding to the visual field of another living thing, or an RGB image including an image corresponding to a visual field intermediate between the human visual field and the visual field of another living thing is observed by the user in accordance with the user's operation.

[0178] This allows the user to view an image that has been converted from the visual characteristics (field of vision) of humans to the visual characteristics of another living thing.

[0179] After the RGB image display is completed, the processor 21 completes the field of view conversion process for one frame of image. The processor 21 then reads the next frame of data contained in the hyperspectral image and performs the above-described steps ST10 to ST15 on the image contained in the next frame of data.

[0180] The above processing completes the operation example (video processing method) of the video processing device 200 of this embodiment.

[0181] (d) Summary The video processing device 200 of this embodiment includes a processing module (visibility degree operation UI) 201 that controls the degree of field of view conversion from one organism to another, and a processing module (organism field of view conversion filter) 202 that performs various computational processes to generate images / videos that show the field of view conversion from one organism to another.

[0182] Organisms with different visual characteristics than humans, such as bees and butterflies, can sometimes have visual discrimination abilities that humans cannot notice, such as being able to distinguish nectar present in flowers.

[0183] The image processing device 200 of this embodiment takes images / videos from a hyperspectral camera as input and continuously changes the images / videos from the world of human visual characteristics that humans are accustomed to seeing to a world that reflects the visual characteristics of other living organisms other than humans.

[0184] As a result, the video processing device 200 of this embodiment can provide the user with a deeper understanding of the vision of other living things and an intuitive understanding of the differences between the world as seen by humans and the world as seen by other living things.

[0185] The video processing device 200 of this embodiment can intuitively control, by operating on one axis (parameter), whether the image is closer to the field of view of a human or another living thing, and how close it is to that of the human or other living thing.

[0186] This allows the video processing device 200 of this embodiment to perform intuitive operations by the user regarding the transformation of the field of view.

[0187] (e) Others The video processing program PRG of the embodiment causes the respective units 201 and 202 (processor 21) of the video processing device 200 of the embodiment to execute the processes of the plurality of steps ST1, ST2, and ST3 in Fig. 9 described above and the processes of the plurality of steps ST10, ST11, ST12, ST13, ST14, and ST15 in Fig. 10 described above by a computer. As a result, the video processing program PRG of the embodiment can obtain the effects described above.

[0188] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0189] 21: Processor 22: ROM 23: RAM 24: Storage 25: Interface 100: User interface device 200: Image processing device 201: Visibility degree operation UI 202: Biological visibility conversion filter 203, 204: Database 300: Display device 400: Input image data 500: Optical spectrum data

Claims

1. An image processing device comprising: a first processing module that sets a first value in response to a user's operation; and a second processing module that uses the first value and input image data including an image captured by dispersing light from a subject by wavelength to generate an image in which the visual characteristics of a first organism are converted into the visual characteristics of a second organism different from the first organism.

2. The video processing device according to claim 1, further comprising: a first database that stores data relating to color matching functions of the first organism; and a second database that stores data relating to spectral sensitivity characteristics of the second organism, wherein the second processing module further uses the color matching functions and the spectral sensitivity characteristics to generate the image.

3. The image processing device according to claim 2, wherein the second processing module further uses a standard white light spectrum to generate the image.

4. The video processing device according to claim 1, wherein the first processing module displays an interface including a one-axis operation unit, and the first value is set based on the user's operation on the operation unit.

5. The video processing device according to claim 1, wherein the image includes the subject exhibiting visual characteristics between the visual characteristics of the first living thing and the visual characteristics of the second living thing.

6. A video processing method comprising: setting a first value; and using the first value and input image data including an image captured by dispersing light from a subject by wavelength, generating an image in which the visual characteristics of a first organism are converted into the visual characteristics of a second organism different from the first organism.

7. A video processing program that causes a computer to execute the processing by each unit of the video processing device according to any one of claims 1 to 5.

Images