Imaging device and method for imaging

The imaging device addresses the challenge of capturing clear images in turbid water by using hyperspectral data analysis and adaptive wavelength selection to optimize imaging conditions, ensuring high-contrast results.

JP2024158758APending Publication Date: 2024-11-08JVC KENWOOD CORP
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023074264
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-04-28
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing imaging devices struggle to capture clear images in turbid water due to light confusion and reflection caused by underwater particles, as conventional infrared cameras fail to select the optimal wavelength for imaging.

Method used

The imaging device employs an acquisition unit to capture hyperspectral data in multiple wavelength ranges, an evaluation unit to calculate contrast values, a calibration unit to select the wavelength range with the maximum contrast as the calibration wavelength, and an update unit to adjust the wavelength range based on environmental changes or object motion, ensuring optimal imaging conditions.

Benefits of technology

This approach allows for the capture of clear images in turbid water by automatically selecting and adjusting the optimal wavelength, providing high-contrast images without user intervention.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024158758000001_ABST
    Figure 2024158758000001_ABST
Patent Text Reader

Abstract

To provide an imaging device and a method for imaging which can obtain a clear image.SOLUTION: The imaging device according to the present disclosure includes: an acquisition unit for acquiring image data taken by dividing light for each of a plurality of wavelength bands; an evaluation unit for calculating the evaluation value of the image data according to each wavelength band; a calibration unit for executing calibration processing of selecting the wavelength band of a first evaluation value showing the largest value of the evaluation value in the plurality of wavelength bands as a calibration wavelength band; and an update unit for updating the calibration wavelength band in a predetermined case.SELECTED DRAWING: Figure 2
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to an imaging device and an imaging method. [Background technology]

[0002] When taking pictures in murky water, it is difficult to obtain clear images due to the scattering and reflection of light caused by particles that cause the water to become murky. In such an environment, it is considered effective to use an infrared camera, and infrared cameras designed for taking pictures in murky water have been commercialized. However, it is not possible to obtain clear images in any murky water using an infrared camera. Thus, there was a problem in that it was difficult to take pictures of objects in the water in a state where they could be recognized.

[0003] For example, Patent Document 1 listed below discloses a processing device that can easily perform a calibration process to remove the effects of reflection from underwater particles, in which a calibration device equipped with a camera is controlled to change from a first state in which the calibration device is not in the field of view of the camera to a second state in which the calibration device is in the field of view of the camera, and the processing device performs the calibration process based on an image captured by the camera when the calibration device is in the second state. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2022-124250 Summary of the Invention [Problem to be solved by the invention]

[0005] However, although the processing device described in Patent Document 1 above can remove the effects of underwater particles through a calibration process, it is not possible to select the optimal wavelength when the wavelength through which light easily transmits in turbid water depends on the size of the underwater particles, and there is not just one wavelength suitable for imaging, but each wavelength is different.

[0006] In view of the above problems, an object of the present disclosure is to provide an imaging device and an imaging method capable of obtaining clear images. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems and achieve the objective, the imaging device disclosed herein includes an acquisition unit that acquires image data captured by dispersing the image data into a plurality of wavelength ranges, an evaluation unit that calculates an evaluation value of the image data for each of the plurality of wavelength ranges, a calibration unit that executes a calibration process to select, as a calibration wavelength range, a wavelength range of a first evaluation value that indicates the maximum evaluation value in all imageable wavelength bands, and an update unit that updates the calibration wavelength range when a predetermined case occurs.

[0008] In order to solve the above-mentioned problems and achieve the objective, the imaging method disclosed herein includes the steps of acquiring image data captured by dispersing the image data into a plurality of wavelength ranges, calculating an evaluation value of the image data for each of the plurality of wavelength ranges, performing a calibration process in which a wavelength range having a first evaluation value that indicates the maximum evaluation value in all imageable wavelength bands is selected as a calibration wavelength range, and updating the calibration wavelength range when a predetermined case occurs. Effect of the Invention

[0009] According to the present disclosure, it is possible to provide an imaging device and an imaging method capable of obtaining a clear image. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of a scene in which an imaging device according to the present disclosure is used. [Diagram 2] FIG. 2 is a diagram showing an example of the configuration of a first embodiment of an imaging device according to the present disclosure. [Diagram 3] FIG. 3 is a diagram illustrating an example of information stored in the evaluation value storage unit of the imaging device according to the present disclosure. [Figure 4] FIG. 4 is a diagram illustrating an example of the configuration of an imaging section of an imaging device according to the present disclosure. [Diagram 5] FIG. 5 is a flowchart showing the flow of a first embodiment of the imaging method according to the present disclosure. [Figure 6] FIG. 6 is a flowchart showing the flow of the calibration process according to the present disclosure. [Figure 7] FIG. 7 is a flowchart showing the flow of a second embodiment of the imaging method according to the present disclosure. [Figure 8] FIG. 8 is a diagram illustrating the processing of the second aspect of the imaging method according to the present disclosure. [Figure 9] FIG. 9 is a diagram illustrating an example of the configuration of a second embodiment of an imaging device according to the present disclosure. [Figure 10] FIG. 10 is a diagram illustrating the processing of the update unit of the imaging device according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the embodiments described below.

[0012] (Overview of the imaging device) First, an overview of the imaging device 100 according to the present disclosure will be described with reference to FIG. 1. FIG. 1 is a diagram that illustrates an example of a scene in which the imaging device according to the present disclosure is used. As shown in FIG. 1, the imaging device 100 according to the present disclosure can be used, for example, when a user U enters underwater UW and captures the state of the underwater UW. As shown in FIG. 1, the user U uses the imaging device 100 to capture an image of a rock R located on the bottom BW of the underwater UW. In such a case, if there are particles that cause turbidity in the underwater UW, the particles cause light scattering and reflection, making it difficult to obtain a clear image.

[0013] It is known that the size of particles, which affect the scattering and reflection of light, affects the length of the wavelength that is optimal for photography. Specifically, it is known that the wavelength that is optimal for photography, which allows light to easily penetrate through turbid water, depends on the size of the particles. In other words, even for turbid water, there is not just one wavelength that is optimal for photography, but each wavelength is different. For this reason, it has been necessary to set a wavelength that is optimal for photography and capture an image.

[0014] (Configuration of imaging device) First embodiment Next, the configuration of a first embodiment of the imaging device 100 according to the present disclosure will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the configuration of the first embodiment of the imaging device according to the present disclosure. As shown in Fig. 2, the imaging device 100 according to the present disclosure includes a connector unit C, a communication unit 110, a storage unit 120, a control unit 130, and an imaging unit 140.

[0015] The connector section C is a connection section (terminal) for connecting an external device to the imaging device 100. The connector section C includes a video signal line and a control signal line. The control signal line may be, for example, a Universal Serial Bus (USB) connection terminal, and if a higher transmission speed is required, a connection terminal appropriate for the speed may be selected. The video signal line may be, for example, an SDI (Serial Digital Interface).

[0016] The communication unit 110 is, for example, a communication module that allows the imaging device 100 to communicate with an external device such as an image processing device. The communication unit 110 communicates data with the external device by wireless communication or wired communication. In the case of wireless communication, the communication unit 110 may include a communication antenna, an RF (Radio Frequency) circuit, or other circuits for communication processing. In the case of wired communication, the communication unit 110 may include, for example, a LAN (Local Area Network) terminal, a transmission circuit, or other circuits for communication processing.

[0017] The storage unit 120 is a storage device that stores various types of information. The storage unit 120 includes a main storage device and an auxiliary storage device. The main storage device may be realized by a semiconductor memory element such as a random access memory (RAM), a read only memory (ROM), or a flash memory. The auxiliary storage device may be realized by a hard disk, a solid state drive (SSD), an optical disk, or the like.

[0018] 2, storage unit 120 includes evaluation value storage unit 121 and frame buffer unit 122. An example of information stored therein will be described below.

[0019] The evaluation value storage unit 121 stores information related to the evaluation value of the image data. Fig. 3 is a diagram showing an example of information stored in the evaluation value storage unit of the imaging device according to the present disclosure.

[0020] In the example shown in FIG. 3, the evaluation value storage unit 121 stores information relating to the items "frame number," "wavelength range," and "evaluation value" in association with each other.

[0021] The "frame number" indicates the frame number of the hyperspectral data. Here, the hyperspectral data indicates image data captured by dispersing the image into a plurality of wavelength ranges, and will be described later in detail. For example, if the frame rate is 30 fps (frames per second), the number indicates the number counted from the first image out of 30 images captured in one second. The "wavelength range" indicates the wavelength range of the image captured by dispersing the image constituting the hyperspectral data, and can capture images from 400 nm to 1000 nm, for example, from 400 nm to 406 nm if the wavelength resolution is 6 nm. The "evaluation value" indicates the evaluation value (contrast) of the image in the wavelength range indicated by the "wavelength range" of the imaging data indicated by the "frame number". The method of calculating the evaluation value will be described later.

[0022] That is, FIG. 3 shows an example in which an evaluation value "V1-1" calculated for an image in the wavelength range "λ1-1" of imaging data in frame number "F1" is stored.

[0023] The evaluation value storage unit 121 is not limited to information relating to the items "frame number," "wavelength range," and "evaluation value," and may store any other information relating to evaluation values.

[0024] The frame buffer unit 122 is a storage area in which image data generated by the imaging unit 140 is temporarily stored. The frame buffer unit 122 may secure a dedicated area in a part of a RAM (Random Access Memory) serving as a main storage device, or may use a dedicated storage device separate from the main storage device. The acquisition unit 131 of the control unit 130 causes the frame buffer unit 122 to sequentially store the image data while causing the imaging unit 140 to capture an image.

[0025] The control unit 130 is a controller that manages and controls the imaging device 100. The control unit 130 is realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like executing various programs stored in the storage unit 120 using a RAM as a working area. The control unit 130 may also be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0026] As shown in Fig. 2, the control unit 130 includes an acquisition unit 131, an evaluation unit 132, a calibration unit 133, and an update unit 134. The control unit 130 realizes these functions and executes these processes by reading and executing a program (software) from the storage unit 120. The control unit 130 may be realized by an electronic circuit. The control unit 130 may execute these processes by one CPU, or may include multiple CPUs and execute these processes in parallel. Below, these configurations will be explained in order.

[0027] The acquisition unit 131 acquires various types of information from inside or outside the imaging device 100. For example, the acquisition unit 131 acquires hyperspectral data in which image data is captured by dispersing light into a plurality of wavelength ranges. The acquisition unit 131 acquires hyperspectral data captured by the imaging unit 140, which will be described later.

[0028] The evaluation unit 132 calculates an evaluation value of the image data for each of a plurality of wavelength ranges. That is, the evaluation unit 132 sequentially calculates an evaluation value of the image data for each wavelength range, and stores the calculated evaluation value in the evaluation value storage unit 121 in association with the wavelength range. Specifically, the evaluation unit 132 calculates contrast as the evaluation value, which is calculated by the following formula (1). If the maximum light intensity of a pixel in a screen is Imax and the minimum light intensity is Imin, the contrast CTR is expressed by the following formula (1).

[0029]

number

[0030] The calibration unit 133 executes a calibration process to select the wavelength range of the first evaluation value, which indicates the maximum evaluation value in a plurality of wavelength bands, as the calibration wavelength range. The process of the calibration unit 133 will be specifically described. First, the calibration unit 133 acquires one frame of hyperspectral data. At this time, the internal counter i is set to 0. i indicates the number of the divided wavelength range, and for example, when the wavelength resolution is 6 nm and the range of 400 nm to 1,000 nm can be photographed, the number of divisions is 100. i takes a value from 0 to 99, and the wavelength range used when i=0 is 400 nm to 406 nm. When i=99, the image data is photographed in the wavelength range from 994 nm to 1,000 nm. While incrementing i, the evaluation values ​​of the images of all bands are compared in sequence. Then, the calibration unit 133 sets the first evaluation value, which is the maximum evaluation value, and the wavelength range at that time as the calibration wavelength range. The calibration unit 133 may execute a calibration process to select, as a calibration wavelength range, a wavelength range having a first evaluation value that indicates a maximum evaluation value among a plurality of wavelength bands.

[0031] The update unit 134 updates the calibration wavelength range when a predetermined case occurs. Here, the predetermined case refers to the following case. The predetermined case in which the update unit 134 updates the calibration wavelength range will be specifically described below.

[0032] The update unit 134 performs the calibration process to update the calibrated wavelength range when the ratio between the first evaluation value and the second evaluation value, which indicates the evaluation value of the calibrated wavelength range in the image data captured after the calibration process, exceeds a first threshold value. The process of the update unit 134 will be specifically described. First, it is assumed that the calibration unit 133 performs the calibration process to obtain the calibrated wavelength range λc and its evaluation value Vc. The update unit 134 acquires one frame of hyperspectral data and extracts image data of the calibrated wavelength range λc from the data. The update unit 134 then calculates the evaluation value Vx of the image data and compares it with the evaluation value Vc obtained by the calibration process. If the difference (Vc-Vx) or the quotient (Vc / Vx) exceeds a first threshold value set in advance, the update unit 134 determines that the current wavelength is no longer optimal due to a change in the environment, and performs the calibration process again. Note that whether the difference or the quotient is used is determined by the parameters used for the evaluation, and either one or both can be used. The parameters to be used include the contrast accumulation value of one frame, the accumulation value of high-frequency components, etc. The above-mentioned first threshold value may be set arbitrarily. For example, it may be set by trial and error, in which the first threshold value is set and then the acquired image data is visually checked to determine the sharpness.

[0033] In addition, when the difference between the second evaluation value indicating the evaluation value of the calibrated wavelength range in the image data captured after the calibration process and the evaluation value of the wavelength range adjacent to the calibrated wavelength range exceeds a predetermined threshold value, the update unit 134 may set the adjacent wavelength range exceeding the second evaluation value as the calibrated wavelength range. Such processing by the update unit 134 will be specifically described. First, as in the above-mentioned case, it is assumed that the calibration process is performed by the calibration unit 133 and the calibrated wavelength range λc and its evaluation value Vc are obtained. The update unit 134 acquires one frame of hyperspectral data and extracts image data of the calibrated wavelength range λc from the data. Then, the update unit 134 acquires the evaluation value of this image data and the evaluation values ​​(Vxl, Vxr) of the adjacent wavelength ranges (λxl, λxr). Here, λxl is a wavelength range adjacent to the calibrated wavelength range λx that is smaller than λx, and λxr is a wavelength range adjacent to the calibrated wavelength range λx that is larger than λx. Then, the update unit 134 compares Vx, Vxl, and Vxr to determine which of them is the maximum value. When Vx is the maximum value, the current wavelength is determined to be optimal and the next frame is photographed. When Vxl is the maximum value, if the value of Vxl-Vx exceeds a predetermined threshold, Vxl is determined to be the optimal wavelength and λxl is newly selected as the calibrated wavelength range. If the predetermined threshold is not exceeded, λx is also selected as the calibrated wavelength range for the next frame. When Vxr is the maximum value, if the maximum value of Vxr-Vx exceeds a predetermined threshold, λxr is determined to be the optimal wavelength and λxr is newly selected as the calibrated wavelength range. If the predetermined threshold is not exceeded, λx is also adopted for the next frame. Note that the evaluation was performed using the difference in the evaluation values ​​in the above, but it may also be determined using the ratio of the evaluation values. In addition, the above-mentioned predetermined threshold may be set arbitrarily. For example, after setting the predetermined threshold, it may be set by trial and error in which the acquired image data is visually confirmed and the sharpness is determined. Note that λxl and λxr do not have to be wavelength ranges adjacent to the calibrated wavelength range λx, and may be wavelength ranges separated from λx by a predetermined wavelength range.

[0034] As described above, the update unit 134 updates the calibration wavelength range when a predetermined case occurs. The update unit 134 may determine whether a predetermined case occurs and have the calibration unit 133 perform a calibration process to update the calibration wavelength range.

[0035] The imaging section 140 generates hyperspectral data in which image data is captured by dispersing light into a plurality of wavelength ranges. Fig. 4 is a diagram showing an example of the configuration of the imaging section of the imaging device according to the present disclosure. As shown in Fig. 4, the imaging section 140 includes an objective lens 141, an optical system 142, an imaging element 143, and an image generating section 144. The configuration of these components will be described below in order.

[0036] Light from a subject is incident on the objective lens 141, which irradiates the incident light onto the optical system 142. The objective lens 141 is driven by a focus drive unit. The focus drive unit is, for example, an actuator, and drives the objective lens 141. The focus drive unit determines whether the light dispersed by the optical system 142 is focused on the surface of the image sensor 143, and drives the objective lens 141 so that the light is focused on the surface of the image sensor 143.

[0037] The optical system 142 has, for example, a slit section, a diffraction grating, and a lens. The slit section transmits light from the subject as linear light extending in the X-axis direction (first direction) and causes the light to be incident on the diffraction grating. The slit section uses an opening having a fine width to narrow down the light that has passed through the objective lens 141, and irradiates the narrowed down light on the diffraction grating. The slit section may be an electronic slit realized by a ferroelectric liquid crystal. This enables high-speed scanning of the subject by high-speed switching of the slit position.

[0038] The diffraction grating separates the light that has passed through the slit into multiple wavelength regions in the Y direction and makes it incident on the lens. The diffraction grating is, for example, a substrate on which a large number of parallel grooves are formed at equal intervals, and separates the lines of light that have passed through the slits in the X-axis direction into multiple wavelength regions. For example, a transmission type diffraction grating can be used as the diffraction grating. Note that a reflective type diffraction grating such as a mechanically ruled diffraction grating or a holographic diffraction grating may also be used as the diffraction grating.

[0039] The imaging element 143 is an element that converts light incident through the optical system 142 into an image signal, which is an electrical signal. The imaging element 143 may be, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0040] The image generating unit 144 generates hyperspectral data (HSD: HyperSpectral Data) for each frame from the image signal generated by the imaging element 143. The hyperspectral data is data in which an image in which light intensity is associated with each pixel in the X direction and the Y direction in one frame is layered for each spectral wavelength band. For example, when 400 nm to 1,000 nm can be captured and the wavelength resolution is 6 nm, data is obtained in which 100 images are connected in layers. The image generating unit 144 may be realized by a program as software, or may be realized by a digital circuit as hardware. Also, it may be realized by an FPGA as hardware. The hyperspectral data may include data of all wavelengths that can be captured, may include data of multiple wavelengths, or may include data of one wavelength.

[0041] According to the first embodiment of the imaging device 100 described above, an evaluation value is calculated for each wavelength range, and the wavelength range in which the evaluation value is maximum is selected as the calibration wavelength range for imaging. As a result, even in an environment such as turbid water where there is an influence of scattering due to particles, a clear image can be obtained by photographing at an optimal wavelength. Therefore, a user can automatically photograph at an optimal wavelength without being aware of the optimal wavelength. Therefore, an imaging device 100 capable of obtaining a clear image can be provided.

[0042] (About the imaging method) Next, an imaging method according to the present disclosure will be described with reference to Fig. 5 to Fig. 8. Fig. 5 is a flowchart showing the flow of a first embodiment of the imaging method according to the present disclosure. As shown in Fig. 5, the first embodiment of the imaging method is composed of, for example, steps S101 to S106. Hereinafter, the first embodiment of the imaging method according to the present disclosure will be described with reference to the flow shown in Fig. 5.

[0043] First, the imaging device 100 executes a calibration process (step S101). The calibration process here refers to the calibration process executed by the calibrator 133 described above. The calibration process will be described in more detail later. Next, the imaging device 100 stores the wavelength range (λc) and evaluation value (Vc) to be used selected by the calibration process (step S102). Next, the imaging device 100 captures one frame of hyperspectral data (step S103). Next, the imaging device 100 extracts image data composed of the wavelength range (λc) from the hyperspectral data (step S104). Next, the imaging device 100 calculates an evaluation value (Vx) of the image data in the wavelength range (λc) (step S105). Next, the imaging device 100 determines whether the difference between Vc and Vx, or the value of Vc / Vx, exceeds a first threshold value (step S106). If the difference between Vc and Vx, or the value of Vc / Vx, exceeds the first threshold value (step S106: Yes), the imaging device 100 returns to step S101 and executes the subsequent processes.

[0044] If the difference between Vc and Vx or the value of Vc / Vx does not exceed the first threshold value (step S106: No), the imaging device 100 returns to step S103 and executes the subsequent processes.

[0045] Also, in step S103, instead of acquiring hyperspectral data, image data captured at λc may be acquired, step S104 may be skipped, and an evaluation value may be calculated in step S105.

[0046] The processes from steps S104 to S106 may be performed for each frame, once every few frames, or once per predetermined time unit. In step S105, evaluation values ​​may be calculated at wavelengths other than λc, or evaluation values ​​may be calculated at multiple wavelengths.

[0047] According to this, when the difference or quotient between the evaluation value of the image in the calibrated wavelength range after the calibration process is performed and the evaluation value of the image in the calibrated wavelength range at the time of the calibration process exceeds a first threshold value, the calibration process can be performed again to update the calibrated wavelength range, thereby making it possible to obtain a clear image, i.e., an image with high contrast.

[0048] Next, the calibration process according to the present disclosure will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the flow of the calibration process according to the present disclosure. As shown in Fig. 6, the calibration process is composed of, for example, steps S201 to S208. Hereinafter, the calibration process according to the present disclosure will be described along the flow shown in Fig. 6.

[0049] First, the imaging device 100 captures one frame of hyperspectral data (step S201). Next, the imaging device 100 sets an internal counter i=0 (step S202). Next, the imaging device 100 determines whether i is less than the wavelength division number (step S203). If i is less than the wavelength division number (step S203: Yes), the imaging device 100 calculates an evaluation value Vi from an image configured in the i-th wavelength band λi (step S204). Next, the imaging device 100 determines whether Vi is greater than Vt (step S205). Here, Vt indicates the maximum evaluation value of images in the wavelength bands up to the i-th, and is set to 0 when i=0. If Vi is greater than Vt (step S205: Yes), the imaging device 100 sets Vt=Vi, λt=λi (step S206). Here, Vt indicates the wavelength band that is the maximum evaluation value of images in the wavelength bands up to the i-th. Next, the imaging device 100 sets the internal counter i to i+1 (step S207). Next, the imaging device 100 returns to step S203 and executes the subsequent processes.

[0050] In step S203, if i is not less than the number of wavelength divisions (step S203: No), the image pickup device 100 sets Vc=Vt and λc=λt (step S208), where Vc indicates the maximum evaluation value obtained by the calibration process, and λc indicates the wavelength range in which the evaluation value obtained by the calibration process is maximum.

[0051] This makes it possible to execute a calibration process in which the wavelength range in which the evaluation value of the image is maximized among the multiple wavelength ranges obtained by dispersion is set as the calibration wavelength range.

[0052] Next, a second embodiment of the imaging method according to the present disclosure will be described with reference to Fig. 7. Fig. 7 is a flowchart showing the flow of the second embodiment of the imaging method according to the present disclosure. As shown in Fig. 7, the second embodiment of the imaging method is composed of, for example, steps S301 to S310. Hereinafter, the second embodiment of the imaging method according to the present disclosure will be described along the flow shown in Fig. 7.

[0053] First, the imaging device 100 executes a calibration process (step S301). Next, the imaging device 100 sets λc to the wavelength range (λx) to be used (step S302). Next, the imaging device 100 captures one frame of hyperspectral data (step S303). Next, the imaging device 100 extracts image data configured in the wavelength range (λx) (step S304). Next, the imaging device 100 calculates evaluation values ​​(Vx, Vxl, Vxr) of image data in the wavelength range (λx) and the adjacent wavelength ranges (λxl, λxr) (step S305). Next, the imaging device 100 determines which of the evaluation values ​​(Vx, Vxl, Vxr) is the maximum (step S306). If Vx is the maximum in step S306, the imaging device 100 returns to step S303 and executes the subsequent processes.

[0054] In step S306, if Vxl is the maximum, the imaging device 100 determines whether the difference between Vxl and Vx exceeds a predetermined threshold value (step S307). If the difference between Vxl and Vx exceeds the predetermined threshold value (step S307: Yes), the imaging device 100 sets λx to λxl (step S308). Then, the imaging device 100 returns to step S303 to execute the subsequent processes. Note that, even if the difference between Vxl and Vx does not exceed the predetermined threshold value in step S307 (step S307: No), the imaging device 100 returns to step S303 to execute the subsequent processes. Note that the threshold value in this case may be set arbitrarily.

[0055] In step S306, if Vxr is the maximum, the imaging device 100 determines whether or not the difference between Vxr and Vx exceeds a predetermined threshold value (step S309). If the difference between Vxr and Vx exceeds the predetermined threshold value (step S309: Yes), the imaging device 100 sets λx to λxr (step S310). Then, the imaging device 100 returns to step S303 to execute the subsequent processes. Note that, even if the difference between Vxr and Vx does not exceed the predetermined threshold value in step S309 (step S309: No), the imaging device 100 returns to step S303 to execute the subsequent processes. Note that the threshold value in this case may be set arbitrarily.

[0056] Here, the process of the second embodiment of the imaging method according to the present disclosure will be described with reference to Fig. 8. Fig. 8 is a diagram for explaining the process of the second embodiment of the imaging method according to the present disclosure. As shown in Fig. 8, if the evaluation value Vx of the calibrated wavelength range set by the calibration process is a maximum value, the calibrated wavelength range is not changed and hyperspectral data of the next frame is acquired. On the other hand, if the difference between the evaluation value Vxl in the wavelength range λxl adjacent to the calibrated wavelength range or the evaluation value Vxr in the wavelength range λxr and the evaluation value Vx in the calibrated wavelength range λx exceeds a predetermined threshold, the calibrated wavelength range is reset to λxl, λxr, respectively, and hyperspectral data of the next frame is acquired.

[0057] This eliminates the need to perform calibration processing for each frame and calculate evaluation values ​​in all wavelength ranges, allowing the imaging processing to be performed quickly, thereby providing an imaging method that can obtain clear images.

[0058] (Configuration of imaging device) Second Embodiment Next, the configuration of the second embodiment of the imaging device 100 according to the present disclosure will be described with reference to FIG. 9. FIG. 9 is a diagram showing a configuration example of the second embodiment of the imaging device according to the present disclosure. As shown in FIG. 9, the second embodiment of the imaging device 100 according to the present disclosure includes a connector unit C, a communication unit 110, a storage unit 120, a control unit 130, an imaging unit 140, and an inertial sensor unit 150. The second embodiment of the imaging device 100 according to the present disclosure is the same as the configuration of the first embodiment of the imaging device 100, except that the processing of the acquisition unit 131 and the update unit 134 is different from that of the first embodiment of the imaging device 100, and an image processing unit 135, a division calibration unit 136, and an inertial sensor unit 150 are added. Therefore, among the configurations of the second embodiment of the imaging device 100 according to the present disclosure, configurations different from the configuration of the first embodiment of the imaging device 100 will be described.

[0059] The acquisition unit 131 acquires image data captured using a calibrated wavelength range set for each divided region. For example, an evaluation value is calculated for each divided region of the screen, and when the wavelength range in which the evaluation value is maximized is different, the wavelength range in which the evaluation value is maximized is set as the calibrated wavelength range and image data captured is acquired from the imaging unit 140. In this case, the screen may be divided by any method, but for example, the screen may be divided into three parts, top and bottom.

[0060] When the magnitude of the motion vector of the photographed object is greater than the second threshold value, or the degree of motion based on the inertial sensor unit 150 is greater than the third threshold value, the update unit 134 performs a calibration process to update the calibrated wavelength range. That is, when the magnitude of the motion vector of the object detected by the image processing unit 135 described later is greater than the second threshold value, or the degree of motion based on the inertial sensor unit 150 described later is greater than the third threshold value, the update unit 134 performs a calibration process based on the image data captured after the calibration process to update the calibrated wavelength range. The third threshold value may be set arbitrarily. For example, after setting the third threshold value, the acquired image data may be visually checked to determine the sharpness by trial and error. This allows an appropriate wavelength range to be set as the calibrated wavelength range when the object moves significantly or the image capture device 100 moves significantly.

[0061] The update unit 134 may determine whether the magnitude of the motion vector of the photographed object is greater than a second threshold value or whether the degree of motion based on the inertial sensor unit 150 is greater than a third threshold value, and may have the calibration unit 133 perform a calibration process to update the calibrated wavelength range.

[0062] Alternatively, when the magnitude of the motion vector of the photographed object is greater than the second threshold value, or the degree of motion based on the inertial sensor unit 150 is greater than the fourth threshold value, the update unit 134 moves the area including the object according to the motion of the object, and selects the wavelength range selected for the area including the object as the calibration wavelength range in the area after the movement. That is, when the magnitude of the motion vector of the object detected by the image processing unit 135 described later is greater than the second threshold value, or the degree of motion based on the inertial sensor unit 150 described later is greater than the fourth threshold value, the update unit 134 moves the area including the object according to the motion of the object, and selects the wavelength range selected for the area including the object as the calibration wavelength range in the area after the movement. The aforementioned fourth threshold value may be set arbitrarily. For example, the fourth threshold value may be set by trial and error, in which the acquired image data is visually checked and the sharpness is determined after the fourth threshold value is set.

[0063] Such processing of the update unit 134 will be described with reference to FIG. 10. FIG. 10 is a diagram for explaining processing of the update unit of the imaging device according to the present disclosure. As shown in FIG. 10, image data IMG1 of frame number F1 is divided into an area D1 of the calibrated wavelength range λ1, an area D2 of the calibrated wavelength range λ2, and an area D3 of the calibrated wavelength range λ3. An object (e.g., an apple) is detected in the area D2 of the image data IMG1. In contrast, it is assumed that the apple has moved downward in the image data IMG2 of frame number F2 with the movement of the imaging device. In such a case, the update unit 134 moves the area of ​​the calibrated wavelength range of each area downward in response to the movement of the apple, as shown in FIG. 10. That is, the calibrated wavelength range λ2 selected for the area D2 including the apple is selected as the calibrated wavelength range in the area D2 after the movement.

[0064] The image processing unit 135 executes image recognition processing based on the image data to obtain the magnitude of the motion vector of the detected object. For example, the image processing unit 135 detects the object by using semantic segmentation, which is a method of classifying pixels of the image data according to which object class they belong to, and classifies the detected object. Then, the image processing unit 135 obtains the magnitude of the motion vector from the amount of movement of the detected object between frames. For example, optical flow may be used as a method for obtaining the magnitude of the motion vector. The optical flow estimates the motion vector by a gradient method, a block matching method, or the like.

[0065] The division calibration unit 136 divides the screen into predetermined regions, and selects the wavelength range in which the evaluation value is maximum for each region as the calibration wavelength range based on the evaluation value for each wavelength range calculated for each divided region. For example, the division calibration unit 136 divides the screen into three regions, upper and lower. For example, when the imaging device 100 is used underwater to capture an image, the lower region corresponds to the seabed, the central region corresponds to seawater, and the upper region corresponds to the sea surface. Then, the division calibration unit 136 selects the wavelength range in which the evaluation value is maximum for each region as the calibration wavelength range based on the evaluation value calculated by the evaluation unit 132 for each divided region. That is, in the example of dividing the screen into three regions, upper and lower, the wavelength range in which the evaluation value is maximum is selected as the calibration wavelength range based on the evaluation values ​​calculated for the upper region, the central region, and the lower region.

[0066] The inertial sensor unit 150 measures information on the degree of movement of the imaging device 100. The inertial sensor unit 150 may be, for example, an IMU (Inertial Measurement Unit) that combines a three-axis gyro sensor and a three-axis acceleration sensor. The gyro sensor generates a primary vibration that vibrates in one direction in a movable electrode, and when the movable electrode is rotated, a Coriolis force acts in a direction 90° from the vibration direction to generate a secondary vibration, which causes a change in capacitance. The gyro sensor may be realized by a capacitive MEMS (Micro Electro Mechanical Systems) gyro sensor that detects this. The acceleration sensor may be, for example, a capacitive acceleration sensor that creates a movable electrode and a fixed electrode using MEMS, and measures acceleration using the relationship between the change in capacitance caused by the movement of the movable electrode and the acceleration.

[0067] The inertial sensor unit 150 calculates the degree of movement based on the three-axis angular velocities and three-axis accelerations measured by the three-axis gyro sensor and the three-axis acceleration sensor. The degree of movement may indicate the magnitude of movement, and may be, for example, the sum of the square root of the sum of the squares of the three-axis accelerations and the square root of the sum of the squares of the three-axis angular velocities.

[0068] According to the second embodiment of the imaging device 100 described above, it is possible to detect an object and update the calibrated wavelength range according to the motion vector or degree of motion of the object, or to divide the screen and set the calibrated wavelength range for each divided area. Therefore, it is possible to set an appropriate wavelength range as the calibrated wavelength range according to changes in the imaging environment. Therefore, it is possible to provide an imaging device 100 that can obtain clear images.

[0069] (Composition and Effects) The imaging device 100 according to the present disclosure includes an acquisition unit 131 that acquires hyperspectral data in which image data is captured by dispersing the image data into a plurality of wavelength ranges, an evaluation unit 132 that calculates an evaluation value of the image data for each of the plurality of wavelength ranges, a calibration unit 133 that executes a calibration process to select a wavelength range of a first evaluation value that indicates the maximum evaluation value in the plurality of wavelength bands as a calibration wavelength range, and an update unit 134 that updates the calibration wavelength range when a predetermined case occurs.

[0070] According to this configuration, an evaluation value is calculated for each wavelength range, and the wavelength range in which the evaluation value is maximum is selected as the calibrated wavelength range for imaging. As a result, even in an environment such as turbid water where there is an influence of scattering due to particles, a clear image can be obtained by imaging at the optimal wavelength. Therefore, a user can automatically image at the optimal wavelength without being aware of the length of the optimal wavelength. Therefore, an imaging device 100 capable of obtaining a clear image can be provided.

[0071] The update unit 134 of the imaging device 100 according to the present disclosure performs a calibration process to update the calibrated wavelength range when a ratio between a first evaluation value and a second evaluation value indicating an evaluation value of the calibrated wavelength range in image data captured after the calibration process exceeds a first threshold value.

[0072] According to this configuration, when the ratio between the second evaluation value of the calibrated wavelength range of the image data captured after the calibration process and the first evaluation value exceeds the first threshold value, the calibration process can be re-executed to update the calibrated wavelength range. Therefore, when the second evaluation value of the calibrated wavelength range of the image data after the calibration process deviates from the first evaluation value at the time of the calibration process due to an environmental change, the calibration process can be re-executed to update the calibrated wavelength range. Therefore, it is possible to provide an imaging device 100 that can obtain clear images.

[0073] The imaging device 100 according to the present disclosure further includes an image processing unit 135 that performs an image recognition process based on the image data to determine the magnitude of the motion vector of the detected object, and the update unit 134 performs a calibration process to update the calibrated wavelength range when the magnitude of the motion vector of the object is greater than a second threshold value or when the degree of movement based on the inertial sensor unit 150 is greater than a third threshold value.

[0074] According to this configuration, when the magnitude of the motion vector of the image data is greater than the second threshold value or the degree of motion is greater than the third threshold value, the calibration process can be re-executed to update the calibrated wavelength range. Therefore, when there is concern about deterioration of transparency due to a large movement of objects underwater or a large movement of the imaging device 100, it is possible to re-execute the calibration process and perform imaging using an optimal wavelength range. Therefore, it is possible to provide an imaging device 100 that can obtain clear images.

[0075] The imaging device 100 according to the present disclosure further includes a divided calibration unit 136 that divides the screen into predetermined regions and selects, for each divided region, the wavelength range with the maximum evaluation value based on the evaluation value for each wavelength range calculated for each divided region as the calibrated wavelength range, wherein the acquisition unit 131 acquires image data captured using the calibrated wavelength range set for each divided region, and when the magnitude of the object's motion vector is greater than a second threshold value or the degree of movement based on the inertial sensor unit 150 is greater than a fourth threshold value, the update unit 134 moves the region containing the object in accordance with the movement of the object and selects the wavelength range selected for the region containing the object as the calibrated wavelength range for the region after the movement.

[0076] According to this configuration, the screen is divided, a calibration wavelength range is set for each divided area, and when the movement of the object in the divided area is large or the degree of movement based on the inertial sensor unit 150 is large, the area including the object is moved according to the movement of the object, and the wavelength range set in the area can be selected as the calibration wavelength range. Therefore, when the movement of the object is large or the movement of the imaging device 100 is large, an appropriate wavelength range can be set as the calibration wavelength range according to the movement of the object. Therefore, it is possible to provide an imaging device 100 that can obtain a clear image.

[0077] The imaging method according to the present disclosure includes the steps of acquiring hyperspectral data in which image data is captured by dispersing the image data into a plurality of wavelength ranges, calculating an evaluation value of the image data for each of the plurality of wavelength ranges, performing a calibration process in which a wavelength range of a first evaluation value that indicates the maximum evaluation value in all wavelength bands that can be captured is selected as a calibrated wavelength range, and updating the calibrated wavelength range when a predetermined case occurs.

[0078] According to this configuration, an evaluation value is calculated for each wavelength range, and the wavelength range in which the evaluation value is maximum is selected as the calibrated wavelength range for imaging. As a result, even in an environment such as turbid water where there is an influence of scattering due to particles, a clear image can be obtained by imaging at the optimal wavelength. Therefore, a user can automatically image at the optimal wavelength without being aware of the length of the optimal wavelength. Therefore, an imaging method capable of obtaining a clear image can be provided.

[0079] Although the embodiment of the present invention has been described above, the embodiment is not limited to the contents of this embodiment. The above-mentioned components include those that a person skilled in the art can easily imagine, those that are substantially the same, and those that are within the so-called equivalent range. Furthermore, the above-mentioned components can be appropriately combined. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the gist of the above-mentioned embodiment. [Explanation of symbols]

[0080] 100 Imaging device 110 Communications Department 120 Storage section 121 Evaluation value storage unit 122 Frame Buffer Section 130 Control section 131 Acquisition Department 132 Evaluation Section 133 Calibration section 134 Update Department 135 Image Processing Unit 136 Split calibration section 140 Imaging unit 141 Objective Lens 142 Optical system 143 Image sensor 144 Image Generation Unit 150 Inertial sensor section C Connector part

Claims

1. an acquisition unit that acquires hyperspectral data in which image data is captured by dispersing the light into a plurality of wavelength ranges; an evaluation unit that calculates an evaluation value of the image data for each of a plurality of wavelength ranges; a calibration unit that executes a calibration process to select a wavelength range of a first evaluation value that indicates a maximum evaluation value among a plurality of wavelength bands as a calibration wavelength range; An update unit that updates the calibration wavelength range when a predetermined case occurs; An imaging device comprising:

2. the updating unit performs the calibration process to update the calibrated wavelength range when a ratio between the first evaluation value and the second evaluation value, which indicates an evaluation value of the calibrated wavelength range in the image data captured after the calibration process, exceeds a first threshold value. The imaging device according to claim 1 .

3. and an image processing unit that performs image recognition processing based on the image data to determine a magnitude of a motion vector of the detected object, the update unit performs the calibration process to update the calibrated wavelength range when a magnitude of the motion vector of the object is greater than a second threshold value or a degree of motion based on the inertial sensor unit is greater than a third threshold value.

3. The imaging device according to claim 1.

4. a division calibration unit that divides the screen into predetermined regions, and selects, for each divided region, a wavelength region in which the evaluation value is maximum, as a calibration wavelength region, based on an evaluation value for each wavelength region calculated for the divided region; The acquisition unit acquires image data captured using the calibration wavelength range set for each divided region, when a magnitude of the motion vector of the object is greater than a second threshold value or a degree of motion based on the inertial sensor unit is greater than a fourth threshold value, the update unit moves a region including the object in accordance with the motion of the object, and selects the wavelength range selected for the region including the object as a calibration wavelength range for the region after the movement. The imaging device according to claim 3 .

5. A step of acquiring hyperspectral data in which image data is captured by dispersing the light into a plurality of wavelength ranges; calculating an evaluation value of the image data for each of a plurality of wavelength ranges; A step of performing a calibration process in which a wavelength range of a first evaluation value that indicates a maximum evaluation value in all imageable wavelength ranges is selected as a calibration wavelength range; updating the calibration wavelength range when a predetermined event occurs; An imaging method comprising:

Citation Information

Patent Citations

  • Processing device, calibration method, camera, and underwater drone

    JP2022124250A