Computing units, information display systems, mobile devices

Abstract

To achieve high accuracy of an automatic estimation, and contribute in an economical viewpoint such as a low birthrate, and a social viewpoint such as development of observation members in an evaluation of information on a field of view.SOLUTION: A computing device makes inferences by a neural network, and the device comprises: a processor and a memory. In the computing device, the processor is configured to: store image data serving as a camera image in the memory; generate a characteristic map serving as an amount-of-characteristic map to be used in the inference on the basis of the image data stored in the memory; calculate a ratio of scattering light and direct light for each pixel on the basis of the image data stored in the memory; generate a gaze map serving as a map of the ratio thereof; store the generated amount-of-characteristic map and gaze map in the memory; make the gaze map reflect the amount-of-characteristic map; and make the inference including the amount-of-characteristic map having the gaze map reflected.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to an arithmetic device, an information presentation system, and a moving body.

Background Art

[0002] In recent years, techniques for evaluating visual field information by utilizing video analysis technology using AI have attracted attention. An example of this type of technique is an automatic weather observation that automates the observation of weather information. Here, among weather information, observations such as runway visual range and the presence or absence of fog can be realized by automatic weather observation by utilizing existing weather observation instruments. On the other hand, it is not easy to automatically observe the visibility range (the distance visible to the naked eye) by current weather observation instruments. Therefore, at present, for example, skilled observers are visually observing the weather on a 24-hour basis. From the economic perspective of labor reduction and the social perspective of supporting the training of new observers, it is considered urgent to automate the observation of the visibility range.

[0003] Here, Patent Document 1 relates to a visibility range evaluation technique, and includes an image acquisition unit that acquires an image including the position of a target object that is an evaluation index for the visibility range within an imaging range, a feature amount calculation unit that calculates a feature amount from the image, a distance information acquisition unit that acquires distance information from the imaging point of the image to the target object, an ambient light information acquisition unit that acquires ambient light information indicating the state of ambient light at the time of imaging the image, and a visibility range evaluation unit that evaluates the visibility range at the time of imaging the image using the feature amount, the distance information, and the ambient light information.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Conventional evaluation methods using deep learning learn the characteristic that targets near the camera are mapped to the bottom of the image, and distant targets are mapped to the top of the image. For example, when evaluating visibility, visibility is estimated based on the height coordinate information and degree of blur relative to the target. Therefore, there was a problem in that the estimation performance deteriorated when large objects such as buildings or aircraft were present near the camera. For example, as shown in Figure 1, further improvement in accuracy is essential for automatic visibility estimation in automated weather observation.

[0006] Therefore, the present invention aims to improve the accuracy of automatic estimation in the evaluation of visual field information, contributing from an economic perspective of reducing the number of personnel and from a social perspective of training observers. [Means for solving the problem]

[0007] According to a first aspect of the present invention, the following computing device is provided. This computing device is a device that performs inference using a neural network and comprises a processor and memory. The processor stores image data, which is a camera image, in memory, generates a feature map, which is a map of features used for inference, based on the image data stored in memory, calculates the ratio of scattered light to direct light for each pixel based on the image data stored in memory, generates a gaze map, which is a map of the ratio, stores the generated feature map and gaze map in memory, reflects the gaze map in the feature map, and performs inference including the feature map that reflects the gaze map.

[0008] According to a second aspect of the present invention, an information presentation system is provided that includes the above-mentioned computing device, a camera that acquires image data to be input to the computing device, and an output device that outputs the results of inference by the computing device.

[0009] A third aspect of the present invention provides a mobile device comprising the above-mentioned computing device and a camera that captures the external environment and acquires image data to be input to the computing device. [Effects of the Invention]

[0010] According to the present invention, it is possible to improve the accuracy of automatic estimation in the evaluation of information related to the field of view, thereby contributing from an economic perspective of reducing the number of personnel and from a social perspective of training observers. Furthermore, other issues, configurations, and effects will be clarified by the following description of embodiments for carrying out the invention. [Brief explanation of the drawing]

[0011] [Figure 1] A figure illustrating an example of the performance of existing visibility estimation approaches. [Figure 2] A diagram illustrating an example of a visibility estimation flow according to an embodiment. [Figure 3] A diagram illustrating an example of a comparison of the effects with conventional technology. [Figure 4] A diagram showing an example of the configuration of the computing device of the first embodiment. [Figure 5] A diagram showing an example configuration of the arithmetic unit of the second embodiment. [Figure 6] A diagram showing an example configuration of the computing device according to the third embodiment. [Figure 7] A diagram showing an example configuration of the computing device according to the fourth embodiment. [Figure 8] A diagram showing an example of the hardware configuration of the computing unit. [Modes for carrying out the invention]

[0012] Embodiments of the present invention will be described below with reference to the drawings. The embodiments are illustrative examples for explaining the present invention, and have been omitted and simplified as appropriate for clarity of explanation. The present invention can also be implemented in various other forms. Unless otherwise specified, each component may be singular or plural. The positions, sizes, shapes, and ranges of the components shown in the drawings may not represent their actual positions, sizes, shapes, and ranges in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the positions, sizes, shapes, and ranges disclosed in the drawings. When there are multiple components with the same or similar function, they may be described using the same symbol but with different subscripts. Furthermore, if it is not necessary to distinguish between these multiple components, the subscripts may be omitted in the description. In embodiments, processing performed by executing a program may be described. Here, the computer executes the program using a processor (e.g., CPU, GPU) and performs processing defined by the program using memory resources (e.g., memory) and interface devices (e.g., communication ports). Therefore, the main entity performing the processing by executing the program may be the processor. Similarly, the main entity performing the processing by executing the program may be a controller, device, system, computer, or node having a processor. The main entity performing the processing by executing the program may be an arithmetic unit, and may include dedicated circuits that perform specific processing. Here, dedicated circuits include, for example, FPGAs (Field Programmable Gate Arrays), ASICs (Application Specific Integrated Circuits), CPLDs (Complex Programmable Logic Devices), etc. The program may be installed on the computer from the program source. The program source may be, for example, a program distribution server or a storage medium readable by the computer. If the program source is a program distribution server, the program distribution server includes a processor and storage resources for storing the program to be distributed, and the processor of the program distribution server may distribute the program to other computers. In addition, in some embodiments, two or more programs may be implemented as a single program, or one program may be implemented as two or more programs.

[0013] First, an example of the processing flow of the arithmetic unit will be described while referring to FIG. 2. The arithmetic unit performs processing using the technology of CNN (Convolutional Neural Network) and outputs an estimation result of the visibility information (in this example, the visibility range) using the acquired camera image. As shown in an example of the visibility range estimation flow in FIG. 2, a fixation map generation unit and a data update unit are added to the conventional visibility range estimation AI. Note that the visibility range indicates the maximum distance visible to the naked eye.

[0014] As shown in FIG. 2, in the CNN, the arithmetic unit uses a plurality of stacked residual blocks (the residual blocks indicated by 0 to n in the figure, which may be referred to as the first plurality of layers) to output a feature map g(x) from the camera image. On the other hand, the arithmetic unit uses a fixation map generation unit to generate a fixation map M(x). The fixation map M(x) is data that reflects information on the fixation area and the non-fixation area in the feature map M(x). In the fixation map generation unit, first, the degree of blurring in the camera image is estimated in pixel units. Next, a fixation map M(x) is output, where the pixel area with a high degree of blurring is the fixation area and the pixel area with a low degree of blurring is the non-fixation area. Here, for example, for the calculation of the degree of blurring, the ratio t of the scattered light and the direct light in pixel units calculated using a technique related to image processing (in this example, Dark Channel Prior) may be used. Generally, for the fixation map M(x), a high degree of blurring is set for distant structures in the fixation area, and a low degree of blurring is set for nearby structures in the non-fixation area.

[0015] Next, in the data update unit, the fixation map M(x) is reflected in the feature map g(x). Specifically, the feature amounts on the feature map g(x) corresponding to the fixation area are emphasized, and conversely, the feature amounts on the feature map g(x) corresponding to the non-fixation area are attenuated.

[0016] Note that it is only necessary to appropriately extract feature amounts from the camera image, and the structure of the residual block (the first plurality of layers) can be changed as appropriate.

[0017] Also, as an example, in the stacked residual blocks, the computing device may reflect the attention map M(x) on the feature map g(x) output by the residual block close to the input (close to the first stage). However, it is not limited to this, and the computing device may perform a process of reflecting the attention map M(x) on the feature map g(x) output by other residual blocks. Also, when one residual block outputs a plurality of feature maps g(x), the computing device may reflect the attention map M(x) on all the output feature maps g(x), or may reflect the attention map M(x) on some of the output feature maps g(x).

[0018] In a CNN, the computing device uses a plurality of stacked residual blocks (the residual blocks shown as n+1 to N in the figure, which may be called the second plurality of layers), generates data for identification from the feature map g(x), and outputs it to the FC (Fully Connected layer). Here, the feature map g(x) processed using the residual blocks (n+1 to N) includes the feature map g(x) on which the attention map M(x) is reflected. As an example, the computing device may perform processing based on the average value of each feature map g(x). That is, the computing device may perform processing based on GAP (Global Average Pooling). However, the computing device may perform processing based on other appropriate methods.

[0019] Note that as long as the data for identification can be appropriately generated from the feature map g(x), the structure of the residual block (the second plurality of layers) can be appropriately changed.

[0020] Next, the computing device outputs an inference result of the visibility based on the data output to the FC. As an example, values of integers from 0 to 2 are appropriately output to the FC. However, as long as appropriate inference can be made, it is not particularly limited to this example.

[0021] The effects of the computing device will be explained with reference to Figure 3. As shown in Figure 3, the visibility estimation performance was evaluated using six different datasets that mapped different landscapes. The results of the evaluation confirmed that visibility estimation performance can be significantly improved by using the computing device. Note that Figure 3 is data with 2000 epochs. R^2 relates to the coefficient of determination.

[0022] <First Embodiment> Next, an example of the configuration of the computing device of the first embodiment will be described with reference to Figure 4. Note that explanations that are the same as those already described may be omitted. This computing device (1a) comprises a memory (11), a feature calculation unit A (21), a gaze map generation unit (22), a data update unit (23), and a feature calculation unit B (24). Image data (31) is input to the computing device (1a) from the camera (12). The computing device (1a) also outputs data used to present the inference result (33) to the output device (13).

[0023] First, the computing unit (1a) stores the image data (31) captured by the camera (12) into the memory (11). Next, the gaze map generation unit (22) receives the image data (31) stored in the memory (11) as input and outputs a gaze map in which the pixel regions to be used as supporting information during estimation are weighted. However, the gaze map is normalized to a continuous range of values ​​from 0 to 1, and pixel regions that store larger values ​​are used more often as supporting information during estimation. Here, the gaze map may be generated based on the image processing technique (Dark Channel Prior) as described above.

[0024] The computing unit (1a) further receives image data (31) as input, applies an n-layer convolution operation in the feature calculation unit A (21), and outputs feature map data (32).

[0025] The arithmetic unit (1a) receives the attention map and feature map data (31) as input, and the data update unit (23) weights the feature map data (31) with the attention map and overwrites the feature map data (32) in memory (11). Here, an attention mechanism may be used for weighting the feature map data (32) with the attention map.

[0026] Finally, the computing unit (1a) receives the weighted feature map data (32) from the data update unit (23) as input, applies an m-layer convolution operation in the feature calculation unit B (24), and outputs the inference result (33). The inference result (33) is displayed outside the computing unit (1a) by the output device (13).

[0027] The user (for example, an observer) can determine the visibility by referring to the data displayed on the output device (13). The display method is not particularly limited, as long as the user can appropriately grasp the visibility of the target.

[0028] The computing unit (1a) may output data to the output device (13) that includes, for example, the inference result data (33), lines indicating targets, arrows, symbols, etc., attached to the camera image for display. In this case, the user can easily grasp the target data based on the camera image with appropriate information attached. In addition, other appropriate display methods may be adopted, such as highlighting the target against the scenery in the camera image (for example, displaying the target with color or outlines).

[0029] According to this embodiment, highly accurate automatic estimation of field of view information is possible. Furthermore, it is possible to support the training of observers using the output results. Therefore, it can contribute from both an economic and social perspective.

[0030] <Second Embodiment> Next, with reference to Figure 5, an example of the configuration of the arithmetic unit of the second embodiment will be described. This arithmetic unit (1b) differs from the first embodiment in that it performs preprocessing by image correction. Note that explanations that have already been described may be omitted.

[0031] The arithmetic unit (1b) includes an image correction unit (25). The image correction unit (25) performs appropriate processing on the image data (31), such as noise reduction, contrast adjustment, and cropping of the gaze area (area with a high degree of blur), by modifying the image data or overwriting pixel values. The arithmetic unit (1b) may then perform processing based on the image data (31) that has undergone such appropriate processing. The image correction unit (25) may also perform color noise reduction. The image correction unit (25) may perform the above processing based on known methods. The image correction unit (25) may be used mainly to improve the accuracy of visibility estimation at night.

[0032] <Third Embodiment> Next, with reference to Figure 6, an example of the configuration of the computing device in the third embodiment will be described. In the third embodiment, the computing device (1c) infers the presence or absence of fog as information about the field of view. Note that explanations that have already been described may be omitted.

[0033] When fog is present, it is assumed that a blurred area with a certain degree of vertical spread is present in the image. Then, in updating the feature map data (32), a weight is set for this area that is highly blurred and becomes a gaze area (i.e., a weight that emphasizes the features). The computing unit (1c) then infers the presence or absence of fog by inferring the presence or absence of a vertically extending area that has features. According to this embodiment, by emphasizing the features, it is possible to accurately infer the presence or absence of fog even if the fog has a heterogeneous distribution in the vertical direction.

[0034] The presence or absence of fog can be indicated by an appropriate display method. The arithmetic unit (1c) may output display data to the output device (13) that includes, for example, text information indicating the presence of fog, lines, arrows, symbols, etc., attached to the camera image. Alternatively, the arithmetic unit (1c) may display a camera image in which the foggy areas are emphasized with color or outlines.

[0035] In this embodiment as well, the visibility of the target may be evaluated based on inference processing. However, from the viewpoint of appropriately reflecting the gaze area when evaluating visibility, it is preferable to have a fog-free environment so that fog does not become noise.

[0036] Furthermore, in this embodiment as well, the image correction unit (25) may perform appropriate processing. The image correction unit (25) may also be used to improve the accuracy of visibility estimation.

[0037] <Fourth Embodiment> Next, with reference to Figure 7, an example of the configuration of the computing device in the fourth embodiment will be described. In the fourth embodiment, the computing device (1d) infers the transparency of the water as visual information. Note that explanations that have already been described may be omitted.

[0038] When the transparency of the water decreases, the camera image becomes blurry. Therefore, when updating the feature map data (32), for example, when the transparency of the water is low, a weight is set that corresponds to a region with a high degree of blurring and therefore a focus area (i.e., a weight that emphasizes the features).

[0039] The computing unit (1d) then performs inference processing, including the updated feature map data (32), to infer the transparency of the water. Here, the computing unit (1d) may, as an example, infer the transparency of the water based on the features of the entire feature map. However, it is sufficient that the transparency of the water can be inferred appropriately, and the transparency of the water may be inferred by other methods, such as inferring based on a representative part of the feature map.

[0040] The transparency of the water can be indicated by an appropriate display method. The calculation unit (1d) may output display data, such as numbers, letters, or symbols indicating the transparency of the water, to the output device (13).

[0041] In this embodiment as well, the visibility of the underwater target may be evaluated based on inference processing. However, from the viewpoint of appropriately reflecting the gaze area when evaluating visibility, it is preferable to have an environment with high transparency so that low transparency does not become noise.

[0042] Furthermore, in this embodiment as well, the image correction unit (25) may perform appropriate processing. The image correction unit (25) may also be used to improve the accuracy of visibility estimation.

[0043] Next, an example of the hardware configuration of the arithmetic unit will be described with reference to Figure 8. The arithmetic unit (1a to 1d) comprises a processor (41), a storage unit (42), and an interface unit (45).

[0044] The processor (41) can be, for example, a CPU, and is the entity that executes predetermined processing. However, the processor (41) only needs to be able to execute predetermined processing, and may be configured using other appropriate semiconductor devices (e.g., a GPU).

[0045] The storage unit (42) can be configured using a main memory (43) and an auxiliary storage device (44). Here, the main memory (43) corresponds to the memory (11) shown in Figure 4. The main memory (43) can be, for example, RAM (Random Access Memory), and the processor (41) temporarily stores data in the main memory (43) during data processing. The auxiliary storage device (44) can be, for example, an HDD (Hard Disk Drive), and stores programs used for various data processing (arithmetic processing, display processing, etc.) performed by the arithmetic units (1a to 1d).

[0046] The auxiliary storage device (44) stores, for example, a program consisting of a feature calculation unit A (21), a gaze map generation unit (22), a data update unit (23), a feature calculation unit B (24), and an image correction unit (25). The auxiliary storage device (44) also stores generated data such as feature maps and gaze maps.

[0047] Note that the configuration of the main memory (43) and auxiliary memory (44) described herein is just one example, and the configuration of the storage unit (42) may be changed as appropriate.

[0048] The interface unit (45) is an interface for inputting and outputting data. The arithmetic units (1a-1d) acquire image data (31), which is a camera image, from the camera (12) via the interface unit (45). The arithmetic units (1a-1d) also output data to be displayed on the output device (13) via the interface unit (45). User operations may also be input to the arithmetic units (1a-1d) via the interface unit (45). User operations are performed using appropriate input devices (not shown), and examples of user operations include selecting a target to evaluate visibility in the camera image, selecting whether or not to determine the presence or absence of fog, and selecting whether or not to evaluate the transparency of the water.

[0049] Although embodiments of the present invention have been described in detail above, the present invention is not limited to the embodiments described above, and various design modifications can be made without departing from the spirit of the invention as described in the claims. For example, the embodiments described above are described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Moreover, it is possible to add, delete, or replace parts of the configuration of each embodiment with other configurations.

[0050] According to one embodiment, a system (information presentation system) is provided that includes a computing device (1a to 1d), a camera (12), and an output device (13).

[0051] A mobile body may be provided that includes a camera (12) for photographing the external environment and a computing device (1a-1d). This mobile body may also include an output device (13). Here, the mobile body is a vehicle (e.g., an automobile), a ship, a submarine, etc., but is not limited to these. The mobile body may also be an aircraft, for example, and the computing device may be used by the pilot to evaluate visibility and the presence or absence of fog during takeoff and landing.

[0052] An interface for communication may be provided as the interface section (45) of the arithmetic units (1a to 1d). The arithmetic units (1a to 1d) may acquire data from the camera (12) via wireless communication and perform data processing.

[0053] When the computing units (1a-1d) and the camera (12) communicate wirelessly, the computing units (1a-1d) and the camera (12) may be located in different locations. For example, the camera (12) may be located in the area to be photographed, and the computing units (1a-1d) may be located in a remote location different from this area. The computing units (1a-1d) may then acquire data from the camera (12) via wireless communication and perform data processing. [Explanation of symbols]

[0054] 1a Arithmetic unit 1b Arithmetic unit 1c arithmetic unit 1d arithmetic unit 11 memory 12 cameras 13 Output device 21 Feature Analysis Unit A 22 Gaze Map Generation Unit 23 Data Update Section 24 Feature Analysis Unit B 25 Image Correction Unit 31 Image data 32 Feature Map Data 33 Inference results 41 processors 42 Storage section 43 Main memory 44 Auxiliary storage device 45 Interface section

Claims

1. A computing device that performs inference using a neural network, Processor and Memory and Equipped with, The aforementioned processor, The image data, which is a camera image, is stored in the memory. Based on the image data stored in the memory, a feature map is generated, which is a map of features used for inference. Based on the image data stored in the memory, the ratio of scattered light to direct light is calculated for each pixel, and a gaze map, which is a map of the ratios, is generated. The generated feature map and gaze map are stored in the memory, and the gaze map is reflected in the feature map so as to enhance the feature quantity in areas where the degree of blurring is high based on the ratio, and attenuate the feature quantity in areas where the degree of blurring is low based on the ratio. Inference is performed including the feature map that reflects the gaze map. A computing device characterized by the following features.

2. The arithmetic device according to claim 1, The aforementioned processor, The image data in the memory is received as input, calculations are performed in the first multiple layers which consist of multiple layers, and the calculation results are output to the memory as the feature map. The feature map in the memory is received as input, and calculations are performed in the second layer, which is a set of layers on the output side of the first layer, and the calculation results are output to the memory as inference results. A computing device characterized by the following features.

3. The arithmetic device according to claim 1, The aforementioned processor, The image data in the memory is received as input, and the ratio of scattered light to direct light is calculated using image processing techniques (Dark Channel Prior). A computing device characterized by the following features.

4. The arithmetic device according to claim 1, The aforementioned processor, The attention map in the memory is received as input, and the attention mechanism reflects the attention map in the feature map. A computing device characterized by the following features.

5. The arithmetic device according to claim 1, The aforementioned processor, The image data in the memory is received as input, and color noise reduction is performed. A computing device characterized by the following features.

6. The arithmetic device according to claim 1, The aforementioned processor, The image data in the memory is received as input, and a region with a high degree of blurring is extracted from the image data. A computing device characterized by the following features.

7. The arithmetic device according to claim 1, The aforementioned processor, Inferring the target's visibility in a camera image. A computing device characterized by the following features.

8. The arithmetic device according to claim 1, The aforementioned processor, By inferring the presence or absence of a vertically extending region containing features, the presence or absence of fog in a camera image is inferred. A computing device characterized by the following features.

9. The arithmetic device according to claim 1, The aforementioned processor, Underwater image data is acquired as camera images, and the transparency of the water is inferred using the said image data. A computing device characterized by the following features.

10. The arithmetic device according to claim 1, A camera that acquires the image data to be input to the aforementioned computing device, The aforementioned computing device outputs the results of its inference, An information presentation system characterized by comprising the following features.

11. An information presentation system according to claim 10, The aforementioned processor, Inferring the target's visibility in the camera image, The inference result is output to the output device. An information presentation system characterized by the following:

12. An information presentation system according to claim 10, The aforementioned processor, By inferring the presence or absence of a vertically extending region containing features, we can infer the presence or absence of fog in the camera image. The inference result is output to the output device. An information presentation system characterized by the following:

13. An information presentation system according to claim 10, The aforementioned processor, Underwater image data is acquired as a camera image, and the transparency of the water is inferred using the said image data. The inference result is output to the output device. An information presentation system characterized by the following:

14. An information presentation system according to claim 10, The aforementioned computing device and the aforementioned camera are, They are each located in different places and are capable of wireless communication. An information presentation system characterized by the following:

15. The arithmetic device according to claim 1, A camera that captures the external environment and acquires the image data to be input to the computing device, A mobile body characterized by having the following features.

Images