Environment recognition device and environment recognition method

The environment recognition device and method address the challenge of achieving accurate recognition across diverse conditions by converting images to match learning conditions, reducing data creation costs and enhancing recognition accuracy.

JP2024164986APending Publication Date: 2024-11-28HITACHI LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023080766
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-16
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Existing environment recognition systems face challenges in achieving highly accurate recognition across diverse environmental conditions, such as nighttime, rain, fog, and different countries, due to the high cost and effort required for creating correct value data for machine learning models.

Method used

An environment recognition device and method that includes an image acquisition unit, environmental condition acquisition unit, image conversion unit, and recognition processing unit, which adapts image recognition models to match learning conditions by converting images based on environmental conditions using convolutional neural networks and deep learning models.

Benefits of technology

Enables highly accurate recognition in various environments while reducing the cost of creating correct value data by adaptively transforming images to match learning conditions, improving recognition performance across different weather and geographical conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024164986000001_ABST
    Figure 2024164986000001_ABST
Patent Text Reader

Abstract

To provide an environment recognition device that achieves highly accurate recognition in various environments while reducing creation costs for correct answer value data in learning.SOLUTION: An environment recognition device 1 of the present invention includes: an image acquisition unit 100; an environment condition acquisition unit 101 that acquires an environment condition for an image acquired by the image acquisition unit; a recognition processing unit 103 that performs recognition processing for the image according to an image recognition model; and an image conversion unit 102 that converts the image acquired by the image acquisition unit according to a learning condition for the image recognition model. The recognition processing unit, when the environment condition acquired by the environment condition acquisition unit is different from the learning condition for the image recognition model in the recognition processing unit, performs the recognition processing for the image obtained by converting the image acquired by the image acquisition unit according to the learning condition by the image conversion unit.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an environment recognition device and an environment recognition method. [Background technology]

[0002] In order to realize preventive safety functions and autonomous driving, it is effective to capture the driving environment with a camera and perform recognition processing such as image processing on the captured images. From the viewpoint of recognition performance, machine learning models such as deep learning are used in recognition processing. Recognition processing is expected to achieve high accuracy in various environmental conditions (weather, etc.).

[0003] In contrast, in a machine learning model, it is effective to create correct answer data for various environmental conditions as learning data and use it for learning, but creating correct answer data for various environmental conditions requires a lot of work. For this reason, for example, in the system for training a prediction system in Patent Document 1, it is described that the prediction system can perform training by using a style model to convert a synthetic image of a low-light environment into a daytime representation as an input to a depth model for generating a depth map (see paragraph 0004). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2022-191188 A Summary of the Invention [Problem to be solved by the invention]

[0005] According to the above prior art, by converting nighttime data into daytime data, nighttime recognition can be performed with a machine learning model trained only on daytime data. Meanwhile, environmental conditions in a driving environment include not only nighttime but also many weather conditions such as rain and fog. In addition, not only the weather but also the country in which the vehicle is driven is diverse, so it is difficult to realize highly accurate recognition processing in various driving environments by simply responding to environmental conditions at night as in the prior art method.

[0006] An object of the present invention is to provide an environment recognition device and an environment recognition method that realize highly accurate recognition in a variety of environments while reducing the cost of creating correct answer data in learning. [Means for solving the problem]

[0007] In order to solve the above problems, the environmental recognition device of the present invention comprises an image acquisition unit, an environmental condition acquisition unit that acquires the environmental conditions of an image acquired by the image acquisition unit, a recognition processing unit that performs recognition processing of the image using an image recognition model, and an image conversion unit that converts the image acquired by the image acquisition unit to match the learning conditions of the image recognition model, and when the environmental conditions acquired by the environmental condition acquisition unit differ from the learning conditions of the image recognition model of the recognition processing unit, the recognition processing unit performs recognition processing of the image acquired by the image acquisition unit by converting it by the image conversion unit to match the learning conditions. Effect of the Invention

[0008] According to the present invention, it is possible to realize highly accurate recognition in a variety of environments while reducing the cost of creating correct answer data in learning. [Brief description of the drawings]

[0009] [Figure 1] 1 is a functional block diagram illustrating a configuration of an environment recognition device according to an embodiment. [Diagram 2] 4 is a flowchart illustrating an operation of the environment recognition device. [Diagram 3]FIG. 4 is a diagram showing an example of a captured image acquired by an image acquisition unit. [Figure 4] FIG. 2 is a diagram illustrating a configuration of a conversion model storage unit. [Diagram 5] FIG. 13 is a diagram showing table information in which execution orders and recognition rates are associated with each other. [Figure 6] FIG. 13 is a diagram showing the process of image conversion. [Figure 7] FIG. 13 is a diagram showing table information indicating an average luminance change amount for each image transformation model. [Figure 8] 11 is a diagram illustrating a method for learning an image conversion model from nighttime to sunny weather. FIG. [Figure 9] FIG. 13 is a diagram illustrating a learning method for another nighttime-to-clear day image conversion model. [Figure 10] FIG. 11 is a functional block diagram showing another configuration of the environment recognition device. [Figure 11] 10A to 10C are diagrams illustrating examples of images before and after image conversion displayed on a display unit. [Figure 12] FIG. 13 is a diagram showing an example of additionally displaying an image in the middle of conversion on the display unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0011] 1 is a functional block diagram showing the configuration of an environment recognition device 1 according to an embodiment. The environment recognition device 1 includes an image acquisition unit 100, an environmental condition acquisition unit 101, an image conversion unit 102, and a recognition processing unit 103, as shown in FIG.

[0012] Specifically, the environment recognition device 1 is composed of a camera, a computer, a memory, a storage device, etc., and the computer executes programs stored in the memory and the storage device to realize various functional units described below.

[0013] The following describes in detail the environment recognition device 1 that is installed in a vehicle and connected to one camera, but is not limited to this and can also be applied to a configuration in which multiple cameras are connected. The environment recognition device 1 can also be used in various environment monitoring systems, such as a vehicle monitoring system for a road measuring instrument that monitors the vehicle's operating status.

[0014] The image acquisition unit 100 is a functional unit that acquires images captured by a camera or the like.

[0015] The environmental condition acquisition unit 101 is a functional unit that acquires the environmental conditions of the surrounding environment of the vehicle, and the acquired environmental conditions are regarded as the environmental conditions of the imaging location of the image acquired by the image acquisition unit 100. In other words, the environmental conditions of the image in this specification refer to environmental information of the imaging location of the image.

[0016] Here, the environmental conditions include information on weather such as sunny, rainy, night (light / dark), snowy, foggy, etc., and information on countries and regions such as Japan, America, China, Europe, India, etc. This environmental information is acquired or determined from images acquired by the image acquisition unit 100, GPS information of the vehicle, car navigation / map system, operation information of the vehicle's wipers, etc.

[0017] Specifically, when using captured images, it is possible to distinguish between sunny, night (light and dark), and snowy conditions based on the overall brightness of the image, and also to determine rain if raindrops are detected. Fog can be determined based on the overall image being blurred and white. Furthermore, by recognizing the characters on signs and road signs from captured images of the driving environment, it is possible to determine the region or country in which the vehicle is traveling.

[0018] Weather information may be obtained from a car navigation system. Also, the region or country in which the vehicle is traveling may be determined based on GPS information or information from a car navigation / map system. Furthermore, by using operation information from wipers, etc., it is possible to grasp the rainy environmental conditions with high accuracy.

[0019] Although details will be described later, when the environmental conditions acquired by the environmental condition acquisition unit 101 do not match the learning conditions of the recognition processing unit 103 described later, the image conversion unit 102 executes image conversion so that the image acquired by the image acquisition unit 100 matches the learning conditions of the recognition processing unit 103. Here, the learning conditions are the environmental conditions of the image used for learning the machine learning model used by the recognition processing unit 103, such as being in Japan and sunny.

[0020] The image conversion unit 102 stores at least one image conversion model to be used for image conversion and selects the one to be used for conversion. These image conversion models may be convolutional neural networks or deep learning models, or may be statistical models trained by other machine learning algorithms, or may be a series of classical filter processes.

[0021] If the current environmental conditions are America and it is raining, and the learning conditions of the recognition processing unit 103 are Japan and it is sunny, the image conversion unit 102 executes image conversion since the current environmental conditions do not match the learning conditions. At this time, from the multiple stored image conversion models, an image conversion model from night to sunny and an image conversion model from America to Japan are selected. By executing these two conversion models in order, the image acquired by the image acquisition unit 100 is converted so that it appears as it does under the environmental conditions of Japan and it is sunny.

[0022] The recognition processing unit 103 executes the recognition process using the image acquired by the image acquisition unit 100 or the image converted by the image conversion unit 102. In the recognition process, a machine learning model is used. This machine learning model is trained using an image under a specific environmental condition (for example, a sunny day in Japan), and the environmental condition, i.e., the learning condition, is stored in the image conversion unit 102. In addition, the machine learning model used in the recognition process may be a segmentation model that identifies each pixel of an image, an object detection model that detects an object appearing in an image, or a depth calculation model that calculates the depth of each pixel of an image.

[0023] Next, the operation of the environment recognition device 1 of the embodiment will be described in detail with reference to the flowchart in Fig. 2. Note that the description will be given on the assumption that the environmental conditions of the driving environment are America and rain, the recognition processing unit 103 uses a segmentation model that recognizes the driving road, and learning data of Japan and sunny weather is used for learning.

[0024] Fig. 3 is a diagram showing an example of a captured image acquired by the image acquisition unit 100. In Fig. 3, reference numeral 300 denotes a road, reference numeral 301 denotes a sidewalk, and reference numeral 302 denotes raindrops. It is assumed that the color of the asphalt of the road 300 differs between America and Japan.

[0025] The environment recognition device 1 sequentially executes an image acquisition process S101, an environmental condition acquisition process S102, a process S103 for checking whether the environmental conditions match the learning conditions, a conversion model selection process S104, a conversion order determination process S105, an image conversion process S106, and a recognition process S107.

[0026] Returning to FIG. 2, in image acquisition processing S101, the image acquisition unit 100 acquires image data from the camera.

[0027] In environmental condition acquisition processing S102, the environmental condition acquisition unit 101 refers to GPS information and determines that the country in which the vehicle is currently traveling is the U.S. After that, it acquires weather information from the car navigation system and determines that the current weather is rain.

[0028] In a process S103 for checking whether the environmental conditions and the learning conditions match, the environmental condition acquisition unit 101 determines that the two conditions do not match (No in S103) because the environmental conditions acquired in the environmental condition acquisition process S102 are America and rain, and the learning conditions of the recognition processing unit 103 stored in the learning condition storage unit 1021 are Japan and sunny, and proceeds to a conversion model selection process S104. If the environmental condition acquisition unit 101 determines that the two conditions match (Yes in S103), it proceeds to a recognition process S107 without performing image conversion.

[0029] In the conversion model selection process S104, the conversion process selection unit 1023 selects an image conversion model to be used this time from among the image conversion models stored in the conversion model storage unit 1022 of the image conversion unit . 4 is a diagram showing the configuration of the conversion model storage unit 1022. The conversion model storage unit 1022 stores, for example, a conversion model for converting countries and regions, and a model for converting weather.

[0030] Specifically, since the environmental conditions acquired in the environmental condition acquisition process S102 are America and rainy, and the learning conditions of the recognition processing unit 103 are Japan and sunny, the conversion model selection process S104 selects two conversion models, one for converting America to Japan and one for converting rainy to sunny, in order to match the environmental conditions with the learning conditions. These conversion models are trained by the learning process described later.

[0031] Returning to Fig. 2, in the conversion order determination process S105, the conversion execution unit 1024 determines the order in which to execute the multiple conversion models selected in the conversion model selection process S104. The order is determined using table information indicating the recognition rate of the segmentation model of the recognition processing unit 103 in each execution order, as shown in Fig. 5. The relationship between this execution order and the recognition rate is determined in advance and prepared as table information.

[0032] Specifically, a large number of evaluation images are prepared in advance, image transformation is performed in each execution order, and the recognition rate when the transformed images are recognized by the segmentation model is calculated. The execution order and the recognition rate are associated to create the table information shown in Figure 5.

[0033] 5, it can be seen that the recognition rate is higher when conversion regarding the country is performed first, and then conversion regarding the weather is performed. Therefore, in the conversion order determination process S105, the conversion order is determined such that image conversion from America to Japan is performed first, and then image conversion from rain to sunny is performed.

[0034] Furthermore, when multiple image transformation models are selected in the transformation model selection process S104, as in this case, the execution order is determined in the transformation order determination process S105. However, when only a single transformation model is selected, it is determined that only that single image transformation model will be executed without accessing table information such as that shown in FIG. 5.

[0035] In the image conversion process S106 in FIG. 2, the conversion execution unit 1024 converts the images acquired in the image acquisition process S101 based on the order determined in the conversion order determination process S105. FIG. 6 is a diagram showing the process of image conversion for processing image data whose environmental conditions are America and it is raining in the recognition processing unit 103 whose learning conditions are Japan and it is sunny.

[0036] First, to perform a conversion from America to Japan, the color of the asphalt on the road in the captured image is changed. This makes it possible to convert an image with environmental conditions of America and rain (US, rain) into an image with environmental conditions of Japan and rain (JP, rain). Then, a conversion from rain to sunny is performed. This removes raindrops and converts the image with environmental conditions of Japan and rain into an image of Japan and sunny (JP, sunny).

[0037] In the recognition process S107 of FIG. 2, the recognition processing unit 103 performs a recognition process on the converted image converted in the image conversion process S106, or the image determined to match the environmental conditions and the learning conditions in the process of confirming the match between the environmental conditions and the learning conditions S103.

[0038] As described above, the environment recognition method of the embodiment selects an image conversion model to be used by referring to the environmental conditions of the acquired image and the learning conditions of the recognition process, and adaptively performs image conversion of the acquired image so that the environmental conditions of the acquired image match the learning conditions of the recognition process. This allows accurate recognition for various environmental conditions (e.g., America and rain) even if the machine learning model used in the recognition process is trained only under specific conditions (e.g., Japan and sunny). In other words, the recognition processing unit can achieve high-precision recognition in various environments by simply creating correct answer data only from the learning data under specific conditions.

[0039] When multiple models are selected as image transformation models to be used for image transformation, the environment recognition device 1 of the embodiment determines the order of application based on a pre-calculated recognition rate of the recognition processing unit 103. This makes it possible to execute image transformation taking into account the recognition performance of the recognition processing unit 103, thereby further improving the performance of the recognition processing unit 103.

[0040] In addition, in the environment recognition device 1 of the embodiment, a conversion model for converting environmental conditions related to country and weather is shown in Fig. 4. Since the appearance of an image changes significantly depending on the country and weather in which the image is taken, the recognition processing unit usually needs to collect learning data for each country and weather in advance, create correct answer data, and learn. However, by using the image conversion model in Fig. 4 to convert the environmental conditions themselves, namely the country and weather, so that they match the learning conditions of the recognition processing unit, the recognition processing unit 103 can perform highly accurate recognition of various countries and weather conditions simply by learning using correct answer data for only specific countries and weather conditions.

[0041] Next, the image transformation model will be described in detail. In the image conversion model of “rainy sky -> sunny” (conversion from rain to sunny) in Figure 4, which performs weather-related conversion, we have explained that the conversion is performed by removing raindrops from the captured image. However, conversion can also be performed by adding raindrops, changing the brightness of the entire image, removing umbrellas held by pedestrians, etc.

[0042] Other environmental conditions than rain include nighttime, snowy, and foggy conditions, which can be converted to sunny. Specifically, at night, the brightness of the entire image is converted, the lighting status of vehicle lights is changed, the lighting status of street lights is changed, and the brightness inside buildings is changed to convert to sunny. In the case of snowy conditions, the image is converted to sunny by removing snow that has accumulated on the road surface, the brightness of the entire image, and umbrellas held by pedestrians. In the case of fog, the fog is removed and the image is converted to sunny.

[0043] As described above, the environmental recognition device 1 of the embodiment converts the image into a sunny image by removing or modifying conditions characteristic of the weather environment, such as rain, nighttime, snowfall, and fog, and absorbs the differences in appearance, enabling the recognition processing unit to perform recognition with even higher accuracy.

[0044] The following describes an image conversion model that performs country-related conversion. It has been explained that the environment recognition device 1 of the embodiment changes the environmental conditions related to a country by converting the color tone of the asphalt in a captured image using an image conversion model.

[0045] In addition, because the presence or absence of white lines on roads (including road paint such as road markings) varies from country to country, the image transformation model can be used to perform image transformation to remove or add white lines on the road surface of the captured image, thereby changing the environmental conditions related to the country.

[0046] Furthermore, since the position where the license plate is attached to the vehicle differs from country to country, the image conversion model can be used to convert the image to change the position where the license plate is attached in the captured image, thereby changing the environmental conditions related to the country. In this case, the characters written on the license plate themselves may be converted. In addition, since the shape of the vehicle itself differs, the height and width of the vehicle, as well as the external shape of the vehicle itself may be converted.

[0047] In addition, the shape of road signs and the characters written on them differ depending on the country. For example, stop signs are hexagonal, triangular, etc. depending on the country. Therefore, the image transformation model can be used to perform image transformation to change the shape of the road sign, and to change the environmental conditions related to the country. In this case, the image transformation model can also be used to perform image transformation of the characters written on the road sign.

[0048] As described above, the environmental recognition device 1 of the embodiment converts the color of the asphalt on the road surface, the partial shape of the vehicle, the presence or absence of white lines, the shapes and characters of road signs, etc., which are characteristic of the environmental conditions of the country of the captured image, using an image conversion model to absorb differences in appearance depending on the country, allowing the recognition processing unit to perform even more accurate recognition.

[0049] In the above, in the environment recognition device 1 of the embodiment, in the conversion order determination process S105 (FIG. 2), the conversion execution unit 1024 performs image conversion in the order of high recognition rate as shown in FIG. 5. However, the conversion execution unit 1024 may perform image conversion in the order of image conversion models with smallest average luminance change amounts.

[0050] Here, for example, in the case of a rainy to sunny image conversion model, the average luminance change is the average value of the luminance difference of each pixel before and after conversion when a rainy image is input and converted into a sunny image. The average luminance change for each image conversion model may be calculated by converting a large number of images in advance, or the image recognized by the conversion execution unit 1024 may be converted in real time to obtain the average luminance change before and after conversion.

[0051] FIG. 7 is a diagram showing table information indicating the average luminance change amount for each image transformation model. In Fig. 7, the average luminance change amount in the image conversion model from America to Japan is 30, and the average luminance change amount in the image conversion model from rainy to sunny is 80. Therefore, the image conversion model from America to Japan has a smaller amount of change in image appearance. Therefore, the conversion execution unit 1024 first executes the image conversion from America to Japan, which has a smaller average luminance change amount, and then executes the image conversion from rainy to sunny.

[0052] The reason why the application order of the above image transformation models is in the order of the smallest average luminance change amount and the smallest change in appearance is based on the fact that a large change in the appearance of an image in image transformation has a high risk of failure and is highly difficult. Therefore, image transformations are executed in order starting with those with the smallest change in appearance, and then image transformations with a large change in appearance and a high degree of difficulty are executed on the image from which those environmental conditions have been removed. This reduces the risk of image transformation failure and enables the recognition processing unit 103 to have even higher accuracy.

[0053] Next, a method for learning a nighttime to sunny image conversion model will be described with reference to Figures 8 and 9. It goes without saying that the following description is not limited to nighttime to sunny image conversion.

[0054] In Figure 8, N101 is the image conversion model from nighttime to sunny. For learning, not only the nighttime to sunny image conversion model N101 but also the sunny to nighttime conversion model N100 is used. N100 and N101 are both convolutional neural networks. For learning, real images taken at night and real images taken on a sunny day are used. Three losses are used for learning losses.

[0055] The first loss is consistency loss. For consistency loss, a real nighttime image is input to N101, which converts it to a sunny image (sunny CG image) to generate it. After that, the sunny CG image is input to N100, which converts it to a nighttime image (nighttime CG image) to generate it. Then, the difference in brightness for each pixel between the input real nighttime image and the nighttime CG image is calculated as the loss. Similarly, a real sunny image is input to N100, the conversion result is input to N101, and the brightness difference between the output and the real sunny image is calculated as the loss. The sum of these two differences is the consistency loss.

[0056] The second loss is the adversarial loss. In the adversarial loss, a real nighttime image is input to N101, and a converted sunny CG image is obtained. The obtained sunny CG image and a real sunny image are input to the sunny classifier N103. The sunny classifier N103 is a classifier that determines whether the input image is a CG image or a real image.

[0057] Similarly, a clear real image is input to N100 to obtain a converted nighttime CG image, which is then input to the nighttime classifier N102. A nighttime real image is also input to the nighttime classifier N102. The nighttime classifier N102 determines whether the input image is a CG image or a real image.

[0058] We calculate the cross-entropy between the classification results of the sunny and nighttime classifiers, and use the sum of the cross-entropy as the adversarial loss. The first loss, the consistency loss, and the second loss, the adversarial loss, are the same as those used in the well-known CycleGAN.

[0059] The third loss is a loss for the classification result of the recognition processing unit, and a method for calculating the loss will be described with reference to Fig. 9. N100 and N101 in Fig. 9 are the same image conversion models as N100 and N101 in Fig. 8. Here, the recognition processing unit 103 is also composed of a convolutional neural network, and has been trained to be able to segment sunny images, and the parameters of the network are not updated in the training of the image conversion model.

[0060] The third loss uses a sunny real image I100 and its ground truth value I101, where I101 is the ground truth value for segmentation.

[0061] First, I100 is input to a sunny-to-nighttime image conversion model N100 to obtain a nighttime CG image I102. Then, I102 is input to a nighttime-to-sunny image conversion model N101 to obtain a sunny CG image I103. Then, the sunny CG image I103 and the correct answer I101 are input to the recognition processor, and cross-entropy is calculated as the loss related to segmentation.

[0062] The loss used in training the image translation model is the sum of the first and second losses and this third loss. In training, the image translation models N100 and N101 and the classifiers N102 and N103 are optimized alternately.

[0063] As described above, in the environment recognition device 1 of the embodiment, models capable of mutual conversion between environmental conditions (not only a conversion model from nighttime to sunny, but also a conversion model from sunny to nighttime) are used in learning the image conversion model.

[0064] As a result, the real images (real images at night and on a sunny day) acquired for each environmental condition required for training the image conversion model do not need to be a pair that matches perfectly except for the environmental conditions (a pair of images, one at night and one on a sunny day, acquired at the exact same location), and can be acquired at different locations for training. This reduces the cost of acquiring data required for training the image conversion model.

[0065] Furthermore, in the environment recognition device 1 of the embodiment, an image (actual image of a sunny day) that meets the learning conditions of the recognition processing unit 103 is input, image conversion is performed twice (conversion from sunny to nighttime, and then conversion from nighttime to sunny), and the loss of the recognition processing unit 103 is calculated using the converted CG image and the correct answer value, and an image conversion model is learned based on the loss.

[0066] This allows the loss of the recognition processing unit 103 to be added to the loss of the image conversion model for learning, and an image conversion model suitable for recognition can be constructed. Also, by utilizing the above-mentioned learning (converting from sunny to nighttime, then converting from nighttime to sunny, and defining the loss of the recognition unit with the converted sunny CG image), the loss of the recognition unit can be calculated using only the correct value of the learning condition (sunny) of the recognition processing unit, and the loss of the recognition processing unit can be added without creating correct value data for the nighttime data, thereby reducing the cost of creating correct value data.

[0067] Next, an example of configuring the environment recognition device 1 of the embodiment as a server-client model will be described. In this configuration, among the components shown in Fig. 1, the conversion model storage unit 1022 and the conversion process selection unit 1023 of the image conversion unit 102 are configured as a server, and the other components are configured as a client. In other words, this is a configuration in which the server determines the image conversion model required for the client to execute the recognition process.

[0068] 2 that the environmental conditions and the learning conditions do not match, the client notifies the server of the environmental conditions and the learning conditions, and the server performs the processes of S104 and S105 in Fig. 2 using the configuration of the conversion model storage unit 1022 and the conversion process selection unit 1023, and notifies the client of the image conversion model selected from the image conversion models stored in the conversion model storage unit 1022. The client executes S106 to perform image conversion using the image conversion model notified from the server, and performs recognition processing in S107.

[0069] In this way, by configuring the environment recognition device 1 as a server-client model, the client does not need to have an area for storing multiple image transformation models, and only needs to have an area for one image transformation model, which reduces the memory cost of the client.

[0070] By adding an image transformation model that converts new environmental conditions to the server, existing clients can perform recognition processing for new environmental conditions, thereby improving the scalability and availability of the recognition processing.

[0071] As described above, when the environmental conditions under which an image is acquired differ from the learning conditions of the recognition processing unit 103, the environment recognition device 1 of the embodiment can perform recognition processing under many environmental conditions by converting the acquired image to match the learning conditions of the recognition processing unit 103. For this reason, the environment recognition device 1 may be configured to display images before and after image conversion so that the image can be confirmed.

[0072] Fig. 10 is a functional block diagram showing the configuration of an environment recognition device 1 that displays images before and after image conversion. In detail, the environment recognition device 1 in Fig. 10 is provided with a display unit 104 in addition to the environment recognition device 1 shown in Fig. 1.

[0073] Fig. 11 is a diagram showing an example of images before and after image conversion displayed on display unit 104. In detail, Fig. 11 is a display example in which an image of America and rain is converted into an image of Japan and sunny weather.

[0074] Furthermore, the environment recognition device 1 may display on the display unit 104 how an image changes when a plurality of image transformation models are applied. Fig. 12 is a diagram showing an example of additionally displaying an image in the middle of transformation. In detail, when transforming an image of America and rain into an image of Japan and sunny, an image (JP, rain) to which an image transformation model for transforming into an image of Japan and rain is applied is displayed, and then the image of Japan and sunny is displayed.

[0075] As shown in FIG. 12, by displaying an image in the middle of conversion, the conversion result of the conversion order determined by the conversion process selection unit 1023 in the conversion order determination process S105 can be confirmed. Therefore, it is also possible to manually change the application order of the image conversion models determined based on the recognition rate described in FIG. 5 or the application order of the image conversion models determined based on the average luminance change amount described in FIG. 7.

[0076] Furthermore, the present invention is not limited to the above-mentioned examples, and various modified examples are included. The above-mentioned examples have been described in detail to easily explain the present invention, and the present invention is not necessarily limited to those having all of the configurations described. Furthermore, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. [Explanation of symbols]

[0077] 100 Image acquisition unit 101 Environmental condition acquisition department 102 Image conversion section 1021 Learning condition storage section 1022 Conversion model storage section 1023 Conversion process selection section 1024 Conversion Execution Unit 103 Recognition processing unit 104 Display section

Claims

1. an image acquisition unit that acquires an image; an environmental condition acquisition unit that acquires an environmental condition of the image acquired by the image acquisition unit; a recognition processing unit that performs image recognition processing using an image recognition model; an image conversion unit that converts the image acquired by the image acquisition unit in accordance with a learning condition of the image recognition model; When the environmental conditions acquired by the environmental condition acquisition unit are different from learning conditions of an image recognition model of the recognition processing unit, the recognition processing unit performs recognition processing on an image obtained by converting the image acquired by the image acquisition unit to match the learning conditions by the image conversion unit. An environment recognition device characterized by:

2. The environment recognition device according to claim 1, The image conversion unit sequentially applies a plurality of image conversion models to the image acquired by the image acquisition unit to convert the image of the environmental condition into an image corresponding to the learning condition. An environment recognition device characterized by:

3. The environment recognition device according to claim 2, The image conversion unit determines an application order with a high recognition rate based on the recognition rate of the recognition processing unit for each application order of the image conversion models, and sequentially applies the image conversion models in the determined application order. An environment recognition device characterized by:

4. The environment recognition device according to claim 2, The image conversion unit calculates an average luminance change amount, which is an average value of the luminance difference of each pixel before and after conversion when converting an image for each image conversion model, and sequentially applies the image conversion models in order of the smallest average luminance change amount. An environment recognition device characterized by:

5. The environment recognition device according to claim 1, The environmental conditions and the learning conditions are a country or a weather An environment recognition device characterized by:

6. 6. The environment recognition device according to claim 5, When converting the environmental conditions of a country as the environmental conditions, the image conversion unit converts the color of the road surface, converts the partial shape of the vehicle, adds white lines, and converts the shape and characters of road signs. An environment recognition device characterized by:

7. 6. The environment recognition device according to claim 5, When converting the weather environmental conditions as the environmental conditions, the image conversion unit adds or removes raindrops, converts brightness, changes the lighting state of vehicle lights, changes the lighting state of street lights, changes the brightness inside buildings, removes snow accumulated on roads, removes umbrellas held by pedestrians, or removes fog. An environment recognition device characterized by:

8. The environment recognition device according to claim 1, The image conversion model of the image conversion unit is a convolutional neural network, and in its learning, a model capable of mutually converting between the environmental conditions is utilized, and learning is performed by minimizing a loss function including the loss of the recognition processing unit. An environment recognition device characterized by:

9. The environment recognition device according to claim 2, The environment recognition device is configured as a server-client model, In the image conversion unit, a conversion model storage unit that stores a plurality of conversion models for converting an image under the environmental conditions into an image corresponding to the learning conditions and a conversion process selection unit that selects an image conversion model to be used for image conversion from the conversion model storage unit are configured as a server, and the other components are configured as a client. An environment recognition device characterized by:

10. The environment recognition device according to claim 1, further comprising: a display unit that displays the image acquired by the image acquisition unit and the image converted by the image conversion unit in accordance with the learning conditions; An environment recognition device comprising:

11. An image acquisition unit acquires an image; acquiring an environmental condition of the image acquired by the image acquisition unit; When a learning condition of an image recognition model of a recognition processing unit is different from the environmental condition, an image conversion unit converts the image acquired by the image acquisition unit in accordance with the learning condition; a step of the recognition processing unit recognizing the image converted by the image conversion unit using the image recognition model; An environment recognition method comprising:

12. The environment recognition method according to claim 11, A step in which the image conversion unit sequentially applies a plurality of image conversion models to the image acquired by the image acquisition unit to convert the image into an image corresponding to the learning conditions. An environment recognition method comprising:

Citation Information

Patent Citations

  • System and method for training prediction system

    JP2022191188A