Environment recognition device and environment recognition method

The environment recognition device addresses the challenge of achieving accurate recognition across diverse conditions by using image conversion models to adapt images, reducing the need for extensive data creation and enhancing recognition performance across different weather and geographical conditions.

WO2024237155A1PCT designated stage expired Publication Date: 2024-11-21HITACHI LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/017200
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-16
Filing Date
2024-05-09
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

Current environment recognition systems for autonomous driving face challenges in achieving highly accurate recognition across various environmental conditions, such as weather and geographical locations, due to the high cost and man-hours required for creating correct value data for machine learning models.

Method used

An environment recognition device with an image acquisition unit, environmental condition acquisition unit, image conversion unit, and recognition processing unit that adapts images by converting environmental conditions to match the learning conditions of the machine learning model, using image conversion models to transform images from diverse conditions to specific learning conditions, thereby enabling accurate recognition.

Benefits of technology

This approach allows for highly accurate recognition in various environments while reducing the cost of creating correct value data, by using image conversion models to transform images from diverse conditions to match the learning conditions, thus improving recognition performance across different weather and geographical conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024017200_21112024_PF_FP_ABST
    Figure JP2024017200_21112024_PF_FP_ABST
Patent Text Reader

Abstract

An environment recognition device (1) of the present invention comprises: an image acquisition unit (100); an environmental condition acquisition unit (101) that acquires an environmental condition of an image acquired by the image acquisition unit; a recognition processing unit (103) that performs image recognition processing by means of an image recognition model; and an image conversion unit (102) that converts, in accordance with a learning condition of the image recognition model, the image acquired by the image acquisition unit. In a case where the environmental condition acquired by the environmental condition acquisition unit differs from the learning condition of the image recognition model of the recognition processing unit, the recognition processing unit performs recognition processing of an image that is acquired by the image acquisition unit and converted by the image conversion unit in accordance with the learning condition, thereby achieving highly accurate recognition in various environments while reducing the cost of creating correct answer value data in learning.
Need to check novelty before this filing date? Find Prior Art

Description

Environment recognition device and environment recognition method

[0001] The present invention relates to an environment recognition device and an environment recognition method.

[0002] In order to realize preventive safety functions and autonomous driving, it is effective to capture the driving environment with a camera and perform recognition processing such as image processing on the captured images. From the perspective of recognition performance, machine learning models such as deep learning are used in recognition processing. Recognition processing is expected to achieve high accuracy under various environmental conditions (weather, etc.).

[0003] In contrast, in a machine learning model, it is effective to create correct value data for various environmental conditions as training data and use the data for learning, but creating correct value data for various environmental conditions requires a lot of work. For this reason, for example, in a system for training a prediction system in Patent Document 1, it is described that the prediction system can perform training by using a style model to convert a synthetic image of a low-light environment into a daytime representation as an input to a depth model for generating a depth map (see paragraph 0004).

[0004] Japanese Patent Application Laid-Open No. 2022-191188

[0005] According to the above-mentioned prior art, by converting nighttime data into daytime data, nighttime recognition can be performed using a machine learning model trained only on daytime data. However, environmental conditions in a driving environment include not only nighttime but also many other weather conditions such as rain and fog. In addition, not only the weather but also the country in which the vehicle is traveling is diverse. Therefore, simply responding to nighttime environmental conditions as in the prior art method makes it difficult to achieve highly accurate recognition processing in diverse driving environments.

[0006] An object of the present invention is to provide an environment recognition device and an environment recognition method that can achieve highly accurate recognition in a variety of environments while reducing the cost of creating correct answer data in learning.

[0007] In order to solve the above problem, the environmental recognition device of the present invention comprises an image acquisition unit, an environmental condition acquisition unit that acquires the environmental conditions of an image acquired by the image acquisition unit, a recognition processing unit that performs recognition processing of the image using an image recognition model, and an image conversion unit that converts the image acquired by the image acquisition unit to match the learning conditions of the image recognition model, and when the environmental conditions acquired by the environmental condition acquisition unit differ from the learning conditions of the image recognition model of the recognition processing unit, the recognition processing unit performs recognition processing of the image obtained by converting the image acquired by the image acquisition unit by the image conversion unit to match the learning conditions.

[0008] According to the present invention, it is possible to reduce the cost of creating correct answer data in learning, while achieving highly accurate recognition in a variety of environments.

[0009] 1 is a functional block diagram showing the configuration of an environment recognition device of an embodiment; FIG. 2 is a flowchart explaining the operation of the environment recognition device; FIG. 3 is a diagram showing an example of a captured image acquired by an image acquisition unit; FIG. 4 is a diagram showing the configuration of a transformation model storage unit; FIG. 5 is a diagram showing table information associating execution orders with recognition rates; FIG. 6 is a diagram showing the process of image transformation; FIG. 7 is a diagram showing table information showing the average luminance change amount for each image transformation model; FIG. 8 is a diagram explaining a learning method for an image transformation model from nighttime to sunny weather; FIG. 9 is a diagram explaining a learning method for another image transformation model from nighttime to sunny weather; FIG. 10 is a functional block diagram showing another configuration of an environment recognition device; FIG. 11 is a diagram showing an example of images before and after image transformation displayed on a display unit; FIG. 12 is a diagram showing an example of additionally displaying an image in the middle of transformation on a display unit.

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0011] 1 is a functional block diagram showing the configuration of an environment recognition device 1 according to an embodiment. As shown in FIG. 1, the environment recognition device 1 includes an image acquisition unit 100, an environmental condition acquisition unit 101, an image conversion unit 102, and a recognition processing unit 103.

[0012] Specifically, the environment recognition device 1 is composed of a camera, a computer, memory, a storage device, etc., and the computer executes programs stored in the memory and storage device to realize various functional units described below.

[0013] The following describes in detail the environment recognition device 1 that is installed in a vehicle and connected to one camera, but the invention is not limited to this and can also be applied to a configuration in which multiple cameras are connected. The environment recognition device 1 can also be used in various environment monitoring systems, such as vehicle monitoring systems that use road measuring instruments to monitor the operating status of vehicles.

[0014] The image acquisition unit 100 is a functional unit that acquires images captured by a camera or the like.

[0015] The environmental condition acquisition unit 101 is a functional unit that acquires the environmental conditions of the surrounding environment of the vehicle, and the acquired environmental conditions are set as the environmental conditions of the imaging location of the image acquired by the image acquisition unit 100. In other words, the environmental conditions of the image in this specification refer to environmental information of the imaging location of the image.

[0016] Here, the environmental conditions include information about weather such as clear skies, rain, night (light / dark), snow, fog, etc., and information about countries and regions such as Japan, America, China, Europe, India, etc. This environmental information is acquired or determined from images acquired by the image acquisition unit 100, GPS information of the vehicle, car navigation / map systems, operation information of the vehicle's wipers, etc.

[0017] Specifically, when using captured images, the brightness of the entire image can be used to distinguish between sunny, night (light and dark), and snowy weather, and if raindrops are detected, it can be determined to be rain. Fog can be determined based on the fact that the entire image is blurred and white. Furthermore, by recognizing the characters on signs and road signs from captured images of the driving environment, it is possible to determine the region or country in which the vehicle is traveling.

[0018] Weather information may be obtained from a car navigation system. The region or country in which the vehicle is traveling may also be determined based on GPS information or information from a car navigation or map system. Furthermore, by using information on the operation of the windshield wipers, etc., the rainy environmental conditions can be grasped with high accuracy.

[0019] Although details will be described later, when the environmental conditions acquired by the environmental condition acquisition unit 101 do not match the learning conditions of the recognition processing unit 103 described later, the image conversion unit 102 performs image conversion so that the image acquired by the image acquisition unit 100 matches the learning conditions of the recognition processing unit 103. Here, the learning conditions are the environmental conditions of the image used for training the machine learning model used by the recognition processing unit 103, such as being in Japan and sunny, for example.

[0020] The image conversion unit 102 stores at least one image conversion model to be used for image conversion and selects the one to be used for conversion. These image conversion models may be convolutional neural networks or deep learning models, statistical models trained by other machine learning algorithms, or a series of classical filter processes.

[0021] If the current environmental conditions are America and rainy, and the learning conditions of the recognition processing unit 103 are Japan and sunny, the image conversion unit 102 executes image conversion because the current environmental conditions do not match the learning conditions. At this time, from among the multiple stored image conversion models, an image conversion model from night to sunny and an image conversion model from America to Japan are selected. By executing these two conversion models in order, the image acquired by the image acquisition unit 100 is converted so that it appears as if it were Japan and sunny.

[0022] The recognition processing unit 103 performs recognition processing using images acquired by the image acquisition unit 100 or images converted by the image conversion unit 102. A machine learning model is used in the recognition processing. This machine learning model is trained using images under specific environmental conditions (for example, sunny weather in Japan), and the environmental conditions, i.e., the learning conditions, are stored in the image conversion unit 102. The machine learning model used in the recognition processing may be a segmentation model that identifies each pixel of an image, an object detection model that detects objects appearing in an image, or a depth calculation model that calculates the depth of each pixel of an image.

[0023] Next, the operation of the environment recognition device 1 of the embodiment will be described in detail with reference to the flowchart in Fig. 2. The description will be made on the assumption that the environmental conditions of the driving environment are America and rainy, the recognition processing unit 103 uses a segmentation model for recognizing driving roads, and learning data for Japan and sunny weather is used for learning.

[0024] Fig. 3 is a diagram showing an example of a captured image acquired by the image acquisition unit 100. In Fig. 3, reference numeral 300 denotes a road, reference numeral 301 denotes a sidewalk, and reference numeral 302 denotes raindrops. It is assumed that the color of the asphalt of the road 300 differs between America and Japan.

[0025] The environment recognition device 1 sequentially executes an image acquisition process S101, an environmental condition acquisition process S102, a process S103 for confirming the match between the environmental conditions and the learning conditions, a conversion model selection process S104, a conversion order determination process S105, an image conversion process S106, and a recognition process S107.

[0026] Returning to FIG. 2, in the image acquisition process S101, the image acquisition unit 100 acquires image data from the camera.

[0027] In environmental condition acquisition processing S102, the environmental condition acquisition unit 101 references GPS information and determines that the country the vehicle is currently traveling in is the U.S.A. Then, it acquires weather information from the car navigation system and determines that the current weather is rain.

[0028] In a process S103 for confirming the match between the environmental conditions and the learning conditions, the environmental condition acquisition unit 101 determines that the two conditions do not match (No in S103) because the environmental conditions acquired in the environmental condition acquisition process S102 are America and rain, and the learning conditions of the recognition processing unit 103 stored in the learning condition storage unit 1021 are Japan and sunny, and proceeds to a conversion model selection process S104. If the environmental condition acquisition unit 101 determines that the two conditions match (Yes in S103), the process proceeds to a recognition process S107 without performing image conversion.

[0029] In the conversion model selection process S104, the conversion process selection unit 1023 selects the image conversion model to be used this time from the image conversion models stored in the conversion model storage unit 1022 of the image conversion unit 102. Fig. 4 is a diagram showing the configuration of the conversion model storage unit 1022. The conversion model storage unit 1022 stores, for example, conversion models for converting countries and regions and models for converting weather.

[0030] Specifically, since the environmental conditions acquired in the environmental condition acquisition process S102 are America and rainy, and the learning conditions of the recognition processing unit 103 are Japan and sunny, the conversion model selection process S104 selects two conversion models, one for converting America to Japan and one for converting rainy to sunny, in order to match the environmental conditions with the learning conditions. These conversion models have been trained by the training described below.

[0031] 2, in the conversion order determination process S105, the conversion execution unit 1024 determines the order in which to execute the multiple conversion models selected in the conversion model selection process S104. The order is determined using table information showing the recognition rates of the segmentation models of the recognition processing unit 103 in each execution order, as shown in Fig. 5. The relationship between this execution order and the recognition rate is determined in advance and prepared as table information.

[0032] Specifically, a large number of evaluation images are prepared in advance, image transformation is performed in each execution order, the recognition rate when the transformed images are recognized using a segmentation model is calculated, and the execution order and recognition rate are associated to create the table information shown in Figure 5.

[0033] 5, it can be seen that the recognition rate is higher when the conversion related to the country is performed first and then the conversion related to the weather is performed. Therefore, in the conversion order determination process S105, the conversion order is determined such that the image conversion from America to Japan is performed first, and then the image conversion from rainy to sunny is performed.

[0034] Furthermore, when multiple image transformation models are selected in the transformation model selection process S104, as in this case, the order of execution is determined in the transformation order determination process S105. However, when only a single transformation model is selected, it is determined that only that single image transformation model will be executed without accessing the table information as shown in FIG. 5.

[0035] 2, the conversion execution unit 1024 converts the images acquired in the image acquisition process S101 based on the order determined in the conversion order determination process S105. Fig. 6 is a diagram showing the image conversion process for processing image data in which the environmental conditions are America and rainy by the recognition processing unit 103 in which the learning conditions are Japan and sunny.

[0036] First, to convert from America to Japan, the color of the asphalt of the road in the captured image is changed. This allows the image with American and rainy environmental conditions (US, rain) to be converted to an image with Japanese and rainy environmental conditions (JP, rain). Then, a rain-to-sunny conversion is performed. This removes raindrops and converts the image with Japanese and rainy environmental conditions into an image of Japan and sunny conditions (JP, clear).

[0037] In the recognition process S107 of Figure 2, the recognition processing unit 103 performs a recognition process on the converted image converted in the image conversion process S106, or an image determined to have a match between the environmental conditions and the learning conditions in the process of confirming the match between the environmental conditions and the learning conditions S103.

[0038] As described above, the environment recognition method of the embodiment selects an image transformation model to use by referring to the environmental conditions of the acquired image and the learning conditions of the recognition process, and adaptively performs image transformation of the acquired image so that the environmental conditions of the acquired image match the learning conditions of the recognition process. This allows accurate recognition under a variety of environmental conditions (e.g., America and rain) even when the machine learning model used in the recognition process is trained only under specific conditions (e.g., sunny in Japan). In other words, simply creating correct answer data from the learning data for specific conditions allows the recognition processing unit to achieve high-precision recognition in a variety of environments.

[0039] When multiple models are selected as image transformation models to be used for image transformation, the environment recognition device 1 of the embodiment determines the order of application based on a pre-calculated recognition rate of the recognition processing unit 103. This allows image transformation to be performed taking into account the recognition performance of the recognition processing unit 103, thereby further improving the performance of the recognition processing unit 103.

[0040] In addition, in the environment recognition device 1 of this embodiment, a conversion model for converting environmental conditions related to country and weather is shown in Figure 4. Because the appearance of an image changes significantly depending on the country and weather in which the image is taken, the recognition processing unit usually needs to collect learning data for each country and weather in advance, create correct value data, and learn from it. However, by using the image conversion model in Figure 4 to convert the environmental conditions themselves, namely, country and weather, so that they match the learning conditions of the recognition processing unit, the recognition processing unit 103 can perform highly accurate recognition of a variety of countries and weather conditions simply by learning from correct value data for only specific countries and weather conditions.

[0041] Next, the image transformation model will be described in detail. In the image transformation model of "rainy sky -> sunny" (conversion from rainy to sunny) in Fig. 4, which performs weather-related transformation, it has been explained that the transformation is performed by removing raindrops from the captured image, but it is also possible to perform transformation by adding raindrops, changing the brightness of the entire image, removing umbrellas held by pedestrians, etc.

[0042] Other environmental conditions besides rain include nighttime, snowy, and foggy conditions, which can be converted to sunny. Specifically, at night, the brightness of the entire image is converted, and the lighting status of vehicle lights, street lights, and the brightness inside buildings are changed to convert to sunny. In the case of snowy conditions, the image is converted to sunny by removing snow accumulated on the road surface, the brightness of the entire image, and umbrellas held by pedestrians. In the case of fog, the fog is removed and the image is converted to sunny.

[0043] As described above, the environmental recognition device 1 of the embodiment converts the image into a clear image by removing or modifying conditions characteristic of the weather environment, such as rain, night, snow, and fog, and absorbs the difference in appearance, allowing the recognition processing unit to perform recognition with even higher accuracy.

[0044] The following describes an image transformation model for performing transformation related to a country. The environment recognition device 1 of the embodiment has been described as changing environmental conditions related to a country by transforming the color tone of the asphalt in a captured image using the image transformation model.

[0045] In addition, since the presence or absence of white lines on roads (including road markings and other road paint) differs depending on the country, the image transformation model can be used to perform image transformation to remove or add white lines on the road surface of the captured image, thereby changing the environmental conditions related to the country.

[0046] Furthermore, since the location of vehicle license plates varies from country to country, the image transformation model can be used to perform image transformation to change the location of the license plate in the captured image, thereby changing the environmental conditions related to the country. In this case, the characters written on the license plate themselves may be transformed. Furthermore, since the shape of the vehicle itself differs, the height and width of the vehicle, as well as the outer shape of the vehicle itself, may be transformed.

[0047] Furthermore, the shape of road signs and the characters written on them vary from country to country. For example, stop signs vary in shape, such as hexagons and triangles, depending on the country. Therefore, the image transformation model can be used to perform image transformation to change the shape of road signs, thereby changing the environmental conditions related to the country. In this case, the image transformation model can also be used to perform image transformation on the characters written on road signs.

[0048] As described above, the environmental recognition device 1 of the embodiment converts the color of the asphalt on the road surface, the partial shape of the vehicle, the presence or absence of white lines, the shape and characters of road signs, etc., which are characteristic of the environmental conditions of the country in the captured image, using an image conversion model to absorb differences in appearance depending on the country, thereby enabling the recognition processing unit to perform even more accurate recognition.

[0049] In the above description, in the environment recognition device 1 of the embodiment, the conversion execution unit 1024 performs image conversion in the order of highest recognition rate as shown in FIG. 5 in the conversion order determination process S105 (FIG. 2). However, the conversion execution unit 1024 may perform image conversion in the order of lowest average luminance change.

[0050] Here, for example, in the case of a rainy-to-sunny image conversion model, the average luminance change is the average value of the luminance difference of each pixel before and after conversion when a rainy image is input and converted into a sunny image. The average luminance change for each image conversion model may be calculated in advance by converting a large number of images, or the image recognized by the conversion execution unit 1024 may be converted in real time to determine the average luminance change before and after conversion.

[0051] 7 is a diagram showing table information indicating the average luminance change amount for each image transformation model. In FIG. 7, the average luminance change amount in the image transformation model from America to Japan is 30, and the average luminance change amount in the image transformation model from rainy to sunny is 80. Therefore, the image transformation model from America to Japan has a smaller amount of change in image appearance. Therefore, the transformation execution unit 1024 first executes image transformation from America to Japan, which has a smaller average luminance change amount, and then executes image transformation from rainy to sunny.

[0052] The reason why the application order of the above image transformation models is in the order of the smallest average luminance change and the smallest change in appearance is that significant changes in the appearance of an image during image transformation have a high risk of failure and are highly difficult. Therefore, image transformations are performed in order starting with the smallest change in appearance, and then image transformations that result in a large change in appearance and are more difficult are performed on the image from which those environmental conditions have been removed. This reduces the risk of image transformation failure and enables further improvement in the accuracy of the recognition processing unit 103.

[0053] Next, a method for learning a nighttime to sunny image conversion model will be described with reference to Figures 8 and 9. It goes without saying that the following description is not limited to nighttime to sunny image conversion.

[0054] In Figure 8, N101 is the image conversion model from nighttime to sunny. For learning, not only the nighttime to sunny image conversion model N101 but also the sunny to nighttime conversion model N100 is used. N100 and N101 are convolutional neural networks. For learning, real images taken at night and real images taken on sunny days are used. Three types of loss are used for learning loss.

[0055] The first loss is consistency loss. For consistency loss, a real nighttime image is input to N101, and an image converted to sunny weather (sunny weather CG image) is generated. Then, the sunny weather CG image is input to N100, and an image converted to nighttime weather (nighttime CG image) is generated. Then, the difference in luminance for each pixel between the input real nighttime image and the nighttime CG image is calculated as the loss. Similarly, a real sunny weather image is input to N100, and the conversion result is input to N101, and the difference in luminance between the output and the real sunny weather image is calculated as the loss. The sum of these two differences is the consistency loss.

[0056] The second loss is an adversarial loss. In the adversarial loss, a real nighttime image is input to N101, and a converted sunny CG image is obtained. The obtained sunny CG image and a real sunny image are input to the sunny classifier N103. The sunny classifier N103 is a classifier that determines whether the input image is a CG image or a real image.

[0057] Similarly, a clear real image is input to N100, a converted nighttime CG image is obtained, and the nighttime CG image is input to the nighttime classifier N102. A nighttime real image is also input to the nighttime classifier N102. The nighttime classifier N102 determines whether the input image is a CG image or a real image.

[0058] The cross-entropy is calculated for the classification results of the sunny and nighttime classifiers, and the sum is used as the adversarial loss. The first loss, the consistency loss, and the second loss, the adversarial loss, are the same as those used in the well-known CycleGAN.

[0059] The third loss is the loss for the classification result of the recognition processing unit, and a method for calculating the loss will be explained with reference to Fig. 9. N100 and N101 in Fig. 9 are the same image transformation models as N100 and N101 in Fig. 8. Here, the recognition processing unit 103 is also configured with a convolutional neural network and has been trained to be able to segment sunny images, and the parameters of this network are not updated in the training of the image transformation model.

[0060] The third loss uses a real sunny image I100 and a ground truth value I101 for that real image, where I101 is the ground truth value for segmentation.

[0061] First, I100 is input into a sunny-to-nighttime image conversion model N100 to obtain a nighttime CG image I102. Then, I102 is input into a nighttime-to-sunny image conversion model N101 to obtain a sunny CG image I103. Then, the sunny CG image I103 and the correct answer value I101 are input into the recognition processing unit, and cross-entropy is calculated as the loss related to segmentation.

[0062] The loss obtained by adding this third loss to the first and second losses is the loss used in training the image translation model. In training, the image translation models N100 and N101 and the classifiers N102 and N103 are alternately optimized.

[0063] As described above, in the environment recognition device 1 of the embodiment, in learning the image transformation model, a model capable of mutual transformation between environmental conditions (not only a transformation model from night to sunny, but also a transformation model from sunny to night) is used.

[0064] As a result, the actual images acquired for each environmental condition (actual images at night and on a sunny day) required for training the image transformation model do not need to be a pair that matches perfectly except for the environmental conditions (a pair of images at night and on a sunny day acquired at the exact same location), and learning is possible even if they are acquired at different locations. This reduces the cost of acquiring the data required for training the image transformation model.

[0065] Furthermore, in the environment recognition device 1 of the embodiment, an image (actual image of a sunny day) that meets the learning conditions of the recognition processing unit 103 is input, image conversion is performed twice (conversion from sunny to nighttime, and then conversion from nighttime to sunny), and the loss of the recognition processing unit 103 is calculated using the converted CG image and the correct value, and an image conversion model is learned based on the loss.

[0066] This allows learning by adding the loss of the recognition processing unit 103 to the loss of the image transformation model, making it possible to build an image transformation model suitable for recognition. Also, by utilizing the above-described learning (converting from sunny to nighttime, then converting from nighttime to sunny, and defining the loss of the recognition unit using the converted sunny CG image), the loss of the recognition unit can be calculated using only the correct value for the learning condition (sunny) of the recognition processing unit, and the loss of the recognition processing unit can be added without creating correct value data for the nighttime data, thereby reducing the cost of creating correct value data.

[0067] Next, an example in which the environment recognition device 1 of the embodiment is configured as a server-client model will be described. In this configuration, of the components shown in Fig. 1, the conversion model storage unit 1022 and the conversion process selection unit 1023 of the image conversion unit 102 are configured as server components, and the other components are configured as client components. In other words, in this configuration, the server determines the image conversion model required for the client to execute the recognition process.

[0068] 2 that the environmental conditions and the learning conditions do not match, the client notifies the server of the environmental conditions and the learning conditions, and the server performs the processes of S104 and S105 in Fig. 2 using the configuration of the conversion model storage unit 1022 and the conversion process selection unit 1023, and notifies the client of the image conversion model selected from the image conversion models stored in the conversion model storage unit 1022. The client performs image conversion in S106 using the image conversion model notified from the server, and performs recognition processing in S107.

[0069] In this way, by configuring the environment recognition device 1 as a server-client model, the client does not need to have an area for storing multiple image transformation models, and only needs to have an area for one image transformation model, which reduces the memory costs of the client.

[0070] By adding an image transformation model that converts new environmental conditions to the server, existing clients can perform recognition processing for new environmental conditions, thereby improving the scalability and availability of recognition processing.

[0071] As described above, when the environmental conditions under which an image is acquired differ from the learning conditions of the recognition processing unit 103, the environment recognition device 1 of the embodiment can perform recognition processing under many environmental conditions by converting the acquired image to match the learning conditions of the recognition processing unit 103. For this reason, the environment recognition device 1 may be configured to display the image before and after the image conversion so that the image can be confirmed.

[0072] 10 is a functional block diagram showing the configuration of an environment recognition device 1 that displays images before and after image conversion. In detail, the environment recognition device 1 in FIG. 10 is configured by adding a display unit 104 to the environment recognition device 1 shown in FIG.

[0073] Fig. 11 is a diagram showing an example of images before and after image conversion displayed on the display unit 104. In detail, Fig. 11 shows a display example in which an image of America and rain is converted into an image of Japan and sunny weather.

[0074] The environment recognition device 1 may also display on the display unit 104 how an image changes when multiple image transformation models are applied. Fig. 12 is a diagram showing an example of additionally displaying an image in the middle of transformation. In more detail, when an image of America and rain is transformed into an image of Japan and sunny weather, an image (JP, rain) to which an image transformation model for transforming into an image of Japan and rain is applied is displayed, and then an image of Japan and sunny weather is displayed.

[0075] As shown in Figure 12, by displaying the image during conversion, it is possible to check the conversion results of the conversion order determined by the conversion process selection unit 1023 in the conversion order determination process S105, so it is also possible to manually change the application order of the image conversion models determined based on the recognition rate described in Figure 5 or the application order of the image conversion models determined based on the average luminance change amount described in Figure 7.

[0076] Furthermore, the present invention is not limited to the above-described examples, and various modifications are included. The above-described examples have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment.

[0077] REFERENCE SIGNS LIST 100 Image acquisition unit 101 Environmental condition acquisition unit 102 Image conversion unit 1021 Learning condition storage unit 1022 Conversion model storage unit 1023 Conversion process selection unit 1024 Conversion execution unit 103 Recognition processing unit 104 Display unit

Claims

1. An environmental recognition device comprising: an image acquisition unit that acquires an image; an environmental condition acquisition unit that acquires the environmental conditions of the image acquired by the image acquisition unit; a recognition processing unit that performs image recognition processing using an image recognition model; and an image conversion unit that converts the image acquired by the image acquisition unit to match the learning conditions of the image recognition model, wherein when the environmental conditions acquired by the environmental condition acquisition unit differ from the learning conditions of the image recognition model of the recognition processing unit, the recognition processing unit performs recognition processing on the image acquired by the image acquisition unit by converting it using the image conversion unit to match the learning conditions.

2. An environmental recognition device as described in claim 1, wherein the image conversion unit sequentially applies a plurality of image conversion models to the image acquired by the image acquisition unit to convert the image of the environmental conditions into an image corresponding to the learning conditions.

3. An environmental recognition device as described in claim 2, characterized in that the image transformation unit determines the application order with the highest recognition rate based on the recognition rate of the recognition processing unit for each application order of the image transformation models, and applies the image transformation models sequentially in the determined application order.

4. An environmental recognition device according to claim 2, wherein the image conversion unit calculates an average luminance change amount, which is the average value of the luminance difference between each pixel before and after image conversion for each image conversion model, and sequentially applies image conversion models in order of the smallest average luminance change amount.

5. An environment recognition device according to claim 1, characterized in that the environmental conditions and the learning conditions are a country or weather.

6. An environmental recognition device as described in claim 5, wherein the image conversion unit, when converting the environmental conditions of a country as the environmental conditions, converts the color of road surfaces, etc., converts partial shapes of vehicles, adds white lines, and converts the shapes and characters of road signs.

7. An environmental recognition device as described in claim 5, wherein the image conversion unit, when converting weather environmental conditions as the environmental conditions, adds or removes raindrops, converts brightness, changes the lighting status of vehicle lights, changes the lighting status of street lights, changes the brightness inside buildings, removes snow accumulated on roads, etc., removes umbrellas held by pedestrians, or removes fog.

8. An environmental recognition device as described in claim 1, wherein the image conversion model of the image conversion unit is a convolutional neural network, and in its learning, a model capable of mutual conversion between the environmental conditions is utilized, and learning is performed by minimizing a loss function including the loss of the recognition processing unit.

9. An environmental recognition device as described in claim 2, wherein the environmental recognition device is configured as a server-client model, and the image conversion unit includes a conversion model storage unit that stores a plurality of conversion models for converting an image of the environmental conditions into an image corresponding to the learning conditions, and a conversion process selection unit that selects an image conversion model to be used for image conversion from the conversion model storage unit, which are configured as a server, and the other components are configured as a client.

10. An environmental recognition device as described in claim 1, further comprising a display unit for displaying the image acquired by the image acquisition unit and the image converted by the image conversion unit to match the learning conditions.

11. An environmental recognition method comprising the steps of: an image acquisition unit acquiring an image; acquiring environmental conditions of the image acquired by the image acquisition unit; and, when the learning conditions of an image recognition model of a recognition processing unit differ from the environmental conditions, an image conversion unit converting the image acquired by the image acquisition unit to match the learning conditions; and a recognition processing unit recognizing the image converted by the image conversion unit using the image recognition model.

12. The environmental recognition method according to claim 11, further comprising a step in which the image conversion unit sequentially applies a plurality of image conversion models to the image acquired by the image acquisition unit to convert the image into an image corresponding to the learning conditions.

Citation Information

Patent Citations

  • Automatic painting creation system by recipe

    JP2017123131A

  • Information processing device, information processing method, and program

    JP2019125116A

  • Information processing apparatus and program

    JP2019213098A

  • Image generator, image generation method, and program

    JP2022115590A