Image processing device and autonomous mobile device

Abstract

This enables robust image recognition with a simple configuration. [Solution] The image processing device 100 includes a storage device SD that stores first image data obtained by imaging and condition information indicating the imaging conditions of the first image data as training data, a recognition device RC having a recognition model constructed by machine learning using the training data, and an image generator IG that applies predetermined image processing to second image data obtained by imaging to generate third image data to be output to the recognition device RC. The image generator IG applies predetermined image processing to the second image data to generate third image data corresponding to the imaging conditions of the first image data.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus for processing image data and an autonomous mobile body.

Background Art

[0002] Conventionally, there is known an apparatus configured to input a captured image from an in-vehicle camera into a neural network and perform object recognition through estimation processing of the neural network (see, for example, Patent Document 1). In the apparatus described in Patent Document 1, in order to obtain robust recognition results, learning data that is difficult to collect from real images is supplemented with CG (computer graphics) images.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in order to obtain robust recognition results, it is necessary to prepare CG images that reflect a wide variety of situations observed in real environments, which requires enormous labor and a high-performance computer environment.

Means for Solving the Problems

[0005] An image processing apparatus according to one aspect of the present invention is an image processing apparatus that processes two-dimensional or three-dimensional images. The image processing apparatus comprises a storage device that stores first image data obtained by imaging and condition information indicating the imaging conditions of the first image data as training data, a recognition device having a recognition model constructed by machine learning using the training data, and an image generator that applies predetermined image processing to second image data obtained by imaging to generate third image data to be output to the recognition device. The image generator applies predetermined image processing to the second image data to generate third image data corresponding to the imaging conditions of the first image data.

[0006] Another aspect of the present invention is an autonomous mobile body equipped with an image processing device that processes two-dimensional or three-dimensional images. The autonomous mobile body comprises a mobile actuator and a mobile control unit. The image processing device comprises a storage device that stores first image data obtained by imaging and condition information indicating the imaging conditions of the first image data as training data, a recognizer having a recognition model constructed by machine learning using the training data, and an image generator that applies predetermined image processing to second image data obtained by imaging to generate third image data to be output to the recognizer. The recognition model is a learning model that recognizes objects in an image indicated by the image data based on the input image data, the image generator applies predetermined image processing to the second image data to generate third image data corresponding to the imaging conditions of the first image data, and the mobile control unit controls the mobile actuator based on the recognition result of the recognition model using the third image data as input. [Effects of the Invention]

[0007] According to the present invention, robust image recognition can be achieved with a simple configuration. [Brief explanation of the drawing]

[0008] [Figure 1] A block diagram schematically showing the overall configuration of an image processing system according to an embodiment of the present invention. [Figure 2]A block diagram illustrating the main components of a vehicle-mounted driving control system. [Figure 3A] A figure showing an example of an acquired image. [Figure 3B] A figure showing other examples of ROM images. [Figure 3C] A figure showing other examples of ROM images. [Figure 4] A figure showing other examples of ROM images. [Figure 5] A flowchart showing an example of the process executed by the controller in Figure 2. [Modes for carrying out the invention]

[0009] Embodiments of the invention will now be described with reference to the drawings. The image processing system (hereinafter also referred to as the image processing device) according to an embodiment of the present invention is a device for generating training data for an image recognition model used in the operation of a vehicle. The vehicle to which the image processing system according to this embodiment is applied may be referred to as the "own vehicle" to distinguish it from other vehicles.

[0010] Figure 1 is a block diagram schematically showing the overall configuration of the image processing system 100 according to this embodiment. As shown in Figure 1, the image processing system 100 includes a learning processing device 1 and an image generation device 2. The learning processing device 1 and the image generation device 2 are connected to each other via a communication network such as CAN (Controller Area Network).

[0011] The learning processing device 1 and the image generation device 2 may be configured as a single device. Furthermore, the image processing system 100 may include a learning processing device 1 mounted on another vehicle. In other words, the image processing system 100 may include multiple learning processing devices 1. Also, the image processing system 100 may include a learning processing device 1 configured as an external server device or the like.

[0012] The learning processing unit 1 has an imaging device CA. When the learning processing unit 1 is mounted on a vehicle, the imaging device CA is an on-board camera. The learning processing unit 1 stores the image data (hereinafter also simply referred to as the image) acquired by the imaging device CA in the storage device SD. The image data stored in the storage device SD is used as training data for model learning (machine learning).

[0013] Furthermore, the learning processing device 1 acquires reference information, which is information about the vehicle itself and its external environment, and identifies the imaging conditions when the imaging device CA acquired an image based on the reference information. The learning processing device 1 stores the information indicating the identified imaging conditions (hereinafter referred to as condition information) in the storage device SD.

[0014] Reference information includes map data, including the roads the vehicle travels on, vehicle information, and external information. Map data may be structured as two-dimensional or three-dimensional data. Vehicle information is information about the vehicle, including location information indicating the vehicle's position. External information is information about the vehicle's external environment, including weather information.

[0015] The learning processing unit 1 constructs a recognition model using machine learning, using the captured images and conditional information stored in the storage device SD as training data. The recognition model is a learning model that recognizes objects in an image indicated by the input image data. The learning processing unit 1 outputs the recognition model and the conditional information used to construct the recognition model to the recognition unit RC.

[0016] The recognition unit RC stores recognition models in association with imaging conditions indicated by the condition information used to construct the recognition model. In other words, the recognition unit RC stores a recognition model for each imaging condition. The recognition unit RC also stores information indicating the correspondence between imaging conditions and recognition models (hereinafter referred to as model correspondence information).

[0017] When a captured image is acquired by the imaging device CA, the image generation device 2 identifies the imaging condition of the captured image. The image generation device 2 inputs the captured image of the imaging device CA together with the identified imaging condition to the image generator IG.

[0018] When a recognition model corresponding to the identified imaging condition (hereinafter referred to as the corresponding recognition model) is not stored in the recognizer RC, the image generator IG applies inverse rendering processing to the captured image of the imaging device CA. The inverse rendering processing is processing that applies at least one of CG processing and image processing by image generation AI (Artificial Intelligence) to an input image to generate a captured image under a specific imaging condition. The image generator IG applies the inverse rendering processing to the captured image of the imaging device CA to generate a captured image under an imaging condition suitable for the recognition model stored in the recognizer RC.

[0019] The image generator IG outputs the image generated by the inverse rendering processing (hereinafter referred to as the generated image) to the recognizer RC. When the corresponding recognition model is stored in the recognizer RC, the inverse rendering processing is not executed, and the captured image of the imaging device CA is directly input to the recognizer RC. The recognizer RC recognizes an object in the generated image using the recognition model and outputs a recognition result. The recognition result output from the recognizer RC is used for travel control of the host vehicle.

[0020] Figure 2 is a block diagram illustrating the essential configuration of the host vehicle, more specifically, the essential configuration of the travel control device mounted on the host vehicle. As shown in Figure 2, the travel control device 3 includes the image processing system 100 (the learning processing device 1 and the image generation device 2) of Figure 1. The travel control device 3 includes a controller 30, a communication unit 33, a camera 34, a lidar 35, a radar 36, a measurement unit 37, and an actuator AC.

[0021] The communication unit 33 is a communication interface that connects the driving control device 3 to CAN or other communication networks. Other communication networks include not only public wireless communication networks such as the Internet and mobile phone networks, but also closed communication networks established for each designated management area, such as wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), etc. The driving control device 3 sends and receives information with external devices via the communication unit 33.

[0022] Camera 34 may be a monocular or stereo camera and captures images of the area around the vehicle. Camera 34 is mounted, for example, at a predetermined position on the front of the vehicle and continuously captures images of the space in front of the vehicle at a predetermined frame rate, outputting image data as detection information to the controller 30. Camera 34 corresponds to the imaging device CA in Figure 1.

[0023] Radar 36 is mounted on the vehicle and detects other vehicles and obstacles in the vicinity of the vehicle by emitting electromagnetic waves and detecting the reflected waves. Radar 36 outputs detection values ​​(detection data) as detection information to the controller 30. Lidar 35 is mounted on the vehicle and detects the distance from the vehicle to surrounding obstacles by measuring the scattered light from the vehicle's omnidirectional illumination. Lidar 35 outputs detection values ​​(detection data) as detection information to the controller 30.

[0024] The measurement unit 37 has a positioning sensor that receives positioning signals transmitted from positioning satellites, and uses the positioning information received by the positioning sensor to measure the current position (latitude, longitude, altitude) of the vehicle. Positioning satellites are artificial satellites such as GPS satellites and quasi-zenith satellites. The measurement unit 37 also has a magnetic sensor (electronic compass) to measure the orientation of the vehicle. The measurement unit 37 may also estimate the orientation of the vehicle based on the changes in the vehicle's position.

[0025] Actuator AC is a drive actuator used to control the movement of the vehicle. When the drive source is an engine, actuator AC includes a throttle actuator that adjusts the opening degree (throttle opening) of the engine's throttle valve. When the drive source is a drive motor, the drive motor is included in actuator AC. Brake actuators that operate the vehicle's braking system and steering actuators that drive the steering system are also included in actuator AC.

[0026] The controller 30 is comprised of a computer having an arithmetic unit 31 such as a CPU (microprocessor), a storage unit 32 such as ROM or RAM, and other peripheral circuits not shown, such as an I / O interface. The storage unit 32 corresponds to the storage device SD in Figure 1. The storage unit 32 stores map data, including road maps. The arithmetic unit 31 functions as a learning data acquisition unit 311, a learning unit 312, an image generation unit 313, an image recognition unit 314, and a driving control unit 315 by executing programs stored in the storage unit 32.

[0027] The communication unit 33, camera 34, lidar 35, radar 36, measurement unit 37, learning data acquisition unit 311, and learning unit 312 are included in the learning processing device 1. The communication unit 33, camera 34, lidar 35, radar 36, measurement unit 37, image generation unit 313, image recognition unit 314, and driving control unit 315 are included in the image generation device 2. The image generation unit 313 functions as the image generator IG in Figure 1. The image recognition unit 314, together with the storage unit 32, functions as the recognition unit RC in Figure 1.

[0028] The learning data acquisition unit 311 stores the captured images acquired by the camera 34 in the storage unit 32. The captured images acquired by the learning data acquisition unit 311 are called learning images. The learning images stored in the storage unit 32 are used as learning data by the learning unit 312.

[0029] Furthermore, the learning data acquisition unit 311 acquires reference information. Specifically, the learning data acquisition unit 311 acquires map data stored in the storage unit 32. The learning data acquisition unit 311 may also acquire map information from a map provisioning server (not shown). Based on the acquired map data and the information obtained by the measurement unit 37, the learning data acquisition unit 311 recognizes (acquires) the position and orientation of the vehicle on the map. In addition, the learning data acquisition unit 311 acquires external information from an external server device or the like via the communication unit 13.

[0030] External information includes information about weather, lighting conditions, surrounding objects, road surface conditions, shadows, and traffic conditions. The learning data acquisition unit 311 acquires weather information from an external weather information server (not shown) via the communication unit 13. Weather information includes information indicating the type of weather (e.g., sunny, cloudy, snowy), whether or not there is rainfall, the type of rain (drizzle, heavy rain, etc.), whether or not there is wind, and the type of wind (e.g., storm, turbulent).

[0031] Information indicating the light environment includes information that can identify the position of celestial bodies (hereinafter referred to as astronomical information). The learning data acquisition unit 311 estimates (calculates) the positional relationship between the vehicle and the celestial body at that time, based on the vehicle's position or orientation and the astronomical information. Based on the estimated positional relationship, the learning data acquisition unit 311 calculates the angle of incidence and characteristics (e.g., light quantity, illuminance) of light from a specific celestial body (e.g., the sun, the moon) as a light source. The learning data acquisition unit 311 may also calculate the angle of incidence and characteristics of light from celestial bodies based on learning images or using sensors not shown, such as an illuminance sensor. Furthermore, information indicating the light environment may also include information that can identify the position of light sources other than celestial bodies, such as lamps, streetlights, etc.

[0032] Information about surrounding objects includes information indicating the surface characteristics (hereinafter referred to as surface characteristics) of structures (e.g., buildings, road signs) and natural objects (e.g., vegetation) around the vehicle. Surface characteristics are surface material and surface texture, and are characteristics that affect the appearance of structures and natural objects, such as gloss, reflection, and unevenness. The learning data acquisition unit 311 recognizes the surface characteristics of surrounding objects based on the learning images. If the surface characteristics information is included in the map data, the learning data acquisition unit 311 may recognize the surface characteristics of surrounding objects based on the map data.

[0033] Information regarding the road surface includes information indicating the road surface condition. Road surface conditions include dry conditions, icy conditions, snowy conditions, etc. The learning data acquisition unit 311 recognizes the road surface condition based on the images used during training. The learning data acquisition unit 311 may also estimate the road surface condition based on information regarding the weather or the vehicle's slip ratio.

[0034] Shadow information indicates whether or not shadows (projected shadows or self-shadows) are being cast by a light source on the road surface, traffic participants, objects, etc. The learning data acquisition unit 311 recognizes shadows formed on the road surface, etc., based on the training images. The learning data acquisition unit 311 may also estimate the position and shape of shadows formed on the road surface, etc., based on the positional relationship between the vehicle and the light source.

[0035] Traffic information includes traffic information indicating road traffic conditions (such as congestion). The learning data acquisition unit 311 acquires traffic information for the road the vehicle is traveling on from an external traffic information distribution server (not shown) via the communication unit 13.

[0036] The learning data acquisition unit 311 identifies the imaging conditions for the learning image, more specifically, the imaging conditions at the time the learning image was acquired by the camera 34, such as weather, lighting environment, surface characteristics of surrounding objects, road surface conditions, shadows, and traffic conditions. The learning image itself may also be used to identify the imaging conditions. Specifically, the learning data acquisition unit 311 may analyze the learning image and identify the imaging conditions for the learning image based on the analysis results.

[0037] The learning data acquisition unit 311 stores condition information indicating the identified imaging conditions in the storage unit 32, associating it with the learning images. The condition information stored in the storage unit 32 is used as learning data by the learning unit 312.

[0038] Figures 3A to 3C show examples of images captured at time T1 by the onboard camera (camera 34) of the vehicle traveling on the highway HW. Vehicles VH1 to VH3 are vehicles ahead of the vehicle traveling in the same direction. Figure 3A shows the captured image when the weather at time T1 was sunny. Figure 3B shows the captured image when the weather at time T1 was snowing. Figure 3C shows the captured image when the weather at time T1 was sunny and the sun SU was in front of the vehicle, creating a backlit condition.

[0039] When the training image is the image captured in Figure 3A, the training data acquisition unit 311 stores condition information indicating that the weather is sunny in the storage unit 32. When the training image is the image captured in Figure 3B, the training data acquisition unit 311 stores condition information indicating that the weather is snowy in the storage unit 32. Furthermore, when the training image is the image captured in Figure 3C, the training data acquisition unit 311 stores condition information indicating that the lighting environment is backlit and that shadows are formed on the road surface behind the vehicles VH1~VH3 and the planted area VG (on the near side in the figure).

[0040] The learning unit 312 constructs a recognition model using machine learning with the learning data (learning images and condition information) stored in the storage unit 32. The learning unit 312 stores the constructed recognition model and the condition information used to construct the recognition model in the storage unit 32. The learning unit 312 also stores model correspondence information in the storage unit 32 that shows the correspondence between the recognition model and the imaging conditions indicated by the condition information. At this time, if a recognition model associated with the same imaging conditions as indicated by the condition information is already stored in the storage unit 32, the learning unit 312 updates that recognition model with the constructed recognition model.

[0041] The image generation unit 313 acquires reference information when an image (hereinafter referred to as the current image) is captured by the camera 34 while the vehicle is in motion. Based on the acquired reference information, the image generation unit 313 identifies the imaging conditions for the current image, more specifically, the imaging conditions (hereinafter referred to as the recognition imaging conditions) that indicate the weather, lighting environment, surface characteristics of surrounding objects, road surface conditions, shadows, and traffic conditions at the time the current image was acquired by the camera 34. The method for acquiring reference information and identifying imaging conditions is the same as that of the learning data acquisition unit 311.

[0042] The image generation unit 313 refers to the model correspondence information and determines whether a corresponding recognition model, specifically a recognition model corresponding to the recognition imaging conditions, is stored in the storage unit 32. If it is determined that a corresponding recognition model is stored in the storage unit 32, the image recognition unit 314 uses the corresponding recognition model to recognize an object in the current image. Specifically, the image recognition unit 314 inputs the current image into the corresponding recognition model and obtains the object recognition result output from the corresponding recognition model.

[0043] On the other hand, if the image generation unit 313 determines that no corresponding recognition model is stored in the storage unit 32, it applies inverse rendering processing to the current image to generate an image with imaging conditions suitable for a recognition model stored in the storage unit 32 (hereinafter referred to as an unsupported recognition model). If multiple unsupported recognition models are stored in the storage unit 32, the image generation unit 313 selects one of the unsupported recognition models and generates an image with imaging conditions suitable for that unsupported recognition model.

[0044] For example, the image generation unit 313 may preferentially select an unsupported recognition model when the content of a predetermined item of the imaging conditions supported by that unsupported recognition model (e.g., weather) matches the imaging conditions at the time of recognition. Alternatively, the image generation unit 313 may calculate the similarity between the imaging conditions supported by the unsupported recognition model and the imaging conditions at the time of recognition, and preferentially select an unsupported recognition model with a high similarity. The similarity may be determined based on the number of matching items among the items of the imaging conditions (weather, light environment, surrounding objects, road surface conditions, shadows, and traffic conditions), or it may be determined based on other criteria.

[0045] Here, we will explain the inverse rendering process. The image generation unit 313 applies CG processing to the current image so that the imaging conditions of the current image (weather, lighting environment, surface characteristics of surrounding objects, road surface conditions, shadows, traffic conditions, etc.) match the imaging conditions supported by the unsupported recognition model.

[0046] Furthermore, images to which CG processing has been applied have different tonal characteristics compared to the actual captured image (hereinafter referred to as the "real image"). More specifically, images to which CG processing has been applied generally show greater changes in brightness and RGB values ​​at the edges compared to the real image, while noise and grayscale changes are less pronounced in areas that are not edges. Therefore, if the current image after CG processing is input to an unsupported recognition model, the desired recognition result may not be obtained from the unsupported recognition model.

[0047] Therefore, the image generation unit 313 converts the tonal characteristics of the current image after CG processing in order to reduce the difference in tonal characteristics between the current image after CG processing and the actual image. More specifically, the image generation unit 313 applies characteristic conversion processing to the current image after CG processing. More specifically, the image generation unit 313 converts the characteristics of the current image after CG processing in order to reduce changes in the edge areas and increase noise and grayscale changes in areas that are not edge areas.

[0048] Furthermore, the image generation unit 313 may apply reverse rendering processing using image generation AI (Artificial Intelligence) to the current image instead of reverse rendering processing using CG processing. More specifically, the image generation unit 313 may apply reverse rendering processing to the current image using image generation AI capable of generating image data corresponding to the input imaging conditions.

[0049] In the reverse rendering process using the image generation AI, the image generation unit 313 inputs the current image and instruction text indicating the imaging conditions supported by the unsupported recognition model to the image generation AI. As a result, the current image to which the reverse rendering process has been applied is output from the image generation AI.

[0050] Figure 4 shows an example of an image captured at time T2 by the onboard camera (camera 34) of the moving vehicle. Figure 4 shows an image captured during rainfall. For example, if the current image is the image captured in Figure 4, and only a recognition model corresponding to sunny weather is stored in the storage unit 32, that is, if a recognition model corresponding to rainfall is not stored, the image generation unit 313 applies inverse rendering processing to the current image. Specifically, the image generation unit 313 applies inverse rendering processing to convert the imaging conditions of the current image from rainfall to sunny weather.

[0051] When applying inverse rendering processing by the image generation AI to the captured image in Figure 4, the image generation unit 313 inputs instructions to the image generation AI such as, "Please change this image from a 'rainy day' to a 'sunny day'. Remove the clouds and falling rain from the sky, and also remove the wet areas and puddles on the ground. Make the sky blue and adjust the brightness to create the atmosphere of a sunny day."

[0052] The image recognition unit 314 recognizes objects in the current image using an unsupported recognition model. Specifically, the image recognition unit 314 inputs the current image, to which inverse rendering has been applied, into the unsupported recognition model and obtains the object recognition result output from the unsupported recognition model.

[0053] The driving control unit 315 controls the actuator AC based on the object recognition results obtained by the image recognition unit 314. More specifically, the driving control unit 315 controls the actuator AC so that the vehicle travels along the lane boundary lines recognized by the image recognition unit 314. The driving control unit 315 also controls the actuator AC to avoid collisions or contact between the vehicle and traffic participants or objects on the road recognized by the image recognition unit 314.

[0054] Figure 5 is a flowchart showing an example of a process executed by the controller 30 of the driving control device 3 according to a predetermined program. The process shown in this flowchart starts when the controller 30 is activated and is repeated at predetermined intervals.

[0055] First, in step S1, the controller 30 determines whether or not an image (hereinafter referred to as a camera image) has been acquired by the camera 34. If the result in step S1 is negative, the controller 30 terminates the process.

[0056] If step S1 is affirmed, in step S2 the controller 30 acquires reference information. In step S3 the controller 30 identifies the camera image acquisition conditions based on the reference information acquired in step S2. In step S4 the controller 30 determines whether or not a recognition model corresponding to the imaging conditions identified in step S3 is stored in the memory unit 32.

[0057] If affirmed in step S4, in step S5 the controller 30 inputs the camera image into the recognition model. If denied in step S4, in step S6 the controller 30 applies inverse rendering to the camera image. In step S7 the controller 30 inputs the image generated by the inverse rendering process, i.e., the camera image to which the inverse rendering process has been applied, into the recognition model.

[0058] In step S8, the controller 30 acquires the object recognition result output from the recognition model. In step S9, the controller 30 controls the actuator AC based on the object recognition result acquired in step S8.

[0059] According to the embodiments described above, the following effects and advantages can be obtained. (1) The image processing system 100 includes a storage device SD (storage unit 32) that stores first image data (training image) obtained by imaging and condition information indicating the imaging conditions of the training image as training data, a recognition unit RC (image recognition unit 314 and storage unit 32) that has a recognition model constructed by machine learning using the training data, and an image generator IG (image generation unit 313) that generates third image data to be output to the recognition unit RC by applying predetermined image processing to second image data (current image) obtained by imaging (Figures 1 and 2). The image generator IG applies predetermined image processing (inverse rendering processing) to the second image data to generate third image data corresponding to the imaging conditions of the first image data. This makes it possible to improve the image recognition accuracy of an image processing device trained with limited training data. As a result, robust image recognition can be realized with a simple configuration.

[0060] (2) The image generator IG applies rendering processing corresponding to the imaging conditions of the first image data to the second image data to generate a third image data as a CG image. This makes the input image closer to the image under the conditions learned by the recognition system. As a result, the image recognition accuracy is improved.

[0061] (3) The image generator IG inputs the second image data and text indicating the imaging conditions of the first image data to an image generation AI capable of generating image data corresponding to the input imaging conditions, and acquires the image data output from the image generation AI as the third image data. This makes the input image closer to the image of the conditions learned by the recognition system. As a result, the image recognition accuracy is improved.

[0062] (4) The image generator IG further recognizes the imaging conditions of the second image data. Based on the imaging conditions of the first image data and the second image data, the image generator IG generates the third image data. In this way, by generating images while considering the imaging conditions at the time of input, images can be generated with greater precision.

[0063] (5) The image processing system 100 further comprises a communication unit 13 and a learning data acquisition unit 311, which is an information acquisition unit that acquires information from the outside via the communication unit 13 (Figure 2). The learning data acquisition unit 311 collects information related to the imaging conditions of the first image data, generates condition information based on the collected information, and stores the condition information in the storage device SD (storage unit 32). This makes it possible to acquire imaging conditions using external information.

[0064] (6) The information related to the imaging conditions of the first image data includes one of the following: weather information, astronomical information, map information, and traffic information. This allows the input image to be made closer to the image under the conditions learned by the recognition system. As a result, the accuracy of image recognition is improved.

[0065] (7) The image processing system 100 further includes a measurement unit 37 as a measuring device for measuring the position or orientation of the imaging device that acquires the second image data (Figure 2). The learning data acquisition unit 311 estimates (calculates) the positional relationship between the imaging device and a specific celestial body based on the position or orientation of the vehicle measured by the measurement unit 37 and astronomical information, and generates conditional information based on the calculation result. This makes it possible to estimate how light from a specific celestial body that serves as a light source, such as the sun or the moon, enters the vehicle, and to generate images with greater precision.

[0066] (8) The memory device SD further stores map data. The image generator IG generates a third image data using the map data. This makes the input image closer to the image conditions learned by the recognition system. As a result, the image recognition accuracy is improved.

[0067] (9) The map data includes either 3D map data or data indicating the surface material or surface texture of the structure. This allows the input image to be brought closer to the image of the conditions the recognizer has learned. As a result, the accuracy of image recognition is improved.

[0068] (10) The vehicle, as an autonomous mobile unit, is equipped with an image processing device for processing images, an actuator AC as a movement actuator, and a driving control unit 315 as a movement control unit. The image processing system 100 includes a storage device SD (storage unit 32) that stores first image data obtained by imaging and condition information indicating the imaging conditions of the first image data as learning data, a recognition unit RC (image recognition unit 314 and storage unit 32) that has a recognition model constructed by machine learning using the learning data, and an image generator IG (image generation unit 313) that applies predetermined image processing to second image data obtained by imaging to generate third image data to be output to the recognition unit RC (Figures 1 and 2). The recognition model is a learning model that recognizes objects in an image indicated by the image data based on the input image data, and the image generator IG applies predetermined image processing to the second image data to generate third image data corresponding to the imaging conditions of the first image data. The driving control unit 315 controls the actuator AC based on the recognition result of the recognition model using the third image data as input. In this way, by mounting an image processing device trained on limited training data onto an autonomous mobile vehicle, autonomous driving of the vehicle can be achieved with a simple configuration.

[0069] The above embodiment can be modified into various forms. Modifications will be described below. In the above embodiment, an image processing system 100 in which a learning processing device 1 and an image generation device 2 process two-dimensional camera images was described as an example. However, the learning processing device 1 and the image generation device 2 may process three-dimensional image data, more specifically, images (three-dimensional point cloud data) acquired by a lidar 35 or radar 36.

[0070] Furthermore, in the above embodiment, the image generator IG (image generation unit 313) applies inverse rendering processing to the current image to generate an image corresponding to the imaging conditions of the training image. However, the image generator, as an object recognition unit, may recognize objects in the current image before applying inverse rendering processing to the current image. The image generator may then apply inverse rendering processing to the region of the current image corresponding to the object. Moreover, the image generator may apply inverse rendering processing to the region of the current image corresponding to a specific object (such as a vehicle). By applying inverse rendering processing only to the region of a specific object in this way, the processing load can be reduced without decreasing the recognition accuracy for that specific object.

[0071] Furthermore, the above embodiment uses the example of applying the image processing system 100 to a vehicle as an autonomous mobile device. However, the autonomous mobile device may be something other than a vehicle, and the image processing system may be applied to an automatic lawnmower, a mobile robot, a drone, or the like.

[0072] The image processing apparatus of the above embodiment can also be configured as an image processing method for processing two-dimensional or three-dimensional images, from another perspective. That is, it can include a first step of storing first image data obtained by imaging and condition information indicating the imaging conditions of the first image data as training data in a storage device, and a second step of applying predetermined image processing to second image data obtained by imaging to generate third image data to be output to a recognition device, wherein the recognition device has a recognition model constructed by machine learning using the training data stored in the storage device, and in the second step, it can also be configured as an image processing method that applies predetermined image processing to second image data to generate third image data corresponding to the imaging conditions of the first image data.

[0073] Furthermore, the present invention can be constructed by replacing the above image processing method with a program that causes a computer to perform processing on two-dimensional or three-dimensional images. Even further, the present invention can be constructed by replacing this with a computer-readable storage medium on which this program is recorded.

[0074] The above description is merely an example, and the present invention is not limited by the embodiments and modifications described above, as long as the features of the present invention are not impaired. It is also possible to arbitrarily combine one or more of the above embodiments and modifications, and to combine modifications with each other. [Explanation of Symbols]

[0075] 1 Learning processing unit, 2 Image generation unit, 3 Driving control unit, 30 Controller, 31 Calculation unit, 32 Storage unit, 33 Communication unit, 34 Camera, 35 Lidar, 36 Radar, 37 Measurement unit, 311 Learning data acquisition unit, 312 Learning unit, 313 Image generation unit, 314 Image recognition unit, 315 Driving control unit, AC Actuator, CA Imaging device, IG Image generator, SD Storage device, RC Recognition device

Claims

1. An image processing apparatus for processing two-dimensional or three-dimensional images, A storage device that stores first image data obtained by imaging and condition information indicating the imaging conditions of the first image data as training data, A recognizer having a recognition model constructed by machine learning using the aforementioned training data, The system includes an image generator that applies predetermined image processing to a second image data obtained by imaging to generate a third image data to be output to the recognition device, The image generator is an image processing device characterized by applying the predetermined image processing to the second image data to generate the third image data corresponding to the imaging conditions of the first image data.

2. In the image processing apparatus according to claim 1, The image generator is characterized by applying rendering processing corresponding to the imaging conditions of the first image data to the second image data to generate the third image data as a computer graphics (CG) image.

3. In the image processing apparatus according to claim 1, The image generator is an image processing device characterized by inputting the second image data and text indicating the imaging conditions of the first image data to an image generation AI (Artificial Intelligence) capable of generating image data corresponding to input imaging conditions, and acquiring the image data output from the image generation AI as the third image data.

4. In the image processing apparatus according to claim 2 or 3, The system further includes an object recognition unit that recognizes objects in the image shown by the second image data, The aforementioned image generator is An image processing apparatus characterized by generating a third image data in which the region of the object in the image corresponds to the predetermined imaging conditions, based on the imaging conditions of the first image data.

5. In the image processing apparatus according to claim 2 or 3, The image generator further recognizes the imaging conditions of the second image data, The image generator is characterized by generating the third image data based on the imaging conditions of the first image data and the imaging conditions of the second image data.

6. In the image processing apparatus according to claim 1, Communications Department and, The system further comprises an information acquisition unit that acquires information from the outside via the aforementioned communication unit, The image processing apparatus is characterized in that the information acquisition unit collects the information related to the imaging conditions of the first image data, generates the condition information based on the collected information, and stores the condition information in the storage device.

7. In the image processing apparatus according to claim 6, The image processing apparatus is characterized in that the information relating to the imaging conditions of the first image data includes one of the following: weather information, astronomical information, map information, and traffic information.

8. In the image processing apparatus according to claim 7, The device further comprises a measuring device for measuring the position or orientation of the imaging device that acquires the second image data, The image processing apparatus is characterized in that the information acquisition unit calculates the positional relationship between the imaging device and a specific celestial body based on the position or orientation measured by the measuring device and the astronomical information, and generates the conditional information based on the calculation result.

9. In the image processing apparatus according to claim 1, The aforementioned storage device further stores map data, The image generator is an image processing device characterized by generating the third image data using the map data.

10. In the image processing apparatus according to claim 9, The image processing apparatus is characterized in that the map data includes either three-dimensional map data or data indicating the surface material or surface texture of a structure.

11. An autonomous mobile vehicle equipped with an image processing device that processes two-dimensional or three-dimensional images, It comprises a moving actuator and a moving control unit, The aforementioned image processing device is A storage device that stores first image data obtained by imaging and condition information indicating the imaging conditions of the first image data as training data, A recognizer having a recognition model constructed by machine learning using the aforementioned training data, The system includes an image generator that applies predetermined image processing to a second image data obtained by imaging to generate a third image data to be output to the recognition device, The aforementioned recognition model is a learning model that recognizes objects in an image represented by the input image data, based on the input image data. The image generator applies the predetermined image processing to the second image data to generate the third image data corresponding to the imaging conditions of the first image data. The autonomous mobile body is characterized in that the movement control unit controls the movement actuator based on the recognition result of the recognition model which takes the third image data as input.

Images