Information processing device, information processing method, information processing system, and program

Abstract

The present disclosure relates to an information processing device, an information processing method, an information processing system, and a program which make it possible to suitably evaluate an object recognition filter in a simpler process. A generation unit, which generates teaching data for a preprocessing filter provided in a front stage of the object recognition filter, is generated by a cyclic generative adversarial network (Cyclic GAN) for unsupervised learning. The teaching data generated by the generated generation unit is put into the object recognition filter, and thus, an evaluation image is generated from a difference between object recognition result images and an evaluation filter, which generates the evaluation image, is generated from the evaluation image and the teaching data. The evaluation filter is put with an input image and thus the evaluation image is generated, and the object recognition filter is evaluated by the generated evaluation image. The present disclosure can be applied to an object recognition device.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, an information processing system, and a program, and in particular to an information processing device, an information processing method, an information processing system, and a program that enable a generation unit or a conversion unit obtained by learning to be appropriately evaluated with simpler processing. [Background technology]

[0002] In recent years, there has been remarkable development in machine learning, typified by deep learning, as a component technology of so-called AI (Artificial Intelligence), and it is now widely applied in fields such as image classification, object detection, and segmentation, as well as natural language processing and speech recognition.

[0003] One of the technologies that has recently attracted attention is Generative Adversarial Networks (GAN). GAN is a type of generative model that can generate non-existent data or convert data based on the characteristics of existing data by learning features from data.

[0004] GANs are attracting attention as a method of "unsupervised learning" that learns features without providing correct data.

[0005] More specifically, GAN is a neural network that learns by having two neural networks compete against each other in an adversarial manner: one consisting of a generation unit that generates realistic data or a conversion unit that converts it into realistic data, and the other consisting of a judgment unit that determines whether the generated or converted realistic data is realistic.

[0006] However, GANs are known to be unstable in their learning process and generate biased data that is difficult to predict, so achieving perfect learning requires repeated trial and error, manual adjustment of parameters, and reconfiguration of the network.

[0007] Therefore, a technology has been proposed in which an unsupervised judgment unit calculates latent feature representations of objects in an image from a realistic image generated by a generation unit or a realistic image converted by a conversion unit, and a supervised judgment unit receives the latent feature representations and attribute labels and calculates the probability that the attribute labels are real, i.e., the likelihood of the object, thereby mechanically evaluating the trained generation unit or conversion unit (see Patent Document 1). [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Japanese Patent Application Publication No. 2019-091440 Summary of the Invention [Problem to be solved by the invention]

[0009] However, the technology of Patent Document 1 requires two judgment units, an unsupervised judgment unit and a supervised judgment unit, as well as manual attribute labels, which makes the configuration and processing for mechanically evaluating the learned generation unit or conversion unit complicated.

[0010] The present disclosure has been made in consideration of such circumstances, and in particular, aims to enable a generation unit or a conversion unit obtained by learning to be appropriately evaluated through simpler processing. [Means for solving the problem]

[0011] An information processing device, an information processing system, and a program according to one aspect of the present disclosure are an information processing device, an information processing system, and a program that include a recognition unit that recognizes an object in an input image and an evaluation unit that evaluates the recognition unit.

[0012] An information processing method according to one aspect of the present disclosure is an information processing method for an information processing device that includes a recognition unit that recognizes an object in an input image and an evaluation unit that evaluates the recognition unit, wherein the evaluation unit includes a step of evaluating the recognition unit.

[0013] In one aspect of the present disclosure, an object in an input image is recognized and the recognition unit is evaluated. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram illustrating an example of the configuration of a vehicle control system. [Figure 2] FIG. 2 is a diagram illustrating an example of a sensing region. [Figure 3] FIG. 10 is a diagram illustrating a configuration example of a recognition unit when the present technology is not applied. [Figure 4] 10 is a diagram illustrating a configuration example of a recognition unit when the present technology is applied. FIG. [Figure 5] 5 is a diagram illustrating an example of the configuration of an evaluation filter generation unit in FIG. 4. FIG. [Figure 6] FIG. 10 is a diagram illustrating learning of an object recognition filter. [Figure 7] FIG. 10 is a diagram illustrating learning of a preprocessing filter. [Figure 8] FIG. 1 is a diagram illustrating a cyclic generative adversarial network. [Figure 9] FIG. 10 is a diagram illustrating generation of a learning set. [Figure 10] FIG. 10 is a diagram illustrating generation of an evaluation image. [Figure 11] FIG. 10 is a diagram illustrating generation of an evaluation filter. [Figure 12] FIG. 10 is a diagram illustrating operation control using an evaluation filter. [Figure 13]10 is a flowchart illustrating an evaluation filter generation process. [Figure 14] 10 is a flowchart illustrating an operation control process. [Figure 15] FIG. 10 is a diagram illustrating a first modified example. [Figure 16] FIG. 10 is a diagram illustrating a second modified example. [Figure 17] FIG. 10 is a diagram illustrating a third modified example. [Figure 18] FIG. 10 is a diagram illustrating a fourth modified example. [Figure 19] FIG. 10 is a diagram illustrating a fifth modified example. [Figure 20] FIG. 1 is a diagram illustrating an example of the configuration of a general-purpose computer. DETAILED DESCRIPTION OF THE INVENTION

[0015] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0016] Hereinafter, embodiments of the present technology will be described in the following order. 1. Example of vehicle control system configuration 2. Example of the configuration of a recognition unit to which the technology of the present disclosure is not applied 3. Example of the configuration of a recognition unit to which the technology disclosed herein is applied 4. Example of evaluation filter generation unit configuration 5. First Modification 6. Second Variant 7. Third Variant 8. Fourth Variant 9. Fifth Variant 10. Software implementation example

[0017] <<1. Example of vehicle control system configuration>> This technology enables the generation unit (conversion unit) obtained through learning to be evaluated using simple processing.

[0018] In this specification, an example in which the present technology is applied to a mobile device control system will be described.

[0019] FIG. 1 is a block diagram showing an example of the configuration of a vehicle control system 11, which is an example of a mobility device control system to which the present technology is applied.

[0020] The vehicle control system 11 is provided in the vehicle 1 and performs processing related to driving assistance and automatic driving of the vehicle 1.

[0021] The vehicle control system 11 includes a processor 21, a communication unit 22, a map information storage unit 23, a GNSS (Global Navigation Satellite System) receiving unit 24, an external recognition sensor 25, an in-vehicle sensor 26, a vehicle sensor 27, a recording unit 28, a driving assistance / autonomous driving control unit 29, a DMS (Driver Monitoring System) 30, an HMI (Human Machine Interface) 31, and a vehicle control unit 32.

[0022] The processor 21, communication unit 22, map information storage unit 23, GNSS receiving unit 24, external recognition sensor 25, in-vehicle sensor 26, vehicle sensor 27, recording unit 28, cruise assist / autonomous driving control unit 29, driver monitoring system (DMS) 30, human-machine interface (HMI) 31, and vehicle control unit 32 are interconnected via a communication network 41. The communication network 41 is configured by an in-vehicle communication network or bus conforming to any standard such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), FlexRay (registered trademark), or Ethernet. Note that the components of the vehicle control system 11 may be directly connected to each other via, for example, near field communication (NFC) or Bluetooth (registered trademark) without using the communication network 41.

[0023] In the following description, when each unit of the vehicle control system 11 communicates via the communication network 41, the description of the communication network 41 will be omitted. For example, when the processor 21 and the communication unit 22 communicate via the communication network 41, it will simply be described that the processor 21 and the communication unit 22 communicate with each other.

[0024] The processor 21 is configured by various processors such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ECU (Electronic Control Unit), etc. The processor 21 controls the vehicle control system 11 as a whole.

[0025] The communication unit 22 communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, etc., and transmits and receives various types of data. As communication with the outside of the vehicle, for example, the communication unit 22 receives from the outside a program for updating software that controls the operation of the vehicle control system 11, map information, traffic information, information about the surroundings of the vehicle 1, etc. For example, the communication unit 22 transmits information about the vehicle 1 (for example, data indicating the state of the vehicle 1, the recognition result by the recognition unit 73, etc.), information about the surroundings of the vehicle 1, etc., to the outside. For example, the communication unit 22 performs communication compatible with a vehicle emergency notification system such as e-call.

[0026] There is no particular limitation on the communication method of the communication unit 22. A plurality of communication methods may be used.

[0027] For example, the communication unit 22 performs wireless communication with devices inside the vehicle using a communication method such as wireless LAN, Bluetooth, NFC, or WUSB (Wireless USB). For example, the communication unit 22 performs wired communication with devices inside the vehicle using a communication method such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface, registered trademark), or MHL (Mobile High-Definition Link) via a connection terminal (and a cable, if necessary) not shown.

[0028] Here, the in-vehicle device refers to, for example, a device that is in the vehicle and is not connected to the communication network 41. Examples of such devices include mobile devices and wearable devices carried by passengers such as the driver, and information devices that are brought into the vehicle and temporarily installed.

[0029] For example, the communication unit 22 communicates with a server or the like existing on an external network (e.g., the Internet, a cloud network, or an operator-specific network) via a base station or an access point using a wireless communication method such as 4G (fourth generation mobile communication system), 5G (fifth generation mobile communication system), LTE (Long Term Evolution), or DSRC (Dedicated Short Range Communications).

[0030] For example, the communication unit 22 communicates with a terminal present in the vicinity of the vehicle (for example, a terminal of a pedestrian or a store, or an MTC (Machine Type Communication) terminal) using P2P (Peer To Peer) technology. For example, the communication unit 22 performs V2X communication. V2X communication includes, for example, vehicle-to-vehicle communication with another vehicle, vehicle-to-infrastructure communication with a roadside unit or the like, vehicle-to-home communication, and vehicle-to-pedestrian communication with a terminal carried by a pedestrian or the like.

[0031] For example, the communication unit 22 receives electromagnetic waves transmitted by a road traffic information communication system (Ethernet (Vehicle Information and Communication System) registered trademark) such as a radio beacon, an optical beacon, or FM multiplex broadcasting.

[0032] The map information storage unit 23 stores maps acquired from an external source and maps created by the vehicle 1. For example, the map information storage unit 23 stores a three-dimensional high-precision map, a global map that is less accurate than a high-precision map and covers a wide area, and the like.

[0033] Examples of high-precision maps include dynamic maps, point cloud maps, and vector maps (also referred to as ADAS (Advanced Driver Assistance System) maps). Dynamic maps are maps consisting of four layers of information, for example, dynamic information, quasi-dynamic information, quasi-static information, and static information, and are provided from an external server or the like. Point cloud maps are maps configured from point clouds (point cloud data). Vector maps are maps in which information such as the positions of lanes and traffic lights is associated with the point cloud map. The point cloud map and the vector map may be provided from an external server or the like, or may be created by the vehicle 1 based on sensing results from the radar 52, the LiDAR 53, and the like as a map for matching with a local map (described later), and stored in the map information storage unit 23. Furthermore, when a high-precision map is provided from an external server or the like, map data of, for example, an area of ​​several hundred square meters related to the planned route along which the vehicle 1 will travel is acquired from the server or the like in order to reduce communication capacity.

[0034] The GNSS receiver 24 receives GNSS signals from GNSS satellites and supplies them to the driving assistance / automated driving control unit 29 .

[0035] The external recognition sensor 25 includes various sensors used to recognize the situation outside the vehicle 1, and supplies sensor data from each sensor to each part of the vehicle control system 11. The type and number of sensors included in the external recognition sensor 25 are arbitrary.

[0036] For example, the external recognition sensor 25 includes a camera 51, a radar 52, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 53, and an ultrasonic sensor 54. The number of cameras 51, radars 52, LiDARs 53, and ultrasonic sensors 54 is arbitrary, and an example of the sensing area of ​​each sensor will be described later.

[0037] As the camera 51, a camera of any imaging method such as a ToF (Time Of Flight) camera, a stereo camera, a monocular camera, or an infrared camera may be used as needed.

[0038] Furthermore, for example, the external recognition sensor 25 includes an environmental sensor for detecting the weather, climate, brightness, etc. The environmental sensor includes, for example, a raindrop sensor, a fog sensor, a sunlight sensor, a snow sensor, an illuminance sensor, etc.

[0039] Furthermore, for example, the external recognition sensor 25 includes a microphone used to detect sounds around the vehicle 1 and the positions of sound sources.

[0040] The interior sensor 26 includes various sensors for detecting information inside the vehicle, and supplies sensor data from each sensor to each unit of the vehicle control system 11. The interior sensor 26 may include any type and any number of sensors.

[0041] For example, the in-vehicle sensors 26 include a camera, radar, a seating sensor, a steering wheel sensor, a microphone, a biometric sensor, etc. The camera may be a camera of any imaging method, such as a ToF camera, a stereo camera, a monocular camera, or an infrared camera. The biometric sensor is provided, for example, on a seat or steering wheel, and detects various types of biometric information of a passenger such as a driver.

[0042] The vehicle sensor 27 includes various sensors for detecting the state of the vehicle 1, and supplies sensor data from each sensor to each unit of the vehicle control system 11. The type and number of sensors included in the vehicle sensor 27 are arbitrary.

[0043] For example, the vehicle sensor 27 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU). For example, the vehicle sensor 27 includes a steering angle sensor that detects the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor that detects the amount of accelerator pedal operation, and a brake sensor that detects the amount of brake pedal operation. For example, the vehicle sensor 27 includes a rotation sensor that detects the number of rotations of the engine or motor, an air pressure sensor that detects tire air pressure, a slip ratio sensor that detects tire slip ratio, and a wheel speed sensor that detects the rotation speed of the wheels. For example, the vehicle sensor 27 includes a battery sensor that detects the remaining battery level and temperature, and an impact sensor that detects external impacts.

[0044] The recording unit 28 includes, for example, a magnetic storage device such as a read-only memory (ROM), a random access memory (RAM), or a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The recording unit 28 records various programs and data used by each component of the vehicle control system 11. For example, the recording unit 28 records a rosbag file including messages transmitted and received by a robot operating system (ROS) on which an application program related to autonomous driving runs. For example, the recording unit 28 includes an event data recorder (EDR) or a data storage system for automated driving (DSSAD), and records information about the vehicle 1 before and after an event such as an accident.

[0045] The driving assistance / automatic driving control unit 29 controls driving assistance and automatic driving of the vehicle 1. For example, the driving assistance / automatic driving control unit 29 includes an analysis unit 61, an action planning unit 62, and an operation control unit 63.

[0046] The analysis unit 61 performs an analysis process of the vehicle 1 and the surrounding situation. The analysis unit 61 includes a self-position estimation unit 71, a sensor fusion unit 72, and a recognition unit 73.

[0047] The self-position estimation unit 71 estimates the self-position of the vehicle 1 based on the sensor data from the external recognition sensor 25 and the high-precision map stored in the map information storage unit 23. For example, the self-position estimation unit 71 generates a local map based on the sensor data from the external recognition sensor 25 and estimates the self-position of the vehicle 1 by matching the local map with the high-precision map. The position of the vehicle 1 is based on, for example, the center of the rear wheel pair axle.

[0048] The local map is, for example, a three-dimensional high-precision map or an occupancy grid map created using a technology such as SLAM (Simultaneous Localization and Mapping). The three-dimensional high-precision map is, for example, the point cloud map described above. The occupancy grid map is a map in which the three-dimensional or two-dimensional space around the vehicle 1 is divided into grids of a predetermined size and the occupancy state of an object is indicated on a grid-by-grid basis. The occupancy state of an object is indicated, for example, by the presence or absence of an object and its probability of existence. The local map is also used, for example, in the detection process and recognition process of the situation outside the vehicle 1 by the recognition unit 73.

[0049] The self-position estimation unit 71 may estimate the self-position of the vehicle 1 based on the GNSS signal and sensor data from the vehicle sensor 27.

[0050] The sensor fusion unit 72 performs sensor fusion processing to obtain new information by combining multiple different types of sensor data (for example, image data supplied from the camera 51 and sensor data supplied from the radar 52). Methods for combining different types of sensor data include integration, fusion, and association.

[0051] The recognition unit 73 performs detection processing and recognition processing of the situation outside the vehicle 1.

[0052] For example, the recognition unit 73 performs detection processing and recognition processing of the situation outside the vehicle 1 based on information from the external recognition sensor 25, information from the self-position estimation unit 71, information from the sensor fusion unit 72, etc.

[0053] Specifically, for example, the recognition unit 73 performs detection processing and recognition processing of objects around the vehicle 1. The object detection processing is, for example, processing to detect the presence or absence, size, shape, position, movement, etc. of an object. The object recognition processing is, for example, processing to recognize attributes such as the type of object, or to identify a specific object. However, the detection processing and the recognition processing are not necessarily clearly separated, and may overlap.

[0054] For example, the recognition unit 73 performs clustering to classify a point cloud based on sensor data such as LiDAR or radar into clusters of points, thereby detecting objects around the vehicle 1. This allows the presence, size, shape, and position of objects around the vehicle 1 to be detected.

[0055] For example, the recognition unit 73 performs tracking to follow the movement of clusters of point clouds classified by clustering, thereby detecting the movement of objects around the vehicle 1. As a result, the speed and traveling direction (movement vector) of the objects around the vehicle 1 are detected.

[0056] For example, the recognition unit 73 performs object recognition processing such as semantic segmentation on the image data supplied from the camera 51 to recognize the type of object around the vehicle 1.

[0057] Note that objects to be detected or recognized may include, for example, vehicles, people, bicycles, obstacles, structures, roads, traffic lights, traffic signs, road markings, and the like.

[0058] For example, the recognition unit 73 performs a recognition process of traffic rules around the vehicle 1 based on the map stored in the map information storage unit 23, the estimation result of the vehicle's own position, and the recognition result of objects around the vehicle 1. Through this process, for example, the positions and states of traffic signals, the contents of traffic signs and road markings, the contents of traffic regulations, and lanes on which travel is permitted are recognized.

[0059] For example, the recognition unit 73 performs a process of recognizing the environment around the vehicle 1. The environment to be recognized may include, for example, the weather, temperature, humidity, brightness, and road surface conditions.

[0060] The behavior planning unit 62 creates a behavior plan for the vehicle 1. For example, the behavior planning unit 62 creates the behavior plan by performing route planning and route following processing.

[0061] Global path planning is a process for planning a rough route from the start to the goal. This route planning also includes a process called trajectory planning, which takes into account the motion characteristics of the vehicle 1 on the route planned by the route planning and generates a trajectory (local path planning) that allows the vehicle 1 to proceed safely and smoothly in the vicinity of the vehicle 1.

[0062] Path following is a process of planning an operation for safely and accurately traveling along a route planned by a route planner within a planned time. For example, a target speed and a target angular velocity of the vehicle 1 are calculated.

[0063] The operation control unit 63 controls the operation of the vehicle 1 in order to realize the action plan created by the action planning unit 62.

[0064] For example, the operation control unit 63 controls the steering control unit 81, the brake control unit 82, and the drive control unit 83 to perform acceleration / deceleration control and direction control so that the vehicle 1 travels along the trajectory calculated by the trajectory plan. For example, the operation control unit 63 performs cooperative control aimed at realizing ADAS functions such as collision avoidance or impact mitigation, following driving, vehicle speed maintenance driving, collision warning for the vehicle itself, and lane departure warning for the vehicle itself. For example, the operation control unit 63 performs cooperative control aimed at automatic driving, which allows the vehicle to travel autonomously without relying on driver operation.

[0065] The DMS 30 performs processes such as authenticating the driver and recognizing the driver's state based on the sensor data from the in-vehicle sensors 26 and the input data input to the HMI 31. Examples of the driver's state to be recognized include physical condition, alertness, concentration, fatigue, gaze direction, level of intoxication, driving operation, and posture.

[0066] The DMS 30 may be configured to perform authentication processing for passengers other than the driver and recognition processing for the conditions of the passengers. Furthermore, for example, the DMS 30 may be configured to perform recognition processing for the conditions inside the vehicle based on sensor data from the in-vehicle sensor 26. Possible conditions inside the vehicle to be recognized include, for example, temperature, humidity, brightness, and odor.

[0067] The HMI 31 is used to input various data, instructions, etc., generates input signals based on the input data, instructions, etc., and supplies them to each component of the vehicle control system 11. For example, the HMI 31 includes operation devices such as a touch panel, buttons, a microphone, switches, and levers, as well as operation devices that allow input by voice, gestures, etc., other than manual operation. The HMI 31 may be, for example, a remote control device that uses infrared or other radio waves, or an externally connected device such as a mobile device or wearable device that supports operation of the vehicle control system 11.

[0068] The HMI 31 also performs output control to generate and output visual information, auditory information, and tactile information for the occupant or the outside of the vehicle, as well as to control the output content, output timing, output method, etc. Visual information is information displayed by images or light, such as an operation screen, a status display of the vehicle 1, a warning display, and a monitor image showing the situation around the vehicle 1. Auditory information is information displayed by sound, such as guidance, warning sounds, and warning messages. Tactile information is information imparted to the occupant's sense of touch by force, vibration, movement, etc.

[0069] Possible devices for outputting visual information include, for example, a display device, a projector, a navigation device, an instrument panel, a CMS (Camera Monitoring System), an electronic mirror, a lamp, etc. The display device may be a device having a normal display, or may be a device that displays visual information within the field of view of a passenger, such as a head-up display, a see-through display, or a wearable device with an AR (Augmented Reality) function.

[0070] Possible devices for outputting auditory information include, for example, audio speakers, headphones, and earphones.

[0071] A device that outputs tactile information may be, for example, a haptic element that uses haptic technology, which may be provided on a steering wheel, a seat, or the like.

[0072] The vehicle control unit 32 controls each part of the vehicle 1. The vehicle control unit 32 includes a steering control unit 81, a brake control unit 82, a drive control unit 83, a body system control unit 84, a light control unit 85, and a horn control unit 86.

[0073] The steering control unit 81 detects and controls the state of the steering system of the vehicle 1. The steering system includes, for example, a steering mechanism including a steering wheel, an electric power steering, etc. The steering control unit 81 includes, for example, a control unit such as an ECU that controls the steering system, an actuator that drives the steering system, etc.

[0074] The brake control unit 82 detects and controls the state of the brake system of the vehicle 1. The brake system includes, for example, a brake mechanism including a brake pedal, an ABS (Antilock Brake System), etc. The brake control unit 82 includes, for example, a control unit such as an ECU that controls the brake system, and an actuator that drives the brake system.

[0075] The drive control unit 83 detects and controls the state of the drive system of the vehicle 1. The drive system includes, for example, an accelerator pedal, a drive force generating device for generating drive force such as an internal combustion engine or a drive motor, and a drive force transmission mechanism for transmitting the drive force to the wheels. The drive control unit 83 includes, for example, a control unit such as an ECU for controlling the drive system, and an actuator for driving the drive system.

[0076] The body system control unit 84 detects and controls the states of the body system systems of the vehicle 1. The body system systems include, for example, a keyless entry system, a smart key system, a power window device, a power seat, an air conditioning system, an airbag, a seat belt, a shift lever, etc. The body system control unit 84 includes, for example, a control unit such as an ECU that controls the body system systems, and an actuator that drives the body system systems.

[0077] The light control unit 85 detects and controls the states of various lights of the vehicle 1. Examples of lights to be controlled include headlights, backlights, fog lights, turn signals, brake lights, projections, and bumper displays. The light control unit 85 includes a control unit such as an ECU that controls the lights, an actuator that drives the lights, and the like.

[0078] The horn control unit 86 detects and controls the state of the car horn of the vehicle 1. The horn control unit 86 includes, for example, a control unit such as an ECU that controls the car horn, and an actuator that drives the car horn.

[0079] FIG. 2 is a diagram showing an example of a sensing area by the camera 51, the radar 52, the LiDAR 53, and the ultrasonic sensor 54 of the external recognition sensor 25 in FIG.

[0080] Sensing area 101F and sensing area 101B are examples of sensing areas of the ultrasonic sensor 54. Sensing area 101F covers the periphery of the front end of the vehicle 1. Sensing area 101B covers the periphery of the rear end of the vehicle 1.

[0081] The sensing results in the sensing area 101F and the sensing area 101B are used for, for example, parking assistance for the vehicle 1.

[0082] Sensing area 102F to sensing area 102B show examples of sensing areas of a short-range or medium-range radar 52. Sensing area 102F covers a position farther in front of the vehicle 1 than sensing area 101F. Sensing area 102B covers a position farther behind the vehicle 1 than sensing area 101B. Sensing area 102L covers the surrounding area behind the left side of the vehicle 1. Sensing area 102R covers the surrounding area behind the right side of the vehicle 1.

[0083] The sensing results in sensing area 102F are used, for example, to detect vehicles, pedestrians, etc. that are present in front of the vehicle 1. The sensing results in sensing area 102B are used, for example, for a collision prevention function behind the vehicle 1. The sensing results in sensing area 102L and sensing area 102R are used, for example, to detect objects in blind spots on the sides of the vehicle 1.

[0084] Sensing area 103F to sensing area 103B show examples of sensing areas sensed by camera 51. Sensing area 103F covers a position farther in front of vehicle 1 than sensing area 102F. Sensing area 103B covers a position farther behind vehicle 1 than sensing area 102B. Sensing area 103L covers the periphery of the left side of vehicle 1. Sensing area 103R covers the periphery of the right side of vehicle 1.

[0085] The sensing results in the sensing area 103F are used, for example, for recognizing traffic lights and traffic signs, lane departure prevention support systems, etc. The sensing results in the sensing area 103B are used, for example, for parking assistance and surround view systems, etc. The sensing results in the sensing areas 103L and 103R are used, for example, for surround view systems, etc.

[0086] Sensing area 104 shows an example of the sensing area of ​​LiDAR 53. Sensing area 104 covers a position farther ahead of vehicle 1 than sensing area 103F. On the other hand, sensing area 104 has a narrower range in the left-right direction than sensing area 103F.

[0087] The sensing results in the sensing area 104 are used for, for example, emergency braking, collision avoidance, pedestrian detection, and the like.

[0088] Sensing area 105 shows an example of the sensing area of ​​the long-distance radar 52. Sensing area 105 covers a position further ahead of the vehicle 1 than sensing area 104. On the other hand, sensing area 105 has a narrower range in the left-right direction than sensing area 104.

[0089] The sensing results in the sensing area 105 are used for, for example, adaptive cruise control (ACC).

[0090] The sensing area of ​​each sensor may have various configurations other than that shown in Figure 2. Specifically, the ultrasonic sensor 54 may also sense the sides of the vehicle 1, and the LiDAR 53 may sense the area behind the vehicle 1.

[0091] <<2. Configuration example of a recognition unit to which the technology of the present disclosure is not applied>> Next, before describing the configuration of the recognition unit 73 to which the technology of the present disclosure is applied, an example configuration of the recognition unit 73 to which the technology of the present disclosure is not applied will be described with reference to FIG.

[0092] In this embodiment, the description will be given taking as an example a configuration in which object recognition processing is performed on an image captured by the camera 51 in the recognition unit 73.

[0093] The recognition unit 73 to which the technology of the present disclosure is not applied includes a preprocessing filter 111 and an object recognition filter 112 .

[0094] The preprocessing filter 111 processes the input image PI, converts it into a preprocessed image PC, and outputs it to the object recognition filter 112 .

[0095] More specifically, the preprocessing filter 111 applies high-quality filter processing to low-quality input images PI that reduce the object recognition accuracy of the object recognition filter 112, such as low-illumination images, images containing noise, cloudy or hazy images, images with blur caused by movement, out-of-focus images, images with saturated brightness, or images with poor lighting conditions in inclement weather such as heavy rain or snow, by adjusting the illumination, removing noise, removing cloudiness or haze, removing blur, adjusting focus, adjusting brightness, and adjusting images in inclement weather to images closer to those in clear weather, and corrects the input images PI to convert them into preprocessed images PC and output them to the object recognition filter 112.

[0096] The pre-processing filter 111 is a filter generated by so-called supervised learning using low-quality images that reduce the accuracy of the filter processing in the object recognition filter 112 and corresponding high-quality images, and is configured, for example, by a neural network.

[0097] The object recognition filter 112 performs object recognition processing such as semantic segmentation on the preprocessed image PC that has been preprocessed by the preprocessing filter 111, recognizes objects on a pixel-by-pixel basis, and outputs an image consisting of the recognition results as an object recognition result image PL.

[0098] The object recognition filter 112 is a filter generated by so-called supervised learning using an input image and information in which the object recognition result is labeled for each corresponding pixel, and is configured by, for example, a neural network.

[0099] The operation control unit 63 recognizes the object of the subject in the image based on the object recognition result in pixel units in this object recognition result image PL, and controls the operation of the vehicle 1 based on the recognition result.

[0100] However, if the pre-processing filter 111 is not sufficiently trained, instability and bias may occur, and there is a risk that appropriate pre-processing may not be performed.

[0101] In other words, if the preprocessing filter 111 is unable to learn sufficiently, for example, if appropriate preprocessing cannot be performed on the input image PI, the object recognition filter 112 may not perform appropriate object recognition, which may result in inappropriate recognition results for each pixel in the object recognition result image PL.

[0102] That is, for example, if it is recognized that there are no objects in the object recognition result image PL that may cause a collision, but in reality, preprocessing by the preprocessing filter 111 is insufficient and an object that may cause a collision is not recognized even though it exists, it may be impossible to take action to avoid the collision.

[0103] As a result, the operation control unit 63 may not be able to properly control the operation of the vehicle 1 based on the object recognition result image PL consisting of this inappropriate recognition result, and in the worst case, this may cause an accident.

[0104] <<3. Configuration example of a recognition unit to which the technology of the present disclosure is applied>> Therefore, in the present disclosure, an evaluation image for evaluating the preprocessing filter 111 corresponding to the object recognition result image can be generated based on the input image, so that operation control can be realized after understanding the evaluation of the recognition result in the object recognition result image.

[0105] More specifically, the recognition unit 73 to which the technology of the present disclosure is applied includes a preprocessing filter 131, an object recognition filter 132, an evaluation filter 133, and an evaluation filter generation unit 134, as shown in FIG.

[0106] The pre-processing filter 131 and the object recognition filter 132 have configurations corresponding to the pre-processing filter 111 and the object recognition filter 112 in FIG. 3 and have the same functions, so a description thereof will be omitted.

[0107] The evaluation filter 133 processes the input image PI to generate and output an evaluation image PE for evaluating the pre-processing filter 131.

[0108] The evaluation image PE is, for example, an image corresponding to a preprocessed image PC obtained by filtering the input image PI with the preprocessing filter 131, and is an image with pixel values ​​so high that preprocessing is inappropriate on a pixel-by-pixel basis.

[0109] That is, if the processing by the preprocessing filter 131 is appropriate, the evaluation image PE will be an entirely black image, and if the processing by the preprocessing filter 131 is inappropriate, the evaluation image PE will be an image in which the inappropriate parts are bright.

[0110] Therefore, the brighter the evaluation image PE is overall, the less sufficient the preprocessing by the preprocessing filter 131 has been, and the lower the evaluation of the preprocessing; conversely, the darker the image is overall, the more sufficient the preprocessing by the preprocessing filter 131 has been, and the higher the evaluation of the preprocessing.

[0111] Furthermore, the more appropriate the preprocessing by the preprocessing filter 131, the more appropriately corrected the preprocessed image PC will be, and therefore the more accurate the object recognition processing by the object recognition filter 132 will be.

[0112] For this reason, areas with low ratings in the evaluation image PE can be considered to have not been sufficiently pre-processed and the object recognition accuracy by the object recognition filter 132 is low, or to have been sufficiently pre-processed but the object recognition accuracy by the object recognition filter 132 is low. Conversely, areas with high ratings in the evaluation image PE can be considered to have been sufficiently pre-processed by the pre-processing filter 131 and the object recognition accuracy by the object recognition filter 132 is high.

[0113] Therefore, the evaluation image PE is an image for evaluating the degree of preprocessing performed by the preprocessing filter 131, and can also be said to be an image for evaluating the object recognition accuracy of the recognition unit 73 as a whole.

[0114] The operation control unit 63 evaluates the object recognition result image PL on a pixel-by-pixel basis based on the evaluation image PE, and controls the operation of the vehicle 1 in accordance with the evaluation.

[0115] That is, for example, the operation control unit 63 may not use an object recognition result image PL in which the area of ​​high pixel values ​​in the corresponding evaluation image PE is larger than a predetermined area when controlling the operation of the vehicle 1.

[0116] At this time, the operation control unit 63 may, for example, use the pixel values ​​of the evaluation image PE to calculate an evaluation score, and if the evaluation score is lower than a predetermined value, may consider the evaluation to be low and may not use the object recognition result image PL when controlling the operation of the vehicle 1.

[0117] Here, the evaluation score may be, for example, the reciprocal of the sum of the pixel values ​​of all pixels. That is, the brighter the evaluation image PE as a whole and the greater the sum of the pixel values ​​of all pixels, the smaller the evaluation score will be, and the lower the evaluation of the preprocessing and object recognition processing will be. Conversely, the darker the evaluation image PE as a whole and the smaller the sum of the pixel values ​​of all pixels, the larger the evaluation score will be, and the higher the evaluation of the preprocessing and object recognition processing will be.

[0118] Furthermore, since the pre-processing filter 131 is constructed, for example, from a neural network, etc., that has been trained by supervised learning, it may be possible to switch to another pre-processing filter 131 that has been trained based on other teacher data and that has received a high evaluation from the evaluation image PE.

[0119] By performing such processing, it becomes possible to realize appropriate operation control of the vehicle 1 according to the reliability of the pre-processing filter 131.

[0120] <<4. Example of the evaluation filter generation unit>> Next, an example of the configuration of the evaluation filter generation unit 134 will be described with reference to FIG.

[0121] The evaluation filter generation unit 134 includes a filter learning unit 151 , a conversion discrimination unit learning unit 152 , a learning set generation unit 153 , an evaluation image generation unit 154 , and an evaluation filter learning unit 155 .

[0122] <Filter learning section> The filter learning unit 151 generates a pre-processing filter PF and an object recognition filter RF corresponding to the pre-processing filter 131 and the object recognition filter 132 through learning.

[0123] More specifically, the filter training unit 151 includes an object recognition filter training unit 171 and a preprocessing filter training unit 172 .

[0124] The object recognition filter learning unit 171 generates an object recognition filter RF based on supervised learning using multiple pairs of an input image PA and an object recognition result image PL to which a label that identifies an object for the input image PA is attached, as shown in FIG. 6, for example, and outputs the object recognition filter RF to the evaluation image generation unit 154.

[0125] That is, the object recognition filter RF generated by the object recognition filter learning unit 171 is composed of, for example, a neural network, recognizes objects on a pixel-by-pixel basis from the input image, and outputs the recognition result image PL to which, for example, a label such as a semantic label has been assigned.

[0126] The input image PA in FIG. 6 is an image of a daytime scene of a pedestrian crossing at an intersection, and the object recognition result image PL is an image in which labels for each object in the input image PA are added as patterns.

[0127] The labels for each object in the object recognition result image PL in FIG. 6 are, for example, road, crosswalk, pedestrian, sidewalk, sign, building, etc., and are made up of information added by, for example, manual input.

[0128] The object recognition filter RF generated by the object recognition filter learning unit 171 has the same configuration as the object recognition filter 132 in the recognition unit 73 in FIG. 4, but is generated solely for the purpose of generating the evaluation filter 133.

[0129] Therefore, the object recognition filter 132 in the recognition unit 73 in FIG. 4 may be the object recognition filter RF generated by the object recognition filter learning unit 171, or may be generated by another configuration.

[0130] The preprocessing filter learning unit 172 generates a preprocessing filter PF based on supervised learning using multiple pairs of low-quality input images PB and corresponding high-quality output images PC, as shown in Figure 7, and outputs the preprocessing filter PF to the evaluation image generation unit 154.

[0131] That is, the pre-processing filter PF generated by the pre-processing filter learning unit 172 is configured, for example, by a neural network, and outputs a high-quality image by improving the image quality of a low-quality input image.

[0132] The low-quality input image PB in FIG. 7 is a dark image with low illumination, and the output image PC is a bright image with high illumination that corresponds to the input image PB.

[0133] In Figure 7, an example of a dark image is shown as an example of a low-quality image, but other low-quality images can also be used, such as an image containing noise, a cloudy or hazy image, an image with blur caused by movement, an out-of-focus image, an image with saturated brightness, or an image with poor lighting conditions in severe weather such as heavy rain or snow.In this case, the corresponding image from which the noise has been removed, the image from which the cloudiness or haze has been removed, the image from which the blur has been removed, the image from which the focus has been adjusted, the image from which the brightness has been adjusted, and the image from which the weather is close to sunny are used as the output image PC, which corresponds to the preprocessed image.

[0134] The preprocessing filter PF generated by the preprocessing filter learning unit 172 has the same configuration as the preprocessing filter 131 in the recognition unit 73 in FIG.

[0135] Therefore, the preprocessing filter 131 in the recognition unit 73 in FIG. 4 may be a preprocessing filter PF generated by the preprocessing filter learning unit 172, or may be generated by another configuration.

[0136] <Conversion discrimination unit / learning unit> The transformation discrimination unit learning unit 152 generates the transformation unit 191, the transformation discrimination unit 192, the inverse transformation unit 193, and the inverse transformation discrimination unit 194 by mutual learning using a cyclic generative adversarial network (Cyclic GAN), and outputs the transformation unit 191 to the learning set generation unit 153.

[0137] Here, the conversion unit 191 is made up of, for example, a neural network, and converts an input image PA used for training the object recognition filter RF into an image PA' that resembles the input image PB used for generating the pre-processing filter PF.

[0138] The transformation determination unit 192 is, for example, composed of a neural network, and determines the error loss by comparing the image PA' transformed by the transformation unit 191 to resemble the input image PB with the input image PB used to train the pre-processing filter PF, and determines whether the image PA' is the input image PB.

[0139] The inverse conversion unit 193 is made up of, for example, a neural network, and converts the input image PB used for training the pre-processing filter PF into an image PB' that resembles the input image PA used for generating the object recognition filter RF.

[0140] The inverse transformation determination unit 194 is, for example, composed of a neural network, and determines the error loss by comparing the image PB' converted by the inverse transformation unit 193 to resemble the input image PA with the input image PA used to train the object recognition filter RF, and determines whether the image PB' is the input image PA.

[0141] A generative adversarial network (GAN) is an adversarial network that trains a conversion unit (generation unit) that converts data into data that resembles specific data based on random noise, and a judgment unit that determines whether the data that resembles specific data converted (generated) by the conversion unit (generation unit) is real specific data.

[0142] As a result, the conversion unit (generation unit) learns to convert (generate data) into data that will be determined by the judgment unit to be genuine specific data, and the judgment unit learns to be able to determine that the data converted by the conversion unit (generation unit) is not genuine data.

[0143] As a result, the conversion unit (generation unit) is trained to be able to convert (generate) data converted (generated) from random noise into data that looks like genuine specific data with greater accuracy, and the judgment unit is trained to be able to judge with high accuracy that data that has been converted (generated) to look like genuine specific data is not genuine specific data.

[0144] A cyclical generative adversarial network (Cyclic GAN) combines multiple conversion units and judgment units trained by the above-mentioned generative adversarial network (GAN), thereby realizing cyclical unsupervised learning and training multiple conversion units (generation units) and judgment units.

[0145] In the transformation determination unit learning unit 152, the transformation unit 191, the inverse transformation unit 193, the transformation determination unit 192, and the inverse transformation determination unit 194 are trained by a cyclic generative adversarial network (Cyclic GAN).

[0146] That is, as shown in FIG. 8, the conversion unit 191 converts the input image PA used for training the object recognition filter RF into an image PA' that resembles the input image PB used for generating the pre-processing filter PF.

[0147] The transformation determination unit 192 determines the error loss by comparing the image PA' transformed by the transformation unit 191 to resemble the input image PB with the set PBS of input images PB used to train the pre-processing filter PF, and determines whether the image PA' is the input image PB.

[0148] The inverse transformation unit 193 uses the image PA' as the input image PB used for training the pre-processing filter PF and transforms it into an image PB' that resembles the input image PA used for generating the object recognition filter RF. That is, the inverse transformation unit 193 inversely transforms the image PA' generated by the transformation unit 191 from the input image PA into an image PB' that resembles the input image PA.

[0149] The inverse transformation determination unit 194 determines the error loss by comparing the image PB' inversely transformed by the inverse transformation unit 193 to resemble the input image PA with the set PAS of input images PA used to train the object recognition filter RF, and determines whether the image PB' is the input image PA.

[0150] The conversion unit 191, conversion discrimination unit 192, inverse conversion unit 193, and inverse conversion discrimination unit 194 configured in this manner are repeatedly mutually trained using a cyclic generative adversarial network (Cyclic GAN), so that the conversion unit 191 is configured to be able to convert with high precision the input image PA used to train the object recognition filter RF into the input image PB used to generate the preprocessing filter PF.

[0151] Moreover, the conversion determination unit 192 can determine with high accuracy whether the image PA' converted from the input image PA by the conversion unit 191 is the input image PB.

[0152] Therefore, it is possible to improve both the conversion accuracy in the conversion unit 191 and the accuracy with which the conversion determination unit 192 determines whether the conversion result of the conversion unit 191 is genuine.

[0153] Similarly, it is possible to improve both the accuracy of the inverse conversion in the inverse conversion unit 193 and the accuracy of the inverse conversion determination unit 194 in determining whether the inverse conversion result of the inverse conversion unit 193 is genuine.

[0154] By such processing, the conversion unit 191 can, for example, convert a high-quality input image PA that has little effect on object recognition and is used to train the object recognition filter RF into a low-quality input image PB that is used to train the pre-processing filter PF with high accuracy.

[0155] In the example of FIG. 8, the conversion unit 191 can convert, with high accuracy, a high-quality bright image used when training the object recognition filter RF into a low-quality dark image used for training the pre-processing filter PF.

[0156] Furthermore, the conversion unit 191 can convert a low-quality image into a high-quality image of a corresponding type depending on the type of the low-quality image.

[0157] That is, types of low-quality images include, for example, dark images, images containing noise, cloudy or hazy images, images with motion blur, out-of-focus images, images with saturated brightness, or images in poor lighting conditions such as in stormy weather such as heavy rain or snow.

[0158] As a result, the conversion unit 191 converts the above-mentioned low-quality images into high-quality images such as a bright image, an image with corresponding noise removed, an image with cloudiness or haze removed, an image with blur removed, an image with focus adjustment, an image with brightness adjustment, and an image similar to that of a sunny day, depending on the type of image.

[0159] That is, the conversion unit 191 converts, for example, a dark image having low brightness quality into a bright, high-quality image with high accuracy. The conversion unit 191 also converts, for example, a low-quality image including noise into a high-quality image from which the noise has been removed with high accuracy. The conversion unit 191 also converts, for example, a low-quality image including cloudiness or haze into a high-quality image from which the cloudiness or haze has been removed. The conversion unit 191 also converts, for example, a low-quality image having blur caused by movement into a high-quality image from which the blur has been removed. The conversion unit 191 also converts, for example, a low-quality image having an out-of-focus state into a high-quality image having an in-focus state. The conversion unit 191 also converts, for example, a low-quality image having saturated brightness into a high-quality image with appropriately adjusted brightness. The conversion unit 191 also converts, for example, a low-quality image having poor lighting conditions due to stormy weather such as heavy rain or snow into a high-quality image having good lighting conditions.

[0160] Similarly, the inverse conversion unit 193 can convert with high precision a low-quality input image PB used to train the preprocessing filter PF into a high-quality input image PA that has little effect on object recognition and is used to train the object recognition filter RF.

[0161] The transformation discrimination unit learning unit 152 outputs the transformation unit 191 to the learning set generation unit 153 out of the transformation unit 191, transformation discrimination unit 192, inverse transformation unit 193, and inverse transformation discrimination unit 194 generated by learning using such a cyclic generative adversarial network (Cyclic GAN).

[0162] <Learning set generation unit> The learning set generation unit 153 uses the conversion unit 191 supplied by the conversion discrimination unit learning unit 152 to convert with high precision, as shown in Figure 9, a high-quality input image PA that has little impact on object recognition and is used to train the object recognition filter RF, into an image PA' that resembles the low-quality input image PB that is used to train the pre-processing filter PF.

[0163] The training set generation unit 153 outputs a pair of the generated high-quality input image PA and an image PA' that resembles the low-quality input image PB used for training the pre-processing filter PF to the evaluation image generation unit 154 as a training set.

[0164] <Evaluation image generation unit> The evaluation image generation unit 154 generates a heat map representing the error of the preprocessing filter PF as an evaluation image PE based on a learning set consisting of a pair of a high-quality input image PA and an image PA' that resembles a low-quality input image PB used to train the preprocessing filter PF, the preprocessing filter PF, and the object recognition filter RF, and outputs the heat map to the evaluation filter training unit 155.

[0165] More specifically, as shown in FIG. 10, the evaluation image generation unit 154 applies a preprocessing filter PF to a high-quality input image PA to generate a preprocessed image PC, and then applies an object recognition filter RF to the image PC to generate an object recognition result image PLa.

[0166] In addition, the evaluation image generation unit 154 generates a preprocessed image PC' by applying a preprocessing filter PF to the low-quality input image PA', and further generates an object recognition result image PLa' by applying an object recognition filter RF to the image PC'.

[0167] Furthermore, the evaluation image generation unit 154 generates, as an evaluation image PE, a heat map whose pixel values ​​are the pixel-by-pixel difference between the object recognition result image PLa obtained based on the high-quality input image PA and the object recognition result image PLa' obtained based on the low-quality input image PA'.

[0168] In other words, since the input images PA and PA' are basically the same high-quality image and low-quality image, respectively, and the same object recognition filter RF is applied to each of them, the evaluation image PE, which consists of a heat map in which the pixel values ​​are the differences between the object recognition result images PLa and PLa' obtained by each process, can be considered to be an image for evaluating the preprocessing filter PF.

[0169] In other words, since the input images PA and PA' correspond to the training data when training the pre-processing filter PF, ideally, if sufficient training has been performed, the images PC and PC' after being subjected to the pre-processing filter PF will both be the same input image PA, and the object recognition result images PLa and PLa', which are the processing results of the object recognition filter RF, will both be the same.

[0170] As a result, ideally, the difference between the pixel values ​​of all pixels in the object recognition result images PLa and PLa' becomes zero, and the evaluation image EP becomes a completely dark image.

[0171] However, if the preprocessing filter PF has not been sufficiently trained, the preprocessing will not be sufficient, and therefore, pixels in areas of the images PC, PC' after the preprocessing filter PF has been applied that have not been sufficiently preprocessed will be labeled with different object recognition results, resulting in differences in pixel values.

[0172] Therefore, in the evaluation image PE, areas with high and bright pixel values ​​and large differences between the object recognition result images PLa and PLa' can be considered to be areas where sufficient preprocessing was not performed by the preprocessing filter PF and proper object recognition was not possible.

[0173] Furthermore, in the evaluation image PE, in areas where the pixel values ​​are dark and close to 0, and the difference between the object recognition result images PLa and PLa' is small, it can be assumed that sufficient preprocessing has been performed by the preprocessing filter PF, and that appropriate object recognition has been performed.

[0174] The evaluation image generation unit 154 calculates an evaluation image PE for each of a plurality of learning sets consisting of a pair of a high-quality input image PA and an image PA' that resembles a low-quality input image PB used for learning the pre-processing filter PF, and outputs the plurality of sets of the image PA' that resembles the low-quality input image PB used for learning the pre-processing filter PF and the corresponding evaluation image PE to the evaluation filter learning unit 155.

[0175] <Evaluation filter learning unit> As shown in Figure 11, the evaluation filter learning unit 155 uses supervised learning to find an evaluation filter EF corresponding to the evaluation filter 133 based on multiple sets of images PA' that resemble low-quality input images PB used to learn the pre-processing filter PF and corresponding evaluation images PE.

[0176] That is, this learning makes it possible to generate an evaluation filter EF that can generate an evaluation image PE by performing a filter process on a low-quality input image PB.

[0177] Through the above series of processes, the evaluation filter EF is generated as the evaluation filter 133 in FIG. 4, and the recognition unit 73 in FIG. 4 can be configured.

[0178] As a result, in the recognition unit 73 of FIG. 4, as shown in FIG. 12, a preprocessing filter 131 is applied to a low-quality input image PI, converting it into a high-quality image PC through preprocessing, and an object recognition filter 132 is further applied to the image PC to generate an object recognition result image PL in which labels corresponding to the object recognition results are attached on a pixel-by-pixel basis.

[0179] Furthermore, the evaluation filter EF generated by the evaluation filter generating unit 134 is configured as the evaluation filter 133, and the evaluation filter 133 is applied to the input image PI to obtain the corresponding evaluation image PE.

[0180] The operation control unit 63 controls the operation of the vehicle 1 based on the object recognition result image PL.

[0181] In this case, for example, if the pixel values ​​in the evaluation image PE are higher than a predetermined value, the preprocessing filter 131 has not performed sufficient preprocessing, the area where proper object recognition is likely to have not been performed is larger than a predetermined area, and the object recognition result image PL is deemed to be less reliable than a predetermined level, the operation control unit 63 may not perform operation control based on the object recognition result image PL.

[0182] Furthermore, with regard to the preprocessing filter 131, the type of image that can be appropriately preprocessed varies depending on the teacher data used in supervised learning. Therefore, a plurality of preprocessing filters 131 corresponding to the type of teacher data may be prepared in advance, and the operation control unit 63 may switch between the preprocessing filters 131 to obtain an evaluation image PE, select the preprocessing filter 131 with the highest evaluation based on the evaluation image PE, and perform operation control based on the object recognition result image PL.

[0183] Furthermore, for example, in the evaluation image PE, for areas where the pixel value is higher than a predetermined value, where the preprocessing filter 131 has not performed sufficient preprocessing, and where appropriate object recognition has not been performed, the object recognition result of the object recognition result image PL is deemed to be insufficient, and the operation control unit 63 may perform operation control based on information about areas of the object recognition result image PL other than the areas where the pixel value is higher than the predetermined value.

[0184] As a result, the operation control unit 63 can evaluate the object recognition result image PL based on the evaluation image PE, and can realize appropriate operation control according to the evaluation result.

[0185] <Evaluation filter generation process> Next, the evaluation filter generation process performed by the evaluation filter generation unit 134 will be described with reference to the flowchart of FIG.

[0186] In step S11, the object recognition filter learning unit 171 of the filter learning unit 151 generates an object recognition filter RF through supervised learning using the method described with reference to FIG.

[0187] In step S12, the pre-processing filter learning unit 172 of the filter learning unit 151 generates a pre-processing filter PF through supervised learning using the method described with reference to FIG.

[0188] In step S13, the transformation discrimination unit learning unit 152 generates a transformation unit 191, a transformation discrimination unit 192, an inverse transformation unit 193, and an inverse transformation discrimination unit 194 by unsupervised learning using a cyclic generative adversarial network (Cyclic GAN) as described with reference to FIG. 8.

[0189] Then, the conversion determination unit learning unit 152 outputs the conversion unit 191 out of the generated conversion unit 191 , conversion determination unit 192 , inverse conversion unit 193 , and inverse conversion determination unit 194 to the learning set generation unit 153 .

[0190] In step S14, as described with reference to Figure 9, the training set generation unit 153 uses the conversion unit 191 to convert with high precision the high-quality input image PA that has little impact on object recognition and is used to train the object recognition filter RF into an image PA' that resembles the low-quality input image PB that is used to train the pre-processing filter PF.

[0191] The training set generation unit 153 then outputs a training set consisting of a pair of a high-quality input image PA and an image PA' that resembles the low-quality input image PB used for training the pre-processing filter PF to the evaluation image generation unit 154. The training set generation unit 153 converts a plurality of different input images PA into images PA', and outputs a training set consisting of a plurality of input images PA and images PA' to the evaluation image generation unit 154.

[0192] In step S15, as described with reference to Figure 10, the evaluation image generation unit 154 generates an evaluation image PE consisting of a heat map representing the error of the preprocessing filter PF based on a training set consisting of a pair of a high-quality input image PA and an image PA' that resembles the low-quality input image PB used to train the preprocessing filter PF, the preprocessing filter PF, and the object recognition filter RF, and outputs this to the evaluation filter training unit 155 together with the image PA' that resembles the low-quality input image PB used to train the preprocessing filter PF.

[0193] In step S16, as described with reference to FIG. 11, the evaluation filter learning unit 155 obtains an evaluation filter EF by supervised learning based on a set of an image PA′ that is likely to be the low-quality input image PB used to learn the preprocessing filter PF and the corresponding evaluation image PE, and configures it as the evaluation filter 133.

[0194] Through the above processing, it is possible to generate an evaluation filter EF for generating an evaluation image PE for evaluating the object recognition result image PL.

[0195] As a result, it becomes possible to appropriately evaluate the preprocessing filter 131 or the object recognition filter 132 simply by applying the determined evaluation filter to the input image input to the preprocessing filter 131 or the object recognition filter 132.

[0196] <Motion control processing> Next, with reference to the flowchart of FIG. 14, an operation control process based on the recognition process by the recognition unit 73 of FIG. 4 using the evaluation filter EF generated by the above-described evaluation filter generation process as the evaluation filter 133 will be described.

[0197] In step S31, the preprocessing filter 131 corresponding to the above-mentioned preprocessing filter PF performs preprocessing on an input image PI consisting of a low-quality image, converts it into a high-quality image PC, and outputs it to the object recognition filter 132.

[0198] In step S32, the object recognition filter 132 corresponding to the above-mentioned object recognition filter RF performs object recognition processing on the high-quality image PC to generate an object recognition result image PL and output it to the operation control unit 63.

[0199] In step S33, the evaluation filter 133 corresponding to the above-mentioned evaluation filter EF generates an evaluation image PE based on the input image PI and outputs it to the operation control unit 63.

[0200] In step S34, the operation control unit 63 controls the operation of the vehicle 1 based on the object recognition result image PL and the evaluation image PE.

[0201] In this case, for example, if the pixel value in the evaluation image PE is higher than a predetermined value, the pre-processing filter 131 has not performed sufficient pre-processing, the area where proper object recognition has not been performed is larger than a predetermined area, and the object recognition result image PL is deemed to be less reliable than a predetermined level, the operation control unit 63 may not perform operation control based on the object recognition result image PL, or may stop only the pre-processing filter 131.

[0202] In addition, if the object recognition result image PL is deemed to be less reliable than a predetermined level, the operation control unit 63 may replace at least one of the preprocessing filter 131 and the object recognition filter 132, which have been learned by supervised learning based on other training data.

[0203] That is, as described above, the conversion unit 191, which is trained by a cyclic generative adversarial network (Cyclic GAN), converts a low-quality image into a corresponding high-quality image depending on the type of the low-quality image.

[0204] Furthermore, since the evaluation image generation unit 154 generates the evaluation image PE based on the learning set generated by the conversion unit 191 in the learning set generation unit 153, the generated evaluation image PE also differs depending on the type of low-quality image.

[0205] Furthermore, since the evaluation image PE is used to generate an evaluation filter EF in the evaluation filter learning unit 155, the evaluation filter EF generated in the evaluation filter learning unit 155 will naturally differ depending on the type of low-quality image.

[0206] Therefore, the appropriate configurations of the pre-processing filter 131, the object recognition filter 132, and the combination of the pre-processing filter 131 and the object recognition filter 132 differ depending on the type of low-quality image.

[0207] For example, the preprocessing filter 131 is configured to be capable of appropriately preprocessing a dark image with low quality in terms of brightness as a low quality category, and is configured to be capable of appropriately preprocessing a noisy image with low quality in terms of noise as a low quality category.

[0208] Similarly, the object recognition filter 132, or the combination of the preprocessing filter 131 and the object recognition filter 132, is different from a configuration that can appropriately perform object recognition processing on a dark image with low brightness as a low-quality type, and a configuration that can appropriately perform object recognition processing on an image with low noise as a low-quality type, including noise.

[0209] Therefore, the evaluation filter 133 that can appropriately evaluate the processing results varies depending on the type of poor quality.

[0210] That is, there is an evaluation filter 133 for the object recognition result image PL for each type of low-quality image, such as a dark image, an image containing noise, a cloudy or hazy image, an image with blur caused by movement, an image that is out of focus, an image with saturated brightness, or an image with poor lighting conditions in inclement weather such as heavy rain or snow.

[0211] Therefore, the operation control unit 63 prepares, for each type of low-quality image, a pre-processing filter 131, an object recognition filter 132, and multiple configurations combining the pre-processing filter 131 and the object recognition filter 132, as well as an evaluation filter 133 for each of the corresponding multiple configurations.

[0212] The operation control unit 63 may then apply multiple evaluation filters 133 to the input image PI, obtain an evaluation image PE using each evaluation filter 133, and select and use either the pre-processing filter 131, the object recognition filter, or the combination of the pre-processing filter 131 and the object recognition filter that has the highest evaluation.

[0213] In this case, if there are many types of low-quality images and many types of evaluation filters 133, the evaluation image PE is obtained by restricting, depending on the type of low-quality image, the pre-processing filter 131, the object recognition filter 132, and one of the multiple evaluation filters 133 corresponding to multiple configurations combining the pre-processing filter 131 and the object recognition filter 132, for each type of low-quality image, depending on the location, time, etc.

[0214] Then, among the evaluation images PE thus obtained, the pre-processing filter 131, the object recognition filter, and the combination of the pre-processing filter 131 and the object recognition filter corresponding to the evaluation filter 133 with the highest evaluation may be selectively used.

[0215] In addition, the processing of the evaluation filter 133 may be performed by a server provided outside the vehicle 1. In this way, even if there are many types of low-quality images and many types of evaluation filters 133, the processing to obtain the evaluation image PE using the evaluation filter 133 can be performed by a server provided outside the vehicle 1, thereby reducing the overall processing load of the recognition unit 73 and increasing the processing speed.

[0216] Furthermore, in the above, an example has been described in which a pre-processing filter 131, an object recognition filter 132, and a plurality of configurations combining the pre-processing filter 131 and the object recognition filter 132 according to the type of low-quality image, and a corresponding plurality of evaluation filters 133 are prepared, but low-quality images in object recognition can also be considered to be those that depend on the imaging environment.

[0217] That is, the brightness and amount of noise in an image change depending on the imaging environment, and for example, there are similar changes in the imaging environment depending on the time of day and weather, such as morning, noon, night, sunny, rainy, and snowy, as mentioned above, and further, changes depending on the season, such as spring, summer, autumn, and winter, are also considered to be low-quality images during recognition. That is, in the case of a recognizer that has trained with images taken on sunny days, images taken on rainy days will be considered to be low-quality images because the colors and brightness are different.

[0218] Similarly, in object recognition processing, for example, images can be considered low quality due to differences in the language used, building style, etc. For example, for a recognizer trained to recognize building objects in images taken in Japan, recognizing buildings in images taken overseas can be considered low quality images.

[0219] Therefore, the operation control unit 63 may prepare a pre-processing filter 131, an object recognition filter 132, and multiple configurations combining the pre-processing filter 131 and the object recognition filter 132, as well as an evaluation filter 133 for each of the multiple corresponding configurations, for each type of low-quality image, such as the source of the data used for learning and the imaging environment, such as date, time, season, location, country, etc., and for each combination thereof, and switch between and use them depending on the evaluation based on the evaluation image PE.

[0220] Furthermore, for example, in the evaluation image PE, areas where pixel values ​​are higher than a predetermined value, where sufficient processing has not been performed by the preprocessing filter 131 and the object recognition filter 132, and where appropriate object recognition has not been performed may be regarded as areas where object recognition processing is not sufficient, and the operation control unit 63 may perform operation control based on information about areas in the object recognition result image PL where object recognition processing has been performed sufficiently, other than areas where pixel values ​​are higher than a predetermined value.

[0221] In either case, the operation control unit 63 can evaluate the object recognition result image PL based on the evaluation image PE, and can realize appropriate operation control according to the evaluation result.

[0222] In the above, an example has been described in which the evaluation filter generation unit 134 is configured within the recognition unit 73 and the evaluation filter EF generated by the evaluation filter generation unit 134 is used as the evaluation filter 133.

[0223] However, the evaluation filter generation unit 134 does not necessarily have to be provided within the recognition unit 73 , and may be provided separately from the vehicle control system 11 including the recognition unit 73 .

[0224] In this case, the evaluation filter 133 may be configured by reading an evaluation filter EF generated by an evaluation filter generation unit 134 provided separately from the vehicle 1 including the recognition unit 73 via a communication network or the like.

[0225] Furthermore, in the above, an example has been described in which an evaluation image PE is generated by the evaluation filter 133 each time an object recognition process is performed by the object recognition filter 132, and the object recognition result image PL is evaluated each time. However, if the conditions do not change much, it is considered that the change in the evaluation image PE will be small. Therefore, the execution frequency may be reduced by executing the evaluation image PE according to the date, time, or location, or by executing the evaluation image PE at predetermined time intervals.

[0226] Here, for example, if an evaluation image PE is generated and an evaluation score of the object recognition result image PL is calculated, and the evaluation score is smaller than a predetermined value, it may be determined that the preprocessing by the preprocessing filter 131 is insufficient, and the processing of the preprocessing filter 131 may be stopped until the next evaluation image PE is calculated and the evaluation score is calculated.

[0227] Then, an evaluation image PE is generated and an evaluation score of the object recognition result image PL is calculated. If the evaluation score is greater than a predetermined value, it is assumed that the preprocessing by the preprocessing filter 131 is sufficient, and the preprocessing filter 131 may be allowed to execute processing until the next evaluation image PE is calculated and the evaluation score is calculated.

[0228] Furthermore, in the above, an example has been described in which the preprocessing filter 131 and the object recognition filter 132 have a configuration obtained by learning, but the configuration does not have to be obtained by learning. Similar evaluation filters may be obtained for various recognizers, and the recognizers may be selectively used depending on the evaluation from the evaluation image PE based on the evaluation filter.

[0229] <<5. First Modification>> 10, the evaluation image generation unit 154 generates a preprocessed image PC by applying a preprocessing filter PF to a high-quality input image PA. Then, the evaluation image generation unit 154 applies an object recognition filter RF to the image PC to generate an object recognition result image PLa, and generates, as an evaluation image PE, a heat map whose pixel values ​​are the pixel-by-pixel differences between the object recognition result image PLa' obtained based on a low-quality input image PA'.

[0230] However, when the preprocessing filter PF processes the object recognition result image PLa for a high-quality input image PA, it is expected that the input image PA itself will ideally be output.

[0231] Therefore, when obtaining the evaluation image PE, as shown in FIG. 15, processing of the pre-processing filter PF for the high-quality input image PA may be omitted, and the input image PA may be used as is instead of the image PC.

[0232] <<6. Second Modification>> In the above, as shown in FIG. 15, an example has been described in which processing of the pre-processing filter PF is omitted for a high-quality input image PA when obtaining an evaluation image PE.

[0233] However, the areas with low evaluations for the pre-processing filter PF are due to the low quality of the input image PA'.

[0234] Therefore, as shown in FIG. 16, when calculating the evaluation image PE, the pre-processing filter PF may not be applied to either the low-quality input image PA' or the high-quality input image PA, but the evaluation image PE may be calculated from the difference between the object recognition result image PLa' obtained by applying the direct object recognition filter RF and the object recognition result image PLa obtained by applying the direct object recognition filter RF.

[0235] <<7. Third Modification>> In the above, an example has been described in which the recognition unit 73 applies the preprocessing filter 131 to the input image PI to obtain a preprocessed image PC, and then applies the object recognition filter 132 to the preprocessed image PC to obtain an object recognition result image PL.

[0236] However, the processing performed on the input image PI in the pre-processing filter 131 is processing performed on values ​​obtained by rounding pixel values ​​in the input image PI, so there is actually a difference between the image PI′ that has been pre-processed and the input image PI.

[0237] Therefore, the difference between the image PI' that is actually preprocessed and the input image PI may be extracted as a residual Pr from the input image PI, added to the preprocessed image PC to obtain an image PCr, and this image PCr may then be multiplied by the object recognition filter 132.

[0238] FIG. 17 shows an example configuration of a recognition unit 73 in which the difference between the image PI' that is actually preprocessed and the input image PI is extracted as a residual Pr from the input image PI, added to the preprocessed image PC to obtain an image PCr, and this image PCr is then multiplied by an object recognition filter 132.

[0239] In the recognition unit 73 of FIG. 17, components having the same functions as those in the recognition unit 73 of FIG. 4 are denoted by the same reference numerals, and the description thereof will be omitted as appropriate.

[0240] That is, the recognition unit 73 in FIG. 17 differs from the recognition unit 73 in FIG. 4 in that it newly includes a residual extraction unit 201 and an addition unit 202.

[0241] The residual extraction unit 201 extracts, from the input image PI, a difference between the input image PI and an image PI′ that is actually processed by the pre-processing filter 131 , as a residual Pr, and outputs the residual Pr to the addition unit 202 .

[0242] The adder 202 generates an image PCr by adding the image PC preprocessed by the preprocessing filter 131 and the residual Pr, and outputs the image PCr to the object recognition filter 132.

[0243] With this configuration, the object recognition filter 132 performs object recognition processing on the image PCr obtained by adding the residual Pr to the image PC preprocessed by the preprocessing filter 131, and outputs an object recognition result image PL.

[0244] By the recognition unit 73 configured as shown in FIG. 17, the difference between the image PI' that is actually preprocessed and the input image PI is extracted as a residual Pr from the input image PI, which is added to the preprocessed image PC to obtain an image PCr, and this image PCr is then multiplied by the object recognition filter 132, thereby eliminating the error associated with the residual Pr and enabling more accurate object recognition processing to be achieved.

[0245] <<8. Fourth Variation>> In the above, we have described an example in which the configuration of the recognition unit 73 includes a residual extraction unit 201 and an addition unit 202, and the object recognition filter 132 performs object recognition processing on an image PCr obtained by adding the residual Pr to an image PC that has been preprocessed by the preprocessing filter 131, and outputs an object recognition result image PL.

[0246] Therefore, the processing of the evaluation image generation unit 154 may also be configured to extract residuals from the input image, add them to the pre-processed image, and then generate a heat map representing the error of the pre-processing filter PF as the evaluation image PE.

[0247] That is, as shown in FIG. 18, the evaluation image generation unit 154 extracts a residual Pr from a high-quality input image PA, and applies a preprocessing filter PF to the high-quality input image PA to generate a preprocessed image PC, and adds the extracted residual Pr to generate an image PCr.

[0248] The evaluation image generation unit 154 extracts a residual Pr' from the low-quality input image PA', applies a preprocessing filter PF to the low-quality input image PA' to generate a preprocessed image PC', and adds the extracted residual Pr' to generate an image PCr'.The evaluation image generation unit 154 then applies an object recognition filter RF to the image PCr' to generate an object recognition result image PLa'.

[0249] Then, the evaluation image generation unit 154 generates, as an evaluation image PE, a heat map whose pixel values ​​are the pixel-by-pixel difference between the object recognition result image PLa obtained based on the high-quality input image PA and the object recognition result image PLa' obtained based on the low-quality input image PA'.

[0250] <<9. Fifth Modification>> Furthermore, we have described an example in which the residuals are calculated, preprocessed, and then added to obtain the evaluation image PE. However, as in the first modified example described with reference to FIG. 15, when obtaining the evaluation image PE, as shown in FIG. 19, processing of the preprocessing filter PF may be omitted for a high-quality input image PA.

[0251] <<10. Example of execution by software>> The above-described series of processes can be executed by hardware, but can also be executed by software. When the series of processes are executed by software, the programs constituting the software are installed from a recording medium into a computer incorporated in dedicated hardware, or into, for example, a general-purpose computer that can execute various functions by installing various programs.

[0252] 20 shows an example of the configuration of a general-purpose computer. This personal computer has a built-in CPU (Central Processing Unit) 1001. An input / output interface 1005 is connected to the CPU 1001 via a bus 1004. A ROM (Read Only Memory) 1002 and a RAM (Random Access Memory) 1003 are connected to the bus 1004.

[0253] Connected to the input / output interface 1005 are an input unit 1006 including input devices such as a keyboard and a mouse through which a user inputs operation commands, an output unit 1007 that outputs a processing operation screen and images of processing results to a display device, a storage unit 1008 including a hard disk drive or the like that stores programs and various data, and a communication unit 1009 including a LAN (Local Area Network) adapter or the like that executes communication processing via a network typified by the Internet. Also connected to the input / output interface 1005 is a drive 1010 that reads and writes data from / to removable storage media 1011 such as a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor memory.

[0254] The CPU 1001 executes various processes in accordance with a program stored in a ROM 1002 or a program read from a removable storage medium 1011 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, installed in a storage unit 1008, and loaded from the storage unit 1008 into a RAM 1003. The RAM 1003 also stores data necessary for the CPU 1001 to execute various processes as appropriate.

[0255] In a computer configured as described above, the CPU 1001 performs the above-described series of processes by, for example, loading a program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it.

[0256] The program executed by the computer (CPU 1001) can be provided by being recorded on a removable storage medium 1011 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0257] In a computer, a program can be installed in the storage unit 1008 via the input / output interface 1005 by inserting a removable storage medium 1011 into the drive 1010. The program can also be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Alternatively, the program can be installed in the ROM 1002 or the storage unit 1008 in advance.

[0258] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0259] 20. The CPU 1001 in FIG. 20 realizes the function of the recognition unit 73 in FIG.

[0260] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0261] The embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure.

[0262] For example, the present disclosure can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network.

[0263] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by multiple devices.

[0264] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0265] The present disclosure can also be configured as follows.

[0266] <1> a recognition unit that recognizes an object in an input image; an evaluation unit that evaluates the recognition unit; An information processing device comprising: <2> The image processing system further includes a conversion unit for converting the input image into a low-quality image. <1> The information processing device described in <3> The conversion unit is generated by unsupervised learning. <2> The information processing device described in <4> The unsupervised learning is learning using a cyclic generative adversarial network (Cyclic GAN). <3> The information processing device described in <5> an evaluation unit generating unit that generates the evaluation unit based on a recognition result by the recognition unit for the input image and a recognition result by the recognition unit for the low-quality image; <2> The information processing device described in <6> further including an evaluation image generation unit that generates an evaluation image consisting of a difference in pixel values ​​between a recognition result image that is the recognition result of the input image by the recognition unit and a recognition result image that is the recognition result of the low-quality image by the recognition unit, The evaluation unit generation unit generates the evaluation unit based on supervised learning using the evaluation image and the input image. <5> The information processing device described in <7> Further comprising a pre-processing unit for performing pre-processing to enhance and correct the image; The recognition unit recognizes an object in a corrected input image obtained by correcting the input image by the preprocessing unit. <6> The information processing device described in <8> The evaluation image generation unit generates the evaluation image consisting of a difference between the recognition result image, which is the recognition result by the recognition unit for a corrected input image obtained by improving and correcting the input image through the pre-processing unit, and the recognition result image, which is the recognition result by the recognition unit for a corrected low-quality image obtained by improving and correcting the low-quality image through the pre-processing unit. <7> The information processing device described in <9> The evaluation image generation unit generates the evaluation image consisting of a difference between the recognition result image, which is the recognition result by the recognition unit for the input image, and the recognition result image, which is the recognition result by the recognition unit for the corrected image obtained by improving and correcting the low-quality image by the pre-processing unit. <7> The information processing device described in <10> a residual extraction unit that extracts a residual related to the preprocessing from the image that has been preprocessed by the preprocessing unit; an adder that adds the residual to the image that has been preprocessed by the preprocessing unit, The recognition unit recognizes an object in a residual-added corrected input image obtained by adding the residual to a corrected input image obtained by correcting the input image by the preprocessing unit. <7> The information processing device described in <11> The evaluation image generation unit generates the evaluation image consisting of a difference between the recognition result image, which is the recognition result by the recognition unit for the residual-added corrected input image obtained by adding the residual to the corrected input image obtained by improving the quality of the input image and correcting it by the pre-processing unit, and the recognition result image, which is the recognition result by the recognition unit for the residual-added corrected low-quality image obtained by adding the residual to the corrected low-quality image obtained by improving the quality of the low-quality image and correcting it by the pre-processing unit. <10> The information processing device described in <12> The evaluation image generation unit generates the evaluation image consisting of a difference between the recognition result image, which is the recognition result by the recognition unit for the input image, and the recognition result image, which is the recognition result by the recognition unit for a residual-added corrected low-quality image, which is obtained by adding the residual to a corrected low-quality image obtained by improving and correcting the low-quality image by the pre-processing unit. <10> The information processing device described in <13> When the evaluation of the recognition unit by the evaluation unit is lower than a predetermined value, the preprocessing unit stops preprocessing and outputs the input image to a subsequent stage in an unprocessed state. <7> The information processing device described in <14> The evaluation unit evaluates the recognition unit at a frequency lower than a frequency at which the recognition unit recognizes an object in the input image. <1> ~ <13> 10. An information processing device according to claim 9, wherein: <15> The evaluation unit evaluates the recognition unit when any one of a predetermined time interval, a predetermined time, and a predetermined position, or a combination thereof, satisfies a predetermined condition. <14> The information processing device described in <16> the recognition unit is present in a plurality of units, and the plurality of recognition units recognize the object in the input image according to different imaging environments, a plurality of the evaluation units are present corresponding to the plurality of recognition units, and the plurality of evaluation units evaluate the corresponding recognition units, a selection unit that selects one of the plurality of recognition units based on evaluation results of the plurality of evaluation units, respectively, of the plurality of recognition units. <1> ~ <15> 10. An information processing device according to claim 9, wherein: <17> The imaging environment is the imaging environment according to the time, season, and location at which the input image was captured. <16> The information processing device described in <18> a recognition unit that recognizes an object in an input image; an information processing method for an information processing device including an evaluation unit that evaluates the recognition unit, The evaluation unit evaluates the recognition unit. An information processing method comprising the steps. <19> a recognition unit that recognizes an object in an input image; an evaluation unit that evaluates the recognition unit; programs that make a computer function. <20> a recognition unit that recognizes an object in an input image; an evaluation unit that evaluates the recognition unit; An information processing system including: [Explanation of symbols]

[0267] 73 recognition unit, 131 preprocessing filter, 132 object recognition filter, 133 evaluation filter, 134 evaluation filter generation unit, 151 filter training unit, 152 transformation discrimination training unit, 153 training set, 154 evaluation image generation unit, 155 evaluation filter training unit, 171 object recognition filter training unit, 172 preprocessing filter training unit, 191 transformation unit, 192 transformation discrimination unit, 193 inverse transformation unit, 194 inverse transformation discrimination unit

Claims

1. a recognition unit that recognizes an object in an input image; an evaluation unit that evaluates the recognition unit; a conversion unit that degrades the input image and converts it into a low-quality image; an evaluation unit generation unit that generates the evaluation unit based on a recognition result by the recognition unit for the input image and a recognition result by the recognition unit for the low-quality image; An information processing device comprising:

2. The conversion unit is generated by unsupervised learning. The information processing device according to claim 1 .

3. The unsupervised learning is learning using a cyclic generative adversarial network (Cyclic GAN). The information processing device according to claim 2 .

4. further including an evaluation image generation unit that generates an evaluation image consisting of a difference in pixel values ​​between a recognition result image that is the recognition result of the input image by the recognition unit and a recognition result image that is the recognition result of the low-quality image by the recognition unit, The evaluation unit generation unit generates the evaluation unit based on supervised learning using the evaluation image and the input image. The information processing device according to claim 1 .

5. Further comprising a pre-processing unit for performing pre-processing to enhance and correct the image; The recognition unit recognizes an object in a corrected input image obtained by correcting the input image by the preprocessing unit. The information processing device according to claim 4 .

6. The evaluation image generation unit generates the evaluation image consisting of a difference between the recognition result image, which is the recognition result by the recognition unit for a corrected input image obtained by improving and correcting the input image through the pre-processing unit, and the recognition result image, which is the recognition result by the recognition unit for a corrected low-quality image obtained by improving and correcting the low-quality image through the pre-processing unit. The information processing device according to claim 5 .

7. The evaluation image generation unit generates the evaluation image consisting of a difference between the recognition result image, which is the recognition result by the recognition unit for the input image, and the recognition result image, which is the recognition result by the recognition unit for the corrected image obtained by improving and correcting the low-quality image by the pre-processing unit. The information processing device according to claim 5 .

8. a residual extraction unit that extracts a residual related to the preprocessing from the image that has been preprocessed by the preprocessing unit; an adder that adds the residual to the image that has been preprocessed by the preprocessing unit, The recognition unit recognizes an object in a residual-added corrected input image obtained by adding the residual to a corrected input image obtained by correcting the input image by the preprocessing unit. The information processing device according to claim 5 .

9. The evaluation image generation unit generates the evaluation image consisting of a difference between the recognition result image, which is the recognition result by the recognition unit for the residual-added corrected input image obtained by adding the residual to the corrected input image obtained by improving the quality of the input image and correcting it by the pre-processing unit, and the recognition result image, which is the recognition result by the recognition unit for the residual-added corrected low-quality image obtained by adding the residual to the corrected low-quality image obtained by improving the quality of the low-quality image and correcting it by the pre-processing unit. The information processing device according to claim 8 .

10. The evaluation image generation unit generates the evaluation image consisting of a difference between the recognition result image, which is the recognition result by the recognition unit for the input image, and the recognition result image, which is the recognition result by the recognition unit for a residual-added corrected low-quality image, which is obtained by adding the residual to a corrected low-quality image obtained by improving and correcting the low-quality image by the pre-processing unit. The information processing device according to claim 8 .

11. When the evaluation of the recognition unit by the evaluation unit is lower than a predetermined value, the preprocessing unit stops preprocessing and outputs the input image to a subsequent stage in an unprocessed state. The information processing device according to claim 5 .

12. The evaluation unit evaluates the recognition unit at a frequency lower than a frequency at which the recognition unit recognizes an object in the input image. The information processing device according to claim 1 .

13. The evaluation unit evaluates the recognition unit when any one of a predetermined time interval, a predetermined time, and a predetermined position, or a combination thereof, satisfies a predetermined condition. The information processing device according to claim 12.

14. the recognition unit is present in plurality, and the plurality of recognition units recognize the object in the input image according to a different imaging environment, a plurality of the evaluation units are present corresponding to the plurality of recognition units, and the plurality of evaluation units evaluate the corresponding recognition units, a selection unit that selects one of the plurality of recognition units based on evaluation results of the plurality of recognition units by the plurality of evaluation units, The information processing device according to claim 1 .

15. The imaging environment is the imaging environment according to the time, season, and location at which the input image was captured. The information processing device according to claim 14.

16. a recognition unit that recognizes an object in an input image; an evaluation unit that evaluates the recognition unit; a conversion unit that degrades the input image and converts it into a low-quality image; an evaluation unit generation unit that generates the evaluation unit based on a recognition result by the recognition unit for the input image and a recognition result by the recognition unit for the low-quality image, the recognition unit performs a recognition process to recognize an object in the input image; the evaluation unit performs an evaluation process to evaluate the recognition unit; the conversion unit performs a conversion process to degrade the input image and convert it into a low-quality image; the evaluation unit generation unit performs an evaluation unit generation process to generate the evaluation unit based on a recognition result by the recognition unit for the input image and a recognition result by the recognition unit for the low-quality image; An information processing method including:

17. a recognition unit that recognizes an object in an input image; an evaluation unit that evaluates the recognition unit; a conversion unit that degrades the input image and converts it into a low-quality image; an evaluation unit generation unit that generates the evaluation unit based on a recognition result by the recognition unit for the input image and a recognition result by the recognition unit for the low-quality image; programs that make a computer function.

18. a recognition unit that recognizes an object in an input image; an evaluation unit that evaluates the recognition unit; a conversion unit that degrades the input image and converts it into a low-quality image; an evaluation unit generation unit that generates the evaluation unit based on a recognition result by the recognition unit for the input image and a recognition result by the recognition unit for the low-quality image; An information processing system including:

Images