Data processing device, data processing method, and program

Abstract

The present technology relates to a data processing device, a data processing method, and a program that can acheive future high functionality. A data amount of sensor data is controlled on the basis of sensor data outputted by a sensing unit that senses light, and a data amount of other data stored in a storage unit that stores data other than the sensor data, and the sensor data is stored in the storage unit. The present technology can be applied, for example, to a camera having a sensor that senses light and outputs image data and the like.

Description

Data processing device, data processing method, and program

[0001] The present technology relates to a data processing device, a data processing method, and a program, and in particular to a data processing device, a data processing method, and a program that enable future high functionality, for example.

[0002] For example, Patent Document 1 describes an imaging device having a first signal processing unit, a second signal processing unit, and a recognition processing unit. The first signal processing unit performs first signal processing, and the second signal processing unit performs second signal processing that is at least partially common to the first signal processing. The recognition processing unit performs recognition processing based on output data of the second signal processing unit.

[0003] International Publication No. 2021 / 070894

[0004] For example, when performing data processing such as recognition processing using images (data) obtained by the imaging device described in Patent Document 1, the images obtained by the imaging device are stored in a memory. In addition to the images, the memory also stores data necessary for the operation of the device, such as firmware (FW) and a data processing library. The data processing library includes, for example, a learning model used in the recognition processing.

[0005] In order to reduce the cost and size of the device, a memory with as small a storage capacity as possible is used, which can store, for example, images obtained by an imaging device and data necessary for the operation of the device.

[0006] Incidentally, the amount of data required for the operation of the device, such as firmware and learning models, may be reduced through future version upgrades. Since the amount of data required for the operation of the device is reduced, there is excess memory storage space, allowing images with a larger amount of data, such as images with a larger (wider, higher) angle of view or resolution, to be stored in the memory. Storing images with a larger angle of view or resolution in the memory allows the device to have more advanced functionality. For example, this enables advanced data processing, such as recognition processing targeting a wider range or recognition processing targeting a greater distance.

[0007] However, in order to obtain an image with a larger angle of view and resolution in an imaging device, it is necessary to change the HW (hardware), such as changing the lens to a wide-angle lens or changing the image sensor to a multi-pixel (and larger if necessary) image sensor.

[0008] As a practical matter, it is difficult to change the hardware, such as the lens or image sensor, of an imaging device after it has been sold, making it difficult to improve the device's functionality.

[0009] The present technology has been developed in light of these circumstances, and will enable future enhancements in functionality.

[0010] The data processing device or program of the present technology is a data processing device that includes a processing unit that controls the amount of sensor data based on the amount of sensor data output by a sensing unit that senses light and the amount of other data stored in a memory unit that stores other data other than the sensor data, and stores the sensor data in the memory unit, or a program for causing a computer to function as such a data processing device.

[0011] A data processing method of the present technology is a data processing method that includes controlling the amount of sensor data output by a sensing unit that senses light and the amount of other data stored in a memory unit that stores other data other than the sensor data, and storing the sensor data in the memory unit.

[0012] In this technology, the amount of sensor data output by a sensing unit that senses light and the amount of other data stored in a memory unit that stores other data other than the sensor data are controlled based on the amount of the sensor data, and the sensor data is stored in the memory unit.

[0013] The data processing device may be an independent device or an internal block constituting one device.

[0014] The program can be provided by transmitting it via a transmission medium or by recording it on a recording medium.

[0015] 1 is a block diagram showing an example of a configuration of a vehicle control system 11 which is an example of a mobile device control system to which the present technology is applied. FIG. 1 is a diagram showing an example of sensing areas by the camera 51, radar 52, LiDAR 53, ultrasonic sensor 54, etc. of the external recognition sensor 25. FIG. 2 is a block diagram showing a first example of a configuration of the camera 51. FIG. 3 is a diagram explaining an example of a storage state of a volatile memory 122. FIG. 4 is a diagram explaining an example of conversion of an original image to a converted image by the image processing unit 121. FIG. 5 is a flowchart explaining an example of processing by the image processing unit 121. FIG. 6 is a diagram showing an example of storage of a converted image in the volatile memory 122. FIG. 7 is a diagram explaining an example of image recognition processing performed before the FW and library are updated. FIG. 8 is a diagram explaining an example of image recognition processing performed after the FW and library are updated in the future. FIG. 9 is a diagram showing an example of the relationship between updates of the FW and library and the forward distance and lateral distance at which an object can be recognized by image recognition processing. FIG. 10 is a flowchart explaining an example of processing for updating data stored in the non-volatile memory 123 of the camera 51. FIG. 11 is a block diagram showing a second example of a configuration of the camera 51. FIG. 12 is a block diagram showing another example of a configuration of the vehicle control system 11. FIG. 13 is a block diagram showing an example of a configuration of an embodiment of a computer to which the present technology is applied.

[0016] <Configuration example of vehicle control system>

[0017] FIG. 1 is a block diagram showing an example of the configuration of a vehicle control system 11, which is an example of a mobility device control system to which the present technology is applied.

[0018] The vehicle control system 11 is provided in the vehicle 1 and performs processing related to automated driving of the vehicle 1. This automated driving includes automated driving of levels 1 to 5, and remote driving and remote assistance of the vehicle 1 by a remote driver.

[0019] The vehicle control system 11 includes a vehicle control ECU (Electronic Control Unit) 21, a communication unit 22, a map information storage unit 23, a location information acquisition unit 24, an external recognition sensor 25, an in-vehicle sensor 26, a vehicle sensor 27, a memory unit 28, a driving automation control unit 29, a DMS (Driver Monitoring System) 30, an HMI (Human Machine Interface) 31, and a vehicle control unit 32.

[0020] The vehicle control ECU 21, communication unit 22, map information storage unit 23, position information acquisition unit 24, external recognition sensor 25, in-vehicle sensor 26, vehicle sensor 27, memory unit 28, driving automation control unit 29, DMS 30, HMI 31, and vehicle control unit 32 are interconnected via a communication network 41 to enable mutual communication. The communication network 41 is configured, for example, by an in-vehicle communication network or bus conforming to a digital two-way communication standard such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), FlexRay (registered trademark), or Ethernet (registered trademark). Different communication networks 41 may be used depending on the type of data being transmitted. For example, CAN may be used for data related to vehicle control, and Ethernet may be used for large-volume data. In addition, each part of the vehicle control system 11 may be directly connected without going through the communication network 41, using wireless communication intended for communication over relatively short distances, such as near field communication (NFC) or Bluetooth (registered trademark).

[0021] In the following description, when each unit of the vehicle control system 11 communicates via the communication network 41, the description of the communication network 41 will be omitted. For example, when the vehicle control ECU 21 and the communication unit 22 communicate via the communication network 41, it will simply be described that the vehicle control ECU 21 and the communication unit 22 communicate with each other.

[0022] The vehicle control ECU 21 is configured with various processors such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), etc. The vehicle control ECU 21 controls the entire or part of the functions of the vehicle control system 11.

[0023] The communication unit 22 communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, etc., and transmits and receives various data. At this time, the communication unit 22 can perform communication using a plurality of communication methods.

[0024] The following provides an overview of communication with the outside of the vehicle that can be performed by the communication unit 22. The communication unit 22 communicates with a server (hereinafter referred to as an external server) or the like on an external network via a base station or an access point using a wireless communication method such as 5G (fifth generation mobile communication system), LTE (Long Term Evolution), or DSRC (Dedicated Short Range Communications). The external network with which the communication unit 22 communicates is, for example, the Internet, a cloud network, or a network specific to a carrier. The communication method used by the communication unit 22 with the external network is not particularly limited as long as it is a wireless communication method that enables digital two-way communication at a communication speed equal to or higher than a predetermined distance.

[0025] Furthermore, for example, the communication unit 22 can communicate with a terminal located near the vehicle using P2P (Peer to Peer) technology. The terminal located near the vehicle can be, for example, a terminal worn by a mobile object moving at a relatively slow speed, such as a pedestrian or a bicycle, a terminal installed at a fixed location in a store, or an MTC (Machine Type Communication) terminal. Furthermore, the communication unit 22 can also perform V2X communication. V2X communication refers to communication between the vehicle and others, such as vehicle-to-vehicle communication with another vehicle, vehicle-to-infrastructure communication with a roadside unit, vehicle-to-home communication, and vehicle-to-pedestrian communication with a terminal carried by a pedestrian.

[0026] The communication unit 22 can receive, for example, a program for updating software that controls the operation of the vehicle control system 11 from the outside (over the air). The communication unit 22 can also receive map information, traffic information, information about the surroundings of the vehicle 1, and the like from the outside. For example, the communication unit 22 can also transmit information about the vehicle 1 and information about the surroundings of the vehicle 1 to the outside. Information about the vehicle 1 that the communication unit 22 transmits to the outside includes, for example, data indicating the state of the vehicle 1 and the recognition results obtained by the external recognition sensor 25. Furthermore, for example, the communication unit 22 performs communication corresponding to a vehicle emergency notification system such as e-call.

[0027] For example, the communication unit 22 receives electromagnetic waves transmitted by a road traffic information and communication system (VICS (Vehicle Information and Communication System) (registered trademark)) such as a radio beacon, an optical beacon, or FM multiplex broadcasting.

[0028] The following provides an overview of communication with the vehicle interior that can be performed by the communication unit 22. The communication unit 22 can communicate with each device in the vehicle using, for example, wireless communication. The communication unit 22 can communicate with each device in the vehicle using a communication method that enables bidirectional digital communication at a predetermined communication speed or higher via wireless communication, such as wireless LAN, Bluetooth (registered trademark), NFC, or Wireless USB (WUSB). The communication unit 22 can also communicate with each device in the vehicle using wired communication. For example, the communication unit 22 can communicate with each device in the vehicle using wired communication via a cable connected to a connection terminal (not shown). The communication unit 22 can communicate with each device in the vehicle using a communication method that enables bidirectional digital communication at a predetermined communication speed or higher via wired communication, such as Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI) (registered trademark), or Mobile High-Definition Link (MHL).

[0029] Here, the in-vehicle device refers to, for example, a device in the vehicle that is not connected to the communication network 41. Possible in-vehicle devices include, for example, a mobile device or wearable device carried by a user in the vehicle, such as a driver, and an information device brought into the vehicle and temporarily installed therein.

[0030] The map information storage unit 23 stores one or both of a map acquired from an external source and a map created by the vehicle 1. For example, the map information storage unit 23 stores a three-dimensional high-precision map, a global map that is less accurate than a high-precision map and covers a wide area, and the like.

[0031] Examples of high-precision maps include dynamic maps, point cloud maps, and vector maps. A dynamic map is a map consisting of four layers of dynamic information, quasi-dynamic information, quasi-static information, and static information, and is provided to the vehicle 1 from an external server or the like. A point cloud map is a map made up of a point cloud (point cloud data). A vector map is a map adapted to automated driving by associating traffic information such as the positions of lanes and traffic lights with the point cloud map.

[0032] The point cloud map and the vector map may be provided, for example, from an external server or the like, or may be created in the vehicle 1 based on sensing results from the camera 51, radar 52, LiDAR 53, etc. as a map for matching with a local map described later, and stored in the map information storage unit 23. Furthermore, when a high-precision map is provided from an external server or the like, map data of, for example, an area of ​​several hundred square meters relating to the planned route along which the vehicle 1 will travel is acquired from the external server or the like in order to reduce communication capacity.

[0033] The location information acquisition unit 24 receives GNSS (Global Navigation Satellite System) signals from GNSS satellites and acquires location information of the vehicle 1. The acquired location information is supplied to the driving automation control unit 29. Note that the method used by the location information acquisition unit 24 is not limited to using GNSS signals, and it may also acquire location information using a beacon, for example.

[0034] The external recognition sensor 25 includes various sensors as a sensing unit that senses various physical quantities used to recognize the situation outside the vehicle 1. The external recognition sensor 25 supplies the sensor data output by each sensor and the results of data processing such as recognition processing performed using the sensor data, etc., to each unit of the vehicle control system 11. The type and number of sensors included in the external recognition sensor 25 are arbitrary.

[0035] For example, the external recognition sensor 25 includes a camera 51, a radar 52, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 53, and an ultrasonic sensor 54. Without being limited to this, the external recognition sensor 25 may be configured to include one or more types of sensors selected from the camera 51, the radar 52, the LiDAR 53, and the ultrasonic sensor 54. The number of cameras 51, radars 52, LiDARs 53, and ultrasonic sensors 54 is not particularly limited as long as the number is a number that can be realistically installed on the vehicle 1. Furthermore, the types of sensors included in the external recognition sensor 25 are not limited to this example, and the external recognition sensor 25 may include other types of sensors. Examples of sensing areas of each sensor included in the external recognition sensor 25 will be described later.

[0036] The photographing (imaging) method of the camera 51 is not particularly limited. For example, cameras using various photographing methods, such as a time-of-flight (ToF) camera, a stereo camera, a monocular camera, or an infrared camera, which are photographing methods capable of distance measurement, can be applied to the camera 51 as needed. The camera 51 is not limited to these, and may simply acquire an image (an image obtained by sensing visible light) without distance measurement. In other words, any device capable of sensing light (including electromagnetic waves) can be used as the camera 51. For simplicity of explanation, the camera 51 is assumed to be, for example, a camera that senses visible light and outputs an image (data) as sensor data.

[0037] Furthermore, for example, the external recognition sensor 25 may include an environmental sensor for detecting the environment for the vehicle 1. The environmental sensor is a sensor for detecting the environment such as weather, climate, brightness, etc., and may include various sensors such as a raindrop sensor, a fog sensor, a sunlight sensor, a snow sensor, and an illuminance sensor.

[0038] Furthermore, for example, the external recognition sensor 25 includes a microphone used to detect sounds around the vehicle 1 and the location of sound sources.

[0039] The interior sensor 26 includes various sensors for detecting information inside the vehicle, and supplies sensor data from each sensor to each unit of the vehicle control system 11. The types and number of the various sensors included in the interior sensor 26 are not particularly limited as long as they are of types and numbers that can be realistically installed in the vehicle 1.

[0040] For example, the interior sensor 26 may include one or more types of sensors selected from the group consisting of a camera, radar, a seating sensor, a steering wheel sensor, a microphone, and a biometric sensor. The camera included in the interior sensor 26 may be a camera using any of various imaging methods capable of measuring distances, such as a Time of Flight (ToF) camera, a stereo camera, a monocular camera, or an infrared camera. The camera included in the interior sensor 26 may also be a camera simply for acquiring images, regardless of distance measurement. The biometric sensor included in the interior sensor 26 may be provided, for example, on a seat, a steering wheel, or the like, and detect (sense) various types of biometric information of the user.

[0041] The vehicle sensor 27 includes various sensors for detecting the state of the vehicle 1, and supplies sensor data output by each sensor to each unit of the vehicle control system 11. The types and number of the various sensors included in the vehicle sensor 27 are not particularly limited as long as they are of types and numbers that can be realistically installed on the vehicle 1.

[0042] For example, the vehicle sensor 27 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU) that integrates these sensors. For example, the vehicle sensor 27 includes a steering angle sensor that detects the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor that detects the amount of accelerator pedal operation, and a brake sensor that detects the amount of brake pedal operation. For example, the vehicle sensor 27 includes a rotation sensor that detects the number of rotations of the engine or motor, an air pressure sensor that detects tire air pressure, a slip ratio sensor that detects tire slip ratio, and a wheel speed sensor that detects the rotation speed of the wheels. For example, the vehicle sensor 27 includes a battery sensor that detects the remaining battery charge and temperature, and an impact sensor that detects external impacts.

[0043] The storage unit 28 includes at least one of a non-volatile storage medium and a volatile storage medium, and stores data and programs. Examples of storage media that can be used include an electrically erasable programmable read-only memory (EEPROM), a random access memory (RAM), a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, and a magneto-optical storage device. The storage unit 28 stores various programs and data used by each component of the vehicle control system 11. For example, the storage unit 28 includes an event data recorder (EDR) and a data storage system for automated driving (DSSAD), and stores information about the vehicle 1 before and after an event such as an accident, and information acquired by the in-vehicle sensors 26.

[0044] The driving automation control unit 29 controls the driving automation function of the vehicle 1. For example, the driving automation control unit 29 includes an action planning unit 62 and an operation control unit 63.

[0045] The behavior planning unit 62 creates a behavior plan for the vehicle 1. For example, the behavior planning unit 62 creates the behavior plan by performing route planning and route tracking processing.

[0046] The route planning includes global path planning and local path planning. Global path planning includes a process of planning a rough route from a start to a goal. Local path planning, also called trajectory planning, includes a process of generating a trajectory that allows the vehicle 1 to proceed safely and smoothly in the vicinity of the vehicle 1 on the planned route, taking into account the motion characteristics of the vehicle 1.

[0047] Path following is a process of planning an operation for safely and accurately traveling along a route planned by a route plan within a planned time. The behavior planning unit 62 can, for example, calculate a target speed and a target angular velocity of the vehicle 1 based on the results of this path following process.

[0048] The operation control unit 63 controls the operation of the vehicle 1 in order to realize the action plan created by the action planning unit 62 .

[0049] For example, the operation control unit 63 controls a steering control unit 81, a brake control unit 82, and a drive control unit 83 included in a vehicle control unit 32 described later, to perform lateral vehicle motion control and longitudinal vehicle motion control so that the vehicle 1 travels along the trajectory calculated by the trajectory plan. For example, the operation control unit 63 performs control aimed at driver assistance functions such as collision avoidance or impact mitigation, following driving, vehicle speed maintenance driving, host vehicle collision warning, host vehicle lane departure warning, and driving automation such as driving without operation by the driver or a remote driver.

[0050] The DMS 30 performs processes such as authenticating the driver and recognizing the driver's state based on sensor data from the in-vehicle sensors 26 and input data input to the HMI 31 (described later). Examples of the driver's state to be recognized include physical condition, level of alertness, level of concentration, level of fatigue, line of sight, level of intoxication, driving operation, and posture.

[0051] The DMS 30 may be configured to perform authentication processing for users other than the driver and recognition processing for the status of the users. Furthermore, for example, the DMS 30 may be configured to perform recognition processing for the status inside the vehicle based on sensor data from the in-vehicle sensor 26. Examples of the status inside the vehicle that may be recognized include temperature, humidity, brightness, and odor.

[0052] The HMI 31 inputs various data and instructions and presents various data to the user.

[0053] The following provides an overview of data input via the HMI 31. The HMI 31 includes input devices for a person to input data. The HMI 31 generates input signals based on data, instructions, and the like input via the input devices and supplies the signals to each component of the vehicle control system 11. The HMI 31 includes, as input devices, controls such as a touch panel, buttons, switches, and levers. The HMI 31 may also include input devices that allow information to be input by voice, gestures, or other means other than manual operation. Furthermore, the HMI 31 may use, as input devices, externally connected devices such as a remote control device using infrared or radio waves, or a mobile or wearable device compatible with the operation of the vehicle control system 11.

[0054] The presentation of data by the HMI 31 will be briefly described. The HMI 31 generates visual information, auditory information, and tactile information for the user or the outside of the vehicle. The HMI 31 also performs output control, controlling the output, output content, output timing, output method, etc. of each piece of generated information. The HMI 31 generates and outputs, as visual information, information indicated by images or lights, such as an operation screen, a status display of the vehicle 1, a warning display, and a monitor image showing the situation around the vehicle 1. The HMI 31 also generates and outputs, as auditory information, information indicated by sounds, such as voice guidance, warning sounds, and warning messages. The HMI 31 also generates and outputs, as tactile information, information imparted to the user's sense of touch by, for example, force, vibration, movement, etc.

[0055] Examples of output devices that the HMI 31 uses to output visual information include a display device that displays an image on its own to present visual information and a projector device that projects an image to present visual information. The display device may be a device that displays visual information within the user's field of vision, such as a head-up display, a transmissive display, or a wearable device with an augmented reality (AR) function, in addition to a display device with a normal display. The HMI 31 may also use display devices included in a navigation system, an instrument panel, a camera monitoring system (CMS), an electronic mirror, a lamp, or the like provided in the vehicle 1 as output devices that output visual information.

[0056] As an output device for the HMI 31 to output auditory information, for example, an audio speaker, a headphone, or an earphone can be applied.

[0057] For example, a haptic element using haptic technology can be applied as an output device for outputting tactile information from the HMI 31. The haptic element is provided on a part that the user touches, such as a steering wheel or a seat.

[0058] The vehicle control unit 32 controls each part of the vehicle 1. The vehicle control unit 32 includes a steering control unit 81, a brake control unit 82, a drive control unit 83, a body system control unit 84, a light control unit 85, and a horn control unit 86.

[0059] The steering control unit 81 detects and controls the state of the steering system of the vehicle 1. The steering system includes, for example, a steering mechanism including a steering wheel, an electric power steering, etc. The steering control unit 81 includes, for example, a steering ECU that controls the steering system, an actuator that drives the steering system, etc.

[0060] The brake control unit 82 detects and controls the state of the brake system of the vehicle 1. The brake system includes, for example, a brake mechanism including a brake pedal, an antilock brake system (ABS), a regenerative brake mechanism, etc. The brake control unit 82 includes, for example, a brake ECU that controls the brake system, an actuator that drives the brake system, etc.

[0061] The drive control unit 83 detects and controls the state of the drive system of the vehicle 1. The drive system includes, for example, an accelerator pedal, a drive force generating device for generating drive force such as an internal combustion engine or a drive motor, and a drive force transmission mechanism for transmitting the drive force to the wheels. The drive control unit 83 includes, for example, a drive ECU for controlling the drive system, and an actuator for driving the drive system.

[0062] The body system control unit 84 detects and controls the states of the body system systems of the vehicle 1. The body system systems include, for example, a keyless entry system, a smart key system, a power window device, a power seat, an air conditioning system, an airbag, a seat belt, a shift lever, etc. The body system control unit 84 includes, for example, a body system ECU that controls the body system systems, an actuator that drives the body system systems, etc.

[0063] The light control unit 85 detects and controls the states of various lights of the vehicle 1. Examples of lights to be controlled include headlights, backlights, fog lights, turn signals, brake lights, projections, and bumper displays. The light control unit 85 includes a light ECU that controls the lights, an actuator that drives the lights, and the like.

[0064] The horn control unit 86 detects and controls the state of the car horn of the vehicle 1. The horn control unit 86 includes, for example, a horn ECU that controls the car horn, an actuator that drives the car horn, and the like.

[0065] Fig. 2 is a diagram showing an example of a sensing area by the camera 51, radar 52, LiDAR 53, ultrasonic sensor 54, etc. of the external recognition sensor 25 in Fig. 1. Note that Fig. 2 shows a schematic view of the vehicle 1 as seen from above, with the left end side being the front end (front) side of the vehicle 1 and the right end side being the rear end (rear) side of the vehicle 1.

[0066] Sensing area 101F and sensing area 101B show examples of sensing areas of the ultrasonic sensors 54. Sensing area 101F covers the periphery of the front end of the vehicle 1 with multiple ultrasonic sensors 54. Sensing area 101B covers the periphery of the rear end of the vehicle 1 with multiple ultrasonic sensors 54.

[0067] The sensing results in the sensing area 101F and the sensing area 101B are used, for example, for parking assistance for the vehicle 1.

[0068] Sensing area 102F to sensing area 102B show examples of sensing areas of a short-range or medium-range radar 52. Sensing area 102F covers a position farther in front of the vehicle 1 than sensing area 101F. Sensing area 102B covers a position farther behind the vehicle 1 than sensing area 101B. Sensing area 102L covers the periphery behind the left side of the vehicle 1. Sensing area 102R covers the periphery behind the right side of the vehicle 1.

[0069] The sensing results in sensing area 102F are used, for example, to detect vehicles, pedestrians, and the like that are present in front of the vehicle 1. The sensing results in sensing area 102B are used, for example, for a collision prevention function behind the vehicle 1. The sensing results in sensing area 102L and sensing area 102R are used, for example, to detect objects in blind spots on the sides of the vehicle 1.

[0070] Sensing areas 103F to 103B show examples of sensing areas sensed by camera 51. Sensing area 103F covers a position farther in front of vehicle 1 than sensing area 102F. Sensing area 103B covers a position farther in the rear of vehicle 1 than sensing area 102B. Sensing area 103L covers the periphery of the left side of vehicle 1. Sensing area 103R covers the periphery of the right side of vehicle 1.

[0071] The sensing results in the sensing area 103F can be used for, for example, recognition of traffic lights and traffic signs, lane departure prevention assistance systems, and automatic headlight control systems. The sensing results in the sensing area 103B can be used for, for example, parking assistance and surround view systems. The sensing results in the sensing areas 103L and 103R can be used for, for example, surround view systems.

[0072] Sensing area 104 shows an example of the sensing area of ​​LiDAR 53. Sensing area 104 covers a position farther ahead of vehicle 1 than sensing area 103F. On the other hand, sensing area 104 has a narrower range in the left-right direction than sensing area 103F.

[0073] The sensing results in the sensing area 104 are used to detect objects such as surrounding vehicles, for example.

[0074] A sensing area 105 shows an example of the sensing area of ​​the long-range radar 52. The sensing area 105 covers a position further ahead of the vehicle 1 than the sensing area 104. On the other hand, the sensing area 105 has a narrower range in the left-right direction than the sensing area 104.

[0075] The sensing results in the sensing area 105 are used for, for example, adaptive cruise control (ACC), emergency braking, collision avoidance, and the like.

[0076] The sensing areas of the cameras 51, radars 52, LiDARs 53, and ultrasonic sensors 54 included in the external recognition sensor 25 may have various configurations other than those shown in FIG. 2 . Specifically, the ultrasonic sensors 54 may also sense the sides of the vehicle 1, and the LiDAR 53 may sense the rear of the vehicle 1. The installation positions of the sensors are not limited to the examples described above. The number of each sensor may be one or more.

[0077] <First Configuration Example of Camera 51>

[0078] FIG. 3 is a block diagram showing a first example of the configuration of the camera 51 in FIG.

[0079] In FIG. 3, the camera 51 has a sensing unit 110 and a data processing unit 120, and the sensing unit 110 and the data processing unit 120 are housed in a single housing.

[0080] The sensing unit 110 has a lens unit 111 and an imaging unit 112, senses light, for example visible light in this case, and outputs an image (data) as sensor data.

[0081] The lens unit 111 focuses light incident on the lens unit 111 onto (the light receiving surface of) the imaging unit 112 .

[0082] The imaging unit 112 is configured with, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor or the like. The imaging unit 112 operates in accordance with a control signal from the image processing unit 121. The imaging unit 112 senses (photoelectrically converts) (receives) light from the lens unit 111 and captures an image to acquire (generate) an image (data) as sensor data. The imaging unit 112 outputs the image obtained by capturing to the data processing unit 120 (the image processing unit 121). The image output by the imaging unit 112 is also referred to as an original image.

[0083] The data processing unit 120 has an image processing unit 121, a volatile memory 122, a non-volatile memory 123, a control unit 124, and a communication interface unit 125, and performs various data processing using the original image output by the imaging unit 112 and other data.

[0084] The image processing unit 121 has an image conversion unit 121A, an image compression unit 121B, and an image recognition unit 121C, and performs various types of image processing using the original image output by the imaging unit 112 in accordance with a control signal from the control unit 124. The image processing unit 121 also stores various pieces of information necessary for the image processing in the volatile memory 122, and performs various types of image processing by referring to the various pieces of information stored in the volatile memory 122 (based on the various pieces of information).

[0085] For example, the image processing unit 121 controls the amount of data of the original image from the imaging unit 112 based on the amount of data other than images (including programs) stored in the volatile memory 122, and stores the image obtained as a result of the data amount control in the volatile memory 122. Furthermore, for example, the image processing unit 121 performs image recognition processing using the image stored in the volatile memory 122. The image obtained as a result of the data amount control and the results of the image recognition processing (image recognition processing results) in the image processing unit 121 are supplied to the control unit 124. Furthermore, for example, when the camera 51 is started, the image processing unit 121 loads the firmware and necessary settings (information) stored in the non-volatile memory 123, and supplies them to the volatile memory 122 for storage (expansion).

[0086] Additionally, the image processing unit 121 supplies a control signal to the image capturing unit 112 to control the exposure time of the image capturing unit 112 under the control of the control unit 124 .

[0087] In the image processing unit 121, the image conversion unit 121A performs distortion correction on the original image from the imaging unit 112 and converts the angle of view of the original image (or the image after processing by the image compression unit 121B). In the angle of view conversion for converting the angle of view of the original image, all or part of the original image is cut out, and the original image is converted into an image with an angle of view equal to or smaller than the angle of view of the original image.

[0088] In the image processing unit 121, the image compression unit 121B converts (compresses) the resolution of the original image (or the image after processing by the image conversion unit 121A) from the imaging unit 112. In the resolution conversion for converting the resolution of the original image, pixels of the original image are thinned out (and smoothed) as necessary, so that the original image is converted into an image with a resolution equal to or lower than the resolution of the original image.

[0089] The control of the data amount of the original image in the image processing unit 121 is performed by converting the angle of view in the image conversion unit 121A and converting the resolution in the image compression unit 121B. By converting the angle of view and resolution of the original image, the original image is converted into an image with a data amount equal to or less than that of the original image, i.e., an image with an angle of view equal to or less than that of the original image and a resolution equal to or less than that of the original image. Here, the image obtained by converting the angle of view and resolution of the original image is also referred to as a converted image. The converted image is an image obtained by converting the angle of view and resolution of the original image as sensor data, and is part of the sensor data. Therefore, the converted image (and even an image generated from the converted image) can be considered sensor data, just like the original image. Converting the angle of view includes converting the angle of view to that of the image before conversion (leaving the angle of view unchanged), as necessary. The same applies to converting the resolution.

[0090] The converted image is supplied from the image processing unit 121 to the volatile memory 122 as (one of) various information and stored therein. The image recognition unit 121C performs image recognition processing using the converted image stored in the volatile memory 122. In the image recognition processing, for example, objects appearing in the converted image, such as other vehicles such as bicycles and automobiles, people (pedestrians), obstacles, structures, roads, traffic lights, traffic signs, road markings, etc., are recognized (detected). In the image recognition processing, the position and type of objects can also be recognized. For example, in the image recognition processing, the type of object around the vehicle 1 appearing in the converted image and the area occupied by the object can be recognized by semantic segmentation or the like.

[0091] The volatile memory 122 is a storage unit that stores sensor data output by the sensing unit 110 and other data (including programs) other than the sensor data. For example, the volatile memory 122 stores various information supplied from the image processing unit 121. That is, the volatile memory 122 stores, as sensor data, a converted image obtained by the image processing unit 121 converting the original image (by controlling the amount of data of the original image), as well as firmware and various settings loaded by the image processing unit 121 from the non-volatile memory 123 as other data.

[0092] The nonvolatile memory 123 stores data that needs to be retained even when the power supply of the camera 51 is turned off. For example, the nonvolatile memory 123 stores firmware, camera parameters as various settings, and various parameters (libraries, etc.) for image recognition processing such as a learning model used in image recognition processing.

[0093] The control unit 124 supplies a control signal for controlling the image processing unit 121 to the image processing unit 121 .

[0094] The control unit 124 controls FOTA (FW Over The Air) and SOTA (Software Over The Air), etc., which update the FW and various settings stored in the nonvolatile memory 123 by OTA (Over The Air), etc. In other words, when the FW and various settings are updated by a version upgrade or the like and the updated FW and various settings are received by the communication interface unit 125 and supplied to the control unit 124, the control unit 124 updates the FW and various settings stored in the nonvolatile memory 123 by, for example, overwriting the nonvolatile memory 123 with the updated FW and various settings from the communication interface unit 125.

[0095] The control unit 124 is supplied with image processing results, i.e., converted images as sensor data, image recognition processing results, etc., from the image processing unit 121. Furthermore, other unit information such as vehicle information is supplied to the control unit 124 from the communication interface unit 125. The vehicle information is sensor data such as the speed (vehicle speed) of the vehicle 1 output by the vehicle sensor 27. The other unit information is information acquired by other units (blocks) other than the camera 51 that constitute the vehicle control system 11. The control unit 124 performs analysis processing of the vehicle 1 and the surrounding situation using the converted images, image recognition processing results, and other unit information supplied to the control unit 124, and supplies the results of the analysis processing (analysis processing results) to the communication interface unit 125.

[0096] For example, the control unit 124 performs self-position estimation processing, sensor fusion processing, detection processing, and recognition processing as the analysis processing.

[0097] In the self-position estimation process, the self-position of the vehicle 1 is estimated based on the transformed image, sensor data of the external recognition sensors 25 other than the camera 51 as other unit information, and a high-precision map stored in the map information storage unit 23. For example, in the self-position estimation process, a local map is generated based on the transformed image and sensor data of the external recognition sensors 25 other than the camera 51, and the local map is matched with the high-precision map, thereby estimating the self-position of the vehicle 1. The position of the vehicle 1 is based on, for example, the center of the rear wheel pair axle.

[0098] The local map is, for example, a three-dimensional high-precision map or an occupancy grid map created using a technology such as SLAM (Simultaneous Localization and Mapping). The three-dimensional high-precision map is, for example, the point cloud map described above. The occupancy grid map is a map in which the three-dimensional or two-dimensional space around the vehicle 1 is divided into grids of a predetermined size and the occupancy state of objects is indicated on a grid-by-grid basis. The occupancy state of objects is indicated, for example, by the presence or absence of an object and the probability of its existence. The local map is also used in the detection process and the recognition process as needed.

[0099] In the self-position estimation process, the self-position of the vehicle 1 may be estimated based on the position information acquired by the position information acquisition unit 24 and the sensor data from the vehicle sensor 27 as other unit information.

[0100] In the sensor fusion process, sensor fusion is performed to obtain information by combining multiple different types of sensor data, for example, the converted image and the sensor data supplied from the radar 52. Methods for combining different types of sensor data include compounding, integration, fusion, and association.

[0101] The detection process detects the situation outside the vehicle 1, and the recognition process recognizes the situation outside the vehicle 1. The detection process and the recognition process are performed based on (using) the converted image, sensor data from the external recognition sensor 25 other than the camera 51, information obtained by the self-position estimation process, information obtained by the sensor fusion process, etc.

[0102] The detection process and recognition process involve detecting and recognizing objects around the vehicle 1. The object detection process is, for example, a process of detecting the presence or absence, size, shape, position, movement, etc. of an object. The object recognition process is, for example, a process of recognizing attributes such as the type of object, or identifying a specific object. However, the detection process and the recognition process are not necessarily clearly separated, and may overlap.

[0103] For example, in the detection process, objects around the vehicle 1 are detected by performing clustering to classify a point cloud based on sensor data from the radar 52, the LiDAR 53, or the like into clusters of points. This allows the presence, size, shape, and position of objects around the vehicle 1 to be detected.

[0104] Furthermore, for example, in the detection process, tracking is performed to follow the movement of the clusters of points classified by clustering, thereby detecting the movement of objects around the vehicle 1. As a result, the speed and traveling direction (movement vector) of the objects around the vehicle 1 are detected.

[0105] The recognition process can recognize traffic rules around the vehicle 1 and other conditions around the vehicle 1 based on the map stored in the map information storage unit 23, the result of the self-location estimation process, the result of the image recognition process, the result of the detection of objects around the vehicle 1 in the detection process, vehicle information such as vehicle speed, etc. For example, the recognition process can recognize the position and state of traffic lights, the contents of traffic signs and road markings, the contents of traffic regulations, and lanes that can be traveled, etc. Furthermore, the recognition process can recognize the time required to reach objects such as intersections, pedestrians, and other vehicles that exist in the traveling direction, etc.

[0106] Additionally, the recognition process can recognize the environment around the vehicle 1, such as the weather, temperature, humidity, brightness, and road surface condition.

[0107] The communication interface unit 125 functions as an interface for exchanging data between the camera 51 and other units constituting the vehicle control system 11. For example, the communication interface unit 125 transmits the analysis processing results from the control unit 124 to other units as needed. Also, for example, the communication interface unit 125 receives other unit information transmitted from other units and supplies the information to the control unit 124.

[0108] The updated firmware stored in the nonvolatile memory 123 is acquired by the communication unit 22 and transmitted to the communication interface unit 125 as other unit information.

[0109] Generally, there is a trade-off between the angle of view and the resolution of an image. The same is true of the relationship between the resolution of an image and the processing speed of image processing using that image. For example, if a wide-angle lens is used as the lens unit 111 in the camera 51, an image with a wide angle of view (a large angle of view) can be captured, but the resolution of distant parts captured in the image will be low. Therefore, in image recognition processing using such an image, it may be difficult to recognize distant objects.

[0110] On the other hand, if the angle of view of the image captured by the camera 51 is made smaller (narrower), the resolution of the distant parts captured in the image can be maintained, and distant objects can be recognized in the image recognition process. However, in this case, the range outside the periphery of the optical axis of the camera 51 (in the direction perpendicular to the optical axis) is not captured in the image, making it difficult to recognize objects present in such a range. In other words, it becomes difficult to recognize objects over a wide range.

[0111] Therefore, a high-performance sensing unit capable of capturing images with a large (wide, high) angle of view and resolution can be used as the sensing unit 110. That is, a wide-angle lens capable of capturing images with a wide angle of view that capture a predetermined wide range can be used as the lens unit 111, and a multi-pixel image sensor capable of capturing images of distant objects up to a predetermined distance at a predetermined high resolution can be used as the imaging unit 112.

[0112] When a high-performance sensing unit 110 is employed, the image (original image) output by the sensing unit 110 will be an image with a large amount of data. Therefore, the volatile memory 122 needs to be a memory that can store such an image with a large amount of data.

[0113] The volatile memory 122 stores images output by the sensing unit 110 as well as data necessary for the operation of the camera 51 , such as firmware and various settings loaded from the non-volatile memory 123 .

[0114] For example, if a deep learning model trained through deep learning is adopted as the learning model for various settings, the volatile memory 122 currently requires a large capacity memory capable of storing a deep learning model, which requires a very large amount of calculations.

[0115] If a large-capacity memory is used as the volatile memory 122, the volatile memory 122 will become larger and consume more power. Furthermore, to accommodate the larger size and increased power consumption of the volatile memory 122, the housing of the camera 51 will become larger, which may make it difficult to install the camera 51 in a vehicle 1 that has limited space for installing the camera 51.

[0116] Incidentally, it is expected that the firmware (as data other than images) stored in the volatile memory 122 and the deep learning model (as a learning model) will be updated as needed. For example, a FOTA update technology has been developed for the firmware. Regarding deep learning models, active research is being conducted into reducing the size of deep learning models. As a result, new technologies are being developed that reduce memory usage and power consumption while maintaining or improving the recognition performance of image recognition processing using deep learning models.

[0117] The amount of data in the FW and learning model (deep learning model) may decrease due to updates. When the amount of data in the FW and learning model decreases, the storage area of ​​the volatile memory 122 corresponding to the decreased amount of data becomes unnecessary. Therefore, if the FW and learning model are updated in the future, after the update, a memory with a storage capacity reduced by the amount of data in the FW and learning model due to the update can be used as the volatile memory 122. In this case, it is possible to suppress an increase in the size of the volatile memory 122 and an increase in power consumption.

[0118] Therefore, in anticipation of a reduction in the amount of data in the FW and learning model due to future updates, a memory with a small storage capacity can be adopted as the volatile memory 122. However, in this case, it becomes difficult to store all of the original images with a large angle of view and resolution output by the high-performance sensing unit 110 until the FW and learning model are updated to FW and learning models with reduced data amounts.

[0119] Therefore, in the camera 51, the image processing unit 121 controls the data amount of the original image based on the data amount of other data other than the image, such as the FW and learning model stored in the volatile memory 122. As a result, the original image is converted into a converted image with a data amount that can be stored in the volatile memory 122 in which other data is stored. That is, in the image processing unit 121, to control the data amount of the original image, the image conversion unit 121A converts the angle of view and / or the image compression unit 121B converts the resolution. As a result, a converted image with an angle of view equal to or smaller than the angle of view of the original image and a resolution equal to or smaller than the resolution of the original image is generated, i.e., a converted image with a data amount equal to or smaller than the data amount of the original image.

[0120] In this case, until other data such as the FW and the learning model are updated, a converted image with a smaller amount of data than the original image, i.e., a converted image with a smaller (narrower, lower) angle of view and / or resolution than the original image, is stored in the volatile memory 122. As a result, in the image recognition process, it may become difficult to recognize distant objects and / or objects over a wide range.

[0121] However, if other data such as the FW and learning model are updated in the future and the amount of other data decreases, images with a larger angle of view and resolution can be stored in the volatile memory 122 by the amount of the decreased amount of data, thereby improving the functionality of the camera 51. For example, it becomes possible to perform high-performance image recognition processing such as image recognition processing targeting a wider range (a range in a direction perpendicular to the optical axis) or image recognition processing targeting a greater distance (in the direction of the optical axis).

[0122] As described above, the camera 51 employs a volatile memory 122 with a small storage capacity in anticipation of future updates to other data such as firmware and learning models, thereby preventing the volatile memory 122 from becoming larger and increasing power consumption.

[0123] Furthermore, the camera 51 employs a high-performance sensing unit 110 that controls the amount of data of the original image based on the amount of data of other data, such as the firmware and learning model, enabling future enhancements to the functionality of the camera 51 without changing the hardware. In other words, when other data, such as the firmware and learning model, is updated and the amount of data of the other data is reduced, an image with a larger angle of view and resolution (ideally, the entire original image) can be stored in the volatile memory 122 by the amount of data reduction. As a result, high-performance image recognition processing, such as image recognition processing targeting a wider range or image recognition processing targeting a longer distance, becomes possible.

[0124] In addition, in the camera 51, the original image is converted into a converted image with a data amount that can be stored in the volatile memory 122 in which other data is stored, based on the data amount of other data other than images, such as firmware and learning models stored in the volatile memory 122. That is, the original image is converted into a converted image with a data amount that can be stored in the available space of the volatile memory 122 when the other data is stored in the volatile memory 122. Therefore, the volatile memory 122 can be a memory with a storage capacity that can store other data, such as the current firmware and learning models, and images that meet or exceed the minimum specifications required for image recognition processing. Furthermore, for example, if the size of the camera 51 is limited, the volatile memory 122 can be a memory with a storage capacity equal to or less than the capacity that can be implemented in a camera 51 of that limited size.

[0125] <Storage state of volatile memory 122>

[0126] FIG. 4 is a diagram illustrating an example of the storage state of the volatile memory 122. As shown in FIG.

[0127] In addition, other data stored in the volatile memory 122 besides the converted image include FW, image recognition processing libraries such as learning models (or dictionaries), various parameters, etc., but for simplicity of explanation, the other data will be referred to as FW and libraries below.

[0128] FIG. 4A shows an example of the storage state of the volatile memory 122 before the FW and library are updated, and FIG. 4B shows an example of the storage state of the volatile memory 122 after the FW and library are (futurely) updated.

[0129] An image storage area for storing the converted image is secured in the volatile memory 122 within the available capacity of the volatile memory 122 when the FW and library are stored, and the converted image is stored in the image storage area. The image storage area also stores images necessary for the image recognition process performed by the image recognition unit 121C, along with the converted image. For example, when the image recognition process is performed using a multi-layer image as a Gaussian pyramid (described later) generated from the converted image, the image storage area stores the converted image and the Gaussian pyramid generated from the converted image.

[0130] Before the FW and library are updated, as shown in A of FIG. 4, the data volume of the pre-update FW and library is large (large), so the image storage area is small (small storage area). The small image storage area that can be secured before the FW and library are updated is also referred to as the pre-update image storage area. The converted image stored in the pre-update image storage area is also referred to as the converted image PA. Since the converted image PA is stored in the small pre-update image storage area, it is an image with a small amount of data. When image recognition processing is performed using a Gaussian pyramid generated from the converted image PA, the image storage area stores the converted image PA and the Gaussian pyramid generated from the converted image PA. The Gaussian pyramid is not shown in A of FIG. 4.

[0131] In the future, if the (pre-update) FW and library are upgraded to FW and library with reduced data volume, the data volume of the updated FW and library will decrease, as shown in Figure 4B. Therefore, the image storage area can be enlarged (made larger) by the amount of data reduction. Here, the large image storage area that can be secured after the FW and library are updated is also referred to as the updated image storage area. The converted image stored in the updated image storage area is also referred to as the converted image PB. Since the converted image PB is stored in the large updated image storage area, it is an image with a large amount of data. When image recognition processing is performed using a Gaussian pyramid generated from the converted image PB, the image storage area stores the converted image PB and a Gaussian pyramid generated from the converted image PB. The Gaussian pyramid is not shown in Figure 4B.

[0132] In FIG. 4A, the volatile memory 122 has a storage capacity large enough to store the converted image PA (second image data) while the pre-update FW and library (first other data) are stored in the volatile memory 122, but not enough to store the converted image PB (third image data). When the image recognition process is performed using a Gaussian pyramid, the volatile memory 122 has a storage capacity large enough to store the converted image PA and the Gaussian pyramid (second image data) generated from the converted image PA while the pre-update FW and library (first other data) are stored in the volatile memory 122, but not enough to store the converted image PB and the Gaussian pyramid (third image data) generated from the converted image PB. Hereinafter, for simplicity's sake, the description of the storage of Gaussian pyramids in the image storage area will be omitted as appropriate.

[0133] <Conversion of original image to converted image>

[0134] FIG. 5 is a diagram illustrating an example of conversion of an original image into a converted image by the image processing unit 121. In FIG.

[0135] When the first other data is stored in the volatile memory 122, the image processing unit 121 converts the first image data having a first angle of view and a first resolution output by the sensing unit 110 into second image data having a second angle of view smaller than the first angle of view and / or a second resolution smaller than the first resolution. Furthermore, when second other data having a smaller data amount than the first other data after updating the first other data is stored in the volatile memory 122, the image processing unit 121 converts the first image data into third image data having a third angle of view larger than the second angle of view and / or a third resolution larger than the second resolution.

[0136] That is, when the FW and library (first other data) before the update are stored in the volatile memory 122, the image processing unit 121 converts the original image (first image data having a first angle of view and a first resolution) into a converted image PA (second image data) having a smaller angle of view (second angle of view) and / or a smaller resolution (second resolution) than the resolution of the original image (first resolution). Furthermore, when the FW and library (second other data) after updating the FW and library (first other data) and having a smaller data volume than the FW and library (first other data) before the update are stored in the volatile memory 122, the image processing unit 121 converts the original image (first image data) into a converted image PB (third image data) having a larger angle of view (third angle of view) than the angle of view (second angle of view) of the converted image PA and / or a larger resolution (third resolution) than the resolution of the converted image PA (second resolution).

[0137] Specifically, before the firmware and library are updated, for example, the image processing unit 121 converts the original image into an angle-of-view reduced image with a smaller angle of view than the original image but with the same resolution as the original image by converting the angle of view of the original image, thereby generating the angle-of-view reduced image as the converted image PA. Alternatively, the image processing unit 121 converts the original image into a resolution-reduced image with a smaller resolution than the original image but with the same angle of view as the original image by converting the resolution of the original image, thereby generating the resolution-reduced image as the converted image PA.

[0138] The reduced-angle image and reduced-resolution image serving as the post-conversion image PA are images with a data amount that can be stored in the pre-update image storage area.

[0139] Before the firmware and library are updated, both the angle of view and the resolution are converted, and an image having a smaller angle of view and a smaller resolution than the original image can be generated as the converted image PA.

[0140] After updating the FW and library, for example, the image processing unit 121 converts the angle of view and / or the resolution of the original image as needed to generate a converted image PB whose angle of view and resolution are equal to or less than those of the original image and whose data amount can be stored in the updated image storage area.

[0141] The post-update image storage area is a larger storage area than the pre-update image storage area. Therefore, the post-conversion image PB that can be stored in the post-update image storage area is an image with a larger amount of data than the post-conversion image PA that can be stored in the pre-update image storage area. For example, the post-conversion image PB is a wide-angle / high-resolution image with a larger angle of view and / or resolution than the post-conversion image PA.

[0142] In addition, if updating the FW and library allows for a storage area larger than the storage area capable of storing the original image to be secured as the updated image storage area, the original image will be used as the converted image PB (treated as the converted image PB) as is.

[0143] <Processing of image processing unit 121>

[0144] FIG. 6 is a flowchart illustrating an example of processing by the image processing unit 121.

[0145] In step S11, the image processing unit 121 calculates the amount of data (required storage capacity required to store other data) of FW and library as data other than the converted image to be stored in the volatile memory 122, and processing proceeds to step S12.

[0146] In step S12, the image processing unit 121 generates a converted image by controlling the amount of data of the original image (reducing the angle of view and / or resolution of the original image) based on the amount of data (required storage capacity) of the FW and library as other data, and the processing proceeds to step S13.

[0147] In step S13, the image processing unit 121 secures an image storage area within the free space of the volatile memory 122, in which the FW and library as other data are stored. Then, the image processing unit 121 stores the converted image obtained by controlling the data amount of the original image in the image storage area of ​​the volatile memory 122, and the process proceeds from step S13 to step S14.

[0148] In step S14, the image processing unit 121 performs image recognition processing using the converted image stored in the volatile memory 122, and supplies the processing result of the image recognition processing to the control unit 124.

[0149] <Storage of converted image in volatile memory 122>

[0150] FIG. 7 is a diagram showing an example of storing a converted image in the volatile memory 122. As shown in FIG.

[0151] The image processing unit 121 uses the converted image obtained from the original image as a first-layer image and generates, for example, a so-called Gaussian pyramid, which is made up of multiple layers of images that have the same angle of view as the first-layer image but different image sizes (resolutions). In Fig. 7, a Gaussian pyramid is generated that is made up of seven layers of images, from the first layer image to the seventh layer image, whose image sizes are gradually reduced from the first layer image.

[0152] The image processing unit 121 stores the Gaussian pyramid (the images of the multiple layers that make up the pyramid) in an image storage area.

[0153] As described above, by storing a Gaussian pyramid in an image storage area and performing image recognition processing using the Gaussian pyramid, it is possible to recognize objects that appear in various sizes in the original image.

[0154] For example, even if the object is the same, if the object is located close to the camera 51, the object will appear large in the original image, and if the object is located far from the camera 51, the object will appear small in the original image. As will be described later, an object that appears large in the original image is recognized by image recognition processing using an image with a small image size, and an object that appears small in the original image is recognized by image recognition processing using an image with a large image size.

[0155] The amount of data (field angle and resolution) of the converted image that becomes the first layer image, the number of layers in the Gaussian pyramid, and the degree to which the image size is gradually reduced in the Gaussian pyramid (hereinafter also referred to as the amount of image size change) are set so that multiple layers of images that make up the Gaussian pyramid can be stored in the image storage area.

[0156] For example, if the number of layers of the Gaussian pyramid and the amount of image size change are each set to a predetermined value in order to maintain the performance of the image recognition process, the amount of data (field angle and resolution) of the converted image that becomes the first layer image is set so that the Gaussian pyramid (the images of the multiple layers that make up the pyramid) with such number of layers and amount of image size change can be stored in the image storage area.

[0157] Furthermore, for example, when the firmware and library are updated and the amount of data in the firmware and library is reduced, an image storage area capable of storing an image exceeding the amount of data in the original image is secured, the number of layers and the amount of image size change can be set so that a Gaussian pyramid can be stored in the image storage area. For example, the larger the image storage area (the larger the capacity), the larger the number of layers can be set.

[0158] <Image recognition processing>

[0159] FIG. 8 is a diagram illustrating an example of image recognition processing performed before the firmware and library are updated.

[0160] For example, the camera 51 is a front camera that captures an image in front of the vehicle 1, and the sensing unit 110 captures an original image showing the front of the vehicle 1.

[0161] The original image (upper left of Figure 8) shows a total of five pedestrians P1 to P5: pedestrian P1 located on the left side in front of vehicle 1, and four pedestrians P2, P3, P4, and P5 located at different distances almost directly in front of vehicle 1. Pedestrian P2 is located the farthest, and pedestrian P3 is located the second farthest. Pedestrian P4 is located the third farthest, and pedestrian P4 is located the closest. Pedestrian P1 is located at a distance from camera 51 similar to that of pedestrian P3. Therefore, in the original image, pedestrian P5 appears the largest, and pedestrian P4 appears the second largest. Pedestrians P1 and P3 appear the third largest, and pedestrian P2 appears the smallest.

[0162] Before the firmware and library are updated, the amount of data in the firmware and library is large, so the pre-update image storage area (top right in FIG. 8) secured in the volatile memory 122 is a small storage area.

[0163] For this reason, the image processing unit 121 reduces the resolution of the original image by, for example, converting the resolution of the original image, and generates a reduced-resolution image ( FIG. 5 ) that has the same angle of view as the original image but a lower resolution as the converted image PA ( FIG. 4 ). Alternatively, the image processing unit 121 cuts out a part of the original image by, for example, converting the angle of view of the original image, and generates a reduced-angle of view image ( FIG. 5 ) that has the same resolution as the original image but a lower angle of view as the converted image PA ( FIG. 4 ).

[0164] The reduced-resolution image (lower left in Figure 8) as the converted image PA includes all of the pedestrians P1 to P5 that appear in the original image. However, because the resolution (image size) of the reduced-resolution image is reduced, the pedestrian P2, who appears farthest, appears very small and unclear.

[0165] On the other hand, in the reduced-angle image (lower right of FIG. 8 ) serving as the converted image PA, pedestrians P1 to P4, which are closest to the optical axis of camera 51 and appear almost directly in front of vehicle 1, including pedestrian P2, who is the furthest away, are captured at the same size and resolution as in the original image. However, in the reduced-angle image, a portion of the original image is cropped out, so pedestrian P1, who is located on the left side in front of vehicle 1 and is far off the optical axis of camera 51, is not captured.

[0166] After generating the transformed image PA, the image processing unit 121 uses the transformed image PA as a first layer image to generate a Gaussian pyramid composed of images in multiple layers, each having the same angle of view as the first layer image but with progressively smaller image sizes, and stores the generated Gaussian pyramid in the pre-update image storage area. In Fig. 8, a Gaussian pyramid composed of four layers of images, the first to fourth layer images, is generated and stored in the pre-update image storage area.

[0167] In the image recognition process, the image processing unit 121 recognizes a pedestrian (person) as a recognition target by, for example, detecting whether a pedestrian is captured within a detection frame while moving a detection frame of a predetermined size over an image stored in the image storage area. In this image recognition process, if the pedestrian's entire body is captured so as to occupy a predetermined proportion or more within the detection frame, the pedestrian is recognized as being present at the position of the detection frame. In other words, in the image recognition process, if the pedestrian is captured at an appropriate size in the image, the pedestrian captured at that appropriate size is recognized.

[0168] For example, if a reduced-resolution image (lower left in Figure 8) is generated as the converted image PA, and a Gaussian pyramid is generated using this reduced-resolution image as the first layer image, consisting of first to fourth layer images with the same angle of view as the first layer image but with progressively smaller image sizes, then in the image recognition process, pedestrians are recognized using this Gaussian pyramid as follows:

[0169] That is, in this case, the reduced-resolution image serving as the first layer image of the Gaussian pyramid shows the second-furthest pedestrians P1 and P3 at an appropriate size, so pedestrians P1 and P3 are recognized from the first layer image. The second layer image shows the third-furthest pedestrian P4 at an appropriate size, so pedestrian P4 is recognized from the second layer image. The third layer image shows the closest pedestrian P5 at an appropriate size, so pedestrian P5 is recognized from the third layer image. The fourth layer image does not show any pedestrians at an appropriate size, so no pedestrians are recognized from the fourth layer image.

[0170] Furthermore, in a Gaussian pyramid generated using a reduced-resolution image as the first layer image, there is no image in which the pedestrian P2, located farthest, is captured at an appropriate size. That is, in the reduced-resolution image, the resolution is reduced, so the pedestrian P2, located farthest, is captured at a size smaller than the appropriate size. And in a Gaussian pyramid generated using such a reduced-resolution image as the first layer image, there is no image in which the pedestrian P2 is captured at an appropriate size. Therefore, when a reduced-resolution image is generated as the converted image PA, the pedestrian P2 is not recognized.

[0171] On the other hand, if a reduced-angle image (lower right of Figure 8) is generated as the transformed image PA, and a Gaussian pyramid is generated using this reduced-angle image as the first layer image, consisting of first to fourth layer images with the same angle of view as the first layer image but with progressively smaller image sizes, then in the image recognition process, pedestrians are recognized using this Gaussian pyramid as follows.

[0172] That is, the reduced-angle image serving as the first layer image of the Gaussian pyramid shows the farthest pedestrian P2 at an appropriate size, allowing the pedestrian P2 to be recognized from the first layer image. The second layer image shows the second-farthest pedestrian P3 at an appropriate size, allowing the pedestrian P3 to be recognized from the second layer image. The third layer image shows the third-farthest pedestrian P4 at an appropriate size, allowing the pedestrian P4 to be recognized from the third layer image. The fourth layer image shows the closest pedestrian P5 at an appropriate size, allowing the pedestrian P5 to be recognized from the fourth layer image.

[0173] Furthermore, in a Gaussian pyramid generated with the angle-of-view reduced image as the first layer image, there is no image of pedestrian P1, who is located on the left side in front of vehicle 1 in the original image. That is, in the angle-of-view reduced image, a portion of the original image is cut out, so that pedestrian P1, who is located far to the left of the optical axis of camera 51, is not captured. And, in a Gaussian pyramid generated with such an angle-of-view reduced image as the first layer image, there is no image of pedestrian P1. Therefore, when the angle-of-view reduced image is generated as the converted image PA, pedestrian P1 is not recognized.

[0174] As described above, before the firmware and library are updated, the image recognition process is performed using an image with a smaller angle of view and / or resolution than the original image, which may result in a decrease in the performance of the image recognition process. For example, a pedestrian P2 located far away and / or a pedestrian P1 located to the side of the vehicle 1 may not be recognized.

[0175] Before the FW and library are updated, either a reduced resolution image or a reduced angle of view image is generated as the converted image PA, and based on the moving speed (vehicle speed) of the vehicle 1, either a reduced resolution image or a reduced angle of view image can be selectively generated as the converted image PA.

[0176] For example, when the vehicle 1 is traveling at high speed, that is, when it is moving at a predetermined speed or above, a pedestrian P1 on the side may not be recognized, but a reduced-angle-of-view image in which a pedestrian P2 in the distance can be recognized can be generated as the converted image PA. This is because when the vehicle 1 is traveling at high speed, for example, when traveling on a highway, it is effective for automated driving to recognize objects that are at a certain distance from the vehicle 1.

[0177] On the other hand, for example, when the vehicle 1 is traveling at a low speed, that is, when it is moving at a speed less than a predetermined speed, a pedestrian P2 in the distance may not be recognized, but a reduced-resolution image in which a pedestrian P1 on the side can be recognized can be generated as the converted image PA. This is because when the vehicle 1 is traveling at a low speed, for example, when attempting to park, recognizing objects around the vehicle 1 is effective for parking assistance.

[0178] Additionally, when the vehicle 1 is traveling at a first speed or less, a reduced-resolution image can be generated as the converted image PA. Furthermore, when the vehicle 1 is traveling at a second speed or more that is faster than the first speed, a reduced-angle image can be generated as the converted image PA. Furthermore, when the vehicle 1 is traveling at a speed faster than the first speed but less than the second speed, the converted image PA can be generated based on the vehicle speed of the vehicle 1. That is, for example, the faster the vehicle speed of the vehicle 1, the more likely it is that an image with a higher resolution and a smaller angle of view than the reduced-resolution image (an image with a resolution and angle of view close to those of the reduced-angle image) can be generated as the converted image PA.

[0179] FIG. 9 is a diagram illustrating an example of image recognition processing that will be performed after the firmware and library are updated in the future.

[0180] For example, as in the case of FIG. 8, the camera 51 is a front camera that captures an image in front of the vehicle 1, and the sensing unit 110 captures an original image showing the front of the vehicle 1.

[0181] In FIG. 9, as in the case of FIG. 8, five pedestrians P1 to P5 are captured in the original image.

[0182] When the FW and library are updated to FW and library with reduced data volume, the post-update image storage area secured in the volatile memory 122 becomes a storage area larger than the pre-update image storage area in FIG.

[0183] Here, for simplicity, the updated image memory area is assumed to be a memory area capable of storing a Gaussian pyramid consisting of the first to fifth layer images, which is generated as the first layer image, with the original image being treated as the transformed image PB (Figure 4).

[0184] In this case, the image processing unit 121 generates the converted image PB by using the original image as it is. In this case, the converted image PB shows the pedestrians P1 to P5 in the same manner as in the original image.

[0185] The image processing unit 121 uses the converted image PB as the first layer image to generate a Gaussian pyramid made up of images in multiple layers, each having the same angle of view as the first layer image but with image sizes that are gradually reduced, and stores the generated Gaussian pyramid in the updated image storage area. In Fig. 9, a Gaussian pyramid made up of five layers of images, from the first layer image to the fifth layer image, is generated and stored in the updated image storage area.

[0186] Thereafter, the image processing unit 121 performs image recognition processing using a detection frame, as described in Figure 8, using, for example, a Gaussian pyramid in which the converted image PB stored in the updated image storage area is used as the first layer image.

[0187] The transformed image PB, serving as the first layer image, has the same angle of view and resolution as the original image. Therefore, the transformed image PB, serving as the first layer image, clearly shows pedestrians P1 to P5. Furthermore, the pedestrians P1 to P5 are also shown in the images of the other layers constituting the Gaussian pyramid, in this case, the second to fifth layer images. Therefore, in image recognition processing using a Gaussian pyramid with the transformed image PB as the first layer image, all of the pedestrians P1 to P5 are recognized as follows:

[0188] That is, in the transformed image PB, which is the original image serving as the first layer image of the Gaussian pyramid, the farthest pedestrian P2 is captured at an appropriate size so that it fits within the detection frame at a predetermined ratio, and therefore pedestrian P2 is recognized from the first layer image. In the second layer image, the second-farthest pedestrians P1 and P3 are captured at an appropriate size, and therefore pedestrians P1 and P3 are recognized from the second layer image. In the third layer image, the third-farthest pedestrian P4 is captured at an appropriate size, and therefore pedestrian P4 is recognized from the third layer image. In the fourth layer image, the closest pedestrian P5 is captured at an appropriate size, and therefore pedestrian P5 is recognized from the fourth layer image.

[0189] As described above, when the FW and library are updated to FW and library with reduced data volume, and an image with a larger angle of view and / or resolution than before the FW and library update, for example, the original image itself, is stored in the volatile memory 122 (in its image storage area) as a converted image, the performance of the image recognition process performed using such a converted image (a Gaussian pyramid with this as the first layer image) is improved compared to before the FW and library update.

[0190] That is, after updating the FW and library, pedestrians P3 to P5 can be recognized in the same way as before the update. Furthermore, pedestrian P2 located at a distance, who could not be recognized by the image recognition process performed using the reduced-resolution image (Gaussian pyramid with the reduced-resolution image as the first layer image) before the FW and library update, can be recognized. Furthermore, pedestrian P1 located to the side, who could not be recognized by the image recognition process performed using the reduced-angle-of-view image (Gaussian pyramid with the reduced-angle-of-view image as the first layer image) before the FW and library update, can be recognized.

[0191] In the image recognition process, the forward distance (range) at which objects such as pedestrians can be recognized is determined by the resolution of the transformed image (which becomes the first layer image), and the lateral distance (range) is determined by the angle of view of the transformed image.

[0192] As the FW and library are updated as needed to reduce the amount of data, the image storage area for storing the converted image is expanded by the amount of data reduction in the FW and library. This allows the original image to be converted into a converted image with a larger angle of view and / or resolution than before the FW and library were updated.

[0193] For example, when the original image is converted into a reduced-angle image having a larger angle of view than before the FW and library are updated, the lateral distance at which an object can be recognized in the image recognition process is increased compared to before the FW and library are updated. In this case, for example, while the vehicle 1 is traveling at high speed, it is possible to detect a pedestrian or the like jumping out from the side more easily than before the FW and library are updated.

[0194] Furthermore, for example, when an original image is converted into a reduced-resolution image as a converted image having a higher resolution than before the FW and library are updated, the forward distance at which objects can be recognized in the image recognition process is extended compared to before the FW and library are updated. In this case, for example, while the vehicle 1 is traveling at a low speed, it is possible to recognize pedestrians and the like at a greater distance than before the FW and library are updated.

[0195] 8 and 9 , an example of image recognition processing has been described assuming that the camera 51 is a front camera capturing an image ahead of the vehicle 1. Alternatively, for example, if the camera 51 is a camera constituting a surround view system, before the firmware and library are updated, the image processing unit 121 can generate a reduced-resolution image as the converted image PA by reducing the resolution through resolution conversion and prioritizing the angle of view. In this case, a certain range is secured as the area around the vehicle 1 in which objects can be recognized. Therefore, for example, during automatic parking at low speeds, it is possible to detect a moving object such as a pedestrian jumping out from the side or to detect a parking space when the moving object or parking space is not very close to the vehicle 1.

[0196] Furthermore, if the firmware and library are updated in the future, reducing the amount of data in the firmware and library and expanding the image storage area by the amount of the data reduction, converted images with higher resolution than those before the firmware and library update can be generated. In this case, it becomes possible to detect moving objects at a greater distance than before the firmware and library update, allowing the vehicle 1 to move faster during automatic parking. Furthermore, if the camera 51 is mounted to capture an area including the area ahead of the vehicle 1, the camera 51 can be used as both a surround view system camera and a front camera. Furthermore, the camera 51 can be used as a replacement for the front camera in the event of a malfunction, contributing to a fail-safe system.

[0197] <Relationship between FW and library updates and the forward and lateral distances at which objects can be recognized by image recognition processing>

[0198] FIG. 10 is a diagram showing an example of the relationship between FW and library updates and the forward and lateral distances at which an object can be recognized by image recognition processing.

[0199] 10, the horizontal axis represents the lateral distance (range) at which an object can be recognized by image recognition processing, and the vertical axis represents the forward distance. Diagonal line segments indicate changes in the forward and lateral distances at which an object can be recognized by image recognition processing.

[0200] 10 , the firmware and library are updated from version 1.0 to version 2.0, 3.0, 4.0, and finally to version 5.0 (software (SW) updated). The higher the version, the smaller the amount of firmware and library data, and the amount of firmware and library data for version 5.0 has been reduced to the data amount that can be stored in the volatile memory 122 when the original image is treated as a converted image (Gaussian pyramid with this as the first layer image).

[0201] When the firmware and library are updated to version 5.0, the image processing unit 121 stores the original image as a converted image in the volatile memory 122, and performs image recognition processing using the converted image. In this case, the converted image stored in the volatile memory 122 has the largest angle of view and resolution among the images used in the image recognition processing, and therefore the lateral distance and forward distance that can be recognized in the image recognition processing are the largest (longest).

[0202] The lateral distance recognizable in the image recognition process is the distance in the direction perpendicular to the optical axis of the camera 51 that can be recognized (within the range) in the image recognition process, and becomes longer as the angle of view of the transformed image (each image of the Gaussian pyramid with the transformed image as the first layer image) becomes larger. The forward distance recognizable in the image recognition process is the distance in the direction of the optical axis of the camera 51 that can be recognized in the image recognition process, and becomes longer as the resolution of the transformed image becomes higher.

[0203] Before the firmware and library were updated to version 5.0, the data volume of the firmware and library had not been reduced to a level that allowed the original image to be stored as a converted image in the volatile memory 122. Therefore, the image processing unit 121 converts the resolution and / or the angle of view of the original image, and stores the converted image in the volatile memory 122, with a resolution and / or angle of view smaller than that of the original image.

[0204] Therefore, since the transformed image has at least one of a field of view and a resolution smaller than that of the original image, the lateral distance and / or forward distance that can be recognized in the image recognition process becomes smaller (shorter) due to the smaller field of view and / or resolution.

[0205] The lower the version of the FW and library, the larger the data volume, and therefore the lower the version of the FW and library, the smaller the data volume of the converted image that can be stored in the volatile memory 122, and therefore the angle of view and / or resolution.

[0206] For example, if the recognizable lateral distance in the image recognition process is to be the same as when the FW and library are version 5.0, the angle of view of the converted image must be maintained at the same angle as the original image. Therefore, the lower the FW and library versions, the lower the resolution of the converted image and the less data the converted image must be. Thus, the lower the FW and library versions, the lower the resolution of the converted image, and the smaller the recognizable forward distance in the image recognition process.

[0207] On the other hand, for example, if the recognizable forward distance in the image recognition process is to be the same as when the FW and library are version 5.0, the resolution of the converted image must be maintained at the same resolution as the original image. Therefore, the lower the FW and library versions, the smaller the angle of view of the converted image must be, and the less data the converted image must be. Thus, the lower the FW and library versions, the smaller the angle of view of the converted image, and the smaller the lateral distance that can be recognized in the image recognition process.

[0208] As described above, when attempting to achieve a certain distance as a recognizable lateral distance in image recognition processing, the lower the FW and library version, the smaller the recognizable forward distance in image recognition processing. Similarly, when attempting to achieve a certain distance as a recognizable forward distance in image recognition processing, the lower the FW and library version, the smaller the recognizable lateral distance in image recognition processing.

[0209] Before updating to version 5.0, which allows the original image to be stored in volatile memory 122 as a converted image, it was necessary to convert the resolution and / or angle of view of the original image, and create an image with a smaller resolution and / or angle of view than the original image as the converted image.

[0210] For the camera 51 mounted on the vehicle 1, the image processing unit 121 can reduce the resolution and / or the angle of view of the converted image based on the speed of the vehicle 1. In other words, the image processing unit 121 can control the resolution and / or the angle of view of the converted image obtained by converting the resolution and / or the angle of view of the original image based on the speed of the vehicle 1.

[0211] For example, the faster the vehicle speed, the higher the resolution and the smaller the angle of view of the converted image that can be generated (toward the upper left of the diagonal line segment), and the slower the vehicle speed, the lower the resolution and the larger the angle of view of the converted image that can be generated (toward the lower right of the diagonal line segment).

[0212] By doing so, it is possible to perform high-performance image recognition processing according to the vehicle speed. For example, the faster the vehicle speed, the more distant the image recognition processing (in the optical axis direction) becomes possible, and the slower the vehicle speed, the more widely the image recognition processing (in the direction perpendicular to the optical axis) becomes possible.

[0213] <Processing for updating data stored in nonvolatile memory 123>

[0214] FIG. 11 is a flowchart illustrating an example of a process performed by the vehicle control system 11 to update data (other data) stored in the nonvolatile memory 123 of the camera 51.

[0215] For example, FIG. 11 is a flowchart illustrating an example of a process for updating software such as firmware stored in the nonvolatile memory 123.

[0216] For example, when updated software (hereinafter also referred to as "update software") is prepared (stored) on a server (not shown) on the Internet or the like, in step S21, the communication unit 22 (FIG. 1) downloads the update software from the server via wireless communication. The communication unit 22 then stores the update software in the storage unit 28 (FIG. 1), and the process proceeds from step S21 to step S22. The download of the update software can be performed while the vehicle 1 is traveling and the camera 15 is operating normally, or can be performed while the vehicle 1 is parked (stopped).

[0217] In step S22, the control unit 124 (FIG. 3) waits until the vehicle 1 is stopped (parked) and the ignition (power) is turned off, and then installs the update software, and the process proceeds to step S23.

[0218] When the engine of the vehicle 1 is running (when the ignition is on), the power supply of the camera 51 is on (power is supplied to each component of the camera 51), and the camera 51 operates normally. Therefore, the control unit 124 waits until the ignition is turned off before installing the update software. The camera 51 is configured to be able to perform predetermined minimum necessary operations even when the ignition is turned off.

[0219] When installing the update software, the control unit 124 reads the update software stored in the storage unit 28 via the communication interface unit 125. Then, the control unit 124 updates the software stored in the non-volatile memory 123 with the update software and activates the update software.

[0220] In step S23, the control unit 124 restarts with the update software stored in the non-volatile memory 123, i.e., it loads the update software stored in the non-volatile memory 123 into the volatile memory 122 and executes it, and the update process is completed.

[0221] Thereafter, when the ignition is turned on, the camera 51 operates normally and performs image processing such as image recognition processing.

[0222] <Second Configuration Example of Camera 51>

[0223] FIG. 12 is a block diagram showing a second example of the configuration of the camera 51 in FIG.

[0224] In the figure, parts corresponding to those in FIG. 3 are given the same reference numerals, and the description thereof will be omitted below as appropriate.

[0225] In FIG. 12, a camera 51 has a sensing unit 210 and a data processing unit 220, and the sensing unit 210 and the data processing unit 220 are housed in separate housings.

[0226] The sensing unit 210 has a lens unit 111, an imaging unit 112, and an image data interface unit 211. Therefore, the sensing unit 210 has the lens unit 111 and the imaging unit 112 in common with the sensing unit 110 in Fig. 3, but differs from the sensing unit 110 in that the image data interface unit 211 is newly provided.

[0227] The image data interface unit 211 functions as an interface for exchanging data (signals) with the data processing unit 220 (the image data interface unit 221 thereof). For example, the image data interface unit 211 converts the original image output by the imaging unit 112 into a camera signal, which is a signal in a predetermined format, and transmits it to the data processing unit 220. The image data interface unit 211 also receives a control signal from the data processing unit 220 that controls the exposure time, etc., of the imaging unit 112, and supplies it to the imaging unit 112.

[0228] The data processing unit 220 has an image processing unit 121 to a communication interface unit 125, and an image data interface unit 221. Therefore, the data processing unit 220 has in common with the data processing unit 120 in Fig. 3 in that it has an image processing unit 121 to a communication interface unit 125, but differs from the data processing unit 120 in that the image data interface unit 221 is newly provided.

[0229] The image data interface unit 221 functions as an interface for exchanging data with the image data interface unit 211 of the sensing unit 210. For example, the image data interface unit 221 receives a camera signal from the image data interface unit 211 and supplies the original image included in the camera signal to the image processing unit 121. In addition, the image data interface unit 211 transmits a control signal from the image processing unit 121 to the image data interface unit 211, the control signal controlling the exposure time of the imaging unit 112 and the like.

[0230] Note that communication between the image data interface units 211 and 221 can be performed by wire, wireless, or a combination of wire and wireless.

[0231] 3 in which the sensing unit 110 and the data processing unit 120 are housed in a single housing, the camera 51 configured with the sensing unit 210 and the data processing unit 220 can be mounted in the vehicle control system 11 as part of the external recognition sensor 25. Furthermore, the camera 51 configured with the sensing unit 210 and the data processing unit 220 can be mounted in separate locations in the vehicle control system 11.

[0232] FIG. 13 is a block diagram showing another example of the configuration of the vehicle control system 11.

[0233] In the figure, parts corresponding to those in FIG. 1 are given the same reference numerals, and the description thereof will be omitted below as appropriate.

[0234] In the vehicle control system 11 of Figure 13, the sensing unit 210 of the camera 51 of Figure 12 is installed as part of the external recognition sensor 25, and the data processing unit 220 is installed as part of the driving automation control unit 29.

[0235] In FIG. 13, camera signals and control signals (FIG. 12) are exchanged between a sensing unit 210 and a data processing unit 220 via image data interface units 211 and 221 .

[0236] In addition to the vehicle 1, the present technology can be applied to mobile bodies that move on land, such as robots, mobile bodies that move in the air, such as drones and artificial satellites, and mobile bodies that move on and / or underwater, such as ships and submarines. That is, for example, the camera 51 can be mounted on various mobile bodies other than the vehicle 1 (the camera 51 can be configured to be mountable on various mobile bodies). Furthermore, the present technology can be applied to systems that detect (recognize) predetermined physical quantities such as distance, temperature, moisture content, chemical composition, etc., by sensing light (electromagnetic waves) of predetermined wavelengths (bands), such as infrared light, ultraviolet light, and other predetermined wavelengths (bands), in addition to visible light.

[0237] <Description of a computer to which this technology is applied>

[0238] Next, the series of processes of the image processing unit 121 described above can be performed by hardware or software. When the series of processes are performed by software, the programs constituting the software are installed in a general-purpose computer or the like.

[0239] FIG. 14 is a block diagram showing an example of the configuration of an embodiment of a computer in which a program for executing the above-described series of processes is installed.

[0240] The program can be recorded in advance on the hard disk 905 or ROM 903 as a recording medium built into the computer.

[0241] Alternatively, the program can be stored (recorded) on a removable recording medium 911 driven by the drive 909. Such a removable recording medium 911 can be provided as a so-called package software. Here, examples of the removable recording medium 911 include a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, and a semiconductor memory.

[0242] The program can be installed into the computer from the removable recording medium 911 as described above, or can be downloaded to the computer via a communication network or a broadcasting network and installed on the built-in hard disk 905. That is, the program can be transferred to the computer wirelessly from a download site via an artificial satellite for digital satellite broadcasting, or transferred to the computer via a wired network such as a LAN (Local Area Network) or the Internet.

[0243] The computer includes a CPU (Central Processing Unit) 902 , to which an input / output interface 910 is connected via a bus 901 .

[0244] When a user inputs a command via an input / output interface 910 by operating an input unit 907, the CPU 902 executes a program stored in a read-only memory (ROM) 903 in accordance with the command. Alternatively, the CPU 902 loads a program stored on a hard disk 905 into a random access memory (RAM) 904 and executes the program.

[0245] As a result, the CPU 902 performs processing according to the flowchart described above or processing performed by the configuration of the block diagram described above. Then, the CPU 902 outputs the processing results from the output unit 906 via the input / output interface 910, or transmits them from the communication unit 908, or further records them on the hard disk 905, as necessary.

[0246] The input unit 907 is made up of a keyboard, a mouse, a microphone, etc. The output unit 906 is made up of an LCD (Liquid Crystal Display), a speaker, etc.

[0247] In this specification, the processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing that is executed in parallel or individually (for example, parallel processing or object-based processing).

[0248] The program may be processed by a single computer (processor), or may be distributed among multiple computers. Furthermore, the program may be transferred to and executed on a remote computer.

[0249] Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0250] It should be noted that the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0251] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0252] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0253] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0254] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0255] The present technology can have the following configurations.

[0256] <1> A data processing device comprising: a processing unit that controls the amount of sensor data based on the amount of sensor data output by a sensing unit that senses light and the amount of other data stored in a storage unit that stores other data other than the sensor data, and stores the sensor data in the storage unit. <2> The data processing device described in <1>, wherein the processing unit converts the sensor data into sensor data with an amount that can be stored in the storage unit that stores the other data. <3> The data processing device described in <1> or <2>, wherein the sensor data is image data. <4> The data processing device described in <3>, wherein the processing unit converts the image data into image data with an angle of view equal to or smaller than the angle of view of the image data and with a resolution equal to or smaller than the resolution of the image data. <5> The data processing device according to <4>, wherein, when first other data is stored in the storage unit, the processing unit converts first image data having a first angle of view and a first resolution output by the sensing unit into second image data having a second angle of view smaller than the first angle of view and / or a second resolution smaller than the first resolution, and when second other data having a smaller data volume than the first other data after updating the first other data is stored in the storage unit, the processing unit converts the first image data into third image data having a third angle of view larger than the second angle of view and / or a third resolution larger than the second resolution. <6> The data processing device according to <5>, wherein, when the first other data is stored in the storage unit, the processing unit converts the first image data into the second image data having the second angle of view or the second image data having the second resolution. <7> The data processing device according to <5> or <6>, wherein, when the first other data is stored in the storage unit, the processing unit converts the first image data into the second image data having the second angle of view or the second image data having the second resolution. <8> The data processing device according to <7>, configured to be mounted on a moving body, wherein the processing unit converts the first image data into the second image data with the second angle of view or the second image data with the second resolution based on a speed of the moving body.<9> The data processing device according to any one of <3> to <8>, wherein the storage unit stores a plurality of image data sets having the same angle of view but different image sizes. <10> The data processing device according to any one of <1> to <9>, wherein the other data includes firmware or a library of image recognition processing using the sensor data. <11> The data processing device according to any one of <1> to <10>, further including the sensing unit. <12> The data processing device according to any one of <1> to <11>, configured to be mounted on a moving object. <13> A data processing method including controlling the amount of sensor data based on the amount of sensor data output by a sensing unit that senses light and the other data stored in a storage unit that stores other data other than the sensor data, and storing the sensor data in the storage unit. <14> A program for causing a computer to function as a processing unit that controls the amount of sensor data based on the amount of sensor data output by a sensing unit that senses light and the other data stored in a storage unit that stores other data other than the sensor data, and storing the sensor data in the storage unit.

[0257] 1 Vehicle, 11 Vehicle control system, 21 Vehicle control ECU, 22 Communication unit, 23 Map information storage unit, 24 Location information acquisition unit, 25 External recognition sensor, 26 In-house sensor, 27 Vehicle sensor, 28 Memory unit, 29 Driving automation control unit, 30 DMS, 31 HMI, 32 Vehicle control unit, 41 Communication network, 51 Camera, 52 Radar, 53 LiDAR, 54 Ultrasonic sensor, 62 Action planning unit, 63 Operation control unit, 81 Steering control unit, 82 Brake control unit, 83 Drive control unit, 84 Body system control unit, 85 Light control unit, 86 Horn control unit, 110 Sensing unit, 111 Lens unit, 112 Imaging unit, 120 Data processing unit, 121 Image processing unit, 121A Image conversion unit, 121B Image compression unit, 121C Image recognition unit, 122 Volatile memory, 123 Non-volatile memory, 124 Control unit, 125 Communication interface unit, 210 Sensing unit, 211 Image data interface unit, 220 Data processing unit, 221 Image data interface unit, 901 Bus, 902 CPU, 903 ROM, 904 RAM, 905 Hard disk, 906 Output unit, 907 Input unit, 908 Communication unit, 909 Drive, 910 Input / output interface, 911 Removable recording medium

Claims

1. A data processing device comprising: a processing unit that controls the amount of sensor data based on the amount of sensor data output by a sensing unit that senses light and the amount of other data stored in a memory unit that stores other data than the sensor data, and stores the sensor data in the memory unit.

2. The data processing device according to claim 1, wherein said processing section converts said sensor data into sensor data of an amount that can be stored in said storage section that stores said other data.

3. The data processing device according to claim 1, wherein the sensor data is image data.

4. The data processing device according to claim 3, wherein said processing section converts said image data into image data having an angle of view equal to or smaller than that of said image data and having a resolution equal to or smaller than that of said image data.

5. The data processing device of claim 4, wherein the processing unit, when first other data is stored in the memory unit, converts first image data having a first angle of view and a first resolution output by the sensing unit into second image data having a second angle of view smaller than the first angle of view and / or a second resolution smaller than the first resolution, and when second other data having a reduced amount of data compared to the first other data after updating the first other data is stored in the memory unit, converts the first image data into third image data having a third angle of view larger than the second angle of view and / or a third resolution larger than the second resolution.

6. A data processing device according to claim 5, wherein the storage unit has a storage capacity capable of storing the second image data but not capable of storing the third image data when the first other data is stored in the storage unit.

7. The data processing device according to claim 5, wherein the processing unit converts the first image data into the second image data having the second angle of view or the second image data having the second resolution when the first other data is stored in the memory unit.

8. The data processing device according to claim 7, configured to be mounted on a moving body, wherein the processing unit converts the first image data into the second image data with the second angle of view or the second image data with the second resolution based on the speed of the moving body.

9. The data processing device according to claim 3, wherein the storage unit stores a plurality of image data having the same angle of view but different image sizes.

10. The data processing device according to claim 1, wherein the other data includes FW (Firmware) or a library for image recognition processing using the sensor data.

11. The data processing device according to claim 1, further comprising the sensing unit.

12. The data processing device according to claim 1, which is configured to be mounted on a moving object.

13. A data processing method including controlling the amount of sensor data based on the amount of sensor data output by a sensing unit that senses light and other data stored in a memory unit that stores other data than the sensor data, and storing the sensor data in the memory unit.

14. A program for causing a computer to function as a processing unit that controls the amount of sensor data based on the amount of sensor data output by a sensing unit that senses light and other data stored in a memory unit that stores other data than the sensor data, and stores the sensor data in the memory unit.

Images