Image processing apparatus

Abstract

To provide an image processing apparatus capable of appropriately setting calculation resources each for executing predetermined processing for images obtained from each of a plurality of on-vehicle cameras.SOLUTION: An image processing apparatus includes: a resource-allocating unit 31 that identifies a camera prioritized from among a plurality of cameras (2-1 to 2-2) by which imaging areas whose peripheries of a vehicle 10 are different from each other are imaged, based on priority information, when the vehicle 10 travels in a predetermined section comprises, and sets calculation resources allocated to predetermined processing for images each produced by a prioritized camera, that are set larger than calculation resources allocated to predetermined processing for images each produced by the other camera; and a calculation-processing unit 32 that executes predetermined processing for images each produced by each of a plurality of cameras in such manner that as there is provided a camera with calculation resources set larger, that are allocated among the plurality of cameras (2-1 to 2-2), a calculation amount of the predetermined processing becomes larger.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image processing device that performs predetermined processing on an image obtained by an on-board camera. [Background technology]

[0002] In a system capable of executing multiple processes, a technique has been proposed for shortening the waiting time of a process with a higher priority (see Patent Document 1).

[0003] The above document describes how, depending on the vehicle's driving conditions, priority is set for object detection processing for images obtained from each of multiple cameras installed on the vehicle, and computing resources are allocated first to the prioritized images to perform object detection processing. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-11822 Summary of the Invention [Problem to be solved by the invention]

[0005] Of the multiple cameras mounted on a vehicle, a camera that can more appropriately capture an object to be detected may be replaced with another camera depending on the section of the road the vehicle is traveling.

[0006] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an image processing device that can appropriately set computational resources for executing predetermined processing on images obtained from each of a plurality of on-board cameras. [Means for solving the problem]

[0007] According to one embodiment, there is provided an image processing device including: a storage unit that stores priority information indicating a camera that is prioritized within a predetermined section among a plurality of cameras that capture different image capture areas around a vehicle within the predetermined section; a resource allocation unit that identifies a prioritized camera among the plurality of cameras based on the priority information while the vehicle is traveling within the predetermined section and allocates more computational resources to a predetermined process performed on images generated by the prioritized camera than to a predetermined process performed on images generated by cameras other than the prioritized camera among the plurality of cameras; and a computation processing unit that performs the predetermined process on images generated by each of the plurality of cameras such that the camera to which more computational resources are allocated receives a larger amount of computation for the predetermined process.

[0008] In one embodiment, the specified section is a section in which the vehicle speed limit in that section is equal to or less than a specified threshold, the multiple cameras include a first camera whose imaging area is in front of the vehicle and a second camera whose imaging area is in front of the vehicle and has a wider angle than the first camera, and the resource allocation unit prioritizes the second camera over the first camera when the vehicle is traveling in the specified section.

[0009] In one embodiment, the predetermined process is a process for detecting a predetermined object, and the calculation processing unit detects the predetermined object by inputting an image generated by a prioritized camera into a classifier that has been pre-trained to detect the predetermined object, while detecting the predetermined object by downsampling or cropping an image generated by another camera and then inputting the image into the classifier.

[0010] In one embodiment, the predetermined processing is processing for tracking predetermined objects, and the calculation processing unit sets the number of objects to be tracked among the plurality of predetermined objects shown in the image generated by the prioritized camera to be greater than the number of objects to be tracked among the plurality of predetermined objects shown in the image generated by the other cameras.

[0011] In one embodiment, the processing unit sets the execution period of the predetermined process for the prioritized camera to be shorter than the execution period of the predetermined process for the other cameras. [Effects of the Invention]

[0012] The image processing device according to the present disclosure can appropriately set computational resources for executing predetermined processing on images obtained from each of a plurality of on-board cameras. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a schematic configuration diagram of a vehicle control system in which an image processing device is implemented. [Figure 2] FIG. 1 is a hardware configuration diagram of an electronic control device that is an example of an image processing device. [Figure 3] FIG. 2 is a functional block diagram of a processor of the electronic control device. [Figure 4] FIG. 10 is an explanatory diagram illustrating an overview of resource allocation for each camera. [Figure 5] 10 is an operational flowchart of a vehicle control process including image processing. DETAILED DESCRIPTION OF THE INVENTION

[0014] An image processing device, an image processing method, and an image processing computer program executed by the image processing device will be described below with reference to the drawings. This image processing device identifies a camera (hereinafter sometimes referred to as a priority camera) that has priority over other cameras when the vehicle is traveling through a predetermined section, based on priority information indicating which camera has priority within a predetermined section among multiple on-board cameras. Then, this image processing device allocates more computing resources to a predetermined process for images generated by the priority camera than to a predetermined process for images generated by the other cameras.

[0015] Fig. 1 is a schematic configuration diagram of a vehicle control system in which an image processing device is implemented. Fig. 2 is a hardware configuration diagram of an electronic control device, which is an example of an image processing device. In this embodiment, a vehicle control system 1 is mounted on a vehicle 10 and controls the vehicle 10. The vehicle control system 1 has two cameras 2-1 to 2-2 for capturing images of the surroundings of the vehicle 10, a GPS receiver 3, a storage device 4, and an electronic control unit (ECU) 5, which is an example of an image processing device. The cameras 2-1 to 2-2, the GPS receiver 3, the storage device 4, and the ECU 5 are communicably connected via an in-vehicle network. The vehicle control system 1 may further have a distance measurement sensor (not shown), such as a LiDAR or radar, for measuring distances to objects around the vehicle 10.

[0016] The cameras 2-1 and 2-2 are attached to the vehicle 10 so as to capture different imaging areas around the vehicle 10. In this embodiment, the camera 2-1 is an example of a first camera, a camera for capturing images of an area in front of the vehicle 10 and in the distance, has a relatively long focal length, and is attached inside the vehicle 10, facing the front of the vehicle 10. The camera 2-2 is an example of a second camera, a camera for capturing images of an area in front of the vehicle 10 and in the vicinity, and has a shorter focal length and a wider angle of view than the camera 2-1. In other words, the camera 2-2 has a wider angle of view than the camera 2-1. Like the camera 2-1, the camera 2-2 is attached inside the vehicle 10, facing the front of the vehicle 10. Note that three or more cameras may be provided on the vehicle 10. For example, a camera may be provided separate from the cameras 2-1 and 2-2, for capturing images of an area behind the vehicle 10 or an area on either the left or right side of the vehicle 10. Each camera captures an image of its own shooting area at a predetermined shooting interval (for example, 1 / 30 to 1 / 10 seconds) and generates an image of that shooting area.

[0017] Each time an image is generated, each of the cameras 2-1 and 2-2 outputs the generated image together with the identification information of the camera to the ECU 5 via the in-vehicle network.

[0018] The GPS receiver 3 receives GPS signals from GPS satellites at predetermined intervals and determines the own position of the vehicle 10 based on the received GPS signals. Then, every time the GPS receiver 3 determines its position, it outputs positioning information indicating the positioning result of the own position of the vehicle 10 based on the GPS signals to the ECU 5 via the in-vehicle network. Note that the vehicle 10 may have a receiver that complies with a satellite positioning system other than the GPS receiver 3. In this case, it is sufficient that the receiver determines the own position of the vehicle 10.

[0019] The storage device 4 is an example of a storage unit, and includes, for example, a hard disk drive, a nonvolatile semiconductor memory, or an optical recording medium and its access device. The storage device 4 stores map information. The map information includes, for each road section, information indicating the location and area of the road section, and priority information indicating the priority of each camera mounted on the vehicle 10 in that road section, i.e., the camera with the highest priority.

[0020] The ECU 5 executes vehicle control processing to control automatic driving of the vehicle 10 or to assist the driver of the vehicle 10 in driving. The vehicle control processing includes predetermined processing of a plurality of time-series images acquired by each of the cameras 2-1 to 2-2. The ECU 5 has a communication interface 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 may be configured as different circuits, or may be integrated into a single integrated circuit.

[0021] The communication interface 21 has an interface circuit for connecting the ECU 5 to the in-vehicle network. Every time the communication interface 21 receives identification information and an image from the camera 2-1 or 2-2, the communication interface 21 passes the received identification information and image to the processor 23. The communication interface 21 also passes the positioning information received from the GPS receiver 3 and the map information received from the storage device 4 to the processor 23.

[0022] The memory 22 is another example of a storage unit and includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 22 stores various data and parameters used in various processes executed by the processor 23 of the ECU 3. For example, the memory 22 stores map information, positioning information, and images. Furthermore, the memory 22 stores various data generated during the vehicle control process, such as information about detected objects, for a certain period of time.

[0023] The processor 23 is an example of a control unit. In this embodiment, the processor 23 includes, for example, one or more central processing units (CPUs) and their peripheral circuits. The processor 23 may further include other arithmetic circuits such as a logic operation unit, a numerical operation unit, or a graphics processing unit. The processor 23 executes vehicle control processing for the vehicle 10.

[0024] 3 is a functional block diagram of the processor 23 of the ECU 3, which is related to vehicle control processing including image processing. The processor 23 has a resource allocation unit 31, an arithmetic processing unit 32, and a vehicle control unit 33. Each of these units in the processor 23 is a functional module realized by a computer program running on the processor 23. Alternatively, each of these units may be a dedicated arithmetic circuit provided in the processor 23.

[0025] The resource allocation unit 31, at predetermined intervals, refers to priority information included in the map information and allocates computational resources to images generated by each of the multiple cameras mounted on the vehicle 10 for predetermined processing executed by the computational processing unit 32. In this embodiment, the resource allocation unit 31 identifies a priority camera from among the cameras 2-1 and 2-2 for the road section on which the vehicle 10 is traveling, and allocates more computational resources to images generated by the priority camera than to images generated by cameras other than the priority camera.

[0026] To this end, the resource allocation unit 31 identifies the road section including the position of the vehicle 10 by referring to the map information and the position of the vehicle 10 indicated by the latest positioning information from the GPS receiver 3. Then, the resource allocation unit 31 identifies the camera with the highest priority as the priority camera based on the priority of each camera for the identified road section indicated in the priority information.

[0027] For example, in road sections with a relatively large number of vulnerable road users, the vehicle 10 is expected to travel at a relatively low speed. Therefore, it is necessary to reliably detect vulnerable road users located near the vehicle 10 rather than far away from the vehicle 10. Therefore, in road sections with a relatively large number of vulnerable road users, the camera 2-2 capturing images of the nearby area is given priority over the camera 2-1 capturing images of the distant area. Note that such road sections are set in advance based on collected data representing traffic conditions. Furthermore, in road sections where the speed limit is equal to or less than a predetermined vehicle speed threshold (e.g., 30 km / h) that is lower than the legal speed limit, it is preferable to give priority to the camera 2-2 capturing images of the nearby area over the camera 2-1 capturing images of the distant area. Furthermore, in road sections without sidewalks, since pedestrians are more likely to walk near the vehicle 10, it is preferable to give priority to the camera 2-2 capturing images of the nearby area over the camera 2-1 capturing images of the distant area.

[0028] Conversely, in road sections where vehicle 10 travels at relatively high speeds, such as expressways, it is necessary to detect objects that may obstruct the travel of vehicle 10 (for example, other vehicles stopped in the lane in which vehicle 10 is traveling) as early as possible. Furthermore, in road sections where an intersection is visible from a distance, it is desirable to be able to detect vulnerable road users passing through the intersection as early as possible, and it is also necessary to improve the recognition accuracy of traffic lights installed at the intersection. Therefore, in road sections where the distance from the expressway or intersection is within a predetermined distance range (for example, several tens of meters to 100 meters), camera 2-1, which captures distant areas, is given priority over camera 2-2, which captures nearby areas. Note that when vehicle 10 approaches the intersection beyond the above distance range, camera 2-2 may be given priority over camera 2-1.

[0029] Depending on the road section, the computational resources may be allocated equally to each camera.

[0030] The resource allocation unit 31 determines the amount of computational resources to be allocated to each camera such that the higher the priority of that camera, the greater the computational resources to be allocated. In doing so, the resource allocation unit 31 determines the amount of computational resources to be allocated to each camera by referring to a lookup table that indicates the relationship between priority and the amount of computational resources to be allocated. The lookup table may be stored in advance in the memory 22 or the storage device 4. The resource allocation unit 31 then notifies the computation processing unit 32 of allocation information that indicates the amount of computational resources allocated to each camera. The allocation information includes, for each camera, a combination of camera identification information and a flag that has a larger value as the amount of computation allocated to that camera increases.

[0031] The arithmetic processing unit 32 performs calculations for predetermined processing on images generated by each camera using the calculation resources indicated by the allocation information notified by the resource allocation unit 31. That is, the arithmetic processing unit 32 performs predetermined processing on images generated by each camera such that the camera to which more calculation resources are allocated has a larger amount of calculation and higher accuracy. In this embodiment, the arithmetic processing unit 32 performs the predetermined processing for avoiding collision of the vehicle 10 with surrounding objects, specifically, object detection processing from images. Furthermore, the arithmetic processing unit 32 performs the predetermined processing, such as tracking detected objects and predicting the future trajectory of the object being tracked, which are related to the object detection processing.

[0032] As an object detection process, the arithmetic processing unit 32 detects a predetermined object depicted in the image by inputting the images generated by each camera into a classifier that has been trained in advance to detect the predetermined object. The predetermined object may be, for example, another vehicle present around the vehicle 10, a pedestrian, a predetermined road marking such as a lane marking or a stop line, various road signs, a traffic light, or another object that may affect the traveling of the vehicle 10. Furthermore, if the detected object is an object that can take multiple states, such as a traffic light, the classifier may also be configured to output a classification result for the state of the object. For example, if the detected object is a traffic light, the classifier is trained in advance to also output the light state of the traffic light, such as green, yellow, or red.

[0033] The classifier used for object detection can be a so-called deep neural network (hereinafter simply referred to as DNN) with a convolutional neural network architecture, such as Single Shot MultiBox Detector or Faster R-CNN. Alternatively, the classifier can be a DNN with an attention mechanism, such as Vision Transformer, or a classifier based on a machine learning method other than DNN. Such a classifier is trained in advance according to a predetermined training method, such as backpropagation, using a large number of images (teacher images) depicting the object to be detected.

[0034] In this embodiment, an image generated by a priority camera to which a relatively large amount of computational resources is allocated (hereinafter, sometimes referred to as a priority image) is input directly to the classifier. In contrast, an image generated by a camera to which a relatively small amount of computational resources is allocated (hereinafter, sometimes referred to as a non-priority camera) (hereinafter, sometimes referred to as a non-priority image) is downsized by downsampling and then input to the classifier. Alternatively, a predetermined area may be cropped from the non-priority image, and the cropped area may be input to the classifier. In this way, the number of pixels of the image input to the classifier for the non-priority image is smaller than that of the original image, so the amount of computation required for object detection processing on the non-priority image is less than the amount of computation required for object detection processing on the priority image. Alternatively, a classifier used for object detection from the priority image and a classifier used for object detection from the non-priority image may be provided separately. In this case, the classifier used for object detection from the priority image (hereinafter, sometimes referred to as a precise classifier) may be a classifier that requires a relatively large amount of computation but has relatively high object detection accuracy. In contrast, a classifier used for object detection from a non-priority image (hereinafter referred to as a simple classifier) can be a classifier that has a relatively low object detection accuracy but requires a relatively small amount of calculation. For example, the number of calculation layers (e.g., convolutional layers or attention mechanisms) that the precise classifier has, or the number of channels output from any layer, is configured to be greater than that of the simple classifier.

[0035] For example, assume that the road section on which vehicle 10 is traveling is a road section with a relatively large number of vulnerable road users, and camera 2-2, which captures the nearby area, is identified as the priority camera. In this case, the image generated by camera 2-2 becomes the priority image, and is input directly to a classifier, thereby enabling accurate detection of pedestrians and other objects near vehicle 10. In contrast, the image generated by camera 2-1 becomes the non-priority image, and is input to the classifier after being downsampled or cropped, thereby attempting to detect objects depicted in the image with a relatively small amount of calculation.

[0036] Furthermore, as a tracking process, the arithmetic processing unit 32 tracks, for each camera, one or more objects detected from each of the multiple images generated in time series by that camera. The objects to be tracked may be movable objects such as other vehicles or pedestrians. However, the objects to be tracked are not limited to movable objects and may also be objects whose state changes over time, such as traffic lights. In this case, the arithmetic processing unit 32 tracks each object by applying a predetermined tracking method, such as KLT tracking, to the object region in which the object is detected from each of the multiple images generated in time series. This allows object regions in which the same object is represented to be associated with each other between the images.

[0037] In this embodiment, the number of objects to be tracked in the multiple priority images generated by the priority camera (hereinafter, sometimes referred to as the tracking upper limit number) is set to be greater than the number of objects to be tracked in the multiple non-priority images generated by the non-priority camera. In other words, the greater the amount of computing resources allocated, the more objects can be tracked. Note that, if the number of objects detected in the latest images of any camera exceeds the tracking upper limit number, the processing unit 32 selects the objects to be tracked up to the tracking upper limit number in the latest images, starting from the object area with the largest object region in the latest image. This is because the larger the object region, the closer the object represented in the object region is to the vehicle 10. Alternatively, the processing unit 32 may select the objects to be tracked up to the tracking upper limit number in the latest images, starting from the object area with the lowest edge closest to the bottom of the image. This is because, if the object to be tracked is an object on the road surface, such as another vehicle or a pedestrian, the bottom edge of the object region in which the object is represented corresponds to the position where the object is in contact with the road surface, and the closer the bottom edge of the object region is to the bottom of the image, the closer the object represented in the object region is to the vehicle 10 is to the object represented in the object region.

[0038] Furthermore, the arithmetic processing unit 32 performs a prediction process to predict the future trajectory of each object being tracked. For each object being tracked, the arithmetic processing unit 32 performs a viewpoint conversion process using information such as the mounting position on the vehicle 10, focal length, and angle of view of the camera that generated the time-series images depicting the object, among the cameras installed on the vehicle 10. The arithmetic processing unit 32 then converts the coordinates of each object being tracked into coordinates on a bird's-eye view image (bird's-eye view coordinates), thereby determining the trajectory of the object. The arithmetic processing unit 32 estimates the position of the detected object at the time each image is generated based on the position and attitude of the vehicle 10 at the time each image is generated, the estimated distance to the detected object, and the direction from the vehicle 10 toward the object. The arithmetic processing unit 32 can estimate the distance to the object based on the position of the bottom edge of the object area depicting the detected object in the image, and parameters such as the camera's shooting direction and installation height. Furthermore, if the vehicle 10 is equipped with a distance measurement sensor, the arithmetic processing unit 32 may use the distance measured by the distance measurement sensor in a direction corresponding to the object area in which the detected object is displayed as the estimated distance to the object. The position and attitude of the vehicle 10 at the time of generating each image may be estimated by comparing the image with map information. In this case, the arithmetic processing unit 32 may project predetermined features, such as road signs detected from the image, onto the map information using the assumed position and attitude of the vehicle 10, and determine the assumed position and attitude of the vehicle 10 when the projected features most closely match the corresponding features displayed in the map information as the actual position and attitude of the vehicle 10. The arithmetic processing unit 32 may then perform prediction processing using a Kalman filter, a particle filter, or the like on the trajectory of the detected object, thereby estimating the predicted trajectory of the object up to a predetermined time ahead.

[0039] In this embodiment, the number of objects to be subjected to prediction processing for one or more objects detected and tracked from multiple priority images generated by the priority camera (hereinafter, sometimes referred to as the prediction upper limit number) is set to be greater than the prediction upper limit number for one or more objects detected and tracked from multiple non-priority images generated by the non-priority camera. In other words, the greater the amount of computational resources allocated, the greater the number of object trajectories that can be predicted. Note that, if the number of objects being tracked for any camera exceeds the prediction upper limit number, the computation processing unit 32 may select the objects to be subjected to prediction processing up to the prediction upper limit number, in order of the objects being tracked that are closest to the vehicle 10 on the trajectory being tracked.

[0040] Furthermore, the execution period of the object detection process and related processes for the priority camera may be set to be shorter than the execution period of the object detection process and related processes for the non-priority camera. The exposure amount of each of the cameras 2-1 and 2-2 may be changed at a predetermined exposure period longer than the shooting period. For example, the exposure amount may be changed in four stages for each exposure period. In this case, the processing unit 32 may use only images generated at a specific exposure amount for the non-priority camera for the object detection process and related tracking process. On the other hand, the processing unit 32 may use images generated at a different exposure amount for the priority camera for the object detection process and related tracking process. As a result, the execution period of the above processes for the priority camera is shorter than the execution period of the above processes for the non-priority camera. Note that even if each camera always shoots at a constant exposure amount, the processing unit 32 may set the execution period of the object detection process and related processes for the priority camera to be relatively shorter. As a result, the processing unit 32 executes the object detection process and related processes for the priority camera at a shorter period, thereby enabling more accurate tracking and prediction of the behavior of objects depicted in images generated by that camera.

[0041] Note that the priority of each camera may be set to one of three or more levels. In this case, the lower the priority set for a non-priority camera, the lower the processing unit 32 may omit predetermined processing for the non-priority image. For example, the lower the priority, the lower the tracking upper limit number and the prediction upper limit number may be set. Alternatively, the processing unit 32 may downsample or crop the non-priority image so that the number of pixels of the non-priority image input to the classifier decreases as the priority decreases. Furthermore, the processing unit 32 may omit execution of object detection processing and related processing for the non-priority image generated by a non-priority camera set to the lowest priority among the settable priorities.

[0042] The calculation processing unit 32 notifies the vehicle control unit 33 of the predicted trajectory of each object for which the predicted trajectory has been calculated. The calculation processing unit 32 also notifies the vehicle control unit 33 of the type, state, and object area of each detected object in the latest image.

[0043] 4 is a diagram showing an example of allocation of computational resources to each camera according to this embodiment. In this example, it is assumed that there are many pedestrians 400 in the road section on which the vehicle 10 is traveling, and therefore camera 2-2, which captures images of the nearby area, is given priority over camera 2-1, which captures images of the distant area. Therefore, more computational resources are allocated to image 402 generated by camera 2-2 than to image 401 generated by camera 2-1. As a result, more precise object detection processing is performed on image 402 than on image 401, and therefore pedestrian 400, the detection target, is more reliably detected in image 402 than in image 401.

[0044] When performing autonomous driving control of vehicle 10, vehicle control unit 33 generates one or more planned driving routes (trajectories) for vehicle 10 in a nearby predetermined section (e.g., 500 m to 1 km) so that vehicle 10 travels along a planned driving route to a destination set by a navigation device (not shown). The planned driving route is expressed, for example, as a set of target positions of vehicle 10 at each time when vehicle 10 travels through the predetermined section. Then, vehicle control unit 33 controls each section of vehicle 10 so that vehicle 10 travels along the planned driving route.

[0045] The vehicle control unit 33 generates a planned driving route for the vehicle 10 based on the predicted trajectory of each object being tracked so that the predicted value of the distance between each of the objects being tracked and the vehicle 10 up to a predetermined time in the future is equal to or greater than a predetermined distance for each object. Furthermore, when a traffic light at an intersection ahead of the vehicle 10 is red, the vehicle control unit 33 generates a planned driving route so that the vehicle 10 will stop before the intersection. Note that the vehicle control unit 33 may generate multiple planned driving routes. In this case, the vehicle control unit 33 may select, from the multiple planned driving routes, a route that minimizes the sum of the absolute values of the acceleration of the vehicle 10.

[0046] Once the planned travel route is set, the vehicle control unit 33 controls each unit of the vehicle 10 so that the vehicle 10 travels along the planned travel route. For example, the vehicle control unit 33 calculates a target acceleration of the vehicle 10 based on the planned travel route and the current vehicle speed of the vehicle 10 measured by a vehicle speed sensor (not shown), and sets the accelerator opening or braking amount so as to achieve the target acceleration. The vehicle control unit 33 then calculates a fuel injection amount based on the set accelerator opening, and outputs a control signal corresponding to the fuel injection amount to a fuel injection device of the engine of the vehicle 10. Alternatively, the vehicle control unit 33 calculates the amount of power to be supplied to the motor based on the set accelerator opening, and controls the motor drive circuit so that the amount of power is supplied to the motor. Alternatively, the vehicle control unit 33 outputs a control signal corresponding to the set braking amount to the brake of the vehicle 10.

[0047] Furthermore, when the vehicle control unit 33 changes the course of the vehicle 10 so that the vehicle 10 travels along the planned travel route, it calculates the steering angle of the vehicle 10 according to the planned travel route and outputs a control signal corresponding to the steering angle to an actuator (not shown) that controls the steering wheels of the vehicle 10.

[0048] Furthermore, when assisting the driver's driving, the vehicle control unit 33 determines whether or not there is a possibility that the vehicle 10 will collide with any of the objects being tracked, based on a predicted trajectory of the vehicle 10 if the vehicle 10 continues traveling at the current vehicle speed and acceleration, and the predicted trajectories of each object being tracked. If it is determined that there is a possibility of a collision, the vehicle control unit 33 decelerates the vehicle 10. Furthermore, the vehicle control unit 33 may notify the driver of the risk of a collision via a user interface provided in the cabin of the vehicle 10. The user interface may be, for example, a display device, a speaker, a light source, or a vibrator.

[0049] FIG. 5 is an operational flowchart of a vehicle control process executed by the processor 23, including image processing of images from each camera mounted on the vehicle.

[0050] The resource allocation unit 31 refers to priority information included in map information to identify a camera from among the multiple cameras on the vehicle 10 that is to be prioritized for the road section on which the vehicle 10 is traveling (step S101). Then, the resource allocation unit 31 allocates computational resources to each camera so that the amount of computational resources allocated to the prioritized camera is greater than the amount of computational resources allocated to the other cameras (step S102).

[0051] The calculation processing unit 32 performs object detection processing and related processing for each camera on an image generated by that camera, with the amount of calculation resources allocated to that camera increasing (step S103).The vehicle control unit 33 then controls the traveling of the vehicle 10 by referring to the results of the object detection processing and the like for the image of each camera (step S104).

[0052] As described above, this image processing device identifies a priority camera to be prioritized when the vehicle is traveling through a predetermined section based on priority information indicating which camera among multiple on-board cameras is prioritized for that section. This image processing device allocates more computing resources to the priority camera than to other cameras, and performs predetermined processing on images generated by each camera using the allocated computing resources. In this way, this image processing device can appropriately allocate resources for performing predetermined processing for each of the multiple on-board cameras.

[0053] The image processing according to the above embodiment or modification may be used for purposes other than vehicle control processing. For example, this image processing may be performed to detect specific features, such as road markings or road signs, in individual road sections in order to generate or update map information used for autonomous driving control, and to upload information about the detected features to a server via an in-vehicle wireless communication terminal (not shown). In this case, the processing of the vehicle control unit 33 may be omitted.

[0054] A computer program that realizes the functions of the processor 23 of the ECU 5 according to the above embodiment or variant may be provided in a form recorded on a computer-readable portable recording medium such as a semiconductor memory, a magnetic recording medium or an optical recording medium.

[0055] As described above, those skilled in the art can make various modifications to the embodiments within the scope of the present invention. [Explanation of symbols]

[0056] 1 Vehicle control system, 2-1, 2-2 Camera, 3 GPS receiver, 4 Storage device, 5 Electronic control device (image processing device), 21 Communication interface, 22 Memory, 23 Processor, 31 Resource allocation unit, 32 Processing unit, 33 Vehicle control unit

Claims

1. a storage unit that stores priority information indicating a camera that is prioritized in a predetermined section among a plurality of cameras that capture different image capture areas around a vehicle in the predetermined section; a resource allocation unit that identifies a prioritized camera among the plurality of cameras based on the priority information while the vehicle is traveling in the predetermined section, and allocates a larger number of computational resources to a predetermined process executed on an image generated by the prioritized camera than the computational resources to be allocated to a predetermined process executed on an image generated by a camera other than the prioritized camera among the plurality of cameras; a processing unit that executes the predetermined processing on the images generated by each of the plurality of cameras such that a camera having a larger amount of allocated computing resources among the plurality of cameras has a larger amount of computation required for the predetermined processing; An image processing device having:

2. 2. The image processing device according to claim 1, wherein the specified section is a section in which the vehicle speed limit in the section is equal to or less than a specified threshold, the plurality of cameras include a first camera having a shooting area in front of the vehicle and a second camera having a shooting area in front of the vehicle and a wider angle than the first camera, and the resource allocation unit prioritizes the second camera over the first camera when the vehicle is traveling in the specified section.

3. the predetermined processing is processing for detecting a predetermined object, 3. The image processing device according to claim 1, wherein the arithmetic processing unit detects the predetermined object by inputting the image generated by the prioritized camera to a classifier that has been trained in advance to detect the predetermined object, and detects the predetermined object by downsampling or cropping the image generated by the other camera and then inputting it to the classifier.

4. the predetermined process is a process of tracking a predetermined object, 3. The image processing device according to claim 1, wherein the processing unit sets the number of objects to be tracked among the plurality of predetermined objects shown in the image generated by the prioritized camera to be greater than the number of objects to be tracked among the plurality of predetermined objects shown in the image generated by the other camera.

5. The image processing device according to claim 1 , wherein the processing unit sets an execution cycle of the predetermined process for the prioritized camera to be shorter than an execution cycle of the predetermined process for the other cameras.

Images