Processor on integrated circuit, control device, target recognition device, and vehicle travel control system

Abstract

A multi-core processor having a plurality of cores each including a plurality of processor elements, wherein: the multi-core processor executes: at least one of Classification processing, Detection processing, Segmentation processing, and Pose Estimation processing, as Deep Neural Network (DNN) processing using a DNN, on vehicle-exterior image data input to a control device; at least one of water droplet / dirt removal processing, distortion correction processing, resizing processing, cutout processing, and luminance standardization processing, as pre-processing for the DNN processing; and at least one of NMS processing and top K processing as post-processing for the DNN processing.

Description

Processor on integrated circuit, control device, target recognition device, and vehicle driving control system CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is based on Japanese Application No. 2023-215857, filed on December 21, 2023, the contents of which are incorporated herein by reference.

[0002] The present disclosure relates to a processor on an integrated circuit, a control device, a target recognition device, and a vehicle driving control system.

[0003] In recent years, there has been a need for improved performance of vehicle-mounted cameras, enlargement of imaging areas, and higher accuracy of target recognition in order to accurately recognize the environment outside a vehicle.

[0004] To meet the above needs, the image data to be calculated has become higher resolution, and the neural network models that process the data have become larger accordingly.

[0005] Patent Document 1 describes a control device that uses a neural network to recognize targets in an image of the exterior of a vehicle captured by an on-board camera in order to realize autonomous driving and more advanced driving assistance.

[0006] U.S. Patent Application Publication No. 2013 / 0063600

[0007] However, the control device described in Patent Document 1 performs pre-processing and post-processing of image data using a 64-bit to 128-bit calculation accelerator such as a DSP (Digital Signal Processor) and a CPU (Central Processing Unit) in target recognition using images captured by an on-board camera, and performs DNN processing using a DNN (Deep Neural Network) in an NPU (Neural Processing Unit) equipped with multiple MAC (Multiply-Accumulate) arithmetic units. As such, in the control device described in Patent Document 1, the target recognition processing is shared among the DSP, CPU, and NPU, and therefore data processing cannot be performed efficiently.

[0008] In view of the above-mentioned problems, an object of the present disclosure is to provide a processor, a control device, a target recognition device, and a vehicle driving control system on an integrated circuit that can efficiently execute data processing.

[0009] The present disclosure employs the following technical solutions to solve the above problems. The reference symbols in parentheses in the claims and in this section are merely examples showing the correspondence with the specific solutions described in the embodiments below as one aspect, and do not limit the technical scope of the present disclosure.

[0010] A processor on an integrated circuit according to one aspect of the present disclosure is a processor on an integrated circuit used in a vehicle that performs autonomous driving assistance based on an outside image captured by a camera that captures the external environment of the vehicle, and includes a multi-core processor having a plurality of cores, each including a plurality of processor elements, and a CPU that controls the multi-core processor, wherein the plurality of cores in the multi-core processor are interconnected, and each of the plurality of processor elements includes an ALU that performs logical operations, addition, and subtraction, and a MAC calculator that performs multiply-and-accumulate operations, and is further interconnected with at least one of the other processor elements so that operation data of the other processor elements can be referenced during operation, and the CPU controls a task schedule of the multi-core processor, The vehicle exterior image is subjected to pre-processing including at least one of a water droplet / dirt removal process for removing water droplets / dirt that are captured in the vehicle exterior image, a distortion correction process for correcting distortion in the vehicle exterior image, a resizing process for changing the size of the vehicle exterior image, a cropping process for cropping a predetermined area from the vehicle exterior image, and a brightness standardization process for standardizing brightness of the vehicle exterior image, and further includes a classification process for classifying external environment information included in the pre-processed image into predetermined target classifications, a detection process for identifying positions, number, and types of targets outside the vehicle from the external environment information included in the pre-processed image, a segmentation process for segmenting the external environment information included in the pre-processed image by targets outside the vehicle, and a pose process for estimating the attitude of targets included in the external environment information included in the pre-processed image. A series of tasks are successively calculated, including a process in which a DNN process consisting of at least one of estimation processes is performed on the pre-processed image, and a post-processing process consisting of at least one of an NMS process that removes some of the overlapping candidate regions from the candidate regions of the target included in the DNN-processed image, and a top K process that extracts the top K candidate regions with the highest confidence from the candidate regions.

[0011] A control device according to one aspect of the present disclosure is a control device for a vehicle that performs autonomous driving assistance, and includes a multi-core processor having a plurality of cores, each including a plurality of processor elements, and a CPU that controls the multi-core processor, wherein the plurality of cores are interconnected, and each of the plurality of processor elements includes an ALU and a MAC calculator, and the multi-core processor performs DNN processing using a DNN (Deep Neural Network) on image data of the outside of the vehicle that is input to the control device, including at least one of classification processing, detection processing, segmentation processing, and pose estimation processing, and as pre-processing for the DNN processing, at least one of water droplet and dirt removal processing, distortion correction processing, resizing processing, cropping processing, and brightness standardization processing, and as post-processing for the DNN processing, at least one of NMS processing and Top-K processing.

[0012] According to this configuration, the control device can execute both neural network calculations and their pre- and post-processing independently without the need for data transfer between DSPs or to the CPU, enabling efficient data processing and faster processing. Specifically, data transfer between the DNN itself and some layers or pre- and post-processing can be omitted, thereby speeding up overall data processing. Furthermore, because the control device of this embodiment is programmable, custom layers can be added, enabling functions to be added via software, improving functional expandability through software, such as route search and signal processing. Furthermore, it is possible to avoid the decline in development efficiency that occurs when development spans CPUs, DSPs, etc., thereby improving development efficiency.

[0013] A target recognition device according to one aspect of the present disclosure is a target recognition device mounted on a vehicle, and includes an imaging unit that acquires image data of an area outside the vehicle, and the control device described above.

[0014] A vehicle driving control system according to one aspect of the present disclosure includes the above-described target recognition device and a driving device, and the control device performs vehicle driving control to control the driving device based on the results of the target recognition processing by the target recognition device.

[0015] The above and other objects, features, and advantages of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which Fig. 1 is a block diagram showing an example of the configuration of a target recognition device according to this embodiment, Fig. 2 is a diagram showing an example of the hardware configuration of a control device provided in the target recognition device according to this embodiment, Fig. 3 is a diagram showing an example of the functional layout of the control device according to this embodiment in comparison with an example of the functional layout of a conventional control device, Fig. 4 is a flowchart showing the target recognition process according to this embodiment executed by a multi-core processor, Fig. 5 is a block diagram showing the configuration of the target recognition device and water droplet removal system according to this embodiment, Fig. 6 is a diagram showing an overview of image processing according to this embodiment, and Fig. 7 is a diagram showing the multi-core processor of the target recognition device according to this embodiment. FIG. 11 is a flowchart showing an example of an object detection process executed by the target recognition device according to this embodiment, FIG. 12 is a schematic diagram showing an example of a result of inference processing by the inference unit, FIG. 13 is a schematic diagram showing an example of a result of NMS processing by the NMS unit, FIG. 14 is a block diagram showing an example of the configuration of a vehicle driving control system according to this embodiment, and FIG. 15 is a diagram for explaining a pyramidal connection pattern.

[0016] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the embodiments described below are examples of how the present disclosure may be implemented, and the present disclosure is not limited to the specific configurations described below. When implementing the present disclosure, specific configurations according to the embodiments may be appropriately adopted.

[0017] 1 is a block diagram showing an example of the configuration of a target recognition device 10 according to the present embodiment. The target recognition device 10 according to the present embodiment is a target recognition device for autonomous driving assistance, is mounted on a vehicle, and performs target recognition processing to recognize targets in the external environment of the vehicle. The target recognition device 10 according to the present embodiment includes at least an imaging unit 20 and a control device 50.

[0018] The imaging unit 20 is, for example, a camera, which captures an image of the external environment around the vehicle and obtains an image of the scenery outside the vehicle (i.e., an outside-vehicle image). The image data captured by the imaging unit 20 is sent to the control device 50. The installation positions and number of imaging units 20 are not particularly limited. For example, the imaging units 10 are installed on the front, side, and rear of the vehicle. The imaging unit 20 includes an imaging element 20a equipped with full HD to 8K pixels.

[0019] The control device 50 is a vehicle processor that performs autonomous driving assistance based on an outside-of-vehicle image captured by the imaging unit 20. The control device 50 is a microcomputer that includes a CPU 100, a multi-core processor 200, and a storage device 300, as will be described in detail later. The control device 50 is also called an ECU (Electronic Control Unit).

[0020] The target object recognition device 10 according to this embodiment may include a LIDAR (Laser Imaging Detection and Ranging), a vehicle condition sensor, and the like. The LIDAR uses laser pulses to measure the position (distance and direction) of a reflection point. More specifically, the LIDAR sequentially outputs (scans) laser pulses in multiple directions. When the laser pulse is reflected at a reflection point, the reflected light of the laser pulse returns to the LIDAR. The LIDAR receives the reflected light of the laser pulse. The distance and direction of the reflection point can be calculated from the reception state of the reflected light. The measurement results from the LIDAR are sent to the control device 50. The vehicle condition sensor detects the state of the vehicle. For example, the vehicle condition sensor includes a wheel speed sensor, a vehicle speed sensor, and the like. The wheel speed sensor detects the rotational speed of each wheel of the vehicle. The vehicle speed sensor detects the speed of the vehicle. Vehicle condition information detected by the vehicle condition sensor is sent to the control device.

[0021] Fig. 2 is a diagram showing an example of the hardware configuration of the control device 50 included in the target object recognition device 10 according to this embodiment. Fig. 3 is a diagram showing an example of the functional layout of the control device 50 according to this embodiment in comparison with an example of the functional layout of a conventional control device.

[0022] In conventional control devices (see the left side of Figure 3), target recognition in image data captured by an onboard camera was performed by a 64-bit to 128-bit calculation accelerator such as a DSP, which performed pre-processing of the image data, and an NPU equipped with multiple MAC calculators performed DNN processing related to target recognition using DNN.As a result, data had to be transferred between the NPU, which performed the DNN processing, and the DSP, which performed the pre-processing for the DNN processing, which resulted in the problem of increased processing time (Problem 1).

[0023] Furthermore, when DNN processing in the NPU involves calculations that cannot be processed by the NPU, the data must be transferred to the CPU for processing, and the time required for this data transfer cannot be ignored (Problem 2).

[0024] Furthermore, in the above-described conventional configuration, data must be input simultaneously to a large number of computing units before and after the DNN processing and / or for calculations in some layers of the neural network, which poses a problem that DSPs, in particular, cannot efficiently execute processing that includes complex control, such as repeating if statements (Problem 3).

[0025] Furthermore, conventional NPU convolution engines are often unable to process rectangular area determination processes such as non-maximum suppression (NMS), which are almost essential for object detection post-processing. Therefore, these processes must be performed by a DSP or CPU with a low number of simultaneous calculations (Problem 4). As a result, even if a control device with specialized NPU functions can execute DNN processing at high speed, the post-processing becomes a bottleneck, resulting in a decrease in overall performance. Post-processing cannot be ignored because it increases the processing load when increasing resolution or using multiple DNN models.

[0026] As described above, in conventional control devices, the target recognition processing is shared among the DSP, CPU, and NPU, and therefore data processing cannot be performed efficiently. Therefore, the control device 50 according to the present embodiment has the following hardware configuration to solve at least one of the above problems 1 to 4 and enable efficient data processing.

[0027] The control device 50 of this embodiment is a processor on a single integrated circuit that includes a CPU 100, a multi-core processor 200, and a storage device 300. The CPU 100 is a multi-core processor control CPU that controls the multi-core processor 200 by executing a control program stored in the storage device 300. The CPU 100 is provided alongside the multi-core processor 200.

[0028] The multi-core processor 200 is a scalable processor that executes target recognition processing. That is, the multi-core processor 200 executes DNN processing using DNN, which is the main processing, pre-processing for the DNN processing, and post-processing for the DNN processing, on the image data of the outside of the vehicle input to the control device 50. Furthermore, the multi-core processor 200 executes, as the DNN processing, at least one of classification processing, detection processing, segmentation processing, and pose estimation processing, as pre-processing, at least one of water droplet and dirt removal processing, distortion correction processing, resizing processing, cropping processing, and brightness standardization processing, and as post-processing, at least one of NMS processing and Top-K processing, on the image data of the outside of the vehicle input to the control device 50. Furthermore, the multi-core processor 200, with its task schedule controlled by the CPU 100, continuously executes a series of tasks, including pre-processing on the exterior image, including at least one of water droplet and dirt removal, distortion correction, resizing, cropping, and brightness standardization; performing DNN processing on the pre-processed image, including at least one of classification, detection, segmentation, and pose estimation; and performing post-processing on the DNN-processed image, including at least one of NMS and top-K processing. Each of these processes will be described later. The multi-core processor 200 is configured to meet safety requirements for applications up to a predetermined level of safety standard (automotive safety level), i.e., ASIL (Automotive Safety Integrity Level) Level C or Level D. ASIL is a vehicle safety standard defined by the ISO 26262 standard, and includes four levels, ASIL-A to ASIL-D, depending on the safety standard. The ASIL level increases from ASIL-A to ASIL-D.That is, ASIL-D has the highest level of safety standards, ASIL-C has the second highest level of safety standards, ASIL-B has the third highest level of safety standards, and ASIL-A has the lowest level of safety standards.

[0029] The multi-core processor 200 is connected to the CPU 100 by loose coupling, which allows the multi-core processor 200 and the CPU 100 to be highly independent from each other, resulting in excellent flexibility, compatibility, scalability, ease of identifying the cause of a malfunction, and the like.

[0030] The multi-core processor 200 has multiple cores 400. The multi-core processor 200 executes DNN processing, pre-processing, and post-processing using at least two or more cores 400. The multiple cores 400 are interconnected. In this embodiment, as an example, the multiple cores 400 are interconnected in a mesh configuration. This enables parallel processing among the multiple cores 400. Instead of interconnecting the multiple cores 400 in a mesh configuration, the multiple cores 400 may be interconnected in a bus configuration, a ring configuration, a tournament configuration, or a pyramid configuration. Here, a state in which the multiple cores 400 are interconnected in a pyramidal configuration will be described with reference to FIG. 15. FIG. 15 is a diagram showing a state in which the multiple cores 400 are interconnected in a pyramidal configuration. As shown in FIG. 15, the multiple cores 400 are organized into multiple groups GR, each including two or more cores 400. In each of the multiple groups GR, the two or more cores 400 are all directly interconnected. A specific core 400GR1 in a first group GR1 among the multiple groups GR is interconnected with a specific core 400GR2 in a second group GR2 among the multiple groups GR. While FIG. 15 shows an example in which all of the multiple groups GR are directly interconnected, they may also be indirectly interconnected. By interconnecting the multiple cores 400 in a pyramidal shape in this manner, it is possible to achieve a trade-off between the complexity of the wiring and the number of hops for data transfer. Each of the multiple cores 400 has multiple processor elements (PEs) 401 and an L2 memory 402, which is a cache memory.

[0031] Each of the multiple processor elements 401 includes an ALU (Arithmetic and Logic Unit) 403, a MAC calculator 404, and an LRM (Local Register Memory) 405. The ALU 403 performs logical operations, addition, and subtraction. The MAC calculator 404 performs multiply-accumulate operations. In this embodiment, as an example, one processor element 401 is equipped with one 32-bit integer ALU 403 and eight 8-bit MAC calculators 404. A 16-bit MAC calculator may be used instead of the 8-bit MAC calculator 404. The number of ALUs 403 and the number of MAC calculators 404 are not limited to this, and there may be multiple of both.

[0032] The multiple processor elements 401 are all interconnected directly or indirectly. The processor element performing an operation is interconnected with at least one of the other processor elements 401 so that the processor element performing the operation can refer to the operation data of the other processor elements. In this embodiment, as an example, the multiple processor elements 401 are interconnected in a mesh configuration. This allows each processor element 401 to become a SIMD (Single Instruction, Multiple Data) processor, enabling parallel processing. Instead of interconnecting the multiple processor elements 401 in a mesh configuration, they may be interconnected in any of a bus, ring, tournament, or pyramid configuration.

[0033] Furthermore, because each processor element 401 has both an ALU 403 and a MAC calculator 404, convolutional and non-convolutional calculations can be performed within each processor element 401 and between multiple processor elements 401. In the case of a network in which the ALU 403 performs a high ratio of non-convolutional calculations, cost performance is high. Furthermore, because each processor element 401, which has both an ALU 403 and a MAC calculator 404, is configured to be able to refer to data from each other, there is no need to transfer some of the processing to a CPU, DPU, or other accelerator.

[0034] Furthermore, because multiple processor elements 401 are interconnected in a mesh configuration, the number of processor elements 401 can be configured, i.e., the number of processor elements 401 can be adjusted to suit the processing. Furthermore, because the multi-core processor 200 is processor-based, everything is programmable, and custom layers can also be added. Therefore, the multi-core processor 200 of this embodiment does not require a DSP. However, if float / double precision is required, a DSP may need to be used.

[0035] As described above, the control device 50 of this embodiment is a control device for a vehicle that performs autonomous driving assistance, and includes a multi-core processor 200 having a plurality of cores 400, each including a plurality of processor elements 401, and a CPU 100 that controls the multi-core processor 200. The plurality of cores 400 are interconnected, and each of the plurality of processor elements 401 includes an ALU 403 and a MAC calculator 404. The multi-core processor 200 performs, on image data of the outside of the vehicle input to the control device 50, at least one of classification processing, detection processing, segmentation processing, and pose estimation processing as DNN processing, at least one of water droplet and dirt removal processing, distortion correction processing, resizing processing, cropping processing, and brightness standardization processing as pre-processing, and at least one of NMS processing and Top-K processing as post-processing.

[0036] The control device 50 of this embodiment, having the above-described configuration, can execute both neural network calculations and their pre- and post-processing by itself without the need for data transfer between DSPs or to a CPU, enabling efficient data processing and faster processing. Specifically, data transfer between the DNN itself and some layers or pre- and post-processing can be omitted, thereby speeding up overall data processing. Furthermore, because the control device 50 of this embodiment is programmable, custom layers can be added, enabling functions to be added via software, improving functional expandability via software, such as route search and signal processing. Furthermore, it is possible to avoid the decline in development efficiency that occurs when development spans CPUs, DSPs, etc., thereby improving development efficiency.

[0037] 4 is a flowchart showing the target recognition process according to this embodiment, which is executed by the multi-core processor 200. First, image data of the outside of the vehicle acquired by the imaging unit 20 is input to the control device 50, and the input image data is subjected to RGB / Ch conversion.

[0038] In step S100, the multi-core processor 200 executes pre-processing for the DNN processing. In the pre-processing, the multi-core processor 200 performs, for example, water droplet and dirt removal processing, then distortion correction, cropping processing, resizing processing, and finally brightness standardization processing.

[0039] In the water droplet / dirt removal process, the multi-core processor 200 uses image data captured by the imaging unit 20 to determine whether water droplets / dirt are attached to the imaging unit 20, and based on the determination result, controls the water droplet / dirt removal device to remove water droplets / dirt attached to the imaging unit 20 (i.e., water droplets / dirt reflected in the vehicle exterior image). The water droplet / dirt removal device includes an air compressor, a hose, and a nozzle, and ejects compressed air generated by the air compressor through the hose and from the nozzle toward the imaging unit 20 to remove water droplets. Note that the water droplet / dirt removal device is not limited to this, and may also eject washer fluid toward the imaging unit 20 or wipe the imaging unit 20 with a camera wiper, for example.

[0040] In the distortion correction process, the multi-core processor 200 performs distortion correction, keystone correction, etc. on image data captured by the imaging unit 20, for example, in accordance with the lens characteristics of the lens included in the imaging unit 20. In the cropping process, the multi-core processor 200 performs cropping to crop a predetermined area that is a part of the image that is to be used for target recognition in the DNN process, for example, on the image data captured by the imaging unit 20. In the resizing process, the multi-core processor 200 performs resizing to resize the image data captured by the imaging unit 20 to a size that is optimal for target recognition in the DNN process, for example.

[0041] In the brightness standardization process, the multi-core processor 200 standardizes (normalizes) the brightness values ​​of the images so as to eliminate the influence of brightness distributions that differ from image to image, for example. At this time, brightness standardization is performed so as not to reduce the contrast between the object portion and the background portion, even if there is an image portion other than the object that has a distinct brightness in the background.

[0042] In step S200, the multi-core processor 200 executes DNN processing as main processing. In the DNN processing, the multi-core processor 200 performs, for example, a classification process (image classification process) that classifies external environment information included in the preprocessed image into predetermined target classifications, a detection process (target detection (object detection) process) that identifies the positions, number, and types of targets outside the vehicle from the external environment information included in the preprocessed image, a segmentation process that segments the external environment information included in the preprocessed image by targets outside the vehicle, and a pose estimation process (pose estimation process) that estimates the poses of targets included in the external environment information included in the preprocessed image.

[0043] In step S300, the multi-core processor 200 executes post-processing for the DNN processing. In the post-processing, the multi-core processor 200 determines the detection area (Top K processing, NMS processing, etc.), paints the areas, etc., using the inference result data inferred in the DNN processing, for example. The NMS processing is a process of removing some of the overlapping candidate areas of the target included in the image after the DNN processing. The Top K processing is a process of extracting the top K candidate areas with the highest confidence from the candidate areas of the target included in the image after the DNN processing. The area painting is a process of painting the image into different areas based on, for example, pixel features.

[0044] In this embodiment, as an example, the water droplet removal process of the water droplet and dirt removal process in the pre-processing of step S100 will be described. Furthermore, a method of determining a detection area using NMS as post-processing of step S300 from multiple candidate areas that are the results obtained by the inference process in the DNN process in step S200 will be described.

[0045] 5 is a block diagram showing the configuration of the target recognition device 10 and the water droplet removal system 11 according to this embodiment. The water droplet removal system 11 includes the target recognition device 10 as an image processing device, and a water droplet removal device 60.

[0046] The target object recognition device 10 determines whether or not water droplets are attached to the imaging unit 20 based on the image data acquired by the imaging unit 20, and sends the determination result to the water droplet removal device 60. Note that although the configuration of Fig. 5 is described with a focus on water droplet removal, the target object recognition device 10 also has configurations other than the function of detecting water droplets shown in Fig. 5.

[0047] The imaging unit 20 from which water droplets are removed by the water droplet removal system 11 may be, for example, an optical sensor that captures images through a lens or acquires information about targets around the vehicle. Specifically, various optical sensors may be used, such as a front camera that captures images in front of the vehicle, a side camera that captures images on the sides of the vehicle, or a backup camera that captures images behind the vehicle.

[0048] The control device 50 includes functional units, a storage unit 50a, a detection unit 50b, an estimation unit 50c, and a determination unit 50d. Each of these functional units is implemented by the multi-core processor 200 in the control device 50. In other words, these functions of the control device 50 are implemented by the multi-core processor 200 (more specifically, the specific core 400 having the L2 memory 402 storing the program) executing processing in accordance with a program stored in the L2 memory 402 provided in the specific core 400.

[0049] The memory unit 50a pre-stores various processing conditions and parameters used when the control device 50 performs detection processing, estimation processing, judgment processing, and the like. Furthermore, the memory unit 50a also stores detection results and estimation results performed within the control device 50. The detection unit 50b receives image data acquired by the imaging unit 20. The detection unit 50b performs edge detection processing on the received image data to derive the edge strength of each pixel. The detection unit 50b detects a group of pixels whose derived edge strength falls within a specific range as a candidate region that is a candidate for a water droplet region blurred by the influence of water droplets. If the candidate region is a water droplet region, this candidate region will have an approximately arc-shaped form. Therefore, it is desirable to detect only arc-shaped regions with an approximately arc-shaped form as candidate regions. The detection unit 50b stores the derived edge strength results and candidate region detection results in the memory unit 50a. The candidate regions stored in the memory unit 50a are subject to judgment by the judgment unit 50d.

[0050] The estimation unit 50c receives the candidate area detection results. Based on the candidate area detection results, the estimation unit 50c estimates a corresponding circle that inscribes the candidate area. The estimation unit 50c stores the circle estimation results in the storage unit 50a.

[0051] The determination unit 50d receives the edge strength derivation results, candidate area detection results, and estimation results of circles that inscribe the candidate areas, all stored in the storage unit 50a. Based on the extraction results, detection results, estimation results, and the like, the determination unit 50d determines whether the candidate areas stored in the storage unit 50a as determination targets are water droplet areas, and determines whether water droplets are adhering to the imaging unit 20. The determination unit 50d sends the determination results to the water droplet removal device 60.

[0052] The image processing in the control device 50 will be described in more detail below with reference to FIGS. 6 and 7. FIG.

[0053] Fig. 6 is a diagram showing an outline of image processing according to this embodiment, and Fig. 7 is a flowchart showing a processing procedure executed by the multi-coprocessor 200 of the target object recognition device 10 according to this embodiment.

[0054] The detection unit 50b receives image data acquired by the imaging unit 20. Next, the detection unit 50d detects edges in the received image data using a well-known edge detection method, such as differentiating the luminance of each pixel. The detection unit 50b derives the edge strength of each pixel in the image data. Note that in this example, edges are detected using the luminance of each pixel, but edges may also be detected for each color component value (e.g., R value, G value, B value) of each pixel. In this case, the sum or maximum value of the edge strengths for each color component value may be used as the edge strength of the pixel.

[0055] Next, the detection unit 50b detects a group of pixels whose derived edge strength is in a specific range equal to or greater than a predetermined first threshold and equal to or less than a predetermined second threshold as candidate region 1, which is a candidate for a water droplet region (step S701). At this time, it is desirable to detect only arc-shaped regions that are approximately arc-shaped as candidate region 1.

[0056] Next, the estimation unit 50c estimates a circle 1a corresponding to the detected candidate area 1 so as to inscribe the candidate area 1 (step S702). In step S702, the following method can be used as a method for estimating the circle 1a.

[0057] As shown in FIG. 6 , the estimation unit 50c extracts multiple attention points 3a-3d (hereinafter collectively referred to as attention points 3) that are the intersections of the candidate area 1 and a rectangle 1b inscribed with the detected candidate area 1. Here, for example, a rectangle with sides extending horizontally and vertically can be selected as the rectangle 1b. In this case, the attention points 3 extracted within the candidate area 1 are the smallest point in the horizontal direction (attention point 3a), the largest point in the vertical direction (attention point 3b), the largest point in the horizontal direction (attention point 3c), and the smallest point in the vertical direction (attention point 3d). Note that the rectangle 1b does not necessarily have to have sides extending horizontally and vertically; a rectangle with sides extending at an angle relative to the horizontal and vertical directions may also be selected.

[0058] Then, by estimating a center 2 that is approximately equidistant from the extracted plurality of points of interest 3, a circle 1a is estimated as a circle that has center 2 and passes through the plurality of points of interest 3. Note that the center 2 estimated here does not need to be perfectly equidistant from the plurality of points of interest 3. For example, if a perfectly equidistant point is located between pixels, multiple pixels contained around it may be estimated as center 2. Furthermore, the estimated circle 1a does not need to pass through all points of interest 3; for example, it may pass through the vicinity of some of the points of interest 3.

[0059] Next, the determination unit 50d determines whether the candidate region is a water droplet region based on the strength of the edges within the estimated circle 1a. For example, as shown in Figure 6, the determination unit 50d determines the strength of the edges of a central region 2a within a predetermined range from the center 2 (step S703).

[0060] If the average edge intensity in the central region 2a is lower than a predetermined threshold (Yes in step S703), the determination unit 50d determines that the candidate region 1 is a water droplet region (more precisely, a part of the water droplet region) (step S704). That is, the determination unit 50d determines that water droplets are attached to the imaging unit 20.

[0061] If the average edge strength in the central region 2a is higher than a predetermined threshold (No at step S703), the candidate region 1 is determined not to be a water droplet region (step S705).

[0062] Then, the determination unit 50d outputs the determination result to the water droplet removal device 60 (step S706), and the process ends.

[0063] The target recognition device 10 according to this embodiment determines whether water droplets are present by utilizing the fact that, among the changes in brightness within the circle 1a, the edge intensity is low in the central region 2a.

[0064] In the present embodiment, the central region 2a may be, for example, a circle having a center at the center 2 and a radius smaller than the radius 4 of the circle 1a. It is preferable that the central region 2a be a circle having a center at the center 2 and a radius smaller than 1 / 2 of 4, and more preferably a circle having a radius smaller than 1 / 4 of 4.

[0065] In this embodiment, the average edge strength in the central region 2a is determined, but it is also possible to determine only the edge strength of one pixel corresponding to the center 2, rather than the central region 2a. In this case, the determination unit 50d determines that candidate region 1 is a water droplet region if the edge strength of the center 2 is lower than a predetermined threshold, and determines that candidate region 1 is not a water droplet region if the edge strength of the center 2 is higher than the predetermined threshold.

[0066] In step 703 of this embodiment, the determination unit 50d makes the determination using the edge strength derived by the detection unit 50b. However, the determination unit 50d may make the determination using a parameter different from the edge strength derived by the detection unit 50b (for example, a parameter that substantially indicates the edge strength, such as spatial frequency).

[0067] If something other than water droplets is attached to the imaging unit 20, the candidate area 1, which is a collection of pixels whose edge intensity falls within a specific range, will not be arc-shaped, but will be rectangular or circular, for example. Furthermore, the edge intensity will not be low at the position corresponding to the central area 2a of the candidate area 1. Therefore, if the candidate area 1 is rectangular or circular, or if the edge intensity of the central area of ​​the candidate area 1 is high, it can be determined that what is attached to the imaging unit 20 is not water droplets but other dirt.

[0068] (NMS Processing) FIG. 8 is a block diagram showing the configuration of the target object recognition device 10 according to this embodiment. The control device 50 includes, as functional units, a storage unit 50a′, an acquisition unit 50b′, an inference unit 50c′, an NMS unit 50d′, and an output unit 50e′. These functional units are implemented by the multi-core processor 200 in the control device 50. That is, these functions of the control device 50 are implemented by the multi-core processor 200 (more specifically, the specific core 400 including the L2 memory 402 storing the program) executing processing in accordance with a program stored in the L2 memory 402 included in the specific core 400. Note that while FIG. 8 describes the configuration with a focus on the NMS processing, the target object recognition device 10 also has components other than the NMS processing shown in FIG. 8.

[0069] The acquisition unit 50 b ′ acquires image data captured by the imaging unit 20 .

[0070] The inference unit 50c' performs inference processing. Specifically, the inference unit 50c' performs object detection using a trained model based on machine learning. The inference unit 50c' performs processing to identify the position (area) and type of a predetermined object from image data using a known object detection algorithm. For example, when the target recognition device 10 is used to control automatic driving of a vehicle, the type of object to be detected by the target recognition device 10 is an object visible from the vehicle. Specifically, the detection targets include automobiles, motorcycles, people, bicycles, traffic lights, white lines on the road, etc.

[0071] Examples of known object detection algorithms include R-CNN, Fast R-CNN, Faster R-CNN, YOLO, and SSD. In this embodiment, the inference unit 50c' uses YOLO as the object detection algorithm. The results obtained by the inference process by the inference unit 50c' include the type of object, a bounding box indicating the area of ​​the object, and a confidence score indicating the probability of the object's existence. The confidence score is, for example, a numerical value between 0 and 1, and the closer the value is to "1," the higher the confidence.

[0072] Note that the bounding box obtained by the inference unit 50c' is a result obtained at a stage prior to the final output of the object detection result, and is not a definitive result. For this reason, the bounding box obtained by the inference unit 50c' is referred to as a candidate area. As can be seen from the above, the control device 50 extracts candidate areas of objects from image data acquired by the imaging unit 20. The object detection method for detecting objects from image data includes an extraction step of extracting candidate areas of objects from the image data. There is not necessarily only one candidate area for each object, and multiple candidate areas may be extracted.

[0073] The NMS unit 50d' performs NMS processing, which is processing to remove parts of overlapping candidate areas. Specifically, overlapping candidate areas refer to multiple candidate areas that partially overlap each other. The NMS processing removes at least one candidate area from the multiple candidate areas that overlap each other. The NMS processing calculates the IoU (Intersection over Union), which is the overlap ratio between the candidate areas. In other words, the control device 50 calculates the overlap ratio between the candidate areas. The object detection method includes a calculation step of calculating the overlap ratio between the candidate areas.

[0074] IoU can be calculated using the following formula (1): IoU = X / Y (1) X: Area of ​​overlap between two candidate regions Y: Total area of ​​the two overlapping candidate regions IoU is an index that indicates how much the candidate regions overlap, and the closer the value is to "1", the greater the overlap.

[0075] Fig. 9 is a schematic diagram for explaining IoU. In Fig. 9, it is assumed that an automobile 91, which is the detection target, is included in the image, and a first candidate region 92 and a second candidate region 93 have been extracted for the automobile 91. The first candidate region 92 and the second candidate region 93 are both rectangular regions. Note that the shape of the candidate regions is not limited to a rectangle and may be other shapes.

[0076] 9 , X in the above formula (1) is the area of ​​the overlapping portion (the hatched portion) of the first candidate region 92 and the second candidate region 93. Y in formula (1) is obtained by adding the area of ​​the first candidate region 92 and the area of ​​the second candidate region 93 and subtracting the area of ​​the overlapping portion of the first candidate region 92 and the second candidate region 93 (the area calculated as X above) from the sum.

[0077] Specifically, the NMS unit 50d' determines which candidate areas to retain from among the overlapping candidate areas based on the calculated IoU value and thresholds included in a threshold table 501a' stored in the storage unit 50a', which includes thresholds used in the object detection process. That is, the control device 50 determines which candidate areas to retain from among the overlapping candidate areas based on the overlapping ratio between the candidate areas and the thresholds stored in the storage unit 50a'. The object detection method includes a determination step of determining which candidate areas to retain from among the overlapping candidate areas based on the overlapping ratio and a preset threshold. The candidate areas determined to be retained are areas that are ultimately output as object areas. The areas that are ultimately output as object areas are areas that are displayed, for example, as bounding boxes.

[0078] In detail, if the calculated IoU value exceeds the threshold value included in the threshold value table 501a', part of the candidate area is removed. The candidate area that is not removed is the candidate area to be retained as described above. That is, the control device 50 removes part of the candidate area if the overlap rate between the candidate areas exceeds the threshold value stored in the storage unit 50a'. With this configuration, it is possible to remove part of multiple candidate areas that have a large overlap rate and are likely to be candidate areas for the same object. That is, unnecessary candidate areas can be appropriately removed, allowing for appropriate object detection.

[0079] More specifically, among the overlapping candidate regions, the candidate region with the highest reliability score is selected as the candidate region to be retained, and the rest are removed. That is, the control device 50 removes all but the candidate region with the highest reliability score from among the overlapping candidate regions. With this configuration, it is possible to retain the most reliable region from among the regions estimated to be candidate regions for the same object, thereby improving the reliability of object detection.

[0080] In this embodiment, the process of removing a portion of overlapping candidate areas is performed on candidate areas estimated to be the same type of object. The IoU is calculated for candidate areas that are estimated to be the same type of object, among the candidate areas that are determined to overlap with each other. The IoU is not calculated for candidate areas that are estimated to be different types of object, even if they overlap with each other. That is, in this embodiment, the control device 50 calculates the overlap ratio for candidate areas of the same type of object. This configuration makes it possible to simplify the NMS process as much as possible and remove unnecessary candidate areas.

[0081] In the example shown in FIG. 9 , it is assumed that the object type of both the first candidate region 92 and the second candidate region 93 is estimated to be an automobile. That is, the first candidate region 92 and the second candidate region 93 are the targets for IoU calculation. The IoU values ​​of the first candidate region 92 and the second candidate region 93 exceed a predetermined threshold. Furthermore, it is assumed that the reliability score of the first candidate region 92 is 0.9, and the reliability score of the second candidate region 93 is 0.5. In this case, of the extracted first candidate region 92 and second candidate region 93, the first candidate region 92, which has the highest reliability score, is retained, and the second candidate region 93 is removed. Note that in the example shown in FIG. 9 , the IoU of the first candidate region 92 and the second candidate region 93 is calculated by comparing the bounding box with the highest score (not shown).

[0082] 9 shows an example in which two candidate regions of the same type overlap, but this is merely an example. There may also be candidate regions in which no overlapping candidate regions exist. A candidate region that does not have any overlapping candidate regions is determined as the candidate region to be retained as described above, and is not removed.

[0083] It is also possible that three or more candidate regions of the same type overlap. In this case, two candidate regions are selected from all overlapping candidate regions, and it is determined whether their IoU values ​​exceed a threshold. If the threshold is exceeded, one candidate region is removed according to its reliability score. If the threshold is not exceeded, both candidate regions are retained. Next, two candidate regions are selected from the remaining candidate regions in a combination different from the previously selected combination, and the IoU values ​​are compared with the threshold and, if necessary, the candidate regions are removed. Subsequently, the same processing as above is performed on the remaining candidate regions. The above processing is repeated for candidate regions whose IoU values ​​exceed the threshold until only one candidate region remains. This allows the number of candidate regions targeting the same object to be narrowed down to one. Furthermore, the remaining candidate regions can be retained as regions targeting different objects.

[0084] The output unit 50e' outputs the results obtained by the inference process and the NMS process described above. The output unit 50e' outputs the final results of the position and type of the detected object to, for example, a display device or a control device that controls the vehicle.

[0085] Next, the thresholds stored in the storage unit 50a' will be described in detail. In this embodiment, the thresholds are stored in the form of a threshold table 501a'. FIG. 10 is a diagram showing an example of the threshold table 501a'. In the example shown in FIG. 10, the object recognition device 10 detects automobiles, people, bicycles, and traffic lights from image data captured by the imaging unit 20 that captures images of the surroundings of the host vehicle. As shown in FIG. 10, thresholds are set for each type of object. In the example shown in FIG. 10, separate thresholds are set for automobiles, people, bicycles, and traffic lights.

[0086] It is conceivable to set the same threshold value for all objects regardless of the type of object. However, such a configuration would make it difficult to set an appropriate threshold value for all objects due to differences in size, shape, etc. depending on the type of object. As a result, it is conceivable that the NMS process will remove too many candidate regions, resulting in undetected objects. Alternatively, it is conceivable that the NMS process will not remove candidate regions that should have been removed, resulting in erroneous detection of objects. In this regard, if the configuration is such that threshold values ​​are set for each type of object, as in the present embodiment, it is possible to appropriately determine the threshold value according to the type of object, thereby reducing the possibility of undetection or erroneous detection of objects. In other words, it is possible to improve object detection performance.

[0087] In this embodiment, since candidate areas estimated to be the same type of object are targeted and some of the overlapping candidate areas are removed, a threshold is set for objects of the same type, but not for objects of different types. For example, a threshold is set for cars, but not for cars and people.

[0088] The threshold value is determined in advance, for example, by using the target object recognition device 10 or by using a device other than the target object recognition device 10, and is stored in the storage unit 50a'. For example, the control device 50 of the target object recognition device 10 may determine the threshold value and store it in the storage unit 50a'. Alternatively, for example, the threshold value may be determined by another device and distributed from that device to the target object recognition device 10. Instead of distribution, the threshold value determined by the other device may be stored in the storage unit 50a' using a portable recording medium. Alternatively, for example, the threshold value may be determined by the other device and stored in the storage unit 50a' by manual input.

[0089] For example, the threshold value may be determined by experimentally changing a number of values, performing an object detection process for each value, and calculating an evaluation index for the results of each object detection process. The object detection process is a process that includes the above-described inference process and NMS process. The evaluation index for the object detection process may be mAP (mean average precision). For example, the threshold value may be determined so that the mAP is the highest. By using this method, an appropriate threshold value can be obtained for each type of object, thereby improving object detection performance.

[0090] The threshold may be determined by, for example, a known technique such as hyperparameter tuning. The threshold may be determined by, for example, Bayesian optimization, which is a type of hyperparameter tuning. In this embodiment, the threshold is a value in the range from 0 to 1. In Bayesian optimization, a plurality of values ​​(for example, 10 values) are randomly selected in the range from 0 to 1, and an object detection process is performed on these values ​​to calculate an evaluation index (mAP). Depending on the result of the evaluation index, a process of determining the next value to be examined is repeated, and an optimal threshold is determined.

[0091] FIG. 11 is a flowchart showing an example of an object detection process executed by the target recognition device 10 according to this embodiment.

[0092] In step S111, the control device 50 initializes the object detection model. The object detection model is the trained model described above. For example, the control device 50 reads out weight parameters stored in the storage unit 50a'. Once the initialization of the object detection model is complete, the process proceeds to the next step S112.

[0093] In step S112, the control device 50 reads out the threshold value table 501a' stored in the storage unit 50a'. After reading out the threshold value table 501a' is completed, the process proceeds to the next step S113.

[0094] In step S113, the control device 50 starts an inference processing loop. The inference processing loop includes processing in step S1131, processing in step S1132, processing in step S1133, processing in step S1134, and processing in step S1135.

[0095] In step S1131, the acquisition unit 50b' acquires image data input from the imaging unit 20. Specifically, the acquisition unit 50b' acquires image data at a predetermined cycle. The predetermined cycle is determined appropriately depending on the purpose of object detection, etc. Once the image data has been acquired, the process proceeds to the next step, S1132.

[0096] In step S1132, the inference unit 50c' performs inference processing using an object detection model on the image data. Through the inference processing, a candidate area of ​​the object being detected, a reliability score, and the type of the object are obtained from the image data. FIG. 12 is a schematic diagram showing an example of the results of the inference processing by the inference unit 50c'. In the example shown in FIG. 12, automobiles and traffic lights are included in the detection target objects, and automobiles and traffic lights are detected as object types. In detail, two candidate areas 400a and 400b classified as automobiles are extracted. Furthermore, two candidate areas 500a and 500b classified as traffic lights are extracted. When the inference processing by the inference unit 50c' is completed, the process proceeds to the next step S1133.

[0097] Hereinafter, for ease of distinction, one of the two candidate areas 400a, 400b classified as automobiles will be referred to as the first vehicle candidate area 400a, and the other as the second vehicle candidate area 400b. Also, one of the two candidate areas 500a, 500b classified as traffic lights will be referred to as the first traffic light candidate area 500a, and the other as the second traffic light candidate area 500b.

[0098] In step S1133, the NMS unit 50d' performs the above-described NMS processing on the inference result obtained in step S1132. Here, it is assumed that the inference result shown in Fig. 12 is obtained, and the threshold value shown in Fig. 10 is used in the NMS processing.

[0099] The NMS unit 50d' calculates the IoU for the first vehicle candidate region 400a and the second vehicle candidate region 400b, which are classified as automobiles, and determines whether the calculated IoU exceeds 0.61, which is the threshold value for vehicles obtained from the threshold table 501a'. If the IoU value exceeds 0.61, the NMS unit 50d' determines that the two vehicle candidate regions 400a and 400b represent the same vehicle region. In this case, the NMS unit 50d' removes the one with the smaller reliability score from the first vehicle candidate region 400a or the second vehicle candidate region 400b. If the IoU value is 0.61 or less, the NMS unit 50d' determines that the first vehicle candidate region 400a and the second vehicle candidate region 400b represent different vehicle regions and decides to retain the two vehicle candidate regions 400a and 400b.

[0100] The NMS unit 50d' also calculates the IoU for the first and second traffic light candidate areas 500a and 500b, which are classified as traffic lights, and determines whether the calculated IoU exceeds 0.62, a threshold value for traffic lights obtained from the threshold table 121. If the IoU value exceeds 0.62, the NMS unit 50d' determines that the two traffic light candidate areas 500a and 500b represent the same traffic light area. In this case, the NMS unit 113 removes the one with the smaller reliability score from the first and second traffic light candidate areas 500a and 500b. If the IoU value is 0.62 or less, the NMS unit 50d' determines that the first and second traffic light candidate areas 500a and 500b represent areas for different traffic lights and decides to retain the two traffic light candidate areas 500a and 500b.

[0101] FIG. 13 is a schematic diagram showing an example of the results of NMS processing by the NMS unit 50d'. FIG. 13 is based on the assumption that the inference results shown in FIG. 12 are obtained. The IoU values ​​calculated for the first vehicle candidate area 400a and the second vehicle candidate area 400b exceed the vehicle-to-vehicle threshold of 0.61. The IoU values ​​calculated for the first traffic light candidate area 500a and the second traffic light candidate area 500b exceed the traffic light-to-traffic light threshold of 0.62.

[0102] 13, as a result of the NMS process, of the two vehicle candidate areas 400a, 400b estimated to represent the same vehicle area, only the first vehicle candidate area 400a, which has the highest reliability score, remains. Also, as a result of the NMS process, of the two traffic light candidate areas 500a, 500b estimated to represent the same traffic light area, only the first traffic light candidate area 500a, which has the highest reliability score, remains. Once the NMS process is complete, the process proceeds to the next step, S1134.

[0103] In step S1134, the output unit 50e' outputs the final result of the object detection process performed on the input image. In the examples shown in Figures 12 and 13, the final result of the object detection process is a result in which one bounding box is attached to each of the automobile and the traffic light. Once the output process is complete, the process proceeds to the next step, S1135.

[0104] In step S1135, the control device 50 checks whether processing has been performed on the final frame. That is, the control device 50 checks whether processing has been completed on all image data. If processing has been completed on all image data (Yes in step S1135), the object detection processing shown in FIG. 11 ends. On the other hand, if processing has not been completed on all image data (No in step S1135), the inference processing loop including the processing of steps S1131 to S1135 is repeated.

[0105] In this embodiment, the NMS process is used to determine the detection area, but the detection area may also be determined using the Top K process, which determines the detection area by extracting only the top K classes with the highest certainty of belonging to the class.

[0106] (Vehicle driving control system) The result of the target recognition processing can be used for vehicle driving control. For example, the result of the target recognition processing can be used for automatic driving control. In automatic driving control, for example, whether or not to execute a lane change is determined based on the recognition result of surrounding vehicles. Therefore, improving the accuracy of the target recognition processing is important for automatic driving control. Furthermore, the result of the target recognition processing may be used for driving assistance control such as ACC (Adaptive Cruise Control) and LTA (Lane Tracing Assist).

[0107] 14 is a block diagram showing an example of the configuration of a vehicle driving control system 140 according to this embodiment. The vehicle driving control system 140 is mounted on a vehicle and performs vehicle driving control to control the driving of the vehicle.

[0108] More specifically, the vehicle driving control system 140 includes a driving device 70 in addition to the above-described target object recognition device 10. The driving device 70 includes a steering device, a drive device, a braking device, etc. The steering device steers the wheels. The drive device is a power source that generates driving force. The drive device is, for example, an electric motor or an engine. The braking device generates braking force.

[0109] The control device 50 performs vehicle driving control based on the result of the target recognition processing by the target recognition device 10. The vehicle driving control includes steering control and acceleration / deceleration control. The control device 50 performs steering control by appropriately operating the steering device. The control device 50 also performs acceleration / deceleration control by appropriately operating the drive device and the braking device.

[0110] (Other Embodiments) Although the embodiments of the present disclosure have been described above, the present disclosure should not be construed as being limited to the above-described embodiments, and can be applied to various embodiments and combinations within the scope that does not deviate from the gist of the present disclosure.

[0111] The control unit and the method described herein may be implemented by a special-purpose computer configured by configuring a processor and memory programmed to execute one or more functions embodied in a computer program. Alternatively, the control unit and the method described herein may be implemented by a special-purpose computer configured by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the control unit and the method described herein may be implemented by one or more special-purpose computers configured by combining a processor and memory programmed to execute one or more functions with a processor configured with one or more hardware logic circuits. Furthermore, the computer program may be stored in a computer-readable non-transitory tangible recording medium as instructions to be executed by a computer. As described above, the present disclosure is not limited to the above embodiments and can be implemented in various forms without departing from the spirit of the present disclosure.

[0112] It may be provided in the following manner.(Aspect 1) A processor (50) on an integrated circuit used in a vehicle that performs autonomous driving assistance based on an outside image captured by a camera that captures the external environment of the vehicle, comprising: a multi-core processor (200) having a plurality of cores (400) each including a plurality of processor elements (401); and a CPU (100) that controls the multi-core processor, wherein the plurality of cores are interconnected in the multi-core processor, and each of the plurality of processor elements includes an ALU (403) that performs logical operations, addition, and subtraction, and a MAC (Media Access Control) unit (404) that performs multiply-and-accumulate operations, and is further interconnected with at least one of the other processor elements so that operation data of the other processor elements can be referenced during the operation, and the CPU controls a task schedule of the multi-core processor, and within the multi-core processor, performing preprocessing on the vehicle exterior image including at least one of a water droplet / dirt removal process for removing water droplets / dirt captured in the vehicle exterior image, a distortion correction process for correcting distortion of the vehicle exterior image, a resizing process for changing the size of the vehicle exterior image, a cropping process for cropping a predetermined area from the vehicle exterior image, and a brightness standardization process for standardizing the brightness of the vehicle exterior image; further performing DNN processing on the preprocessed image including at least one of a classification process for classifying external environment information included in the preprocessed image into predetermined target classifications, a detection process for identifying the positions, number, and types of targets outside the vehicle from the external environment information included in the preprocessed image, a segmentation process for segmenting the external environment information included in the preprocessed image by targets outside the vehicle, and a pose estimation process for estimating the poses of targets included in the external environment information included in the preprocessed image; and a processor on an integrated circuit that continuously operates a series of tasks including a process of performing post-processing on the image after the DNN processing, the post-processing comprising at least one of NMS processing that removes some of the overlapping candidate regions from the candidate regions of the target included in the image after the DNN processing, and top K processing that extracts the top K candidate regions with the highest confidence from the candidate regions.

[0113] (Aspect 2) A control device (50) for a vehicle that performs autonomous driving assistance, comprising: a multi-core processor (200) having a plurality of cores (400) each including a plurality of processor elements (401); and a CPU (100) that controls the multi-core processor, wherein the plurality of cores are interconnected, and each of the plurality of processor elements includes an ALU (403) and a MAC calculator (404), and the multi-core processor performs, on image data of the outside of the vehicle input to the control device, at least one of classification processing, detection processing, segmentation processing, and pose estimation processing as DNN processing using a DNN (Deep Neural Network), and at least one of water droplet and dirt removal processing, distortion correction processing, resizing processing, cropping processing, and brightness standardization processing as preprocessing for the DNN processing. A control device that executes at least one of an NMS process and a top K process as a post-processing for the DNN process.

[0114] According to this embodiment, the control device can execute both neural network calculations and their pre- and post-processing using only the control device 50, eliminating the need for data transfer between DSPs or to the CPU. This allows for efficient data processing and faster processing times. Specifically, data transfer between the DNN itself and some layers or pre- and post-processing can be omitted, thereby speeding up overall data processing. Furthermore, because the control device 50 of this embodiment is programmable, custom layers can be added, enabling functions to be added using software, improving functional expandability through software, such as route search and signal processing. Furthermore, it is possible to avoid the decline in development efficiency that occurs when development spans multiple CPUs, DSPs, etc., thereby improving development efficiency.

[0115] (Aspect 3) The control device according to aspect 2, wherein the control device is a processor on a single integrated circuit.

[0116] According to this aspect, both the neural network calculations and the pre- and post-processing can be executed by a single processor on an integrated circuit.

[0117] (Aspect 4) The control device according to aspect 2 or 3, wherein the multi-core processor is connected to the CPU by loose coupling.

[0118] According to this aspect, the multi-core processor and the CPU are highly independent from each other, which provides excellent flexibility, compatibility, scalability, and ease of identifying the cause of a malfunction.

[0119] (Aspect 5) The control device according to any one of aspects 2 to 4, wherein the multi-core processor performs the DNN processing, the pre-processing, and the post-processing on the image data using at least two or more cores of the plurality of cores.

[0120] According to this aspect, the control device can perform DNN processing, pre-processing, and post-processing on image data for autonomous driving assistance using at least two or more cores.

[0121] (Aspect 6) The control device according to any one of aspects 2 to 5, wherein the plurality of processor elements are interconnected such that a processor element among the plurality of processor elements that is performing a calculation can refer to calculation data of the other processor elements.

[0122] According to this aspect, the number of processor elements can be adjusted to suit the processing.

[0123] (Aspect 7) The control device according to aspect 6, wherein the plurality of processor elements are interconnected in any one of a bus, mesh, ring, and tournament topology.

[0124] According to this aspect, the number of processor elements can be adjusted more flexibly to suit the processing.

[0125] (Aspect 8) The control device according to aspect 6, wherein the plurality of processor elements are comprised of a plurality of groups, each of which includes two or more processor elements, and in each of the plurality of groups, the two or more processor elements are all directly interconnected, and a specific processor element in a first group of the plurality of groups is interconnected with a specific processor element in a second group of the plurality of groups.

[0126] According to this aspect, the number of processor elements can be adjusted more flexibly to suit the processing.

[0127] (Aspect 9) The control device according to any one of aspects 2 to 8, wherein the plurality of cores are interconnected in any one of a bus, mesh, ring, and tournament topology.

[0128] According to this aspect, parallel processing among multiple cores becomes possible.

[0129] (Aspect 10) The control device according to any one of Aspects 2 to 8, wherein the plurality of cores are made up of a plurality of groups each including two or more cores, and in each of the plurality of groups, the two or more cores are all directly interconnected, and a specific core in a first group among the plurality of groups is interconnected with a specific core in a second group among the plurality of groups.

[0130] According to this aspect, parallel processing among multiple cores becomes possible.

[0131] (Aspect 11) A target recognition device (10) mounted on a vehicle, comprising: an imaging unit (20) that acquires image data of an area outside the vehicle; and the control device (50) according to any one of aspects 2 to 10.

[0132] (Aspect 12) The target recognition device according to claim 11, wherein the imaging unit is provided on the vehicle so as to be able to capture an image of a scene outside the vehicle, and includes an imaging element (20a) having full HD to 8K pixels.

[0133] (Aspect 13) A vehicle driving control system (140) comprising: the target object recognition device (10) according to Aspect 11 or 12; and a driving device (70), wherein the control device performs vehicle driving control to control the driving device based on a result of the target object recognition processing by the target object recognition device.

[0134] Although the present disclosure has been described with reference to the embodiments, it is understood that the present disclosure is not limited to the embodiments or structures. The present disclosure also encompasses various modifications and equivalent modifications. In addition, various combinations and forms, including only one element, more than one element, or less than one element, are also within the scope and spirit of the present disclosure.

Claims

1. A processor (50) on an integrated circuit used in a vehicle that performs automatic driving assistance based on an outside image captured by a camera that captures the external environment of the vehicle, comprising: a multi-core processor (200) having a plurality of cores (400) each including a plurality of processor elements (401); and a CPU (100) that controls the multi-core processor, wherein the plurality of cores are interconnected in the multi-core processor, and each of the plurality of processor elements includes an ALU (403) that performs logical operations, addition, and subtraction, and a MAC calculator (404) that performs multiply-and-accumulate operations, and is further interconnected with at least one of the other processor elements so that the calculation data of the other processor elements can be referenced during calculation, and the CPU controls the task schedule of the multi-core processor, and within the multi-core processor, performing pre-processing on the vehicle exterior image including at least one of a water droplet / dirt removal process for removing water droplets / dirt captured in the vehicle exterior image, a distortion correction process for correcting distortion of the vehicle exterior image, a resizing process for changing the size of the vehicle exterior image, a cropping process for cropping a predetermined area from the vehicle exterior image, and a luminance standardization process for standardizing the luminance of the vehicle exterior image; further performing DNN processing on the pre-processed image including at least one of a classification process for classifying external environment information included in the pre-processed image into a predetermined target classification, a detection process for identifying the positions, number, and types of targets outside the vehicle from the external environment information included in the pre-processed image, a segmentation process for dividing the external environment information included in the pre-processed image by targets outside the vehicle, and a pose estimation process for estimating the posture of targets included in the external environment information included in the pre-processed image; The processor on the integrated circuit continuously operates a series of tasks including a process of performing post-processing on the image after the DNN processing, which includes at least one of an NMS process for removing some of the overlapping candidate regions among the candidate regions for targets contained in the image after the DNN processing, and a top K process for extracting the top K candidate regions with high confidence from the candidate regions.

2. A control device (50) for a vehicle that performs automatic driving assistance, comprising: a multi-core processor (200) having a plurality of cores (400) each including a plurality of processor elements (401); and a CPU (100) that controls the multi-core processor, wherein the plurality of cores are interconnected, and each of the plurality of processor elements includes an ALU (403) and a MAC calculator (404), and the multi-core processor performs, on image data of the outside of the vehicle input to the control device, at least one of classification processing, detection processing, segmentation processing, and pose estimation processing as DNN processing using a DNN (Deep Neural Network), and at least one of water droplet / dirt removal processing, distortion correction processing, resizing processing, cropping processing, and brightness standardization processing as pre-processing for the DNN processing, A control device that executes at least one of an NMS process and a top K process as a post-processing for the DNN process.

3. The control device of claim 2, wherein the control device is a processor on a single integrated circuit.

4. The control device according to claim 2, wherein the multi-core processor is connected to the CPU by loose coupling.

5. The control device according to claim 2, wherein the multi-core processor performs the DNN processing, the pre-processing, and the post-processing on the image data using at least two or more of the multiple cores.

6. The control device according to claim 2, wherein said plurality of processor elements are interconnected so that a processor element among said plurality of processor elements which is performing a calculation can refer to calculation data of the other processor elements.

7. The control device according to claim 6, wherein the plurality of processor elements are interconnected in any one of a bus, mesh, ring, and tournament topology.

8. The control device according to claim 6, wherein the plurality of processor elements are comprised of a plurality of groups, each of which includes two or more processor elements, and in each of the plurality of groups, the two or more processor elements are all directly interconnected, and a specific processor element in a first group among the plurality of groups is interconnected with a specific processor element in a second group among the plurality of groups.

9. The control device according to claim 2, wherein the multiple cores are interconnected in any one of a bus, mesh, ring, or tournament configuration.

10. The control device of claim 2, wherein the plurality of cores are comprised of a plurality of groups each including two or more cores, and in each of the plurality of groups, the two or more cores are all directly interconnected, and a specific core in a first group among the plurality of groups is interconnected with a specific core in a second group among the plurality of groups.

11. A target recognition device (10) mounted on a vehicle, comprising: an imaging unit (20) that acquires image data of the outside of the vehicle; and a control device (50) according to any one of claims 2 to 10.

12. The target recognition device according to claim 11, wherein the imaging unit is mounted on the vehicle so as to be able to capture images of the scenery outside the vehicle, and includes an imaging element having full HD to 8K pixels.

13. A vehicle driving control system (140) comprising: a target object recognition device (10) according to claim 11; and a driving device (70), wherein the control device performs vehicle driving control to control the driving device based on the result of the target object recognition processing by the target object recognition device.

Images