Information processing device, information processing system, information processing program, and information processing method

Abstract

The objective of the present invention is to enable a reduction in processing load when a plurality of different sensors are used. An information processing device according to the present disclosure is provided with a recognition processing unit (15, 40b) which performs recognition processing for recognizing a target object, by adding to the output of a first sensor (23) region information generated in accordance with an object likelihood detected in the course of object recognition processing based on the output of a second sensor (21) different from the first sensor.

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing system, an information processing program, and an information processing method. [Background technology]

[0002] There are known technologies that use sensors such as image sensors and millimeter wave radars to detect objects. There are various detection methods for sensors to detect objects, and each method is suitable for different situations. For this reason, a technology has been proposed that uses multiple sensors with different detection methods to detect objects. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] WO 17 / 057056 Summary of the Invention [Problem to be solved by the invention]

[0004] When multiple sensors with different detection methods are used, if the detection process is performed using all of the outputs of the multiple sensors, the load of the detection process may become large. In order to avoid this increase in the load of the detection process, a method of setting a detection window for the output of the sensor and limiting the range of the detection process is conceivable. However, in the past, the method of setting this detection window had not been defined.

[0005] An object of the present disclosure is to provide an information processing device, an information processing system, an information processing program, and an information processing method that are capable of reducing the processing load when a plurality of different sensors are used. [Means for solving the problem]

[0006] The information processing device according to the present disclosure includes a recognition processing unit that performs a recognition process for recognizing an object by adding area information generated according to an object likelihood detected in the course of object recognition processing based on an output of a second sensor different from the first sensor to an output of a first sensor. The recognition processing unit performs the recognition processing using an object recognition model obtained by machine learning, and the object recognition model generates the region information in one or more layers included in a first convolutional layer generated based on an output of the second sensor, and adds the generated region information to one or more layers of a second convolutional layer generated based on the output of the first sensor and having one-to-one correspondence with the one or more layers that generated the region information. . [Brief description of the drawings]

[0007] [Figure 1] 1 is a block diagram showing an example of a schematic configuration of a vehicle control system; [Diagram 2] FIG. 2 is a functional block diagram illustrating an example of a function of an outside-vehicle information detection unit in the vehicle control system. [Diagram 3] FIG. 2 is a diagram illustrating an example of the configuration of an object recognition model used in a recognition processing unit. [Figure 4] FIG. 1 is a block diagram showing an example of the configuration of a learning system. [Diagram 5] 2 is a block diagram showing an example of a hardware configuration of an outside-vehicle information detection unit applicable to each embodiment. FIG. [Figure 6] FIG. 1 is a schematic diagram illustrating an object recognition model according to an embodiment of the present disclosure. [Figure 7] FIG. 2 is a diagram showing a configuration of an example of an object recognition model according to the first embodiment. [Figure 8] 4 is a diagram illustrating an example of a configuration of a synthesis unit according to the first embodiment. FIG. [Figure 9] FIG. 2 is a schematic diagram for explaining a first example of an attention map based on an object recognition model according to the first embodiment. [Figure 10] 4 is a schematic diagram for explaining a second example of an attention map based on an object recognition model according to the first embodiment. FIG. [Figure 11] FIG. 11 is a diagram illustrating an example of a configuration of an object recognition model according to a second embodiment. [Figure 12] FIG. 13 is a diagram illustrating an example of a configuration of an object recognition model according to a third embodiment. [Figure 13] FIG. 13 is a diagram illustrating an example of a configuration of a synthesis unit according to the third embodiment. [Figure 14]FIG. 13 is a diagram illustrating an example of a configuration of an object recognition model according to a fourth embodiment. [Figure 15] FIG. 13 is a diagram illustrating an example of a configuration of an object recognition model according to a fifth embodiment. [Figure 16] FIG. 13 is a block diagram showing a first example of an outside-vehicle information detection unit and a data acquisition unit according to a sixth embodiment. [Figure 17] FIG. 23 is a block diagram showing a second example of an outside-vehicle information detection unit and a data acquisition unit according to the sixth embodiment. [Figure 18] FIG. 23 is a block diagram showing a third example of an outside-vehicle information detection unit and a data acquisition unit according to the sixth embodiment. [Figure 19] FIG. 23 is a block diagram showing a fourth example of an outside-vehicle information detection unit and a data acquisition unit according to the sixth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0009] Hereinafter, embodiments of the present disclosure will be described in the following order. 1. Technologies Applicable to Each Embodiment 1-1. Example of an in-vehicle system 1-2. Overview of functions 1-3. Hardware configuration example 2. Overview of the embodiments of the present disclosure 3. First embodiment 3-1.Specific examples 4. Second embodiment 5. Third embodiment 6. Fourth embodiment 7. Fifth embodiment 8. Sixth embodiment 8-1. First Example 8-2. Second Example 8-3.Third Example 8-4.Fourth Example 8-5. Fifth Example 8-6. Sixth Example

[0010] [1. Technologies Applicable to Each Embodiment] Prior to describing each embodiment of the present disclosure, in order to facilitate understanding, techniques applicable to each embodiment of the present disclosure will be described.

[0011] (1-1. Example of an in-vehicle system) First, an in-vehicle system applicable to each embodiment of the present disclosure will be described in brief. Fig. 1 is a block diagram showing a schematic configuration example of a vehicle control system which is an example of an in-vehicle system applicable to each embodiment of the present disclosure.

[0012] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in Fig. 1, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside-vehicle information detection unit 10, an inside-vehicle information detection unit 12040, and an integrated control unit 12050. Also, as functional configurations of the integrated control unit 12050, a microcomputer 12051, an audio / video output unit 12052, and an in-vehicle network I / F (interface) 12053 are illustrated.

[0013] The drivetrain control unit 12010 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 12010 functions as a control device for a drive force generating device for generating a drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.

[0014] The body system control unit 12020 controls the operation of various devices mounted on the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as head lamps, back lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves or signals of various switches transmitted from a portable device that replaces a key may be input to the body system control unit 12020. The body system control unit 12020 receives the input of these radio waves or signals and controls the door lock device, power window device, lamps, and the like of the vehicle.

[0015] The outside-vehicle information detection unit 10 detects information outside the vehicle equipped with the vehicle control system 12000. For example, a data acquisition unit 20 is connected to the outside-vehicle information detection unit 10. The outside-vehicle information detection unit 10 includes various sensors for acquiring the situation outside the vehicle. For example, the data acquisition unit 20 may include an optical sensor that receives visible light or non-visible light such as infrared light and outputs an electrical signal according to the amount of received light, and the outside-vehicle information detection unit 10 receives an image captured by the optical sensor. In addition, the data acquisition unit 20 may further include a sensor that acquires the outside situation by other methods, such as a millimeter wave radar, LiDAR (Light Detection and Ranging, or Laser Imaging Detection and Ranging), or an ultrasonic sensor.

[0016] The data acquisition unit 20 is provided, for example, on the front nose, side mirror, or upper part of the windshield of the vehicle 12100, with the front of the vehicle as the data acquisition direction. The outside-vehicle information detection unit 10 may perform object detection processing or distance detection processing of people, cars, obstacles, signs, characters on the road surface, etc., based on various sensor outputs received from the data acquisition unit 20.

[0017] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 that detects the state of the driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing.

[0018] The microcomputer 12051 can calculate control target values ​​for the driving force generating device, the steering mechanism, or the braking device based on the information inside and outside the vehicle acquired by the outside-of-vehicle information detection unit 10 or the inside-vehicle information detection unit 12040, and output a control command to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the inter-vehicle distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane departure warning, etc.

[0019] In addition, the microcomputer 12051 can perform cooperative control for the purpose of autonomous driving, in which the vehicle travels autonomously without relying on the driver's operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle acquired by the outside vehicle information detection unit 10 or the inside vehicle information detection unit 12040.

[0020] Furthermore, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the vehicle exterior information detection unit 10. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the vehicle exterior information detection unit 10, and perform cooperative control for the purpose of preventing glare, such as switching high beams to low beams.

[0021] The audio / video output unit 12052 transmits at least one output signal of audio and video to an output device capable of visually or audibly notifying information to passengers in the vehicle or the outside of the vehicle. In the example of Fig. 1, an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are exemplified as the output device. The display unit 12062 may include at least one of an on-board display and a head-up display, for example.

[0022] (1-2. Overview of functions) Next, an example of the function of the vehicle outside information detection unit 10 applicable to each embodiment of the present disclosure will be briefly described.

[0023] Fig. 2 is a functional block diagram of an example for explaining the function of the outside information detection unit 10 in the vehicle control system 12000 of Fig. 1. In Fig. 2, the data acquisition unit 20 includes a camera 21 and a millimeter wave radar 23. The outside information detection unit 10 includes an information processing unit 11. The information processing unit 11 includes an image processing unit 12, a signal processing unit 13, a geometric transformation unit 14, and a recognition processing unit 15.

[0024] The camera 21 includes an image sensor 22. Any type of image sensor, such as a CMOS image sensor or a CCD image sensor, can be used for the image sensor 22. The camera 21 (image sensor 22) captures an image of the area ahead of the vehicle in which the vehicle control system 12000 is mounted, and supplies the captured image (hereinafter, referred to as a captured image) to the image processing unit 12.

[0025] The millimeter wave radar 23 senses the area ahead of the vehicle, and at least a part of the sensing range overlaps with the camera 21. For example, the millimeter wave radar 23 transmits a transmission signal consisting of millimeter waves ahead of the vehicle, and receives a reception signal, which is a signal reflected by an object (reflector) ahead of the vehicle, by a receiving antenna. For example, a plurality of receiving antennas are provided at predetermined intervals in the lateral direction (width direction) of the vehicle. A plurality of receiving antennas may also be provided in the height direction. The millimeter wave radar 23 supplies data (hereinafter referred to as millimeter wave data) indicating the strength of the reception signal received by each receiving antenna in time series to the signal processing unit 13.

[0026] The transmission signal of the millimeter wave radar 23 is scanned in a predetermined angle range on a two-dimensional plane, for example, to form a fan-shaped sensing range. By scanning this range vertically, a bird's-eye view with three-dimensional information can be obtained.

[0027] The image processing unit 12 performs a predetermined image processing on the captured image. For example, the image processing unit 12 reduces the number of pixels of the captured image (reducing the resolution) by performing a process such as pixel thinning or filtering on the captured image to match the size of the image that can be processed by the recognition processing unit 15. The image processing unit 12 supplies the captured image with reduced resolution (hereinafter referred to as a low-resolution image) to the recognition processing unit 15.

[0028] The signal processing unit 13 performs predetermined signal processing on the millimeter wave data to generate a millimeter wave image which is an image showing the sensing result of the millimeter wave radar 23. The signal processing unit 13 generates millimeter wave images of multiple ch (channels) including, for example, a signal intensity image and a speed image. The signal intensity image is a millimeter wave image which shows the position of each object ahead of the vehicle and the intensity of the signal (received signal) reflected by each object. The speed image is a millimeter wave image which shows the position of each object ahead of the vehicle and the relative speed of each object with respect to the vehicle.

[0029] The geometric transformation unit 14 performs a geometric transformation on the millimeter-wave image to transform the millimeter-wave image into an image in the same coordinate system as the captured image. In other words, the geometric transformation unit 14 transforms the millimeter-wave image into an image viewed from the same viewpoint as the captured image (hereinafter referred to as a geometrically transformed millimeter-wave image). More specifically, the geometric transformation unit 14 transforms the coordinate systems of the signal intensity image and velocity image from the coordinate system of the millimeter-wave image to the coordinate system of the captured image. Note that, hereinafter, the signal intensity image and velocity image after the geometric transformation are referred to as a geometrically transformed signal intensity image and a geometrically transformed velocity image. The geometric transformation unit 14 supplies the geometrically transformed signal intensity image and the geometrically transformed velocity image to the recognition processing unit 15.

[0030] The recognition processing unit 15 performs recognition processing of an object ahead of the vehicle based on the low-resolution image, the geometric transformation signal strength image, and the geometric transformation velocity image using a recognition model previously obtained by machine learning. The recognition processing unit 15 supplies data indicating the recognition result of the object to the integrated control unit 12050 via the communication network 12001.

[0031] The target object is an object to be recognized by the recognition processing unit 15, and any object can be the target object. However, it is preferable to select an object that includes a portion having a high reflectivity of the transmission signal of the millimeter wave radar 23. The following description will be given by taking a case where the target object is a vehicle as an example.

[0032] FIG. 3 shows an example of the configuration of an object recognition model 40 used in the recognition processing unit 15. As shown in FIG.

[0033] The object recognition model 40 is a model obtained by machine learning. Specifically, the object recognition model 40 is a model obtained by deep learning, which is one type of machine learning, using a deep neural network. More specifically, the object recognition model 40 is configured by SSD (Single Shot Multibox Detector), which is one type of object recognition model using a deep neural network. The object recognition model 40 includes a feature extraction unit 44 and a recognition unit 45.

[0034] The feature extraction unit 44 includes feature extraction layers 41a to 41c, which are convolution layers using a convolutional neural network, and an adder 42. The feature extraction layer 41a extracts features of the captured image Pa, and generates a feature map (hereinafter referred to as a captured image feature map) that represents the distribution of the features in two dimensions. The feature extraction layer 41a supplies the captured image feature map to the adder 42.

[0035] The feature extraction layer 41b extracts features of the geometrically transformed signal strength image Pb and generates a feature map (hereinafter, referred to as a signal strength image feature map) that represents the distribution of the features in two dimensions. The feature extraction layer 41b supplies the signal strength image feature map to the adder .

[0036] The feature extraction layer 41c extracts features of the geometrically transformed velocity image Pc, and generates a feature map (hereinafter referred to as a velocity image feature map) that represents the distribution of the features in two dimensions. The feature extraction layer 41c supplies the velocity image feature map to the adder .

[0037] The adder 42 generates a composite feature map by adding the captured image feature map, the signal intensity image feature map, and the velocity image feature map. The adder 42 supplies the composite feature map to the recognition unit 45.

[0038] The recognition unit 45 includes a convolutional neural network. Specifically, the recognition unit 45 includes convolutional layers 43a to 43c.

[0039] The convolution layer 43a performs a convolution operation on the composite feature map. The convolution layer 43a performs an object recognition process based on the composite feature map after the convolution operation. The convolution layer 43a supplies the composite feature map after the convolution operation to the convolution layer 43b.

[0040] The convolution layer 43b performs a convolution operation on the composite feature map supplied from the convolution layer 43a. The convolution layer 43b performs an object recognition process based on the composite feature map after the convolution operation. The convolution layer 43a supplies the composite feature map after the convolution operation to the convolution layer 43c.

[0041] The convolution layer 43c performs a convolution operation on the composite feature map supplied from the convolution layer 43b. The convolution layer 43b performs a recognition process on the object based on the composite feature map after the convolution operation.

[0042] The object recognition model 40 outputs data indicating the results of object recognition by the convolutional layers 43a to 43c.

[0043] The size (number of pixels) of the composite feature map decreases from the convolutional layer 43a onwards, and is smallest at the convolutional layer 43c. The larger the size of the composite feature map, the higher the recognition accuracy of an object that is small as seen from the vehicle (camera), and the smaller the size of the composite feature map, the higher the recognition accuracy of an object that is large as seen from the vehicle. Therefore, for example, if the object is a vehicle, a large composite feature map makes it easier to recognize a small vehicle that is far away, and a small composite feature map makes it easier to recognize a large vehicle that is close by.

[0044] Fig. 4 is a block diagram showing an example configuration of learning system 30. Learning system 30 performs learning processing of object recognition model 40 in Fig. 3. Learning system 30 includes input unit 31, image processing unit 32, answer data generation unit 33, signal processing unit 34, geometric transformation unit 35, teacher data generation unit 36, and learning unit 37.

[0045] The input unit 31 includes various input devices and is used for inputting data necessary for generating teacher data, user operation, etc. For example, when a captured image is input, the input unit 31 supplies the captured image to the image processing unit 32. For example, when millimeter wave data is input, the input unit 31 supplies the millimeter wave data to the signal processing unit 34. For example, the input unit 31 supplies data indicating a user instruction input by a user operation to the correct answer data generation unit 33 and the teacher data generation unit 36.

[0046] The image processing unit 32 performs the same processing as the image processing unit 12 in Fig. 2. That is, the image processing unit 32 performs a predetermined image processing on the captured image to generate a low-resolution image. The image processing unit 32 supplies the low-resolution image to the answer data generation unit 33 and the teacher data generation unit 36.

[0047] The correct data generating unit 33 generates correct data based on the low-resolution image. For example, a user specifies the position of a vehicle in the low-resolution image via the input unit 31. The correct data generating unit 33 generates correct data indicating the position of the vehicle in the low-resolution image based on the position of the vehicle specified by the user. The correct data generating unit 33 supplies the correct data to the teacher data generating unit 36.

[0048] The signal processing unit 34 performs the same processing as the signal processing unit 13 in Fig. 2. That is, the signal processing unit 34 performs predetermined signal processing on the millimeter wave data to generate a signal intensity image and a velocity image. The signal processing unit 34 supplies the signal intensity image and the velocity image to the geometric transformation unit 35.

[0049] The geometric transformation unit 35 performs the same processing as the geometric transformation unit 14 in Fig. 2. That is, the geometric transformation unit 35 performs geometric transformation of the signal strength image and the velocity image. The geometric transformation unit 35 supplies the geometrically transformed signal strength image and the geometrically transformed velocity image after the geometric transformation to the teacher data generation unit 36.

[0050] The teacher data generating unit 36 ​​generates teacher data including input data including a low-resolution image, a geometrically transformed signal strength image, and a geometrically transformed velocity image, as well as correct answer data. The teacher data generating unit 36 ​​supplies the teacher data to the learning unit 37.

[0051] The learning unit 37 uses the teacher data to perform a learning process for the object recognition model 40. The learning unit 37 outputs the object recognition model 40 after learning.

[0052] Here, the object recognition model learning process executed by the learning system 30 will be described.

[0053] Before the start of this process, data used to generate the teacher data is collected. For example, while the vehicle is actually traveling, the camera 21 and the millimeter wave radar 23 installed on the vehicle sense the area ahead of the vehicle. Specifically, the camera 21 captures an image of the area ahead of the vehicle and stores the captured image in the storage unit. The millimeter wave radar 23 detects an object ahead of the vehicle and stores the captured millimeter wave data in the storage unit. The teacher data is generated based on the captured image and millimeter wave data stored in the storage unit.

[0054] First, the learning system 30 generates teacher data. For example, a user inputs a captured image and millimeter wave data acquired at approximately the same time to the learning system 30 via the input unit 31. That is, the captured image and millimeter wave data obtained by sensing at approximately the same time are input to the learning system 30. The captured image is supplied to the image processing unit 32, and the millimeter wave data is supplied to the signal processing unit 34.

[0055] The image processing unit 32 performs image processing such as thinning on the captured image to generate a low-resolution image. The image processing unit 32 supplies the low-resolution image to the answer data generating unit 33 and the teacher data generating unit .

[0056] The signal processing unit 34 performs a predetermined signal processing on the millimeter wave data to estimate the position and speed of an object that reflects the transmission signal in front of the vehicle. The position of the object is represented, for example, by the distance from the vehicle to the object, and the direction (angle) of the object relative to the optical axis direction of the millimeter wave radar 23 (the traveling direction of the vehicle). Note that, for example, the optical axis direction of the millimeter wave radar 23 is equal to the center direction of the radiated range when the transmission signal is transmitted radially, and is equal to the center direction of the scanned range when the transmission signal is scanned. The speed of the object is represented, for example, by the relative speed of the object with respect to the vehicle.

[0057] The signal processing unit 34 generates a signal strength image and a velocity image based on the estimation results of the object position and velocity. The signal processing unit 34 supplies the signal strength image and the velocity image to the geometric transformation unit 35. Although not shown, the velocity image is an image that represents the positions of objects ahead of the vehicle and the distribution of the relative velocities of each object in a bird's-eye view, similar to the signal strength image.

[0058] The geometric transformation unit 35 performs geometric transformation on the signal strength image and the velocity image, and converts the signal strength image and the velocity image into images in the same coordinate system as the captured image, thereby generating a geometrically transformed signal strength image and a geometrically transformed velocity image. The geometric transformation unit 35 supplies the geometrically transformed signal strength image and the geometrically transformed velocity image to the teacher data generation unit 36.

[0059] In the geometrically transformed signal strength image, areas with stronger signal strength are brighter, and areas with weaker signal strength are darker. In the geometrically transformed velocity image, areas with faster relative velocities are brighter, areas with slower relative velocities are darker, and areas where the relative velocity cannot be detected (where no object exists) are painted black. In this way, by performing a geometric transformation on the millimeter wave images (signal strength image and velocity image), not only the positions of objects in the lateral and depth directions, but also the positions of objects in the height direction can be displayed.

[0060] However, the millimeter wave radar 23 has a lower resolution in the height direction as the distance increases, so the height of a distant object may be detected as higher than it actually is.

[0061] In response to this, the geometric transformation unit 35 limits the height of an object that is a predetermined distance or more away when performing geometric transformation of a millimeter wave image. Specifically, when performing geometric transformation of a millimeter wave image, if the height of an object that is a predetermined distance or more away exceeds a predetermined upper limit value, the geometric transformation unit 35 limits the height of the object to the upper limit value and performs geometric transformation. This prevents, for example, when the target object is a vehicle, erroneous recognition caused by the height of the distant vehicle being detected as being higher than it actually is.

[0062] The teacher data generating unit 36 ​​generates teacher data including input data including the captured image, the geometric transformation signal intensity image, and the geometric transformation velocity image, as well as the correct answer data. The teacher data generating unit 36 ​​supplies the generated teacher data to the learning unit 37.

[0063] Next, the learning unit 37 learns the object recognition model 40. Specifically, the learning unit 37 inputs input data included in the teacher data to the object recognition model 40. The object recognition model 40 performs a recognition process for the target object, and outputs data indicating the recognition result. The learning unit 37 compares the recognition result of the object recognition model 40 with the correct answer data, and adjusts the parameters of the object recognition model 40 so as to reduce the error.

[0064] Next, the learning unit 37 judges whether or not to continue learning. For example, if the learning of the object recognition model 40 has not converged, the learning unit 37 judges to continue learning, and the process returns to the initial teacher data generation process. After that, each of the above-mentioned processes is repeatedly executed until it is judged that the learning is to be ended.

[0065] On the other hand, if the result of the judgment by the learning unit 37 is, for example, that the learning of the object recognition model 40 has converged, it is judged that the learning is to be ended, and the object recognition model learning process is ended. In this manner, the trained object recognition model 40 is generated.

[0066] (1-3. Hardware configuration example) Next, an example of a hardware configuration of the vehicle outside information detection unit 10 applicable to each embodiment of the present disclosure will be described. Fig. 5 is a block diagram showing an example of a hardware configuration of the vehicle outside information detection unit 10 applicable to each embodiment. In Fig. 5, the vehicle outside information detection unit 10 includes a CPU (Central Processing Unit) 400, a ROM (Read Only Memory) 401, a RAM (Random Access Memory) 402, and interfaces (I / F) 403, 404, and 405, which are communicatively connected to each other via a bus 410. The vehicle outside information detection unit 10 may further include a storage device such as a flash memory.

[0067] The CPU 400 controls the overall operation of the vehicle outside information detection unit 10 in accordance with programs and data pre-stored in the ROM 401, using the RAM 402 as a work memory. Here, the ROM 401 or the RAM 402 pre-stores programs and data for realizing the object recognition model 40 described with reference to Figs. 2 to 4. The object recognition model 40 is constructed in the vehicle outside information detection unit 10 by the CPU 400 executing the programs.

[0068] The interface 403 is an interface for connecting the camera 21. The interface 404 is an interface for connecting the millimeter wave radar 23. The outside information detection unit 10 controls the camera 21 and the millimeter wave radar 23 via these interfaces 403 and 404, and acquires captured image data (hereinafter referred to as image data) captured by the camera 21 and millimeter wave data acquired by the millimeter wave radar 23. The outside information detection unit 10 applies the image data and millimeter wave data as input data to the object recognition model 40, thereby executing recognition processing to recognize objects.

[0069] 5, the interface 405 is an interface for communicating between the vehicle outside information detection unit 10 and the communication network 12001. The vehicle outside information detection unit 10 transmits information indicating the object recognition result output by the object recognition model 40 from the interface 405 to the communication network 12001.

[0070] 2. Overview of the embodiments of the present disclosure Next, an outline of an embodiment of the present disclosure will be described. In each embodiment of the present disclosure, a detection window for detecting an object based on the output of a first sensor for detecting an object is set based on the output of a second sensor for detecting the object in a manner different from that of the first sensor, and a recognition process is performed to recognize the object based on the output of a region of the second sensor that corresponds to the detection window.

[0071] 6 is a diagram illustrating an object recognition model 40 according to an embodiment of the present disclosure. In the object recognition model 40a, image data 100 acquired from the camera 21 is input to a feature extraction layer 110. In addition, millimeter wave image data 200 based on a millimeter wave image acquired from the millimeter wave radar 23 is input to a feature extraction layer 210.

[0072] Image data 100 input to object recognition model 40a is shaped into data including one or more channels of feature amounts, for example, in image processing unit 12. The image data 100 undergoes feature extraction by feature extraction layer 110 in object recognition model 40a, and is converted into data with its size changed as necessary and channels of feature amounts added. The image data 100 from which features have been extracted by feature extraction layer 110 undergoes convolution processing in object recognition layer 120, and multiple object recognition layer data are generated that are sequentially convolved.

[0073] The object recognition model 40a creates an attention map 130 based on a plurality of object recognition layer data. The attention map 130 includes information indicating a detection window for limiting an area to be subjected to object recognition, for example, within a range indicated by the image data 100. The created attention map 130 is input to the multiplication unit 220.

[0074] On the other hand, the millimeter wave image data 200 input to the object recognition model 40a is shaped into data including features of one or more channels by, for example, the signal processing unit 13 and the geometric transformation unit 14. The millimeter wave image data 200 undergoes feature extraction by the feature extraction layer 210 in the object recognition model 40a, and is resized as necessary (for example, to the same size as the image data 100) and is converted into data with additional channels of features. The millimeter wave image data 200 of each channel, the features of which have been extracted by the feature extraction layer, is input to the multiplication unit 220, and multiplication is performed pixel by pixel with the attention map 130. This limits the area in the millimeter wave image data 200 where object recognition is performed. Furthermore, the output of the multiplication unit 220 is input to the addition unit 221, and the output of the feature extraction layer 210 is added. The output of the addition unit 221 is input to the object recognition layer 230, and is subjected to convolution processing.

[0075] In this way, by performing object recognition processing on an area restricted by the attention map 130, the amount of processing required for the object recognition processing can be reduced.

[0076] It should be noted that by using data of a past frame 101 as the image data 100, it is possible to speed up the processing.

[0077] [3. First embodiment] Next, a first embodiment of the present disclosure will be described. Fig. 7 is a diagram showing an example of the configuration of an object recognition model according to the first embodiment. In Fig. 7, the object recognition model 40b has feature extraction layers 110 and 210 shown on the left side of the figure, and object recognition layers 120 and 230, and the processes therein are the same as those in Fig. 6, so that the description thereof will be omitted here.

[0078] 7, an object recognition layer 230 based on millimeter wave image data 200 and an object recognition layer 120 based on image data 100 are diagrammatically shown. The object recognition layer 230 is a layer of object recognition layer data 230 that is sequentially convolution-processed based on the millimeter wave image data 200. 0 , 230 1 , 230 2 , 230 3 , 230 4 , 230 5 and ~230 6 The object recognition layer 120 includes each object recognition layer data 120 that is sequentially convolution-processed based on the image data 100. 0 , 120 1 , 120 2 , 120 3 , 120 4 , 120 5 and 120 6 Includes.

[0079] In the following, each object recognition layer data 120 0 ~120 6 If there is no need to distinguish between them, they are treated as object recognition layer data 120. x In the same manner, each object recognition layer data 230 0 ~230 6 If there is no need to distinguish between them, they are treated as object recognition layer data 230. x The following explanation will be given using the above as a representative example.

[0080] In FIG. 7, each object recognition layer data 120 0 ~120 7 Specific examples are shown as layer images #0, #1, #2, #3, #4, #5, and #6, which correspond to the attention maps, respectively. Details will be described later, but among the layer images, the white parts shown in layer images #1 and #2 indicate the detection windows.

[0081] That is, the object recognition layer 120 obtains object likelihood based on the features of each layer image #0, #1, #2, #3, #4, #5, and #6, and determines an area where the obtained object likelihood is high. The object recognition layer 120 obtains object likelihood based on pixel information for, for example, layer image #1. Then, the object likelihood obtained is compared with a threshold value, and a region where the object likelihood is higher than the threshold value is determined. In the example of FIG. 7, a region expressed in white in layer image #1 indicates a region where the object likelihood is higher than the threshold value. The object recognition layer 120 generates region information indicating the region. This region information includes information indicating a position in layer image #1 and a value indicating the object likelihood at that position. The object recognition layer 120 sets a detection window based on the region indicated in this region information, and creates an attention map.

[0082] Here, each object recognition layer data 120 0 ~120 6 The size of each layer image is successively reduced by convolution. For example, in the example of FIG. 7, layer image #0 (object recognition layer data 120 0 For example, if the size of layer image #0 is 640 pixels x 384 pixels, the size of layer image #6 becomes 1 pixel x 1 pixel after seven layers of convolution (and shaping).

[0083] As described above, a layer image with a smaller number of convolutions and a larger size can detect smaller (farther away) objects, and a layer image with a larger number of convolutions and a smaller size can detect larger (closer) objects. This is because each object recognition layer data 230 based on millimeter wave data 0 ~230 6 The same applies to.

[0084] Layer images with a large number of convolutions and a small number of pixels, or layer images with a small number of convolutions that recognize objects as small, may not be suitable for object recognition processing. Therefore, in the example of Fig. 7, attention maps may not be created for all seven layers, but may be created using a number of layer images according to the purpose (for example, three layers, layer images #1 to #3).

[0085] Each object recognition layer data 120 0 ~120 7 are input to the corresponding synthesis unit 300. In addition, each object recognition layer data 230 based on the millimeter wave image data 200 0 ~230 6 Similarly, each of the object recognition layer data 120 is input to the corresponding synthesis unit 300. 0 ~120 7 Each of the object recognition layer data 230 0 ~230 6 Each of and is synthesized to obtain the synthetic object recognition layer data 310 0 ~310 6 Generate.

[0086] 8 is a diagram showing an example of the configuration of the synthesis unit 300 according to the first embodiment. The synthesis unit 300 includes a multiplication unit 220 and an addition unit 221. The multiplication unit 220 receives object recognition layer data 120 using an attention map based on image data 100 at one input terminal. x The other input terminal of the multiplication unit 220 receives object recognition layer data 230 based on the millimeter wave image data 200. x The multiplication unit 220 multiplies the object recognition layer data 120 input to one of the input terminals. x and object recognition layer data 230 input to the other input terminal. x The multiplication unit 220 calculates the product of each pixel of the above. The millimeter wave image data 200 (object recognition layer data 230 x ) that corresponds to the detection window will be highlighted.

[0087] Without being limited to this, the object recognition model 40a may suppress the area outside the detection window in the millimeter wave image data 200.

[0088] The multiplication result of the multiplication unit 220 is input to one input terminal of the addition unit 221. The other input terminal of the addition unit 221 receives object recognition layer data 230 based on the millimeter wave image data 200. xThe adder 221 receives the multiplication result of the multiplier 220 input to one input terminal and the object recognition layer data 230. x Calculate the matrix sum for and .

[0089] In this way, through the processing of the multiplication unit 220 and the addition unit 221, area information generated according to the object likelihood detected during the object recognition processing based on the image data 100 from the camera 21 as a second sensor different from the first sensor is added to the millimeter wave image data 200 from the millimeter wave radar 23 as a first sensor.

[0090] Here, the adder 221 performs a process of adding the original image to the multiplication result of the multiplier 220. For example, when the attention map is expressed by a value of 0 or 1 for each pixel, for example, when the attention map is all 0 in a certain layer image, or in an area of ​​the attention map where the values ​​are 0, information is lost. Therefore, in the process in the prediction unit 150 described later, recognition processing for that area becomes impossible. Therefore, the adder 221 adds the object recognition layer data 230 based on the millimeter wave image data 200. x This prevents data loss in that area.

[0091] Returning to FIG. 7, the synthetic object recognition layer data 310 output from each synthesis unit 300 0 ~310 6 The prediction unit 150 predicts each of the input synthetic object recognition layer data 310 0 ~310 6 The prediction unit 150 performs object recognition processing based on the received signal, and predicts the class of the recognized object, etc. The prediction result by the prediction unit 150 is output from the outside-vehicle information detection unit 10 as data indicating the recognition result of the target object, and is passed to the integrated control unit 12050 via the communication network 12001, for example.

[0092] (3-1. Specific Examples) The attention map based on the object recognition model 40a according to the first embodiment will be described more specifically with reference to FIGS.

[0093] FIG. 9 is a schematic diagram for explaining a first example of an attention map based on the object recognition model 40a according to the first embodiment.

[0094] In Fig. 9, an example of original image data 100a is shown on the left side. The right side of Fig. 9 shows, from the top, object recognition layer data 230. x , Object recognition layer data 230 x ,Synthetic object recognition layer data 310 x From the left, layer image #1 (object recognition layer data 120 1 ), and layer image #2 (object recognition layer data 120 2 ) and #3 (Object Recognition Layer Data 120 3 ) to the object recognition layer data 230 x , Object recognition layer data 230 x and synthetic object recognition layer data 310 x is shown.

[0095] 9 is a feature map showing features based on the millimeter wave image data 200, and the middle part shows an attention map created from the features of the image data 100. The lower part shows the composite object recognition layer data 310 obtained by combining the feature map based on the millimeter wave image data 200 and the attention map based on the image data 100 in the combining unit 300. x It is as follows.

[0096] Below, the object recognition layer data 230 corresponding to layer image #X x The object recognition layer data of layer image #X is 230 x Also, the synthetic object recognition layer data 310 corresponding to the layer image #X is called x The synthetic object recognition layer data 310 of layer image #X x It is called.

[0097] In FIG. 9, object recognition layer data 230 x Among them, the object recognition layer data 230 of the layer image #1 1 In the figure, the area 23110 The recognition result that seems to be an object appears in the part indicated by . Also, the layer image #1 has an area 121 10 and 121 11 The object likelihood of these regions 121 is set to be equal to or greater than a threshold. 10 and 121 11 In contrast, the synthetic object recognition layer data 310 of layer image #1 is 1 So, area 231 10 Corresponding area 230 10 ' and area 121 10 and 121 11 121 corresponding to 10 ' and 121 11 ' and recognition results that resemble objects appear.

[0098] Similarly, for the layer image #2, the object recognition layer data 230 of the layer image #2 is 2 In area 231 11 The layer image #1 has a region 121 13 The object likelihood of the region 121 is determined to be equal to or greater than the threshold. 13 In contrast, the synthetic object recognition layer data 310 of layer image #2 is 2 So, area 231 11 Corresponding area 230 11 ' and area 121 13 Corresponding to 121 13 ' and recognition results that resemble objects appear.

[0099] For layer image #3, the object recognition layer data 230 of layer image #3 3 In area 231 12 A recognition result that looks like an object appears in the area indicated by , and in layer image #1, no area with an object likelihood equal to or greater than the threshold is detected, and no detection window is created. 3 So, area 231 12 Corresponding area 230 12', recognition results that resemble objects appear.

[0100] Also, area 121 10 and 121 11 , and area 121 13 In the figure, the white and gray regions correspond to the detection windows. In this case, for example, the whiter the region, the higher the object likelihood. 13 In the figure, the area with a strong white color where the light gray vertically elongated rectangle and the dark gray horizontally elongated rectangle intersect is area 121. 13 The detection window is the region with the highest object likelihood within the layer image. As described above, the detection window is set based on region information including, for example, information indicating a corresponding position within the layer image and a value indicating the object likelihood.

[0101] In this way, in the layer images #1 and #2, the object recognition layer data 230 based on the millimeter wave image data 200 x The object recognition layer 310 emphasizes the area where the object-like recognition result appears based on the millimeter wave image data 200 without calculating the object likelihood for the object. x can be generated.

[0102] In addition, the adder 221 generates object recognition layer data 230 based on the millimeter wave image data 200. x Since the above-mentioned addition is performed, even if a detection window is not set in layer image #2, as in layer image #3, it is possible to emphasize an area in which an object-like recognition result appears based on the millimeter wave image data 200.

[0103] Fig. 10 is a schematic diagram for explaining a second example of an attention map by the object recognition model 40a according to the first embodiment. The meanings of the parts in Fig. 10 are the same as those in Fig. 9 described above, and therefore will not be explained here. In Fig. 10, an example of original image data 100b is shown on the left side.

[0104] In FIG. 10, object recognition layer data 230 xAmong them, the object recognition layer data 230 of the layer image #1 1 In the figure, the area 231 20 The recognition result that seems to be an object appears in the part indicated by . Also, the layer image #1 has an area 121 20 and 121 21 The object likelihood of these regions 121 is set to be equal to or greater than a threshold. 20 and 121 21 In contrast, the synthetic object recognition layer data 310 of layer image #1 is 1 So, area 231 20 Corresponding area 230 20 ' and area 121 20 and 121 21 121 corresponding to 20 ' and 121 21 ' and recognition results that resemble objects appear.

[0105] Similarly, for the layer image #2, the object recognition layer data 230 of the layer image #2 is 2 In area 231 21 The layer image #2 has a recognition result that looks like an object in the area indicated by 22 The object likelihood of the region 121 is determined to be equal to or greater than the threshold. 22 In contrast, the synthetic object recognition layer data 310 of layer image #2 is 2 So, area 231 21 Corresponding area 230 21 ' and area 121 22 Corresponding to 121 22 ' and recognition results that resemble objects appear.

[0106] For the layer image #3, the object recognition layer data 230 of the layer image #3 3 In area 231 22 The layer image #1 has a region 121 23 The object likelihood of the region 121 is determined to be equal to or greater than the threshold. 23In contrast, the synthetic object recognition layer data 310 of layer image #3 is 3 So, area 231 23 Corresponding area 230 21 ' and area 121 23 Corresponding to 121 23 ' and recognition results that resemble objects appear.

[0107] In the second example, as in the first example described above, in the layer images #1 to #3, the object recognition layer data 230 based on the millimeter wave image data 200 is x The object recognition layer 310 emphasizes the area where the object-like recognition result appears based on the millimeter wave image data 200 without calculating the object likelihood for the object. x can be generated.

[0108] As described above, according to the first embodiment, even if the millimeter wave image data 200 alone has a weak feature, the feature can be enhanced by using the attention map based on the image data 100 captured by the camera 21, thereby improving the performance of object recognition. This also reduces the load associated with the recognition process when multiple different sensors are used.

[0109] In the example of FIG. 7, the object recognition layer data 120 have corresponding convolution layers. x and object recognition layer data 230 x The synthesis unit 300 synthesizes the above data to generate the synthesized object recognition layer data 310 of each convolution layer. x Each of them is input to the prediction unit 150, but this is not limited to this example. For example, x and object recognition layer data 230 x (For example, object recognition layer data 120 1 and object recognition layer data 230 2 ) and the synthesized object recognition layer data 310 synthesized by the synthesis unit 300 xcan be input to the prediction unit 150. In this case, the object recognition layer data 120 to be synthesized by the synthesis unit 300 x and object recognition layer data 230 x It is preferable to make the size of each object recognition layer data 120 uniform. x and each object recognition layer data 230 x A part of the data is synthesized by the synthesis unit 300 to generate synthesized object recognition layer data 310. x At this time, each object recognition layer data 120 x and each object recognition layer data 230 x Alternatively, each of the convolutional layers may select corresponding data one by one from the input data and combine them in the combiner 300. Alternatively, each of the convolutional layers may select multiple pieces of data and combine them in the combiner 300.

[0110] [4. Second embodiment] Next, a second embodiment of the present disclosure will be described. The second embodiment is an example of creating an attention map by a method different from that of the first embodiment. Fig. 11 is a diagram showing an example of a configuration of an object recognition model according to the second embodiment.

[0111] In FIG. 11, in the object recognition model 40c, the object recognition layer 120a performs convolution processing based on the image data 100, and each object recognition layer data 120 0 ~120 6 Here, the object recognition layer 120a has the deepest convolutional layer and generates the object recognition layer data 120 having the smallest size. 6 For example, double the size of the data, and then use the next layer, object recognition layer data 122. 1 Generate.

[0112] In this case, the newly generated object recognition layer data 122 1 The object recognition layer 120 0 ~120 6 The object recognition layer data with the smallest size among the 120 6 Therefore, the object recognition layer 120a inherits the features of the object recognition layer data 120. 6The next deepest convolutional layer is the object recognition layer data 120 6 For example, the object recognition layer data 120 is twice as large as the 5 Object recognition layer data 120 6 , and new object recognition layer data 122 1 Generate.

[0113] Similarly, the object recognition layer 120a generates the object recognition layer data 122 1 For example, the size of the object recognition layer data 120 is doubled. 5 , and new object recognition layer data 122 2 In this way, the object recognition layer 120a according to the second embodiment generates the generated object recognition layer data 122. x For example, the size of the object recognition layer data 120 is doubled. x The object recognition layer data 122 is newly created by combining x+1 The process of generating is repeated.

[0114] The object recognition layer 120a is a set of object recognition layer data 120 generated by sequentially doubling the size as described above. 6 , 122 1 , 122 2 , 122 3 , 122 4 , 122 5 and 122 6 Attention maps are created based on the maximum size of the object recognition layer data 122 6 is fitted to layer image #0 to create an attention map for layer image #0. 5 is inserted into layer image #1 to create an attention map for layer image #1. 4 , 122 3 , 122 2 , 122 1 and 120 6 are embedded into each of the layer images #2, #3, #4, #5 and #6 in order of decreasing size to create an attention map for each of the layer images #2 to #6.

[0115] In this way, in the second embodiment, the object recognition layer 120a generates a new attention map by creating it by machine learning and fitting it in. This reduces FPs (False Positives) due to strong reflection objects other than the recognition target, such as guardrails and curbs, and improves the performance of object recognition using only the millimeter wave image data 200. On the other hand, in the second embodiment, the object recognition layer data 120, which is obtained by performing convolution on the image data 100 up to a deep convolution layer, is generated. 6 Since the attention map is created by linking the data, the features of objects that are difficult to capture with the camera 21 are weakened. For example, it becomes difficult to recognize objects hidden by water droplets or fog. Therefore, it is preferable to switch between the method of creating the attention map according to the second embodiment and the method of creating the attention map according to the first embodiment, for example, according to the environment.

[0116] [5. Third embodiment] Next, a third embodiment of the present disclosure will be described. In the third embodiment, each object recognition layer data 230 based on millimeter wave image data 200 is 0 ~230 6 For each attention map based on the image data 100 (each object recognition layer data 120 0 ~120 6 12 is a diagram showing an example of a configuration of an object recognition model according to the third embodiment.

[0117] In the object recognition model 40d shown in FIG. 12, the object recognition layer 230 generates each object recognition layer data 230 based on the millimeter wave image data 200 in the same manner as in the first embodiment described above. 0 ~230 6 On the other hand, the object recognition layer 120b generates each object recognition layer data 120 based on the image data 100. 0 ~120 6 and each object recognition layer data 120 0 '~120 6 ' and generate.

[0118] Here, each object recognition layer data 120 0 ~120 6 is data in which parameters are adjusted so that object recognition is performed using the image data 100 alone. 0 '~120 6 ' is data in which parameters have been adjusted so that object recognition is performed using both the millimeter wave image data 200 and the image data 100. For example, in the learning system 30 described with reference to Fig. 4, learning for performing object recognition using the image data 100 alone and learning for performing object recognition together with the millimeter wave image data 200 are performed on the same image data 100, and parameters for each are generated.

[0119] As in the first embodiment, each synthesis unit 301 synthesizes each object recognition layer data 120 generated in the object recognition layer 120b. 0 ~120 6 and each object recognition layer data 120 0 '~120 6 ' and each object recognition layer data 230 generated in the object recognition layer 230 0 ~230 6 and are synthesized with corresponding data.

[0120] Fig. 13 is a diagram showing an example of the configuration of a synthesis unit 301 according to the third embodiment. As shown in Fig. 13, the synthesis unit 301 has a connection unit 222 added to the configuration of the multiplication unit 220 and the addition unit 221 in the synthesis unit 300 in Fig. 8.

[0121] In the synthesis unit 301, the multiplication unit 220 receives, at one input end, the object recognition layer data 120 whose parameters have been adjusted so that object recognition can be performed using the image data 100 alone. x is input to the other input terminal, and object recognition layer data 230 x The multiplication unit 220 multiplies the object recognition layer data 120 input to one of the input terminals. x and object recognition layer data 230 input to the other input terminal. xThe multiplication result of the multiplication unit 220 is input to one input terminal of the addition unit 221. The other input terminal of the addition unit 221 receives the object recognition layer data 230. x The adder 221 receives the multiplication result of the multiplier 220 input to one input terminal and the object recognition layer data 230. x Calculate the matrix sum for and .

[0122] The output of the addition unit 221 is input to one input terminal of a connection unit 222. The other input terminal of the connection unit 222 is supplied with object recognition layer data 120, the parameters of which have been adjusted so that object recognition is performed using the image data 100 and the millimeter wave image data 200. x The connection unit 222 receives the output of the addition unit 221 and the object recognition layer data 120. x Concatenate ' and .

[0123] This concatenation process is performed by combining the output of the adder 221 and the object recognition layer data 120. x The output of the adder 221 and the object recognition layer data 120 are listed. x As a result, the data output from the connection unit 222 is, for example, a feature amount included in the output of the addition unit 221 and the object recognition layer data 120. x The data includes the sum of the features possessed by

[0124] The synthesis process in the synthesis unit 301 creates an attention map showing the presence or absence of an object using only the image data 100, and multiplies the created attention map by only the feature amount based on the millimeter wave image data 200. This limits the feature amount based on the millimeter wave image data 200, making it possible to suppress FP.

[0125] Therefore, according to the object recognition model 40d of the third embodiment, it is possible to create an attention map based on image data 100 acquired by the camera 21 alone, and to perform object recognition based on the integrated output of the camera 21 and the millimeter wave radar 23.

[0126] [6. Fourth embodiment] Next, a fourth embodiment of the present disclosure will be described. The fourth embodiment uses object recognition layer data 120 based on image data 100. x and object recognition layer data 230 based on millimeter wave image data 200 x In this example, concatenated data is generated by concatenating and, and object recognition is performed using this concatenated data.

[0127] 14 is a diagram showing an example of the configuration of an object recognition model according to the fourth embodiment. In the object recognition model 40e according to the fourth embodiment, each piece of linked data for performing the object recognition process is already stored in the object recognition layer data 120. x and object recognition layer data 230 x Therefore, in each connected data, the object recognition layer data 230 based on the millimeter wave image data 200 is included. x Therefore, in the object recognition model 40e according to the fourth embodiment, the object recognition layer data 120 x and object recognition layer data 230 x A process for suppressing the area outside the detection window in the millimeter wave image data 200 is performed before the connection unit 222 that connects the above.

[0128] In the object recognition model 40e shown in FIG. 14, each object recognition layer data 230 generated in the object recognition layer 230 based on the millimeter wave image data 200 0 ~230 6 (not shown) are input to the synthesis unit 300. On the other hand, the object recognition layer 120c synthesizes each object recognition layer data 120 based on the image data 100. 0 ~120 6 The generated object recognition layer data 120 0 ~120 6 A predetermined number of pieces of data are superimposed to create an attention map. This attention map is input to the synthesis unit 300.

[0129] In the example of FIG. 14, the object recognition layer 120c includes each object recognition layer data 120 0 ~120 6 The convolutional layers are sequentially arranged in three adjacent object recognition layer data sets, starting from 120 0 , 120 1 and 120 2 The attention map is created by the image data 123 on which the object recognition layer data 120 is superimposed. This is not limited to this example. For example, the object recognition layer 120c may be 0 ~120 6 The attention map can be created by the image data 123 in which all of the above are superimposed. x The attention map may be created by image data on which the convolutional layers are superimposed. x In addition, multiple object recognition layer data 120 selected at different convolutional layers are used. x It is also possible to create an attention map using image data 123 on which the above is superimposed.

[0130] In the synthesis unit 300, the image data 123 and each object recognition layer data 230 are multiplied by the multiplication unit 220 in the same manner as described with reference to FIG. 0 ~230 6 and the product is calculated, and the product is calculated by the adder 221 for each object recognition layer data 230 0 ~230 6 The synthesis unit 300 synthesizes the image data 123 and each object recognition layer data 230. 0 ~230 6 The combined data obtained by combining the above is input to one input terminal of the connecting unit 222.

[0131] The other input terminal of the connection portion 222 is connected to each of the object recognition layer data 120 generated by the object recognition layer 120c based on the image data 100. 0 ~120 6 The connection unit 222 receives the synthetic data input to one input terminal and the object recognition layer data 120 input to the other input terminal. 0 ~1206 and each object recognition layer data 120 0 ~120 6 2 Corresponding consolidated data 242 0 , 242 1 , 242 2 , 242 3 , 242 4 , 242 5 and 242 6 Generate.

[0132] Each linked data 242 output from the linking unit 222 0 ~242 6 are each input to the prediction unit 150.

[0133] With this configuration, the prediction unit 150 can recognize each linked data 242 0 ~242 6 Therefore, according to the object recognition model 40e according to the fourth embodiment, it is possible to create an attention map based on the image data 100 acquired by the camera 21 alone, and to perform object recognition based on the integrated output of the camera 21 and the millimeter wave radar 23.

[0134] [7. Fifth embodiment] Next, a fifth embodiment according to the present disclosure will be described. The object recognition model according to the fifth embodiment is an example in which the image data 100 of the previous frame is used as the image data 100 for creating an attention map.

[0135] Fig. 15 is a diagram showing an example of the configuration of an object recognition model according to the fifth embodiment. Note that an object recognition model 40f shown in Fig. 15 is an example in which the configuration of the fifth embodiment is applied to the object recognition model 40d (see Fig. 12) according to the above-mentioned third embodiment.

[0136] In the object recognition model 40f shown in FIG. 15, the object recognition layer 120d, in the same manner as in FIG. 12 described above, generates each object recognition layer data 120 based on image data 100 (called image data 100 of the current frame) acquired by the camera 21 as frame image data of a certain frame (called the current frame). 0 ~120 6 Further, the object recognition layer 230 generates each object recognition layer data 230 based on the millimeter wave image data 200 acquired by the millimeter wave radar 23 corresponding to the current frame (hereinafter referred to as millimeter wave image data 200 of the current frame). 0 ~230 6 Generate.

[0137] At this time, each object recognition layer data 120 generated based on the image data 100 of the current frame 0 ~120 6 The object recognition layer data 120 is stored in the memory 420. For example, the RAM 402 shown in FIG. 5 can be used as the memory 420. 0 ~120 6 In the above description, all of the convolution layers are stored in the memory 420, but this is not limited to the above example. For example, 0 Alternatively, only the above-mentioned information may be stored.

[0138] On the other hand, the object recognition layer 120d is generated based on the image data 100 (referred to as the image data 100 of the past frame 101) acquired by the camera 21 in the past (for example, the frame immediately before the current frame) and stored in the memory 420. 0 ~120 6 Here, the attention map is created based on the object recognition layer data 120 of the shallowest convolution layer for the memory 420. 0 If only the object recognition layer data 120 is stored, 0 , and the convolution process is sequentially performed on each object recognition layer data 120 1 ~120 6can be generated.

[0139] Each object recognition layer data 120 corresponding to the current frame 0 ~120 6 and each object recognition layer data 230 0 ~230 6 are input to the corresponding synthesis unit 301. Also, each object recognition layer data 120 generated based on the image data 100 of the past frame 101 is 0 ~120 6 are input to the synthesis unit 301 as attention maps.

[0140] In the synthesis unit 301, as described with reference to FIG. 13, the multiplication unit 220 multiplies each of the object recognition layer data 120 0 ~120 6 and each object recognition layer data 230 0 ~230 6 and each of the object recognition layer data 230 is multiplied by the adder 221. 0 ~230 6 The object recognition layer data 120 generated based on the image data 100 of the past frame 101 is added to each of the addition results of the addition unit 221 in the connection unit 222. 0 ~120 6 are concatenated.

[0141] In this way, by creating an attention map using data from the past frame 101 as the image data 100, one or more convolution processes can be omitted in the object recognition layer 120c, making it possible to speed up processing.

[0142] [8. Sixth embodiment] Next, a sixth embodiment will be described. In the above-mentioned first to fifth embodiments, the data acquisition unit 20 has been described as including the camera 21 and the millimeter wave radar 23 as sensors, but the combination of sensors included in the data acquisition unit 20 is not limited to this example. In the sixth embodiment, other examples of combinations of sensors included in the data acquisition unit 20 will be described.

[0143] (8-1. First example) Fig. 16 is a block diagram showing a first example of the outside-vehicle information detection unit and the data acquisition unit according to the sixth embodiment. As shown in Fig. 16, the first example is an example in which the data acquisition unit 20a includes a camera 21 and a LiDAR 24 as sensors. The LiDAR 24 is a light reflection distance measurement sensor for measuring distance by a LiDAR method in which light emitted from a light source is reflected on an object to measure distance, and includes a light source and a light receiving unit.

[0144] The signal processing unit 13a creates, for example, three-dimensional point cloud information based on the RAW data output from the LiDAR 24. The geometric transformation unit 14a converts the three-dimensional point cloud information created by the signal processing unit 13a into an image viewed from the same viewpoint as the image captured by the camera 21. More specifically, the geometric transformation unit 14a converts the coordinate system of the three-dimensional point cloud information based on the RAW data output from the LiDAR 24 into the coordinate system of the captured image. The output data of the LiDAR 24, whose coordinate system has been converted into the coordinate system of the captured image by the geometric transformation unit 14a, is supplied to the recognition processing unit 15a. The recognition processing unit 15a performs object recognition processing using the output data of the LiDAR 24, whose coordinate system has been converted into the coordinate system of the captured image, instead of the millimeter wave image data 200 in the recognition processing unit 15 described above.

[0145] (8-2. Second Example) FIG. 17 is a block diagram showing a second example of the outside-vehicle information detection unit and the data acquisition unit according to the sixth embodiment. As shown in FIG. 17, the second example is an example in which the data acquisition unit 20b includes a camera 21 and an ultrasonic sensor 25 as sensors. The ultrasonic sensor 25 measures distance by transmitting sound waves (ultrasound waves) in a frequency band higher than the audible frequency band and receiving the reflected waves of the ultrasound, and has, for example, a transmitting element that transmits the ultrasound and a receiving element that receives the ultrasound. In some cases, one element transmits and receives the ultrasound. The ultrasonic sensor 25 can obtain three-dimensional point cloud information by, for example, repeatedly transmitting and receiving the ultrasound at a predetermined period while scanning the transmission direction of the ultrasound.

[0146] The signal processing unit 13b creates, for example, three-dimensional point cloud information based on the data output from the ultrasonic sensor 25. The geometric transformation unit 14b converts the three-dimensional point cloud information created by the signal processing unit 13b into an image viewed from the same viewpoint as the image captured by the camera 21. More specifically, the geometric transformation unit 14b converts the coordinate system of the three-dimensional point cloud information based on the data output from the ultrasonic sensor 25 into the coordinate system of the captured image. The output data of the ultrasonic sensor 25, whose coordinate system has been converted into the coordinate system of the captured image by the geometric transformation unit 14b, is supplied to the recognition processing unit 15b. The recognition processing unit 15b performs object recognition processing using the output data of the ultrasonic sensor 25, whose coordinate system has been converted into the coordinate system of the captured image, instead of the millimeter wave image data 200 in the recognition processing unit 15 described above.

[0147] (8-3. Third Example) Fig. 18 is a block diagram showing a third example of the outside-vehicle information detection unit and the data acquisition unit according to the sixth embodiment. As shown in Fig. 18, the third example is an example in which the data acquisition unit 20c includes a camera 21, a millimeter-wave radar 23, and a LiDAR 24 as sensors.

[0148] In the outside vehicle information detection unit 10 shown in Fig. 18, millimeter wave data output from the millimeter wave radar 23 is input to the signal processing unit 13. The signal processing unit 13 performs the same processing as that described using Fig. 2 on the input millimeter wave data to generate a millimeter wave image. The geometric transformation unit 14 performs a geometric transformation on the millimeter wave image generated by the signal processing unit 13 to convert the millimeter wave image into an image in the same coordinate system as the captured image. The image into which the millimeter wave image has been converted by the geometric transformation unit 14 (referred to as a converted millimeter wave image) is supplied to the recognition processing unit 15c.

[0149] In addition, in the vehicle exterior information detection unit 10, the RAW data output from the output of the LiDAR 24 is input to the signal processing unit 13c. The signal processing unit 13c creates, for example, three-dimensional point cloud information based on the RAW data input from the LiDAR 24. The geometric transformation unit 14c converts the three-dimensional point cloud information created by the signal processing unit 13c into an image viewed from the same viewpoint as the image captured by the camera 21. The image into which the three-dimensional point cloud information has been converted by the geometric transformation unit 14 (referred to as a converted LiDAR image) is supplied to the recognition processing unit 15c.

[0150] The recognition processing unit 15c integrates the converted millimeter wave image and the converted LiDAR image input from the geometric transformation units 14 and 14c, respectively, and performs object recognition processing by using the integrated image in place of the millimeter wave image data 200 in the above-described recognition processing unit 15. Here, the recognition processing unit 15c can link the converted millimeter wave image and the converted LiDAR to integrate the converted millimeter wave image and the converted LiDAR.

[0151] (8-4. Fourth Example) Fig. 19 is a block diagram showing a fourth example of the outside-vehicle information detection unit and the data acquisition unit according to the sixth embodiment. As shown in Fig. 19, the fourth example applies the data acquisition unit 20a including the camera 21 and the millimeter-wave radar 23 described with reference to Fig. 16. Meanwhile, in the outside-vehicle information detection unit 10, the image processing unit 12 and the geometric transformation unit 14d are connected to the output of the camera 21, and only the signal processing unit 13 is connected to the millimeter-wave radar 23.

[0152] In the outside information detection unit 10, the image processing unit 12 performs predetermined image processing on the captured image output from the camera 21. The captured image processed by the image processing unit 12 is supplied to a geometric transformation unit 14d. The geometric transformation unit 14d converts the coordinate system of the captured image into the coordinate system of the millimeter wave data output from the millimeter wave radar 23. The captured image converted into the coordinate system of the millimeter wave data by the geometric transformation unit 14d (referred to as a converted captured image) is supplied to a recognition processing unit 15d.

[0153] Meanwhile, in the vehicle exterior information detection unit 10, millimeter wave data output from the millimeter wave radar 23 is input to the signal processing unit 13. The signal processing unit 13 performs predetermined signal processing on the input millimeter wave data and generates a millimeter wave image based on the millimeter wave data. The millimeter wave image generated by the signal processing unit 13 is supplied to the recognition processing unit 15d.

[0154] The recognition processing unit 15d can use, for example, millimeter wave image data of a millimeter wave image supplied from the signal processing unit 13 instead of the image data 100 in the above-mentioned recognition processing unit 15, and can use the converted captured image supplied from the geometric transformation unit 14d instead of the millimeter wave image data 200. For example, when the performance of the millimeter wave radar 23 is high and the performance of the camera 21 is low, it is considered to employ the configuration according to this fourth example.

[0155] (8-5. Fifth Example) In the first to fourth examples of the sixth embodiment described above, the camera 21 is combined with a sensor of a type different from that of the camera 21, but this is not limited to this example. For example, as a fifth example of the sixth embodiment, a combination of cameras 21 with different characteristics can be applied. As an example, a combination of a first camera 21 using a telephoto lens with a narrow angle of view capable of capturing images at long distances and a second camera 21 using a wide-angle lens with a wide angle of view capable of capturing images over a wide range can be considered.

[0156] (8-6. 6th Example) Next, a fifth example of the sixth embodiment will be described. The fifth example is an example in which the configuration of the recognition processing unit 15 is switched depending on conditions. For the sake of explanation, the following will be described by taking the recognition processing unit 15 (object recognition model 40a) according to the first embodiment as an example.

[0157] As an example, it is possible to switch between using and not using the attention map depending on the weather and the scene. For example, under nighttime and rainy conditions, it may be difficult to recognize objects in the images captured by the camera 21. In this case, object recognition is performed using only the output of the millimeter wave radar 23. As another example, it is possible to change the way the attention map is used when one of the multiple sensors included in the data acquisition unit 20 does not operate normally. For example, when normal image data 100 is not output due to a malfunction of the camera 21, object recognition is performed at the same recognition level as when the attention map is not used. As yet another example, when the data acquisition unit 20 includes three or more sensors, it is possible to create multiple attention maps based on the outputs of the multiple sensors. In this case, it is possible to integrate the multiple attention maps created based on the outputs of the multiple sensors.

[0158] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0159] The present technology can also be configured as follows. (1) a recognition processing unit that performs a recognition process for recognizing an object by adding area information generated according to an object likelihood detected in the course of object recognition processing based on an output of a second sensor different from the first sensor to an output of the first sensor; Equipped with Information processing device. (2) The recognition processing unit is The recognition process is performed using an object recognition model obtained by machine learning; The object recognition model generates the region information in one layer of a first convolutional layer generated based on the output of the second sensor, and adds the generated region information to a layer of a second convolutional layer generated based on the output of the first sensor, the layer corresponding to the layer that generated the region information. The information processing device according to (1). (3) The recognition processing unit is The recognition process is performed using an object recognition model obtained by machine learning; The object recognition model generates the region information in a plurality of layers included in a first convolutional layer generated based on an output of the second sensor, and adds the generated region information to each of a plurality of layers of a second convolutional layer generated based on the output of the first sensor and corresponding one-to-one to each of the plurality of layers that generated the region information. The information processing device according to (1). (4) The recognition processing unit is generating the region information in each of a predetermined number of the first convolutional layers; The information processing device according to (3). (5) the second sensor is an image sensor; 5. The information processing device according to any one of (1) to (4). (6) The first sensor is any one of a millimeter wave radar, a light reflection distance measuring sensor, and an ultrasonic sensor. The information processing device according to (5). (7) The first sensor includes: The system includes two or more sensors selected from an image sensor, a millimeter wave radar, a light reflection distance sensor, and an ultrasonic sensor, and the output of the first sensor is an output obtained by integrating the outputs of the two or more sensors. The information processing device according to (5). (8) the first sensor is an image sensor; The second sensor is any one of a millimeter wave radar, a light reflection distance measuring sensor, and an ultrasonic sensor. 5. The information processing device according to any one of (1) to (4). (9) The recognition processing unit is highlighting regions in the output of the first sensor that correspond to regions in the output of the second sensor where the object likelihood is equal to or greater than a first threshold. 9. An information processing device according to any one of (1) to (8). (10) The recognition processing unit is suppressing regions in the output of the first sensor that correspond to regions in the output of the second sensor where the object likelihood is below a second threshold. 10. The information processing device according to any one of (1) to (9). (11) The recognition processing unit is generating the region information using an output of the second sensor one frame before; 11. The information processing device according to any one of (1) to (10). (12) The recognition processing unit is Concatenating an output of the second sensor to the area information. 12. The information processing device according to any one of (1) to (11). (13) A first sensor; a second sensor different from the first sensor; an information processing device including a recognition processing unit that performs a recognition process to recognize an object by adding area information generated according to an object likelihood detected in the process of object recognition processing based on the output of the second sensor to the output of the first sensor; An information processing system comprising: (14) A recognition processing step of performing a recognition processing for recognizing an object by adding area information generated according to an object likelihood detected in the process of object recognition processing based on an output of a second sensor different from the first sensor to the output of the first sensor. An information processing program for causing a computer to execute the above. (15) Executed by a processor, a recognition processing step of performing a recognition processing for recognizing an object by adding area information generated according to an object likelihood detected in the course of object recognition processing based on an output of a second sensor different from the first sensor to an output of the first sensor; Including, Information processing methods. [Explanation of symbols]

[0160] 10. External information detection unit 11 Information Processing Section 12 Image processing section 13, 13a, 13b, 13c Signal processing section 14, 14a, 14b, 14c, 14d Geometric transformation section 15a, 15b, 15c, 15d Recognition processing section 20, 20a, 20b, 20c Data acquisition section 21 Camera 22 Image Sensor 23 Millimeter wave radar 24 LiDAR 25 Ultrasonic Sensor 30 Learning System 40, 40a, 40b, 40c, 40d, 40e, 40f Object recognition model 41a, 41b, 41c, 110, 210 Feature extraction layer 100,100a,100b Image data 120,120a,120b,120c Object recognition layer 120 0 ,120 1 ,120 2 ,120 3 ,120 4 ,120 5 ,120 6 ,120 x ,120 0 ',120 1 ',120 2 ',120 3 ',120 4 ',120 5 ',120 6 ',122 1 ,122 2 ,122 3 ,122 4 ,122 5 ,122 6 ,230 0 ,230 1 ,230 2 ,230 3 ,230 4 ,230 5 ,230 6,230 x Object Recognition Layer Data 150 Prediction Department 200 mmWave Image Data 220 Multiplication section 221 Addition section 222 Connecting part 230 Object recognition layer 242 0 ,242 1 ,242 2 ,242 3 ,242 4 ,242 5 ,242 6 Consolidated Data 300,301 Synthesis section 310 0 ,310 1 ,310 2 ,310 3 ,310 4 ,310 5 ,310 6 Synthetic Object Recognition Layer Data

Claims

1. a recognition processing unit that performs a recognition process for recognizing an object by adding area information generated according to an object likelihood detected in the course of object recognition processing based on an output of a second sensor different from the first sensor to an output of the first sensor; Equipped with The recognition processing unit is The recognition process is performed using an object recognition model obtained by machine learning; The object recognition model generates the region information in one or more layers included in a first convolutional layer generated based on an output of the second sensor, and adds the generated region information to one or more layers of a second convolutional layer generated based on the output of the first sensor and in one-to-one correspondence with the one or more layers that generated the region information. Information processing device.

2. The recognition processing unit is generating the region information in each of a predetermined number of the first convolutional layers; The information processing device according to claim 1 .

3. the second sensor is an image sensor; The information processing device according to claim 1 .

4. The first sensor is any one of a millimeter wave radar, a light reflection distance measuring sensor, and an ultrasonic sensor. The information processing device according to claim 2 .

5. The first sensor includes: The vehicle includes two or more sensors selected from an image sensor, a millimeter wave radar, a light reflection distance sensor, and an ultrasonic sensor, and an output obtained by integrating the outputs of the two or more sensors is set as an output of the first sensor. The information processing device according to claim 2 .

6. the first sensor is an image sensor; The second sensor is any one of a millimeter wave radar, a light reflection distance measuring sensor, and an ultrasonic sensor. The information processing device according to claim 1 .

7. The recognition processing unit is highlighting regions in the output of the first sensor that correspond to regions in the output of the second sensor where the object likelihood is equal to or greater than a first threshold. The information processing device according to claim 1 .

8. The recognition processing unit is suppressing regions of the output of the first sensor that correspond to regions in the output of the second sensor where the object likelihood is below a second threshold. The information processing device according to claim 1 .

9. The recognition processing unit is generating the region information using an output of the second sensor one frame before; The information processing device according to claim 1 .

10. The recognition processing unit is coupling an output of the second sensor to the area information; The information processing device according to claim 1 .

11. A first sensor; a second sensor different from the first sensor; an information processing device including a recognition processing unit that performs a recognition process to recognize an object by adding area information generated according to an object likelihood detected in the course of object recognition processing based on an output of the second sensor to an output of the first sensor; Including, The recognition processing unit is The recognition process is performed using an object recognition model obtained by machine learning; The object recognition model generates the region information in one or more layers included in a first convolutional layer generated based on an output of the second sensor, and adds the generated region information to one or more layers of a second convolutional layer generated based on the output of the first sensor and in one-to-one correspondence with the one or more layers that generated the region information. Information processing system.

12. A recognition processing step of performing a recognition processing for recognizing an object by adding area information generated according to an object likelihood detected in the process of object recognition processing based on an output of a second sensor different from the first sensor to the output of the first sensor. Run the following on your computer: The recognition processing step includes: The recognition process is performed using an object recognition model obtained by machine learning; The object recognition model generates the region information in one or more layers included in a first convolutional layer generated based on an output of the second sensor, and adds the generated region information to one or more layers of a second convolutional layer generated based on the output of the first sensor and in one-to-one correspondence with the one or more layers that generated the region information. Information processing program for.

13. Executed by a processor, a recognition processing step of performing a recognition processing for recognizing an object by adding area information generated according to an object likelihood detected in the course of object recognition processing based on an output of a second sensor different from the first sensor to an output of the first sensor; Including, The recognition processing step includes: The recognition process is performed using an object recognition model obtained by machine learning; The object recognition model generates the region information in one or more layers included in a first convolutional layer generated based on an output of the second sensor, and adds the generated region information to one or more layers of a second convolutional layer generated based on the output of the first sensor and in one-to-one correspondence with the one or more layers that generated the region information. Information processing methods.

Images