Information processing apparatus, information processing system, information processing method, and program product

Abstract

[Problem] To provide a technology that makes it possible to quickly and accurately recognize an object. [Solution] An information processing apparatus according to the present technology includes a control section. The control section recognizes an object based on event information detected by an event-based sensor, and transmits a result of the recognition to a sensor apparatus including a sensor section capable of acquiring information about the object.

Description

Technical Field

[0001] This technology relates to techniques for identifying objects, such as those used to control autonomous driving. Background Technology

[0002] Autonomous driving for automobiles is divided into six levels, from Level 0 to Level 5, and it is expected that automobiles will develop in stages, progressing from manual driving at Level 0 to fully autonomous driving at Level 5. Currently, Level 2 partially autonomous driving technology has been put into practical use, and the next stage, Level 3 conditional autonomous driving, is being implemented.

[0003] In autonomous driving control, it is necessary to identify the environment around the vehicle (such as other vehicles, people, traffic lights, and traffic signs). Various sensors, such as cameras, light detection and ranging (LiDAR), millimeter-wave radar, and ultrasonic sensors, are used to perform sensing of the environment around the vehicle.

[0004] Patent document 1, shown below, discloses a technique for monitoring the road surface on which a vehicle intends to travel using an event-based (visual) sensor. The event-based sensor is a sensor capable of detecting changes in the brightness of each pixel. At the timing of a brightness change occurring in a portion, the event-based sensor can only output information about that portion.

[0005] Here, a conventional image sensor that outputs the entire image at a fixed frame rate is also called a frame-based sensor. Compared to frame-based sensors, the aforementioned type of sensor is called an event-based sensor. Event-based sensors capture changes in brightness as events.

[0006] Citation List

[0007] Patent documents

[0008] Patent Document 1: Japanese Patent Application Publication No. 2013-79937 Summary of the Invention

[0009] Technical issues

[0010] In this field, there is a need for technologies that enable the rapid and accurate identification of objects.

[0011] In view of the above, the purpose of this technology is to provide a technique that enables the rapid and accurate identification of objects.

[0012] Solution to the problem

[0013] The information processing apparatus according to this technology includes a controller.

[0014] The controller identifies the object based on event information detected by the event-based sensor and sends the identification result to a sensor device, which includes a sensor unit capable of acquiring information about the object.

[0015] Therefore, for example, by obtaining information about the part corresponding to the object from the sensor device, the object identified using event information can be quickly and accurately identified.

[0016] In the information processing device, the controller can identify the object, specify the location of the region of interest (ROI) corresponding to the object, and send the ROI location as the identification result to the sensor device.

[0017] In the information processing device, the sensor device can extract ROI information corresponding to the ROI location from the information acquired by the sensor unit, and can send the ROI information to the information processing device.

[0018] In an information processing device, the controller can identify objects based on ROI information obtained from sensor devices.

[0019] In the information processing device, the controller can design an autonomous driving plan based on information about objects identified based on ROI information.

[0020] In the information processing device, the controller can design an autonomous driving plan based on information about objects identified based on event information.

[0021] In the information processing device, the controller can determine whether an autonomous driving plan is scalable based solely on information about objects identified based on event information.

[0022] In the information processing device, when the controller has determined that the autonomous driving plan is not feasible, the controller can obtain ROI information and design an autonomous driving plan based on information about the objects identified based on the ROI information.

[0023] In the information processing device, when the controller has determined that the autonomous driving plan is designable, the controller can design the autonomous driving plan based on information about the objects identified based on event information without obtaining ROI information.

[0024] In the information processing device, the sensor unit may include an image sensor capable of acquiring an image of an object, and the ROI information may be an ROI image.

[0025] In the information processing device, the sensor unit may include a supplementary sensor that can acquire supplementary information, which is information about objects that the controller did not recognize using event information.

[0026] In the information processing device, the controller can acquire supplementary information from the sensor device, and based on the supplementary information, the controller can identify objects that were not identified using the event information.

[0027] In the information processing device, the controller can design an autonomous driving plan based on information about objects identified based on supplementary information.

[0028] In the information processing device, the controller can acquire information about the movement of the moving body, which is the goal of the autonomous driving plan, and based on the information about the movement, the controller can change the cycle of identifying objects based on supplementary information.

[0029] In information processing devices, controllers can make the cycle shorter as the movement of the moving body becomes slower.

[0030] In an information processing device, a sensor device can modify the cutout position of the ROI information based on the offset of the object in the ROI information.

[0031] The information processing system according to this technology includes an information processing device and a sensor device. The information processing device includes a controller. The controller identifies an object based on event information detected by an event-based sensor and sends the identification result to the sensor device, which includes a sensor unit capable of acquiring information about the object.

[0032] The information processing method according to this technology includes identifying an object based on event information detected by an event-based sensor; and sending the identification result to a sensor device, the sensor device including a sensor unit capable of acquiring information about the object.

[0033] The procedure according to this technology causes a computer to perform processing, including identifying an object based on event information detected by an event-based sensor; and sending the identification result to a sensor device, which includes a sensor unit capable of acquiring information about the object. Attached Figure Description

[0034] Figure 1 The figure illustrates an automatic driving control system according to a first embodiment of the present technology.

[0035] Figure 2 This is a block diagram illustrating the internal configuration of an automatic driving control system.

[0036] Figure 3The illustration shows vehicles, including DVS, driving on ordinary roads.

[0037] Figure 4 The illustration shows information about the edge of the vehicle ahead, obtained by DVS.

[0038] Figure 5 The illustration shows an image of a vehicle ahead, captured by an image sensor.

[0039] Figure 6 This is a flowchart illustrating the processes performed by the controller of the automatic driving control device.

[0040] Figure 7 This is a flowchart illustrating the processes performed by the controller of the sensor device.

[0041] Figure 8 The diagram illustrates the state of the generated recognition model.

[0042] Figure 9 The illustration shows an example of a specific block configuration in an autonomous driving control system.

[0043] Figure 10 This illustration shows another example of a specific block configuration in an autonomous driving control system. Detailed Implementation

[0044] Embodiments according to the present technology will now be described with reference to the accompanying drawings.

[0045] <<First Embodiment>>

[0046] <Overall configuration and configuration of each structural element>

[0047] Figure 1 The figure illustrates an automatic driving control system 100 according to a first embodiment of the present technology. Figure 2 This is a block diagram illustrating the internal configuration of the automatic driving control system 100.

[0048] The first embodiment describes an example in which an automated driving control system 100 (information processing system) is included in a vehicle to control the driving of the vehicle. Note that the moving body including the automated driving control system 100 (whether the moving body is manned or unmanned) is not limited to a car, but can be, for example, a motorcycle, train, airplane or helicopter.

[0049] like Figure 1 and 2As shown, the autonomous driving control system 100 according to the first embodiment includes a dynamic vision sensor (DVS) 10, a sensor device 40, an autonomous driving control device (information processing device) 30, and an autonomous driving execution device 20. The autonomous driving control device 30 can communicate with the DVS 10, the sensor device 40, and the autonomous driving execution device 20 via wired or wireless means.

[0050] [DVS]

[0051] The DVS10 is an event-based sensor. The DVS10 can detect changes in the brightness of incident light for each pixel. When a brightness change occurs in the area corresponding to a pixel, the DVS10 can output coordinate information and corresponding time information; the coordinate information is about the coordinates representing that area. The DVS10 generates timing data including coordinate information related to the brightness change at the microsecond level and sends this data to the automatic driving control unit 30. Note that the timing data acquired by the DVS10 and including coordinate information related to the brightness change is referred to as event information below.

[0052] Because the DVS10 only outputs information about the portions where brightness changes occur, it requires less data and has a much faster output speed (microseconds) compared to frame-based image sensors. Furthermore, the DVS10 performs logarithmic scaling and has a wide dynamic range. Therefore, the DVS10 can detect brightness changes in bright backlit conditions without highlight clipping, and conversely, it can also appropriately detect brightness changes in dark conditions.

[0053] [Example of event information obtained from DVS]

[0054] (While this vehicle is in motion)

[0055] This section describes what kind of event information is obtained from the DVS10 when it is included in a vehicle. Figure 3 The illustration shows vehicles, including the DVS10, driving on ordinary roads.

[0056] exist Figure 3 In the example shown, vehicle 1 (hereinafter referred to as vehicle 1), including DVS10 (automatic driving control system 100), is traveling in the left lane, while another vehicle 2 (hereinafter referred to as the preceding vehicle 2) is traveling in the same lane in front of vehicle 1. Additionally, in Figure 3 In the example shown, another vehicle 3 (hereinafter referred to as oncoming vehicle 3) is traveling towards vehicle 1 from its direction of travel in the opposite lane. Additionally, in Figure 3 For example, there are traffic lights 4, traffic signs 5, pedestrians 6, crosswalks 7, and lane markings 8 used to mark the boundaries between lanes.

[0057] Because the DVS10 can detect changes in brightness, the edge of an object with a velocity difference between itself (DVS10) and the object can essentially be detected as event information. Figure 3 In the example shown, there are speed differences between vehicle 1 and the vehicle in front 2, the oncoming vehicle 3, the traffic light 4, the traffic sign 5, the pedestrian 6, and the crosswalk 7. Therefore, the edges of these objects are detected as event information by DVS10.

[0058] Figure 4 The illustration shows information about the edge of the vehicle 2 ahead, obtained by DVS10. Figure 5 The illustration shows an example of an image of the vehicle 2 ahead, acquired by an image sensor.

[0059] exist Figure 3 In the example shown, Figure 4 The edges of the vehicle 2 ahead, as well as the edges of other vehicles such as the opposite vehicle 3, traffic light 4, traffic sign 5, pedestrian 6, and crosswalk 7, are detected as event information by DVS10.

[0060] Furthermore, DVS10 can detect objects whose brightness changes due to, for example, light emission, regardless of whether there is a speed difference between the vehicle 1 (DVS10) and the object. For example, the light section 4a in traffic light 4 flashes continuously for a period of time when the flashing is imperceptible to a human. Therefore, DVS10 can detect the light section 4a in traffic light 4 as a part with a change in brightness, regardless of whether there is a speed difference between the vehicle 1 and the light section 4a.

[0061] On the other hand, even if there is a speed difference between the vehicle 1 (DVS10) and the target object, there are exceptions where the target object is not captured as a portion of it that has changed brightness. There is a possibility that DVS10 may not be able to detect such a target object.

[0062] For example, when there is a straight dividing line 8, such as Figure 3 As shown, and when the vehicle 1 is traveling parallel to the partition line 8, the appearance of the partition line 8 does not change. Therefore, from the perspective of the vehicle 1, the brightness of the partition line 8 does not change. Thus, in this case, there is a possibility that the DVS10 will not detect the partition line 8 as having a change in brightness. Note that when the partition line 8 is not parallel to the direction of travel of the vehicle 1, the DVS10 can still detect the partition line 8 as usual.

[0063] For example, even if there is a speed difference between vehicle 1 and partition line 8, there is a possibility that partition line 8 may not be detected as a portion with a change in brightness. Therefore, in the first embodiment, supplementary information is performed based on supplementary information obtained by the supplementary sensor described later regarding such objects not detected by DVS10.

[0064] (When this vehicle stops)

[0065] Next, for example, in Figure 3 Assume that vehicle 1 is stopped while waiting for the traffic light to change. In this case, the object with a speed difference between vehicle 1 and the object (i.e., the edges of oncoming vehicle 3 and pedestrian 6 (when he / she is moving)) is detected as event information by DVS10. Additionally, regardless of whether there is a speed difference between vehicle 1 and light section 4a, DVS10 will detect the illuminated light section 4a in traffic light 4 as event information.

[0066] On the other hand, regarding objects where there is no speed difference between vehicle 1 (DVS10) and the target object due to vehicle 1 being stationary, there is a possibility that the edge of the target object may not be detected. For example, when a vehicle 2 in front of vehicle 1 stops to wait for a traffic light to change, similar to vehicle 1, the edge of vehicle 2 in front may not be detected. In addition, the edges of traffic light 4 and traffic sign 5 may also not be detected.

[0067] Note that in the first embodiment, supplementation is performed based on supplementary information about objects not detected by the DVS10, obtained by the supplementary sensor described later.

[0068] [Automatic driving control device 30]

[0069] Refer again Figure 2 The autonomous driving control device 30 includes a controller 31. The controller 31 performs various calculations based on various programs stored in a storage device (not shown) and performs overall control over the autonomous driving control device 30. The storage device stores various programs and various data necessary for the processing performed by the controller 31 of the autonomous driving control device 30.

[0070] The controller 31 of the autonomous driving control device 30 is implemented through hardware or a combination of hardware and software. The hardware is configured as part or all of the controller 31, and examples of hardware include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), and combinations of two or more of these. Note that this also applies to the controller 41 of the sensor device 40, which will be described later.

[0071] For example, the controller 31 of the autonomous driving control device 30 uses the DVS 10 to perform object recognition processing, performs processing to specify the location of the region of interest (ROI) corresponding to the object recognized by the DVS 10, and requests to acquire the ROI image corresponding to the ROI location. Additionally, for example, the controller 31 of the autonomous driving control device 30 performs object recognition based on the ROI image, designs a driving plan based on the object recognized based on the ROI image, and generates operation control data based on the designed driving plan.

[0072] Note that the processing performed by the controller 31 of the automatic driving control unit 30 will be described in detail later when the operation is described.

[0073] [Sensor device 40]

[0074] The sensor device 40 includes a controller 41 and a sensor unit 42 (sensor section). The sensor unit 42 can acquire information about the target object necessary for designing a driving plan. The sensor unit 42 includes sensors other than DVS10, and specifically, the sensor unit 42 includes an image sensor 43, a lidar 44, a millimeter-wave radar 45, and an ultrasonic sensor 46.

[0075] The controller 41 of the sensor device 40 performs various calculations based on various programs stored in a storage device (not shown) and performs overall control of the sensor device 40. The storage device stores various programs and various data necessary for the processing performed by the controller 41 of the sensor device 40.

[0076] For example, the controller 41 of the sensor device 40 performs ROI cutting out processing, which cuts out the portion corresponding to the ROI position from the overall image acquired by the image sensor 43, and modification processing, which modifies the ROI cutting out position.

[0077] Note that the processing performed by the controller 41 of the sensor device 40 will be described in detail later when the operation is described.

[0078] Image sensor 43 includes imaging devices such as charge-coupled device (CCD) sensors and complementary metal-oxide-semiconductor (CMOS) sensors, as well as optical systems such as image-forming lenses. Image sensor 43 is a frame-based sensor that outputs an overall image at a specific frame rate.

[0079] The lidar 44 includes a light emitting unit that emits laser light in the form of pulses, and a light receiving unit that can receive waves reflected from an object. The lidar 44 measures the time from the emission of laser light by the light emitting unit to the time the laser light is reflected by the object and received by the light receiving unit. Accordingly, the lidar 44 can detect, for example, the distance and orientation of the object. The lidar 44 can record the reflection direction and distance of the pulsed laser light in the form of points in a three-dimensional point set, and can acquire information about the environment surrounding the vehicle 1 in the form of a three-dimensional point set.

[0080] The millimeter-wave radar 45 includes a transmitting antenna capable of emitting millimeter waves (electromagnetic waves) with wavelengths on the order of millimeters, and a receiving antenna capable of receiving waves reflected from an object. For example, the millimeter-wave radar 45 can detect the distance to and orientation of an object based on the difference between the millimeter waves emitted by the transmitting antenna and received by the receiving antenna and the millimeter waves reflected from the object.

[0081] The ultrasonic sensor 46 includes a transmitter capable of emitting ultrasonic waves and a receiver capable of receiving waves reflected from an object. The ultrasonic sensor 46 measures the time from the emission of ultrasonic waves by the transmitter to the reflection of the ultrasonic waves from the object and their reception by the receiver. Accordingly, the ultrasonic sensor 46 can, for example, detect the distance to the object and the orientation of the object.

[0082] The five sensors, namely the four sensors 43, 44, 45 and 46 in sensor unit 42 and DVS 10, use protocols such as Precision Time Protocol (PTP) to synchronize with each other at the microsecond level.

[0083] The overall image captured by image sensor 43 is output to controller 41 of sensor device 40. Additionally, the overall image captured by image sensor 43 is sent as sensor information to the autonomous driving control unit. Similarly, information acquired by lidar 44, millimeter-wave radar 45, and ultrasonic sensor 46 is output as sensor information to autonomous driving control unit 30.

[0084] The sensor information acquired by each of the four sensors 43, 44, 45, and 46 is used to identify objects that were not identified using the event information acquired by the DVS10. In this sense, the sensor information acquired by each sensor is supplementary information.

[0085] In this description, the sensor that acquires ROI-cutout-target information is referred to as the ROI target sensor. Additionally, the sensor that acquires information (supplementary information) for identifying objects not identified using event information acquired by DVS 10 is referred to as the supplementary sensor.

[0086] In the first embodiment, image sensor 43 is a ROI target sensor because it acquires image information corresponding to the ROI cutout target. Additionally, image sensor 43 is also a supplementary sensor because it acquires image information as supplementary information. In other words, image sensor 43 serves as both a ROI target sensor and a supplementary sensor.

[0087] In addition, in the first embodiment, the lidar 44, the millimeter-wave radar 45, and the ultrasonic sensor 46 are supplementary sensors because each of the lidar 44, the millimeter-wave radar 45, and the ultrasonic sensor 46 acquires sensor information as supplementary information.

[0088] It should be noted that the ROI target sensor is not limited to image sensor 43. For example, instead of image sensor 43, LiDAR 44, millimeter-wave radar 45, or ultrasonic sensor 46 can be used as ROI target sensors. In this case, ROI cutout processing can be performed on the information acquired by LiDAR 44, millimeter-wave radar 45, or ultrasonic sensor 46 to obtain ROI information.

[0089] At least two of the four sensors—image sensor 43, lidar 44, millimeter-wave radar 45, and ultrasonic sensor 46—can be used as ROI target sensors.

[0090] In the first embodiment, the image sensor 43, lidar 44, millimeter-wave radar 45, and ultrasonic sensor 46 are used as supplementary sensors, and generally, it is sufficient if at least one of the four sensors is used as a supplementary sensor. Note that at least two of the four sensors can be used as both ROI target sensors and supplementary sensors.

[0091] [Automatic driving actuator 20]

[0092] Based on the operation control data from the autonomous driving control device 30, the autonomous driving actuator 20 performs autonomous driving by controlling, for example, the accelerator mechanism, the braking mechanism, and the steering mechanism.

[0093] <Description of Operation>

[0094] Next, the processes performed by the controller 31 of the autonomous driving control device 30 and the processes performed by the controller 41 of the sensor device 40 will be described. Figure 6 This is a flowchart illustrating the process performed by the controller 31 of the automatic driving control device 30. Figure 7 This is a flowchart illustrating the processing by the controller 41 of the sensor device 40.

[0095] refer to Figure 6First, the controller 31 of the automatic driving control device 30 obtains event information (including time-series data of coordinate information related to brightness changes) from the DVS 10. Figure 4 (See the edge information shown) (Step 101). Next, the controller 31 of the autonomous driving control device 30 identifies the objects necessary for designing the driving plan based on the event information (Step 102). Examples of objects necessary for designing the driving plan include the vehicle in front 2, the oncoming vehicle 3, the traffic light 4 (including the light part 4a), the traffic sign 5, the pedestrian 6, the crosswalk 7, and the lane markings 8.

[0096] Here, for example, when there is a speed difference between the vehicle 1 (DVS10) and each of the following: the vehicle ahead 2, the oncoming vehicle 3, the traffic light 4, the pedestrian 6, the crosswalk 7, and the lane marking 8, the autonomous driving controller 31 can essentially identify the vehicle ahead 2, the oncoming vehicle 3, the traffic light 4, the pedestrian 6, the crosswalk 7, and the lane marking 8 based on event information from the DVS. On the other hand, for example, there is a possibility that even if there is a speed difference between the vehicle 1 (DVS10) and the lane marking 8, the controller 31 of the autonomous driving control device 30 might exceptionally fail to identify the lane marking 8 based on event information from the DVS 10. Note that the controller 31 of the autonomous driving control device 30 can identify the illuminated portion 4a of the traffic light 4 based on event information from the DVS 10, regardless of whether there is a speed difference between the vehicle 1 (DVS10) and the illuminated portion 4a.

[0097] In step 102, the controller 31 of the autonomous driving control device 30 identifies the object by comparing it with a pre-stored first recognition model. Figure 8 The diagram illustrates the state of the generated recognition model.

[0098] like Figure 8 As shown, firstly, training data for the objects necessary for designing the driving plan is provided. Training data based on event information obtained when images of the objects are captured using DVS10 is used as training data for the objects. For example, data obtained by creating a database of information about movements performed on a timeline is used as training data, with information about movement included in time-series data that includes coordinate information (such as edges) related to changes in the brightness of the objects. Using the training data, learning is performed by machine learning, such as using a neural network, and a first recognition model is generated.

[0099] After identifying the objects necessary for designing a driving plan based on event information from DVS10, the controller 31 of the autonomous driving control unit 30 determines whether the driving plan is designable, without acquiring ROI images, using only information about the objects identified based on event information from DVS10 (step 103).

[0100] For example, when in Figure 2 When the vehicle 2 in front may collide with the vehicle 1 due to sudden braking, the controller 31 of the automatic driving control device 30 learns from the event information that the vehicle 2 in front is likely to collide with the vehicle 1 (because the edge of the vehicle 2 in front is approaching the vehicle 1).

[0101] Additionally, for example, when Figure 2 If pedestrian 6 is likely to run in front of vehicle 1, the controller 31 of the automatic driving control device 30 can know from the event information that pedestrian 6 is likely to run in front of vehicle 1 (because the edge of pedestrian 6 is about to cross in front of vehicle 1).

[0102] For example, in such an emergency, the controller 31 of the autonomous driving control device 30 determines that the driving plan is designable without acquiring an ROI image, using only information about the object identified based on event information from DVS10 ("Yes" in step 103).

[0103] In this scenario, the controller 31 of the autonomous driving control device 30 does not send a ROI image acquisition request to the sensor device 40, and only uses information about the objects identified by the DVS 10 to design an autonomous driving plan (step 110). Then, the controller 31 of the autonomous driving control device 30 generates operation control data consistent with the designed autonomous driving plan based on the autonomous driving plan (step 111), and sends the generated operation control data to the autonomous driving execution device 20 (step 112).

[0104] Here, as described above, the event information is output by DVS10 at high speed, and the amount of event information is small. Therefore, for example, identifying an object takes less time compared to globally analyzing the entire image from image sensor 43 to identify the object. Therefore, for example, in the aforementioned emergency situation, by quickly designing a driving plan using only information about the object identified based on the event information, the emergency can be avoided.

[0105] When it has been determined in step 103 that the autonomous driving plan cannot be designed using only information about objects identified based on event information from DVS10 (No in step 103), the controller 31 of the autonomous driving control device 30 moves to step 104, which follows step 103. Note that, except in the aforementioned emergency situations, it is generally determined that the autonomous driving plan is undesignable.

[0106] In step 104, the controller 31 of the autonomous driving control device 30 designates a specific region corresponding to an object from the coordinate positions included in the event information from DVS 10 as a Region of Interest (ROI). The number of ROIs designated as corresponding to an object can be one, two, or more. For example, when there is one object identified based on the event information from DVS 10, there is one ROI corresponding to the number of object objects. When there are two or more object objects identified based on the event information from DVS 10, there are two or more ROI locations corresponding to the number of object objects.

[0107] Next, the controller 31 of the autonomous driving control device 30 sends an ROI image acquisition request, including information about the ROI location, to the sensor device 40 (step 105).

[0108] refer to Figure 7 The controller 41 of the sensor device 40 determines whether it has received an ROI image acquisition request from the autonomous driving control device 30 (step 201). When the controller 41 of the sensor device 40 determines that it has not received an ROI image acquisition request ("No" in step 201), the controller 41 of the sensor device 40 determines again whether it has received an ROI image acquisition request from the autonomous driving control device 30. In other words, the controller 41 of the sensor device 40 waits to receive an ROI image acquisition request.

[0109] When the controller 41 of the sensor device 40 has determined that it has received a ROI image acquisition request from the autonomous driving control device 30 ("Yes" in step 201), the controller 41 of the sensor device 40 acquires the overall image from the image sensor 43 (step 202). Next, the controller 41 of the sensor device 40 selects one of the ROI locations included in the ROI image acquisition request (step 203).

[0110] Next, the controller 41 of the sensor device 40 sets the cutout position for the ROI image in the overall image (step 204), and cuts out the ROI image corresponding to the ROI position from the overall image (step 205).

[0111] Next, the controller 41 of the sensor device 40 analyzes the ROI image to determine the offset of the object within the ROI image (step 206). In other words, the controller 41 of the sensor device 40 determines whether the object is correctly located within the ROI image.

[0112] Next, the controller 41 of the sensor device 40 determines whether the offset is less than or equal to a specified threshold (step 207). When the controller 41 of the sensor device 40 determines that the offset is greater than the specified threshold ("No" in step 207), the controller of the sensor device 40 modifies the ROI cutout position according to the offset (step 208). Then, the controller 41 of the sensor device 40 cuts out the ROI image from the overall image again, corresponding to the modified ROI cutout position.

[0113] When the controller 41 of the sensor device 40 determines in step 207 that the offset is less than or equal to a specified threshold ("Yes" in step 207), the controller 41 of the sensor device 40 determines whether there is another ROI location that has not yet been cut out of the ROI image (step 209). When the controller 41 of the sensor device 40 has determined that there is another ROI location ("Yes" in step 209), the controller 41 of the sensor device 40 returns to step 203, selects one of the remaining ROI locations, and cuts out the ROI image corresponding to the selected ROI location from the overall image.

[0114] Note that, as can be seen from the description in this article, the ROI image (ROI information) is a partial image that is cut out from the overall image acquired by the image sensor 43 as the part corresponding to the ROI location.

[0115] For example, suppose that when, for instance, the vehicle ahead 2, the oncoming vehicle 3, the traffic light 4 (including the light portion 4a), the traffic sign 5, the pedestrian 6, the crosswalk 7, and the dividing line 8 are identified as objects based on event information from DVS 10, the locations corresponding to these objects are determined as ROI locations. In this case, the portions corresponding to, for instance, the vehicle ahead 2, the oncoming vehicle 3, the traffic light 4 (including the light portion 4a), the traffic sign 5, the pedestrian 6, the crosswalk 7, and the dividing line 8 are cut out from the overall image acquired by image sensor 43, and corresponding ROI images are generated. Note that one ROI image corresponds to one object (one ROI location).

[0116] Note that the controller 41 of the sensor device 40 can determine not only the offset of the object in the ROI image, but also the amount of exposure performed when the image sensor 43 captures the image from which the ROI image is generated. In this case, the controller 41 of the sensor device 40 analyzes the ROI image to determine whether the amount of exposure performed when capturing the image from which the ROI image is generated is within an appropriate range. When the controller 41 of the sensor device 40 has determined that the exposure is not within an appropriate range, the controller 41 of the sensor device 40 generates information about the exposure for modifying the exposure, and adjusts the amount of exposure performed by the image sensor 43.

[0117] When the controller 41 of the sensor device 40 determines in step 209 that the ROI images corresponding to all ROI locations have been cut out ("No" in step 209), the controller 41 of the sensor device 40 determines whether there are multiple generated ROI images (step 210). When the controller 41 of the sensor device 40 determines that there are multiple ROI images ("No" in step 210), the controller 41 of the sensor device 40 generates ROI-related information (step 211) and moves to step 212, which follows step 211.

[0118] Describe ROI-related information. When multiple ROI images exist, the ROI images of the multiple ROI images are combined to send to the autonomous driving control unit 30 as a single combined image. ROI-related information is used to identify which parts of the single combined image correspond to which ROI images.

[0119] When the controller 41 of the sensor device 40 determines in step 210 that a single ROI image exists (No in step 210), the controller 41 of the sensor device 40 does not generate ROI-related data and moves to step 212.

[0120] In step 212, the controller 41 of the sensor device 40 performs image processing on the ROI image. This image processing enables the controller 31 of the autonomous driving control device 30 to accurately identify the object in step 109, which will be described later (see [link to article]). Figure 6 ).

[0121] Examples of image processing include digital gain processing, white balance, lookup table (LUT) processing, color matrix transformation, defect correction, shot correction, denoising, gamma correction, and demosaic (e.g., returning from a Bayer arrangement output by an imaging device to an RGB arrangement).

[0122] After performing image processing on the ROI image, the controller 41 of the sensor device 40 sends the ROI image information to the autonomous driving control device 30 (step 213). Note that when a single ROI image exists, the controller 41 of the sensor device 40 sends the single ROI image as ROI image information to the autonomous driving control device 30. On the other hand, when multiple ROI images exist, the controller 41 of the sensor device 40 combines the ROI images of the multiple ROI images to obtain a single combined image, and sends the single combined image as ROI image information to the autonomous driving control device 30. In this case, ROI-related information is included in the ROI image information.

[0123] When the controller 41 of the sensor device 40 sends the ROI image information to the autonomous driving control device 30, the controller 41 of the sensor device 40 returns to step 201 and determines whether it has received the ROI image acquisition request from the autonomous driving control device 30.

[0124] Refer again Figure 6 After sending a ROI image acquisition request to the sensor device 40, the controller 31 of the autonomous driving control device 30 determines whether it has received ROI image information from the sensor device 40 (step 106).

[0125] When the controller 31 of the autonomous driving control device 30 has determined that it has not yet received ROI image information ("No" in step 106), the controller 31 of the autonomous driving control device 30 determines again whether it has received ROI image information. In other words, the controller 31 of the autonomous driving control device 30 waits to receive ROI image information after making an ROI image acquisition request.

[0126] When the controller 31 of the autonomous driving control device 30 has determined that it has received ROI image information ("Yes" in step 106), the controller 31 of the autonomous driving control device 30 determines whether the received ROI image information is a combined image obtained by combining multiple ROI images (step 107).

[0127] When the controller 31 of the autonomous driving control device 30 determines that the received ROI image information is a combined image obtained by combining multiple ROI images ("Yes" in step 107), the controller 31 of the autonomous driving control device 30 moves to step 109 based on the ROI, which follows step 108. On the other hand, when the controller 31 of the autonomous driving control device 30 determines that the received ROI image information is a single ROI image ("No" in step 107), the controller 31 of the autonomous driving control device 30 does not perform the separation process and moves to step 109.

[0128] In step 109, the controller 31 of the autonomous driving control device 30 designs the object necessary for the driving plan based on ROI image recognition. In this case, the object recognition process is performed by comparing the object with a pre-stored second recognition model.

[0129] refer to Figure 8 The second recognition model is essentially generated based on a concept similar to that of the first recognition model. However, unlike the first recognition model, which uses data based on event information obtained when capturing images of the object using the DVS10 as training data, the second recognition model uses data based on image information obtained when capturing images of the object by the image sensor 43 as training data. Using this image-based training data, learning is performed using machine learning, such as a neural network, to generate the second recognition model.

[0130] When the controller 31 of the autonomous driving control unit 30 performs object recognition processing based on ROI image, it makes it possible to identify objects in greater detail compared to identifying objects based on event information. For example, the controller can identify, for example, the license plate numbers and brake light colors of each of the vehicles in front 2 and oncoming 3, the color of the light portion 4a in the traffic light 4, the lettering on the traffic sign 5, the direction of the pedestrian 6's face, and the color of the dividing line 8.

[0131] After identifying the object based on the ROI image, the controller 31 of the autonomous driving control device 30 designs an autonomous driving plan based on information about the object identified based on the ROI image (and information about the object identified based on event information) (step 110). Then, the controller 31 of the autonomous driving control device 30 generates operation control data consistent with the designed autonomous driving plan based on the autonomous driving plan (step 111), and sends the generated operation control data to the autonomous driving execution device 20 (step 112).

[0132] In other words, this embodiment employs a method of acquiring ROI images by specifying the ROI locations corresponding to objects necessary for designing a driving plan based on event information from DVS10, and identifying objects based on the acquired ROI images.

[0133] As described above, instead of acquiring the entire image, this embodiment acquires a ROI image to identify the object. Therefore, compared to acquiring the entire image each time, this embodiment has the advantage of smaller data volume and therefore shorter image acquisition time.

[0134] Furthermore, object identification is achieved using ROI images, which reduce data volume through ROI processing. Therefore, this embodiment has the advantage of shorter object identification time compared to globally analyzing the entire image. Moreover, this embodiment makes accurate object identification possible because object identification is based on ROI images. In other words, this embodiment makes it possible to identify object quickly and accurately.

[0135] Here, there is a possibility that using event information from DVS10 will fail to identify an object with no speed difference between the vehicle 1 (DVS10) and the target object. Therefore, there is a possibility that such an object cannot be identified using the ROI image. Therefore, in this embodiment, the controller 31 of the autonomous driving control device 30 identifies the target object necessary for designing the driving plan based not only on the ROI image but also on supplementary information from the sensor unit 42 in the sensor device 40.

[0136] For example, the partition line 8 extending parallel to the moving vehicle 1, and objects that are no longer captured as parts of which brightness changes because the vehicle 1 has stopped, are identified by the controller 31 of the autonomous driving control device 30 based on supplementary information from the sensor unit 42.

[0137] The controller 31 of the autonomous driving control device 30 repeatedly performs a series of processes at a specified cycle, including specifying the ROI location in the event information, acquiring the ROI image, and identifying the objects necessary for designing a driving plan based on the ROI image, as described above. Figure 6 (Steps 101 to 109). Note that this series of processes is referred to below as a series of recognition processes based on the ROI image.

[0138] In addition, in parallel with performing a series of recognition processes based on the ROI image, the controller 31 of the autonomous driving control device 30 repeatedly performs a series of processes at specified intervals, including acquiring supplementary information from the sensor device 40 and identifying objects necessary for designing a driving plan based on the supplementary information. Note that this series of processes is referred to below as a series of recognition processes based on supplementary information.

[0139] In a series of recognition processes based on supplementary information, the controller 31 of the autonomous driving control device 30 identifies the object by globally analyzing the corresponding supplementary information from the four sensors in the sensor unit 42. Therefore, the controller 31 of the autonomous driving control device 30 can also appropriately identify objects that were not identified using event information or ROI images.

[0140] In a series of recognition processes based on supplementary information, it is necessary to globally analyze each piece of supplementary information from the sensors. Therefore, the series of recognition processes based on supplementary information takes longer than when analyzing ROI images. Consequently, the series of recognition processes based on supplementary information is performed on a cycle that is several times longer than the cycle of performing a series of recognition processes based on ROI images.

[0141] For example, each time a series of recognition processes based on the ROI image is repeated several times, a series of recognition processes based on supplementary information is performed once. In other words, when an object is identified based on the ROI image through a series of recognition processes based on the ROI image (see step 109), the object is identified once based on supplementary information each time the series of recognition processes based on the ROI image is repeated several times. At this time, an autonomous driving plan is designed using information about the object identified based on the ROI image and information about the object identified based on supplementary information (as well as information about the object identified based on event information) (see step 110).

[0142] Here, compared to when vehicle 1 is moving, there are more instances where there is no speed difference between vehicle 1 and the object when vehicle 1 is stationary. Therefore, it is more difficult to identify the object in the event information when vehicle 1 is stationary compared to when vehicle 1 is moving.

[0143] Therefore, the controller 31 of the automatic driving control device 30 can acquire information about the movement of the vehicle 1, and can adjust the cycle of performing a series of identification processes based on supplementary information based on the information about the movement of the vehicle 1. The information about the movement of the vehicle 1 can be obtained from information about the speedometer and information such as that from the Global Positioning System (GPS).

[0144] In this scenario, for example, the cycle of performing a series of recognition processes based on supplementary information can be shortened as the vehicle 1 moves slower. This makes it possible, for example, to use the supplementary information to appropriately identify objects that the DVS10 failed to capture due to the slower movement of the vehicle 1 as portions of them exhibiting changes in brightness.

[0145] It should be noted that, conversely, the cycle of performing a series of identification processes based on supplementary information can become shorter as the vehicle 1 moves faster. This is based on the idea that if the vehicle 1 moves faster, then more accurate object identification will be required.

[0146] <Specific Block Configuration: First Example>

[0147] Next, the configuration of specific blocks in the automatic driving control system 100 will be described. Figure 9 The illustration shows an example of a specific block configuration in the automatic driving control system 100.

[0148] It should be noted that, Figure 9 In the middle, the following was omitted. Figure 2 The sensor unit 42 contains four sensors: a lidar 44, a millimeter-wave radar 45, and an ultrasonic sensor 46. Only the image sensor 43 is shown. Additionally, in... Figure 9 In the middle, it was also omitted Figure 2 The flow of sensor information (supplementary information) in sensor unit 42 is illustrated, and only the flow of ROI image is shown.

[0149] like Figure 9 As shown, the automatic driving control device 30 includes an object recognition unit 32, an automatic driving planning unit 33, an operation controller 34, a synchronization signal generator 35, an image data receiver 36, and a decoder 37.

[0150] Additionally, the sensor device 40 includes a sensor block 47 and a signal processing block 48. The sensor block 47 includes an image sensor 43, a central processing unit 49, a ROI cutout unit 50, an ROI analyzer 51, an encoder 52, and an image data transmitter 53. The signal processing block 48 includes a central processing unit 54, an information extraction unit 55, an ROI image generator 56, an image analyzer 57, an image processor 58, an image data receiver 59, a decoder 60, an encoder 61, and an image data transmitter 62.

[0151] Note that, Figure 2 The controller 31 of the autonomous driving control device 30 shown herein is, for example, Figure 9 The object recognition unit 32, the automatic driving planning unit 33, the operation controller 34, and the synchronization signal generator 35 shown correspond to each other. Additionally, Figure 2 The controller 41 of the sensor device 40 shown herein is, for example, Figure 9 The sensor block 47 shown includes a central processing unit 49, an ROI cutout unit 50, and an ROI analysis unit 51; and Figure 9 The signal processing block 48 shown corresponds to the central processing unit 54, information extraction unit 55, ROI image generator 56, image analyzer 57 and image processor 58.

[0152] "Autopilot control device"

[0153] First, the autonomous driving control unit 30 is described. A synchronization signal generator 35 is configured to generate synchronization signals according to a protocol such as Precise Time Protocol (PTP) and output these synchronization signals to the DVS10, image sensor 43, lidar 44, millimeter-wave radar 45, and ultrasonic sensor 46. Accordingly, the five sensors, including the DVS10, image sensor 43, lidar 44, millimeter-wave radar 45, and ultrasonic sensor 46, are synchronized with each other, for example, at the microsecond level.

[0154] The object recognition unit 32 is configured to acquire event information from the DVS 10 and identify objects necessary for designing the driving plan based on the event information (refer to steps 101 and 102). The object recognition unit 32 is configured to output information about the objects identified based on the event information to the autonomous driving planning unit 33.

[0155] Furthermore, the object recognition unit 32 is configured to determine whether the ROI image information is a composite image obtained by combining multiple ROI images after receiving ROI image information from the sensor device 40 (refer to step 107). When the ROI image information is a composite image obtained by combining multiple ROI images, the object recognition unit 32 is configured to separate the composite image into corresponding ROI images based on ROI-related information (refer to step 108).

[0156] Furthermore, the object recognition unit 32 is configured to identify objects necessary for the autonomous driving plan based on ROI image recognition (refer to step 109). Additionally, the object recognition unit 32 is configured to output information about the object identified based on the ROI image to the autonomous driving plan unit 33.

[0157] Furthermore, the object recognition unit 32 is configured to identify objects necessary for designing an autonomous driving plan based on supplementary information acquired by the sensor device 40. The object recognition unit 32 outputs information about the object identified based on the supplementary information to the autonomous driving plan unit 33.

[0158] The autonomous driving planning unit 33 is configured to determine whether a driving plan is designable using only the information about the object identified based on event information after acquiring information about the object identified based on event information, without acquiring an ROI image. The information about the object identified based on event information is acquired from the object recognition unit 32 (see step 103).

[0159] The autonomous driving planning unit 33 is configured to design an autonomous driving plan using only the information about the object identified based on event information when the driving plan can be designed (refer to the processing from "Yes" in step 103 to step 110).

[0160] In addition, the autonomous driving planning unit 33 is configured to designate a specific area as the ROI location when a driving plan cannot be designed using only this information. This specific area comes from the coordinate location included in the event information obtained from DVS10 and corresponds to the object (see step 104).

[0161] Additionally, the autonomous driving planning unit 33 is configured to send an ROI image acquisition request to the sensor device 40 after specifying the ROI location. This ROI image acquisition request includes information about the ROI location (refer to step 105). Furthermore, the autonomous driving planning unit 33 is configured to send a supplementary information acquisition request to the sensor device 40.

[0162] In addition, the autonomous driving planning unit 33 is configured to design an autonomous driving plan based on the information about the object identified in the ROI image (and the information about the object identified based on event information) after acquiring information about the object identified in the ROI image. The information about the object identified in the ROI image is obtained from the object identification unit 32 (see steps 109 and 110).

[0163] In addition, the autonomous driving planning unit 33 is configured to design an autonomous driving plan based on the information about the object identified based on the ROI image and the information about the object identified based on the supplementary information (as well as the information about the object identified based on the event information) after acquiring information about the object identified based on the supplementary information. The information about the object identified based on the supplementary information is acquired from the object identification unit 32.

[0164] In addition, the autonomous driving planning unit 33 is configured to output the designed autonomous driving plan to the operation controller 34.

[0165] The operation controller 34 generates operation control data consistent with the acquired autonomous driving plan based on the autonomous driving plan obtained from the autonomous driving planning unit 33 (step 111), and outputs the generated operation control data to the autonomous driving execution device 20 (step 112).

[0166] The image data receiver is configured to receive ROI image information transmitted from the sensor device 40 and output the received information to the decoder. The decoder is configured to decode the ROI image information and output the information obtained through decoding to the object recognition unit 32.

[0167] "Sensor device"

[0168] (Sensor block)

[0169] Next, the sensor block 47 of the sensor device 40 will be described. The central processing unit 49 of the sensor block 47 is configured to set the ROI cut-out position based on information about the ROI location included in the ROI acquisition request sent from the autonomous driving control device 30 (refer to step 204). In addition, the central processing unit 49 of the sensor block 47 is configured to output the set ROI cut-out position to the ROI cut-out section 50.

[0170] Additionally, the central processing unit 49 of the sensor block 47 is configured to modify the ROI cutout position based on the offset of the object in the ROI image analyzed by the image analyzer 57 of the signal processing block 48 (refer to steps 207 and 208). Furthermore, the central processing unit 49 of the sensor block 47 is configured to output the modified ROI cutout position to the ROI cutout section 50.

[0171] Additionally, the central processing unit 49 of the sensor block 47 is configured to adjust the amount of exposure performed with respect to the image sensor 43 based on the amount of exposure performed when capturing an image from which a ROI image is generated, the ROI image being analyzed by the image analyzer 57 of the signal processing block.

[0172] The ROI cutout unit 50 is configured to acquire an overall image from the image sensor 43 and cut out a portion corresponding to the ROI cutout position from the overall image to generate an ROI image (refer to step 205). Additionally, the ROI cutout unit 50 is configured to output information about the generated ROI image to the encoder 52.

[0173] Furthermore, the ROI cutout unit 50 is configured to combine the ROI images of the multiple ROI images to generate a combined image when multiple ROI images are generated from the overall image, and output the combined image to the encoder 52 as ROI image information. The ROI cutout unit 50 is also configured to generate ROI-related information at this time (refer to step 211) and output the ROI-related information to the ROI analyzer 51.

[0174] The ROI analyzer 51 is configured to convert the ROI-related information obtained from the ROI cutout section 50 into ROI-related information for encoding, and output the ROI-related information for encoding to the encoder 52.

[0175] Encoder 52 is configured to encode ROI image information and output the encoded ROI image information to image data transmitter 53. Additionally, encoder 52 is configured to encode ROI-related information when such information exists, and to include the encoded ROI-related information in the encoded ROI image information, outputting the encoded ROI image information to image data transmitter 53.

[0176] Image data transmitter 53 is configured to send encoded ROI image information to signal processing block 48.

[0177] (Signal processing block)

[0178] Next, the signal processing block 48 in the sensor device 40 will be described. The image data receiver 59 is configured to receive encoded ROI image information and output the received encoded ROI image information to the decoder 60.

[0179] Decoder 60 is configured to decode the encoded ROI image information. Furthermore, decoder 60 is configured to output the decoded ROI image information to ROI image generator 56. Additionally, decoder 60 is configured to generate ROI-related information for decoding when ROI-related information is included in the ROI image information (when the ROI image information is a composite image obtained by combining multiple ROI images), and output the generated ROI-related information for decoding to information extraction unit 55.

[0180] The information extraction unit 55 is configured to convert ROI-related information used for decoding into ROI-related information, and output the ROI-related information obtained through the conversion to the ROI image generator 56. The ROI image generator 56 is configured to separate the combined image into corresponding ROI images based on the ROI-related information when the ROI image information is a composite image obtained by combining multiple ROI images. Furthermore, the ROI image generator 56 is configured to output the ROI images to the image analyzer 57.

[0181] Image analyzer 57 is configured to analyze the ROI image to determine the offset of the object within the ROI image (refer to step 206), and output the offset to central processing unit 54. Additionally, image analyzer 57 is configured to analyze the ROI image to determine the amount of exposure performed when capturing the image from which the ROI image is generated, and output the exposure amount to central processing unit 54. Furthermore, image analyzer 57 is configured to output the ROI image to image processor 58.

[0182] Image processor 58 is configured to perform image processing on the ROI image based on image processing control information from central processor 54 (see step 212). Additionally, image processor 58 is configured to output the ROI image to encoder.

[0183] The central processing unit 54 is configured to receive an ROI acquisition request, including the ROI location, from the autopilot control unit 30 and send the ROI acquisition request to the sensor block 47. Additionally, the central processing unit 54 is configured to send to the sensor block 47 information about the alignment of the object and information about the exposure, obtained through analysis performed by the image analyzer 57.

[0184] Additionally, the central processing unit 54 is configured to output image processing control information to the image processor 58. For example, the image processing control information is information used to cause the image processor 58 to perform image processing such as digital gain processing, white balance, lookup table (LUT) processing, color matrix transformation, defect correction, shot correction, noise reduction, gamma correction, and depigmentation.

[0185] In addition, the central processing unit 54 is configured to acquire supplementary information from the sensor unit 42 in response to a supplementary information acquisition request from the autonomous driving control unit 30, and send the supplementary information to the autonomous driving control unit 30.

[0186] Encoder 61 is configured to encode ROI image information and output the encoded ROI image information to image data transmitter 62. Additionally, encoder 61 is configured to encode ROI-related information when such information exists, including the encoded ROI-related information in the encoded ROI image information, so that the encoded ROI image information is output to image data transmitter 62.

[0187] Image data transmitter 62 is configured to send encoded ROI image information to autonomous driving control unit 30.

[0188] <Specific Block Configuration: Second Example>

[0189] Next, another example of the feature block configuration in the automatic driving control system 100 will be described. Figure 10 The illustration shows another example of a specific block configuration in the automatic driving control system 100.

[0190] exist Figure 10 In the example shown, the description focuses on... Figure 9 At different points. Figure 9 In the example shown, the ROI cutout portion 50 and the ROI analyzer 51 are provided to the sensor block 47 of the sensor device 40, while Figure 10In the example shown, they are provided to the signal processing block 48 of the sensor device 40.

[0191] In addition, Figure 9 In the example shown, the information extraction unit 55, the ROI image generator 56, the image analyzer 57, and the image processor 58 are provided to the signal processing block 48 of the sensor device 40, while... Figure 10 In the example shown, those are provided to the autonomous driving control unit 30.

[0192] Here, Figure 2 The controller 31 of the automatic driving control device 30 corresponds to the synchronization signal generator 35, the object recognition unit 32, the automatic driving planning unit 33, the operation controller 34, the information extraction unit 55, the ROI image generator 56, the image analyzer 57, and the image processor 58. Furthermore, Figure 2 The controller 41 of the sensor device 40 and the central processing unit 49 of the sensor block 47; and Figure 10 The signal processing block 48 corresponds to the central processing unit 49, the ROI cutout unit 50, and the ROI analyzer 51.

[0193] exist Figure 10 In the example shown, the image analyzer 57 and image processor 58 are not provided on the sensor device 40 side, but on the autonomous driving control device 30 side. Therefore, the determination of the offset of the object in the ROI image, the determination of the exposure performed with respect to the image sensor 43, and the image processing of the ROI image are not performed on the sensor side, but are performed on the autonomous driving control device 30 side. In other words, these processes can be performed on the sensor device 40 side or on the autonomous driving control device 30 side.

[0194] exist Figure 10 In the example shown, the ROI image is not cut out by sensor block 47, but by signal processing block 48. Therefore, it is not the ROI image, but the entire image that is sent from sensor block 47 to signal processing block 48.

[0195] Signal processing block 48 is configured to receive the overall image from sensor block 47 and generate an ROI image corresponding to the ROI location from the overall image. Furthermore, signal processing block 48 is configured to output the generated ROI image as ROI image information to the autonomous driving control device 30.

[0196] Additionally, signal processing block 48 is configured to generate ROI-related information and a combined image when generating multiple ROI images from a single overall image. The combined image is obtained by combining multiple ROI images. In this case, signal processing block 48 is configured to use the combined image as ROI image information and include ROI-related information in the ROI image information to send the ROI image information to the autonomous driving control device 30.

[0197] exist Figure 10 In the example shown, by Figure 9 In the example shown, part of the processing performed by the central processing unit 49 of the sensor block 47 is performed by the central processing unit 54 of the signal processing block 48.

[0198] In other words, the central processing unit 54 of the signal processing block 48 is configured to set the ROI cut-out position 30 based on information about the ROI position included in the ROI acquisition request sent from the autonomous driving control device. Furthermore, the central processing unit 54 of the signal processing block 48 is configured to output the set ROI cut-out position to the ROI cut-out unit 50.

[0199] Furthermore, the central processing unit 54 of the signal processing block 48 is configured to modify the ROI cutout position based on the offset of the object in the ROI image analyzed by the image analyzer 57 of the automatic driving control device 30. Then, the central processing unit 54 of the signal processing block 48 is configured to output the modified ROI cutout position to the ROI cutout unit 50.

[0200] exist Figure 10 In the example shown, apart from the addition of an information extraction unit 55, a ROI image generator 56, an image analyzer 57, and an image processor 58, the autonomous driving control device 30 is essentially similar to Figure 9 The automatic driving control device 30 shown. However, in Figure 10 In the example shown, by Figure 9 In the example shown, part of the processing performed by the central processing unit 54 of the signal processing block 48 in the sensor device 40 is performed by the autonomous driving planning unit 33 of the autonomous driving control device 30.

[0201] In other words, the autonomous driving planning unit 33 is configured to send information about the alignment of the object and information about the exposure obtained through analysis performed by the image analyzer 57 to the sensor device 40. Additionally, the autonomous driving planning unit 33 is configured to output image processing control information to the image processor 58.

[0202] <Effects and Others>

[0203] As described above, this embodiment employs a method of acquiring ROI images by specifying the ROI locations corresponding to objects necessary for designing a driving plan based on event information from DVS10, and identifying objects based on the acquired ROI images.

[0204] In other words, in this embodiment, instead of acquiring the entire image, a ROI image is obtained to identify the object. Therefore, compared to acquiring the entire image each time, this embodiment has the advantage of smaller data volume and therefore shorter image acquisition time.

[0205] Furthermore, object identification is achieved using ROI images, which have reduced data volume through ROI processing. Therefore, this embodiment has the advantage of shorter object identification time compared to globally analyzing the entire image. Moreover, this embodiment makes accurate object identification possible because it is based on ROI images. In other words, this embodiment makes fast and accurate object identification possible.

[0206] Note that this embodiment includes a process of acquiring event information from the DVS10 to specify the ROI location, which differs from the case of acquiring the entire image and globally analyzing it to identify the object. Therefore, to compare the time spent identifying the object using these two methods, it is necessary to consider both the time spent acquiring the event information and the time spent specifying the ROI location. However, as mentioned above, the DVS10 outputs event information at high speed, and the amount of event information is small. Therefore, specifying the ROI location corresponding to the object also takes less time. Thus, even considering the above points, this embodiment, which acquires and analyzes the ROI image to identify the object, makes it possible to reduce the time spent identifying the object compared to acquiring and analyzing the entire image.

[0207] Furthermore, this embodiment makes it possible to design autonomous driving plans based on information about objects quickly and accurately identified from ROI images. This leads to improved safety and reliability of autonomous driving.

[0208] Furthermore, in this embodiment, the ROI position is set based on event information from DVS10. Therefore, appropriate positions corresponding to the object can be cut from each overall image in the left, right, up, and down directions to generate ROI images.

[0209] Furthermore, in this embodiment, the ROI cutout position for the ROI image is modified based on the offset of the object in the ROI image. This makes it possible to generate an ROI image by appropriately cutting out the object.

[0210] In addition, in this embodiment, when the autonomous driving plan does not require obtaining ROI images and can be designed using only information about objects identified based on event information from DVS10, the autonomous driving plan is designed using only this information.

[0211] Here, as mentioned above, the event information is output at high speed by DVS10, and the amount of event information is small. Therefore, for example, identifying an object takes less time compared to globally analyzing the entire image from image sensor 43 to identify the object. Therefore, for example, in emergency situations such as a possible collision between another vehicle and vehicle 1, or a pedestrian 6 potentially running in front of vehicle 1, the emergency can be avoided by quickly designing a driving plan using only information about the object identified based on the event information.

[0212] In addition, in this embodiment, supplementary information is acquired from supplementary sensors, and object recognition is based on the supplementary information. This also makes it possible to properly recognize object objects that were not recognized based on event information or ROI images (such as partition lines 8 extending parallel to the moving vehicle 1, or object objects that are no longer captured as having a portion with brightness changes due to the vehicle 1 stopping).

[0213] Furthermore, this embodiment makes it possible to design autonomous driving plans based on information about objects accurately identified using supplementary information. This leads to further improvements in the safety and reliability of autonomous driving.

[0214] Furthermore, in this embodiment, the period for identifying objects based on supplementary information is changed based on information about the movement of the vehicle 1. This makes it possible to appropriately change the period according to the movement of the vehicle 1. In this case, when the period becomes shorter as the movement of the vehicle 1 becomes slower, it makes it possible, for example, to appropriately identify objects that were not captured by the DVS10 as having a change in brightness due to the slower movement of the vehicle 1.

[0215] <<Various Revisions>>

[0216] The above has described examples of using the object recognition technology according to this invention to identify objects in autonomous driving control. On the other hand, the object recognition technology according to this invention can also be used for purposes other than autonomous driving control. For example, the object recognition technology according to this invention can be used to detect product defects caused on a production line, or it can be used to identify objects as overlay targets when applying augmented reality (AR). Generally, the object recognition technology according to this invention can be applied to any purpose of identifying objects.

[0217] This technology can also be configured as follows.

[0218] (1) An information processing device, comprising

[0219] Controller, its

[0220] Identifying objects based on event information detected by event-based sensors, and

[0221] The recognition results are sent to a sensor device, which includes a sensor unit capable of acquiring information about the object.

[0222] (2) The information processing apparatus according to (1), wherein

[0223] controller

[0224] Identify objects.

[0225] Specify the location of the region of interest (ROI) corresponding to the object, and

[0226] The ROI location is sent as the identification result to the sensor device.

[0227] (3) The information processing apparatus according to (2), wherein

[0228] Sensor device

[0229] Extract the ROI information corresponding to the ROI location from the information acquired by the sensor unit.

[0230] The ROI information is sent to the information processing device.

[0231] (4) The information processing apparatus according to (3), wherein

[0232] The controller identifies objects based on ROI information obtained from sensor devices.

[0233] (5) The information processing apparatus according to (4), wherein

[0234] The controller designs an autonomous driving plan based on information about objects identified using ROI-based information.

[0235] (6) The information processing apparatus according to (5), wherein

[0236] The controller designs an autonomous driving plan based on information about objects identified through event-based information.

[0237] (7) The information processing apparatus according to (6), wherein

[0238] The controller determines whether an autonomous driving plan can be designed based solely on information about objects identified through event-based information.

[0239] (8) The information processing apparatus according to (7), wherein

[0240] In the absence of an autonomous driving plan

[0241] controller

[0242] Obtain ROI information, and

[0243] Autonomous driving plans are designed based on information about objects identified using ROI-based information.

[0244] (9) The information processing apparatus according to (7) or (9), wherein

[0245] Given the ability to design autonomous driving plans

[0246] The controller designs an autonomous driving plan based on information about objects identified from event-based information, without acquiring ROI information.

[0247] (10) The information processing apparatus according to any one of (3) to (9), wherein

[0248] The sensor unit includes an image sensor capable of acquiring images of an object, and

[0249] ROI information is the ROI image.

[0250] (11) The information processing apparatus according to any one of (5) to (10), wherein

[0251] The sensor unit includes a supplementary sensor that can acquire supplementary information about objects that the controller cannot recognize using event information.

[0252] (12) The information processing apparatus according to (11), wherein

[0253] The controller obtains supplementary information from the sensor device, and

[0254] Based on supplementary information, the controller identifies objects that cannot be identified using event information.

[0255] (13) The information processing apparatus according to (12), wherein

[0256] The controller designs an autonomous driving plan based on information about the objects identified using supplementary information.

[0257] (14) The information processing apparatus according to (13), wherein

[0258] The controller acquires information about the movement of the moving body that becomes the object of the autonomous driving plan, and

[0259] Based on information about movement, the controller changes the cycle of object recognition based on supplementary information.

[0260] (15) The information processing apparatus according to (14), wherein

[0261] The controller shortens the cycle as the moving object moves slower.

[0262] (16) The information processing apparatus according to any one of (3) to (15), wherein

[0263] The sensor device modifies the cutout position of the ROI information based on the offset of the object in the ROI information.

[0264] (17) An information processing system, comprising:

[0265] Information processing device, including

[0266] controller

[0267] Identifying objects based on event information detected by event-based sensors, and

[0268] The recognition result is sent to a sensor device, which includes a sensor unit capable of acquiring information about the object; and

[0269] Sensor device.

[0270] (18) An information processing method, comprising:

[0271] Identifying objects based on event information detected by event-based sensors; and

[0272] The recognition results are sent to a sensor device, which includes a sensor unit capable of acquiring information about the object.

[0273] (19) A program that causes a computer to perform the following processes:

[0274] Identifying objects based on event information detected by event-based sensors; and

[0275] The recognition results are sent to a sensor device, which includes a sensor unit capable of acquiring information about the object.

[0276] List of reference numerals

[0277] 10 DVS

[0278] 20 Automatic driving actuators

[0279] 30 Automatic driving control device

[0280] 31 Controller of the automatic driving control device

[0281] 40 Sensor Devices

[0282] 41 Controller of the sensor device

[0283] 42 sensor units

[0284] 43 Image Sensor

[0285] 44 LiDAR

[0286] 45 mm wave radar

[0287] 46 Ultrasonic Sensors

[0288] 100 Automatic Automated Driving Control System

Claims

1. An information processing device, comprising: Controller, the controller Objects are identified based on event information detected by event-based sensors, where... The event-based sensor includes a dynamic vision sensor that detects changes in the brightness of incident light for each pixel, and The identification result is sent to a sensor device, which includes a sensor unit capable of acquiring information about the object. Wherein, the controller determines whether an autonomous driving plan for controlling the moving body can be designed solely based on information about the object identified based on the event information; and wherein, Given the ability to design the aforementioned autonomous driving plan The controller designs the autonomous driving plan based on information about objects identified based on the event information, without acquiring information about the region of interest (ROI).

2. The information processing apparatus according to claim 1, wherein... The controller Identify the object. Specify the region of interest (ROI) location corresponding to the object, and The ROI location is sent to the sensor device as the identification result.

3. The information processing apparatus according to claim 2, wherein... The sensor device Extract the ROI information corresponding to the ROI location from the information acquired by the sensor unit. The ROI information is sent to the information processing device.

4. The information processing apparatus according to claim 3, wherein The controller identifies the object based on ROI information obtained from the sensor device.

5. The information processing apparatus according to claim 4, wherein The controller designs the autonomous driving plan based on information about the objects identified based on the ROI information.

6. The information processing apparatus according to claim 5, wherein The controller designs an autonomous driving plan based on information about objects identified based on the event information.

7. The information processing apparatus according to claim 6, wherein In the absence of the aforementioned autonomous driving plan The controller Obtain the ROI information, and An autonomous driving plan is designed based on information about the objects identified based on the ROI information.

8. The information processing apparatus according to claim 3, wherein The sensor unit includes an image sensor capable of acquiring an image of the object, and The ROI information is the ROI image.

9. The information processing apparatus according to claim 5, wherein The sensor unit includes a supplementary sensor capable of acquiring supplementary information, which is information about objects that the controller cannot recognize using the event information.

10. The information processing apparatus according to claim 9, wherein The controller obtains the supplementary information from the sensor device, and Based on the supplementary information, the controller identifies objects that cannot be identified using the event information.

11. The information processing apparatus according to claim 10, wherein The controller designs an autonomous driving plan based on information about the objects identified based on the supplementary information.

12. The information processing apparatus according to claim 11, wherein The controller acquires information about the movement of the mobile body that becomes the object of the autonomous driving plan, and Based on information about the movement, the controller changes the period for recognizing objects based on the supplementary information.

13. The information processing apparatus according to claim 12, wherein The controller shortens the cycle as the moving body moves slower.

14. The information processing apparatus according to claim 3, wherein The sensor device modifies the cutout position of the ROI information based on the offset of the object in the ROI information.

15. An information processing system, comprising: Information processing device, including Controller, the controller Objects are identified based on event information detected by an event-based sensor, wherein the event-based sensor includes a dynamic vision sensor that detects changes in the brightness of incident light for each pixel, and The identification result is sent to a sensor device, the sensor device including a sensor unit capable of acquiring information about the object; and The sensor device, Wherein, the controller determines whether an autonomous driving plan for controlling the moving body can be designed solely based on information about the object identified based on the event information; and wherein, Given the ability to design the aforementioned autonomous driving plan The controller designs the autonomous driving plan based on information about objects identified based on the event information, without acquiring information about the region of interest (ROI).

16. An information processing method, comprising: Objects are identified based on event information detected by an event-based sensor, wherein the event-based sensor includes a dynamic vision sensor that detects changes in the brightness of incident light for each pixel; and The identification result is sent to a sensor device, which includes a sensor unit capable of acquiring information about the object. Among these, determining whether an autonomous driving plan for controlling a moving body can be designed solely based on information about an object identified based on the event information; and wherein... Given the ability to design the aforementioned autonomous driving plan The autonomous driving plan is designed based on information about objects identified based on the event information, without acquiring Region of Interest (ROI) information.

17. A program product that causes a computer to perform processing, said processing comprising: Objects are identified based on event information detected by an event-based sensor, wherein the event-based sensor includes a dynamic vision sensor that detects changes in the brightness of incident light for each pixel; and The identification result is sent to a sensor device, which includes a sensor unit capable of acquiring information about the object. Among these, determining whether an autonomous driving plan for controlling a moving body can be designed solely based on information about an object identified based on the event information; and wherein... Given the ability to design the aforementioned autonomous driving plan The autonomous driving plan is designed based on information about objects identified based on the event information, without acquiring Region of Interest (ROI) information.

Images