A device that recognizes objects

Abstract

To provide a device capable of lowering the computational load for recognizing an object.SOLUTION: An object recognizing device includes: an image sensor which acquires an image in the travelling direction of a vehicle; a controller which cuts out, from an image captured by the image sensor, an object region containing a first region being lower than the image center and a second region being upper than image center, adjacent to the first region, and having an area smaller than the first region, and sets them as a target image; and a model which is configured to take as input the positions of the target image and the object region and output a recognition result regarding road lane lines and at least one of a traffic signal and a sign-board.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an apparatus for recognizing an object.

Background Art

[0002] Patent Document 1 discloses an apparatus for recognizing an object based on an image captured by a camera. This apparatus uses a CNN (Convolutional Neural Network) to recognize an object in the image. This apparatus generates a flipped image obtained by flipping the camera image vertically, and inputs the flipped image into a CNN model. This apparatus preferentially recognizes an object appearing in the lower half region of the screen by using the flipped image in an environment where the upper limit of the number of recognizable objects is determined.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] During driving of a vehicle, objects that a driver should pay attention to include signals or signs appearing in the upper half region of the screen. Therefore, it is required to recognize not only objects appearing in the lower half region of the screen but also signals or signs appearing in the upper half region of the screen. However, when the entire screen is to be processed, there is a risk of an increase in the calculation load. The present disclosure provides an apparatus capable of reducing the load of calculations for recognizing an object.

Means for Solving the Problems

[0005] An object recognition device according to one embodiment of the present disclosure comprises an image sensor, a controller, and a model. The image sensor acquires an image in the direction of travel of a vehicle. The controller extracts a target region from the image acquired by the image sensor, which includes a first region below the center of the image and a second region above the center of the image that is adjacent to the first region and has a smaller area than the first region, and uses this as a target image. The model is configured to take the target image and the position of the target region as input and to output recognition results regarding road lane lines and at least one of traffic lights and signs. [Effects of the Invention]

[0006] This disclosure provides a technology that can reduce the computational load required for object recognition. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 is a block diagram showing an example of the configuration of a vehicle equipped with a recognition device according to one embodiment. [Figure 2] Figure 2 shows an example of a target area in a camera image. [Figure 3] Figure 3 shows another example of a target area in a camera image. [Figure 4] Figure 4 shows another example of a target region in a camera image. [Figure 5] Figure 5 is a flowchart showing an example of the operation of the recognition device. [Modes for carrying out the invention]

[0008] Embodiments of the present invention will be described below with reference to the drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant descriptions are omitted.

[0009] [Vehicle configuration] Figure 1 is a block diagram showing an example of the configuration of a vehicle equipped with a device according to one embodiment. As shown in Figure 1, the recognition device 1 (an example of a device) is mounted on a vehicle 2, for example. Vehicle 2 is an autonomous vehicle, for example. Vehicle 2 is not limited to an autonomous vehicle, and may be a vehicle that assists with driving operations. Assistance with driving operations includes cases where only information is provided.

[0010] The recognition device 1 includes an image sensor 10, a yaw rate sensor 11, a turn signal sensor 12, a processing ECU 13 (an example of a controller), and a recognition model 14 (an example of a model). The ECU (Electronic Control Unit) is an electronic control unit having a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), CAN (Controller Area Network) communication circuit, etc. The recognition device 1 does not necessarily have to include the yaw rate sensor 11 and the turn signal sensor 12.

[0011] Image sensor 10 is, for example, a camera. Image sensor 10 acquires an image of the direction of travel of vehicle 2. Hereinafter, the image acquired by image sensor 10 will be referred to as a camera image. Yaw rate sensor 11 detects the yaw rate of vehicle 2. Turn signal sensor 12 detects the direction of the turn signal that indicates the direction of travel of vehicle 2.

[0012] The processing ECU 13 extracts a target region from the camera image to generate a target image. The target image is the image that will be processed by the recognition model 14. The target region includes a first region and a second region. The first region is the area below the center of the camera image. The second region is the area above the center of the camera image. The second region is adjacent to the first region and has a smaller area than the first region. In other words, the target region is convex, L-shaped, or an inverted L-shape.

[0013] The processing ECU 13, as an example, divides the camera image into blocks and specifies a region on the image based on the identifier assigned to each block. Figure 2 shows an example of a target region in a camera image. As shown in Figure 2, the camera image G is divided into blocks. As an example, the camera image is divided into an 8x8 matrix. Each block is assigned an identifier. For example, the identifier of the leftmost block in the top row is B1. The identifier of the block adjacent to the right of block B1 is B2. In this way, each block is assigned an identifier Bn. The processing ECU 13 can specify a region at any position using the block identifier Bn.

[0014] The processing ECU 13 determines the position of the first region R1, which is the area below the center of the camera image G, and the position of the second region R2, which is the area above the center of the camera image G. The first region R1 is located below the center of the camera image G. Generally, road lane lines L1 and L2 are captured below the center of the camera image G. In other words, the first region R1 is set with the expectation that road lane lines L1 and L2 will be captured. The first region R1 may also be set to include a preceding vehicle V1 and an adjacent vehicle V2.

[0015] The second region R2 is positioned above the center of the camera image G. Generally, the sign S and the traffic signals are captured above the center of the camera image G. In other words, the second region R2 is set up with the expectation that the sign S and the signals will be captured. Also, generally, the sign S and the sign are smaller than the road lane lines L1 and L2. For this reason, the area of ​​the second region R2 can be smaller than that of the first region R1. The second region R2 is adjacent to the first region R1. The target region R, which is formed by connecting the first region R1 and the second region R2, can be treated as a single image.

[0016] The machining ECU 13 determines the positions of the first region R1 and the second region R2 based on, for example, a predetermined block identifier as a default.

[0017] The processing ECU 13 may change the position of the first region R1 and the position of the second region R2 depending on the type or performance of the image sensor 10. The type or performance of the image sensor 10 refers to camera performance such as wide-angle or telephoto. The type or performance of the image sensor 10 can be distinguished by an identifier. For example, the processing ECU 13 has a table that stores the identifier of the image sensor 10 in association with the position of the first region R1 and the position of the second region R2. The table is stored in the memory unit of the ECU. The processing ECU 13 obtains the identifier of the image sensor 10, refers to the table based on the obtained identifier, and determines the position of the first region R1 and the position of the second region R2. This makes it possible to set the optimal target region R for each image sensor 10.

[0018] The processing ECU 13 may determine the positions of the first region R1 and the second region R2 based on the detection result of the yaw rate sensor 11. For example, the processing ECU 13 determines the turning direction of the vehicle 2 based on the detection result of the yaw rate sensor 11. The processing ECU 13 moves the second region R2 in the turning direction of the vehicle 2. Figure 3 shows another example of the target region in a camera image. Assume that the scene in Figure 3 is a scene in which the vehicle 2 is driving while turning to the left. In this case, as shown in Figure 3, the position of the second region R2 has moved one block to the left compared to the target region R shown in Figure 2. As a result, the left part of the sign S fits into the second region R2. The processing ECU 13 may also change the position of the first region R1. The processing ECU 13 determines the amount of movement of the first region R1 and the amount of movement of the second region R2 according to the magnitude of the yaw rate of the vehicle 2, and the amount of movement of the second region R2 may be set to be greater than the amount of movement of the first region R1. As a result, the processing ECU 13 can include lane lines L1 and L2 in the first region R1, while more accurately capturing the sign S and signals, and including the sign S and signals in the second region R2. In this way, the optimal target region R can be set according to the movement of the vehicle 2.

[0019] The processing ECU 13 may determine the positions of the first region R1 and the second region R2 based on the direction indicated by the direction indicator detected by the direction indicator sensor 12. For example, the processing ECU 13 moves the second region R2 in the same direction as the direction of the direction indicator. FIG. 4 is a diagram showing another example of the target region in the camera image. Assume that the scene in FIG. 4 is a scene where the direction indicator is flashing so that the vehicle 2 moves into the right lane. In this case, as shown in FIG. 4, compared with the target region R shown in FIG. 2, the positions of the first region R1 and the second region R2 have moved one block to the right. As a result, the adjacent lane and the adjacent vehicle V2 come to be within the first region R1, and the right part of the signboard S comes to be within the second region R2. In this way, an optimal target region R can be set according to the movement of the vehicle 2.

[0020] The recognition model 14 is a learned model. The recognition model 14 is learned by machine learning or the like based on the teacher data. The recognition model 14 may be a CNN model. The recognition model 14 inputs the target image generated by the processing ECU 13 and the position of the target region. The position of the target region is the identifier Bn of the block described above. When the recognition model 14 inputs the target image and the position of the target region, it is learned to output a recognition result regarding at least one of the road lane lines L1, L2, the signboard S, and the signal. Such learning is executed by a server or the like, and the learned recognition model 14 is provided to the vehicle 2 via communication or the like.

[0021] The recognition model 14 outputs, as recognition results, the positions of the road lane lines L1, L2, the information of the signboard S, the signal information of the traffic signal, and the like. The recognition result is output to the user interface 3. The user interface 3 is, for example, a display, a speaker, or the like. The user interface 3 notifies the driver of the recognition result. Alternatively, the recognition result is output to another ECU 4. The other ECU 4 is, for example, an automatic driving ECU or the like.

[0022] The processing ECU 13 may determine the positions of the first region R1 and the second region R2 based on the positions of lane lines L1 and L2 recognized by the recognition model 14. For example, suppose the scene in Figure 4 is one in which vehicle 2 moves to the right lane. In this case, lane lines L1 and L2 move relatively to the left. Therefore, the processing ECU 13 moves the positions of the first region R1 and the second region R2 in the opposite direction to the direction in which the recognized lane lines L1 and L2 move. For example, as shown in Figure 4, the positions of the first region R1 and the second region R2 have moved one block to the right compared to the target region R shown in Figure 2. As a result, the adjacent lane and adjacent vehicle V2 will be contained within the first region R1, and the right portion of signboard S will be contained within the second region R2. In this way, the optimal target region R can be set to match the movement of vehicle 2.

[0023] [Operation of the recognition device] Figure 5 is a flowchart illustrating an example of the operation of the recognition device. The flowchart shown in Figure 5 is initiated when the recognition device 1 receives a start command operation.

[0024] As shown in Figure 5, first, the image sensor 10 of the recognition device 1 acquires a camera image G in step S10. Subsequently, the processing ECU 13 determines the target area R in step S12.

[0025] In step S14, the processing ECU 13 extracts the target region R from the camera image G to obtain the target image. In step S16, the recognition model 14 receives the target image extracted in step S14 as input. In step S18, the recognition model 14 outputs recognition results regarding the road lane lines L1 and L2, and at least one of the sign S and the traffic light. When step S18 is completed, the flowchart shown in Figure 5 is finished.

[0026] [Summary of Embodiments] According to the recognition device 1, a target region R is extracted from the camera image G acquired by the image sensor 10, including a first region R1 below the center of the camera image G and a second region R2 above the center of the camera image G that is adjacent to the first region R1 and has a smaller area than the first region R1. This target region R is then used as the target image. By making the target image a convex shape or similar, the area to be calculated can be reduced while capturing the lane lines L1, L2 and at least one of the traffic lights and signs S within the image. Therefore, the recognition model 14 can reduce the computational load for recognizing the lane lines L1, L2 and at least one of the traffic lights and signs S compared to when the entire image is used as the target.

[0027] Although exemplary embodiments have been described above, the invention is not limited to the exemplary embodiments described above, and various omissions, substitutions, and modifications may be made.

[0028] This disclosure includes the following aspects: [Clause 1] An image sensor that acquires an image of the direction of travel of the vehicle, A controller extracts a target region from an image acquired by the image sensor, which includes a first region below the center of the image and a second region above the center of the image that is adjacent to the first region and has a smaller area than the first region, and uses this as the target image. A model configured to take the aforementioned target image and the location of the aforementioned target area as input and output recognition results relating to at least one of road lane lines, traffic lights, and signs, A device that recognizes objects, equipped with the necessary features. [Clause 2] The apparatus according to Clause 1, wherein the controller has a table that stores the identifier of the image sensor in association with the position of the first region and the position of the second region, and determines the position of the first region and the position of the second region based on the identifier of the image sensor and the table. [Clause 3] The vehicle further comprises a yaw rate sensor for detecting the yaw rate of the vehicle, The device according to Clause 1 or 2, wherein the controller determines the position of the first region and the position of the second region based on the yaw rate of the vehicle. [Clause 4] The vehicle further comprises a sensor that detects the direction of a turn signal indicating the direction of travel of the vehicle, The device according to any one of the clauses 1 to 3, wherein the controller determines the position of the first region and the position of the second region based on the direction indicated by the direction indicator. [Clause 5] The apparatus according to any one of claims 1 to 4, wherein the controller determines the position of the first region and the position of the second region based on the position of the lane lines recognized by the model. [Explanation of Symbols]

[0029] 1... Recognition device (an example of a device), 2... Vehicle, 10... Image sensor, 11... Yaw rate sensor, 12... Turn signal sensor, 13... Processed ECU (an example of a controller), 14... Recognition model (an example of a model).

Claims

1. An image sensor that acquires an image of the direction of travel of a vehicle, A controller extracts a target region from an image acquired by the image sensor, which includes a first region below the center of the image and a second region above the center of the image that is adjacent to the first region and has a smaller area than the first region, and uses this as the target image. A model configured to take the aforementioned target image and the location of the aforementioned target area as input and output recognition results relating to at least one of road lane lines, traffic lights, and signs, Equipped with, The controller has a table that stores the identifier of the image sensor in association with the position of the first region and the position of the second region, and determines the position of the first region and the position of the second region based on the identifier of the image sensor and the table. A device that recognizes objects.

2. An image sensor that acquires an image of the direction of travel of a vehicle, A controller extracts a target region from an image acquired by the image sensor, which includes a first region below the center of the image and a second region above the center of the image that is adjacent to the first region and has a smaller area than the first region, and uses this as the target image. A model configured to take the aforementioned target image and the location of the aforementioned target area as input and output recognition results relating to at least one of road lane lines, traffic lights, and signs, A yaw rate sensor for detecting the yaw rate of the vehicle, Equipped with, The controller is an object recognition device that determines the position of the first region and the position of the second region based on the yaw rate of the vehicle.

3. An image sensor that acquires an image of the direction of travel of a vehicle, A controller extracts a target region from an image acquired by the image sensor, which includes a first region below the center of the image and a second region above the center of the image that is adjacent to the first region and has a smaller area than the first region, and uses this as the target image. A model configured to take the aforementioned target image and the location of the aforementioned target area as input and output recognition results relating to at least one of road lane lines, traffic lights, and signs, The vehicle comprises a sensor that detects the direction of a turn signal that indicates the direction of travel of the vehicle, The controller is an object recognition device that determines the position of the first region and the position of the second region based on the orientation indicated by the direction indicator.

4. An image sensor that acquires an image of the direction of travel of a vehicle, A controller extracts a target region from an image acquired by the image sensor, which includes a first region below the center of the image and a second region above the center of the image that is adjacent to the first region and has a smaller area than the first region, and uses this as the target image. A model configured to take the aforementioned target image and the location of the aforementioned target area as input and output recognition results relating to at least one of road lane lines, traffic lights, and signs, Equipped with, The controller is an object recognition device that determines the position of the first region and the position of the second region based on the position of the lane lines recognized by the model.

Images