Information processing device

Abstract

To accurately estimate a distance to a vehicle in an image.SOLUTION: An information processing device comprises a control unit which executes the steps of: acquiring a type of a vehicle appearing in a photographed image of a camera; specifying a number plate of the vehicle from the photographed image; acquiring the vertical or horizontal pixel number of the number plate of the vehicle in the photographed image; and estimating the distance to the vehicle from the camera on the basis of the vertical or horizontal dimension of the number plate according to the type of the vehicle, and the vertical or horizontal pixel number of the number plate.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an information processing device. [Background technology]

[0002] A distance estimation device has been disclosed that calculates a distribution of estimated distances to an object from each image in a time series based on the size of the object area, the degree of variation in size of the object type in real space, and the confidence that the object is that type of object, and estimates the distance to the object at the time each image is captured by a prediction process using the distribution of estimated distances for each image as an observation error model of the distance to the object (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-026683 Summary of the Invention [Problem to be solved by the invention]

[0004] One aspect of the disclosure has an object to provide an information processing device capable of accurately estimating a distance to a vehicle in an image. [Means for solving the problem]

[0005] One aspect of the present disclosure is Obtaining the type of vehicle appearing in the image captured by the camera; Identifying a license plate number of the vehicle from the captured image; Obtaining the number of vertical or horizontal pixels of the license plate of the vehicle in the captured image; Estimating a distance from the camera to the vehicle based on a vertical or horizontal dimension of a license plate according to the type of the vehicle and a vertical or horizontal pixel number of the license plate; A control unit that executes It is an information processing device. Effect of the Invention

[0006] According to one aspect of the present disclosure, the distance to a vehicle in an image can be estimated with high accuracy. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration of a distance estimation system according to the first embodiment. [Diagram 2] FIG. 2 shows an example of the hardware configuration of an ECU. [Diagram 3] FIG. 3 is a diagram illustrating an example of a functional configuration of a vehicle. [Figure 4] FIG. 4 is a diagram illustrating an example of distance estimation. [Diagram 5] FIG. 5 is an example of a flowchart of the distance estimation process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] One aspect of the present disclosure is to obtain the type of vehicle appearing in an image captured by a camera, identify the license plate of the vehicle from the captured image, obtain the number of pixels in the vertical or horizontal direction of the license plate of the vehicle in the captured image, and The information processing device is equipped with a control unit that executes the following: estimating the distance from the camera to the vehicle based on the vertical or horizontal dimensions of the license plate according to the type and the number of vertical or horizontal pixels of the acquired license plate.

[0009] The camera is, for example, a camera mounted on a vehicle and installed facing the outside of the vehicle. Vehicles equipped with a camera include, for example, automobiles, trains, motorcycles, and bicycles. When the camera is mounted on a vehicle, the information processing device is, for example, an in-vehicle device, an ECU (Electronic Control Unit), a smart device, or the like that is connected to the camera by wire or wirelessly. The information processing device may be a computer such as a smartphone, a tablet terminal, or a wearable terminal, but is not limited thereto, and may be a server capable of communicating with a vehicle equipped with a camera.

[0010] Furthermore, the camera is not limited to one mounted on a vehicle, and may be, for example, a fixed camera installed at a predetermined position, etc. In the case where the camera is a fixed camera installed at a predetermined position, the information processing device may be, for example, a user terminal such as an OC (Persona Computer) that is connected to the camera by wire or wirelessly, and a server.

[0011] Furthermore, the vehicles appearing in the image are, for example, vehicles equipped with license plates of automobiles, motorcycles, mopeds, etc. In Japan, the vertical and horizontal dimensions of license plates are regulated for large vehicles, ordinary automobiles, motorcycles, and mopeds, respectively. Therefore, the types of vehicles acquired by the information processing device are, for example, some or all of large vehicles, ordinary automobiles, motorcycles, and mopeds.

[0012] In one aspect of the present disclosure, the dimensions of a vehicle's license plate are known values, so the distance to the vehicle can be accurately estimated by using the license plate corresponding to the type of vehicle in the camera image as a reference.

[0013] Another aspect of the present disclosure can be specified as a method for causing a computer to execute the processing of the information processing device, or as a program for causing a computer to execute the processing of the information processing device, and a non-transitory computer-readable recording medium for recording the program.

[0014] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The configurations of the following embodiments are merely examples, and the present disclosure is not limited to the configurations of the embodiments.

[0015] First Embodiment 1 is a diagram showing an example of a system configuration of a distance estimation system 100 according to the first embodiment. The distance estimation system 100 according to the first embodiment is a system that estimates the distance to a vehicle 50 from an image captured by a camera 2 mounted on the vehicle 10, using the dimensions of the license plate of the vehicle 50 that appears in the captured image.

[0016] In the first embodiment, the distance estimation system 100 includes an on-board camera 2 mounted on a vehicle 10, and an ECU (Electronic Control Unit) 1 that controls the camera 2. The U1 and the camera 2 are connected to each other via, for example, a CAN (Controller Area Network) or are connected via in-vehicle Ethernet (registered trademark) or the like.

[0017] The camera 2 is, for example, a camera installed on the upper part of the windshield or on the back side of the rearview mirror of the vehicle 10 so as to capture a predetermined range in front of the vehicle 10. The camera 2 may be provided exclusively for the distance estimation system 100, or may be a camera used in another system such as a drive recorder. The target cameras of the distance estimation system 100 are not limited to those having an imaging range in front of the vehicle 10. For example, the target cameras of the distance estimation system 100 may include a camera that is installed on the upper part of the rear window of the vehicle 10 and has an imaging range in the rear of the vehicle 10.

[0018] The ECU 1 is, for example, a control device that controls the camera 2. In the first embodiment, the ECU 1 executes image recognition processing on image data captured by the camera 2, acquires the type of vehicle included in the image, and estimates the distance to the vehicle in the image using the dimensions of the license plate according to the type of vehicle. In Japan, the dimensions of the license plate are regulated according to the type of vehicle. In the first embodiment, a four-wheeled vehicle is assumed as the target vehicle to be detected from the image captured by the camera 2. In this case, there are two types of vehicles: large vehicles and regular vehicles. A large vehicle is a four-wheeled vehicle with a total vehicle weight of 8 tons or more and a maximum load capacity of 5 tons or more, or a four-wheeled vehicle with a capacity of 30 or more. A large vehicle is, for example, a truck, a large bus, or a large special vehicle (such as a crane truck) that meets these conditions. A regular vehicle is a four-wheeled vehicle that is not classified as a large vehicle, and a light vehicle. Note that a vehicle to be detected from the image captured by the camera 2 is not limited to a four-wheeled vehicle, and may include a motorcycle and a motorized bicycle. In this case, the types of vehicles may be four types: large vehicles, regular cars, motorcycles, and motorbikes.

[0019] In the first embodiment, the dimensions of the license plate are regulated for each type of vehicle, and the dimensions of the license plate appearing in the captured image are used as reference dimensions to estimate the distance from the camera 2 to the license plate. The distance from the camera 2 to the license plate is set as the distance from the camera 2 to the vehicle having the license plate. This can improve the accuracy of estimating the distance from the camera 2 to the vehicle 10.

[0020] 2 is an example of a hardware configuration of the ECU 1. The ECU 1 includes, as its hardware configuration, a CPU 101, a memory 102, an auxiliary storage device 103, and an interface 104. The memory 102 and the auxiliary storage device 103 are computer-readable recording media. The ECU 1 is an example of an "information processing device."

[0021] The auxiliary storage device 103 stores various programs and data used by the CPU 101 when executing each program. The auxiliary storage device 103 is, for example, an EPROM (Erasable Programmable ROM) or a flash memory. The programs to be executed include, for example, an image recognition program and a distance estimation program. The image recognition program is a program that detects a target object from an image. The distance estimation program is a program that estimates the distance from the camera 2 to the target object from the result of the image recognition process.

[0022] The memory 102 is a storage device that provides the CPU 101 with a storage area for loading programs stored in the auxiliary storage device 103 and a working area, and is used as a buffer. The memory 102 may be, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), or the like. This includes semiconductor memories such as IEEE 802.11b / g and IEEE 802.11b / g.

[0023] The CPU 101 executes various processes by loading the OS and various other programs stored in the auxiliary storage device 103 into the memory 102 and executing them. The number of CPUs 101 is not limited to one, and may be multiple. The CPU 101 is an example of a "control unit."

[0024] The interface 104 is, for example, a circuit having a port used for connection to a CAN, an in-vehicle Ethernet (registered trademark), or other network. Note that the hardware configuration of the ECU 1 is not limited to that shown in FIG.

[0025] The camera 2 is, for example, a monocular camera. The camera 2 includes an image sensor and a lens. The image sensor of the camera 2 may be, for example, a CCD (Charge Coupled Device), and CMOS (Complementary Metal Oxide Semiconductor). The camera 2 is connected to the CPU 101 through a network such as a CAN or an in-vehicle Ethernet, but is not limited to this. The camera 2 may be connected to the CPU 101 directly by a cable or the like without going through a network.

[0026] The DCM 3 is a communication device having a communication function between the vehicle 10 and an external device. The DCM 3 supports, for example, LTE (Long Term Evolution), 5G (5th Generation), and 6G. The vehicle is connected to a wireless network by a wireless communication method such as LTE (6th Generation), Wi-Fi, or DSCR, and is connected to the Internet through the wireless network. Note that the distance estimation process is not limited to being executed by the ECU 1, and may be executed by the DCM 3, for example.

[0027] 3 is a diagram showing an example of the functional configuration of the vehicle 10. The vehicle 10 includes, as its functional configuration, a control unit 11, an object detection unit 12, a license plate detection unit 13, an image acquisition unit 14, a dimension information storage unit 15, and a camera information storage unit 16. The processing of these functional configurations is achieved by the CPU 101 executing a distance estimation program held in the auxiliary storage device 103.

[0028] The image acquisition unit 14 acquires image data within an imaging range in front of the vehicle 10, which is captured by the camera 2 at a predetermined cycle, from the camera 2. The imaging rate of the camera 2 is set, for example, in the range of 15 fps to 60 fps according to the performance of the camera 2 or according to a setting by the owner or manager of the vehicle 10. The image acquisition unit 14 outputs the acquired image data to the control unit 11.

[0029] The dimension information storage unit 15 and the camera information storage unit 16 are created, for example, in a memory area of ​​the auxiliary storage device 103. The dimension information storage unit 15 holds information on the dimensions of the license plate. The dimension information storage unit 15 holds information on the dimensions of the license plate for each type of vehicle. Specifically, the dimensions of a license plate for a large vehicle are 22 cm in height and 44 cm in width. The dimensions of a license plate for a standard vehicle are 16.5 cm in height and 33 cm in width. The camera information storage unit 16 holds information about the camera 2. The information about the camera 2 held in the camera information storage unit 16 includes, for example, the focal length, the number of pixels, and the pixel size.

[0030] The control unit 11 estimates the distance to other vehicles included in the image captured by the camera 2. The control unit 11 receives input of image data captured by the camera 2 from the image acquisition unit 14. The control unit 11 instructs the object detection unit 12 to start object detection processing for the image captured by the camera 2. The control unit 11 acquires the detection result of the object included in the image captured by the camera 2 from the object detection unit 12. As the detection result of the object in the image by the object detection unit 12, the type of object included in the image and position information of the object in the image are obtained. The position information in the image is, for example, coordinate information of the pixel in the image that corresponds to the object. If a vehicle is included in the image, the type of vehicle is obtained as the type of detected object.

[0031] When the object detection result in the image captured by camera 2 indicates that a vehicle is included in the image captured by camera 2, control unit 11 acquires from dimensional information storage unit 15 dimensional information of the license plate corresponding to the type of vehicle detected. Control unit 11 instructs license plate detection unit 13 to start a process of detecting the license plate from the image captured by camera 2, and acquires the detection result of the license plate from license plate detection unit 13. Position information of the license plate within the image is acquired as the detection result of the license plate from license plate detection unit 13. Control From the result of the detection of the license plate by the license plate detection unit 13, the control unit 11 acquires, for example, the number of vertical pixels of the license plate in the image captured by the camera 2. The control unit 11 estimates the distance from the camera 2 to the license plate using the number of vertical pixels of the license plate in the image captured by the camera 2 and the focal length of the camera 2. The method of distance estimation will be described in detail later.

[0032] The object detection unit 12 receives an instruction from the control unit 11 and executes a process of detecting an object from an image captured by the camera 2. Specifically, the object detection unit 12 is an AI (artificial intelligence) and machine learning model that detects an object from an image and outputs the type of the object and position information in the image where the object appears. Machine learning models used in the object detection unit 12 include, for example, R-CNN (Region Based Convolutional Neural Networks), YOLO (You Only Look Once), This model is constructed using algorithms such as SSD (Single Shot MultiBox Detector) and DETR (End-to-End Object Detection with Transformers).

[0033] The object detection unit 12 receives an image captured by the camera 2, executes a predetermined process on the input image, and outputs the type of object and position information of the object within the image for one or more objects appearing in the input image. For example, if a vehicle 50 is included in the image captured by the camera 2, the object detection unit 12 outputs the vehicle type of the vehicle 50. The object detection unit 12 outputs the result of the object detection process for the image captured by the camera 2 to the control unit 11.

[0034] The license plate detection unit 13 receives an instruction from the control unit 11 and executes a detection process of the license plate from the captured image of the camera 2. The license plate detection unit 13 is a machine learning model or AI (artificial intelligence) that is trained to detect the license plate from the image. The machine learning model used in the license plate detection unit 13 is, for example, a model constructed by an algorithm such as R-CNN, YOLO, SSD, and DETR. The license plate detection unit 13 receives a part or all of the captured image of the camera 2 as input, executes a predetermined process on the input image, and outputs position information of the license plate in the input image in the image. The license plate detection unit 13 outputs the detection result of the license plate in the captured image of the camera 2 to the control unit 11. Note that the object detection unit 12 and the license plate detection unit 13 are not limited to the machine learning model or AI, and may be a predetermined image recognition process.

[0035] 4 is a diagram showing an example of distance estimation. In camera 2, there is a distance from the center point of the lens to the image sensor (film surface). The distance from the center point of the lens to the image sensor is called the focal length. The distance from the center point of the lens to the subject is hereinafter called the subject distance.

[0036] For example, as shown in Fig. 4, the figure formed by the center point of the lens and the height of the license plate as the subject is symmetrical to the figure formed by the center point of the lens and the height of the license plate in the image. Therefore, the ratio of the license plate height to the subject distance in the real world is equal to the ratio of the license plate height to the focal length in the image.

[0037] The height (vertical dimension) of the license plate and the focal length are known values. The height of the license plate in the image is obtained by multiplying the vertical pixel value of the license plate by the pixel size of the camera. The pixel size of the camera is the dimension per pixel. The pixel size of the camera is a parameter specific to each camera model and is a known value. Therefore, the subject distance can be calculated by multiplying the height of the license plate in the real world by the focal length and dividing the value by the height of the license plate in the image.

[0038] The control unit 11 receives the vertical dimension of the license plate according to the type of the vehicle from the dimension information storage unit 15. The focal length and pixel size are obtained from the camera information storage unit 16, and the height of the license plate in the image is obtained based on the pixel size of the camera 2 from the result of the license plate detection in the image captured by the camera 2, and the distance to the vehicle 50 is estimated.

[0039] <Processing flow> Fig. 5 is an example of a flowchart of the distance estimation process. The process shown in Fig. 5 is executed repeatedly at a predetermined cycle, for example. The cycle at which the distance estimation process is executed may be the same as the image capture cycle of the camera 2, or may be a cycle designated by the owner or manufacturer of the vehicle 10. The process shown in Fig. 5 is executed by the CPU 101, but for convenience, the process will be described mainly with respect to the functional components.

[0040] In OP101, the control unit 11 receives an input of image data captured by the camera 2 from the image acquisition unit 14. In OP102, the control unit 11 instructs the object detection unit 12 to start an object detection process for the image captured by the camera 2, and acquires, from the object detection unit 12, the type of object detected from the image captured by the camera 2 and position information of the object within the image. When a vehicle is detected from the image captured by the camera 2, it is detected as a large vehicle or an ordinary vehicle.

[0041] In OP103, the control unit 11 determines whether or not a vehicle is detected from the image captured by the camera 2. If a vehicle is detected from the image captured by the camera 2 (OP103: YES), the process proceeds to OP104. If a vehicle is not detected from the image captured by the camera 2 (OP103: NO), the process shown in FIG. 5 ends.

[0042] In OP104, the control unit 11 acquires, from the dimension information storage unit 15, the dimensional information of the license plate corresponding to the type of vehicle detected from the image captured by the camera 2. When a large vehicle is detected, the dimensional information of the license plate for large vehicles is acquired. When a standard vehicle is detected, the dimensional information of the license plate for standard vehicles is acquired.

[0043] In OP105, the control unit 11 cuts out a portion corresponding to the vehicle detected by the object detection unit 12 from the image captured by the camera 2. This is a process for reducing the size of the image input to the license plate detection unit 13 and reducing the processing load. The position of the vehicle in the image captured by the camera 2 is acquired by the object detection unit 12 in OP102.

[0044] In OP106, the control unit 11 instructs the license plate detection unit 13 to start a detection process of the license plate from the image captured by the camera 2, and acquires a detection result of the license plate from the license plate detection unit 13. As a detection result of the license plate by the license plate detection unit 13, position information of the license plate within the image is acquired.

[0045] In OP107, the control unit 11 obtains the number of vertical pixels of the license plate in the image from the result of the detection of the license plate by the license plate detection unit 13. The number of vertical pixels of the license plate may be obtained, for example, by cutting out an area corresponding to the license plate from the image captured by the camera 2. Furthermore, when the license plate is tilted in the image captured by the camera 2, the control unit 11 obtains the tilt of the license plate from the aspect ratio of the numbers written on the license plate, corrects the image of the license plate to a front view using the obtained tilt, and then obtains the number of vertical pixels of the license plate in the captured image. The tilt of the license plate in the image is obtained, for example, by comparing the aspect ratio of a known number with the aspect ratio of the number in the image. By using this method, even if the vertices of the square of the license plate are not clear in the captured image, if even one of the numbers on the license plate can be detected, the coordinates of the vertices of the square can be obtained based on the tilt of the license plate.

[0046] In OP108, the control unit 11 estimates the distance to the vehicle detected in OP102 using the size information of the license plate acquired in OP104, the information of the license plate in the captured image acquired in OP107, and the focal length and pixel size of the camera 2 acquired from the camera information storage unit 16 (see FIG. 4). Then, the process shown in FIG. 5 ends. The estimated distance to the vehicle is used, for example, for an ACC (Adaptive Cruise Control System: vehicle distance control) of an ADAS (Advanced Driver Assistance System) and a FCW (Forward collision warning: forward vehicle approach warning).

[0047] When a plurality of vehicles are detected from the image captured by the camera 2 by the object detection process of OP 102, the processes of OP 104 to OP 108 may be executed for each of the detected vehicles. Note that the distance estimation process shown in Fig. 5 is an example, and the distance estimation process executed by the ECU 1 is not limited to the process shown in Fig. 5.

[0048] <Effects of the First Embodiment> According to the first embodiment, the size of the license plate corresponding to the type of vehicle appearing in the image captured by the camera 2 can be used to improve the accuracy of estimating the distance to the vehicle.

[0049] <Other Modifications> The above-described embodiment is merely an example, and the present disclosure can be modified and implemented as appropriate without departing from the spirit and scope of the present disclosure.

[0050] In the first embodiment, the distance estimation process is executed by the vehicle 10, but the distance estimation process may be executed by, for example, a server. In the case where the server executes the distance estimation process, the DCM of the vehicle 10 3 transmits the captured image of camera 2 to the server, and receives the distance to vehicle 10, which is the result of the distance estimation process, from the server. In this case, the server is an example of an "information processing device." The distance estimation process described in the first embodiment may be executed by, for example, a user terminal such as a smartphone or tablet terminal equipped with a camera, or a PC capable of importing captured images. In this case, the device that executes the distance estimation process described in the first embodiment is an example of an "information processing device."

[0051] In the first embodiment, the vertical dimension of the license plate is used to estimate the distance to the vehicle in the captured image, but the horizontal dimension of the license plate may also be used. Since motorcycles and mopeds also have license plates, motorcycles and mopeds may also be included as vehicles. In this case, the object detection unit 12 detects motorcycles and mopeds as vehicle types from the captured image. In this case, the dimension information storage unit 15 also holds dimensional information of the license plates of the motorcycles and mopeds.

[0052] In the first embodiment, the target of distance estimation is the distance to a vehicle in the captured image of the camera 2, but the target of distance estimation is not limited to a vehicle. For example, if the object is a sign, a white line on a road, a safety fence, or the like, whose dimensions can be acquired in advance, the distance to the object appearing in the captured image can be estimated.

[0053] The processes and means described in this disclosure can be freely combined and implemented as long as no technical contradiction occurs.

[0054] Furthermore, a process described as being performed by one device may be shared and executed by multiple devices. Alternatively, a process described as being performed by different devices may be executed by one device. In a computer system, the hardware configuration (server configuration) by which each function is realized can be flexibly changed.

[0055] The present disclosure can also be realized by supplying a computer program implementing the functions described in the above embodiments to a computer, and having one or more processors of the computer read and execute the program. Such a computer program may be provided to the computer by a non-transitory computer-readable storage medium connectable to the system bus of the computer, or may be provided to the computer via a network. Non-transitory computer-readable storage media include, for example, any type of disk, such as a magnetic disk (floppy disk, hard disk drive (HDD), etc.), an optical disk (CD-ROM, DVD disk, Blu-ray disk, etc.), a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic card, a flash memory, an optical card, and any type of medium suitable for storing electronic instructions. [Explanation of symbols]

[0056] 1··ECU 2. Camera 3. DCM 10. Vehicle 11 Control section 12. Object detection unit 13 License plate detector 14. Image acquisition section 15. Dimension information storage section 16 Camera information storage section 100 Distance Estimation System 101··CPU 102 Memory 103...Auxiliary storage device 104 Interface

Claims

[Claim 1] Obtaining the type of vehicle appearing in the image captured by the camera; Cutting out a portion corresponding to the vehicle from the captured image; Identifying a license plate of the vehicle from an image of a portion corresponding to the vehicle; Obtaining the number of pixels in the vertical or horizontal direction of the license plate of the vehicle in the image of the portion corresponding to the vehicle; Estimating a distance from the camera to the vehicle based on a vertical or horizontal dimension of a license plate according to the type of the vehicle and the number of pixels in the vertical or horizontal direction of the license plate; A control unit that executes When the license plate is tilted in the image of the portion corresponding to the vehicle, the control unit obtains the tilt of the license plate from the aspect ratio of the numbers written on the license plate, corrects the image of the portion corresponding to the vehicle to an image of the license plate viewed from the front using the tilt of the license plate, and obtains the number of vertical or horizontal pixels of the license plate in the corrected image of the portion corresponding to the vehicle. Information processing device.

Images