Method

Abstract

To improve a technique for detecting an object from a video captured by a vehicle.SOLUTION: A method executed by an information processing apparatus includes: acquiring a video captured by a vehicle; continuously detecting a stationary object from the acquired video; and, if the detection of the object fails, searching for the object in the video again until a tracking time set according to a speed of the vehicle passes. The tracking time is set longer as compared with the speed higher than a threshold value if the speed is lower than the threshold value.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a method. [Background technology]

[0002] Conventionally, there are known techniques for detecting objects from images captured by a vehicle camera. For example, Patent Document 1 discloses a technique for estimating whether an object that was once lost due to obstruction is the same object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-228303 A Summary of the Invention [Problem to be solved by the invention]

[0004] When searching for an object from a video using image analysis technology such as machine learning, if the search target is lost and a certain time has passed, the search is terminated and the next object detected during the search is often determined to be a different object. However, for example, when a vehicle is stopped or moving at a low speed at a traffic light, etc., and an obstruction such as a large vehicle appears in front of the vehicle, the stationary object in the shadow of the obstruction may be lost and the tracking time may reach the upper limit. In this case, there is a risk that the same object may be erroneously determined to be a different object by a subsequent re-search. As such, there is room for improvement in technology for detecting objects from video captured by a vehicle.

[0005] An object of the present disclosure, made in consideration of the above circumstances, is to improve technology for detecting objects from images captured by a vehicle. [Means for solving the problem]

[0006] A method according to one embodiment of the present disclosure is executed by an information processing device and includes acquiring video captured by a vehicle, continuously detecting stationary objects from the acquired video, and, if the object cannot be detected, searching again for the object in the video with a tracking time as an upper limit set according to the speed of the vehicle, wherein the tracking time is set longer when the speed is equal to or less than a threshold compared to when the speed is above the threshold. Effect of the Invention

[0007] According to one embodiment of the present disclosure, it is possible to improve the technology for detecting objects from images captured by a vehicle. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating a schematic configuration of a system according to an embodiment of the present disclosure. [Diagram 2] 1 is a flowchart illustrating an operation of an information processing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. In each drawing, the same or corresponding parts are denoted by the same reference numerals. In the description of this embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate.

[0010] 1, a system 1 according to an embodiment of the present disclosure includes an information processing device 10 and a vehicle 20. The devices are communicably connected to each other via a network 30 including, for example, the Internet.

[0011] The information processing device 10 is installed in a facility such as a data center. The information processing device 10 is, for example, a server belonging to a cloud computing system or other computing system. The information processing device 10 may be configured as an in-vehicle device provided in a vehicle 20.

[0012] The vehicle 20 is any type of automobile that uses electricity for at least a part of its drive, such as a gasoline vehicle, a diesel vehicle, a BEV (Battery Electric Vehicle), a HEV (Hybrid Electric Vehicle), a PHEV (Plug-in Hybrid Electric Vehicle), or a FCEV (Fuel Cell Electric Vehicle). The vehicle 20 may be capable of moving by autonomous driving. The autonomous driving includes, for example, levels 1 to 5 defined by the Society of Automotive Engineers (SAE), but is not limited to these and may be defined arbitrarily. The vehicle 20 includes an imaging unit including a camera or the like, and is capable of capturing images of the surroundings and transmitting the generated images to the information processing device 10. The vehicle 20 includes a sensor unit that detects the speed of the vehicle 20, and is capable of transmitting information indicating the speed of the vehicle 20 to the information processing device 10. The sensor unit may be capable of detecting the acceleration or angular velocity of the vehicle 20.

[0013] The network 30 includes the Internet, at least one WAN, at least one MAN, or a combination thereof. "WAN" is an abbreviation for wide area network. "MAN" is an abbreviation for metropolitan area network. The network 30 may include at least one wireless network, at least one optical network, or a combination thereof. The wireless network is, for example, an ad-hoc network, a cellular network, a wireless LAN, a satellite communication network, or a terrestrial microwave network. "LAN" is an abbreviation for local area network. The network 30 may include a CAN (Controller Area Network), etc.

[0014] First, an overview of this embodiment will be described, and details will be described later. The information processing device 10 acquires an image captured by the vehicle 20, continuously detects a stationary object from the acquired image, and when the object cannot be detected, searches for the object in the image again with a tracking time set according to the speed of the vehicle 20 as an upper limit. When the speed is equal to or less than a threshold, the tracking time is set to be longer than when the speed is above the threshold.

[0015] Stationary objects include, for example, traffic lights, signs, billboards, etc. According to this embodiment, when an object cannot be detected, i.e., when the object is lost, the upper limit of the tracking time can be changed according to the moving state of the vehicle 20. This makes it possible to continuously search and detect objects while reducing the possibility of misjudging the same stationary object as a different stationary object by re-searching. Therefore, it becomes possible to improve the technology for detecting objects from images captured by a vehicle.

[0016] Next, the configuration of the information processing device 10 will be described in detail with reference to Fig. 1. The information processing device 10 includes a control unit 11, a storage unit 12, and a communication unit 13.

[0017] The control unit 11 includes at least one processor, at least one dedicated circuit, or a combination of these. The processor is a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for specific processing. "CPU" is an abbreviation for central processing unit. "GPU" is an abbreviation for graphics processing unit. The dedicated circuit is, for example, an FPGA or ASIC. "FPGA" is an abbreviation for field-programmable gate array. "ASIC" is an abbreviation for application specific integrated circuit. The control unit 11 executes processing related to the operation of the information processing device 10 while controlling each part of the information processing device 10.

[0018] The storage unit 12 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a RAM or a ROM. "RAM" is an abbreviation for random access memory. "ROM" is an abbreviation for read only memory. The RAM is, for example, an SRAM or a DRAM. "SRAM" is an abbreviation for static random access memory. "DRAM" is an abbreviation for dynamic random access memory. The ROM is, for example, an EEPROM. "EEPROM" is an abbreviation for electrically erasable programmable read only memory. The storage unit 12 functions as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 12 stores information used in the operation of the information processing device 10 and information obtained by the operation of the information processing device 10. For example, the storage unit 12 may store a system program, an application program, a database, map information, and the like. The information stored in the storage unit 12 may be updated with information acquired from the network 30 via the communication unit 13.

[0019] The communication unit 13 includes at least one communication interface. The communication interface is, for example, a LAN interface, a LAN interface, or an interface compatible with a mobile communication standard such as LTE, 4G, or 5G. "LTE" is an abbreviation for Long Term Evolution. "4G" is an abbreviation for 4th generation. "5G" is an abbreviation for 5th generation. The communication unit 13 receives data used in the operation of the information processing device 10, and transmits data obtained by the operation of the information processing device 10.

[0020] The functions of the information processing device 10 are realized by executing a program according to this embodiment on a processor corresponding to the control unit 11. That is, the functions of the information processing device 10 are realized by software. The program causes a computer to execute the operations of the information processing device 10, thereby causing the computer to function as the information processing device 10. That is, the computer functions as the information processing device 10 by executing the operations of the information processing device 10 in accordance with the control program.

[0021] The program can be stored in a non-transitory computer-readable medium. In this embodiment, the non-transitory computer-readable medium is a ROM. Non-transitory computer-readable media include, for example, a magnetic recording device, an optical disk, and a magneto-optical recording medium. The program is distributed, for example, by selling, transferring, or lending a portable medium such as a DVD or CD-ROM on which the program is stored. "DVD" is an abbreviation for digital versatile disc. "CD-ROM" is an abbreviation for compact disc read only memory. The program may be distributed by storing the program in the storage of a server and transferring the program from the server to another computer. The program may be provided as a program product.

[0022] A computer temporarily stores a program stored in a portable medium or transferred from a server in a main storage device. Then, the computer reads the program stored in the main storage device with a processor and executes processing according to the read program with the processor. The computer may read the program directly from a portable medium and execute processing according to the program. The computer may execute processing according to the received program each time a program is transferred from the server to the computer. Processing may be executed by a so-called ASP-type service that realizes functions only by execution instructions and result acquisition without transferring a program from the server to the computer. "ASP" is an abbreviation for application service provider. Programs include information used for processing by a computer and equivalent to a program. For example, data that is not a direct command to a computer but has the property of defining computer processing falls under " equivalent to a program.

[0023] A part or all of the functions of the information processing device 10 may be realized by a programmable circuit or a dedicated circuit as the control unit 11. In other words, a part or all of the functions of the information processing device 10 may be realized by hardware.

[0024] An operation of the information processing device 10 according to this embodiment will be described with reference to Fig. 2. The operation corresponds to a method. In the following, the information processing device 10 transmits and receives information to and from the vehicle 20 via the communication unit 13 and the network 30.

[0025] 2, the control unit 11 of the information processing device 10 receives and acquires video captured by the vehicle 20. The video may be generated by an imaging unit such as a camera provided in the vehicle 20 by capturing the video at a predetermined frame rate.

[0026] In S2, the control unit 11 detects stationary objects from the video. Any image analysis technology such as machine learning may be used for the detection. The control unit 11 can detect multiple objects from the video. The processing operations of the control unit 11 from S3 onwards are performed independently for each of the multiple objects detected from the video.

[0027] In S3, the control unit 11 judges whether or not the object has been lost from the image. The control unit 11 judges whether or not each of the multiple objects detected in S2 has been lost. Any method may be adopted for the judgment. For example, the control unit 11 may use any image analysis technology to judge that the object has been lost due to the presence of an obstruction such as another vehicle in front. For example, the control unit 11 may judge that the object has been lost when the object detected as a stationary object has disappeared and the stationary object gradually shifts out of the shooting area and disappears, that is, other than the situation where the object goes out of frame due to the movement of the vehicle 20. If it is judged that the object has been lost, the process of the control unit 11 proceeds to S4. If it is judged that the object has not been lost, the control unit 11 repeats the process of S3. In this way, the control unit 11 continuously detects the object.

[0028] In S4, the control unit 11 acquires information indicating the speed of the vehicle 20. The control unit 11 acquires the information by receiving the information from the vehicle 20, the information being detected by a sensor unit included in the vehicle 20.

[0029] In S5, the control unit 11 acquires information indicating a threshold value, and judges whether or not the speed of the vehicle 20 indicated by the information acquired in S4 is equal to or less than the threshold value. The control unit 11 may acquire information indicating a preset threshold value by reading it from the storage unit 12. If it is greater than the threshold value, the processing operation of the control unit 11 proceeds to S6. If it is equal to or less than the threshold value, the processing operation of the control unit 11 proceeds to S9.

[0030] In S6, the control unit 11 searches for the object again from the video, with the tracking time t1 as the upper limit. Any image analysis technique such as machine learning may be adopted for the search. The control unit 11 may obtain information indicating the preset tracking time t1 by reading it from the storage unit 12. After that, the process of the control unit 11 proceeds to S7.

[0031] In S7, the control unit 11 judges whether or not an object has been detected. If an object has been detected, the control unit 11 regards the object as being the same object that was lost in S3. Then, the processing operation of the control unit 11 returns to S3. If an object has not been detected, the processing operation of the control unit 11 proceeds to S8.

[0032] In S8, the control unit 11 judges whether the tracking time t1 has elapsed. If the tracking time t1 has elapsed, the processing operation of the control unit 11 for the object that is one of the multiple objects detected in S2 and that was lost in S3 ends. If the tracking time t1 has not elapsed, the processing operation of the control unit 11 returns to S7.

[0033] In S9, the control unit 11 searches for the object again from the video, with the tracking time t2 as the upper limit. Any image analysis technique such as machine learning may be adopted for the search. The tracking time t2 is longer than the tracking time t1. The control unit 11 may obtain information indicating the preset tracking time t2 by reading it from the storage unit 12. After that, the process of the control unit 11 proceeds to S9.

[0034] As shown in S3 to S9, when the control unit 11 is unable to detect the object, it searches again for the object in the video, with the upper limit being the tracking time set according to the speed of the vehicle 20. When the speed is equal to or less than the threshold, the tracking time t2 is set to be longer than when the speed exceeds the threshold, i.e., compared to the tracking time t1.

[0035] In S10, the control unit 11 judges whether an object has been detected. If an object has been detected, the control unit 11 regards the object as the same object that was lost in S3. Then, the processing operation of the control unit 11 returns to S3. If an object has not been detected, the processing operation of the control unit 11 proceeds to S11.

[0036] In S11, the control unit 11 judges whether the tracking time t2 has elapsed. If the tracking time t2 has elapsed, the processing operation of the control unit 11 for the object that is one of the multiple objects detected in S2 and that was lost in S3 ends. If the tracking time t2 has not elapsed, the processing operation of the control unit 11 returns to S10.

[0037] As described above, the method executed by the information processing device 10 according to the present embodiment includes acquiring an image captured by the vehicle 20, continuously detecting a stationary object from the acquired image, and searching again for the object in the image when the object cannot be detected, with a tracking time set according to the speed of the vehicle 20 as an upper limit, and the tracking time is set longer when the speed is equal to or less than a threshold value compared to when the speed exceeds the threshold value. According to the present embodiment, when the vehicle 20 is moving at a low speed or stopped in a traffic jam or the like, and an object once detected from the image is lost, the control unit 11 can search again for the object with a tracking time set longer than normal as an upper limit. By setting the tracking time longer, it is possible to reduce the possibility that the control unit 11 will erroneously determine the same object as a different object. Therefore, it is possible to improve the technology for detecting objects from images captured by a vehicle.

[0038] The present disclosure is not limited to the above-described embodiments. For example, two or more blocks shown in the block diagram may be integrated, or one block may be divided. Instead of executing two or more steps shown in the flowchart in chronological order as described, they may be executed in parallel or in a different order depending on the processing capacity of the device executing each step, or as necessary. Other modifications are possible without departing from the spirit of the present disclosure.

[0039] The information processing device 10 may be communicably connected to the vehicle 20 via an in-vehicle network such as a CAN. In this case, the information processing device 10 is provided in the vehicle 20, and can directly transmit and receive images or various information to and from the vehicle 20. [Explanation of symbols]

[0040] 1 System 10. Information processing device 11 Control section 12 Storage section 13. Communications Department 20 Vehicles 30 Network

Claims

[Claim 1] A method executed by an information processing device, comprising: Acquiring video captured by a vehicle; continuously detecting stationary objects from the captured images; and when the object cannot be detected, searching again for the object in the image, with a tracking time set according to a speed of the vehicle as an upper limit. The tracking time is set to be longer when the speed is equal to or less than a threshold value compared to when the speed is greater than the threshold value. method.

Images