Information processing device

Abstract

To provide an information processing apparatus for improving accuracy of depth estimation of an image captured by a camera.SOLUTION: In a depth estimation system including a vehicle and a server, the server is an information processing apparatus that learns a machine learning model for estimating the depth of an object in an image captured by a camera mounted on a vehicle, using at least two consecutive captured images in a moving image captured by the camera. The information processing apparatus is configured to: obtain a first image by removing, from the images captured by the camera, areas with little change between the captured images included in the moving image captured by the camera and learn the machine learning model using the first image; or obtain a first image by removing, from the images captured by the camera, the areas with little change between the captured images included in the moving image captured by the camera and estimate the depth for the first image by the machine learning model.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present disclosure relates to an information processing device. [Background technology]

[0002] A method has been disclosed for creating a depth image in which each pixel has a depth, by assigning a single representative depth to each pixel in a window to generate a depth image, and filtering each pixel in the depth image to correct outlier depths without blurring depth discontinuities (e.g., Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2010-218549 A Summary of the Invention [Problem to be solved by the invention]

[0004] One aspect of the disclosure has an object to provide an information processing device capable of improving the accuracy of estimating the depth of an object in an image captured by a camera. [Means for solving the problem]

[0005] One aspect of the present disclosure is A control unit is provided that performs learning of a machine learning model that estimates the depth of an object in an image captured by a camera using at least two consecutive captured images in a video captured by the camera, or performs estimation of the depth of an object in an image captured by the camera using the trained machine learning model; The control unit is obtaining a first image by removing an area having little change between captured images included in a moving image captured by the camera from the captured image of the camera; performing the learning or the estimation using the first image; It is an information processing device. Effect of the Invention

[0006] According to one aspect of the present disclosure, it is possible to improve the accuracy of estimating the depth of an object in an image captured by a camera. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an example of a depth estimation method according to the first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a system configuration of the depth estimation system according to the first embodiment and a hardware configuration of a vehicle and a server. [Diagram 3] FIG. 3 is a diagram illustrating an example of a functional configuration of a vehicle. [Figure 4] FIG. 4 is a diagram illustrating an example of a functional configuration of the server. [Diagram 5] FIG. 5 is a diagram showing an example of removing the fixed region performed in the first embodiment. [Figure 6] FIG. 6 is a diagram showing an example of removing the fixed region performed in the first embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of learning of a depth estimation model. [Figure 8] FIG. 8 is an example of a flowchart of a vehicle depth estimation process. [Figure 9] FIG. 9 is an example of a flowchart of a learning process of a depth estimation model of the server. [Figure 10] FIG. 10 is a diagram showing an example of a depth estimation result with and without removal of the fixed region. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] One method of estimating depth in an image captured by a monocular camera is to use a machine learning model. In learning the machine learning model, multiple images of the same scene captured from multiple viewpoints by the same camera are used. For example, multiple consecutive images in a video captured by the camera are used as the multiple images of the same scene captured from multiple viewpoints. For example, when the camera is installed on a vehicle, a part of the vehicle body may be included in the imaging range of the camera. In that case, a part of the vehicle reflected in the multiple consecutive images in the video captured by the camera becomes an area with little change. An area with little change between multiple consecutive images in the video may be determined as infinity by the machine learning model, and the accuracy of the depth estimation of an object in the captured image by the machine learning model may be reduced.

[0009] In one aspect of the present disclosure, an area with little change in a video captured by a camera is removed from an image used for learning the machine learning model or estimation by the machine learning model, thereby suppressing the area from being determined as being at infinity. More specifically, one aspect of the present disclosure is an information processing device including a control unit. The control unit executes learning of a machine learning model that estimates the depth of an object in an image captured by a camera using at least two consecutive captured images in a video captured by the camera, or executes estimation of the depth of an object in an image captured by the camera using the trained machine learning model. The control unit removes an area with little change between captured images included in the video captured by the camera from the image captured by the camera to obtain a first image, and uses the first image to train the machine learning model or perform estimation by the trained machine learning model.

[0010] The camera is a monocular camera with one lens. The camera is, for example, mounted or installed on a moving body, or held or worn by a human, and the distance to surrounding objects changes as the camera moves. In other words, the camera in one aspect of the present disclosure is a camera that does not include a fixed camera or the like and is not fixed to a non-moving object. The moving body on which the camera is mounted or installed may be, for example, a vehicle, a ship, or an airplane. Vehicles on which a camera is mounted include, for example, automobiles, bicycles, motorcycles, and railroad cars. Areas with little change in a video captured by a camera are, for example, areas included within the imaging range of the camera, which correspond to a part of the body of the moving body on which the camera is mounted or installed, a part of the external or internal parts of the moving body, a sticker attached to the window of the moving body, and an object loaded on the moving body.

[0011] In addition, the camera that captures the video used to train the machine learning model and the camera that captures the images for which depth is estimated by the machine learning model may be the same or different individuals.

[0012] The machine learning model that estimates the depth of an object in a captured image is a model that is realized by combining algorithms used in image processing, such as a Convolutional Neural Network (CNN), a Residual Network (ResNet), and PackNet. The machine learning model outputs the depth of each pixel in response to an input of the pixel value of each pixel in a captured image. The depth is the distance from the center point of the camera lens to the object, that is, the so-called depth.

[0013] Removing from a camera's captured image an area that has little change between captured images contained in a video captured by the camera may include, for example, cutting out the area from the captured image and masking the area in the captured image.

[0014] The information processing device is, for example, a computer mounted on a moving object on which the camera is mounted or installed, or a dedicated computer such as a server. The information processing device is a terminal device such as a smartphone, a tablet terminal, a PC (Personal Computer), or a game terminal. The computer mounted on the moving object may be, for example, an ECU (Electronic Control Unit). Unit), data communication devices, car navigation systems, and drive recorders. In addition, the information processing device that learns the machine learning model and the information processing device that performs estimation using the machine learning model may be different devices. Alternatively, the same information processing device may learn the machine learning model and perform estimation using the machine learning model. The control unit of the information processing device is, for example, a processor such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit).

[0015] According to one aspect of the present disclosure, a machine learning model is trained using images in which areas with little change in a video, such as a part of a moving object captured in an image captured by a camera, have been removed, thereby making it possible to prevent such areas from being determined to be at infinity, thereby improving the accuracy of estimating the depth of objects in the captured image.

[0016] Another aspect of the present disclosure can be specified as a method for causing a computer to execute the processing of the information processing device, or as a program for causing a computer to execute the processing of the information processing device, and a non-transitory computer-readable recording medium for recording the program.

[0017] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The configurations of the following embodiments are merely examples, and the present disclosure is not limited to the configurations of the embodiments.

[0018] First Embodiment FIG. 1 is a diagram showing an example of a depth estimation method according to the first embodiment. In the first embodiment, the depth of an object appearing in an image captured by a camera 2 mounted on a vehicle 10 is estimated using a machine learning model. It is assumed that the camera 2 is installed near the front of the roof of the vehicle 2. The imaging range of the camera 2 includes the bonnet of the vehicle 10. Therefore, as shown in an image P1 captured by the camera 2, a part of the bonnet of the vehicle 10 is reflected below the image captured by the camera 2.

[0019] The bonnet part of the vehicle 10 that is reflected in the video captured by the camera 2 does not change. Therefore, if a machine learning model is trained using frame images in the video captured by the camera 2 and the depth of the image captured by the camera 2 is estimated using the machine learning model, there is a high possibility that the bonnet part of the vehicle 10 will be determined to be infinitely far away.

[0020] In the first embodiment, an area with little change in the video, such as the bonnet of the vehicle 10 reflected in the image captured by the camera 2, is removed, and the machine learning model is trained and estimated using the image captured after removal. This makes it possible to prevent an area with little change in the video, such as the bonnet of the vehicle 10 reflected in the image captured by the camera 2, from being determined to be infinitely far away, and improves the accuracy of estimating the depth of an object in the image captured by the camera 2.

[0021] The camera 2 is not limited to being installed near the front of the roof of the vehicle 2, and may be, for example, a camera installed inside the vehicle 10 on the upper part of the windshield or on the back side of the rearview mirror so as to capture a predetermined range in front of the vehicle 10. Alternatively, the camera 2 may be, for example, a camera installed on the upper part of the rear window of the vehicle 10 so as to capture an image of the rear of the vehicle 10. The camera 2 may be provided exclusively for depth estimation, or may be a camera used in another system such as a drive recorder.

[0022] FIG. 2 is a diagram showing an example of a system configuration of the depth estimation system 100 according to the first embodiment and a hardware configuration of the vehicle 10 and the server 5. The depth estimation system 100 includes the vehicle 10 and the server 5. The server 5 learns a machine learning model. The vehicle 10 estimates the depth of an object in an image captured by the camera 2 using the learned machine learning model. The vehicle 10 and the server 5 are connected to a network such as the Internet, for example, and can communicate with each other. Note that in FIG. 2, the hardware configuration of the vehicle 10 is shown by extracting components related to the depth estimation process, and the vehicle 10 also includes a hardware configuration related to running in addition to the hardware components shown in FIG. 2.

[0023] The vehicle 10 includes an ECU 1, a camera 2, and a DCM 3 as hardware components related to depth estimation processing. The ECU 1, the camera 2, and the DCM 3 are connected to each other via, for example, a Controller Area Network (CAN) or an in-vehicle Ethernet (registered trademark) or the like. do.

[0024] The ECU 1 is, for example, a control device that controls the camera 2. The ECU 1 includes, as its hardware configuration, a CPU 101, a memory 102, an auxiliary storage device 103, and an interface 104. The memory 102 and the auxiliary storage device 103 are computer-readable recording media. The ECU 1 is an example of an "information processing device."

[0025] The auxiliary storage device 103 stores various programs and data used by the CPU 101 when executing each program. The auxiliary storage device 103 is, for example, an EPROM (Erasable Programmable ROM) or a flash memory. The programs that are used include, for example, a depth estimation program. The depth estimation program is a program that estimates the distance to an object in an image captured by the camera.

[0026] The memory 102 is a storage device that provides the CPU 101 with a storage area for loading programs stored in the auxiliary storage device 103 and a working area, and is used as a buffer. The memory 102 may be, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), or the like. This includes semiconductor memories such as IEEE 802.11b / g and IEEE 802.11b / g.

[0027] The CPU 101 executes various processes by loading the OS and various other programs stored in the auxiliary storage device 103 into the memory 102 and executing them. The number of CPUs 101 is not limited to one, and may be multiple. The CPU 101 is an example of a "control unit."

[0028] The interface 104 is, for example, a circuit having a port used for connection to a CAN, an in-vehicle Ethernet (registered trademark), or other network. Note that the hardware configuration of the ECU 1 is not limited to that shown in FIG.

[0029] The camera 2 is, for example, a monocular camera. The camera 2 includes an image sensor and a lens. The image sensor of the camera 2 may be, for example, a CCD (Charge Coupled Device), and CMOS (Complementary Metal Oxide Semiconductor). The camera 2 is connected to the ECU 1 through a network such as CAN or in-vehicle Ethernet, but is not limited to this. The camera 2 may be connected to the ECU 1 directly via a cable or the like without going through a network.

[0030] The DCM 3 is a communication device having a communication function between the vehicle 10 and an external device. The DCM 3 supports, for example, LTE (Long Term Evolution), 5G (5th Generation), and 6G. Mobile radio communication systems such as (6th Generation), Wi-Fi, or wireless communication systems such as DSCR The depth estimation process is not limited to being executed by the ECU 1, and may be executed by the DCM 3, for example.

[0031] The server 5 includes, as its hardware configuration, a CPU 501, a memory 502, an auxiliary storage device 503, and a communication unit 504. The CPU 501, the memory 502, and the auxiliary storage device 503 are similar to the CPU 101, the memory 102, and the auxiliary storage device 103, respectively. However, the auxiliary storage device 503 may be, for example, a solid state drive (SSD), The auxiliary storage device 503 is a function for performing depth estimation. The server 5 stores a learning control program for the machine learning model. The server 5 may further include a GPU (Graphics Processing Unit) instead of the CPU 501 or in addition to the CPU 501.

[0032] The communication unit 504 is connected to the network by wire or wirelessly. The communication unit 504 is, for example, a network interface card (NIC), an optical communication circuit, or a wireless communication circuit. The communication unit 504 converts data from the CPU 501 into a signal and transmits it to the network, and converts a signal received via the network into data and outputs it to the CPU 501. The server 5 is an example of an "information processing device" that performs learning of a machine learning model. Note that the hardware configuration of the vehicle 10 and the server 5 is not limited to that shown in FIG. 2.

[0033] 3 is a diagram showing an example of a functional configuration of the vehicle 10. The vehicle 10 includes, as functional components, an image acquisition unit 11, a speed acquisition unit 12, an image processing unit 13, a depth estimation unit 14, a communication unit 15, and a recorded data storage unit 16. The processing of these functional components is achieved by the CPU 101 executing a depth estimation program stored in the auxiliary storage device 103.

[0034] The image acquisition unit 11 acquires image data captured by the camera 2 at a predetermined cycle from the camera 2. The imaging rate of the camera 2 is set, for example, in the range of 15 fps to 60 fps according to the performance of the camera 2 or according to a setting by the owner or manager of the vehicle 10. The image acquisition unit 11 outputs the acquired images to the speed acquisition unit 12 and the image processing unit 13.

[0035] When the image captured by the camera 2 is input from the image acquisition unit 11, the speed acquisition unit 12 associates the speed of the vehicle 10 with the image captured by the camera 2 and stores the image in the recorded data storage unit 16. The speed acquisition unit 12 acquires the speed of the vehicle 10, for example, at a predetermined cycle or when the image captured by the camera 2 is input from the image acquisition unit 11. Since the input of the captured image from the image acquisition unit 11 is performed, for example, at the same cycle as the imaging rate of the camera 2, the acquisition cycle of the speed of the vehicle 10 by the speed acquisition unit 12 may be the same as the imaging rate of the camera 2. However, the acquisition cycle of the speed of the vehicle 10 by the speed acquisition unit 12 may be different from the imaging rate of the camera 2. The speed of the vehicle 10 is acquired, for example, from a speedometer provided in the vehicle 10.

[0036] The image processing unit 13 performs a process of removing an area with little change in the video from the captured image of the camera 2 input from the image acquisition unit 11. The area with little change in the video captured by the camera 2 is, for example, an area corresponding to a part of the vehicle 10 reflected in the video. The area with little change in the video captured by the camera 2 is hereinafter referred to as a fixed area. The fixed area may be, for example, designated in advance by an administrator of the depth estimation system 100, an owner of the vehicle 10, or the like, or the image processing unit 13 may identify a frame image in the video of the camera 2 by image recognition processing or a machine learning model, etc. In addition to the hood and roof of the vehicle 10, when the camera 2 is installed inside the vehicle, the fixed area may be an area where a sticker attached to the window glass and interior parts such as a mirror and dashboard are reflected in the captured image of the camera 2. Details of removing the fixed area from the captured image of the camera 2 will be described later. The image processing unit 13 The captured image from which the fixed region has been removed is output to the depth estimation unit 14. The captured image from which the fixed region has been removed is an example of a "first image."

[0037] The depth estimation unit 14 performs depth estimation of an object in an image captured by the camera 2 using a trained machine learning model. Hereinafter, the machine learning model used for depth estimation is referred to as a depth estimation model. The depth estimation model is a model realized by combining algorithms used in image processing, such as a convolutional neural network (CNN), ResNet (Residual Network), and PackNet. When the depth estimation unit 14 receives an input of an image captured by the camera 2 from which a fixed region has been removed from the image processing unit 13, the depth estimation unit 14 inputs the captured image as an input image to the depth estimation model. For example, the depth estimation model outputs an estimation result of the depth of each pixel in response to an input of a pixel value of each pixel of the input image. The estimation result of the depth estimation model is, for example, an estimated value of the depth for each pixel, or a probability of classification into each of a plurality of depth levels. The depth level may be set in units of 1 m to 10 m from the camera 2, for example. For example, the depth estimation unit 14 may create and output a depth map for the input image by coloring or shading each pixel based on the depth estimation result of each pixel. The result of the depth estimation may be output to another ECU that uses depth, or may be stored in a predetermined database. In the depth map, the fixed region removed from the captured image may be displayed, for example, in a color or shade assigned to a depth estimate value or depth level that indicates a distance closer to the camera 2.

[0038] The communication unit 15 communicates with the server 5. For example, the communication unit 15 receives update data of the depth estimation model from the server 5. The communication unit 15 outputs the update data of the depth estimation model to the depth estimation unit 14, and the depth estimation unit 14 updates the depth estimation model based on the update data. The communication unit 15 also transmits the recorded data of the camera 2 stored in the recorded data storage unit 16 to the server 5 at a predetermined cycle. For example, the server 5 learns the depth estimation model based on the recorded data of the camera 2 and creates update data. The cycle of transmitting the recorded data to the server 5 is, for example, once a day, once a week, or once a month. The recorded data transmitted to the server 5 may be all or part of the data for a predetermined period between the previous transmission and the current transmission. When a portion of the recorded data is transmitted, for example, recorded data for a predetermined length of time immediately before the transmission time, recorded data for a predetermined length of time every hour between the previous transmission and the current transmission, etc. may be transmitted.

[0039] The recorded data storage unit 16 is created in a predetermined storage area of ​​the auxiliary storage device 103. The recorded data storage unit 16 holds the recorded data of the camera 2. The recorded data is associated with the speed of the vehicle 10 acquired by the speed acquisition unit 12 for each frame image. Note that the functional configuration of the vehicle 10 is not limited to that shown in FIG.

[0040] 4 is a diagram showing an example of the functional configuration of the server 5. The server 5 includes, as its functional configuration, a camera information storage unit 51, an image processing unit 52, a learning processing unit 53, a communication unit 54, and a learning data storage unit 55. The processing of these functional configurations is achieved by the CPU 501 executing a learning control program for the depth estimation model held in the auxiliary storage device 503.

[0041] The communication unit 54 communicates with the vehicle 10. When the communication unit 54 receives recorded data from the vehicle 10, the communication unit 54 stores the recorded data as learning data in the learning data storage unit 55. Recorded data for a predetermined period is received from the vehicle 10. In addition, when the communication unit 54 receives update data for the depth estimation model from the learning processing unit 53, the communication unit 54 transmits the update data to the vehicle 10.

[0042] The camera information storage unit 51 and the learning data storage unit 55 are created in the storage area of ​​the auxiliary storage device 503. The camera information storage unit 51 holds the image capturing rate of the camera 2. The rate may be transmitted from the vehicle 10, for example, when the depth estimation process is executed for the first time, when the imaging rate of the camera 2 is changed, etc., and may be received and acquired by the communication unit 54. Alternatively, the rate may be stored in the camera information storage unit 51 by an administrator of the depth estimation system 100. The learning data storage unit 55 holds the video recording data of the camera 2. In the video recording data in the learning data storage unit 55, for example, the speed of the vehicle 10 is associated with each frame image.

[0043] The image processing unit 52 reads out the recorded data of the camera 2 as learning data from the learning data storage unit 55, and removes the fixed area from each frame image in the recorded data in the same manner as the image processing unit 13. Information on the fixed area may be acquired from the vehicle 10, or the image processing unit 52 may identify the fixed area from the recorded data using a predetermined machine learning model or a predetermined image recognition process, or may be specified by an administrator of the depth estimation system 100. The image processing unit 52 outputs the frame image from which the fixed area has been removed to the learning processing unit 53.

[0044] The learning processing unit 53 receives an input of a frame image from which a fixed region has been removed from the image processing unit 52, and learns a depth estimation model using the frame image as learning data. Details of learning the depth estimation model will be described later. When learning of the depth estimation model is completed, the learning processing unit 53 creates update data for the depth estimation model. The update data for the depth estimation model includes, for example, parameter values ​​of the depth estimation model after learning. The learning processing unit 53 outputs the update data for the depth learning model to the communication unit 54, and the communication unit 54 transmits the update data for the depth learning model to the vehicle 10. Note that the functional configuration of the server 5 is not limited to that shown in FIG. 4.

[0045] 5 and 6 are diagrams showing an example of removing a fixed region performed in the first embodiment. The example shown in Fig. 5 is an example in which a fixed region is removed by cutting out the fixed region from an image captured by camera 2. In the example shown in Fig. 5, the blackened area at the bottom of image P1 is the fixed region, and image P1-1 with the blackened portion cut out is input to the depth estimation model.

[0046] The example shown in FIG. 6 is an example in which the fixed region is removed by masking the captured image of camera 2 using a mask image of the fixed region. In the example shown in FIG. 6, the blackened portion at the bottom right of the captured image P2 of camera 2 is the fixed region. In the mask image M2 of the fixed region, the same portion as the captured image P2 of camera 2 is blackened, indicating the portion to be masked. By multiplying the captured image P2 of camera 2 by the mask image M2, a masked image PM2 is obtained in which the portion of the captured image P2 other than the fixed region is extracted. The masked image PM2 is input to the depth estimation model.

[0047] The image processing unit 13 of the vehicle 10 and the image processing unit 52 of the server 5 remove the fixed region from the captured image of the camera 2 by either a method of cutting out the fixed region or a method of masking the fixed region. The image processing unit 13 of the vehicle 10 and the image processing unit 52 of the server 5 may remove the fixed region from the captured image of the camera 2 by the same method, or may remove the fixed region from the captured image of the camera 2 by different methods. In either case of the method of cutting out the fixed region or the method of masking the fixed region, the cut out region and the masked region are not reflected in the loss calculation when the depth estimation model is trained. In addition, for the cut out region and the masked region, at the time of depth estimation, for example, since there is no input value, no depth estimation result is obtained as an output value from the depth estimation model, or even if some depth estimation result is obtained, it is ignored because it is a random value, and is displayed with a predetermined color or density.

[0048] FIG. 7 is a diagram showing an example of learning of a depth estimation model. The learning processing unit 53 performs learning of the depth estimation model, for example, as follows. In step S1, the learning processing unit 53 inputs the t-th frame image to the depth estimation model. In step S2, the learning processing unit 53 acquires a depth map for the t-th frame image as an output of the depth estimation model. In FIG. 7, the depth is indicated by shading in the depth map. Note that the depth map in FIG. 7 In depth maps, lighter colors indicate closer objects and darker colors indicate further away objects. However, the representation of perspective in a depth map is not limited to this.

[0049] In step S3, the learning processing unit 53 generates a composite image of the t+1-th frame image based on the t-th frame image, the speed of the vehicle 10 associated with the t-th frame image, and the imaging rate of the camera 2. The distance traveled by the vehicle 10 from the t-th frame to the t+1-th frame can be obtained from the speed of the vehicle 10 and the imaging rate of the camera 2, and the t+1-th frame image can be estimated according to the travel distance. Note that the method of generating the composite image of the t+1-th frame image is not limited to a specific method, and any well-known technology may be used.

[0050] In step S4, the learning processing unit 53 compares the t+1th frame of the synthetic image with the t+1th frame image of the learning data, and performs a loss calculation. In addition to the absolute value of the error for each pixel between images, smoothness loss and other calculations are performed to obtain the loss. In step S5, the learning processing unit 53 adjusts parameters of the depth estimation model based on the loss acquired in step S4.

[0051] The learning processing unit 53 repeatedly executes the processes from step S1 to step S5 until a learning end condition is satisfied. The processes from step S1 to step S5 are regarded as one learning, and the learning end condition is, for example, that the loss obtained in step S5 is less than a threshold value or that the number of learning times reaches a predetermined value.

[0052] In the case of depth estimation using the depth estimation model, the partial processing enclosed by the dashed line in Fig. 7, i.e., the processing of steps S1 and S2, is executed for the target image. Therefore, the depth estimation unit 14 of the vehicle 10 executes the processing of steps S1 and S2 for the image captured by the camera 2.

[0053] The learning method of the depth estimation model is not limited to the method described in Fig. 7. For example, when the speed and imaging rate of the vehicle 10 are not acquired, a machine learning model that estimates a change in the attitude of the camera 2 from frame images before and after the t-th frame image may be used to estimate the change in the attitude of the camera 2, and a composite image of the t+1-th frame image may be generated using the estimation result. The machine learning model that estimates the change in the attitude of the camera is, for example, a model that uses CNN.

[0054] <Processing flow> Fig. 8 is an example of a flowchart of a depth estimation process of the vehicle 10. The process shown in Fig. 8 is executed repeatedly at a predetermined cycle while the vehicle 10 is traveling, for example. Without being limited to this, the process shown in Fig. 8 may be executed, for example, when an instruction to start the depth estimation process is input. The process shown in Fig. 8 is executed by the CPU 101 of the ECU 1, but for convenience, the process will be described mainly with respect to the functional components.

[0055] In OP101, the image acquisition unit 11 acquires an image captured by the camera 2. The image acquisition unit 11 outputs the image captured by the camera 2 to the image processing unit 13. In OP102, the image processing unit 13 removes the fixed region from the image captured by the camera 2. The image processing unit 13 outputs the image captured by the camera 2 from which the fixed region has been removed to the depth estimation unit 14. In OP103, the depth estimation unit 14 inputs the image captured by the camera 2 from which the fixed region has been removed to a trained depth estimation model, and acquires a depth estimation result for the input image from the depth estimation model. In OP104, the depth estimation unit 14 supplements the depth of the fixed region and outputs a depth estimation result for the image captured by the camera 2. A depth map may be output as the depth estimation result. The depth of the fixed region may be, for example, a value indicating a predetermined proximity, or a value indicating the closest position in a region other than the fixed region. The smaller depth may be adopted, or the process shown in FIG.

[0056] Fig. 9 is an example of a flowchart of a learning process of a depth estimation model of the server 5. The process shown in Fig. 9 is started, for example, at a predetermined timing or when a start instruction is input. The process shown in Fig. 9 is executed at a predetermined date and time, for example, once a day, once a week, or once a month. The process shown in Fig. 9 is executed by the CPU 501 of the server 5, but for convenience, the process will be described mainly with respect to the functional components.

[0057] In OP201, the image processing unit 52 reads out the recording data of the camera 2, which is the learning data, from the learning data storage unit 55. The read-out recording data is, for example, recording data of a predetermined length of time. In OP202, the image processing unit 52 removes fixed areas from each frame image included in the recording data of the camera 2, and outputs the removed data to the learning processing unit 53.

[0058] The processes of OP203 and OP204 are repeatedly executed for each image frame included in the recorded data. In OP203, the learning processing unit 53 executes learning of a depth estimation model for one frame image in the recorded data. In OP203, the processes of steps S1 to S5 in FIG. 7 are executed, for example, with one frame image in the recorded data as the t-th frame image.

[0059] In OP204, the learning processing unit 53 determines whether or not a condition for ending learning of the depth estimation model is satisfied. If the condition for ending learning of the depth estimation model is satisfied (OP204: YES), the process proceeds to the next frame image, or if learning of all frame images in the recorded data is completed, the process proceeds to OP205. If the condition for ending learning of the depth estimation model is not satisfied (OP204: NO), the process of OP203 is repeatedly executed.

[0060] In OP205, the learning processing unit 53 creates update data for the depth estimation model and transmits it to the vehicle 10 through the communication unit 54. The update data for the depth estimation model includes parameter values ​​of the depth estimation model after the learning of the depth estimation model is completed. Then, the process shown in FIG. 9 ends.

[0061] FIG. 10 is a diagram showing an example of a depth estimation result with and without removal of the fixed region. FIG. 10 shows a depth map P10 and a depth map P20 as a result of depth estimation for the same captured image P1 (see FIG. 1) of the camera 2. The depth map P10 is a depth map based on a depth estimation result performed for an image captured by the camera 2 in which the fixed region has not been removed, using a depth estimation model learned from an image captured by the camera 2 in which the fixed region has not been removed. The depth map P20 is a depth map based on a depth estimation result performed for an image captured by the camera 2 in which the fixed region has been removed, using a depth estimation model learned from an image captured by the camera 2 in which the fixed region has been removed. In the depth map P10 and the depth map P20 in FIG. 10, the perspective is indicated by light and dark, and the lighter the color, the closer, and the darker the color, the farther.

[0062] In the depth map P10, the area (fixed area) at the bottom corresponding to the bonnet of the vehicle 10 reflected in the captured image is darkened and determined to be infinitely far away. In contrast, in the depth map P20, the area (fixed area) corresponding to the bonnet of the vehicle 10 reflected in the captured image is excluded from the depth estimation process, and is therefore shown, for example, in shades corresponding to a predetermined depth value indicating closeness. Therefore, the depth map P20 shows a natural depth overall, with the change in depth corresponding to the perspective of an actual object. Note that the fixed area may be excluded from the depth map.

[0063] Therefore, according to the first embodiment, by using an image obtained by removing fixed areas from an image captured by camera 2 for learning the depth estimation model and estimation by the depth estimation model, it is possible to prevent the depth of an area with little change in the video captured by camera 2 from being determined to be infinite. This makes it possible to improve the estimation accuracy of the image captured by camera 2 by the depth estimation model.

[0064] <Other Modifications> The above-described embodiment is merely an example, and the present disclosure can be modified and implemented as appropriate without departing from the spirit and scope of the present disclosure.

[0065] In the first embodiment, the learning and estimation of the depth estimation model are performed by different devices, but the learning and estimation of the depth estimation model may be performed by the same device. For example, the ECU 1 of the vehicle 10 may learn the depth estimation model and use the depth estimation model to perform depth estimation of the image captured by the camera 2.

[0066] In the first embodiment, it is assumed that images captured by the same camera 2 are used for learning and estimation of the depth estimation model. This allows the depth estimation model to be specialized for the camera 2. However, this is not limited to the above, and a camera capturing an image used in learning the depth estimation model and a camera capturing an image to be estimated by the depth estimation model may be different cameras as long as the camera specifications and installation conditions are similar. Furthermore, the depth estimation model may be learned using video data from multiple cameras with similar specifications and installation conditions. This allows the depth estimation model to be versatile.

[0067] The processes and means described in this disclosure can be freely combined and implemented as long as no technical contradiction occurs.

[0068] Furthermore, a process described as being performed by one device may be shared and executed by multiple devices. Alternatively, a process described as being performed by different devices may be executed by one device. In a computer system, the hardware configuration (server configuration) by which each function is realized can be flexibly changed.

[0069] The present disclosure can also be realized by supplying a computer program implementing the functions described in the above embodiments to a computer, and having one or more processors of the computer read and execute the program. Such a computer program may be provided to the computer by a non-transitory computer-readable storage medium connectable to the system bus of the computer, or may be provided to the computer via a network. Non-transitory computer-readable storage media include, for example, any type of disk, such as a magnetic disk (floppy disk, hard disk drive (HDD), etc.), an optical disk (CD-ROM, DVD disk, Blu-ray disk, etc.), a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic card, a flash memory, an optical card, and any type of medium suitable for storing electronic instructions. [Explanation of symbols]

[0070] 1··ECU 2. Camera 3. DCM 5. Server 10. Vehicle 11. Image acquisition section 12...Speed ​​acquisition section 13. Image Processing Department 14...Depth estimation part 16 Recording data storage section 51 Camera information storage section 52··Image Processing Department 53 Learning processing unit 55 Learning data storage unit 100 Depth Estimation System 101, 501·CPU 102, 502...Memory 103, 503...Auxiliary storage device 104 Interface 504 Communications Department

Claims

[Claim 1] A method for detecting a depth of an object in an image comprising: learning a machine learning model for estimating a depth of an object in an image; and estimating a depth of the object in an input image using the learned machine learning model; The control unit is removing areas with little change between images included in the video captured by a camera mounted on a moving object from at least two consecutive images in the video captured by the camera to obtain a first image and a second image following the first image; generating a third image following the first image from the first image, a moving distance of the moving object between the times when the two consecutive images were captured, the moving distance being calculated based on the speed of the moving object and the shooting rate of the camera, and an estimated depth of the object in the first image obtained by inputting the first image into the machine learning model; adjusting parameters of the machine learning model based on a comparison between the second image and the third image; The learning is performed by Information processing device.

Images