An unsupervised monocular depth estimation method and system for moving objects

Abstract

The present invention discloses an unsupervised monocular depth estimation method and system for a moving object, wherein the method comprises: obtaining frame sequence data from a monocular camera; extracting an intermediate frame of the frame sequence data as a reference frame; obtaining motion foreground data and motion optical flow data based on the frame sequence data; inputting the motion foreground data into a depth estimation module to obtain depth sequence data; inputting the motion foreground data into a camera hyperparameter estimation module to obtain camera hyperparameter values; and calculating the motion optical flow data, depth sequence data, and camera hyperparameter values ​​based on a loss function to obtain motion detection data and depth estimation data of the moving object. By extracting a reference frame from the frame sequence, obtaining motion foreground data and motion optical flow data through the frame sequence data, and then obtaining depth sequence data and camera hyperparameter values, and finally obtaining monocular depth estimation data through comprehensive calculation, the effect of achieving accurate monocular depth estimation with a simple system is achieved.

Description

Technical Field

[0001] The present invention relates to the field of monocular depth estimation technology for autonomous driving, and in particular to an unsupervised monocular depth estimation method and system for moving objects. Background Art

[0002] Autonomous driving, the most promising technology in the current mobility industry and a key application scenario for artificial intelligence, will undoubtedly significantly boost social productivity, optimize road conditions, and improve traffic safety. The realization of autonomous driving relies on breakthroughs in computer vision technology. As the most important perception method, vision technology must identify roads, moving objects, and surrounding scenes, and return their real-world 3D coordinate information. However, identifying moving objects and extracting their 3D coordinate information is particularly challenging.

[0003] Methods for stereoscopic object recognition are categorized into supervised and unsupervised approaches. Supervised approaches involve manually assigned labels for videos and images. This approach offers the advantage of being completely consistent with human perception. However, due to the ever-changing variety of moving objects in the real world, supervised approaches require a tremendous amount of manpower to label the data, a burden that industrial applications cannot sustain. Therefore, the development of unsupervised visual recognition technology that does not rely on human labeling is considered a crucial breakthrough in the field of artificial intelligence. Currently, methods for visual depth estimation of moving objects are categorized into binocular, multi-camera, and monocular approaches based on the number of cameras used. Compared to the first two approaches, monocular vision lacks spatial information and is the most challenging to implement. However, monocular vision offers high research value due to its ease of use, low cost, and proximity to real-world application scenarios.

[0004] The existing unsupervised monocular depth estimation method generally takes a continuous image sequence as input and outputs the predicted depth. This method is affected by a large number of moving targets, and its accuracy will be affected. It also relies on camera parameter calibration, resulting in a more complex processing flow and affecting the processing efficiency of depth estimation. Summary of the Invention

[0005] In order to overcome the technical defects of the above-mentioned existing unsupervised monocular depth estimation methods, such as being unsuitable for a large number of moving targets, relying on camera parameter calibration and having a complex processing flow, the present invention provides an unsupervised monocular depth estimation method and system for moving objects.

[0006] In order to solve the above problems, the present invention is implemented according to the following technical solutions:

[0007] In a first aspect, the present invention discloses an unsupervised monocular depth estimation method for a moving object, which specifically comprises the following steps:

[0008] Get frame sequence data from a monocular camera;

[0009] Extracting the middle frame of the frame sequence data as a reference frame;

[0010] Acquire motion foreground data and motion optical flow data based on frame sequence data;

[0011] Input the motion foreground data into the depth estimation module to obtain depth sequence data;

[0012] Input the motion foreground data into the camera hyperparameter estimation module to obtain the camera hyperparameter values;

[0013] Based on the loss function, the motion optical flow data, depth sequence data and camera hyperparameter values ​​are calculated to obtain motion detection data and depth estimation data of moving objects.

[0014] As a first preferred implementation manner for the first aspect, acquiring motion foreground data and motion optical flow data based on frame sequence data specifically includes:

[0015] The frame sequence data is transmitted to the optical flow module, and the optical flow sequence data is obtained after processing;

[0016] The optical flow sequence data is sent to the motion detection module, and after processing, the motion mask sequence data is obtained;

[0017] After correspondingly multiplying the frame sequence data and the motion mask sequence data, the motion foreground data is obtained;

[0018] After the optical flow sequence data and the motion mask sequence data are correspondingly multiplied, the motion optical flow data is obtained.

[0019] As a second preferred implementation manner for the first aspect, the calculation of the motion optical flow data, the depth sequence data, and the camera hyperparameter values ​​based on the loss function specifically includes:

[0020] The motion optical flow data, depth sequence data and camera hyperparameter values ​​are input into the reconstruction module respectively, and the reconstructed image is obtained after reconstruction;

[0021] Based on the loss function, the moving objects in the reference frame and the reconstructed image are calculated to obtain motion detection data and depth estimation data of the moving objects.

[0022] As a third preferred implementation manner for the first aspect: the loss function is specifically: Loss = |I 预测 -M L / / 2 *I L / / 2 | 2 ;

[0023] Wherein, the I 预测 To reconstruct the picture, the I L / / 2 is the reference frame, the M L / / 2is a reference motion mask, which is an intermediate mask of the motion mask sequence data.

[0024] As a fourth preferred implementation manner for the first aspect, obtaining frame sequence data of a monocular camera specifically includes:

[0025] Obtain video data with moving objects captured by a monocular camera;

[0026] Decoding video data;

[0027] Get preset length information;

[0028] The video data that meets the preset length is intercepted from the decoded video data to form frame sequence data.

[0029] In a second aspect, the present invention further discloses an unsupervised monocular depth estimation system for a moving object, comprising:

[0030] A data acquisition module is used to acquire frame sequence data from a monocular camera;

[0031] A data extraction module, which is used to extract an intermediate frame of the frame sequence data as a reference frame;

[0032] A data processing module, which is used to obtain motion foreground data and motion optical flow data based on frame sequence data;

[0033] A depth estimation module, which is used to input the motion foreground data into the depth estimation module to obtain depth sequence data;

[0034] A hyperparameter estimation module is used to input the motion foreground data into the camera hyperparameter estimation module to obtain camera hyperparameter values;

[0035] A comprehensive calculation module is used to calculate motion optical flow data, depth sequence data and camera hyperparameter values ​​based on the loss function to obtain motion detection data and depth estimation data of moving objects.

[0036] As a first preferred implementation manner for the second aspect, the data processing module specifically includes:

[0037] The optical flow data unit is used to transmit the frame sequence data to the optical flow module, and obtain the optical flow sequence data after processing;

[0038] A motion detection unit, which is used to transmit the optical flow sequence data to the motion detection module and obtain motion mask sequence data after processing;

[0039] A motion foreground unit, configured to obtain motion foreground data by correspondingly multiplying the frame sequence data with the motion mask sequence data;

[0040] The motion optical flow unit is used to obtain motion optical flow data by correspondingly multiplying the optical flow sequence data with the motion mask sequence data.

[0041] As a second preferred implementation manner for the second aspect, the comprehensive calculation module specifically includes:

[0042] The image reconstruction unit is used to input motion optical flow data, depth sequence data and camera hyperparameter values ​​into the reconstruction module respectively, and obtain a reconstructed image after reconstruction;

[0043] The motion detection and depth estimation unit is used to calculate the moving object and the reconstructed image in the reference frame based on the loss function to obtain motion detection data and depth estimation data of the moving object.

[0044] As a third preferred implementation for the second aspect:

[0045] The loss function is specifically: Loss = |I 预测 -M L / / 2 *I L / / 2 | 2 ;

[0046] Wherein, the I 预测 To reconstruct the picture, the I L / / 2 is the reference frame, the M L / / 2 is a reference motion mask, which is an intermediate mask of the motion mask sequence data.

[0047] As a fourth preferred implementation manner for the second aspect, when the data acquisition module is running, specifically performs:

[0048] Obtain video data with moving objects captured by a monocular camera;

[0049] Decoding video data;

[0050] Get preset length information;

[0051] The video data that meets the preset length is intercepted from the decoded video data to form frame sequence data.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] The unsupervised monocular depth estimation method for moving objects described in the present invention creatively uses monocular video as input, so that the method does not need to calibrate camera parameters, and can embed camera parameters into the modeling process in an unsupervised manner, thereby greatly simplifying the processing flow of monocular depth estimation. On the other hand, the present invention detects moving objects in the scene by generating optical flow without relying on labels, thereby improving the robustness of the algorithm while ensuring detection accuracy. In addition, the present invention detects moving objects based on optical flow and combines parameter-free monocular depth estimation without relying on labels, and the accuracy of the two predictions is significantly improved. The present method provides a method for motion detection and depth estimation for a monocular camera that can be applied to situations where there are a large number of moving targets, and the overall system can remain simple, while the detection accuracy is improved, thereby enabling monocular vision to be better applied in autonomous driving to reduce costs and improve ease of use. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, wherein:

[0055] Figure 1 1 is a flow chart of the unsupervised monocular depth estimation method for a moving object according to the present invention;

[0056] Figure 2 Schematic diagram of the structure of the unsupervised monocular depth estimation system for moving objects of the present invention;

[0057] The specific contents of the drawings of the present invention are explained in detail in the following specific embodiments. DETAILED DESCRIPTION

[0058] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0059] As used herein, the term "including" and its variations represent open inclusion, i.e., "including but not limited to." Unless otherwise stated, the term "or" means "and / or." The term "based on" means "based at least in part on." The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment." The term "another embodiment" means "at least one additional embodiment." The terms "first," "second," etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0060] The access device and server can be connected directly or indirectly via wired or wireless communication. The access device can be a terminal or a server. The target application is running on the access device. The target application is an application program capable of initiating data requests to the server, such as a social networking application, a payment application, or a gaming application. The server can be an application server providing services for the target application, or a proxy server distinct from the application server corresponding to the target application. The server is responsible for identifying whether each access device is a malicious device and intercepting data packets from malicious devices. When the server is a proxy server, the proxy server forwards data packets that are not from malicious devices to the application server. The terminal can be a desktop terminal or a mobile terminal. Mobile terminals can be, but are not limited to, smartphones, tablets, laptops, desktop computers, smart speakers, smart watches, etc. The server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0061] Example 1

[0062] like Figure 1 As shown, in the first aspect, the present invention discloses an unsupervised monocular depth estimation method for moving objects, which requires at least one monocular camera, a vehicle with autonomous driving function, and a processing system to implement. The processing system is a system that can receive and process video data from the monocular camera, and can perform motion detection and depth estimation of moving objects based on video processing based on the video data, thereby assisting the vehicle in more accurate and reliable autonomous driving. The embodiment of the present invention specifically includes the following steps:

[0063] Step S1: Obtain frame sequence data from a monocular camera.

[0064] Specifically, the monocular camera starts and collects video data, the processing system obtains the video data with moving objects collected by the monocular camera, and then decodes the video data. The system downloads and updates to obtain the length information of the preset frame sequence data. The processing system intercepts the video segment that meets the preset length from the decoded video data to form frame sequence data. In the embodiment of the present invention, the frame sequence data is represented as {I1, ..I L}.

[0065] Step S2: extracting the middle frame of the frame sequence data as a reference frame.

[0066] Specifically, the processing system obtains the frame sequence data {I1, ..I LExtract the intermediate frame I L / / 2 as a reference frame.

[0067] Step S3: Acquire motion foreground data and motion optical flow data based on the frame sequence data.

[0068] Specifically, the processing system converts the frame sequence data {I1, ..I L} is sent to the optical flow module (FlowModule), and after processing, the optical flow sequence data {F1..F L-1};

[0069] The processing system converts the optical flow sequence data {F1..F L-1} is sent to the motion detection module (MotionModule), and after processing, the motion mask sequence data {M1..M L-1};

[0070] The processing system converts the frame sequence data {I1, ..I L} and motion mask sequence data {M1..M L-1}After corresponding multiplication, the motion foreground data {O1..O L-1}, the corresponding multiplication is to multiply the values ​​with the same subscript in the two sequences in sequence;

[0071] The processing system converts the optical flow sequence data {F1..F L-1} and motion mask sequence data {M1..M L-1}After corresponding multiplication, the motion optical flow data {Q1..Q L-1}. Motion optical flow data {Q1..Q L-1} is used to display the changing trend of motion optical flow in frame sequence data, so as to facilitate subsequent calculation and obtain reconstructed images.

[0072] Step S4: Convert the motion foreground data {O1..O L-1} is input to the depth estimation module (DepthModule) to obtain the depth sequence data {D1..D L-1}.

[0073] Specifically, the processing system starts the depth estimation module (DepthModule) and converts the motion foreground data {O1..O L-1} is input into the depth estimation module (DepthModule) to perform depth estimation and obtain depth sequence data {D1..D L-1}, depth sequence data {D1..D L-1} and the data with the same subscript as the frame sequence data {I1, ..I L} one-to-one correspondence is used to show the depth of the moving object and the monocular camera in the frame sequence data, as well as the depth change trend, so as to facilitate subsequent calculations to obtain reconstructed images.

[0074] Step S5: Convert the motion foreground data {O1..O L-1} Input to the camera hyperparameter estimation module (CamModule) to obtain the camera hyperparameter value (P cam ).

[0075] Step S6: Based on the loss function (Loss) the motion optical flow data {Q1..Q L-1}、Depth sequence data {D1..D L-1} and camera hyperparameter value P cam Calculations are performed to obtain motion detection data and depth estimation data of moving objects.

[0076] Among them, based on the loss function (Loss) of motion optical flow data {Q1..Q L-1}、Depth sequence data {D1..D L-1} and camera hyperparameter value P cam Calculation is performed, specifically including: converting motion optical flow data {Q1..Q L-1}、Depth sequence data {D1..D L-1} and camera hyperparameter value P cam Input them into the reconstruction module respectively, and reconstruct the image (I 预测 ); Based on the loss function (Loss), the moving objects in the reference frame and the reconstructed image (I 预测 ) is calculated to obtain the motion detection data and depth estimation data of the moving object. The loss function is specifically: Loss = |I 预测 -M L / / 2 *I L / / 2 | 2 ;

[0077] Wherein, the I 预测 To reconstruct the picture, the I L / / 2 is the reference frame, the M L / / 2 is a reference motion mask, which is an intermediate mask of the motion mask sequence data.

[0078] The present invention uses motion optical flow data {Q1..Q L-1}、Depth sequence data {D1..D L-1} and camera hyperparameter value P cam Perform comprehensive dimensional reconstruction to obtain reliable reconstructed images (I 预测), the reconstructed image is an image that does not exist in the frame sequence data and is used as a prediction for the subsequent image. It is obtained by integrating the image information in the frame sequence data to make a reliable prediction. The reconstructed image can make motion prediction more intuitive, thereby significantly improving the prediction accuracy of the moving object of the monocular camera. Based on the depth estimation data, it is possible to detect the spatial distance relationship between the moving object and the monocular camera in the video data collected by the monocular camera; based on the motion detection data, it is possible to predict the motion trend of the moving object in the video data collected by the monocular camera, thereby predicting the spatial relationship between the autonomous driving vehicle and external objects, and conducting risk warnings to assist the autonomous driving system in making autonomous driving operation decisions, so that the more cost-effective monocular camera can be better adapted to the field of autonomous driving.

[0079] In summary, the unsupervised monocular depth estimation method for moving objects disclosed in the embodiments of the present invention creatively uses monocular video as input, so that the method does not need to calibrate camera parameters and can embed camera parameters into the modeling process in an unsupervised manner, thereby greatly simplifying the processing flow of monocular depth estimation.

[0080] On the other hand, the present invention detects moving objects in the scene by generating optical flow without relying on labels, thereby improving the robustness of the algorithm while ensuring detection accuracy, making the detection more stable and reliable.

[0081] Furthermore, without relying on labels, the present invention significantly improves the accuracy of both predictions by detecting moving objects based on optical flow and combining it with parameter-free monocular depth estimation. This method provides a method for motion detection and depth estimation for monocular cameras that is applicable to situations where there are a large number of moving targets, while maintaining the overall simplicity of the system and improving detection accuracy. This allows monocular vision to be better applied in autonomous driving, reducing costs and improving ease of use.

[0082] The present invention extracts reference frames from the frame sequence, obtains motion foreground data and motion optical flow data through the frame sequence data, and then obtains depth sequence data and camera hyperparameter values. Finally, the monocular depth estimation data is obtained through comprehensive calculation, thereby achieving accurate monocular depth estimation with a simple system.

[0083] For other steps of the unsupervised monocular depth estimation method for moving objects described in this embodiment, refer to the prior art.

[0084] Example 2

[0085] like Figure 2As shown, in a second aspect, the present invention further discloses an unsupervised monocular depth estimation system for a moving object, comprising a data acquisition module M1, a data extraction module M2, a data processing module M3, a depth estimation module M4, a hyperparameter estimation module M5, and a comprehensive calculation module M6, wherein:

[0086] The data acquisition module M1 is used to obtain frame sequence data from a monocular camera. Specifically, it performs the following steps: obtaining video data with moving objects captured by the monocular camera; decoding the video data; obtaining preset length information; and intercepting the video that meets the preset length from the decoded video data to form frame sequence data.

[0087] The data extraction module M2 is used to extract the middle frame of the frame sequence data as a reference frame;

[0088] The data processing module M3 is used to obtain motion foreground data and motion optical flow data based on the frame sequence data;

[0089] The depth estimation module M4 is used to input the motion foreground data into the depth estimation module to obtain depth sequence data;

[0090] The hyperparameter estimation module M5 is used to input the motion foreground data into the camera hyperparameter estimation module to obtain the camera hyperparameter value;

[0091] The comprehensive calculation module M6 is used to calculate the motion optical flow data, depth sequence data and camera hyperparameter values ​​based on the loss function to obtain motion detection data and depth estimation data of the moving object.

[0092] As an optional implementation of the embodiment of the present invention, further, the data processing module M3 specifically includes:

[0093] The optical flow data unit is used to transmit the frame sequence data to the optical flow module, and obtain the optical flow sequence data after processing;

[0094] A motion detection unit, which is used to transmit the optical flow sequence data to the motion detection module and obtain motion mask sequence data after processing;

[0095] A motion foreground unit, configured to obtain motion foreground data by correspondingly multiplying the frame sequence data with the motion mask sequence data;

[0096] The motion optical flow unit is used to obtain motion optical flow data by correspondingly multiplying the optical flow sequence data with the motion mask sequence data.

[0097] The comprehensive calculation module M6 specifically includes:

[0098] The image reconstruction unit is used to input motion optical flow data, depth sequence data and camera hyperparameter values ​​into the reconstruction module respectively, and obtain a reconstructed image after reconstruction;

[0099] The motion detection and depth estimation unit is used to calculate the moving object in the reference frame and the reconstructed image based on the loss function to obtain the motion detection data and depth estimation data of the moving object. The loss function is specifically: Loss = |I 预测 -M L / / 2 *I L / / 2 | 2 , wherein said I 预测 To reconstruct the picture, the I L / / 2 is the reference frame, the M L / / 2 is a reference motion mask, which is an intermediate mask of the motion mask sequence data.

[0100] To sum up, when the unsupervised monocular depth estimation system for moving objects described in the embodiment of the present invention is running, it can implement all the steps of the unsupervised monocular depth estimation method for moving objects described in Example 1, thereby realizing information collection based on a monocular camera, and achieving the effect of obtaining corresponding motion detection data and depth estimation data, thereby providing accurate data for autonomous driving, so as to better apply the monocular vision mode to the field of autonomous driving, and reduce the overall cost of autonomous driving.

[0101] For other structures of the unsupervised monocular depth estimation system for moving objects described in this embodiment, refer to the prior art.

[0102] Example 3

[0103] The present invention also discloses an electronic device, at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor. When the at least one processor executes the instructions, it specifically implements the following steps: obtaining frame sequence data from a monocular camera; extracting the intermediate frame of the frame sequence data as a reference frame; obtaining motion foreground data and motion optical flow data based on the frame sequence data; inputting the motion foreground data into a depth estimation module to obtain depth sequence data; inputting the motion foreground data into a camera hyperparameter estimation module to obtain camera hyperparameter values; calculating the motion optical flow data, depth sequence data and camera hyperparameter values ​​based on a loss function to obtain motion detection data and depth estimation data of a moving object.

[0104] Example 4

[0105] The present invention also discloses a storage medium storing a computer program. When the computer program is executed by a processor, the following steps are specifically implemented: obtaining frame sequence data from a monocular camera; extracting an intermediate frame of the frame sequence data as a reference frame; obtaining motion foreground data and motion optical flow data based on the frame sequence data; inputting the motion foreground data into a depth estimation module to obtain depth sequence data; inputting the motion foreground data into a camera hyperparameter estimation module to obtain camera hyperparameter values; and calculating the motion optical flow data, depth sequence data and camera hyperparameter values ​​based on a loss function to obtain motion detection data and depth estimation data of a moving object.

[0106] The present disclosure may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0107] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0108] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0109] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Java, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0110] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0111] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0112] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0113] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the above-mentioned module, program segment or instruction includes one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a special hardware-based system that performs the prescribed function or action, or can be implemented with a combination of special hardware and computer instructions.

[0114] The embodiments of the present disclosure have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons skilled in the art to understand the embodiments disclosed herein.

Claims

1. An unsupervised monocular depth estimation method for moving objects, characterized in that: The specific steps include: Get frame sequence data from a monocular camera; Extracting the middle frame of the frame sequence data as a reference frame; Acquire motion foreground data and motion optical flow data based on frame sequence data; Input the motion foreground data into the depth estimation module to obtain depth sequence data; Input the motion foreground data into the camera hyperparameter estimation module to obtain the camera hyperparameter values; Calculate motion optical flow data, depth sequence data, and camera hyperparameter values ​​based on the loss function to obtain motion detection data and depth estimation data of moving objects; The calculation of motion optical flow data, depth sequence data and camera hyperparameter values ​​based on the loss function specifically includes: The motion optical flow data, depth sequence data and camera hyperparameter values ​​are input into the reconstruction module respectively, and the reconstructed image is obtained after reconstruction; Based on the loss function, the moving objects in the reference frame and the reconstructed image are calculated to obtain motion detection data and depth estimation data of the moving objects; The acquiring of motion foreground data and motion optical flow data based on the frame sequence data specifically includes: The frame sequence data is transmitted to the optical flow module, and the optical flow sequence data is obtained after processing; The optical flow sequence data is transmitted to the motion detection module, and after processing, the motion mask sequence data is obtained; After correspondingly multiplying the frame sequence data and the motion mask sequence data, the motion foreground data is obtained; After the optical flow sequence data and the motion mask sequence data are correspondingly multiplied, the motion optical flow data is obtained.

2. The unsupervised monocular depth estimation method for a moving object according to claim 1, wherein: The loss function is specifically: Loss=|I 预测 -M L / / 2 *I L / / 2 | 2 ; Wherein, the I 预测 To reconstruct the picture, the I L / / 2 is the reference frame, the M L / / 2 is a reference motion mask, which is an intermediate mask of the motion mask sequence data.

3. The unsupervised monocular depth estimation method for a moving object according to claim 1, wherein: The acquiring of frame sequence data from the monocular camera specifically includes: Obtain video data with moving objects captured by a monocular camera; Decoding video data; Get preset length information; The video data that meets the preset length is intercepted from the decoded video data to form frame sequence data.

4. An unsupervised monocular depth estimation system for moving objects, characterized in that include: A data acquisition module is used to acquire frame sequence data from a monocular camera; A data extraction module, which is used to extract an intermediate frame of the frame sequence data as a reference frame; A data processing module, which is used to obtain motion foreground data and motion optical flow data based on frame sequence data; A depth estimation module, which is used to input the motion foreground data into the depth estimation module to obtain depth sequence data; A hyperparameter estimation module is used to input the motion foreground data into the camera hyperparameter estimation module to obtain camera hyperparameter values; A comprehensive calculation module is used to calculate motion optical flow data, depth sequence data and camera hyperparameter values ​​based on a loss function to obtain motion detection data and depth estimation data of moving objects; The comprehensive calculation module specifically includes: The image reconstruction unit is used to input motion optical flow data, depth sequence data and camera hyperparameter values ​​into the reconstruction module respectively, and obtain a reconstructed image after reconstruction; A motion detection and depth estimation unit, which is used to calculate the moving object in the reference frame and the reconstructed image based on the loss function to obtain motion detection data and depth estimation data of the moving object; The data processing module specifically includes: The optical flow data unit is used to transmit the frame sequence data to the optical flow module, and obtain the optical flow sequence data after processing; A motion detection unit, which is used to transmit the optical flow sequence data to the motion detection module and obtain motion mask sequence data after processing; A motion foreground unit, configured to obtain motion foreground data by correspondingly multiplying the frame sequence data with the motion mask sequence data; The motion optical flow unit is used to obtain motion optical flow data by correspondingly multiplying the optical flow sequence data with the motion mask sequence data.

5. The unsupervised monocular depth estimation system for moving objects according to claim 4, wherein: The loss function is specifically: Loss=|I 预测 -M L / / 2 *I L / / 2 | 2 ; Wherein, the I 预测 To reconstruct the picture, the I L / / 2 is the reference frame, the M L / / 2 is a reference motion mask, which is an intermediate mask of the motion mask sequence data.

6. The unsupervised monocular depth estimation system for moving objects according to claim 4, characterized in that When the data acquisition module is running, it specifically performs the following operations: Obtain video data with moving objects captured by a monocular camera; Decoding video data; Get preset length information; The video data that meets the preset length is intercepted from the decoded video data to form frame sequence data.

Images