Autoencoder training method and system, and deep image generation method

Abstract

The application provides a self-encoder training method, system and a deep image generation method. The self-encoder training method comprises: obtaining a stereo image in vehicle operation, wherein the stereo image comprises a left image and a right image; generating a stereo disparity map according to the left image; generating a predicted right image according to the left image and the stereo disparity map; and calculating a first mean square error between the predicted right image and the right image. The self-encoder can be trained according to the image collected by a stereo camera, and the trained self-encoder can output a depth image according to the image of a monocular camera.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a self-encoder training method and system and a deep image generation method. BACKGROUND

[0002] With the advancement of technology, functions such as assisted driving or autonomous driving are becoming more and more popular on vehicles. When implementing functions such as autonomous driving or partial assisted driving, the distance between the vehicle and pedestrians, vehicles, and obstacles needs to be determined through deep images, and corresponding operations are performed through the distance between the vehicle and the pedestrians, vehicles, and obstacles.

[0003] Currently, deep images are usually obtained using a stereo camera, and then the left and right images of the stereo camera are used to obtain the deep image. However, the use of a stereo camera increases the additional cost and has poor applicability. SUMMARY

[0004] Therefore, it is necessary to provide a self-encoder training method and system and a deep image generation method. The self-encoder can be trained according to the images collected by a stereo camera, and the trained self-encoder can output a depth image according to the images of a monocular camera.

[0005] In a first aspect, an embodiment of the present application provides a self-encoder training method, comprising:

[0006] obtaining stereo images in the running of a vehicle, the stereo images comprising left images and right images;

[0007] generating a stereo disparity map according to the left images;

[0008] generating a predicted right image according to the left images and the stereo disparity map;

[0009] calculating a first mean square error between the predicted right image and the right image.

[0010] According to some embodiments of the present application, the stereo images further comprise time sequence information and time sequence left images;

[0011] The self-encoder training method further comprises:

[0012] generating a time sequence disparity map according to the left images and the time sequence information;

[0013] generating a predicted left image according to the left images and the time sequence disparity map;

[0014] calculating a second mean square error between the predicted left image and the time sequence left image.

[0015] According to some embodiments of the present application, the method further comprises:

[0016] setting a first mean square error threshold;

[0017] determining whether the first mean square error is lower than the first mean square error threshold;

[0018] if the first mean square error is lower than the first mean square error threshold, determining that the auto-encoder training is completed.

[0019] According to some embodiments of the present application, the method further comprises:

[0020] setting a second mean square error threshold;

[0021] determining whether the second mean square error is lower than the second mean square error threshold;

[0022] if the second mean square error is lower than the second mean square error threshold, determining that the auto-encoder training is completed.

[0023] According to some embodiments of the present application, the method further comprises:

[0024] if the first mean square error is lower than the first mean square error threshold or the second mean square error is lower than the second mean square error threshold, determining that the auto-encoder training is completed.

[0025] According to some embodiments of the present application, the method further comprises:

[0026] generating the stereo disparity map according to the right image;

[0027] generating a predicted left image according to the right image and the stereo disparity map.

[0028] According to some embodiments of the present application, the method further comprises:

[0029] generating the temporal disparity map according to the right image;

[0030] generating a predicted right image according to the right image and the temporal disparity map.

[0031] In a second aspect, embodiments of the present application provide an auto-encoder training system, comprising:

[0032] an image acquisition module, configured to acquire a stereo image in vehicle operation, the stereo image comprising a left image and a right image;

[0033] an auto-encoder, configured to generate a stereo disparity map according to the left image;

[0034] an output module, configured to generate a predicted right image according to the left image and the stereo disparity map;

[0035] a calculation module configured to calculate a first mean square error between the predicted right image and the right image.

[0036] According to some embodiments of the present application, the stereoscopic image further comprises timing information and a timing left image.

[0037] The autoencoder is further configured to generate a timing disparity map according to the left image and the timing information.

[0038] The output module is further configured to generate a predicted left image according to the left image and the timing disparity map.

[0039] The calculation module is further configured to calculate a second mean square error between the predicted left image and the timing left image.

[0040] In a third aspect, embodiments of the present application provide a depth image generation method, applied to a vehicle, comprising:

[0041] acquiring an image collected by a monocular camera;

[0042] inputting the image collected by the monocular camera to an autoencoder derived from the autoencoder training method of the first aspect;

[0043] generating a stereoscopic disparity map and a timing disparity map;

[0044] fusing the stereoscopic disparity map and the timing disparity map to generate a fused disparity map;

[0045] generating a depth image according to the fused disparity map.

[0046] The autoencoder training method, system and depth image generation method provided by the embodiments of the present application can train an autoencoder according to images collected by a stereo camera, and the trained autoencoder can output a depth image according to images of a monocular camera. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a schematic diagram of an autoencoder training system module provided by an embodiment of the present application.

[0048] Figure 2 is a schematic diagram of an autoencoder training method flow provided by an embodiment of the present application.

[0049] Figure 3 is a schematic diagram of an autoencoder training method flow provided by another embodiment of the present application.

[0050] Figure 4 is a schematic diagram of a depth image generation system module provided by an embodiment of the present application.

[0051] Figure 5 is a flowchart of a method for generating a depth image according to an embodiment of the present application.

[0052] Main element symbol explanation

[0053] Autoencoder training system 10

[0054] Image acquisition module 100; 100a

[0055] Autoencoder 200; 200a

[0056] Output module 300

[0057] Computing module 400

[0058] Fusion module 500

[0059] Generation module 600

[0060] Depth image generation system 20 DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application.

[0062] It should be noted that “at least one” in the embodiments of the present application means one or more, and more means two or more. Unless otherwise defined, all technical and scientific terms used in the present application have the same meanings as those commonly understood by those skilled in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing the specific embodiments of the present application, and are not intended to limit the present application.

[0063] It should be noted that “first”, “second”, and the like in the embodiments of the present application are only used for the purpose of distinguishing description, and cannot be understood as indicating or implying relative importance, nor can be understood as indicating or implying sequence. The features limited by “first”, “second” can explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the words “exemplary” or “for example” are used to mean as an example, illustration, or description. Any embodiment or design scheme described as “exemplary” or “for example” in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the words “exemplary” or “for example” are used to present the relevant concept in a specific manner.

[0064] Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0065] With the advancement of technology, functions such as assisted driving or automatic driving are becoming more and more popular on vehicles. When implementing functions such as automatic driving or partial assisted driving, the distance of pedestrians, vehicles and obstacles from the vehicle needs to be determined through depth images, and corresponding operations are performed through the distance from the vehicle.

[0066] Currently, depth images are usually obtained using a stereo camera, and then depth images are obtained according to the left and right images of the stereo camera. Setting a stereo camera increases additional costs and has poor applicability.

[0067] Therefore, the present application provides a self-encoder training method and system and a depth image generation method. The self-encoder can be trained according to the images collected by the stereo camera (Stereo Camera), and the trained self-encoder can output a depth image according to the images of a monocular camera.

[0068] Some embodiments of the application will be described in detail below with reference to the accompanying drawings. In the case of no conflict, the embodiments described below and the features in the embodiments can be combined with each other.

[0069] Figure 1 is a self-encoder training system 10 module schematic diagram provided by an embodiment of the present application. The self-encoder training system 10 includes the following parts: an image acquisition module 100, a self-encoder 200, an output module 300 and a calculation module 400.

[0070] It can be understood that the image acquisition module 100 includes a stereo camera (Stereo Camera). The self-encoder training system 10 can perform model training through the stereo images collected by the image acquisition module 100. The stereo camera has a binocular camera, i.e. left and right two cameras, which can obtain images with different left and right viewing angles (hereinafter referred to as left image and right image).

[0071] In the embodiment of the present application, the image information during the driving of the vehicle can be collected by the image acquisition module 100 first. Exemplarily, the image information during the driving of the vehicle can include image information of the vehicle in different driving scenes, such as closed test road sections, highway road sections and urban road sections, etc. driving scenes, so as to improve the adaptability of the self-encoder training system 10 to different driving scenes, which is not limited in the present application.

[0072] In some embodiments, the image acquisition module 100 will also add a time sequence label to the collected image when collecting the image information during the driving of the vehicle. Exemplarily, the image acquisition module 100 can add a time sequence label t to the image collected at time t, and add a time sequence label t+1 to the image collected at time t+1.

[0073] It can be understood that the image acquisition module 100 transmits the left image captured by the stereo camera to the autoencoder 200. In some embodiments, the autoencoder 200 is an artificial neural network capable of learning an efficient representation of input data through unsupervised learning. In some embodiments, the autoencoder 200 provided by the embodiments of the present application is a trained autoencoder, which can output a stereo disparity map at time t according to the input left image at time t, and transmit the stereo disparity map to the output module 300.

[0074] It can be understood that the disparity map is the positional deviation of the pixels of the same scene imaged by two cameras. Exemplarily, the corresponding predicted right image can be obtained by performing addition operation on the left image and the corresponding stereo disparity map.

[0075] It can be understood that the left image, the right image and the disparity map in the autoencoder training system 10 are represented by a matrix. Exemplarily, taking the image collected by the image acquisition module 100 as a gray image as an example, the rows of the matrix correspond to the height (in pixels) of the image, the columns of the matrix correspond to the width (in pixels) of the image, and the elements of the matrix correspond to the pixels of the image. The value of the matrix element is the gray value of the pixel. In some embodiments, the image collected by the image acquisition module 100 can also be a binary image or an RGB image, etc., which is not limited by the present application.

[0076] It can be understood that the output module 300 can calculate the predicted right image at time t according to the left image at time t and the stereo disparity map at time t. In some embodiments, the output module 300 can perform addition operation on the left image matrix at time t and the stereo disparity map matrix at time t to obtain the predicted right image matrix at time t. The output module 300 outputs the predicted right image matrix at time t to the calculation module 400.

[0077] It can be understood that the image acquisition module 100 transmits the right image captured by the stereo camera to the calculation module 400. It can be understood that the calculation module 400 performs mean square error (MSE) operation on the predicted right image matrix at time t and the actual right image matrix at time t to calculate the first MSE between the predicted right image matrix at time t and the actual right image matrix at time t. The first MSE can be calculated by formula (1).

[0078]

[0079] Wherein, n is the number of numbers in the image matrix, y i is the value of the pixel point in the predicted right image at time t, is the value of the pixel point in the actual right image at time t.

[0080] It can be understood that the smaller the value of the first MSE is, the smaller the difference between the predicted right image matrix at time t and the actual right image matrix at time t is. When the value of the first MSE is 0, it means that the predicted right image matrix at time t is the same as the actual right image matrix at time t.

[0081] In some embodiments, the first MSE threshold value can be set in the calculation module 400. If the first MSE value calculated by the calculation module 400 is lower than the first MSE threshold value, it means that the training of the autoencoder is completed.

[0082] In some embodiments, the calculation module 400 can also be set to determine that the training of the autoencoder is completed when the first MSE is continuously lower than the first MSE threshold value. For example, if the first MSE value calculated by the calculation module 400 is continuously lower than the first MSE threshold value for three times, it means that the training of the autoencoder is completed.

[0083] In some embodiments, the autoencoder 200 outputs the temporal disparity map of the left image at time t and the left image at time t+1 according to the input left image at time t, and transmits the temporal disparity map to the output module 300.

[0084] It can be understood that the output module 300 can calculate the predicted left image at time t+1 according to the left image at time t and the temporal disparity map at time t+1. In some embodiments, the output module 300 can add the left image matrix at time t and the matrix at time t+1 to obtain the predicted left image matrix at time t+1. The output module 300 transmits the predicted left image matrix at time t+1 to the calculation module 400.

[0085] It can be understood that the image acquisition module 100 transmits the actual left image matrix at time t+1 captured by the stereo camera to the calculation module 400. The calculation module 400 calculates the second MSE between the predicted left image matrix at time t+1 and the actual left image matrix at time t+1 by formula (1). It can be understood that the calculation method of the second MSE is the same as that of the first MSE, which will not be repeated here.

[0086] It can be understood that the second MSE threshold value can be set in the calculation module 400, and the setting method of the threshold value is the same as that of the first MSE threshold value, which will not be repeated here.

[0087] It can be understood that the calculation module 400 can determine whether the auto-encoder 200 is trained according to the first MSE threshold and the second MSE threshold. For example, if the first MSE is lower than the first MSE threshold and the second MSE is higher than the second MSE threshold, the calculation module 400 can determine that the auto-encoder 200 is not trained, and the auto-encoder training system 10 continues to train the auto-encoder 200 until the second MSE is lower than the second MSE threshold. For example, if the first MSE is lower than the first MSE threshold or the second MSE is lower than the second MSE threshold, the calculation module 400 can determine that the auto-encoder 200 is trained.

[0088] In some embodiments, the image acquisition module 100 can also input the right image into the auto-encoder 200, and the auto-encoder 200 outputs a stereo disparity map for the right image according to the input right image, and then compares the right image with the left image and calculates the MSE. For details of the calculation method, please refer to the calculation method of the first MSE, which will not be repeated here.

[0089] The auto-encoder training system 10 provided by the embodiments of the present application can generate corresponding stereo disparity maps and time series disparity maps according to the input images of the monocular camera.

[0090] Figure 2 is a flowchart of an auto-encoder training method provided by an embodiment of the present application. The auto-encoder training method at least includes the following steps:

[0091] S100: Acquire stereo images in vehicle operation.

[0092] It can be understood that in step S100, the image acquisition module 100 is used to acquire stereo images in vehicle operation, and the stereo images include left images and right images. For details of the acquisition method, please refer to Figure 1 , which will not be repeated here.

[0093] S200: Generate a stereo disparity map according to the left image.

[0094] It can be understood that in step S200, the auto-encoder 200 is used to generate a stereo disparity map according to the left image in the stereo image. For details of the generation method, please refer to Figure 1 , which will not be repeated here.

[0095] S300: Generate a predicted right image according to the left image and the stereo disparity map.

[0096] It can be understood that in step S300, the output module 300 is used to generate a predicted right image according to the left image and the stereo disparity map. For details of the generation method, please refer to Figure 1 , which will not be repeated here.

[0097] S400: Calculate a first mean square error between the predicted right image and the right image.

[0098] It can be understood that in step S400, the calculation module 400 is configured to calculate a first mean square error between the predicted right image and the right image, and the calculation manner can be known from Figure 1 , which will not be repeated here.

[0099] Figure 3 is a flowchart of a self-encoder training method provided by another embodiment of the present application. The self-encoder training method further includes the following steps:

[0100] S500: Generating a temporal disparity map according to the left image and the temporal information.

[0101] It can be understood that in step S500, the self-encoder 200 is configured to generate a temporal disparity map according to the left image and the temporal information, and the generation manner can be known from Figure 1 , which will not be repeated here.

[0102] S600: Generating a predicted left image according to the left image and the temporal disparity map.

[0103] It can be understood that in step S600, the output module 300 is configured to generate a predicted left image according to the left image and the temporal disparity map, and the generation manner can be known from Figure 1 , which will not be repeated here.

[0104] S700: Calculate a second mean square error between the predicted left image and the temporal left image.

[0105] It can be understood that in step S700, the calculation module 400 is configured to calculate a second mean square error between the predicted left image and the temporal left image, and the calculation manner can be known from Figure 1 , which will not be repeated here.

[0106] Figure 4 is a schematic diagram of a depth image generation system 20 module provided by an embodiment of the present application. As shown in Figure 4 , the depth image generation system 20 includes an image acquisition module 100a, a self-encoder 200a, a fusion module 500, and a generation module 600.

[0107] It can be understood that the image acquisition module 100a is a monocular camera, which can acquire image information during vehicle driving and transmit the image information to the self-encoder 200a.

[0108] It can be understood that the self-encoder 200a is a self-encoder 200a trained by the self-encoder training system 10 in Figure 1 . The specific training manner can be known from Figure 1 , which will not be repeated here.

[0109] It can be understood that the auto-encoder 200 generates the stereo disparity map and the time-series disparity map according to the image information collected by the image acquisition module 100a, and the specific generation manner is refer to Figure 1 , which will not be repeated here.

[0110] It can be understood that the fusion module 500 performs disparity map fusion processing on the stereo disparity map and the time-series disparity map to generate a fused disparity map. It can be understood that the fusion module 500 can set different weight values for the stereo disparity map and the time-series disparity map when performing disparity map fusion processing, which is not limited by the present application.

[0111] It can be understood that the generation module 600 generates a depth image according to the fused disparity map generated by the fusion module 500. The method of generating a depth image according to a disparity map can be according to the encoding manner in the related art, which is not limited by the present application.

[0112] Figure 5 is a flowchart of a depth image generation method provided by an embodiment of the present application. As shown in Figure 5 , the depth image generation method at least includes the following steps:

[0113] S10: acquiring an image collected by a monocular camera.

[0114] It can be understood that in step S10, the image acquisition module 100a is used to acquire an image collected by a monocular camera, and the acquisition manner is refer to Figure 1 and Figure 4 , which will not be repeated here.

[0115] S20: inputting the image collected by the monocular camera to the trained auto-encoder.

[0116] It can be understood that in step S20, the image acquisition module 100a is used to output the collected image to the auto-encoder 200a.

[0117] S30: generating a stereo disparity map and a time-series disparity map.

[0118] It can be understood that in step S30, the auto-encoder 200a is used to generate a stereo disparity map and a time-series disparity map, and the generation manner is refer to Figure 1 and Figure 4 , which will not be repeated here.

[0119] S40: fusing the stereo disparity map and the time-series disparity map to generate a fused disparity map.

[0120] It can be understood that in step S40, the fusion module 500 is configured to fuse the stereoscopic disparity map and the temporal disparity map to generate a fused disparity map, and the specific fusion manner can be referred to in combination with Figure 4 , which will not be repeated here.

[0121] S50: generating a depth image according to the fused disparity map.

[0122] It can be understood that in step S50, the generation module 600 is configured to generate a depth image according to the fused disparity map, and the specific generation manner can refer to the depth image generation method in the related art, which is not limited herein.

[0123] Those skilled in the art should recognize that the above embodiments are only used to illustrate the present application, and are not used as a limitation to the present application. Any appropriate changes and variations made to the above embodiments within the spirit and principle of the present application fall within the scope of the present application.

Claims

1. An autoencoder training method, characterized in that, include: Acquire a stereoscopic image of the vehicle in operation, the stereoscopic image including a left image, a right image, time sequence information, and a time sequence left image corresponding to the time sequence information; The following steps are performed using the autoencoder: A stereo disparity map is generated based on the left image, and a predicted right image is generated based on the left image and the stereo disparity map. A temporal disparity map is generated based on the left image and the temporal information, and a predicted left image is generated based on the left image and the temporal disparity map. Calculate the first mean square error between the predicted right image and the right image, and calculate the second mean square error between the predicted left image and the time-series left image; Set a first mean square error threshold and a second mean square error threshold; If the first mean square error is lower than the first mean square error threshold, or the second mean square error is lower than the second mean square error threshold, then the autoencoder training is considered complete.

2. An autoencoder training method, characterized in that, include: Acquire a stereoscopic image of the vehicle in operation, the stereoscopic image including a left image, a right image, time sequence information, and a time sequence left image corresponding to the time sequence information; The following steps are performed using the autoencoder: A stereo disparity map is generated based on the right image, and a predicted left image is generated based on the right image and the stereo disparity map. A temporal disparity map is generated based on the right image and the temporal information, and a predicted right image is generated based on the right image and the temporal disparity map. Among them, the first mean square error between the predicted left image and the left image is calculated; Set a first mean squared error threshold. If the first mean squared error is lower than the first mean squared error threshold, then the autoencoder training is considered complete.

3. An autoencoder training system for implementing the method as described in claim 1, characterized in that, include: The image acquisition module is used to acquire stereoscopic images of the vehicle in motion, the stereoscopic images including a left image and a right image; An autoencoder is used to generate a stereo disparity map based on the left image; The output module is used to generate a predicted right image based on the left image and the stereo disparity map; The calculation module is used to calculate the first mean square error between the predicted right image and the right image.

4. The autoencoder training system as described in claim 3, characterized in that, The stereoscopic image also includes time sequence information and a time sequence left image; The autoencoder is also used to generate a temporal disparity map based on the left image and the temporal information; The output module is also used to generate a predicted left image based on the left image and the temporal disparity map; The calculation module is also used to calculate the second mean square error between the predicted left image and the time-series left image.

5. A depth image generation method, applied to automobiles, characterized in that, include: Acquire images captured by a monocular camera; The image captured by the monocular camera is input into the autoencoder trained by the autoencoder training method as described in claim 1 or 2; Generate stereo disparity maps and time-series disparity maps; The stereo disparity map and the temporal disparity map are fused to generate a fused disparity map; A depth image is generated based on the fused disparity map.

Images