Depth image generation method, system, electronic device and readable storage medium

Abstract

The application provides a depth image generation method, system, electronic device and computer readable storage medium. The method comprises the following steps: obtaining a plurality of groups of original binocular images, and establishing a data set containing instance segmentation labels based on the plurality of groups of original binocular images; training a self-encoder network by using the data set containing the instance segmentation labels to obtain a trained self-encoder network; obtaining a monocular image, inputting the monocular image into the trained self-encoder network to obtain a first disparity map; and converting the first disparity map to obtain a depth image corresponding to the monocular image. According to the application, the binocular image and the instance segmentation image are combined as training data for training the self-encoder network, the monocular image is only needed to be input into the self-encoder network, and then the disparity map can be output, the depth image corresponding to the monocular image can be generated through conversion of the disparity map, and the depth estimation precision can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a depth image generation method and system, electronic equipment and computer readable storage medium. BACKGROUND

[0002] With the development of machine learning technology, deep learning networks have been applied to many real-world scenarios, such as autonomous driving. In these applications, the prediction result is very important. If the pixels predicted incorrectly appear in a very important area, for example, a relatively small obstacle, it may cause the avoidance system based on deep learning to malfunction.

[0003] Therefore, monocular depth estimation is a key technology in deep learning that needs to be solved urgently. The specific task of monocular depth estimation is to predict the depth of each pixel point in an image. The image composed of the depth values of each pixel point is also called a depth map. Monocular depth estimation is of great significance for obstacle detection, three-dimensional scene reconstruction, and scene stereo analysis in autonomous driving. SUMMARY

[0004] Therefore, the present application provides a depth image generation method and system, electronic equipment and computer readable storage medium, which can generate a depth image corresponding to a monocular image and improve the depth estimation accuracy.

[0005] In a first aspect, an embodiment of the present application provides a depth image generation method, comprising:

[0006] Obtaining a plurality of groups of original binocular images, and establishing a data set containing instance segmentation labels based on the plurality of groups of original binocular images;

[0007] Training a self-encoder network using the data set containing instance segmentation labels to obtain a trained self-encoder network;

[0008] Obtaining a monocular image, inputting the monocular image into the trained self-encoder network to obtain a first disparity map;

[0009] Converting the first disparity map to obtain a depth image corresponding to the monocular image.

[0010] In an embodiment, each of the plurality of groups of original binocular images comprises a first image and a second image, the training of the auto-encoder network using the dataset containing instance segmentation labels comprises: inputting the first image into the auto-encoder network to obtain a second disparity map; performing data processing on the second disparity map using the instance segmentation labels to obtain a third disparity map; adding the first image and the third disparity map to obtain a predicted image of the second image; calculating an error between the second image and the predicted image using a preset mean square error formula; taking the error as a training loss of the auto-encoder network until the training loss converges, and obtaining a trained auto-encoder network.

[0011] In an embodiment, the data processing on the second disparity map using the instance segmentation labels to obtain a third disparity map comprises: generating an attention image based on the instance segmentation labels, and fusing and correcting the second disparity map based on the attention image to obtain a third disparity map.

[0012] In an embodiment, the preset mean square error formula is:

[0013]

[0014] wherein, MSE is the error between the predicted image and the second image, y i is the i-th pixel point of the second image, is the average pixel value of the second image, and n is the number of pixel points of the second image.

[0015] In an embodiment, the first image is a left image of an original binocular image, and the second image is a right image of the original binocular image.

[0016] In an embodiment, the first image is a right image of an original binocular image, and the second image is a left image of the original binocular image.

[0017] In a second aspect, an embodiment of the present application provides a depth image generation system, comprising: an image acquisition module configured to acquire a plurality of groups of original binocular images, and establish a dataset containing instance segmentation labels based on the plurality of groups of original binocular images; a model training module configured to train an auto-encoder network using the dataset containing instance segmentation labels to obtain a trained auto-encoder network; an image inference module configured to acquire a monocular image, and input the monocular image into the trained auto-encoder network to obtain a first disparity map; and an image conversion module configured to convert the first disparity map to obtain a depth image corresponding to the monocular image.

[0018] In an embodiment, each of the plurality of groups of original binocular images comprises a first image and a second image, and the model training module comprises: an image generation unit configured to input the first image into the auto-encoder network to obtain a second disparity map; a data processing unit configured to perform data processing on the second disparity map using the instance segmentation label to obtain a third disparity map; a first calculation unit configured to add the first image and the third disparity map to obtain a predicted image of the second image; a second calculation unit configured to calculate an error between the second image and the predicted image using a preset mean square error formula; and a model training unit configured to use the error as a training loss of the auto-encoder network until the training loss converges, thereby obtaining a trained auto-encoder network.

[0019] In a third aspect, an embodiment of the present application provides an electronic device, which comprises a processor and a memory, the memory is configured to store instructions, and the processor is configured to invoke the instructions in the memory, so that the electronic device executes the depth image generation method in the first aspect.

[0020] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer instructions, when the computer instructions are executed on an electronic device, the electronic device executes the depth image generation method in the first aspect.

[0021] The depth image generation method, system, electronic device and computer readable storage medium provided by the embodiments of the present application can improve the depth estimation precision by combining binocular images and instance segmentation images as training data for training an auto-encoder network, inputting monocular images into the auto-encoder network, outputting disparity maps, and converting the disparity maps to generate depth images corresponding to the monocular images. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 FIG. 1 is a schematic diagram of a step flow of a depth image generation method according to an embodiment of the present application.

[0023] Figure 2 FIG. 2 is a detailed flow diagram of a step flow of the depth image generation method shown in FIG. 1. Figure 1

[0024] Figure 3 FIG. 3 is a schematic diagram of a functional module of a depth image generation system according to an embodiment of the present application.

[0025] Figure 4 FIG. 4 is a detailed functional diagram of the functional module of the depth image generation system shown in FIG. 3. Figure 3

[0026] MAIN ELEMENT SYMBOL EXPLANATION​​

[0027] Depth image generation system 20

[0028] Image acquisition module 21

[0029] Model training module 22

[0030] Image inference module 23

[0031] Image conversion module 24

[0032] Image generation unit 221

[0033] Data processing unit 222

[0034] First calculation unit 223

[0035] Second calculation unit 224

[0036] Model training unit 225

[0037] The following detailed description will further describe the present application in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, not all the embodiments of the present application.

[0039] It should be noted that “at least one” in the embodiments of the present application means one or more, and more means two or more. Unless otherwise defined, all technical and scientific terms used in the present application have the same meanings as those commonly understood by those skilled in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing the specific embodiments of the present application, and are not intended to limit the present application.

[0040] It should be noted that in the embodiments of the present application, the terms “first”, “second”, etc. are only used for the purpose of distinguishing description, and cannot be understood as indicating or implying relative importance, nor can it be understood as indicating or implying sequence. The features limited by “first”, “second” can explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the terms “exemplary” or “for example” are used to represent as an example, illustration or description. Any embodiment or design scheme described as “exemplary” or “for example” in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the terms “exemplary” or “for example” are intended to present the relevant concept in a specific manner.

[0041] Please refer to Figure 1A schematic flowchart of a step of a depth image generation method provided by an embodiment of the present application is shown.

[0042] Specifically, the depth image generation method can include:

[0043] S100, a plurality of groups of original binocular images are acquired, and a dataset containing instance segmentation labels is established based on the plurality of groups of original binocular images.

[0044] In the embodiment, the original binocular images can be images directly acquired from an image acquisition device, for example, the image acquisition device is a camera, etc., can be pre-stored images acquired from a storage device, for example, the storage device is a U disk, etc., or images acquired from a website server through a network, and the acquisition manner of the original binocular images is not limited in the embodiment.

[0045] In the embodiment, instance segmentation is that multiple instances belonging to the same category in an image need to be distinguished. For example, for a target image, the target image can include multiple persons belonging to the human category, that is, multiple persons are included, and in instance segmentation, the multiple persons need to be distinguished, and each person can obtain a corresponding instance segmentation label. The original binocular images are subjected to instance segmentation processing to obtain instance segmentation labels of the original binocular images. The dataset containing the instance segmentation labels is obtained according to the instance segmentation labels of the original binocular images.

[0046] S200, training a self-encoder network by using the dataset containing the instance segmentation labels to obtain a trained self-encoder network.

[0047] In the embodiment, the self-encoder network is trained by using the dataset containing the instance segmentation labels established based on the plurality of groups of original binocular images, and a better training effect can be obtained, and specific training steps can be referred to the training flowchart shown in Figure 2

[0048] S300, acquiring a monocular image, inputting the monocular image into the trained self-encoder network to obtain a first disparity map.

[0049] In the embodiment, the monocular image can be acquired by a monocular camera, and the monocular image can contain pictures of any scene.

[0050] The sample data of the self-encoder network comes from binocular images, that is, the disparity of the binocular images is used to guide the prediction of the depth of the monocular image, and the accuracy of the depth prediction can be improved.

[0051] S400, converting the first disparity map to obtain a depth image corresponding to the monocular image.

[0052] ​In the embodiment, after the monocular image is acquired, the monocular image is input into the trained auto-encoder network, and the auto-encoder network outputs a first disparity map corresponding to the monocular image instead of a depth image. Therefore, the first disparity map needs to be converted according to the first disparity map output by the auto-encoder network, a lens baseline distance of a monocular camera for capturing the monocular image, and a lens focal length, so as to obtain a depth image corresponding to the monocular image.

[0053] Further combining Figure 2 As shown in the embodiment, each of the plurality of groups of original binocular images includes a first image and a second image, and the auto-encoder network can be trained by the following steps:

[0054] S210, inputting the first image into the auto-encoder network to obtain a second disparity map.

[0055] S220, performing data processing on the second disparity map by using the instance segmentation label to obtain a third disparity map.

[0056] In the embodiment, the attention image can be generated based on the instance segmentation label, and the second disparity map can be fused and corrected based on the attention image as a guide of the attention mechanism, so as to obtain a more fine third disparity map.

[0057] S230, adding the first image and the third disparity map to obtain a predicted image of the second image.

[0058] S240, calculating an error of the second image and the predicted image by using a preset mean square difference formula.

[0059] In the embodiment, the preset mean square difference formula can be:

[0060]

[0061] wherein, MSE is the error of the predicted image and the second image, y i is an i-th pixel point of the second image, is an average pixel value of the second image, and n is a pixel point number of the second image.

[0062] S250, taking the error as a training loss of the auto-encoder network until the training loss converges, so as to obtain the trained auto-encoder network.

[0063] In the embodiment, the first image can be a left image of the original binocular image, and the second image can be a right image of the original binocular image. In other embodiments, the first image can be a right image of the original binocular image, and the second image can be a left image of the original binocular image. That is, the disparity map can be a left-view corresponding disparity map obtained based on a left view as a reference, or a right-view corresponding disparity map obtained based on a right view as a reference, and the embodiment does not limit this.

[0064] The training sample data of the auto-encoder network in the application comes from the original binocular image containing instance segmentation labels, that is, the application utilizes binocular disparity to guide the prediction of monocular image depth. Therefore, the depth image generation method of the application can obtain good training effect without a large amount of labeled data.

[0065] Further combining Figure 3 As shown, based on the same idea as the monocular camera based object detection method in the above embodiment, the application further provides a depth image generation system 20, which can be used to execute the above monocular camera based object detection method. For ease of illustration, only the part related to the embodiment of the application is shown in the functional module schematic diagram of the depth image generation system 20, and those skilled in the art can understand that the structure shown does not constitute a limitation on the depth image generation system 20, and can include more or fewer components than shown, or combine certain components, or different component arrangements.

[0066] Specifically, in the embodiment, the depth image generation system 20 can be applied in an electronic device, and the depth image generation system 20 can include: an image acquisition module 21, configured to acquire a plurality of groups of original binocular images, and establish a data set containing instance segmentation labels based on the plurality of groups of original binocular images; a model training module 22, configured to train an auto-encoder network by using the data set containing instance segmentation labels, and obtain a trained auto-encoder network; an image inference module 23, configured to acquire a monocular image, input the monocular image into the trained auto-encoder network, and obtain a first disparity map; and an image conversion module 24, configured to convert the first disparity map to obtain a depth image corresponding to the monocular image.

[0067] In the embodiment, each of the plurality of groups of original binocular images includes a first image and a second image, and further combining Figure 4As shown, the model training module 22 can include: an image generation unit 221, configured to input the first image into the auto-encoder network to obtain a second disparity map; a data processing unit 222, configured to perform data processing on the second disparity map by using the instance segmentation label to obtain a third disparity map; a first calculation unit 223, configured to add the first image and the third disparity map to obtain a predicted image of the second image; a second calculation unit 224, configured to calculate an error between the second image and the predicted image by using a preset mean square error formula; and a model training unit 225, configured to take the error as a training loss of the auto-encoder network until the training loss converges, so as to obtain the trained auto-encoder network.

[0068] The application further discloses an electronic device, which comprises a processor and a memory. The memory is configured to store instructions, and the processor is configured to invoke the instructions in the memory to enable the electronic device to perform the steps of the depth image generation method in the above embodiments. The electronic device can be a desktop computer, a notebook computer, a palm computer, a cloud server, or the like. The terminal device can interact with a user through a keyboard, a mouse, a remote controller, a touchpad, a voice control device, or the like.

[0069] The application further discloses a computer readable storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the steps of the depth image generation method in the above embodiments. The storage medium can be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, or the like.

[0070] The depth image generation method, system, electronic device, and computer readable storage medium provided by the embodiments of the application can train an auto-encoder network by combining binocular images and instance segmentation images, and can output a disparity map by using a monocular image as an input. The depth image can be obtained by converting the disparity map, the depth estimation precision can be improved, and the problem of poor recovery effect of a detail part in depth reconstruction is solved.

[0071] Those skilled in the art should understand that the above embodiments are only used to illustrate the application, and are not used as a limitation to the application. Any appropriate changes and modifications made to the above embodiments within the spirit and principle of the application shall fall within the scope of protection of the application.

Claims

1. A method of generating a depth image, the method comprising: The method comprises the following steps: obtaining a plurality of groups of original binocular images, and establishing a data set containing instance segmentation labels based on the plurality of groups of original binocular images, wherein each group of original binocular images in the plurality of groups of original binocular images comprises a first image and a second image; training a self-encoder network by using the data set containing instance segmentation labels to obtain a trained self-encoder network, comprising: inputting the first image into the self-encoder network to obtain a second disparity map; performing data processing on the second disparity map by using the instance segmentation labels to obtain a third disparity map; adding the first image and the third disparity map to obtain a predicted image of the second image; calculating an error between the second image and the predicted image by using a preset mean square difference formula; taking the error as a training loss of the self-encoder network until the training loss converges, and obtaining the trained self-encoder network; obtaining a monocular image, and inputting the monocular image into the trained self-encoder network to obtain a first disparity map; converting the first disparity map to obtain a depth image corresponding to the monocular image.

2. The depth image generating method of claim 1, wherein, The data processing on the second disparity map by using the instance segmentation labels to obtain the third disparity map comprises: generating an attention image based on the instance segmentation labels, and fusing and correcting the second disparity map based on the attention image to obtain the third disparity map.

3. The depth image generating method of claim 1, wherein, The preset mean square difference formula is: wherein, is an error of the predicted image and the second image, is an i-th pixel point of the second image, is an average pixel value of the second image, and n is a number of pixel points of the second image.

4. The depth image generation method of claim 1, wherein the first image is a left image of the original binocular image, and the second image is a right image of the original binocular image.

5. The depth image generation method of claim 1, wherein the first image is a right image of the original binocular image, and the second image is a left image of the original binocular image.

6. A depth image generation system characterized by, The method comprises the following steps: an image acquisition module is configured to obtain a plurality of groups of original binocular images, and establish a data set containing instance segmentation labels based on the plurality of groups of original binocular images, wherein each group of original binocular images in the plurality of groups of original binocular images comprises a first image and a second image; a model training module is configured to train a self-encoder network by using the data set containing instance segmentation labels to obtain a trained self-encoder network, and the model training module comprises: an image generation unit configured to input the first image into the self-encoder network to obtain a second disparity map; a data processing unit configured to perform data processing on the second disparity map by using the instance segmentation labels to obtain a third disparity map; a first calculation unit configured to add the first image and the third disparity map to obtain a predicted image of the second image; a second calculation unit configured to calculate an error between the second image and the predicted image by using a preset mean square difference formula; and a model training unit configured to take the error as a training loss of the self-encoder network until the training loss converges, and obtain the trained self-encoder network; an image inference module is configured to obtain a monocular image, and input the monocular image into the trained self-encoder network to obtain a first disparity map. An image conversion module is configured to convert the first disparity map to obtain a depth image corresponding to the monocular image pair.

7. An electronic device, comprising: The electronic device includes a processor and a memory. The memory is configured to store instructions. The processor is configured to invoke the instructions in the memory, so that the electronic device executes the depth image generation method in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions. When the computer instructions run on the electronic device, the electronic device executes the depth image generation method in any one of claims 1 to 5.

Images