Training method of image depth estimation model and processing method of image depth information
By combining environmental images and point cloud data, the depth information and characteristic relationship of each pixel point in the environmental image are determined and optimized, and the problem of low accuracy of depth information in the prior art is solved, and higher accuracy and reliability.
Patent Information
- Application Number
- CN202011546584.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-23
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2040-12-23
AI Technical Summary
In the prior art, the accuracy of optimizing the depth information of the environment based on coordinate calculation method may be low, and there are problems of incomplete depth information or completion errors.
By combining environmental images and environmental point cloud data, the depth information of each pixel point and its characteristic relationship with neighboring pixel points are determined, and optimization processing is performed to improve the accuracy and reliability of depth information.
Accurate completion and optimization of depth information of each pixel in the environmental image is achieved, and the accuracy and reliability of depth information is improved.
Smart Images

Figure CN112634343B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to computers and image processing technology, and in particular to a training method for an image depth estimation model, a method and device for processing image depth information, an autonomous driving vehicle, an electronic device, a program product, and a storage medium, which can be applied to artificial intelligence fields such as autonomous driving, computer vision, and deep learning. Background Art
[0002] The depth information of an image can be understood as the distance from the image collector to each point in the scene corresponding to the image.
[0003] In the prior art, depth information can be widely used in various fields, such as autonomous driving vehicles, robotics, etc. When optimizing the depth information of environmental images, the method usually adopted is to complete the depth information of environmental images based on coordinate calculation.
[0004] However, the accuracy of depth information obtained by optimization based on coordinate calculation and other methods may be low. Summary of the invention
[0005] The present application provides a training method for an image depth estimation model for improving the accuracy of optimized image depth information, a method for processing image depth information, a device, an autonomous driving vehicle, an electronic device, a program product, and a storage medium.
[0006] According to one aspect of the present application, a method for training an image depth estimation model is provided, comprising:
[0007] Inputting the acquired sample environment image, sample edge information of the sample environment image, and sample environment point cloud data into the model to be trained;
[0008] The model to be trained determines the initial depth information of each pixel in the sample environment image and the characteristic relationship between each pixel and its corresponding neighboring pixel according to the sample environment image, the sample edge information of the sample environment image, and the sample environment point cloud data, and optimizes the initial depth information of each pixel according to the characteristic relationship to obtain the optimized depth information of each pixel, and adjusts the parameters of the model to be trained according to the optimized depth information to obtain an image depth estimation model;
[0009] The image depth estimation model is used to obtain optimized depth information of each pixel in the environment image.
[0010] According to another aspect of the present application, a method for processing image depth information is provided, comprising:
[0011] Acquire an environmental image and environmental point cloud data, input the environmental image and the environmental point cloud data into an image depth estimation model trained and generated by the method described in the above embodiment, and output optimized depth information of each pixel in the environmental image.
[0012] According to another aspect of the present application, a training device for an image depth estimation model is provided, comprising:
[0013] An input module, used to input the acquired sample environment image, the sample edge information of the sample environment image, and the sample environment point cloud data into the model to be trained;
[0014] A determination module, configured to determine, by the model to be trained, the initial depth information of each pixel in the sample environment image and the characteristic relationship between each pixel and its corresponding neighboring pixel according to the sample environment image, the sample edge information of the sample environment image, and the sample environment point cloud data;
[0015] An optimization module, used for optimizing the initial depth information of each pixel according to the characteristic relationship to obtain optimized depth information of each pixel;
[0016] An adjustment module, used to adjust the parameters of the model to be trained according to the optimized depth information to obtain an image depth estimation model;
[0017] The image depth estimation model is used to obtain optimized depth information of each pixel in the environment image.
[0018] According to another aspect of the present application, a device for processing image depth information is provided, comprising:
[0019] An acquisition module is used to acquire environmental images and environmental point cloud data;
[0020] An estimation module is used to input the environment image and the environment point cloud data into an image depth estimation model trained and generated based on the method described in the above embodiment, and output optimized depth information of each pixel in the environment image.
[0021] According to another aspect of the present application, there is provided an electronic device, including:
[0022] at least one processor; and
[0023] a memory communicatively connected to the at least one processor; wherein,
[0024] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor so that the at least one processor can perform the method described in the above embodiment.
[0025] According to another aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the above embodiment.
[0026] According to another aspect of the present application, an autonomous driving vehicle is provided, comprising: an image acquisition device, a radar sensor, a controller, and an image depth information processing device as described in the above embodiment; wherein,
[0027] The image acquisition device is used to acquire an image of the environment of the autonomous driving vehicle;
[0028] The radar sensor is used to collect environmental point cloud data of the autonomous driving vehicle;
[0029] The image depth information processing device is used to output optimized depth information of each pixel in the environment image according to the environment image of the autonomous driving vehicle and the environment point cloud data;
[0030] The controller is used to perform autonomous driving control processing on the autonomous driving vehicle based on the optimized depth information of each pixel point in the environmental image.
[0031] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.
[0033] Figure 1 is a schematic diagram according to the first embodiment of the present application;
[0034] Figure 2 is a schematic diagram of an 8-neighborhood according to the present application;
[0035] Figure 3 is a schematic diagram according to a second embodiment of the present application;
[0036] Figure 4 is a schematic diagram according to a third embodiment of the present application;
[0037] Figure 5 It is a schematic diagram of an application scenario that can implement the embodiment of the present application
[0038] Figure 6 is a schematic diagram according to a fourth embodiment of the present application;
[0039] Figure 7 is a schematic diagram according to a fifth embodiment of the present application;
[0040] Figure 8 is a schematic diagram according to a sixth embodiment of the present application;
[0041] Fig. 9 is a schematic diagram according to a seventh embodiment of the present application;
[0042] Fig.10 is a schematic diagram according to an eighth embodiment of the present application;
[0043] Fig.11 is a schematic diagram according to a ninth embodiment of the present application;
[0044] Fig.12 is a schematic diagram according to the tenth embodiment of the present application;
[0045] Fig.13 It is a schematic diagram according to the eleventh embodiment of the present application. DETAILED DESCRIPTION
[0046] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0047] It is worth noting that optimizing the depth information of the environment image based on coordinate calculation and other methods in related technologies may result in low accuracy and reliability of the optimized depth information, that is, there may be problems with incomplete or incomplete depth information.
[0048] The inventors of this application have obtained the inventive concept of the embodiments of this application through creative work: combining environmental images and environmental point cloud data, determining the depth information of each pixel in the environmental image, and the characteristic relationship between each pixel and its corresponding neighborhood pixel, optimizing the depth information of each pixel, that is, completing it, so as to achieve the accuracy and reliability of the optimized depth information.
[0049] Based on the above-mentioned inventive concept, the present application provides an image depth estimation model training method, an image depth information processing method, an apparatus, an autonomous driving vehicle, an electronic device, a program product and a storage medium, which are applied to artificial intelligence fields such as autonomous driving, computer vision, and deep learning in computer and image processing technology to achieve the technical effect of optimizing the accuracy and reliability of depth information.
[0050] Figure 1is a schematic diagram according to the first embodiment of the present application, such as Figure 1 As shown, the training method of the image depth estimation model of this embodiment includes:
[0051] S101: Input the acquired sample environment image, sample edge information of the sample environment image, and sample environment point cloud data into the model to be trained.
[0052] Exemplarily, the executor of this embodiment may be a training device for an image depth estimation model (hereinafter referred to as a training device), and the training device may be a server (such as a cloud server or a local server, etc.), a computer, a terminal device, a processor, and a chip, etc.
[0053] Among them, the model to be trained can be a convolutional neural network model, and this embodiment does not limit the parameters of the convolutional neural network model (such as the number of channels, convolution kernel coefficients, etc.).
[0054] Among them, the sample environment image can represent the image of the environment in the scene with depth information optimization. For example, for the application scenario of autonomous driving, the environment image can represent the images of various objects in the driving environment including the driving process of the autonomous driving vehicle.
[0055] Similarly, the sample environmental point cloud data can represent the point cloud data of each object in the environment in the depth information optimized scene. For example, for the application scenario of autonomous driving, the environmental point cloud data can represent the point cloud data of each object in the driving environment including the driving process of the autonomous driving vehicle.
[0056] The sample edge information of the sample environment image can be represented by edge information obtained by performing edge feature extraction on the sample environment image.
[0057] S102: The model to be trained determines the initial depth information of each pixel in the sample environment image and the characteristic relationship between each pixel and its corresponding neighboring pixel based on the sample environment image, the sample edge information of the sample environment image, and the sample environment point cloud data.
[0058] It is worth noting that in this step, two dimensions of content can be determined based on the sample environment image, the edge information of the sample environment image, and the environment point cloud data. The content of one dimension is the initial depth information of each pixel in the sample environment image, and the content of the other dimension is the characteristic relationship between each pixel and its corresponding neighborhood pixel.
[0059] The neighborhood can be 8 neighborhoods, 4 neighborhoods, or 16 neighborhoods, which is not limited in this embodiment. Figure 2The characteristic relationship between each pixel and its corresponding neighboring pixel is described as follows:
[0060] like Figure 2 As shown, pixel a is the center point, and the eight points closest to pixel a are the neighboring pixels. Figure 2 The neighborhood pixel points 1 to 8 shown in FIG.
[0061] It is worth mentioning that Figure 2 It is only used to exemplarily illustrate the positional relationship between each pixel point and its neighboring pixel points in this embodiment, and should not be understood as a limitation on the positional relationship between each pixel point and its neighboring pixel points.
[0062] Each pixel point has corresponding features, such as color features (such as RGB values, etc.), texture features, and shape features. In this embodiment, the correlation between the features of the center point a and the features of each of the eight neighboring pixel points can be determined, such as the correlation between color features, the correlation between texture features, and the correlation between shape features.
[0063] S103: Optimizing the initial depth information of each pixel according to the feature relationship to obtain optimized depth information of each pixel, and adjusting the parameters of the model to be trained according to the optimized depth information to obtain an image depth estimation model.
[0064] Among them, the image depth estimation model is used to obtain the optimized depth information of each pixel in the environment image.
[0065] It is worth noting that this embodiment introduces the feature of feature relationship, and optimizes the initial depth information of each pixel point according to the feature relationship, thereby obtaining depth optimization information of each pixel point.
[0066] For example, combining the above example and Figure 2 The feature relationship may be a correlation relationship between the feature of pixel a and the features of each pixel in the neighborhood pixels 1 to 8. By optimizing the depth information of pixel a based on the correlation relationship, it is equivalent to adjusting (such as correcting) and / or completing the depth information of pixel a based on the neighborhood pixels, so that the depth information of pixel a is as consistent as possible with the relevant information of the actual object. Therefore, the technical effect of improving the accuracy and reliability of the depth optimization information can be achieved.
[0067] Figure 3 is a schematic diagram according to the second embodiment of the present application, such as Figure 3 As shown, the training method of the image depth estimation model of this embodiment includes:
[0068] S301: Input the acquired sample environment image, sample edge information of the sample environment image, and sample environment point cloud data into the model to be trained.
[0069] For example, the description of S301 may refer to S101, which will not be repeated here.
[0070] S302: Generate image features of each pixel by splicing based on the sample environment image, sample edge information, and sample environment point cloud data.
[0071] For example, the image features may represent the color features, texture features, shape features, and spatial relationship features of each pixel.
[0072] S303: Determine, according to the image features of each pixel, initial depth information and feature relationships when each pixel is used as the center point of the sample environment image.
[0073] In this embodiment, for any pixel point, the initial depth information and feature relationship of the pixel point when the pixel point is taken as the center point of the environment image are determined.
[0074] In this embodiment, the image features of each pixel are obtained by splicing the environmental image and the environmental point cloud data, so that the image features of each pixel include both the image features corresponding to the environmental image and the image features corresponding to the environmental point cloud data, so that the image features of each pixel are richer, thereby improving the technical effect of the comprehensiveness and reliability of the determined depth information and feature relationship.
[0075] S304: determining adjustment information according to the feature relationship, and optimizing the initial depth information of each pixel according to the adjustment information to obtain optimized depth information of each pixel.
[0076] From the above examples, it can be seen that the feature relationship can characterize the correlation between the features of each pixel and the features of neighboring pixels. In this embodiment, by determining the adjustment information based on the feature relationship, the accuracy and reliability of the adjustment information can be improved, so that when the initial depth information of each pixel is optimized based on the adjustment information, the technical effect of the accuracy and reliability of the optimized depth information of each pixel can be improved.
[0077] In some embodiments, at least one of the similarity between each pixel and its neighboring pixels, the relationship between each pixel and its neighboring pixels to which it belongs, and the confidence of the initial depth information of each pixel; determining the adjustment information based on the feature relationship includes: determining the adjustment information based on at least one of the similarity, the relationship between the belonging objects, and the confidence.
[0078] For example, combining the above example and Figure 2 :
[0079] The pixel degree may be the pixel degree between pixel a and each of the neighboring pixel points 1 to 8, and may specifically be the similarity between the feature of pixel a and the feature of pixel 1, and more specifically, may be the similarity between the RGB values of the two.
[0080] The belonging object relationship may be whether pixel point a and each of neighboring pixel points 1 to 8 belong to the same object. For example, if pixel point a and neighboring pixel point 1 are both pixels of the signboard, then pixel point a and neighboring pixel point 1 belong to the same object.
[0081] The confidence level may be the possibility that the initial depth information of the pixel point a is accurate depth information.
[0082] It is worth noting that, in this embodiment, the feature relationship may include content of one dimension or content of multiple dimensions, thereby achieving the technical effect of diversity and flexibility in determining adjustment information based on the feature relationship.
[0083] In some embodiments, if the feature relationship includes a degree of similarity, then determining the feature relationship based on the image feature of each pixel includes: determining the degree of similarity based on the image feature of each pixel and the image features of neighboring pixels.
[0084] Exemplarily, similarity calculation is performed based on the image features of each pixel and the image features of neighboring pixels, such as similarity calculation of RGB values, to obtain the degree of similarity.
[0085] It is worth noting that, in this embodiment, by determining the degree of similarity based on image features, the convenience and speed of determining the degree of similarity can be improved, thereby improving the efficiency of determining the optimized depth information.
[0086] In some embodiments, if the feature relationship includes an belonging object relationship, the feature relationship is determined based on the image features of each pixel, including: determining the objects to which each pixel and the neighboring pixel belong respectively based on the image features of each pixel, and determining the belonging object relationship based on the objects to which they belong.
[0087] For example, combining the above examples and Figure 2 , according to the image features of pixel a, determine the object to which pixel a belongs, and according to the image features of each of the neighborhood pixels 1 to 8, determine the object to which each of the neighborhood pixels 1 to 8 belongs, and determine the relationship between the object to which pixel a belongs and the object to which each of the neighborhood pixels 1 to 8 belongs (such as belonging to the same object or belonging to different objects).
[0088] It is worth noting that in this embodiment, for each pixel point and the image features of the corresponding neighborhood pixels, the objects to which they belong are determined respectively, and the object relationship is determined based on the objects to which they belong, which can improve the technical effect of the accuracy and reliability of the determined objects.
[0089] In some embodiments, according to the image features of each pixel, the objects to which each pixel and the neighboring pixel belong respectively are determined, and the belonging-object relationship is determined based on the objects to which each pixel belongs, including the following steps:
[0090] Step 1: According to the image features of the neighborhood pixels, the edge information of the neighborhood pixels is determined, and based on the edge information of the neighborhood pixels, the belonging object of the neighborhood pixels is determined.
[0091] Step 2: According to the image features of each pixel, the edge information of each pixel is determined, and the object to which each pixel belongs is determined according to the edge information of each pixel.
[0092] Exemplarily, the edge information is used to represent information related to the edge of an image, that is, information about an image of a region where one image is connected to another image.
[0093] Step 3: According to the belonging object of the neighborhood pixel point and the belonging object of each pixel point, determine whether the neighborhood pixel point and each pixel point belong to the same object or belong to different objects.
[0094] It is worth noting that in this embodiment, by determining the belonging object by determining the edge information, the determined belonging object can have a technical effect of high accuracy and reliability, thereby achieving the technical effect of determining the accuracy and reliability of neighborhood pixels and each pixel point belonging to the same object, or belonging to different objects.
[0095] In some embodiments, if the feature relationship includes confidence, and the image features of at least some pixels include depth information, determining the feature relationship according to the image features of each pixel includes the following steps:
[0096] Step 1: Determine the depth information of each pixel according to the depth information of at least part of the pixels.
[0097] For example, combining the above example and Figure 2 , at least some of the image features of the neighborhood pixels 1 to 8 have depth information. If the image features of the neighborhood pixels 1 to 8 all have depth information, the depth information of the pixel a can be determined based on the depth information of the neighborhood pixels 1 to 8.
[0098] Step 2: Determine a confidence level according to the depth information of each pixel determined based on the depth information of at least part of the pixels and the initial depth information of each pixel determined based on the image feature of each pixel.
[0099] Accordingly, the depth information of pixel a is determined according to the depth information of neighboring pixels 1 to 8, and the initial depth information of pixel a is determined based on the image features of pixel a, and the confidence of the depth information of pixel a is obtained.
[0100] It is worth noting that, in this embodiment, by determining the depth information of each pixel based on the depth information of at least some of the neighborhood pixels, and determining the confidence level in combination with the determined depth information and the initial depth information, the confidence level can have a strong characteristic of characterizing the reliability of the depth information, thereby achieving the technical effect of improving the accuracy and reliability of the confidence level.
[0101] S305: Adjust the parameters of the model to be trained according to the optimized depth information to obtain an image depth estimation model.
[0102] In some embodiments, S305 may include the following steps:
[0103] Step 1: Determine the loss information between the optimized depth information and the preset reference depth information.
[0104] Exemplarily, the reference depth information may be first-order difference information of the reference depth information. The optimized depth information and the reference depth information may be calculated based on a preset loss function to obtain corresponding loss information.
[0105] Step 2: Adjust the parameters of the model to be trained based on the loss information to obtain the image depth estimation model.
[0106] Exemplarily, the magnitude of adjustment to be made to the parameters of the model to be trained may be determined based on the magnitude of the loss information.
[0107] Figure 4 is a schematic diagram according to the third embodiment of the present application, such as Figure 4 As shown, the method for processing image depth information of this embodiment includes:
[0108] S401: Acquire environmental images and environmental point cloud data.
[0109] Exemplarily, the executor of this embodiment may be an image depth information processing device (hereinafter referred to as the processing device), and the processing device may be a server (such as a cloud server or a local server, etc.), a computer, a terminal device, a processor, and a chip, etc.
[0110] The processing device of this embodiment may be the same device as the training device, or may be a different device, which is not limited in this embodiment.
[0111] The method of this embodiment can be applied to Figure 5 The application scenario described above is Figure 5 In the application scenario shown, an autonomous driving vehicle 501 is traveling on a road 502, and the autonomous driving vehicle 501 may be provided with a camera 5011 and a radar sensor 5012.
[0112] The camera 5011 can capture objects within the camera's field of view, such as Figure 5 The environment image includes lane lines 503, traffic lights 504, and signs 505 shown in FIG.
[0113] The radar sensor 5012 may collect environmental point cloud data corresponding to the radar sensor parameters, such as environmental point cloud data corresponding to the lane line 503 , the traffic light 504 , and the sign 505 .
[0114] It is worth noting that the above examples are only used to illustrate application scenarios to which the present embodiment can be applied, and should not be construed as limiting the application scenarios of the present embodiment.
[0115] Accordingly, combined with Figure 1 In the application scenario shown, the environment image may include lane lines, traffic lights, and signs.
[0116] Similarly, combined with Figure 1 In the application scenario shown, the environment image may be point cloud data including lane lines, traffic lights, and signs.
[0117] And when the method of this embodiment is applied to Figure 5 In the application scenario shown, the processing device may be an autonomous driving vehicle, and specifically may be a computer, an on-board terminal, a processor, and a chip (such as a vehicle networking chip) arranged in the autonomous driving vehicle.
[0118] Now combined Figure 1 The application scenario shown is described as follows:
[0119] The processing device can be connected to the camera and the radar sensor respectively. The camera can send the environmental image it collects to the processing device, and accordingly, the processing device can obtain the environmental image sent by the camera; similarly, the radar sensor can send the environmental point cloud data it collects to the processing device, and accordingly, the processing device can obtain the environmental point cloud data sent by the radar sensor.
[0120] S402: Inputting the environment image and the environment point cloud data into the image depth estimation model trained and generated by any of the above methods, and outputting the optimized depth information of each pixel in the environment image.
[0121] It is worth noting that the method for determining the optimized depth information of each pixel in the environment image based on the image depth estimation model can be referred to the principle of the method for determining the optimized depth information of each pixel in the environment image during the image depth estimation model training in the above example, which will not be repeated here.
[0122] Figure 6 is a schematic diagram according to the fourth embodiment of the present application, such as Figure 6 As shown, the method for processing image depth information of this embodiment includes:
[0123] S601: Acquire environmental images and environmental point cloud data.
[0124] For example, the description of S601 may refer to S401, which will not be described again here.
[0125] S602: Extract edge information of the environment image.
[0126] In this embodiment, after acquiring the environment image, the processing device may extract edge information of the environment image.
[0127] For example, edge information of an environment image can be extracted based on a canny operator.
[0128] S603: Inputting the environment image, the environment point cloud data, and the edge information into the image depth estimation model trained and generated by any of the above methods, and outputting the optimized depth information of each pixel in the environment image.
[0129] For example, the description of S603 may refer to the principle of S602, which will not be repeated here.
[0130] Figure 7 is a schematic diagram according to the fifth embodiment of the present application, such as Figure 7 As shown, the training device 700 of the image depth estimation model of this embodiment includes:
[0131] The input module 701 is used to input the acquired sample environment image, the sample edge information of the sample environment image, and the sample environment point cloud data into the model to be trained.
[0132] Determination module 702 is used to determine the initial depth information of each pixel in the sample environment image and the characteristic relationship between each pixel and its corresponding neighboring pixel based on the sample environment image, the sample edge information of the sample environment image, and the sample environment point cloud data by the model to be trained.
[0133] The optimization module 703 is used to optimize the initial depth information of each pixel point according to the feature relationship to obtain the optimized depth information of each pixel point.
[0134] The adjustment module 704 is used to adjust the parameters of the model to be trained according to the optimized depth information to obtain an image depth estimation model.
[0135] Among them, the image depth estimation model is used to obtain the optimized depth information of each pixel in the environment image.
[0136] Combination Figure 8 It can be seen that, in some embodiments, the second optimization module 703 includes:
[0137] The first determining submodule 7031 is used to determine adjustment information according to the feature relationship.
[0138] The optimization submodule 7032 is used to optimize the initial depth information of each pixel according to the adjustment information to obtain the optimized depth information of each pixel.
[0139] In some embodiments, the feature relationship includes: the degree of similarity between each pixel and its neighboring pixels, the relationship between each pixel and its neighboring pixels, and at least one of the confidence levels of the initial depth information of each pixel; the first determination submodule 7031 is used to determine the adjustment information based on at least one of the degree of similarity, the relationship between the belonging objects, and the confidence level.
[0140] Combination Fig. 9 It can be seen that, in some embodiments, the determination module 702 includes:
[0141] The generation submodule 7021 is used to splice and generate the image features of each pixel according to the sample environment image, the sample edge information, and the sample environment point cloud data.
[0142] The second determination submodule 7022 is used to determine the initial depth information and feature relationship when each pixel is used as the center point of the sample environment image according to the image feature of each pixel.
[0143] In some embodiments, the second determination submodule 7022 is used to determine the degree of similarity based on the image features of each pixel and the image features of neighboring pixels.
[0144] In some embodiments, the second determination submodule 7022 is used to determine the objects to which each pixel and the neighboring pixel belong respectively according to the image features of each pixel, and determine the belonging-object relationship based on the objects to which they belong.
[0145] In some embodiments, the second determination submodule 7022 is used to determine the edge information of the neighborhood pixels according to the image features of the neighborhood pixels, and determine the belonging objects of the neighborhood pixels based on the edge information of the neighborhood pixels, and determine the edge information of each pixel according to the image features of each pixel, and determine the belonging object of each pixel according to the edge information of each pixel, and determine whether the neighborhood pixels and each pixel belong to the same object or different objects based on the belonging object of the neighborhood pixels and the belonging object of each pixel.
[0146] In some embodiments, if the feature relationship includes confidence, the image features of at least some pixels include depth information; the second determination submodule 7022 is used to determine the depth information of each pixel based on the depth information of at least some pixels, and determine the confidence based on the depth information of each pixel determined based on the depth information of at least some pixels, and the initial depth information of each pixel determined based on the image features of each pixel.
[0147] Combination Fig.10 It can be seen that, in some embodiments, the adjustment module 704 includes:
[0148] The third determining submodule 7041 is used to determine loss information between the optimized depth information and the preset reference depth information.
[0149] The adjustment submodule 7042 is used to adjust the parameters of the model to be trained based on the loss information to obtain an image depth estimation model.
[0150] Fig.11 is a schematic diagram according to the ninth embodiment of the present application, Fig.11 As shown, the image depth information processing device 1100 of this embodiment includes:
[0151] An acquisition module 1101 is used to acquire environmental images and environmental point cloud data;
[0152] The estimation module 1102 is used to input the environment image and the environment point cloud data into the image depth estimation model trained and generated by the method described in any of the above embodiments, and output the optimized depth information of each pixel in the environment image.
[0153] Combination Fig.12 It can be seen that, in some embodiments, the image depth information processing device 1100 further includes:
[0154] An extraction module 1103 is used to extract edge information of the environment image;
[0155] Furthermore, the estimation module 1102 is used to input the environmental image, the environmental point cloud data, and the edge information into an image depth estimation model trained and generated based on the method described in any of the above embodiments, and output optimized depth information of each pixel in the environmental image.
[0156] According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.
[0157] According to an embodiment of the present application, the present application also provides a computer program product, the program product includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.
[0158] Fig.13 A schematic block diagram of an example electronic device 1300 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0159] like Fig.13 As shown, the electronic device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1302 or a computer program loaded from a storage unit 1308 into a random access memory (RAM) 1303. In the RAM 1303, various programs and data required for the operation of the device 1300 can also be stored. The computing unit 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.
[0160] A number of components in the device 1300 are connected to the I / O interface 1305, including: an input unit 1306, such as a keyboard, a mouse, etc.; an output unit 1307, such as various types of displays, speakers, etc.; a storage unit 1308, such as a disk, an optical disk, etc.; and a communication unit 1309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1309 allows the device 1300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0161] The computing unit 1301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1301 performs the various methods and processes described above, such as the training method of the image depth estimation model and / or the processing method of the image depth information. For example, in some embodiments, the training method of the image depth estimation model and / or the processing method of the image depth information may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1308. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1300 via the ROM 1302 and / or the communication unit 1309. When the computer program is loaded into the RAM 1303 and executed by the computing unit 1301, one or more steps of the training method of the image depth estimation model and / or the processing method of the image depth information described above may be performed. Alternatively, in other embodiments, the computing unit 1301 may be configured to execute a training method for an image depth estimation model and / or a method for processing image depth information in any other appropriate manner (eg, by means of firmware).
[0162] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0163] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0164] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0165] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0166] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.
[0167] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.
[0168] According to another aspect of the embodiment of the present application, the embodiment of the present application further provides an autonomous driving vehicle, comprising: an image acquisition device, a radar sensor, a controller, and an image depth information processing device as described in any of the above embodiments; wherein,
[0169] An image acquisition device is used to acquire environmental images of the autonomous driving vehicle.
[0170] Exemplarily, the image acquisition device includes a camera or other device that can be used to acquire images.
[0171] Radar sensors are used to collect point cloud data of the environment for autonomous vehicles.
[0172] The image depth information processing device is used to output optimized depth information of each pixel in the environment image based on the environment image and environment point cloud data of the autonomous driving vehicle.
[0173] The controller is used to perform autonomous driving control processing on the autonomous driving vehicle based on the optimized depth information of each pixel in the environment image.
[0174] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.
[0175] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.
Claims
1. A training method for an image depth estimation model, include: Inputting the acquired sample environment image, sample edge information of the sample environment image, and sample environment point cloud data into the model to be trained; The model to be trained generates image features of each pixel in the sample environment image by splicing according to the sample environment image, the sample edge information of the sample environment image, and the sample environment point cloud data; and determines the initial depth information when each pixel is the center point of the sample environment image and the feature relationship between each pixel and its corresponding neighboring pixel according to the image features of each pixel; and determines adjustment information according to the feature relationship; and optimizes the initial depth information of each pixel according to the adjustment information to obtain the optimized depth information of each pixel; and adjusts the parameters of the model to be trained according to the optimized depth information to obtain the image depth estimation model; wherein the image depth estimation model is used to obtain the optimized depth information of each pixel in the environment image; The characteristic relationship includes: at least one of the similarity between each pixel and the neighboring pixel, the belonging object relationship between each pixel and the neighboring pixel, and the confidence of the initial depth information of each pixel; determining the adjustment information according to the characteristic relationship includes: The adjustment information is determined according to at least one of the similarity level, the relationship between the belonging objects, and the confidence level.
2. The method according to claim 1, in, If the characteristic relationship includes the similarity degree, determining the characteristic relationship according to the image feature of each pixel point includes: The similarity level is determined according to the image features of each pixel and the image features of the neighborhood pixels.
3. The method according to claim 1, in, If the feature relationship includes the belonging object relationship, determining the feature relationship according to the image feature of each pixel point includes: According to the image features of each pixel, the objects to which each pixel and the neighboring pixel belong respectively are determined, and the belonging-object relationship is determined based on the objects to which each pixel belongs.
4. The method according to claim 3, in, Determining the objects to which each pixel and the neighboring pixel belong respectively according to the image feature of each pixel, including: Determine edge information of the neighborhood pixel points according to image features of the neighborhood pixel points, and determine the belonging object of the neighborhood pixel points based on the edge information of the neighborhood pixel points; Determine edge information of each pixel according to the image feature of each pixel, and determine the object to which each pixel belongs according to the edge information of each pixel; And, determining the belonging object relationship based on the respective objects, including: determining whether the neighborhood pixel point and each pixel point belong to the same object or different objects according to the belonging object of the neighborhood pixel point and the belonging object of each pixel point.
5. The method according to any one of claims 1 to 4, in, If the feature relationship includes the confidence level, and the image features of at least some of the pixels include depth information, determining the feature relationship according to the image features of each pixel includes: Determine the depth information of each pixel according to the depth information of at least part of the pixels; The confidence level is determined according to the depth information of each pixel determined based on the depth information of at least part of the pixels and the initial depth information of each pixel determined based on the image feature of each pixel.
6. The method according to any one of claims 1 to 4, in, Adjusting the parameters of the model to be trained according to the optimized depth information to obtain an image depth estimation model, including: Determining loss information between the optimized depth information and preset reference depth information; The parameters of the model to be trained are adjusted based on the loss information to obtain the image depth estimation model.
7. A method for processing image depth information, include: Acquire environmental images and environmental point cloud data; The environment image and the environment point cloud data are input into an image depth estimation model trained and generated based on the method described in any one of claims 1 to 6, and the optimized depth information of each pixel in the environment image is output.
8. The method according to claim 7, after acquiring the environment image and the environment point cloud data, the method further include: Extracting edge information of the environment image; And, the environmental image and the environmental point cloud data are input into an image depth estimation model trained and generated based on the method described in any one of claims 1 to 6, and the optimized depth information of each pixel in the environmental image is output, including: the environmental image, the environmental point cloud data, and the edge information are input into an image depth estimation model trained and generated based on the method described in any one of claims 1 to 6, and the optimized depth information of each pixel in the environmental image is output.
9. A training device for an image depth estimation model, include: An input module, used to input the acquired sample environment image, the sample edge information of the sample environment image, and the sample environment point cloud data into the model to be trained; A determination module, configured to determine, by the model to be trained, the initial depth information of each pixel in the sample environment image and the characteristic relationship between each pixel and its corresponding neighboring pixel according to the sample environment image, the sample edge information of the sample environment image, and the sample environment point cloud data; An optimization module, used for optimizing the initial depth information of each pixel according to the characteristic relationship to obtain optimized depth information of each pixel; An adjustment module, used to adjust the parameters of the model to be trained according to the optimized depth information to obtain an image depth estimation model; wherein the image depth estimation model is used to obtain the optimized depth information of each pixel in the environment image; The optimization module includes: A first determination submodule, configured to determine adjustment information according to the characteristic relationship; An optimization submodule, configured to optimize the initial depth information of each pixel according to the adjustment information to obtain optimized depth information of each pixel; The characteristic relationship includes at least one of the similarity between each pixel and the neighboring pixel, the belonging object relationship between each pixel and the neighboring pixel, and the confidence of the initial depth information of each pixel; The first determination submodule is used to determine the adjustment information according to at least one of the similarity, the relationship between the belonging objects, and the confidence; Wherein, the determination module includes: A generating submodule, used for splicing and generating the image features of each pixel point according to the sample environment image, the sample edge information, and the sample environment point cloud data; The second determination submodule is used to determine the initial depth information when each pixel is used as the center point of the sample environment image and the feature relationship according to the image feature of each pixel.
10. The device according to claim 9, in, The second determination submodule is used to determine the similarity level according to the image features of each pixel and the image features of the neighborhood pixels.
11. The device according to claim 9, in, The second determination submodule is used to determine the objects to which each pixel and the neighboring pixel belong respectively according to the image features of each pixel, and determine the belonging-to-object relationship based on the objects to which each pixel belongs.
12. The device according to claim 11, in, The second determination submodule is used to determine the edge information of the neighborhood pixels according to the image features of the neighborhood pixels, and determine the belonging objects of the neighborhood pixels based on the edge information of the neighborhood pixels, and determine the edge information of each pixel according to the image features of each pixel, and determine the belonging object of each pixel according to the edge information of each pixel, and determine whether the neighborhood pixel and each pixel belong to the same object or different objects according to the belonging object of the neighborhood pixel and the belonging object of each pixel.
13. The device according to any one of claims 9 to 12, in, If the feature relationship includes the confidence level, the image features of at least some pixels include depth information; the second determination submodule is used to determine the depth information of each pixel based on the depth information of at least some pixels, and determine the confidence level based on the depth information of each pixel determined based on the depth information of at least some pixels, and the initial depth information of each pixel determined based on the image features of each pixel.
14. The device according to any one of claims 9 to 12, in, The adjustment module comprises: A third determination submodule, configured to determine loss information between the optimized depth information and a preset reference depth information; An adjustment submodule is used to adjust the parameters of the model to be trained based on the loss information to obtain the image depth estimation model.
15. A device for processing image depth information, include: An acquisition module is used to acquire environmental images and environmental point cloud data; An estimation module is used to input the environmental image and the environmental point cloud data into an image depth estimation model trained and generated based on the method described in any one of claims 1 to 6, and output optimized depth information of each pixel in the environmental image.
16. The device according to claim 15, further comprising: include: An extraction module, used to extract edge information of the environment image; Furthermore, the estimation module is used to input the environmental image, the environmental point cloud data, and the edge information into an image depth estimation model trained and generated based on the method described in any one of claims 1 to 6, and output optimized depth information of each pixel in the environmental image.
17. An electronic device, include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 6; or, So that the at least one processor can execute the method of any one of claims 7-8.
18. A non-transitory computer-readable storage medium storing computer instructions, in, The computer instructions are used to make the computer execute the method according to any one of claims 1 to 6; or The computer instructions are used to cause the computer to execute the method according to any one of claims 7 to 8.
19. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented; or The computer program implements the method according to any one of claims 7-8 when executed by a processor.
20. An autonomous driving vehicle, include: An image acquisition device, a radar sensor, a controller, and an image depth information processing device as claimed in claim 15 or 16; wherein, The image acquisition device is used to acquire an image of the environment of the autonomous driving vehicle; The radar sensor is used to collect environmental point cloud data of the autonomous driving vehicle; The image depth information processing device is used to output optimized depth information of each pixel in the environment image according to the environment image of the autonomous driving vehicle and the environment point cloud data; The controller is used to perform autonomous driving control processing on the autonomous driving vehicle based on the optimized depth information of each pixel point in the environmental image.
Citation Information
Patent Citations
Visual depth of field estimation method, apparatus, devcie, and storage medium
CN109118532A
Image processing method, device, electronic equipment and storage medium
CN110400337A
Monocular vision odometer method fusing edge features and deep learning
CN111311666A
Deep estimation network training method and device for automatic driving scene and autonomous vehicle
CN111428859A
Model training method and device and prediction method and device
CN111753961A
