An image depth analysis method based on end-to-end deep learning

Abstract

The present invention belongs to the technical field of image processing, and specifically discloses an image depth analysis method based on end-to-end deep learning, comprising: calibrating a camera and a depth camera for verifying the camera at a monocular shooting end; obtaining a coded image and a depth map of an object in a target scene through the shooting end, and generating a training set; designing a neural network whose input at the input end is the coded image and whose output at the output end is the depth map, and training the neural network through the training set; removing the depth camera at the shooting end, and inputting the real-time coded image obtained at the shooting end into the trained neural network, so as to obtain a real-time depth map corresponding to the real-time coded image; and having the following advantages: realizing depth acquisition through a monocular camera and deep learning technology, significantly reducing cost, volume and power consumption, improving work efficiency and depth map accuracy, and being suitable for space and power limited devices.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image depth analysis method based on end-to-end deep learning. Background Art

[0002] In recent years, 3D camera computational imaging technology has developed rapidly, and different 3D sensors have corresponding 3D computational imaging technologies. For example, structured light cameras obtain depth information of objects by calculating the shape changes of projected speckle images, and binocular cameras obtain parallax by calculating matching pixels in the image to obtain depth information of objects. Mainstream 3D sensor solutions mainly include TOF, structured light, binocular, and lidar. These 3D sensor technologies have the disadvantages of high cost, large size, and high power consumption, which not only increase the computational complexity and processing time, but also put forward higher requirements on hardware performance.

[0003] To this end, an image depth parsing method based on end-to-end deep learning is proposed to solve the above problems. Summary of the invention

[0004] The present invention aims to provide an image depth analysis method based on end-to-end deep learning to solve or improve at least one of the above-mentioned technical problems.

[0005] In view of this, a first aspect of the present invention is to provide an image depth analysis method based on end-to-end deep learning.

[0006] A first aspect of the present invention provides an image depth analysis method based on end-to-end deep learning, comprising the following steps: calibrating a camera and a depth camera used to verify the camera at a single-purpose shooting end; obtaining a coded image and a depth map of objects in a target scene through the shooting end, and generating a training set; designing a neural network whose input end has a coded image as input and whose output end has a depth map as output, and training the neural network through the training set; removing the depth camera at the shooting end, and inputting the real-time coded image obtained by the shooting end into the trained neural network to obtain a real-time depth map corresponding to the real-time coded image.

[0007] In any of the above technical solutions, the step of generating a training set specifically includes: respectively obtaining the first contour line of the object in the encoded image and the second contour line of the object in the depth map; projecting the depth map onto the encoded image to obtain the distance value between the first contour line and the second contour line; judging whether the depth map is qualified according to the distance value, and generating the training set with the qualified depth map and the corresponding encoded image.

[0008] In any of the above technical solutions, the neural network includes an encoder and a decoder; and the step of the neural network generating the real-time depth map through the real-time encoded image specifically includes: the encoder gradually passes the real-time encoded image through multiple feature dimensions, and extracts feature information through a convolution kernel of size n×n in the receptive field of each feature dimension; the decoder integrates the corresponding receptive field and the extracted feature information in each feature dimension, and obtains the depth value corresponding to the real-time encoded image through upsampling; and generates the real-time depth map according to the depth value.

[0009] In any of the above technical solutions, n is a positive integer; for different target scenes, the size of n is positively correlated with the distance value.

[0010] In any of the above technical solutions, the step of extracting feature information specifically includes: obtaining the third contour line of the real-time encoded image, and setting the value of n according to the target scene corresponding to the real-time encoded image; in each feature extraction, obtaining the correlation relationship between the third contour line and the first pixel area on the convolution kernel; the convolution kernel extracts the feature information within the current receptive field multiple times according to the correlation relationship.

[0011] In any of the above technical solutions, the association relationship is whether the first pixel area corresponds to the second pixel area on the real-time encoded image and the third contour line, and whether there is at least one overlap; if there is an overlap, there is an association relationship; if there is no overlap, there is no association relationship; wherein, the association relationship is updated each time feature information within the current receptive field is extracted.

[0012] In any of the above technical solutions, the step of extracting the feature information in the current receptive field according to the position relationship by the convolution kernel specifically includes: determining whether the second pixel area corresponding to each first pixel area coincides with the third contour line; traversing n of the convolution kernels; 2 A first pixel area is marked, and the first pixel area that does not have the association relationship with the third contour line is marked; the marked first pixel area is offset to the third contour line on the real-time encoded image; and the feature information is extracted through the offset convolution kernel.

[0013] In any of the above technical solutions, the step of offsetting the marked first pixel area on the real-time encoded image to the third contour line specifically includes: obtaining the feature map channel value of the real-time encoded image in the current feature dimension; using the feature map channel value as an offset, and updating the second pixel area corresponding to the marked first pixel area by the offset.

[0014] In any of the above technical solutions, in the training of the neural network, the marked first pixel area is offset to the second contour line on the encoded image.

[0015] In any of the above technical solutions, the real-time depth map is constructed by the depth value, and the depth value is obtained as: distance = f*b / disparity; wherein the distance represents the depth value, f represents the real focal length of the camera, b represents the distance between the binoculars, and monocular b = 1 is used, and the dispaity represents the disparity obtained by the feature information.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] By integrating multiple key steps such as data acquisition, preprocessing, neural network training and reasoning, and post-processing into one framework, the entire process from image acquisition to depth map generation is automated. This highly integrated design greatly improves work efficiency and reduces operational complexity, making it easy for users to use even without deep expertise.

[0018] By using a deep learning model to parse depth information, it is possible to accurately extract depth information from the encoded image and generate a high-quality depth map. In particular, by introducing offset matching convolution, the performance in feature extraction has been significantly improved, thus ensuring the accuracy and reliability of the final depth map.

[0019] Traditional depth acquisition technology often relies on the collaborative work of multiple sensors, such as a binocular camera system that requires two cameras, and a lidar system that requires expensive laser scanning equipment. The present invention only requires a monocular camera with advanced depth encoding and parsing technology, which significantly reduces hardware costs. The size of a single camera is much smaller than the combination of multiple sensors, which makes the system more compact and easy to integrate into space-constrained devices such as mobile devices, wearable devices, and small robots. Fewer hardware components means lower overall power consumption, which is especially important for portable and battery-powered devices, and can extend the use time of the device.

[0020] Additional aspects and advantages of embodiments according to the present invention will become apparent in the following description or may be learned through practice of embodiments according to the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0022] Figure 1 is a flow chart of the method steps of the present invention;

[0023] Figure 2 It is a data processing screw connection block diagram of the present invention;

[0024] Figure 3 The coding diagram collected by the present invention;

[0025] Figure 4 A depth map collected by the present invention;

[0026] Figure 5 is the projected depth map of the present invention;

[0027] Figure 6 It is a schematic diagram of the superposition of the coding map and the depth map of the present invention;

[0028] Figure 7 It is a schematic diagram of the neural network structure of the present invention;

[0029] Figure 8 This is an example diagram of the receptive field of the present invention;

[0030] Fig. 9 The relationship between the receptive fields of different layers of the present invention;

[0031] Fig.10 This is an example diagram of the common convolution of the present invention;

[0032] Fig.11 Schematic diagram of convolution adopted by the present invention;

[0033] Fig.12 is a system logic block diagram of the present invention;

[0034] Fig.13 The figure is a schematic diagram of the structure of an electronic device of the present invention. DETAILED DESCRIPTION

[0035] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0036] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0037] See also Figure 1-Figure 13 , the following describes an image depth parsing method based on end-to-end deep learning according to some embodiments of the present invention.

[0038] The embodiment of the first aspect of the present invention proposes an image depth parsing method based on end-to-end deep learning. In some embodiments of the present invention, such as Figure 1-Figure 11 As shown, the method includes:

[0039] S101, calibrating a camera and a depth camera used to verify the camera at a single-purpose shooting end.

[0040] Here, the purpose of camera calibration is to determine the internal parameters of the camera (such as focal length, optical center position) and external parameters (such as the position and orientation of the camera relative to a reference coordinate system). These parameters are necessary for converting from image coordinates to actual world coordinates, and can be used to correct distortion in images captured by the camera and improve the accuracy of measurement and reconstruction. The purpose of depth camera verification is to verify the accuracy and consistency of the depth information captured by the depth camera.

[0041] As can be seen from the above, a calibration plate with known geometric properties (such as a chessboard) is usually used as a reference, and the calibration plate is photographed by the camera at multiple different positions and orientations. In each captured picture, the control points on the calibration plate (such as the corners of the chessboard) are automatically or manually identified. Through the positions of the control points on the image and their known positions in the world coordinate system, a mathematical model (such as a pinhole camera model) is used to calculate the internal and external parameters of the camera. According to the calculated parameters, the image captured by the camera is distorted, such as radial distortion and tangential distortion. A depth camera is used to capture depth maps in the same environment, which directly provide the depth information of each pixel relative to the camera. If a monocular camera is used for 3D reconstruction or depth estimation, the depth information captured by the depth camera can be compared with the results of the monocular camera to verify its accuracy. According to the results of the comparison, it may be necessary to adjust the processing flow or parameters of the monocular vision system to improve the accuracy of the depth estimation.

[0042] In any of the above embodiments, the sensor device for collecting the encoded image and depth data, the camera above is the point spread function depth coding camera, and the black one below is the depth camera. The point spread function depth coding camera and the depth camera are calibrated with internal and external parameters to obtain the internal parameters of the point spread function depth coding camera and the depth camera, as well as the external parameters of the two.

[0043] In this embodiment, the point spread function depth coding camera uses specific optical elements (such as a special lens or grating) to encode the light field information entering the camera, allowing the depth information of the scene to be recovered through a subsequent decoding process. This camera design is able to capture and encode depth information in a single frame image, making it possible to estimate depth from a monocular image. The depth camera directly measures the distance from each pixel to the camera through different technical means (such as structured light, ToF (time of flight) or stereo vision) to provide an accurate depth map.

[0044] Depth information is encoded into the captured image by applying specific distortion or scattering to the light entering the camera through optical elements. This distortion or scattering is related to the distance of the objects in the scene. The depth information of each pixel can be decoded from the encoded image through the known point spread function model and encoding strategy. A specific light pattern is projected into the scene, and the depth information is calculated by analyzing the deformation of the pattern on the surface of the object. A light pulse is emitted into the scene, and the time required for the light to be emitted and returned is measured to calculate the distance. Use two or more cameras to capture images from different perspectives and estimate the depth by comparing the differences between the images.

[0045] S102, obtaining the encoded image and depth map of the object in the target scene through the shooting end, and generating a training set.

[0046] Here, a point spread function depth encoding camera captures an encoded image of a scene, where depth information is encoded into the image by means of a point spread function. A depth camera directly captures a depth map of the scene, providing distance information for each pixel relative to the camera.

[0047] As can be seen above, specially designed optical elements (such as lenses or gratings) are used to scatter the light entering the camera, so that objects at different depths produce different blur effects on the image. This blur effect changes with the depth of the object, and the depth information can be decoded through subsequent processing. Depending on the type of depth camera (such as ToF, structured light, or stereo vision), the depth of each point in the scene is directly measured or calculated. These measurements or calculations are based on the propagation time of light, deformation of light patterns, or parallax between images.

[0048] In any of the above embodiments, step S102 specifically includes:

[0049] (1) Encoded image and depth data acquisition, including:

[0050] Data collection is performed to collect coded images and depth images respectively. The coded images collected are as follows: Figure 3 As shown in the figure, the depth map obtained is Figure 4 shown.

[0051] (2) Projecting the depth data onto the coded image to obtain the true depth of the coded image includes:

[0052] Using the point spread function depth coding camera and the internal and external parameters obtained by the depth camera calibration, the depth image is projected onto the coded image. The projection formula is shown in formulas 1, 2, and 3. The projected depth map is as follows Figure 5 As shown in Figure 2, the projected depth map and the encoded image are superimposed together, and it can be seen that the two correspond to each other, indicating that the projection is accurate, such as Figure 6 shown.

[0053] Formula 1,

[0054] Formula 2,

[0055] Formula 3,

[0056] In formula 1, u1 and v1 represent the pixel coordinates of camera 1, and f x1 , f y1 Represents the focal length of camera 1, u 01 , v 01 is the principal point position of camera 1, X c1 , Y c1 , Z c1 is the coordinate position in the coordinate system of camera 1. In formula 2, R represents the rotation matrix, T represents the translation matrix, and X c2 , Y c2 , Z c2 is the coordinate position in the camera 2 coordinate system. In formula 3, f x2 , f y2 Represents the focal length of camera 2, u 02 , v 02 is the principal point position of camera 2, u2, v2 represent the pixel coordinates of camera 2.

[0057] (3) Collect data sets, including:

[0058] Repeat steps (1) and (2) above to acquire a large number of encoded images and projected depth images, which will be used as deep learning datasets.

[0059] In this embodiment, .

[0060] S103, designing a neural network whose input end inputs the encoded image and whose output end outputs the depth map, and training the neural network using a training set.

[0061] Here, a point spread function depth encoding camera is used to capture the encoded image. This image contains depth information encoded by optical elements, but this information is not directly available and needs to be restored through a decoding process. A depth camera is used to directly capture the depth map of the scene. The depth map directly provides the distance information of each point in the scene relative to the camera. The depth image is projected onto the encoded image using the internal and external parameters of the two cameras to verify and align the depth information and the encoded image. By repeating the above process, a large number of encoded images and corresponding depth image data are collected and generated to construct a dataset for deep learning.

[0062] As can be seen from the above, the point spread function depth coding camera encodes depth information through special optical design (such as using a lens with a specific point spread function). Depth cameras (such as ToF or structured light cameras) directly calculate depth information by measuring the flight time of light or analyzing the deformation of light patterns. Through the intrinsic parameters (focal length, principal point position) and extrinsic parameters (rotation and translation relationship between cameras) obtained by camera calibration, the depth information obtained from the depth camera can be accurately mapped to the encoded image captured by the point spread function depth coding camera. Use the above formula to convert the three-dimensional points captured by the depth camera into the image coordinate system of the point spread function depth coding camera to complete the accurate projection of the depth information to the encoded image. Collect data in different scenes and conditions to ensure the diversity and richness of the data set. Pair and synchronize the projected depth map with the corresponding encoded image to ensure that each pair of images is spatially aligned, providing accurate annotation data for deep learning model training.

[0063] Specifically, the steps to generate a training set include:

[0064] A first contour line of the object in the encoded image and a second contour line of the object in the depth map are obtained respectively.

[0065] The depth map is projected onto the encoded image to obtain a distance value between the first contour line and the second contour line.

[0066] Whether the depth map is qualified is determined according to the distance value, and a training set is generated by combining the qualified depth map and the corresponding encoded image.

[0067] Here, the contour of the object is extracted from the encoded image. This step can be achieved by an edge detection algorithm (such as Canny edge detection), the purpose of which is to identify the boundaries of the object in the encoded image. The contour of the same object is extracted from the depth map. Since the depth map directly reflects the depth information of each pixel in the scene, its contour extraction may use the change in depth value to identify the boundary of the object. The depth map is projected onto the encoded image, the purpose of which is to match the contours identified in the two images and calculate the distance value between the matching contours. According to the distance value between the contours, it is judged whether the quality of the depth map meets the predetermined standard. Only when the degree of contour matching reaches a certain standard, the depth map is considered qualified. The screened qualified depth map and its corresponding encoded image are combined to construct the final deep learning training set.

[0068] As can be seen above, the edge detection algorithm is applied to the coded image and the depth map to identify the contours of the object. This process needs to take into account the local feature changes of the image and determine the boundaries through depth or color gradients. The object contours in the depth map are mapped to the coordinate space of the coded image using the camera internal and external parameters obtained previously. The distance between the mapped depth map contour and the original coded image contour is calculated. This distance can be calculated by the Euclidean distance between corresponding points on the contour. A threshold is set to evaluate the accuracy of the contour matching. Distance values ​​below this threshold mean that the two contours match well and the depth information is considered accurate. Only when the contour distance between the depth map and the coded image meets this criterion, the pair of images will be selected into the final training set. The combination process involves preprocessing, formatting and possible data augmentation operations to optimize the quality and diversity of the training set. The final training set will be used to train the deep learning model, which uses this data to learn how to accurately predict or recover from the coded image.

[0069] Specifically, the neural network includes an encoder and a decoder; and the steps of the neural network generating a real-time depth map by encoding an image in real time include:

[0070] The encoder progressively encodes the real-time image through multiple feature dimensions and extracts feature information through a convolution kernel of size n×n in the receptive field of each feature dimension.

[0071] The decoder integrates the corresponding receptive field and extracted feature information in each feature dimension, and obtains the depth value corresponding to the real-time encoded image through upsampling.

[0072] Generate real-time depth map based on depth values.

[0073] Here, the encoder is responsible for gradually extracting and compressing feature information from the real-time encoded image. It processes the image through multiple feature dimensions and uses n×n convolution kernels to extract features in the receptive field of each feature dimension. The task of the decoder is to integrate the features extracted in the encoding stage and restore the feature information to the spatial size of the original image through an upsampling process to obtain the depth value of each pixel. Based on the depth value output by the decoder, a depth map corresponding to the real-time encoded image is generated.

[0074] As can be seen above, the encoder consists of multiple layers of convolutional layers, each designed to capture different features of the image through a convolution kernel of a specific size. These features gradually transition from basic visual elements (such as edges and textures) to more complex structures (such as the partial or overall shape of an object). Each neuron in the convolutional layer has a receptive field, that is, the size of the area in the input image that it can "see". As the data passes through deeper convolutional layers, the receptive field of the neuron increases, and it can capture wider contextual information. The decoder gradually restores the size of the feature map through deconvolution (transposed convolution) or upsampling layers, while integrating feature information at different levels. In this process, the jump connection from the encoder (if any) can help restore the lost detail information and improve the accuracy of depth prediction. The decoder uses upsampling techniques to restore the compressed feature map to a size close to the input image. This process involves increasing the resolution of the feature map by interpolation or learning. The final output of the decoder is a depth map that matches the size of the input image, where the value of each pixel represents the depth information of the point. This depth map can be directly used in various applications such as 3D reconstruction, augmented reality, etc.

[0075] In any of the above embodiments, n is a positive integer; for different target scenes, the size of n is positively correlated with the distance value.

[0076] In this embodiment, according to different target scenes, a convolution kernel size n that is positively correlated with the distance value is selected to meet the requirements of feature extraction at different scales.

[0077] During feature extraction, farther objects or scenes may require larger convolution kernels to capture enough contextual information for accurate depth estimation. Therefore, the choice of n is related to the distance between the target object and the camera to ensure the accuracy of depth estimation and the quality of the depth map. For different target scenes, the most suitable convolution kernel size n can be selected according to the average distance or depth range of objects in the scene, thereby finding the best balance between detail extraction and context understanding.

[0078] S104, dismantling the depth camera at the shooting end, and inputting the real-time coded image obtained by the shooting end into the trained neural network to obtain a real-time depth map corresponding to the real-time coded image.

[0079] Here, after the depth camera is removed, the system relies on the captured real-time coded images as the input data source. These coded images contain depth information implied by specific coding techniques (such as point spread function depth coding). The real-time coded images are processed using the trained neural network model to automatically infer and restore the depth value of each pixel, and finally generate a depth map corresponding to the real-time coded image.

[0080] As can be seen above, the real-time encoded image may first undergo preprocessing steps, such as cropping, scaling, or normalization, to adapt to the input requirements of the neural network model. The neural network in the encoder part processes the real-time encoded image through a series of convolution and activation layers, extracting the features of the image layer by layer. These features contain the visual information of the image, including but not limited to edges, textures, shapes, etc., as well as the depth information implied by the encoding technology. As the data flows through the network, the decoder part begins to integrate these features, and gradually restores the depth information of the image through more convolutional layers and upsampling layers. This step may use specific deep learning strategies, such as skip connections (skipping some layers and directly transmitting data) to retain more details and reduce information loss. Finally, the depth map output by the network is the depth value of each pixel inferred based on the features of the encoded image. These depth values ​​are integrated into the final depth map for visualization or further analysis. The generated depth map may also undergo post-processing steps, such as depth range adjustment, smoothing, etc., to improve the quality and usability of the depth map.

[0081] Specifically, the step of extracting feature information includes:

[0082] A third contour line of the real-time coded image is obtained, and a value of n is set according to a target scene corresponding to the real-time coded image.

[0083] In each feature extraction, the correlation relationship between the third contour line and the first pixel area on the convolution kernel is obtained.

[0084] The convolution kernel extracts feature information within the current receptive field multiple times based on the association relationship.

[0085] Here, first, the third contour of the object is extracted from the real-time coded image. The contour provides basic information about the shape of the object, which is essential for understanding the object and scene structure in the image. According to the target scene corresponding to the real-time coded image, the size n of the convolution kernel is intelligently set. This step is based on the needs of scene features and depth information to select the convolution kernel size that best suits the current task in order to achieve the best feature extraction effect. During the feature extraction process, the correlation between the third contour and the first pixel area on the convolution kernel is analyzed. This correlation guides the behavior of the convolution kernel when extracting feature information within the receptive field.

[0086] As can be seen above, edge detection algorithms (such as the Canny algorithm) or deep learning models are used to identify and extract contours in the encoded image. The identification of contours provides important prior knowledge for subsequent feature extraction. The size n of the convolution kernel is dynamically adjusted based on the specific needs of the target scene. This may mean using convolution kernels of different sizes in different image regions or at different processing stages to adapt to the specific needs of the scene. For example, for areas with more details, smaller convolution kernels are used to capture more detailed features; while in areas with more uniform features, larger convolution kernels are used to improve processing efficiency and capture a wider range of contextual information. The process of the convolution operation is optimized by analyzing the relationship between the contour and the first pixel area of ​​the convolution kernel. This may involve adjusting the application of the convolution kernel according to the position and shape of the contour, for example, giving priority to the pixel area associated with the contour to more accurately extract the feature information around the contour. The convolution kernel extracts feature information in the current receptive field multiple times after considering the contour association relationship. This iterative process allows the network to deeply explore and integrate the detailed information around the contour, thereby generating a richer and more accurate feature representation.

[0087] In any of the above embodiments, the association relationship is whether the second pixel area on the real-time encoded image corresponding to the first pixel area overlaps with the third contour line at least one place; if overlap occurs, there is an association relationship, and if no overlap occurs, there is no association relationship.

[0088] Wherein, each time feature information within the current receptive field is extracted, the association relationship is updated.

[0089] In this embodiment, the relationship between a certain pixel position (first pixel area) in the convolution kernel and the corresponding pixel position (second pixel area) in the encoded image is determined, as well as whether this position overlaps with the contour line (third contour line) of the image. If there is an overlap, it is considered that the position has an associated relationship with the contour line. When performing the convolution operation, priority is given to those pixel areas that have an associated relationship with the contour line, because these areas are more important for understanding the structure and depth information of the image. In the continuous feature extraction process, as the convolution kernel moves on the image, the association relationship between each pixel position and the contour line is continuously updated to ensure that the feature extraction process always focuses on the key structure of the image.

[0090] First, the contours of the object are identified from the real-time encoded image using edge detection algorithms or other image processing techniques. For each receptive field position where the convolution kernel is performing feature extraction, the overlap between the pixels in the receptive field and the contour is analyzed. This analysis is based on the geometric position information of the contour and the current application position of the convolution kernel. When a pixel position of the convolution kernel is detected to coincide with the contour, the weight or priority of this position is increased, because the pixels near the contour are particularly important for understanding the image content. This may involve adjusting the convolution kernel parameters or using special processing logic to emphasize the features of these key areas. During the extraction process, the association between the pixels and the contour in each receptive field is dynamically updated according to the movement of the convolution kernel on the image. This ensures that the network can adapt to changes in image content and continue to focus on areas rich in structural information.

[0091] Specifically, the steps of extracting feature information within the current receptive field based on the position relationship of the convolution kernel include:

[0092] It is determined whether the second pixel area corresponding to each first pixel area coincides with the third contour line.

[0093] Traversing the convolution kernel n 2 A first pixel region is marked, and the first pixel region has no association relationship with the third contour line.

[0094] The marked first pixel region is shifted onto the third contour line on the real-time encoded image.

[0095] Feature information is extracted through the offset convolution kernel.

[0096] Here, for each first pixel region in the convolution kernel (i.e., each pixel position in the convolution operation), it is determined whether its corresponding second pixel region (the corresponding position in the real-time encoded image) overlaps with the third contour line. The purpose of this step is to identify those convolution kernel pixel positions that are directly associated with the image contour line. Pixel region marking: traverse all n×n pixel regions of the convolution kernel and mark those pixel regions that do not overlap with the third contour line, i.e., have no association relationship. These marked regions are considered to be relatively unimportant in the current receptive field. For pixel regions marked as unimportant, their positions on the real-time encoded image are offset to the nearest third contour line. This means adjusting the focus of the convolution kernel so that it is more focused on features near the contour line. Through this offset convolution kernel, feature information within the receptive field is extracted. This step utilizes the adjusted convolution kernel to more effectively capture features related to the contour line.

[0097] As can be seen from the above, it is necessary to accurately identify the contours in the real-time coded image first, which can be achieved through edge detection algorithms or deep learning-based methods. By analyzing the spatial relationship between the positions of each pixel in the convolution kernel and the contours, the application strategy of the convolution kernel is dynamically adjusted. This involves adjusting the weights of certain pixels in the convolution kernel or redistributing the focus of the convolution kernel to optimize the target area for feature extraction. By shifting the focus of the convolution kernel to the contours, the network's ability to capture image structural information is enhanced, especially for the recognition of boundaries and contours. Such an offset operation ensures that the network can focus more on areas that are critical to depth information, improving the accuracy and efficiency of feature extraction. After this adjusted convolution operation, the network can better understand the geometric structure and spatial depth of the image when processing real-time coded images, thereby achieving higher accuracy when generating depth maps.

[0098] Specifically, the step of shifting the marked first pixel area to the third contour line on the real-time encoded image specifically includes:

[0099] Get the feature map channel value of the real-time encoded image in the current feature dimension.

[0100] The feature map channel value is used as an offset, and the second pixel area corresponding to the marked first pixel area is updated by the offset.

[0101] Here, first, the channel values ​​are obtained from the feature map corresponding to the current feature dimension. These values ​​reflect the feature intensity or the richness of the feature information at each position of the real-time encoded image under the feature dimension. Then, these feature map channel values ​​are used as offsets to adjust the focus of the convolution operation. Specifically, these values ​​are used to update the position of the second pixel area corresponding to the first pixel area marked in the convolution kernel on the real-time encoded image. By dynamically updating the application position of the convolution kernel based on the offset of the feature intensity, the convolution operation can be more focused on the area that coincides with or is close to the image contour, thereby more effectively capturing the feature information near the edge or contour.

[0102] As can be seen from the above, in deep learning models, feature maps of different channels often represent different types of feature responses in the image. Using the intensity of these feature responses as an offset means that the focus of the convolution kernel can be adjusted according to the saliency of the features in the image. Specifically, for each marked pixel area, the offset is determined according to the value of the corresponding channel in the feature map, and then the position of the pixel area is adjusted according to this offset to make it closer to or overlap with the contour of the image. This dynamic offset based on the value of the feature map channel enables the convolution kernel to adaptively focus on the most important or most informative part of the image when extracting features, especially the contour and edge areas, which is crucial for tasks such as depth estimation and object recognition. In actual implementation, this offset may need to be fine-tuned according to the specific content of the real-time encoded image and the specific structure of the feature map to ensure that the offset operation can effectively enhance the model's ability to extract key features.

[0103] In any of the above embodiments, the real-time depth map is constructed by depth values, and the depth values ​​are obtained as follows:

[0104] distance = f*b / disparity;

[0105] Wherein, distance represents the depth value, f represents the real focal length of the camera, b represents the distance between the two eyes, and monocular b=1 is used, and disparity represents the disparity directly obtained from the feature information.

[0106] In this embodiment, the depth value of each pixel is calculated by using known camera parameters and parallax values, thereby constructing a real-time depth map, which provides basic data for subsequent applications such as 3D reconstruction and augmented reality.

[0107] Parameter definition: f: The real focal length of the camera, which is part of the camera's intrinsic parameters and is usually obtained through the camera calibration process. b: In binocular stereo vision, it represents the baseline distance between the two cameras. In the monocular case mentioned here, it is assumed that b = 1 to simplify the calculation. In fact, this does not change the essential method of calculating depth, but is a normalization of the depth scale. Disparity: represents disparity, that is, the horizontal position difference between the corresponding pixels of the same scene point in the two camera views. The size of the disparity is inversely proportional to the distance from the object to the camera.

[0108] In a binocular vision system, two cameras shooting the same scene will produce parallax due to different observation angles. According to the geometric relationship, the farther the object is from the camera, the smaller the parallax between the two camera views; the closer the distance, the larger the parallax. The above formula can be used to convert the disparity value into the actual depth value. In a monocular vision system, there is no direct disparity information because there is only one observation point. However, disparity information can also be estimated through certain technical means, such as disparity predicted by a deep learning model, using special coded images, or assuming certain scene constraints. Here, the assumption of b = 1 is actually a simplification that facilitates the application of similar calculation methods to monocular systems. A complete depth map can be constructed by calculating the depth value for each pixel in the coded image. This depth map describes in detail the distance of each point in the scene relative to the camera, providing rich information for further image analysis and processing.

[0109] The present invention provides an image depth analysis method based on end-to-end deep learning.

[0110] In any of the above embodiments, during the training of the neural network, the marked first pixel region is shifted to the second contour line on the encoded image.

[0111] In this embodiment, during the training process of the neural network, certain pixel areas (first pixel areas) are marked as areas that require special attention. These areas are usually due to their location in the image, or because they contain key information that helps with depth estimation. In the encoded image, the contour lines of the object (second contour lines) are identified by the algorithm. The identification of contour lines helps to reveal structural information in the image and provide important clues for depth estimation. The marked pixel areas are offset on the encoded image so that they are closer to or directly on the contour lines. This offset process means that the information of these pixel areas will be given priority consideration when extracting features and training models.

[0112] By offsetting the key pixel region to the contour line, the model is able to more intensively extract features related to the image boundary. Pixels near the contour line usually contain important information about the shape, size, and relative depth of the object, which is crucial for generating accurate depth maps. During model training, this offset strategy can be used as a data augmentation technique to help the model learn how to better interpret the structural information and depth clues in the image. By adaptively adjusting the pixel region of interest, the model is able to more effectively utilize the available visual information, thereby improving its performance. In deep learning models, the accuracy of depth estimation depends largely on the model's ability to extract and interpret image features. Offsetting the key pixel region to the contour line strengthens the model's understanding of the image boundary, thereby improving the accuracy of depth estimation. This method provides a way to dynamically adjust the learning focus, allowing the network to automatically focus on the areas that are most important for depth estimation when processing different images. This dynamic adjustment mechanism allows the model to maintain high accuracy and robustness when dealing with various scenarios.

[0113] In any of the above embodiments, a deep learning convolutional neural network is designed, the network input is a coded image, and the output is a depth map. In some embodiments, the network structure is as follows: Figure 7 The network consists of two parts, the encoder and the decoder. The encoder is used to extract image features and encode them, while downsampling to extract feature information of different dimensions and receptive fields. The receptive field example is shown in Figure 8 As shown in the figure, on a feature map of size 8*8, a 3*3 convolution kernel is used to perform convolution to extract features. The 3*3 area represents the size of the receptive field corresponding to the feature extraction. Fig. 9 It represents the relationship between the receptive fields of different layers. The encoder downsamples the feature map, reduces the size of the feature map, and changes the size of the receptive field. Therefore, feature information of different dimensions and receptive fields can be extracted through the feature layer operation in the encoder. The decoder is used to integrate information from the extracted features of different dimensions and receptive fields, and upsample to restore the original feature size to output the depth information of the original size.

[0114] In this embodiment, encoder: feature extraction and encoding. The encoder is responsible for gradually extracting the features of the input encoded image. Through a series of convolutional layers and downsampling (pooling) layers, the encoder is able to capture image features from simple to complex. Convolutional layer: Each convolutional layer uses a convolution kernel of size n×n to slide on the input feature map, calculates the dot product between the convolution kernel and the feature map, and thus extracts local features. These local features include visual information such as edges and textures. Downsampling: Downsampling layers (such as maximum pooling layers) reduce the spatial dimensions of feature maps while retaining important features. This step helps reduce the amount of computation and increase the receptive field size of individual neurons in subsequent layers.

[0115] Decoder: Feature Integration and Depth Recovery. The task of the decoder is to integrate the features extracted by the encoder and gradually restore the spatial size of the feature map to finally generate a depth map corresponding to the original image. Feature Integration: The decoder integrates feature information of different dimensions through a series of upsampling operations and convolutional layers. These layers not only restore the spatial dimensions of the feature map, but also fuse features at different levels to more accurately reconstruct depth information. Upsampling: Upsampling layers (such as transposed convolutional layers) increase the spatial dimensions of the feature map to approach the size of the original image. In this process, skip connections can be used to directly connect high-resolution features in the encoder to the corresponding layers of the decoder to retain more detailed information.

[0116] Depth map generation, depth value recovery: After the features are successfully integrated, the last layer of the network maps these features to depth values ​​to generate a depth map. Each pixel value in the depth map represents the predicted depth of the corresponding pixel in the original image.

[0117] In any of the above embodiments, the transfer matching convolution is as follows: Fig.11 As shown, the two curves represent ghost lines in the image. Ghost features are the image features that the network of the present invention needs to focus on extracting, so the pixel area containing ghost lines needs to be extracted through convolution for key features. Fig.11 As can be seen in the left figure, in the area of ​​the convolution of a 3*3 convolution kernel, the ghost line passes through the five pixel areas 1, 2, 3, 5, and 6, and the pixel areas 4, 7, 8, and 9 do not pass through the ghost line. Therefore, for this convolution kernel, the five pixel areas 1, 2, 3, 5, and 6 are more important, and the four areas 4, 7, 8, and 9 are less important. Therefore, through the transfer matching convolution algorithm, the four unimportant areas 4, 7, 8, and 9 are transferred to the four pixel areas with ghost lines closest to the 3*3 area, such as Fig.11 As shown by the arrows in the left figure, the 9 convolution areas become Fig.11 The gray area shown in the right figure. Fig.11 The new convolution area in the right figure contains the ghost line, which is more important for feature extraction. Therefore, the ghost feature can be better extracted through the transfer matching convolution operation.

[0118] In this embodiment, during the image processing, ghost lines are used as key visual features and need to be accurately identified and extracted by the neural network. These features are very important in some application scenarios, such as when performing image depth estimation or stereo matching. Dynamic adjustment of convolution kernels: Traditional convolution operations are performed on a fixed 3×3 area and may not fully cover all important features. Transfer matching convolution improves the relevance and effect of feature extraction by dynamically adjusting the convolution kernel's area of ​​interest to focus more on the pixel area containing ghost lines.

[0119] First, we analyze which pixel regions contain ghost lines in the area covered by the 3×3 convolution kernel. In the given example, positions 1, 2, 3, 5, and 6 are considered important regions containing ghost lines, while positions 4, 7, 8, and 9 do not cover ghost lines and are considered relatively unimportant. For those pixel regions that do not cover ghost lines (4, 7, 8, and 9), the focus of these regions is transferred to the adjacent regions containing ghost lines through the transfer matching convolution algorithm. This transfer is based on the spatial relationship between regions to ensure that the transferred convolution kernel can completely cover the ghost features. After the above transfer, the new area covered by the convolution kernel (i.e., the transferred area) can be more focused on the ghost lines of the image. In this way, when performing the convolution operation, the network can more effectively extract feature information related to the ghost lines, improving the quality of feature extraction and the accuracy of network recognition. In the deep learning model, this optimized feature extraction mechanism can directly affect the generation of the final depth map. In particular, when dealing with applications that require accurate depth information, such as 3D reconstruction or augmented reality, transfer matching convolution can provide more accurate depth estimation.

[0120] In any of the above embodiments, the specific steps of implementing the transfer matching convolution are:

[0121] ① The original feature map size is (b*h*w*c), denoted as U, where b represents the batch size, h represents the feature map height, w represents the feature map width, and c represents the number of feature map channels. After a normal convolution, the output feature map size is (b*h*w*2c), denoted as V. The output result refers to the offset (x offset and y offset) of each pixel in the original image batch, so it is 2c. The offset here is the offset of the unimportant pixel transfer, for example Fig.11 The offset of pixel 4 is (0,-2), the offset of pixel 7 is (-1,-2), the offset of pixel 8 is (2,0), and the offset of pixel 9 is (1,-1).

[0122] ② Add the pixel index value of the image in U to V to get the pixel position after transfer.

[0123] ③Perform convolution operation on the transferred pixels, thus realizing transfer matching convolution.

[0124] In this embodiment, there is an original feature map of size (b*h*w*c), where b represents the batch size, h and w represent the height and width of the feature map, respectively, and c represents the number of channels. The output feature map size is obtained by applying a common convolution layer to the original feature map U, and the 2c here represents the x and y offsets of each pixel, that is, the offsets of the pixel position of each channel in the x and y directions. Each pixel index value in the original feature map U is added to the offset in its corresponding offset feature map V to calculate the pixel position after the transfer. After obtaining the pixel position transfer, a convolution operation is performed on these transferred pixels to extract the feature information of the transfer match.

[0125] The initial convolution operation not only extracts the features of the image, but also predicts the offset in which direction each pixel should be moved. This offset is learned based on the features of the original image, and the purpose is to move unimportant pixels (i.e. pixels that do not overlap with the target features, such as ghost lines) to more important positions. By adding the original index value of each pixel to its offset, we are actually repositioning these pixels so that they focus on the key feature areas in the image. This process is dynamic and relies on the offset information learned by the convolution operation. The adjusted pixel positions are used as new inputs and the convolution operation is performed again. This time the convolution is performed on the spatially adjusted feature map. Since the pixel positions have been optimized to make the key features (such as ghost lines) more prominent, this step can more effectively capture and extract information related to these features.

[0126] In any of the above embodiments, converting depth into parallax specifically includes:

[0127] Binocular parallax refers to the difference in the position of an object caused by the different distances between two eyes when they see the same object. This difference can be represented by a parameter, namely the parallax value. The closer the object is to the observer, the greater the difference between the two eyes, which forms binocular parallax. By calculating the parallax size, the distance of the object can be obtained. The closer the object is, the greater the parallax value. The formula for converting depth to parallax is as follows, where distance represents the depth value, f represents the focal length, and b represents the distance between the two eyes;

[0128] disparity = f*b / istance;

[0129] Convert depth to disparity, where b is 1 and f is the actual focal length of the camera. After converting depth to disparity, the closer the distance, the larger the disparity value, so it is more sensitive to the disparity value of close objects, and the closer the object, the higher the accuracy.

[0130] In this embodiment, the depth information obtained by binocular disparity calculation is essential for tasks such as understanding scene structure, object positioning and navigation. Depth information can help robots or autonomous vehicles determine the distance of surrounding objects in order to respond appropriately. By converting depth information into disparity values, the spatial relationship of objects can be more intuitively represented, especially in image reconstruction or augmented reality applications. Disparity maps are often used in binocular stereo matching algorithms to derive the three-dimensional structure of the scene.

[0131] By converting the depth values ​​of the depth map in the training set into disparity, the feature information extracted by the neural network is marked as disparity, and then the disparity is compared and marked to realize the training of the neural network.

[0132] In a binocular stereo vision system, there is a direct mathematical relationship between the depth distance of an object and its disparity dispaTity on two imaging planes: dispaTity = f × b / distance, where f is the focal length of the camera and b is the baseline distance between the two eyes (i.e., the distance between the two cameras). This formula shows that the disparity is inversely proportional to the distance from the object to the camera. In some applications, such as deep learning, we may first obtain the depth information of the object. In order to take advantage of the disparity map, the depth value needs to be converted into a disparity value. This process is to reverse the above formula and convert the known depth value distance into a disparity value dispaTity. Since the disparity is inversely proportional to the depth, this means that for closer objects, the disparity value will be larger. This makes the disparity map more sensitive to changes in nearby objects, thereby providing higher accuracy when reconstructing objects or estimating their positions.

[0133] Specifically, for example, in an autonomous driving system, the vehicle needs to quickly and accurately determine the position and distance of surrounding objects. By converting the depth information in the image captured by the camera into parallax information, the system can more accurately estimate the position of other vehicles, pedestrians or obstacles, and make safer and more effective driving decisions. In virtual reality (VR) or augmented reality (AR) applications, the conversion of depth to parallax can help create a more realistic three-dimensional scene and enhance the user's immersion by enhancing the fine processing of nearby objects.

[0134] In any of the above embodiments, training the deep learning network includes:

[0135] The designed neural network model is trained using the collected data set. First, the collected true depth values ​​are converted into disparity values. The network output is obtained through forward calculation. The difference Loss between the network input and the true depth is calculated. Then, the Loss is propagated back to each layer through reverse calculation. The weight of each layer is updated using the gradient descent method to achieve the purpose of reducing the Loss. After training, any encoded image is input, and the network calculation can output the depth map corresponding to the image.

[0136] In this embodiment, first, the real depth values ​​in the collected data set are converted into disparity values. This step is because in some cases, using disparity values ​​as training targets instead of directly using depth values ​​can simplify the problem or improve the learning efficiency of the network. The encoded image is input into the designed neural network model, and the network outputs the corresponding depth map (or disparity map) through a series of forward propagation calculations of encoder and decoder layers. The difference between the depth map output by the network and the real depth (or disparity) is calculated, and a loss function (such as mean square error MSE) is usually used to quantify this difference. According to the loss value calculated by the loss function, the gradient of each layer in the network is calculated by the back propagation algorithm, that is, the partial derivative of the loss with respect to the weight of each layer. The weight parameters of each layer in the network are updated using gradient descent or other optimization algorithms, with the aim of minimizing the loss function, that is, reducing the difference between the network output and the real depth.

[0137] The purpose of converting the real depth value into disparity value is to utilize the geometric relationship in binocular vision, which simplifies the problem of depth estimation to a certain extent and makes it easier for the network to learn the mapping relationship from image to depth. Through forward calculation, the network tries to capture the mapping relationship from the input encoded image to the output depth map. This process involves extracting complex features from the image and inferring the depth value of each pixel in the image based on these features. The loss function is the key to evaluating the performance of the network, which directly affects the direction and efficiency of network training. Choosing a suitable loss function can help the network learn depth information more accurately. Gradient descent is the core algorithm for optimizing network weights. It adjusts the weights according to the partial derivative (gradient) of the loss function to the weight parameters in order to reduce the loss function value. Back propagation is an effective way to calculate these gradients, ensuring that the weights of each layer can be adjusted according to its contribution to the final loss. Through repeated forward calculation, loss calculation, back propagation and weight update processes, the model gradually learns how to predict accurate depth maps from encoded images. Multiple iterations may be required during training, as well as the use of validation sets to adjust hyperparameters to achieve the best learning effect.

[0138] In any of the above embodiments, deep learning reasoning performs image depth information analysis and calculation. Once the deep learning network is trained, the depth information of objects in any image taken by a point spread function depth coding camera can be analyzed. For a coded image, it is put into the deep learning network as input, and the trained deep learning network is used for reasoning calculation. The output is the depth map corresponding to the image. Then, the parallax is converted into depth using the following formula to obtain the depth map;

[0139] distance = f*b / disparity;

[0140] In this embodiment, the coded images captured by the point spread function depth coding camera are used as the input of the deep learning network. These coded images contain depth information implied by specific optical coding technology. The coded images are input into the trained deep learning model. The network parses the depth information in the image through the forward propagation process and outputs the disparity value of each pixel. Using the disparity value and known camera parameters (such as the true focal length f and the distance b between the binoculars), the disparity is converted into a depth value through the formula distance = f × b / disparity, thereby obtaining the corresponding depth map.

[0141] Since the feature information extracted by the neural network is marked as disparity, in actual use it is necessary to convert the disparity into a depth value for generating a depth map.

[0142] In a deep learning network, the input encoded image is processed through multiple convolutional layers, activation layers, and possible pooling layers. Each layer performs a specific transformation on the input data with the goal of gradually extracting and abstracting the depth information in the image. The encoder part of the network is responsible for extracting the features of the image, while the decoder part converts these features into a disparity map. This process involves complex nonlinear mappings that are obtained through network training. During the training process, the difference (Loss) between the output of the network (i.e. the predicted disparity map) and the true disparity map is used to calculate the gradient, and the weights of the network are updated through the back-propagation algorithm to minimize this difference. After the inference process is over, the obtained disparity map is converted into a depth map. This conversion is based on geometric relationships and the physical parameters of the camera and is a relatively simple mathematical calculation process.

[0143] Another embodiment of the first aspect of the present invention provides an image depth parsing system for implementing the image depth parsing method based on end-to-end deep learning in any of the above embodiments. In some embodiments of the present invention, Fig.12 As shown, the image depth analysis system 2 includes:

[0144] The data acquisition module 201 is responsible for acquiring coded images and depth data from a point spread function depth coding camera and / or other depth sensors, including sensor devices such as a point spread function depth coding camera and a depth camera, as well as a data receiving and preprocessing unit.

[0145] The data preprocessing module 202 performs preprocessing on the collected data, including operations such as image cropping, scaling, and normalization, and converting depth values ​​into disparity values, including an image processing unit and a depth-to-disparity calculation unit.

[0146] The neural network training module 203 uses the preprocessed data to train the deep learning model, and continuously updates the weight of the model through steps such as forward calculation, loss calculation and back propagation to reduce the error between the prediction and the actual. It includes a model training unit, a loss calculation unit, and an optimizer (such as a gradient descent algorithm).

[0147] The neural network inference module 204 uses the trained deep learning model to perform inference calculations on the new encoded image and outputs the corresponding disparity map or directly outputs the depth map. It includes a model inference unit and a disparity to depth conversion unit.

[0148] The disparity to depth conversion module 205 converts the disparity map output by the neural network into a depth map, and completes the calculation from the disparity value to the depth value according to the known camera parameters. It includes a disparity calculation unit and a depth calculation unit.

[0149] The depth map post-processing module 206 performs post-processing operations on the generated depth map, such as smoothing filtering, boundary optimization, etc., to improve the quality and usability of the depth map. It includes a depth map processing unit.

[0150] The user interface module 207 provides a user interaction interface, allowing the user to configure parameters, start data collection, trigger model training and reasoning processes, and view depth map results. It includes a configuration management unit, an operation interface unit, and a result display unit.

[0151] The present invention provides an image depth analysis system, which can adapt to different data collection requirements and environmental conditions by including multiple sensors and preprocessing methods. This flexibility ensures that the system can work efficiently in a variety of scenarios, from autonomous driving to robot navigation, and even complex three-dimensional scene reconstruction.

[0152] Through sophisticated data preprocessing and advanced disparity estimation methods based on deep learning, the system is able to generate high-precision depth maps. This not only improves the accuracy of subsequent applications, but also enhances the reliability of the system.

[0153] The entire process from data acquisition to depth map generation is highly automated, reducing the need for manual intervention and significantly improving work efficiency. At the same time, the modular design of the system also means that each component can be optimized independently, further improving overall processing speed and performance.

[0154] The user interface module provides a simple and clear interface, allowing non-expert users to easily configure parameters, start processing and view results. In addition, the modular structure of the system also facilitates maintenance and upgrades. Each module can be updated independently without affecting the stability of the overall system.

[0155] The system design takes into account the development of future technologies, and the modular architecture allows new sensors, algorithms or other components to be easily integrated, ensuring the long-term availability and compatibility of the system.

[0156] Due to the versatility and flexibility of the system design, it can be applied to a wide range of fields, including but not limited to self-driving cars, drone navigation, virtual reality content creation, medical imaging analysis, etc., providing powerful capabilities of depth perception and scene understanding.

[0157] Embodiments of the second aspect of the present invention provide electronic equipment. In some embodiments of the present invention, such as Fig.13 As shown, an electronic device is provided, which includes: electronic devices such as desktop computers, notebooks, handheld computers, and cloud servers. The electronic device 3 may include but is not limited to a processor 301 and a memory 302. Those skilled in the art will understand that Fig.13 The electronic device 3 is merely an example and does not limit the electronic device 3 , and may include more or less components than those shown in the figure, or different components.

[0158] The processor 301 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0159] The memory 302 may be an internal storage unit of the electronic device 3, for example, a hard disk or memory of the electronic device 3. The memory 302 may also be an external storage device of the electronic device 3, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 3. The memory 302 may also include both an internal storage unit of the electronic device 3 and an external storage device. The memory 302 is used to store computer programs and other programs and data required by the electronic device.

[0160] The embodiment of the third aspect of the present invention proposes a computer-readable storage medium. In some embodiments of the present invention, a computer-readable storage medium is provided, and when the computer-readable storage medium is executed by the processor 301, the steps of the above method are implemented. Therefore, the computer-readable storage medium provided by the third aspect of the present invention has all the technical effects of the above steps, which will not be repeated here.

[0161] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0162] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0163] In the embodiments provided in the present disclosure, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. Multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection of devices or units, which may be electrical, mechanical or other forms.

[0164] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0165] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should all be included in the protection scope of the present disclosure.

Claims

1. An image depth analysis method based on end-to-end deep learning, characterized in that: The steps include: Calibrate the camera and the depth camera used to verify the camera at the single-purpose shooting end; Acquire a coded image and a depth map of an object in a target scene through the shooting end; respectively acquire a first contour line of the object in the coded image and a second contour line of the object in the depth map; Projecting the depth map onto the encoded image to obtain a distance value between the first contour line and the second contour line; Determine whether the depth map is qualified according to the distance value, and generate a training set with the qualified depth map and the corresponding encoded image; Design a neural network whose input end inputs the encoded image and whose output end outputs the depth map, and train the neural network using the training set; The depth camera of the shooting end is removed, and the real-time coded image obtained by the shooting end is input into the trained neural network; the neural network includes an encoder and a decoder, the encoder gradually passes the real-time coded image through multiple feature dimensions, and extracts feature information through a convolution kernel of size n×n in the receptive field of each feature dimension; the decoder integrates the corresponding receptive field and the extracted feature information in each feature dimension, and obtains the depth value corresponding to the real-time coded image by upsampling; generating a real-time depth map according to the depth value; The method includes: obtaining a third contour line of the real-time coded image, and setting a value of n according to a target scene corresponding to the real-time coded image; obtaining an association relationship between the third contour line and a first pixel region on the convolution kernel in each feature extraction; determining whether a second pixel region on the real-time coded image corresponding to each first pixel region overlaps with the third contour line at least one place, if an overlap occurs, there is an association relationship, and if no overlap occurs, there is no association relationship; traversing the convolution kernel a first pixel region, marking the first pixel region that has no association relationship with the third contour line; shifting the marked first pixel region on the real-time coded image to the third contour line; extracting the feature information through the shifted convolution kernel; Wherein, n is a positive integer; for different target scenes, the size of n is positively correlated with the distance value.

2. The image depth analysis method according to claim 1, characterized in that: Each time feature information within the current receptive field is extracted, the association relationship is updated.

3. The image depth analysis method according to claim 1, characterized in that: The step of shifting the marked first pixel area on the real-time coded image to the third contour line specifically includes: Obtain a feature map channel value of the real-time encoded image at the current feature dimension; The feature map channel value is used as an offset, and a second pixel area corresponding to the marked first pixel area is updated by the offset.

4. The image depth analysis method according to claim 2, characterized in that: During the training of the neural network, the marked first pixel region is shifted to the second contour line on the encoded image.

5. The image depth analysis method according to claim 1, characterized in that: The real-time depth map is constructed by the depth values, and the depth values ​​are obtained as: ； Among them, the represents the depth value, f represents the real focal length of the camera, b represents the distance between the two eyes, and monocular b=1 is used, represents the disparity obtained through the feature information.

Images