Deep recognition model training method, image depth recognition method, and related device

Abstract

The application relates to image processing and provides a depth recognition model training method, an image depth recognition method and related equipment. In the application, a to-be-recognized image and a depth recognition network are acquired, a first image and a second image are acquired, a ground plane segmentation network is used to segment the first image to obtain a ground plane region, a projection image is generated based on an initial depth image corresponding to the first image and a preset pose matrix, a target height loss is generated based on a shooting device, the initial depth image and the ground plane region, a depth loss is calculated according to a gradient loss between the initial depth image and the first image and a photometric loss between the projection image and the first image, the depth recognition network is adjusted based on the depth loss and the target height loss to obtain a depth recognition model, the to-be-recognized image is input into the depth recognition model, and a recognition result of the to-be-recognized image is obtained. The application can improve the depth recognition efficiency of the image.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more particularly to a method for training a depth recognition model, an image depth recognition method, and related equipment. Background Technology

[0002] Current solutions for depth recognition of vehicle images utilize image segmentation networks to accurately segment the images, followed by depth recognition based on the segmented images. However, the numerous and complex parameters of image segmentation networks lead to a lengthy segmentation process, resulting in low efficiency for depth recognition. Therefore, improving the efficiency of depth recognition has become a crucial technical challenge. Summary of the Invention

[0003] In view of the above, it is necessary to provide a deep recognition model training method, an image depth recognition method, and related equipment to solve the technical problem of low efficiency in image depth recognition.

[0004] This application provides a depth recognition model training method, which includes: acquiring a first image and a second image captured by an imaging device; constructing a ground plane segmentation network; segmenting the first image using the ground plane segmentation network to determine the ground plane region in the first image; generating a projection image of the first image based on the first image, an initial depth image corresponding to the first image, and a pose matrix corresponding to both the first image and the second image; generating a target height loss of the depth recognition network based on a pre-acquired depth recognition network, the imaging device, the initial depth image, and the ground plane region in the first image; calculating a depth loss of the depth recognition network based on the gradient loss between the initial depth image and the first image and the photometric loss between the projection image and the first image; and adjusting the depth recognition network based on the depth loss and the target height loss to obtain a depth recognition model.

[0005] According to an optional embodiment of this application, constructing the ground plane segmentation network includes: obtaining a high-resolution network, wherein the high-resolution network includes a first-stage network, a second-stage network, a third-stage network, a fourth-stage network, and an output layer; removing the fourth-stage network and adjusting the number of branch networks in the output layer to the number of branch networks in the third-stage network to obtain the output layer corresponding to each branch network in the third-stage network; counting the number of first channels in the last convolutional layer of the backbone network of the first-stage network and adjusting the number of first channels to a first preset value to obtain a first resolution network; adjusting the number of channels in the last convolutional layer of the second-stage network to obtain a second resolution network; adjusting the number of channels in the last convolutional layer of the third-stage network to obtain a third resolution network; splicing the first resolution network, the second resolution network, and the third resolution network to obtain an image resolution network; and splicing the image resolution network and the output layer corresponding to each branch network of the third-stage network to obtain the ground plane segmentation network.

[0006] According to an optional embodiment of this application, the ground plane segmentation network further includes a classification layer. The step of segmenting the first image using the ground plane segmentation network to obtain the ground plane region in the first image includes: inputting the first image into the image resolution network in the ground plane segmentation network for processing to obtain the third branch feature map output by the third resolution network in the image resolution network at each branch network; inputting multiple third branch feature maps into the corresponding output layer for convolution operation to obtain the target feature map output by each output layer; fusing multiple target feature maps to obtain a fused feature map; inputting the fused feature map into the classification layer for classification to obtain the initial object corresponding to each preset category in the first image; and selecting the region occupied by the initial object whose preset category is the ground plane category as the ground plane region.

[0007] According to an optional embodiment of this application, the step of inputting the first image into the image resolution network in the ground plane segmentation network for processing to obtain the third branch feature map output by the third resolution network in each branch network of the image resolution network includes: inputting the first image into the backbone network of the first resolution network for feature extraction to obtain the first backbone feature map output by the backbone network of the first resolution network; inputting the first backbone feature map into each branch network of the first resolution network for convolution operation to obtain the first branch feature map output by each branch network of the first resolution network; inputting multiple first branch feature maps into the second resolution network for feature extraction to obtain the second branch feature map output by the second resolution network; and inputting multiple second feature maps into the third resolution network for feature extraction to obtain the third branch feature map.

[0008] According to an optional embodiment of this application, the first image and the second image include the same initial object. Generating a projection image of the first image based on the first image, an initial depth image corresponding to the first image, and a pose matrix corresponding to both the first image and the second image includes: determining the pixel point corresponding to the same initial object in the first image as a first pixel point, and determining the pixel point corresponding to the same initial object in the second image as a second pixel point; obtaining a first homogeneous coordinate matrix of the first pixel point and a second homogeneous coordinate matrix of the second pixel point; obtaining the inverse matrix of the intrinsic parameter matrix of the imaging device; and calculating the first homogeneous coordinate matrix of the first pixel point based on the first homogeneous coordinate matrix and the inverse matrix of the intrinsic parameter matrix. The camera coordinates are calculated, and the second camera coordinates of the second pixel are calculated based on the second homogeneous coordinate matrix and the inverse of the intrinsic parameter matrix. The first camera coordinates and the second camera coordinates are calculated based on the preset epipolar constraint relationship to obtain the rotation matrix and the translation matrix. The rotation matrix and the translation matrix are concatenated to obtain the pose matrix. The homogeneous coordinate matrix of each pixel in the first image is obtained, and the depth value of each pixel in the first image is obtained from the initial depth image. The projection coordinates of each pixel in the first image are calculated based on the pose matrix, the homogeneous coordinate matrix of each pixel, and the depth value of each pixel. The projection coordinates of each pixel are arranged according to the projection coordinates of each pixel to obtain the projection image.

[0009] According to an optional embodiment of this application, generating the target height loss of the depth recognition network based on the shooting device, the initial depth image, and the ground plane region in the first image includes: obtaining the real-world height from the optical center of the shooting device to the ground plane region; constructing a camera coordinate system based on the first image and the shooting device; calculating the projected height based on the coordinates of each ground pixel in the ground plane region in the camera coordinate system; and calculating the target height loss based on the pixel coordinates of the pixels in the initial depth image, the projected height, and the real-world height.

[0010] According to an optional embodiment of this application, calculating the projection height based on the coordinates of each ground pixel in the ground plane region of the camera coordinate system includes: obtaining the coordinates of any ground pixel in the ground plane region of the camera coordinate system; calculating a unit normal vector based on the coordinates of the any ground pixel; determining the vector formed by taking the optical center of the shooting device as the starting point and each ground pixel as the ending point as the target vector of the ground pixel; calculating the projection distance corresponding to each ground pixel based on the target vector of each ground pixel and the unit normal vector; and performing a weighted average calculation on the projection distances corresponding to all ground pixels to obtain the projection height.

[0011] According to an optional embodiment of this application, generating the target height loss of the depth recognition network based on the pre-acquired depth recognition network, the shooting device, the initial depth image, and the ground plane region in the first image includes: calculating the height ratio of the real-world height to the projected height; multiplying the height ratio by the pixel coordinates of each pixel in the initial depth image to obtain the depth coordinates corresponding to each pixel; generating a first height loss based on the pixel coordinates and corresponding depth coordinates of each pixel in the initial depth image; multiplying the translation matrix by the height ratio to obtain a multiplication matrix; generating a second height loss based on the multiplication matrix and the translation matrix; and generating the target height loss based on the first height loss and the second height loss.

[0012] This application provides an image depth recognition method, which includes: acquiring an image to be recognized, inputting the image to be recognized into a depth recognition model, obtaining a target depth image of the image to be recognized and depth information of the image to be recognized, wherein the depth recognition model is obtained by executing the depth recognition model training method as described above.

[0013] This application provides an electronic device, the electronic device comprising:

[0014] Memory, storing at least one instruction; and

[0015] The processor executes at least one instruction to implement the depth recognition model training method or the image depth recognition method.

[0016] This application provides a computer-readable storage medium storing at least one instruction, which is executed by a processor in an electronic device to implement the depth recognition model training method and the image depth recognition method.

[0017] In summary, the ground plane segmentation network constructed in this application is generated by reducing some parameters of a high-resolution network while ensuring segmentation accuracy. This reduces the network's runtime, enabling accurate and rapid segmentation of the ground plane region. A target height loss for the depth recognition network is generated based on the imaging device, the initial depth image, and the ground plane region in the first image. This target height loss is generated based on the predicted height between pixels in the ground plane region and the imaging device, as well as the actual height between pixels in the ground plane region and the imaging device. Adjusting the depth network using this target height loss makes the predicted height for each pixel in the image to be recognized more accurate. Based on the depth loss and the target height loss, the depth recognition network is adjusted to obtain a depth recognition model. This model comprehensively performs depth recognition on the image to be recognized from multiple aspects, including luminance, gradient, and the predicted height between each pixel and the imaging device. This allows for rapid and accurate identification of depth information in the image, thereby improving the efficiency of depth recognition. Attached Figure Description

[0018] Figure 1 This is an application environment diagram provided by an embodiment of this application.

[0019] Figure 2 This is a flowchart of a deep recognition model training method provided in an embodiment of this application.

[0020] Figure 3 This is a schematic diagram of the structure of the ground plane segmentation network provided in an embodiment of this application.

[0021] Figure 4 This is a schematic diagram of the pixel coordinate system and camera coordinate system provided in the embodiments of this application.

[0022] Figure 5 This is a flowchart of the image depth recognition method provided in the embodiments of this application.

[0023] Figure 6This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] like Figure 1 The diagram shown illustrates the application environment provided by an embodiment of this application. The depth recognition model training method and image depth recognition method provided in this application can be applied to one or more electronic devices 1. The electronic device 1 communicates with a shooting device 2, which can be a monocular camera or other devices for shooting.

[0026] The electronic device 1 is a device capable of automatically calculating parameter values ​​and / or processing information according to pre-set or stored instructions. Its hardware includes, but is not limited to: microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0027] The electronic device 1 can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, etc.

[0028] The electronic device 1 may further include network devices and / or user devices. The network devices include, but are not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0029] The network in which the electronic device 1 is located includes, but is not limited to: the Internet, wide area network, metropolitan area network, local area network, virtual private network (VPN), etc.

[0030] like Figure 2The diagram shown is a flowchart of a deep recognition model training method provided in an embodiment of this application. Depending on different needs, the order of the steps in the flowchart can be adjusted according to actual detection requirements, and some steps can be omitted. The execution subject of the method is an electronic device, such as... Figure 1 Electronic device 1 shown.

[0031] 101. Acquire the first image and the second image captured by the shooting device.

[0032] In at least one embodiment of this application, the shooting device may be a monocular camera, the first image and the second image are adjacent frames of three primary color light (Red, Green, Blue, RGB) images, the generation time of the second image is longer than the generation time of the first image, the first image and the second image may contain initial objects such as vehicles, ground, pedestrians, sky, trees, etc., and the first image and the second image contain the same initial objects.

[0033] 102. Construct a ground plane segmentation network.

[0034] In at least one embodiment of this application, the ground plane segmentation network refers to a network that segments the ground plane in an image.

[0035] In at least one embodiment of this application, the electronic device constructs a ground plane segmentation network comprising:

[0036] The electronic device acquires a high-resolution network, wherein the high-resolution network includes a first-stage network, a second-stage network, a third-stage network, a fourth-stage network, and an output layer. The electronic device removes the fourth-stage network and adjusts the number of branch networks in the output layer to the number of branch networks in the third-stage network, obtaining an output layer corresponding to each branch network in the third-stage network. The electronic device counts the number of the first channel in the last convolutional layer of the backbone network of the first-stage network and adjusts the number of the first channel to a first preset value, obtaining a first-resolution network. The electronic device adjusts the number of channels in the last convolutional layer of the second-stage network to obtain a second-resolution network, and adjusts the number of channels in the last convolutional layer of the third-stage network to obtain a third-resolution network. Further, the electronic device concatenates the first-resolution network, the second-resolution network, and the third-resolution network to obtain an image-resolution network, and concatenates the image-resolution network and the output layer corresponding to each branch network in the third-stage network to obtain the ground plane segmentation network.

[0037] The high-resolution network (High-Resolution Netv2, HRNetv2) can be obtained from the Internet, and the first preset value is less than the first number of channels. For example, the first preset value can be half the number of channels.

[0038] In at least one embodiment of this application, the number of output layers is the same as the number of branch networks in the third-stage network, and each output layer is connected to a corresponding branch network in the third-stage network. For example, if the number of branch networks in the third-stage network is three, the number of output layers is adjusted to three.

[0039] In at least one embodiment of this application, the number of branches in the previous resolution network is the same as the number of backbone networks in the next resolution network, and when there are multiple backbone networks, each backbone network is connected to each other in parallel.

[0040] like Figure 3The diagram shown is a schematic representation of the ground plane segmentation network provided in an embodiment of this application. The first resolution network includes one backbone network and two branch networks, with the backbone network connected to each branch network. The second resolution network includes two parallel-connected backbone networks and three branch networks, with each backbone network connected to each branch network. The third resolution network includes three parallel-connected backbone networks and three branch networks. The output layer has three layers. Each backbone network in the first, second, and third resolution networks contains four convolutional layers. When the number of channels in the last convolutional layer of the backbone network in the first stage network (stage 1) is 64, the number of channels in the last convolutional layer of the backbone network in the first resolution network (branch 1) can be 32. When the number of channels in the last convolutional layer of the first backbone network in the second stage network (stage 2) is 48, the number of channels in the first backbone network in the second resolution network (branch 2) can be 32. The number of channels in the last convolutional layer can be 24; when the number of channels in the last convolutional layer of the second backbone network in the second stage network stage2 is 96, the number of channels in the last convolutional layer of the second backbone network in the second resolution network branch2 can be 48; when the number of channels in the last convolutional layer of the first backbone network in the third stage network stage3 is 48, the number of channels in the last convolutional layer of the first backbone network in the third resolution network branch3 can be 24; when the number of channels in the last convolutional layer of the second backbone network in the third stage network stage3 is 96, the number of channels in the last convolutional layer of the second backbone network in the third resolution network branch3 can be 48; when the number of channels in the last convolutional layer of the third backbone network in the third stage network stage3 is 192, the number of channels in the last convolutional layer of the third backbone network in the third resolution network branch3 can be 96.

[0041] In this embodiment, by reducing the number of channels in the last convolutional layer of the backbone network of each resolution network, a large number of parameters of the ground plane segmentation network can be reduced.

[0042] 103. The ground plane segmentation network is used to segment the first image to obtain the ground plane region in the first image.

[0043] In at least one embodiment of this application, the ground plane segmentation network further includes a classification layer, and the electronic device uses the ground plane segmentation network to segment the first image to obtain the ground plane region in the first image, including:

[0044] The electronic device inputs the first image into the image resolution network of the ground plane segmentation network for processing, obtaining the third branch feature map output by the third resolution network in each branch network of the image resolution network. Further, the electronic device inputs multiple third branch feature maps into the corresponding output layers for convolution operation, obtaining the target feature map output by each output layer. Further still, the electronic device fuses multiple target feature maps to obtain a fused feature map. Further still, the electronic device inputs the fused feature map into the classification layer for classification, obtaining the initial object corresponding to each preset category in the first image, and selecting the area occupied by the initial object corresponding to the preset category as the ground plane category as the ground plane region.

[0045] The preset categories may include, but are not limited to: ground plane, road, pedestrian, sky, etc.

[0046] The electronic device inputs the multiple target feature maps into a 1x1 convolutional layer for convolution to obtain the fused feature map. The classification layer can be a softmax layer.

[0047] Specifically, the electronic device inputs the first image into the image resolution network of the ground plane segmentation network for processing, and obtains the third branch feature map output by the third resolution network in each branch network of the image resolution network, including:

[0048] The electronic device inputs the first image into the backbone network of the first resolution network for feature extraction, obtaining a first backbone feature map output by the backbone network of the first resolution network. The first backbone feature map is then input into each branch network of the first resolution network for convolution operation, obtaining a first branch feature map output by each branch network of the first resolution network. Further, the electronic device inputs multiple first branch feature maps into the second resolution network for feature extraction, obtaining a second branch feature map output by the second resolution network. The multiple second feature maps are then input into the third resolution network for feature extraction, obtaining the third branch feature map.

[0049] In this embodiment, the feature extraction processes of the second and third resolution networks are basically the same as those of the first resolution network, and will not be described in detail here. The resolution of each backbone network in the second and third resolution networks is different.

[0050] Through the above implementation method, since the parameters in the ground plane segmentation network are reduced, the operating speed of the ground plane segmentation network can be improved, thereby enabling the ground plane region to be segmented quickly.

[0051] 104. Generate a projection image of the first image based on the first image, the initial depth image corresponding to the first image, and the pose matrix corresponding to both the first image and the second image.

[0052] In at least one embodiment of this application, the initial depth image refers to an image containing depth information, wherein the depth information refers to the distance between the initial object corresponding to each pixel in the first image and the shooting device, wherein the shooting device can be a monocular camera.

[0053] In at least one embodiment of this application, the pose matrix refers to the transformation relationship between the camera coordinates of the pixels in the first image and the corresponding camera coordinates of the pixels in the second image, and the camera coordinates refer to the coordinates of each pixel in the camera coordinate system.

[0054] like Figure 4 The diagram shown is a schematic of the pixel coordinate system and camera coordinate system provided in an embodiment of this application. The electronic device uses the pixel point O in the first row and first column of the first image as the reference. uv A pixel coordinate system is constructed with the origin as the origin, the parallel line containing the first row of pixels as the u-axis, and the vertical line containing the first column of pixels as the v-axis. Furthermore, the electronic device uses the light spot O of the monocular camera... XY The camera coordinate system is constructed with the optical axis of the monocular camera as the X-axis, the line parallel to the u-axis of the pixel coordinate system as the Y-axis, and the line parallel to the v-axis of the pixel coordinate system as the Z-axis.

[0055] In at least one embodiment of this application, the projected image represents an image of a transformation process, wherein the transformation process refers to the transformation process between the pixel coordinates of a pixel in the first image and the corresponding pixel coordinates in the second image.

[0056] In at least one embodiment of this application, the electronic device generates a projection image of the first image based on the first image, an initial depth image corresponding to the first image, and a pose matrix corresponding to both the first image and the second image, including:

[0057] The electronic device identifies the pixels corresponding to the same initial object in the first image as first pixels and the pixels corresponding to the same initial object in the second image as second pixels. Further, the electronic device obtains the first homogeneous coordinate matrix of the first pixel and the second homogeneous coordinate matrix of the second pixel. The electronic device also obtains the inverse matrix of the intrinsic parameter matrix of the shooting device. Further, the electronic device calculates the first camera coordinates of the first pixel based on the first homogeneous coordinate matrix and the inverse matrix of the intrinsic parameter matrix, and calculates the second camera coordinates of the second pixel based on the second homogeneous coordinate matrix and the inverse matrix of the intrinsic parameter matrix. Even further, the electronic... The device calculates the coordinates of the first camera and the second camera based on a preset epipolar constraint relationship to obtain a rotation matrix and a translation matrix. The electronic device concatenates the rotation matrix and the translation matrix to obtain the pose matrix. The electronic device obtains the homogeneous coordinate matrix of each pixel in the first image and obtains the depth value of each pixel in the first image from the initial depth image. Further, the electronic device calculates the projection coordinates of each pixel in the first image based on the pose matrix, the homogeneous coordinate matrix of each pixel, and the depth value of each pixel. Further still, the electronic device arranges each pixel according to the projection coordinates of each pixel to obtain the projected image.

[0058] The depth value refers to the pixel value of each pixel in the initial depth image.

[0059] In at least one embodiment of this application, the method of inputting the first image into the depth recognition network to generate the initial depth image is prior art, and will not be described in detail here.

[0060] Wherein, the first homogeneous coordinate matrix of the first pixel is a matrix with one more dimension than the pixel coordinate matrix, and the element value of the extra dimension is 1. The pixel coordinate matrix is ​​a matrix generated based on the first pixel coordinates of the first pixel, where the first pixel coordinates refer to the coordinates of the first pixel in the pixel coordinate system. For example, if the first pixel coordinates of the first pixel in the pixel coordinate system are (u, v), then the pixel coordinate matrix of the first pixel is... Then the homogeneous coordinate matrix of the pixel is Multiply the first homogeneous coordinate matrix and the inverse of the intrinsic parameter matrix to obtain the first camera coordinates of the first pixel, and multiply the second homogeneous coordinate matrix and the inverse of the intrinsic parameter matrix to obtain the second camera coordinates of the second pixel.

[0061] The generation method of the second homogeneous coordinate matrix is ​​basically the same as that of the first homogeneous coordinate matrix, and will not be described in detail here.

[0062] The rotation matrix can be represented as:

[0063]

[0064] Where pose is the pose matrix, which is a 4x4 matrix; R is the rotation matrix, which is a 3x3 matrix; and t is the translation matrix, which is a 3x1 matrix.

[0065] The formulas for calculating the translation matrix and the rotation matrix are as follows:

[0066] K -1 p1(txR)(K -1 p2) T =0;

[0067] Among them, K -1 p1 is the coordinate of the first camera, K -1 p2 represents the coordinates of the second camera, p1 represents the first homogeneous coordinate matrix, p2 represents the second homogeneous coordinate matrix, and K represents the coordinates of the second camera. -1 is the inverse of the intrinsic parameter matrix.

[0068] In this embodiment, the translation matrix and the rotation matrix are calculated. The pose of each pixel in the first image can be determined based on the rotation matrix, and the position of each pixel in the first image can be determined based on the translation matrix.

[0069] Specifically, the formula for calculating the projection coordinates of each pixel in the projected image is as follows:

[0070] P = K * pose * Z * K -1 *H;

[0071] Where P represents the projected coordinates of each pixel, K represents the intrinsic parameter matrix of the imaging device, pose represents the pose matrix, and K -1 Let K be the inverse matrix, H be the target homogeneous coordinate matrix of each pixel in the first image, and Z be the depth value of the corresponding pixel in the initial depth image.

[0072] 105. Based on the pre-acquired depth recognition network, the shooting device, the initial depth image, and the ground plane region in the first image, generate the target height loss of the depth recognition network.

[0073] In at least one embodiment of this application, the depth recognition network refers to a network capable of recognizing depth information in an image. In at least one embodiment of this application, the target height loss refers to the difference between the predicted height and the real-world height, wherein the predicted height refers to the distance between each pixel in the first image predicted by the depth recognition network and the capturing device, and the real-world height refers to the distance between the initial object corresponding to the pixel in the first image and the capturing device in reality.

[0074] In at least one embodiment of this application, the electronic device generates a target height loss for the depth recognition network based on a pre-acquired depth recognition network, the imaging device, the initial depth image, and the ground plane region in the first image, including:

[0075] The electronic device acquires the real-world height from the optical center of the shooting device to the ground plane region. The electronic device constructs a camera coordinate system based on the first image and the shooting device. Further, the electronic device calculates the projected height based on the coordinates of each ground pixel in the ground plane region in the camera coordinate system. Even further, the electronic device calculates the target height loss based on the pixel coordinates of the pixels in the initial depth image, the projected height, and the real-world height.

[0076] Among them, constructing the camera coordinate system is as follows Figure 4 As shown.

[0077] Specifically, the electronic device calculates the projection height based on the coordinates of each ground pixel in the ground plane region of the camera coordinate system, including:

[0078] The electronic device acquires the coordinates of any ground pixel in the ground plane region of the camera coordinate system. Further, the electronic device calculates a unit normal vector based on the coordinates of any ground pixel. The electronic device determines the target vector of the ground pixel as the vector formed by the optical center of the shooting device as the starting point and each ground pixel as the ending point. Further, the electronic device calculates the projection distance corresponding to each ground pixel based on the target vector of each ground pixel and the unit normal vector. Further still, the electronic device performs a weighted average calculation on the projection distances corresponding to all ground pixels to obtain the projection height.

[0079] The formula for calculating the unit normal vector is as follows:

[0080] N t =(P t P t T ) -1 P t ;

[0081] Where, N t This refers to the unit normal vector, P. t P refers to the coordinates of any ground pixel in the ground plane region within the camera coordinate system. t T This refers to the target vector.

[0082] In this embodiment, the projection height refers to the weighted average of multiple projection distances between each pixel in the first image and the shooting device. Since the coordinates of all pixels in the ground plane area are included in the calculation, the projection height can be made more accurate.

[0083] Specifically, the electronic device calculates the target height loss based on the pixel coordinates of pixels in the initial depth image, the projected height, and the real-world height, including:

[0084] The electronic device calculates the height ratio of the real-world height to the projected height. Further, the electronic device multiplies the height ratio by the pixel coordinates of each pixel in the initial depth image to obtain the depth coordinates corresponding to each pixel. Further still, the electronic device generates a first height loss based on the pixel coordinates and corresponding depth coordinates of each pixel in the initial depth image. The electronic device multiplies the translation matrix by the height ratio to obtain a multiplication matrix. Further still, the electronic device generates a second height loss based on the multiplication matrix and the translation matrix. Further still, the electronic device generates the target height loss based on the first height loss and the second height loss.

[0085] The formula for calculating the first height loss is:

[0086]

[0087] Wherein, the L d This refers to the first height loss, where n is the total number of pixels in the initial depth image, and i is the i-th pixel in the initial depth image. The D... i t (u, v) refers to the depth coordinates corresponding to the i-th pixel in the initial depth image, D i (u, v) refers to the pixel coordinates of the i-th pixel in the initial depth image.

[0088] The formula for calculating the second altitude loss is:

[0089] L ts =|t s -t|;

[0090] Among them, L ts This refers to the second altitude loss, t s t refers to the multiplication matrix, and t refers to the translation matrix.

[0091] The electronic device performs a weighted average calculation on the first height loss and the second height loss to obtain the target height loss.

[0092] Through the above implementation method, the target height loss is calculated based on the pixel coordinates of the pixels in the initial depth image, the projected height, and the real-world height. Since the projected height is more accurate, the target height loss can be reduced more quickly.

[0093] 106. Calculate the depth loss of the depth recognition network based on the gradient loss between the initial depth image and the first image and the photometric loss between the projected image and the first image.

[0094] In at least one embodiment of this application, the depth loss includes photometric loss and gradient loss.

[0095] Specifically, the formula for calculating the depth loss is as follows:

[0096] Lc = Lt + Ls;

[0097] Lc represents the depth loss, Lt represents the photometric loss, and Ls represents the gradient loss.

[0098] The formula for calculating the photometric loss is as follows:

[0099] Lt=αSSIM(x,y)+(1-α)||x i -y i ||;

[0100] Where Lt represents the photometric loss, α is a preset balance parameter, typically set to 0.85, and SSIM(x, y) represents the structural similarity index between the projected image and the first image, ||x i -y i || represents the grayscale difference between the projected image and the first image, x i y represents the pixel value of the i-th pixel in the projected image. i This represents the pixel value of the pixel corresponding to the i-th pixel in the first image.

[0101] The formula for calculating the structural similarity index is as follows:

[0102]

[0103] c1 = (K1L) 2 ;

[0104] c2 = (K2L) 2 ;

[0105] Where SSIM(x, y) is the structural similarity index, x is the projected image, y is the first image, and μ x The average grayscale value of the projected image is μ. y σ is the average grayscale value of the first image. x σ is the grayscale standard deviation of the projected image. y Let σ be the grayscale standard deviation of the first image. xy Let c1 and c2 be the grayscale covariance between the projected image and the first image, where c1 and c2 are preset parameters, L is the maximum pixel value in the first image, and K1 and K2 are preset constants, where K1 << 1 and K2 << 1.

[0106] Specifically, the electronic device calculates the gradient loss between the initial depth image and the first image by:

[0107] The formula for calculating the gradient loss is:

[0108]

[0109] Where Ls represents the gradient loss, x represents the initial depth image, y represents the first image, D(u, v) represents the pixel coordinates of the i-th pixel in the initial depth image, and I(u, v) represents the pixel coordinates of the i-th pixel in the first image.

[0110] In this embodiment, since the depth loss includes the changes in luminance and gradient of each pixel in the first image to the corresponding pixel in the second image, the depth loss can more accurately reflect the differences between the first image and the second image.

[0111] 107. Based on the depth loss and the target height loss, adjust the depth recognition network to obtain a depth recognition model.

[0112] In at least one embodiment of this application, the deep recognition model refers to a model generated after adjusting the deep recognition network.

[0113] In at least one embodiment of this application, the electronic device adjusts the depth recognition network based on the depth loss and the target height loss to obtain a depth recognition model, including:

[0114] The electronic device calculates the overall loss of the depth recognition network based on the depth loss and the target height loss. Furthermore, the electronic device adjusts the depth recognition network based on the overall loss until the overall loss is reduced to the minimum, thereby obtaining the depth recognition model.

[0115] In at least one embodiment of this application, the depth loss and the target height loss are weighted and averaged to obtain the total loss.

[0116] In this embodiment, the overall loss includes the depth loss and the target height loss. Since the depth loss can more accurately reflect the difference between the first image and the second image, adjusting the depth network based on the overall loss can improve the learning ability of the depth network and make the recognition accuracy of the depth recognition model higher.

[0117] like Figure 5 The diagram shown is a flowchart of the image depth recognition method provided in an embodiment of this application.

[0118] Depending on the specific requirements, the order of the steps in the flowchart can be adjusted according to actual testing requirements, and some steps can be omitted. The method is executed by an electronic device, for example... Figure 1 Electronic device 1 shown.

[0119] 108. Obtain the image to be recognized.

[0120] In at least one embodiment of this application, the image to be identified refers to an image for which depth information needs to be identified.

[0121] In at least one embodiment of this application, the electronic device acquires the image to be identified by:

[0122] The electronic device retrieves the image to be identified from a preset database.

[0123] The preset database can be the KITTI database, the Cityscapes database, the vKITTI database, etc. The deep recognition network can be a deep neural network, which can be obtained from Internet databases.

[0124] 109. The image to be identified is input into the depth recognition model to obtain the target depth image of the image to be identified and the depth information of the image to be identified. The depth recognition model is obtained by executing the depth recognition model training method as described above.

[0125] In at least one embodiment of this application, the target depth image refers to an image containing depth information of each pixel in the image to be identified, and the depth information of each pixel in the image to be identified refers to the distance between the object to be identified corresponding to each pixel in the image to be identified and the shooting device that captured the image to be identified.

[0126] In at least one embodiment of this application, the method of generating the target depth image is basically the same as the method of generating the initial depth image, so this application will not elaborate further.

[0127] In at least one embodiment of this application, the electronic device acquires the pixel value of each pixel in the target depth image as the depth information of the corresponding pixel in the image to be identified.

[0128] By implementing the above methods, the accuracy of depth recognition model is improved, thereby enhancing the accuracy of depth recognition of the image to be recognized.

[0129] In summary, the ground plane segmentation network constructed in this application is generated by reducing some parameters of a high-resolution network while ensuring segmentation accuracy. This reduces the network's runtime, enabling accurate and rapid segmentation of the ground plane region. A target height loss for the depth recognition network is generated based on the imaging device, the initial depth image, and the ground plane region in the first image. This target height loss is generated based on the predicted height between pixels in the ground plane region and the imaging device, as well as the actual height between pixels in the ground plane region and the imaging device. Adjusting the depth network using this target height loss makes the predicted height for each pixel in the image to be recognized more accurate. Based on the depth loss and the target height loss, the depth recognition network is adjusted to obtain a depth recognition model. This model comprehensively performs depth recognition on the image to be recognized from multiple aspects, including luminance, gradient, and the predicted height between each pixel and the imaging device. This allows for rapid and accurate identification of depth information in the image, thereby improving the efficiency of depth recognition.

[0130] like Figure 6 The diagram shown is a schematic diagram of the structure of the electronic device provided in an embodiment of this application.

[0131] In one embodiment of this application, the electronic device 1 includes, but is not limited to, a memory 12, a processor 13, and a computer program stored in the memory 12 and executable on the processor 13, such as an image depth recognition program and a depth recognition model training program.

[0132] Those skilled in the art will understand that the schematic diagram is merely an example of electronic device 1 and does not constitute a limitation on electronic device 1. It may include more or fewer components than shown in the diagram, or combine certain components, or different components. For example, electronic device 1 may also include input / output devices, network access devices, buses, etc.

[0133] The processor 13 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 13 is the computing core and control center of the electronic device 1, connecting various parts of the electronic device 1 through various interfaces and lines, and acquiring the operating system and installed applications and program code of the electronic device 1. For example, the processor 13 can acquire the first image captured by the imaging device 2 through an interface.

[0134] The processor 13 acquires the operating system and various installed applications of the electronic device 1. The processor 13 acquires these applications to implement the steps in the aforementioned deep recognition model training methods and image deep recognition method embodiments, for example... Figure 2 and Figure 5 The steps are shown.

[0135] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and retrieved by the processor 13 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the process of retrieving the computer program from the electronic device 1.

[0136] The memory 12 can be used to store the computer programs and / or modules. The processor 13 implements various functions of the electronic device 1 by running or retrieving the computer programs and / or modules stored in the memory 12, and by calling the data stored in the memory 12. The memory 12 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 12 may include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.

[0137] The memory 12 can be the external memory and / or internal memory of the electronic device 1. Furthermore, the memory 12 can be a physical memory, such as a memory module, a TF card (Trans-flash Card), etc.

[0138] If the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when the computer program is acquired by a processor, it can implement the steps of the various method embodiments described above.

[0139] The computer program includes computer program code, which may be in the form of source code, object code, accessible file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, and read-only memory (ROM).

[0140] Combination Figure 2The memory 12 in the electronic device 1 stores multiple instructions to implement a depth recognition model training method. The processor 13 can acquire the multiple instructions to implement: acquiring a first image and a second image captured by the imaging device; constructing a ground plane segmentation network; using the ground plane segmentation network to segment the first image; determining the ground plane region in the first image; generating a projection image of the first image based on the first image, the initial depth image corresponding to the first image, and the pose matrix corresponding to both the first image and the second image; generating a target height loss of the depth recognition network based on the pre-acquired depth recognition network, the imaging device, the initial depth image, and the ground plane region in the first image; calculating the depth loss of the depth recognition network based on the gradient loss between the initial depth image and the first image and the photometric loss between the projection image and the first image; and adjusting the depth recognition network based on the depth loss and the target height loss to obtain a depth recognition model.

[0141] Combination Figure 5 The memory 12 in the electronic device 1 stores multiple instructions to implement an image depth recognition method. The processor 13 can acquire the multiple instructions to achieve: acquiring an image to be recognized, inputting the image to be recognized into a depth recognition model, and obtaining the target depth image of the image to be recognized and the depth information of the image to be recognized.

[0142] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 2 and Figure 5 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0143] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0144] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0145] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0146] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0147] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in this application may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. A deep recognition model training method, applied to an electronic device, wherein the electronic device communicates with a shooting device, characterized in that, The deep recognition model training method includes: Acquire the first image and the second image captured by the imaging device; Construct a ground plane segmentation network; The ground plane segmentation network is used to segment the first image to determine the ground plane region in the first image; Based on the first image, the initial depth image corresponding to the first image, and the pose matrix corresponding to both the first image and the second image, a projection image of the first image is generated. Based on a pre-acquired depth recognition network, the imaging device, the initial depth image, and the ground plane region in the first image, the target height loss of the depth recognition network is generated, including: inputting the first image into the depth recognition network to generate the initial depth image; obtaining the real-world height from the optical center of the imaging device to the ground plane region; constructing a camera coordinate system based on the first image and the imaging device; calculating the projected height based on the coordinates of each ground pixel in the ground plane region in the camera coordinate system; and calculating the target height loss based on the pixel coordinates of the pixels in the initial depth image, the projected height, and the real-world height. The depth loss of the depth recognition network is calculated based on the gradient loss between the initial depth image and the first image and the photometric loss between the projected image and the first image. Based on the depth loss and the target height loss, the depth recognition network is adjusted to obtain a depth recognition model.

2. The deep recognition model training method as described in claim 1, characterized in that, The construction of the ground plane segmentation network includes: Obtain a high-resolution network, wherein the high-resolution network includes a first-stage network, a second-stage network, a third-stage network, a fourth-stage network, and an output layer: Remove the fourth-stage network and adjust the number of branch networks in the output layer to the number of branch networks in the third-stage network to obtain the output layer corresponding to each branch network in the third-stage network; The number of the first channel in the last convolutional layer of the backbone network of the first stage network is counted, and the number of the first channel is adjusted to a first preset value to obtain the first resolution network; The number of channels in the last convolutional layer of the second-stage network is adjusted to obtain the second-resolution network, and the number of channels in the last convolutional layer of the third-stage network is adjusted to obtain the third-resolution network. By splicing the first resolution network, the second resolution network, and the third resolution network, an image resolution network is obtained. The image resolution network and the output layer corresponding to each branch network in the third stage network are spliced ​​together to obtain the ground plane segmentation network.

3. The deep recognition model training method as described in claim 2, characterized in that, The ground plane segmentation network further includes a classification layer, and the step of segmenting the first image using the ground plane segmentation network to obtain the ground plane region in the first image includes: The first image is input into the image resolution network in the ground plane segmentation network for processing, and the third branch feature map output by the third resolution network in the image resolution network in each branch network is obtained. Multiple third-branch feature maps are input into the corresponding output layers for convolution operations to obtain the target feature map output by each output layer. Multiple target feature maps are fused to obtain a fused feature map; The fused feature map is input into the classification layer for classification to obtain the initial object corresponding to each preset category in the first image; The area occupied by the initial object corresponding to the preset category of ground plane is selected as the ground plane area.

4. The deep recognition model training method as described in claim 3, characterized in that, The step of inputting the first image into the image resolution network of the ground plane segmentation network for processing, and obtaining the third branch feature map output by the third resolution network in each branch network of the image resolution network, includes: The first image is input into the backbone network of the first resolution network for feature extraction, and the first backbone feature map output by the backbone network of the first resolution network is obtained. The first backbone feature map is input into each branch network of the first resolution network for convolution operation to obtain the first branch feature map output by each branch network of the first resolution network. Multiple first branch feature maps are input into the second resolution network for feature extraction to obtain the second branch feature map output by the second resolution network. Multiple second branch feature maps are then input into the third resolution network for feature extraction to obtain the third branch feature map.

5. The deep recognition model training method as described in claim 1, characterized in that, The first image and the second image include the same initial object. Generating a projection image of the first image based on the first image, the initial depth image corresponding to the first image, and the pose matrix corresponding to both the first image and the second image includes: The pixel corresponding to the same initial object in the first image is determined as the first pixel, and the pixel corresponding to the same initial object in the second image is determined as the second pixel; Obtain the first homogeneous coordinate matrix of the first pixel, and obtain the second homogeneous coordinate matrix of the second pixel; Obtain the inverse matrix of the intrinsic parameter matrix of the shooting device; The first camera coordinates of the first pixel are calculated based on the first homogeneous coordinate matrix and the inverse of the intrinsic parameter matrix, and the second camera coordinates of the second pixel are calculated based on the second homogeneous coordinate matrix and the inverse of the intrinsic parameter matrix. The rotation matrix and translation matrix are calculated based on the preset epipolar constraint relationship for the first camera coordinates and the second camera coordinates. The rotation matrix and the translation matrix are concatenated to obtain the pose matrix; Obtain the homogeneous coordinate matrix of each pixel in the first image, and obtain the depth value of each pixel in the first image from the initial depth image; Based on the pose matrix, the homogeneous coordinate matrix of each pixel, and the depth value of each pixel, the projected coordinates of each pixel in the first image are calculated. The projected image is obtained by arranging each pixel according to its projected coordinates.

6. The deep recognition model training method as described in claim 1, characterized in that, The step of calculating the projection height based on the coordinates of each ground pixel in the ground plane region of the camera coordinate system includes: Obtain the coordinates of any ground pixel in the ground plane region within the camera coordinate system; Calculate the unit normal vector based on the coordinates of any ground pixel; The vector formed by taking the optical center of the shooting device as the starting point and each ground pixel as the ending point is determined as the target vector of that ground pixel. The projection distance corresponding to each ground pixel is calculated based on the target vector of each ground pixel and the unit normal vector. The projection height is obtained by performing a weighted average calculation on the projection distances corresponding to all ground pixels.

7. The deep recognition model training method as described in claim 5, characterized in that, The step of calculating the target height loss based on the pixel coordinates of pixels in the initial depth image, the projected height, and the real-world height includes: Calculate the height ratio of the real-world height to the projected height; The height ratio is multiplied by the pixel coordinates of each pixel in the initial depth image to obtain the depth coordinates of each pixel. A first height loss is generated based on the pixel coordinates and corresponding depth coordinates of each pixel in the initial depth image; Multiply the translation matrix by the height ratio to obtain the multiplication matrix; A second height loss is generated based on the multiplication matrix and the translation matrix; The target height loss is generated based on the first height loss and the second height loss.

8. An image depth recognition method, characterized in that, The image depth recognition method includes: Acquire the image to be recognized; The image to be identified is input into a depth recognition model to obtain a target depth image of the image to be identified and depth information of the image to be identified. The depth recognition model is obtained by performing the depth recognition model training method as described in any one of claims 1 to 7.

9. An electronic device, characterized in that, The electronic device includes: Memory, storing at least one instruction; and The processor executes the at least one instruction to implement the depth recognition model training method as described in any one of claims 1 to 7, or the image depth recognition method as described in claim 8.

Images