Image depth recognition method, electronic device, and storage medium

Abstract

The application relates to image processing and provides an image depth recognition method, an electronic device and a storage medium. The method comprises the following steps: acquiring a to-be-recognized image, a first initial image and a second initial image; recognizing the first initial image based on a depth recognition network to obtain an initial depth image; preprocessing the first initial image and the second initial image, and generating a pose absolute value matrix based on the preprocessed images and a pose network; generating an initial projection image according to the first initial image and the second initial image, the pose network and the initial depth image; recognizing a target image and a target projection image according to the pose absolute value matrix and a preset threshold matrix; adjusting the depth recognition network based on the errors among the initial depth image, the target image and the target projection image to obtain a depth recognition model; inputting the to-be-recognized image into the depth recognition model to obtain depth information of the to-be-recognized image; and the application can improve the accuracy of image depth recognition.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more particularly to an image depth recognition method, electronic device, and storage medium. Background Technology

[0002] Current methods for depth recognition of in-vehicle images employ training images to train deep networks. However, since the training images typically include both static and dynamic objects, the presence of dynamic objects can negatively impact the training accuracy of the deep network. Consequently, the trained depth recognition model may fail to accurately identify the depth information in the in-vehicle image, making it difficult to determine the true distance between the vehicle and various objects or obstacles in the surrounding environment, thus affecting driving safety. Summary of the Invention

[0003] In view of the above, it is necessary to provide an image depth recognition method, electronic device and storage medium to solve the technical problem of inaccurate depth information recognition of vehicle images.

[0004] This application provides an image depth recognition method, the image depth recognition method comprising:

[0005] The process involves acquiring an image to be recognized, a first initial image, and a second initial image, and obtaining a depth recognition network and a pose network. Depth recognition is performed on the first initial image using the depth recognition network to obtain an initial depth image. The first initial image is preprocessed to obtain a first static image and a first dynamic image corresponding to the first initial image. The second initial image is also preprocessed to obtain a second static image and a second dynamic image corresponding to the second initial image. An absolute pose matrix is ​​generated based on the first static image, the first dynamic image, the second static image, the second dynamic image, and the pose network. The first initial image and the second initial image are then input into the system. The target pose matrix is ​​obtained by inputting the pose network; an initial projection image of the first initial image is generated based on the first initial image, the initial depth image, and the target pose matrix; the target image of the first initial image and the target projection image of the initial projection image are identified according to the pose absolute value matrix and a preset threshold matrix; the depth recognition network is adjusted based on the gradient error between the initial depth image and the target image and the photometric error between the target projection image and the target image to obtain a depth recognition model; the image to be recognized is input into the depth recognition model to obtain the target depth image of the image to be recognized and the depth information of the image to be recognized.

[0006] According to an optional embodiment of this application, the preprocessing of the first initial image to obtain a first static image and a first dynamic image corresponding to the first initial image includes:

[0007] Calculate a single score value for each pixel in the first initial image based on the pixel value of each pixel; calculate the category probability of each pixel in the first initial image for each preset object based on the single score value and multiple preset objects; determine the preset object corresponding to the category probability with the highest value as the pixel object corresponding to that pixel; determine the pixel region formed by pixels of the same pixel object in the first initial image as the initial object; classify the initial objects according to preset rules to obtain dynamic objects corresponding to dynamic categories and static objects corresponding to static categories in the first initial image; perform masking processing on the dynamic objects in the first initial image to obtain the first static image; perform masking processing on the static objects in the first initial image to obtain the first dynamic image.

[0008] According to an optional embodiment of this application, generating a pose absolute value matrix based on the first static image, the first dynamic image, the second static image, the second dynamic image, and the pose network includes:

[0009] The first static image and the second static image are input into the pose network to obtain a static pose matrix; the first dynamic image and the second dynamic image are input into the pose network to obtain a dynamic pose matrix; each matrix element in the static pose matrix is ​​subtracted from the corresponding matrix element in the dynamic pose matrix to obtain a pose difference; the absolute value of the pose difference is taken to obtain the absolute pose value of each matrix element in the static pose matrix; the absolute pose values ​​are arranged according to the element position of each matrix element in the static pose matrix to obtain the absolute pose value matrix.

[0010] According to an optional embodiment of this application, the first initial image and the second initial image are images captured by the same imaging device, and the step of inputting the first initial image and the second initial image into the pose network to obtain the target pose matrix includes:

[0011] The pixel corresponding to the dynamic object in the first initial image is determined as the first pixel; the first homogeneous coordinate matrix of the first pixel is obtained, and the second homogeneous coordinate matrix of the second pixel corresponding to the first pixel in the second initial image is obtained; the inverse matrix of the intrinsic parameter matrix of the shooting device is obtained; the first camera coordinate of the first pixel is calculated based on the first homogeneous coordinate matrix and the inverse matrix of the intrinsic parameter matrix, and the second camera coordinate of the second pixel is calculated based on the second homogeneous coordinate matrix and the inverse matrix of the intrinsic parameter matrix; the first camera coordinate and the second camera coordinate are calculated based on a preset epipolar constraint relationship to obtain a rotation matrix and a translation matrix; the rotation matrix and the translation matrix are concatenated to obtain the target pose matrix.

[0012] According to an optional embodiment of this application, the step of identifying the target image of the first initial image and the target projection image of the initial projection image based on the pose absolute value matrix and the preset threshold matrix includes:

[0013] Each pose absolute value in the pose absolute value matrix is ​​compared with the corresponding threshold in the preset threshold matrix; if there is at least one pose absolute value in the pose absolute value matrix that is greater than the corresponding threshold, then the first static image is determined as the target image, the dynamic position of the dynamic object in the first initial image is identified, the region in the initial projection image corresponding to the dynamic position is determined as the projection object, and the projection object is masked to obtain the target projection image; or, if each pose absolute value in the pose absolute value matrix is ​​less than or equal to the corresponding threshold, the first initial image is determined as the target image, and the initial projection image is determined as the target projection image.

[0014] According to an optional embodiment of this application, generating an initial projection image of the first initial image based on the first initial image, the initial depth image, and the target pose matrix includes:

[0015] Obtain the target homogeneous coordinate matrix of each pixel in the first initial image, and obtain the depth value of each pixel in the first initial image from the initial depth image; calculate the projection coordinates of each pixel in the first initial image based on the target pose matrix, the target homogeneous coordinate matrix of each pixel, and the depth value of each pixel; arrange each pixel according to the projection coordinates of each pixel to obtain the initial projection image.

[0016] According to an optional embodiment of this application, adjusting the depth recognition network based on the gradient error between the initial depth image and the target image and the photometric error between the target projection image and the target image to obtain a depth recognition model includes:

[0017] The depth loss value of the depth recognition network is calculated based on the gradient error and the photometric error; the depth recognition network is adjusted based on the depth loss value until the depth loss value is reduced to the minimum, thus obtaining the depth recognition model.

[0018] According to an optional embodiment of this application, the formula for calculating the photometric error is:

[0019]

[0020] Where Lt represents the photometric error, α is a preset balance parameter, SSIM(x,y) represents the structural similarity index between the target projection image and the target image, ||x i -y i || represents the grayscale difference between the target projection image and the target image, x i y represents the pixel value of the i-th pixel in the target projection image. i This represents the pixel value of the pixel corresponding to the i-th pixel in the target image.

[0021] This application provides an electronic device, the electronic device comprising:

[0022] Memory, storing at least one instruction; and

[0023] The processor executes the at least one instruction to implement the image depth recognition method.

[0024] This application provides a computer-readable storage medium storing at least one instruction, which is executed by a processor in an electronic device to implement the image depth recognition method.

[0025] In summary, this application preprocesses the first initial image to accurately identify dynamic and static objects within it. When at least one pose absolute value is greater than a corresponding threshold, the dynamic object is determined to have moved. When all pose absolute values ​​in the pose absolute value matrix are less than or equal to the corresponding threshold, the dynamic object is determined not to have moved. Therefore, masking of non-moving dynamic objects in the initial image is avoided. When a dynamic object in the first initial image moves, it is masked to obtain a target image. Similarly, the dynamic object in the initial projection image is masked to obtain the target projection image. Based on the gradient error between the initial depth image and the target image, and the photometric error between the target projection image and the target image, the depth recognition network is adjusted to obtain a depth recognition model. Since adjusting the deep neural network based on the gradient error and the photometric error avoids masking of non-moving dynamic objects in the initial image, the accuracy of the depth recognition model is improved, thereby enhancing the precision of image depth recognition. Attached Figure Description

[0026] Figure 1 This is an application environment diagram of a preferred embodiment of the image depth recognition method of this application.

[0027] Figure 2 This is a flowchart of a preferred embodiment of the image depth recognition method of this application.

[0028] Figure 3 This is a schematic diagram of the pixel coordinate system and camera coordinate system of the image depth recognition method of this application.

[0029] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the image depth recognition method of this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this application clearer, the application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] like Figure 1 The diagram shown illustrates an application environment of a preferred embodiment of the image depth recognition method of this application. The image depth recognition method can be applied to one or more electronic devices 1, which communicate with a shooting device 2. The shooting device 2 can be a monocular camera or other devices used for shooting.

[0032] The electronic device 1 is a device capable of automatically calculating parameter values ​​and / or processing information according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0033] The electronic device 1 can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, etc.

[0034] The electronic device 1 may further include network devices and / or user devices. The network devices include, but are not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0035] The network in which the electronic device 1 is located includes, but is not limited to: the Internet, wide area network, metropolitan area network, local area network, virtual private network (VPN), etc.

[0036] like Figure 2 The diagram shown is a flowchart of a preferred embodiment of an image depth recognition method according to this application. Depending on different needs, the order of the steps in the flowchart can be adjusted according to actual detection requirements, and some steps can be omitted. The method is executed by an electronic device, such as... Figure 1 Electronic device 1 shown.

[0037] 101. Obtain the image to be recognized, the first initial image, and the second initial image, and obtain the depth recognition network and the pose network.

[0038] In at least one embodiment of this application, the image to be identified refers to an image that requires depth information recognition.

[0039] In at least one embodiment of this application, the first initial image and the second initial image are three primary color light (Red, Green, Blue, RGB) images of adjacent frames, the generation time of the second initial image is longer than the generation time of the first initial image, the first initial image and the second initial image may contain initial objects such as vehicles, ground, pedestrians, sky, and trees, and the first initial image and the second initial image contain the same initial objects.

[0040] In at least one embodiment of this application, the electronic device acquires the image to be identified by:

[0041] The electronic device controls the shooting device to capture the target scene and obtain the image to be identified.

[0042] The target scene may include target objects such as vehicles, ground, and pedestrians.

[0043] In at least one embodiment of this application, the electronic device obtains the first initial image and the second initial image from a preset database, which may be a KITTI database, a Cityscapes database, a vKITTI database, etc.

[0044] In at least one embodiment of this application, the deep recognition network can be a deep neural network, and the pose network refers to a convolutional neural network for recognizing poses. Both the deep recognition network and the pose network can be obtained from databases on the Internet.

[0045] 102. Based on the depth recognition network, perform depth recognition on the first initial image to obtain an initial depth image.

[0046] In at least one embodiment of this application, the initial depth image refers to an image containing depth information, wherein the depth information refers to the distance between the initial object corresponding to each pixel in the first initial image and the imaging device of the first initial image, wherein the imaging device may be a monocular camera.

[0047] In at least one embodiment of this application, the deep recognition network includes convolutional layers and deconvolutional layers.

[0048] In at least one embodiment of this application, the electronic device performs depth recognition on the first initial image based on the depth recognition network to obtain an initial depth image, including:

[0049] The electronic device inputs the first initial image into the convolutional layer for convolution operation to obtain an initial feature map corresponding to the first initial image, and inputs the initial feature map into the deconvolutional layer for deconvolution operation to obtain a high-dimensional feature map. Further, the electronic device maps the distance between each pixel and the shooting device to the depth value of each pixel in the high-dimensional feature map. Furthermore, the electronic device generates the initial depth image based on each pixel and the pixel value of each pixel.

[0050] Through the above implementation method, since the depth value of each pixel can reflect the real distance between each pixel on the first initial image and the shooting device, the projection coordinates can be accurately calculated through the initial depth image.

[0051] 103. The first initial image is preprocessed to obtain a first static image and a first dynamic image corresponding to the first initial image, and the second initial image is preprocessed to obtain a second static image and a second dynamic image corresponding to the second initial image.

[0052] In at least one embodiment of this application, the first initial image comprises a plurality of initial objects.

[0053] In at least one embodiment of this application, the first static image refers to an image generated after masking a dynamic object in the first initial image, and the second static image refers to an image generated after masking a dynamic object in the second initial image. The dynamic object refers to a movable object, such as a pedestrian or a vehicle. The first dynamic image refers to an image generated after masking a static object in the first initial image, and the second dynamic image refers to an image generated after masking a static object in the second initial image. The static object refers to a non-movable object, such as a tree or the ground.

[0054] In at least one embodiment of this application, the electronic device preprocesses the first initial image to obtain a first static image and a first dynamic image corresponding to the first initial image, including:

[0055] The electronic device calculates a single score value for each pixel in the first initial image based on the pixel value of each pixel in the first initial image. Further, based on the single score value and multiple preset objects, the electronic device calculates the category probability of each pixel in the first initial image for each preset object. Even further, the electronic device determines the preset object corresponding to the category probability with the highest value as the pixel object corresponding to that pixel, and determines the pixel region formed by pixels of the same pixel object in the first initial image as the initial object. Even further, the electronic device classifies the initial objects according to preset rules to obtain dynamic objects corresponding to dynamic categories and static objects corresponding to static categories in the first initial image. Even further, the electronic device performs masking processing on the dynamic objects in the first initial image to obtain the first static image, and performs masking processing on the static objects in the first initial image to obtain the first dynamic image.

[0056] In at least one embodiment of this application, the preset rules determine initial objects such as vehicles, people, or animals as movable initial objects and assign them to the dynamic category. Initial objects such as plants or fixed objects are determined as immovable initial objects, and the corresponding initial categories of the immovable initial objects are assigned to the static category. For example, movable initial objects such as pedestrians, cats, dogs, bicycles, and cars are assigned to the dynamic category, while immovable initial objects such as trees, streetlights, and buildings are assigned to the static category.

[0057] Specifically, the formula for calculating the category probability is:

[0058]

[0059] Among them, S i This represents the probability that each pixel belongs to the category of the i-th preset object. z represents the single score value of the j-th pixel in the first initial image. j This represents the pixel value of the j-th pixel in the first initial image. The total score of all pixels in the first initial image is represented by , i represents the i-th preset object, and k represents the number of the plurality of preset objects.

[0060] In at least one embodiment of this application, the generation process of the second static image is basically the same as that of the first static image, and the generation process of the second dynamic image is basically the same as that of the first dynamic image, so this application will not elaborate further here.

[0061] Through the above implementation method, the pixel region formed by the pixels of the same pixel object in the first initial image is determined as the initial object, and the initial object is initially classified according to the preset rules. The positions of dynamic objects and static objects in the first initial image can be initially determined, so that the dynamic objects and static objects can be accurately masked by the position.

[0062] 104. Generate a pose absolute value matrix based on the first static image, the first dynamic image, the second static image, the second dynamic image and the pose network.

[0063] In at least one embodiment of this application, the pose absolute value matrix refers to a matrix generated from multiple pose absolute values, the pose absolute value refers to the absolute value of the difference between the elements in the static pose matrix and the corresponding elements in the dynamic pose matrix, the static pose matrix refers to a matrix generated based on the first static image and the second static image, and the dynamic pose matrix refers to a matrix generated based on the first dynamic image and the second dynamic image.

[0064] In at least one embodiment of this application, the electronic device generates a pose absolute value matrix based on the first static image, the first dynamic image, the second static image, the second dynamic image, and the pose network, including:

[0065] The electronic device inputs the first static image and the second static image into the pose network to obtain a static pose matrix, and inputs the first dynamic image and the second dynamic image into the pose network to obtain a dynamic pose matrix. Further, the electronic device subtracts each matrix element in the static pose matrix from the corresponding matrix element in the dynamic pose matrix to obtain a pose difference. Further still, the electronic device takes the absolute value of the pose difference to obtain the absolute pose value of each matrix element in the static pose matrix. Further still, the electronic device arranges the absolute pose values ​​according to the element position of each matrix element in the static pose matrix to obtain the absolute pose value matrix.

[0066] In this embodiment, the generation methods of the static pose matrix and the dynamic pose matrix are basically the same as those of the target pose matrix in the following text, so this application will not repeat them here.

[0067] Through the above implementation, since the static pose matrix contains the position and orientation of the static object, and the dynamic pose matrix contains the position and orientation of the dynamic object, the static pose matrix can accurately reflect the state of the static object, and the dynamic pose matrix can accurately reflect the state of the dynamic object. When the dynamic object does not move, the dynamic pose matrix is ​​basically the same as the static pose matrix. By using the calculation result of the absolute value of the pose and the corresponding threshold to determine whether the dynamic object has moved, the influence of reasonable error can be avoided.

[0068] 105. Input the first initial image and the second initial image into the pose network to obtain the target pose matrix.

[0069] In at least one embodiment of this application, the target pose matrix refers to the transformation relationship from camera coordinates to world coordinates for each pixel in the first initial image, and the camera coordinates for each pixel in the first initial image refer to the coordinates of each pixel in the camera coordinate system.

[0070] like Figure 3 The diagram shown is a schematic of the pixel coordinate system and camera coordinate system of the depth recognition method of this application. The electronic device uses the pixel point O in the first row and first column of the first initial image as the reference. uv A pixel coordinate system is constructed with the origin as the origin, the parallel line containing the first row of pixels as the u-axis, and the vertical line containing the first column of pixels as the v-axis. Furthermore, the electronic device uses the light spot O of the monocular camera... XY The camera coordinate system is constructed with the origin as the origin, the optical axis of the monocular camera as the Z-axis, the line parallel to the u-axis of the pixel coordinate system as the X-axis, and the line parallel to the v-axis of the pixel coordinate system as the Y-axis.

[0071] In at least one embodiment of this application, the electronic device inputs the first initial image and the second initial image into the pose network to obtain a target pose matrix, including:

[0072] The electronic device determines the pixel corresponding to the dynamic object in the first initial image as the first pixel. Further, the electronic device obtains the first homogeneous coordinate matrix of the first pixel and the second homogeneous coordinate matrix of the second pixel corresponding to the first pixel in the second initial image. Further, the electronic device obtains the inverse matrix of the intrinsic parameter matrix of the shooting device. Further still, the electronic device calculates the first camera coordinates of the first pixel based on the first homogeneous coordinate matrix and the inverse matrix of the intrinsic parameter matrix, and calculates the second camera coordinates of the second pixel based on the second homogeneous coordinate matrix and the inverse matrix of the intrinsic parameter matrix. Further still, the electronic device calculates the first camera coordinates and the second camera coordinates based on a preset epipolar constraint relationship to obtain a rotation matrix and a translation matrix, and concatenates the rotation matrix and the translation matrix to obtain the target pose matrix.

[0073] Wherein, the first homogeneous coordinate matrix of the first pixel is a matrix with one more dimension than the pixel coordinate matrix, and the element value of the extra dimension is 1. The pixel coordinate matrix is ​​a matrix generated based on the first pixel coordinates of the first pixel, where the first pixel coordinates refer to the coordinates of the first pixel in the pixel coordinate system. For example, if the first pixel coordinates of the first pixel in the pixel coordinate system are (v, v), then the pixel coordinate matrix of the first pixel is... Then the homogeneous coordinate matrix of the pixel is Multiply the first homogeneous coordinate matrix and the inverse of the intrinsic parameter matrix to obtain the first camera coordinates of the first pixel, and multiply the second homogeneous coordinate matrix and the inverse of the intrinsic parameter matrix to obtain the second camera coordinates of the second pixel.

[0074] The generation method of the second homogeneous coordinate matrix is ​​basically the same as that of the first homogeneous coordinate matrix, and will not be described in detail here.

[0075] The target rotation matrix can be represented as:

[0076]

[0077] Where pose is the target pose matrix, which is a 4x4 matrix; R is the rotation matrix, which is a 3x3 matrix; and t is the translation matrix, which is a 3x1 matrix.

[0078] The formulas for calculating the translation matrix and the rotation matrix are as follows:

[0079] K -1 p1(txR)(K-1 p2) T =0;

[0080] Among them, K -1 p1 is the coordinate of the first camera, K -1 p2 represents the coordinates of the second camera, p1 represents the first homogeneous coordinate matrix, p2 represents the second homogeneous coordinate matrix, and K represents the coordinates of the second camera. -1 is the inverse of the intrinsic parameter matrix.

[0081] Through the above implementation method, the two-dimensional pixel coordinates of each pixel in the first initial image and the second initial image are converted into three-dimensional camera coordinates in the camera coordinate system according to the camera intrinsic parameter matrix. The rotation matrix and translation matrix can be accurately calculated through the camera coordinates, thereby accurately generating the target pose matrix according to the rotation matrix and the translation matrix.

[0082] 106. Generate an initial projection image of the first initial image based on the first initial image, the initial depth image, and the target pose matrix.

[0083] In at least one embodiment of this application, the initial projected image represents an image of the transformation process, wherein the transformation process refers to the transformation process between the pixel coordinates of a pixel in the first initial image and the corresponding pixel coordinates in the second initial image.

[0084] In at least one embodiment of this application, the electronic device generates an initial projection image of the first initial image based on the first initial image, the initial depth image, and the target pose matrix, including:

[0085] The electronic device acquires the target homogeneous coordinate matrix of each pixel in the first initial image and obtains the depth value of each pixel in the first initial image from the initial depth image. Further, the electronic device calculates the projection coordinates of each pixel in the first initial image based on the target pose matrix, the target homogeneous coordinate matrix of each pixel, and the depth value of each pixel. Furthermore, the electronic device arranges each pixel according to the projection coordinates of each pixel to obtain the initial projection image.

[0086] The depth value refers to the pixel value of each pixel in the initial depth image.

[0087] Specifically, the formula for calculating the projection coordinates of each pixel in the initial projected image is as follows:

[0088] P = K * pose * Z * K -1 *H;

[0089] Where P represents the projected coordinates of each pixel, K represents the intrinsic parameter matrix of the imaging device, and pose represents the target pose matrix. -1 Let K be the inverse matrix, H be the target homogeneous coordinate matrix of each pixel in the first initial image, and Z be the depth value of the corresponding pixel in the initial depth image.

[0090] 107. Identify the target image of the first initial image and the target projection image of the initial projection image based on the pose absolute value matrix and the preset threshold matrix.

[0091] In at least one embodiment of this application, the preset threshold matrix refers to a matrix with the same dimension as the pose absolute value matrix that is set in advance, and the preset threshold matrix contains multiple thresholds.

[0092] In at least one embodiment of this application, the electronic device identifies the target image of the first initial image and the target projection image of the initial projection image based on the pose absolute value matrix and the preset threshold matrix, including:

[0093] The electronic device compares each pose absolute value in the pose absolute value matrix with the corresponding threshold in the preset threshold matrix. If there is at least one pose absolute value in the pose absolute value matrix that is greater than the corresponding threshold, the electronic device determines the first static image as the target image, identifies the dynamic position of the dynamic object in the first initial image, determines the region in the initial projection image corresponding to the dynamic position as the projection object, and performs masking processing on the projection object to obtain the target projection image. Alternatively, if each pose absolute value in the pose absolute value matrix is ​​less than or equal to the corresponding threshold, the electronic device determines the first initial image as the target image and the initial projection image as the target projection image.

[0094] Through the above implementation, when at least one pose absolute value is greater than the corresponding threshold, it is determined that the dynamic object has moved. The first static image is determined as the target image, and the dynamic object corresponding to the dynamic category in the initial projection image is masked. Since the position of the dynamic object corresponding to the dynamic category has changed, the depth value of the pixel corresponding to the dynamic object has changed. The depth value is not used in the calculation of the loss value, which can avoid the influence of the moving dynamic object on the calculation of the loss value. When all pose absolute values ​​in the pose absolute value matrix are less than or equal to the corresponding threshold, it is determined that the dynamic object has not moved. The first initial image is determined as the target image, and the initial projection image is determined as the target projection image, which can accurately calculate the loss value.

[0095] 108. Based on the gradient error between the initial depth image and the target image and the photometric error between the target projection image and the target image, the depth recognition network is adjusted to obtain a depth recognition model.

[0096] In at least one embodiment of this application, the deep recognition model refers to a model generated after adjusting the deep recognition network.

[0097] In at least one embodiment of this application, the electronic device adjusts the depth recognition network based on the gradient error between the initial depth image and the target image and the photometric error between the target projection image and the target image to obtain a depth recognition model, including:

[0098] The electronic device calculates the depth loss value of the depth recognition network based on the gradient error and the photometric error. Further, the electronic device adjusts the depth recognition network based on the depth loss value until the depth loss value is reduced to the minimum, thereby obtaining the depth recognition model.

[0099] Specifically, the formula for calculating the depth loss value is as follows:

[0100] Lc = Lt + Ls;

[0101] Lc represents the depth loss value, Lt represents the photometric error, and Ls represents the gradient error.

[0102] The formula for calculating the photometric error is as follows:

[0103]

[0104] Where Lt represents the photometric error, α is a preset balance parameter, typically set to 0.85, and SSIM(x,y) represents the structural similarity index between the target projection image and the target image, ||x i -y i || represents the grayscale difference between the target projection image and the target image, x i y represents the pixel value of the i-th pixel in the target projection image. i This represents the pixel value of the pixel corresponding to the i-th pixel in the target image.

[0105] The formula for calculating the structural similarity index is as follows:

[0106]

[0107] c1 = (K1L) 2 ;

[0108] c2 = (K2L) 2 ;

[0109] Where SSIM(x,y) is the structural similarity index, x is the target projected image, y is the target image, and μ x The grayscale average value of the target projected image, μ y σ is the average grayscale value of the target image. x σ is the grayscale standard deviation of the target projected image. y Let σ be the grayscale standard deviation of the target image. xy Let c1 and c2 be the grayscale covariance between the target projection image and the target image, where c1 and c2 are preset parameters, L is the maximum pixel value in the target image, and K1 and K2 are preset constants, with K1 << 1 and K2 << 1.

[0110] The formula for calculating the gradient error is:

[0111]

[0112] Where Ls represents the gradient error, x represents the initial depth image, y represents the target image, D(u, v) represents the pixel coordinates of the i-th pixel in the initial depth image, and I(u, v) represents the pixel coordinates of the i-th pixel in the target image.

[0113] By implementing the above methods, the accuracy of the deep recognition model can be improved because the influence of moving dynamic objects on the calculation of the loss value of the deep recognition network is avoided.

[0114] 109. Input the image to be identified into the depth recognition model to obtain the target depth image of the image to be identified and the depth information of the image to be identified.

[0115] In at least one embodiment of this application, the target depth image refers to an image containing depth information of each pixel in the image to be identified, and the depth information of each pixel in the image to be identified refers to the distance between the object to be identified corresponding to each pixel in the image to be identified and the shooting device.

[0116] In at least one embodiment of this application, the method of generating the target depth image is basically the same as the method of generating the initial depth image, so this application will not elaborate further.

[0117] In at least one embodiment of this application, the electronic device acquires the pixel value of each pixel in the target depth image as the depth information of the corresponding pixel in the image to be identified.

[0118] By implementing the above methods, the accuracy of depth recognition model is improved, thereby enhancing the accuracy of depth recognition of the image to be recognized.

[0119] In summary, this application preprocesses the first initial image to accurately identify dynamic and static objects within it. When at least one pose absolute value is greater than a corresponding threshold, the dynamic object is determined to have moved. When all pose absolute values ​​in the pose absolute value matrix are less than or equal to the corresponding threshold, the dynamic object is determined not to have moved. Therefore, masking of non-moving dynamic objects in the initial image is avoided. When a dynamic object in the first initial image moves, it is masked to obtain a target image. Similarly, the dynamic object in the initial projection image is masked to obtain the target projection image. Based on the gradient error between the initial depth image and the target image, and the photometric error between the target projection image and the target image, the depth recognition network is adjusted to obtain a depth recognition model. Since adjusting the deep neural network based on the gradient error and the photometric error avoids masking of non-moving dynamic objects in the initial image, the accuracy of the depth recognition model is improved, thereby enhancing the precision of image depth recognition.

[0120] like Figure 4 The diagram shown is a schematic representation of the structure of an electronic device that implements the image depth recognition method of this application.

[0121] In one embodiment of this application, the electronic device 1 includes, but is not limited to, a memory 12, a processor 13, and a computer program, such as a depth recognition program, stored in the memory 12 and executable on the processor 13.

[0122] Those skilled in the art will understand that the schematic diagram is merely an example of electronic device 1 and does not constitute a limitation on electronic device 1. It may include more or fewer components than shown in the diagram, or combine certain components, or different components. For example, electronic device 1 may also include input / output devices, network access devices, buses, etc.

[0123] The processor 13 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 13 is the computing core and control center of the electronic device 1, connecting various parts of the electronic device 1 through various interfaces and lines, and acquiring the operating system and installed applications and program code of the electronic device 1. For example, the processor 13 can acquire the image to be identified captured by the imaging device 2 through an interface.

[0124] The processor 13 acquires the operating system and various installed applications of the electronic device 1. The processor 13 acquires these applications to implement the steps in the above-described embodiments of the image depth recognition method, for example... Figure 2 The steps are shown.

[0125] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and retrieved by the processor 13 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the process of retrieving the computer program from the electronic device 1.

[0126] The memory 12 can be used to store the computer programs and / or modules. The processor 13 implements various functions of the electronic device 1 by running or retrieving the computer programs and / or modules stored in the memory 12, and by calling the data stored in the memory 12. The memory 12 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 12 may include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.

[0127] The memory 12 can be the external memory and / or internal memory of the electronic device 1. Furthermore, the memory 12 can be a physical memory, such as a memory module, a TF card (Trans-flash Card), etc.

[0128] If the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when the computer program is acquired by a processor, it can implement the steps of the various method embodiments described above.

[0129] The computer program includes computer program code, which may be in the form of source code, object code, accessible file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, and read-only memory (ROM).

[0130] Combination Figure 2The memory 12 in the electronic device 1 stores multiple instructions to implement an image depth recognition method. The processor 13 can acquire the multiple instructions to implement: acquiring an image to be recognized, a first initial image, and a second initial image, and acquiring a depth recognition network and a pose network; performing depth recognition on the first initial image based on the depth recognition network to obtain an initial depth image; preprocessing the first initial image to obtain a first static image and a first dynamic image corresponding to the first initial image, and preprocessing the second initial image to obtain a second static image and a second dynamic image corresponding to the second initial image; and based on the first static image, the first dynamic image, the second static image, the second dynamic image, and the pose network... Generate an absolute pose matrix; input the first initial image and the second initial image into the pose network to obtain a target pose matrix; generate an initial projection image of the first initial image based on the first initial image, the initial depth image, and the target pose matrix; identify the target image of the first initial image and the target projection image of the initial projection image according to the absolute pose matrix and a preset threshold matrix; adjust the depth recognition network based on the gradient error between the initial depth image and the target image and the photometric error between the target projection image and the target image to obtain a depth recognition model; input the image to be recognized into the depth recognition model to obtain the target depth image of the image to be recognized and the depth information of the image to be recognized.

[0131] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 2 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0132] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0133] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0134] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0135] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0136] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in this application may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. An image depth recognition method, applied to electronic devices, characterized in that, The image depth recognition method includes: Acquire the image to be recognized, the first initial image, and the second initial image, and acquire the depth recognition network and the pose network; Based on the depth recognition network, depth recognition is performed on the first initial image to obtain an initial depth image; The first initial image is preprocessed to obtain a first static image and a first dynamic image corresponding to the first initial image, and the second initial image is preprocessed to obtain a second static image and a second dynamic image corresponding to the second initial image. Generate a pose absolute value matrix based on the first static image, the first dynamic image, the second static image, the second dynamic image, and the pose network; The first initial image and the second initial image are input into the pose network to obtain the target pose matrix; An initial projection image of the first initial image is generated based on the first initial image, the initial depth image, and the target pose matrix; Identify the target image of the first initial image and the target projection image of the initial projection image based on the pose absolute value matrix and the preset threshold matrix; Based on the gradient error between the initial depth image and the target image and the photometric error between the target projection image and the target image, the depth recognition network is adjusted to obtain a depth recognition model, including: calculating the depth loss value of the depth recognition network based on the gradient error and the photometric error, adjusting the depth recognition network based on the depth loss value until the depth loss value is reduced to the minimum, and obtaining the depth recognition model; The image to be identified is input into the depth recognition model to obtain the target depth image of the image to be identified and the depth information of the image to be identified.

2. The image depth recognition method as described in claim 1, characterized in that, The step of preprocessing the first initial image to obtain a first static image and a first dynamic image corresponding to the first initial image includes: Calculate a single score value for each pixel in the first initial image based on the pixel value of each pixel in the first initial image; Based on the single score value and multiple preset objects, calculate the category probability of each pixel in the first initial image on each preset object; The preset object corresponding to the category probability with the highest value is determined as the pixel object corresponding to this pixel point; The pixel region formed by the pixels of the same pixel object in the first initial image is determined as the initial object; The initial objects are classified according to preset rules to obtain dynamic objects corresponding to the dynamic category and static objects corresponding to the static category in the first initial image; The dynamic object in the first initial image is masked to obtain the first static image; The static object in the first initial image is masked to obtain the first dynamic image.

3. The image depth recognition method as described in claim 1, characterized in that, The generation of the pose absolute value matrix based on the first static image, the first dynamic image, the second static image, the second dynamic image, and the pose network includes: The first static image and the second static image are input into the pose network to obtain a static pose matrix; The first dynamic image and the second dynamic image are input into the pose network to obtain a dynamic pose matrix; Subtract the corresponding element in the dynamic pose matrix from each element in the static pose matrix to obtain the pose difference. The absolute value of the pose difference is taken to obtain the absolute pose value of each matrix element in the static pose matrix; Based on the position of each element in the static pose matrix, the absolute pose values ​​are arranged to obtain the absolute pose value matrix.

4. The image depth recognition method as described in claim 2, characterized in that, The first initial image and the second initial image are images captured by the same imaging device. The step of inputting the first initial image and the second initial image into the pose network to obtain the target pose matrix includes: The pixel corresponding to the dynamic object in the first initial image is determined as the first pixel. Obtain the first homogeneous coordinate matrix of the first pixel, and obtain the second homogeneous coordinate matrix of the second pixel corresponding to the first pixel in the second initial image; Obtain the inverse matrix of the intrinsic parameter matrix of the shooting device; The first camera coordinates of the first pixel are calculated based on the first homogeneous coordinate matrix and the inverse of the intrinsic parameter matrix, and the second camera coordinates of the second pixel are calculated based on the second homogeneous coordinate matrix and the inverse of the intrinsic parameter matrix. The rotation matrix and translation matrix are obtained by calculating the coordinates of the first camera and the second camera based on the preset epipolar constraint relationship. The rotation matrix and the translation matrix are concatenated to obtain the target pose matrix.

5. The image depth recognition method as described in claim 2, characterized in that, The step of identifying the target image of the first initial image and the target projection image of the initial projection image based on the pose absolute value matrix and the preset threshold matrix includes: Each absolute value of the pose in the pose absolute value matrix is ​​compared with the corresponding threshold in the preset threshold matrix; If the pose absolute value matrix contains at least one pose absolute value greater than the corresponding threshold, then the first static image is determined as the target image, the dynamic position of the dynamic object in the first initial image is identified, the region in the initial projection image corresponding to the dynamic position is determined as the projection object, and the projection object is masked to obtain the target projection image; or If the absolute value of each pose in the pose absolute value matrix is ​​less than or equal to the corresponding threshold, the first initial image is determined as the target image, and the initial projected image is determined as the target projected image.

6. The image depth recognition method as described in claim 1, characterized in that, The process of generating the initial projection image of the first initial image based on the first initial image, the initial depth image, and the target pose matrix includes: Obtain the target homogeneous coordinate matrix of each pixel in the first initial image, and obtain the depth value of each pixel in the first initial image from the initial depth image; Based on the target pose matrix, the target homogeneous coordinate matrix of each pixel, and the depth value of each pixel, the projected coordinates of each pixel in the first initial image are calculated. The initial projected image is obtained by arranging each pixel according to its projected coordinates.

7. The image depth recognition method as described in claim 1, characterized in that, The formula for calculating the photometric error is: in, This indicates the photometric error. The preset balance parameters, The structural similarity index represents the relationship between the projected image of the target and the target image. This represents the grayscale difference between the target projection image and the target image. Indicates the first in the target projection image The pixel value of each pixel. Indicates that the target image is related to the first The pixel value of the pixel corresponding to each pixel.

8. An electronic device, characterized in that, The electronic device includes: Memory, storing at least one instruction; and The processor executes the at least one instruction to implement the image depth recognition method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which is executed by a processor in an electronic device to implement the image depth recognition method as described in any one of claims 1 to 7.

Images