Spatial three-dimensional structure information recovery method, model training method and device

Abstract

The invention relates to the technical field of computer vision, and provides a spatial three-dimensional structure information recovery method and device and a model training method and device, and a spatial three-dimensional structure recovery model comprises an encoder, a decoder and a back projection layer. The model training method comprises the following steps: performing feature extraction on a to-be-trained target scene image shot by a color camera through an encoder; estimating from the extracted features through a decoder to obtain an association graph obtained by associating the scale depth map and the instance segmentation map, and back-projecting the association graph to a point cloud space through a back-projection layer to obtain predicted spatial three-dimensional structure information; acquiring spatial three-dimensional structure information serving as a supervision signal; and determining a value of a loss function of the spatial three-dimensional structure recovery model based on the predicted spatial three-dimensional structure information and the spatial three-dimensional structure information serving as the supervision signal, and updating model parameters according to the value. According to the method, the three-dimensional space structure information can be recovered by inputting one color image to the model, and the cost is low.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method for recovering spatial three-dimensional structure information, a model training method and a device. Background Art

[0002] The three-dimensional structure information of space is very important for augmented reality (AR), virtual reality (VR) and some other tasks that require the use of the three-dimensional structure information of space, such as depth-based camera autofocus and spatial perception-based microphone array initial parameter estimation. The image captured by a color camera is a two-dimensional signal obtained by perspective projection of the three-dimensional space, so the three-dimensional structure information of space cannot be directly obtained from it.

[0003] At present, the three-dimensional structure restoration scheme of indoor space mainly uses depth sensors such as TOF (Time-of-Fl igh) and structured light to generate the depth information of the space, and uses color cameras to generate RGB color information, and calibrates and aligns the above two information. Finally, the depth information is back-projected in combination with the internal parameters of the camera to generate a point cloud of the three-dimensional structure of the space. However, the cost of equipment equipped with depth sensors is relatively high, and in industrial scenarios, the cost of components is often very sensitive. Therefore, there is an urgent need for a solution that can achieve three-dimensional structure restoration of indoor space and is relatively low-cost. Summary of the invention

[0004] In response to the above technical problems, the purpose of this application is to provide a method for recovering spatial three-dimensional structure information, a model training method and a device, aiming to solve the technical problem that the existing indoor space three-dimensional structure recovery requires the use of equipment equipped with depth sensors, resulting in relatively high costs.

[0005] In a first aspect, an embodiment of the present application provides a method for training a spatial three-dimensional structure information recovery model, wherein the spatial three-dimensional structure recovery model includes an encoder, a decoder, and a back-projection layer, and the method includes:

[0006] Extracting features from the target scene image to be trained taken by the color camera by using the encoder;

[0007] The decoder estimates the associated map obtained by associating the scaled depth map and the instance segmentation map from the extracted features.

[0008] Back-projecting the association graph into the point cloud space through the back-projection layer to obtain predicted spatial three-dimensional structure information;

[0009] Obtain spatial three-dimensional structure information as a supervisory signal;

[0010] Determining a value of a loss function of the spatial three-dimensional structure restoration model based on the predicted spatial three-dimensional structure information and the spatial three-dimensional structure information as a supervisory signal;

[0011] The model parameters of the spatial three-dimensional structure restoration model are updated based on the value of the loss function.

[0012] Furthermore, the step of back-projecting the association graph into the point cloud space through the back-projection layer to obtain predicted spatial three-dimensional structure information includes:

[0013] Obtaining internal parameters of the color camera;

[0014] Performing distortion correction on the correlation graph using the internal parameters to obtain a distortion-corrected correlation graph;

[0015] Based on the pixel coordinates of the distortion-corrected association map, the internal parameters and the depth information of the scaled depth map, the distortion-corrected association map is back-projected into the point cloud space using a back-projection formula to obtain predicted spatial three-dimensional structure information.

[0016] Furthermore, the spatial three-dimensional structure information includes the three-dimensional coordinates of the instance point cloud and the instance label of the instance point cloud, and the step of determining the value of the loss function of the spatial three-dimensional structure restoration model based on the predicted spatial three-dimensional structure information and the spatial three-dimensional structure information as the supervisory signal includes:

[0017] Determine a first loss value based on the predicted three-dimensional coordinates of the instance point cloud and the three-dimensional coordinates of the instance point cloud as a supervisory signal;

[0018] Determine a second loss value based on the instance label of the predicted instance point cloud and the instance label of the instance point cloud as the supervisory signal;

[0019] The value of the loss function of the spatial three-dimensional structure restoration model is determined based on the first loss value and the second loss value.

[0020] Furthermore, before the step of obtaining the spatial three-dimensional structure information as a supervisory signal, the method further includes:

[0021] Acquire the target scene image to be processed taken by a color camera and the internal parameters of the color camera;

[0022] Predicting a scaled depth map corresponding to the target scene image to be processed;

[0023] Performing instance segmentation on the target scene image to be processed to obtain an instance segmentation map corresponding to the target scene image to be processed;

[0024] According to the pixel coordinate position, the scaled depth map corresponding to the target scene image to be processed and the instance segmentation map corresponding to the target scene image to be processed are associated to obtain an association map corresponding to the target scene image to be processed;

[0025] Based on the internal parameters, the association graph corresponding to the target scene image to be processed is back-projected into the point cloud space to obtain preliminary spatial three-dimensional structure information;

[0026] The preliminary spatial three-dimensional structure information is stored as the spatial three-dimensional structure information used as a supervisory signal.

[0027] Furthermore, before the step of obtaining the spatial three-dimensional structure information as a supervisory signal, the step includes:

[0028] Acquire the target scene image to be processed taken by the color camera and the internal parameters of the color camera;

[0029] Predicting a scaled depth map corresponding to the target scene image to be processed;

[0030] Performing instance segmentation on the target scene image to be processed to obtain an instance segmentation map corresponding to the target scene image to be processed;

[0031] According to the pixel coordinate position, the scaled depth map corresponding to the target scene image to be processed and the instance segmentation map corresponding to the target scene image to be processed are associated to obtain an association map;

[0032] Based on the internal parameters, the association graph corresponding to the target scene image to be processed is back-projected into the point cloud space to obtain preliminary spatial three-dimensional structure information;

[0033] According to the instance labels in the instance segmentation map corresponding to the target scene image to be processed, a point cloud plane fitting method is used to perform plane fitting on the point clouds of each plane category in the preliminary spatial three-dimensional structure information to obtain optimized spatial three-dimensional structure information;

[0034] The optimized spatial three-dimensional structure information is stored as the spatial three-dimensional structure information used as a supervisory signal.

[0035] Furthermore, the step of predicting a scaled depth map corresponding to the target scene image to be processed includes:

[0036] Inputting the target scene image to be processed into a first depth estimation model, estimating scale-free depth information of the target scene image to be processed by the first depth estimation model, and obtaining a scale-free depth map;

[0037] The scale of the scale-free depth map is restored by using a priori calibration objects to obtain a scaled depth map corresponding to the target scene image to be processed.

[0038] In a second aspect, an embodiment of the present application provides a method for restoring spatial three-dimensional structure information, comprising:

[0039] Acquire the target scene image to be analyzed captured by a color camera;

[0040] The target scene image to be analyzed is input into a spatial three-dimensional structure information recovery model, and the spatial three-dimensional structure information of the target scene is recovered by the spatial three-dimensional structure information recovery model; wherein the spatial three-dimensional structure information recovery model is trained using the spatial three-dimensional structure information recovery model training method described in any one of claims 1 to 6.

[0041] In a third aspect, an embodiment of the present application provides a spatial three-dimensional structure information recovery model training device, wherein the spatial three-dimensional structure recovery model includes an encoder, a decoder, and a back-projection layer, and the device includes:

[0042] A feature extraction module, used for extracting features from the target scene image to be trained obtained by taking a color camera through the encoder;

[0043] A correlation map estimation module, used to estimate the correlation map obtained by associating the scaled depth map and the instance segmentation map from the extracted features through the decoder

[0044] A back-projection module, used for back-projecting the association map into a point cloud space through the back-projection layer to obtain predicted spatial three-dimensional structure information;

[0045] A spatial three-dimensional structure information acquisition module, used to acquire spatial three-dimensional structure information as a supervisory signal;

[0046] A loss value determination module, used to determine the value of the loss function of the spatial three-dimensional structure restoration model based on the predicted spatial three-dimensional structure information and the spatial three-dimensional structure information as a supervisory signal;

[0047] A model parameter updating module is used to update the model parameters of the spatial three-dimensional structure restoration model based on the value of the loss function.

[0048] In a fourth aspect, an embodiment of the present application provides a spatial three-dimensional structure information recovery device, comprising:

[0049] An image acquisition module is used to acquire the target scene image to be processed taken by a color camera;

[0050] A spatial three-dimensional structure information recovery module is used to input the target scene image to be processed into a spatial three-dimensional structure information recovery model, and recover the spatial three-dimensional structure information of the target scene through the spatial three-dimensional structure information recovery model; wherein, the spatial three-dimensional structure information recovery model is trained using any of the spatial three-dimensional structure information recovery model training methods described above.

[0051] In a fifth aspect, an embodiment of the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any of the methods described above are implemented.

[0052] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the methods described above.

[0053] The embodiment of the present invention provides a method for training a spatial three-dimensional structure information recovery model, the spatial three-dimensional structure recovery model includes an encoder, a decoder and a back-projection layer, and the method includes: extracting features of a target scene image to be trained obtained by a color camera through the encoder; estimating a correlation map obtained by associating a scaled depth map and an instance segmentation map from the extracted features through the decoder, back-projecting the correlation map to the point cloud space through the back-projection layer to obtain predicted spatial three-dimensional structure information; obtaining spatial three-dimensional structure information as a supervisory signal; determining the value of the loss function of the spatial three-dimensional structure recovery model based on the predicted spatial three-dimensional structure information and the spatial three-dimensional structure information as a supervisory signal; and updating the model parameters of the spatial three-dimensional structure recovery model based on the value of the loss function. The embodiment of the present invention trains a spatial three-dimensional structure information recovery model, so that a target scene image obtained by a color camera is input into the trained spatial three-dimensional structure information recovery model to recover its spatial three-dimensional structure information, that is, end-to-end three-dimensional structure information output is realized, which greatly reduces the cost and has a wider range of use compared to the recovery of spatial three-dimensional structure information based on hardware such as TOF and structured light, and has great advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the implementation manner will be briefly introduced below. Obviously, the drawings described below are only some implementation manners of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 It is a flowchart of a spatial three-dimensional structure information recovery model training method provided in an embodiment of the present application;

[0056] Figure 2 It is a structural schematic diagram of a spatial three-dimensional structure information recovery model provided in an embodiment of the present application;

[0057] Figure 3 It is a structural schematic diagram of a spatial three-dimensional structure information recovery model training device provided in an embodiment of the present application;

[0058] Figure 4 It is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "above", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of features, integers, steps, operations, elements, modules, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, components, and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may also be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any module and all combinations of one or more associated listed items.

[0061] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0062] Embodiment 1:

[0063] like Figure 1 , Figure 2 As shown, an embodiment of the present invention provides a spatial three-dimensional structure information recovery model training method, the spatial three-dimensional structure recovery model includes an encoder, a decoder and a back projection layer, and the method includes steps S1 to S6:

[0064] S1, extracting features from the target scene image to be trained obtained by taking a color camera through the encoder;

[0065] S2, estimating, by the decoder, from the extracted features, a correlation map obtained by associating the scaled depth map and the instance segmentation map

[0066] S3, back-projecting the association graph into the point cloud space through the back-projection layer to obtain predicted spatial three-dimensional structure information;

[0067] S4, obtaining spatial three-dimensional structure information as a supervisory signal;

[0068] S5. Determine a value of a loss function of the spatial three-dimensional structure restoration model based on the predicted spatial three-dimensional structure information and the spatial three-dimensional structure information as a supervisory signal;

[0069] S6. Update the model parameters of the spatial three-dimensional structure restoration model based on the value of the loss function.

[0070] As in the above steps S1 to S6, the color camera may be a monocular camera, and the target scene image to be trained obtained by the color camera is a color image, and the color image refers to an image in which each pixel is composed of R (red), G (blue), and B (green) components. Generally, the target scene image is an indoor scene image, that is, the spatial three-dimensional structure information recovery model training method provided in the embodiment of the present invention can be specifically a spatial three-dimensional structure information recovery model training method. The target scene image to be trained refers to the target scene image used for training. By initializing the model parameters of the encoder a and the decoder b, the encoder and decoder to be trained are obtained. Based on the input image, the initial depth map and the initial instance segmentation map corresponding to the input image can be obtained by the encoder and decoder to be trained. Associating the depth map with the instance segmentation map is to associate the depth information and instance information corresponding to the depth map and the instance segmentation map, that is, the depth information and instance information of each pixel in the association map are one-to-one corresponding. It should be understood that the sizes of the target scene image, the depth map, and the instance segmentation map are the same.

[0071] The model composed of the initialized encoder, decoder and back-projection layer can be regarded as the spatial three-dimensional structure information recovery model to be trained. By training the spatial three-dimensional structure information recovery model to be trained, a trained spatial three-dimensional structure recovery model can be obtained when the model training is completed. It should be noted that the trained spatial three-dimensional structure recovery model is a multi-task model.

[0072] The embodiment of the present invention trains a spatial three-dimensional structure information recovery model. In this way, a target scene image taken by a color camera is input into the trained spatial three-dimensional structure information recovery model to recover its spatial three-dimensional structure information, thereby realizing end-to-end three-dimensional structure information output. In addition, the spatial three-dimensional structure information recovery model of the embodiment of the present application has a streamlined structure, which greatly reduces the cost and has a wider range of applications compared to hardware-based spatial three-dimensional structure information recovery based on TOF, structured light, etc., and has great advantages.

[0073] In one embodiment, the step of back-projecting the association map into the point cloud space through the back-projection layer to obtain predicted spatial three-dimensional structure information includes steps S31-S33:

[0074] S31, obtaining internal parameters of the color camera;

[0075] S32, performing distortion correction on the correlation graph using the internal parameters to obtain a distortion-corrected correlation graph;

[0076] S33. Based on the pixel coordinates of the distortion-corrected association map, the internal parameters, and the depth information of the scaled depth map, the distortion-corrected association map is back-projected into the point cloud space using a back-projection formula to obtain predicted spatial three-dimensional structure information.

[0077] In the embodiment of the present application, the distortion correction of the correlation graph can be performed using opencv and other public functions, such as using the undistort function to correct image distortion. Suppose the correlation graph after distortion correction is P', then P' = cv2.undistort(P, np.array([k1, k2, k3, p1, p2]), np.array(K)); np.array represents matrixing the array, and K is the internal parameter of the camera. Different cameras have different internal parameters. The internal parameter K of the pinhole camera includes f x , f y , c x , c y , k1, k2, k3, p1 and p2; among them, f x Indicates the horizontal focal length of the camera, f y Indicates the vertical focal length of the camera, c x represents the horizontal center of the camera, c y represents the vertical center of the camera, k1, k2, k3 represent the radial distortion coefficients of the lens, and p1, p2 represent the tangential distortion coefficients of the lens. The internal parameter K of the fisheye camera is f x , f y , c x , c y , f1, f2, f3, f4; among them, fx Indicates the horizontal focal length of the camera, f y Indicates the vertical focal length of the camera, c x represents the horizontal center of the camera, c y represents the vertical center of the camera, and f1, f2, f3, and f4 represent the distortion coefficients of the fisheye lens.

[0078] In the embodiments of the present application, the back-projection process varies depending on the camera.

[0079] When a pinhole camera is used, the back-projection formula (1) is used

[0080]

[0081] The distortion-corrected association map is back-projected into the point cloud space to obtain preliminary spatial three-dimensional structure information. Specifically, the coordinates in the distortion-corrected association map P are substituted into the back-projection formula (1) where u, v, u, v represent the horizontal and vertical coordinates of the pixels of the image respectively, the corresponding depth information is substituted into D(u, v), and the internal parameters are substituted into K, then (x', y', z') can be calculated, where (x', y', z') represents the coordinates of the pixel points at the positions u and v transformed into the three-dimensional space. Since the scaled depth map D and the instance segmentation map C are associated, the predicted spatial three-dimensional structure information contains semantic information, and the predicted spatial three-dimensional structure information is J(i) = (x', y', z', c'); where c' is the predicted instance label at the same pixel coordinate, and i represents the number of the point cloud.

[0082] When a fisheye camera is used, the distortion-corrected correlation map is back-projected into the point cloud space using the back-projection formula (2) to obtain preliminary spatial three-dimensional structure information.

[0083]

[0084]

[0085]

[0086]

[0087] θ * =d -1 (dθ)

[0088] d(θ)=θ+f1θ 3 +f2θ 5 +f3θ 7 +f4θ 9

[0089] Specifically, substitute the coordinates in the distortion-corrected correlation map P' into u and v in the back-projection formula (2), where u and v represent the horizontal and vertical coordinates of the pixels in the image, respectively. Substitute the corresponding depth information into D(u, v), and substitute the horizontal focal length of the camera f x , the vertical focal length of the camera f y , the horizontal center of the camera c x , the vertical center of the camera c y , and the fisheye lens distortion coefficients f1, f2, f3, f4 are substituted into the above formula to obtain (x', y', z'), (x', y', z') represents the coordinates of the pixel point at the u and v positions transformed into the three-dimensional space. x , m y Represent the normalized coordinates of the pixels in the matrix in the horizontal and vertical coordinates, respectively, and are intermediate variables. In addition, the camera is a monocular camera. It should be noted that since the scale depth map D and the instance segmentation map C are associated, the predicted spatial three-dimensional structure information contains semantic information, and the predicted spatial three-dimensional structure information is J(i) = (x', y', z', c'); where c' is the predicted instance label at the same pixel coordinate, and i represents the number of the point cloud.

[0090] The embodiments of the present application are adaptable to various types of color cameras, including pinhole cameras and fisheye cameras, and are widely used.

[0091] In one embodiment, the spatial three-dimensional structure information includes the three-dimensional coordinates of the instance point cloud and the instance label of the instance point cloud, and the step of determining the value of the loss function of the spatial three-dimensional structure restoration model based on the predicted spatial three-dimensional structure information and the spatial three-dimensional structure information as a supervisory signal includes steps S61-S63:

[0092] S61, determining a first loss value based on the predicted three-dimensional coordinates of the instance point cloud and the three-dimensional coordinates of the instance point cloud as a supervisory signal;

[0093] S62, determining a second loss value based on the instance label of the predicted instance point cloud and the instance label of the instance point cloud serving as a supervisory signal;

[0094] S63. Determine a value of a loss function of the spatial three-dimensional structure restoration model based on the first loss value and the second loss value.

[0095] As in step S61 above, the three-dimensional coordinates of the predicted instance point cloud are (x', y', z'), and the three-dimensional coordinates of the instance point cloud serving as the supervisory signal are (x, y, z). It should be understood that the three-dimensional coordinates of the instance point cloud serving as the supervisory signal are (x, y, z), which is the true value. The first loss value is calculated using any one of the SmoothsL1 loss function, the L1 loss function, or the L2 loss function. It should be noted that the SmoothsL1 loss function can converge faster than the L1 loss function and is more stable when trained than the L2 loss function. Therefore, it is preferred to use the SmoothsL1 loss function to determine the first loss value. Specifically, SmoothL1 is used for the x, y, and z respectively, and the mean is calculated as the loss value of the regression term. The loss value of the regression term is the first loss value. As shown in formula (3):

[0096]

[0097] Among them, LossRegresion represents the loss value of the regression term, and the SmoothsL1 loss function is shown in formula (4):

[0098]

[0099] Among them, a1 represents the true value and a2 represents the model predicted value.

[0100] If the L1 loss function is used, the first loss value is calculated using formula (5), which is:

[0101]

[0102] Among them, the L1 loss function is shown in formula (6):

[0103] L1(a1,a2)=|a1-a2| (6)

[0104] Among them, a1 represents the true value and a2 represents the model predicted value.

[0105] If the L2 loss function is used, the first loss value is calculated using formula (7), which is:

[0106]

[0107] Among them, the L2 loss function is shown in formula (8):

[0108] L2(a1,a2)=(a1-a2) 2 (8)

[0109] Among them, a1 represents the true value and a2 represents the model predicted value.

[0110] As in step S62 above, the instance label c is trained using the cross entropy loss function as the classification item loss. Assuming that the instance label of the predicted instance point cloud is c', and the instance label of the instance point cloud as the supervisory signal is c, the cross entropy loss function is specifically used, that is, formula (9) is used to calculate the classification item loss value, that is, the second loss value:

[0111] LossClassification=-[clogc'+(1-c)log(1-c')] (9)

[0112] As in step S63 above, the regression term loss and the classification term loss are weighed by the λ factor, as shown in formula (8). Formula (10) is the loss function of the spatial three-dimensional structure restoration model:

[0113] loss=λ*LossRegresion+(1-λ)LossClassification (10)

[0114] Among them, λ can be set to 0.5, and loss is the loss value of the spatial three-dimensional structure recovery model.

[0115] As shown in step S63 above, the loss function of the embodiment of the present invention supervises both the geometric structure of the point cloud and the instance labels, so that the training target is located in the three-dimensional point cloud space, which is closer to the learning target, thereby improving the accuracy of the spatial three-dimensional structure output by the model.

[0116] In one embodiment, before the step of obtaining the spatial three-dimensional structure information as a supervisory signal, steps S71-S76 are also included:

[0117] S71, acquiring a target scene image to be processed taken by a color camera and internal parameters of the color camera;

[0118] S72, predicting a scaled depth map corresponding to the target scene image to be processed;

[0119] S73, performing instance segmentation on the target scene image to be processed to obtain an instance segmentation map corresponding to the target scene image to be processed;

[0120] S74, according to the pixel coordinate position, associating the scaled depth map corresponding to the target scene image to be processed and the instance segmentation map corresponding to the target scene image to be processed to obtain an association map;

[0121] S75, based on the internal parameters, back-projecting the association graph corresponding to the target scene image to be processed into the point cloud space to obtain preliminary spatial three-dimensional structure information;

[0122] S76. Store the preliminary three-dimensional spatial structure information as the three-dimensional spatial structure information used as a supervisory signal.

[0123] As in the above step S71, the color camera may be a common pinhole camera, a fisheye camera, or other cameras, which is not limited in the embodiment of the present application.

[0124] As in step S72 above, the scaled depth map corresponding to the target scene image to be processed is the scaled depth map of the target scene. The pixel value of the depth map represents the depth, and the depth represents the distance between a certain point in the scene and the camera.

[0125] As shown in step S73 above, according to the different segmentation granularities, image segmentation is divided into semantic segmentation and instance segmentation. Semantic segmentation refers to classification at the pixel level, and pixels of the same category are divided into the same category. Compared with the use of box annotations in target detection, semantic segmentation is more refined. Instance segmentation is more detailed than semantic segmentation, and can distinguish objects of the same category but belonging to different individuals. The instance segmentation map is composed of instance-level mask labels of each thing, and the instance-level mask labels of things are usually called instance labels. It should be noted that the embodiment of the present application does not limit the order of steps that have no logical relationship. For example, step S73 can be executed before step S72, or after step S72 is executed, step S73 can be executed.

[0126] As in step S74 above, if the scaled depth map is D, the instance segmentation map is C, and the association map is P, then the scaled depth map D and the instance segmentation map C are associated according to the pixel coordinate position, specifically: according to the pixel coordinate position (x, y), D(x, y) and C(x, y) corresponding to each coordinate position are associated to form P(x, y), then the association map P is the set of all P(x, y). Preferably, the scaled depth map D and the instance segmentation map C are matrices with a scale of W*H, and each number in the matrix is ​​a floating point number, P = {D, C}, where {} is a concatenation symbol, and the matrix dimension of P is W*H*2.

[0127] As in step S75 above, the three-dimensional structure information of the space is composed of point cloud information. The point cloud is a collection of points. Each point in the point cloud represents a set of x, y, z coordinates and other information. Since the embodiment of the present application associates the scaled depth map with the instance segmentation map, the spatial three-dimensional structure information obtained after projecting the association map into the point cloud space includes semantic information, which is described in mathematical language. The spatial three-dimensional structure information J(i) = (x, y, z, c); wherein (x, y, z) represents the three-dimensional coordinates, c is the instance label under the same pixel coordinates, and i represents the number of the point cloud.

[0128] As in step S76 above, the preliminary spatial three-dimensional structure information is stored as the spatial three-dimensional structure information used as a supervisory signal. Specifically, it can be stored in the first storage space. In this way, when necessary, the spatial three-dimensional structure information can be obtained from the first storage space as a supervisory signal.

[0129] The embodiment of the present application generates supervisory information such as a depth map and a semantic map through steps S71 to S76, and training can be completed efficiently without expensive data acquisition equipment and manual data annotation.

[0130] In one embodiment, before the step of obtaining the spatial three-dimensional structure information as a supervisory signal, steps S81 to S87 are included:

[0131] S81, acquiring a target scene image to be processed taken by a color camera and internal parameters of the color camera;

[0132] S82, predicting a scaled depth map corresponding to the target scene image to be processed;

[0133] S83, performing instance segmentation on the target scene image to be processed to obtain an instance segmentation map corresponding to the target scene image to be processed;

[0134] S84, according to the pixel coordinate position, associating the scaled depth map corresponding to the target scene image to be processed and the instance segmentation map corresponding to the target scene image to be processed to obtain an association map corresponding to the target scene image to be processed;

[0135] S85, based on the internal parameters, back-projecting the association graph corresponding to the target scene image to be processed into the point cloud space to obtain preliminary spatial three-dimensional structure information;

[0136] S86, performing plane fitting on the point clouds of each plane category in the preliminary spatial three-dimensional structure information using a point cloud plane fitting method according to the instance labels in the instance segmentation map corresponding to the target scene image to be processed, to obtain optimized spatial three-dimensional structure information;

[0137] S87: Store the optimized three-dimensional spatial structure information as the three-dimensional spatial structure information used as a supervisory signal.

[0138] As in the above step S81, the color camera may be a common pinhole camera, a fisheye camera, or other cameras, which is not limited in the embodiment of the present application.

[0139] As in step S82 above, the scaled depth map corresponding to the target scene image to be processed is the scaled depth map of the target scene. The pixel value of the depth map represents the depth, and the depth represents the distance between a certain point in the scene and the camera.

[0140] As shown in step S83 above, according to the different segmentation granularities, image segmentation is divided into semantic segmentation and instance segmentation. Semantic segmentation refers to classification at the pixel level, and pixels of the same category are divided into the same category. Compared with the use of box annotations in target detection, semantic segmentation is more refined. Instance segmentation is more detailed than semantic segmentation, and objects of the same category but belonging to different individuals can be distinguished. The instance segmentation map is composed of instance-level mask labels of each thing, and the instance-level mask labels of things are usually called instance labels. It should be noted that the embodiment of the present application does not limit the order of steps that have no logical relationship. For example, step S83 can be executed before step S82, or after step S82 is executed, step S83 can be executed.

[0141] As in step S84 above, if the scaled depth map is D, the instance segmentation map is C, and the association map is P, then the scaled depth map D and the instance segmentation map C are associated according to the pixel coordinate position, specifically: according to the pixel coordinate position (x, y), D(x, y) and C(x, y) corresponding to each coordinate position are associated to form P(x, y), then the association map P is the set of all P(x, y). Preferably, the scaled depth map D and the instance segmentation map C are matrices with a scale of W*H, and each number in the matrix is ​​a floating point number, P = {D, C}, where {} is a concatenation symbol, and the matrix dimension of P is W*H*2.

[0142] As in step S85 above, the three-dimensional structure information of the space is composed of point cloud information. The point cloud is a collection of points. Each point in the point cloud represents a set of x, y, z coordinates and other information. Since the embodiment of the present application associates the scaled depth map with the instance segmentation map, the spatial three-dimensional structure information obtained after projecting the association map into the point cloud space includes semantic information, which is described in mathematical language. The spatial three-dimensional structure information J(i) = (x, y, z, c); wherein (x, y, z) represents the three-dimensional coordinates, c is the instance label under the same pixel coordinates, and i represents the number of the point cloud.

[0143] As in the above step S86, specifically, according to the instance labels of the objects in the instance segmentation graph, the point cloud plane fitting method such as the least square method can be used to perform plane fitting on the point clouds of plane categories such as walls, desktops, ceilings, floors, etc. in the preliminary spatial three-dimensional structure information, so as to obtain spatial three-dimensional structure information that is more in line with the actual situation. It should be noted that since the indoor space contains a large number of weak texture areas, such as walls, desktops, ceilings, and floors, it is generally difficult for the depth estimation model to obtain accurate depth information, resulting in a certain degree of error in the preliminary spatial three-dimensional structure information obtained above. The embodiment of the present application determines the weak texture area according to the instance label information in the preliminary spatial three-dimensional structure information, and performs plane fitting on the point cloud level, thereby improving the accuracy of the spatial three-dimensional structure information corresponding to the weak texture area.

[0144] As in step S87, the optimized spatial three-dimensional structure information is stored as the spatial three-dimensional structure information used as the supervisory signal, specifically, it can be stored in the second storage space, so that when necessary, the spatial three-dimensional structure information can be obtained from the second storage space as the supervisory signal. Compared with using the preliminary spatial three-dimensional structure information as the supervisory signal, the spatial three-dimensional structure recovery model trained by using the optimized spatial three-dimensional structure information as the spatial three-dimensional structure information used as the supervisory signal is more accurate.

[0145] It should be noted that the supervisory signal can be obtained according to steps SS71 to S76, or according to steps S81 to S87, or can be obtained using other methods. For example, the depth value can also be the true value provided by the hardware depth sensor, and the semantic label can also be a manually annotated semantic label.

[0146] In one embodiment, the step of predicting the scaled depth map corresponding to the target scene image to be processed includes S721-SS723:

[0147] S721, inputting the target scene image to be processed into a first depth estimation model, estimating scale-free depth information of the target scene image to be processed by using the first depth estimation model, and obtaining a scale-free depth map;

[0148] S722: Use a priori calibration objects to restore the scale of the scale-free depth map to obtain a scaled depth map corresponding to the target scene image to be processed.

[0149] As in step S721 above, the first depth estimation model can be any model that can predict the scale-free depth information of the target scene image to be processed, including but not limited to a model based on statistical methods and a model based on deep learning methods. For example, the structure of the first depth estimation model includes a second encoder and a second decoder, the second encoder is used to extract features of the target scene image to be processed, and the second decoder is used to estimate the scale-free depth information of the target scene image to be processed from the extracted features, that is, to estimate the scale-free depth information of the scene, and its value range is [0,1]. When the camera is a monocular camera, the depth estimation model is a monocular depth estimation model.

[0150] As in step S722 above, the scale-free depth map is scaled and restored to obtain a scaled depth map. The scaled depth map represents the scale information of the target scene. It should be understood that by performing scale restoration on the scale-free depth information corresponding to each pixel of the image, the scale information of the corresponding pixel is obtained, and then the scale information of the image, that is, the scaled depth map corresponding to the image, can be obtained. The prior calibration object can be some planar objects in the room, such as a wall or a table. Specifically, the width or height of the prior calibration object can be used to scale the scale-free information of the scale-free depth map.

[0151] In one embodiment, the step of restoring the scale of the scale-free depth map using a priori calibration objects comprises:

[0152] Obtaining the actual height of the prior calibration object and the number of pixels of the height of the prior calibration object in the scale-free depth map;

[0153] Calculating the actual depth of the prior calibration object according to the actual height of the prior calibration object, the number of pixels of the height of the prior calibration object in the scale-free depth map, and the vertical focal length of the camera;

[0154] Calculating a scale factor according to the actual depth and the corresponding scale-free depth information;

[0155] The scale-free depth map is scaled according to the scale factor.

[0156] In an embodiment of the present application, taking the center coordinates of the prior calibration object as an example, the actual height of the (center coordinates (x, y)) at the center of the prior calibration object and the number of pixels at the height of the center coordinates of the prior calibration object in the scale-free depth map are obtained, and then the actual depth at the center coordinates of the prior calibration object is calculated based on the actual height at the center of the prior calibration object, the number of pixels at the height of the center coordinates of the prior calibration object in the scale-free depth map, and the vertical focal length of the camera, and then the scale-free depth information corresponding to the center coordinates of the prior calibration object is obtained, and then the scale factor is calculated based on the actual depth at the center coordinates of the prior calibration object and the scale-free depth information corresponding to the center coordinates of the prior calibration object, and the scale-free depth map is restored according to the scale factor. Specifically, the actual depth is calculated using the formula d=(h / p)*fy, where h represents the actual height corresponding to the prior calibration object in the physical world, p represents the number of pixels at the height of the prior calibration object in the scale-free depth map, fy represents the vertical focal length of the camera, and d represents the actual depth of the prior calibration object. The scale factor is calculated using factor = d / D'(x, y), where D'(x, y) represents the scale-free depth information of the calibration object (it should be noted that the scale-free depth information of a certain point of the prior calibration object, such as the center point of the prior calibration object, is used as the scale-free depth information of the prior calibration object). The depth map is scaled using D = D'*factor. Where D' represents the scale-free depth map and D represents the scaled depth map.

[0157] The scale restoration method of the embodiment of the present application is simple and can efficiently realize scale restoration without scale depth information.

[0158] In one embodiment, the step of restoring the scale of the scale-free depth map using a priori calibration objects comprises:

[0159] Obtaining the actual width of the prior calibration object and the number of pixels of the height of the prior calibration object in the scale-free depth map;

[0160] Calculating the actual depth of the prior calibration object according to the actual width of the prior calibration object, the number of pixels of the height of the prior calibration object in the scale-free depth map, and the horizontal focal length of the camera;

[0161] Calculating a scale factor according to the actual depth and the corresponding scale-free depth information;

[0162] The scale-free depth map is scaled according to the scale factor.

[0163] In an embodiment of the present application, taking the center coordinates of the prior calibration object as an example, the actual width of the (center coordinates (x, y)) at the center of the prior calibration object and the number of pixels at the height of the center coordinates of the prior calibration object in the scale-free depth map are obtained, and then the actual depth at the center coordinates of the prior calibration object is calculated based on the actual width at the center of the prior calibration object, the number of pixels at the height of the center coordinates of the prior calibration object in the scale-free depth map, and the horizontal focal length of the camera, and then the scale-free depth information corresponding to the center coordinates of the prior calibration object is obtained, and then the scale factor is calculated based on the actual depth at the center coordinates of the prior calibration object and the scale-free depth information corresponding to the center coordinates of the prior calibration object, and the scale-free depth map is restored according to the scale factor. Specifically, the actual depth is calculated using the formula d=(w / p)*fy, where w represents the actual width corresponding to the prior calibration object in the physical world, p represents the number of pixels at the height of the prior calibration object in the scale-free depth map, fy represents the horizontal focal length of the camera, and d represents the actual depth of the prior calibration object. The scale factor is calculated using factor = d / D'(x, y), where D'(x, y) represents the scale-free depth information of the calibration object (it should be noted that the scale-free depth information of a certain point of the prior calibration object, such as the center point of the prior calibration object, is used as the scale-free depth information of the prior calibration object). The depth map is scaled using D = D'*factor. Where D' represents the scale-free depth map and D represents the scaled depth map.

[0164] The scale restoration method of the embodiment of the present application is simple and can efficiently restore scale-free depth information.

[0165] It should be noted that when performing monocular scale restoration, for the sake of simplicity, only the derivation process of a single point (x, y) is discussed. In fact, multiple points can be sampled on the calibration object, and a more accurate scale factor can be calculated through the least squares method to obtain more accurate depth information.

[0166] In one embodiment, in one embodiment, the step of predicting the scaled depth map corresponding to the target scene image includes:

[0167] The target scene image to be processed is input into a second depth estimation model, and the scaled depth map corresponding to the target scene image to be processed is predicted by the second depth estimation model.

[0168] In the embodiment of the present application, the second depth estimation model can be any model that can predict the scale-free depth information of the target scene image to be processed, including but not limited to a model based on a statistical method or a model based on a deep learning method. For example, based on a deep learning method, the second depth estimation model to be trained is trained by using the predicted scaled depth map and the corresponding label (true value).

[0169] In one embodiment, performing instance segmentation on the target scene image to be processed to obtain an instance segmentation map corresponding to the target scene image to be processed includes:

[0170] The target scene image to be processed is input into an instance segmentation model, and instance segmentation is performed on the target scene image to be processed by the instance segmentation model to obtain an instance segmentation map corresponding to the target scene image to be processed.

[0171] In the embodiment of the present application, the instance labels of the objects in the image can be directly obtained through the instance segmentation model, and then the instance segmentation map can be obtained. It should be understood that the instance segmentation map is composed of the instance labels of each object. It should be noted that the instance segmentation model can be any model that can perform instance segmentation on the target scene image to be processed, and the embodiment of the present application does not limit this.

[0172] In one embodiment, the step of performing instance segmentation on the target scene image to obtain an instance segmentation map corresponding to the target scene image includes:

[0173] Predicting the semantic category of each pixel in the target scene image to be processed by a semantic segmentation model to obtain a semantic segmentation map;

[0174] According to the semantic category, a connected region search algorithm is used to search for connected regions in the semantic segmentation graph;

[0175] Each connected region is taken as an instance label to obtain an instance segmentation map corresponding to the target scene image to be processed.

[0176] In an embodiment of the present application, the semantic segmentation model is similar in structure to the depth estimation model, including a third encoder and a third decoder, the encoder is used to extract features from the target scene image, and the decoder is used to classify the semantic information of the scene from the extracted features. It should be noted that the semantic segmentation model can be any model with such functions, including but not limited to implementation based on statistical methods and implementation based on deep learning methods. It should be noted that the second encoder and the third encoder can be the same, and the second decoder and the third decoder are different.

[0177] In addition, since the semantic categories of pixels are distinguished by labels, the mask map of each type of semantic label can be obtained by obtaining the semantic category of each pixel. According to the semantic category, the specific steps of using the connected domain search algorithm to find the connected area in the semantic segmentation map are: for the mask map of each type of semantic label (semantic segmentation map and mask map of each type of label), first set the pixel value with label in the mask map to 255, and set the pixel value without label to zero to obtain a binary map. Input the binary map into the connected domain search algorithm, extract the connected domain therein, and regard each type of connected domain as an instance label. It should be understood that the connected domain search algorithm can be any algorithm that can realize the connected domain search, for example, a seed filling algorithm can be used or a two-pass scanning method can be used. If the seed filling algorithm is used, first select a pixel point in the foreground as a seed, and gradually expand outward according to the connection conditions until all the pixels in the area are traversed, and search other areas in the same way until all areas on the image have been found. The condition for connectivity may be: if the eight neighboring values ​​of the 3*3 area around the current pixel position are all the same as the central pixel value, then the 3*3 area is determined to be connected. The condition for connectivity may also be other conditions, which can be set according to actual needs, and the embodiment of the present application does not limit this.

[0178] Embodiment 2:

[0179] The present application provides a method for restoring spatial three-dimensional structure information, including:

[0180] Acquire the target scene image to be analyzed captured by a color camera;

[0181] The target scene image to be analyzed is input into a spatial three-dimensional structure information recovery model, and the spatial three-dimensional structure information of the target scene is recovered by the spatial three-dimensional structure information recovery model; wherein the spatial three-dimensional structure information recovery model is trained using any of the spatial three-dimensional structure information recovery model training methods described above.

[0182] Embodiment three:

[0183] like Figure 3 As shown, an embodiment of the present application provides a spatial three-dimensional structure information recovery model training device, the spatial three-dimensional structure recovery model includes an encoder, a decoder and a back projection layer, and the device includes:

[0184] A feature extraction module 1 is used to extract features from the target scene image to be trained obtained by taking a color camera through the encoder;

[0185] A correlation map estimation module 2 is used to estimate the correlation map obtained by associating the scaled depth map and the instance segmentation map from the extracted features through the decoder

[0186] A back-projection module 3 is used to back-project the association map into the point cloud space through the back-projection layer to obtain predicted spatial three-dimensional structure information;

[0187] A spatial three-dimensional structure information acquisition module 4 is used to acquire spatial three-dimensional structure information as a supervisory signal;

[0188] A loss value determination module 5, used to determine the value of the loss function of the spatial three-dimensional structure restoration model based on the predicted spatial three-dimensional structure information and the spatial three-dimensional structure information as a supervisory signal;

[0189] The model parameter updating module 6 is used to update the model parameters of the spatial three-dimensional structure restoration model based on the value of the loss function.

[0190] In one embodiment, the back projection module comprises:

[0191] An acquisition unit, used for acquiring internal parameters of the color camera;

[0192] A distortion correction unit, used for performing distortion correction on the correlation graph using the internal parameters to obtain a distortion-corrected correlation graph;

[0193] A projection unit is used to back-project the distortion-corrected association map to a point cloud space using a back-projection formula based on the pixel coordinates of the distortion-corrected association map, the internal parameters, and the depth information of the scaled depth map to obtain predicted spatial three-dimensional structure information.

[0194] Furthermore, the loss value determination module includes:

[0195] A first loss value determination module, used to determine a first loss value based on the predicted three-dimensional coordinates of the instance point cloud and the three-dimensional coordinates of the instance point cloud as a supervisory signal;

[0196] A second loss value determination module, configured to determine a second loss value based on the instance label of the predicted instance point cloud and the instance label of the instance point cloud as a supervisory signal;

[0197] A total loss value determination module is used to determine the value of the loss function of the spatial three-dimensional structure restoration model based on the first loss value and the second loss value.

[0198] In one embodiment, before the step of obtaining the spatial three-dimensional structure information as a supervisory signal, the method further includes:

[0199] Acquire the target scene image to be processed taken by a color camera and the internal parameters of the color camera;

[0200] Predicting a scaled depth map corresponding to the target scene image to be processed;

[0201] Performing instance segmentation on the target scene image to be processed to obtain an instance segmentation map corresponding to the target scene image to be processed;

[0202] According to the pixel coordinate position, the scaled depth map corresponding to the target scene image to be processed and the instance segmentation map corresponding to the target scene image to be processed are associated to obtain an association map corresponding to the target scene image to be processed;

[0203] Based on the internal parameters, the association graph corresponding to the target scene image to be processed is back-projected into the point cloud space to obtain preliminary spatial three-dimensional structure information;

[0204] The preliminary spatial three-dimensional structure information is stored as the spatial three-dimensional structure information used as a supervisory signal.

[0205] In one embodiment, before the step of obtaining the spatial three-dimensional structure information as a supervisory signal, the step includes:

[0206] Acquire the target scene image to be processed taken by a color camera and the internal parameters of the color camera;

[0207] Predicting a scaled depth map corresponding to the target scene image to be processed;

[0208] Performing instance segmentation on the target scene image to be processed to obtain an instance segmentation map corresponding to the target scene image to be processed;

[0209] According to the pixel coordinate position, the scaled depth map corresponding to the target scene image to be processed and the instance segmentation map corresponding to the target scene image to be processed are associated to obtain an association map corresponding to the target scene image to be processed;

[0210] Based on the internal parameters, the association graph corresponding to the target scene image to be processed is back-projected into the point cloud space to obtain preliminary spatial three-dimensional structure information;

[0211] According to the instance labels in the instance segmentation map corresponding to the target scene image to be processed, a point cloud plane fitting method is used to perform plane fitting on the point clouds of each plane category in the preliminary spatial three-dimensional structure information to obtain optimized spatial three-dimensional structure information;

[0212] The optimized spatial three-dimensional structure information is stored as the spatial three-dimensional structure information used as a supervisory signal.

[0213] In one embodiment, the step of predicting the scaled depth map corresponding to the target scene image to be processed includes:

[0214] Inputting the target scene image to be processed into a first depth estimation model, estimating scale-free depth information of the target scene image to be processed by the first depth estimation model, and obtaining a scale-free depth map;

[0215] The scale of the scale-free depth map is restored by using a priori calibration objects to obtain a scaled depth map corresponding to the target scene image to be processed.

[0216] It should be noted that the spatial three-dimensional structure information recovery device provided in the embodiment of the present application corresponds to the spatial three-dimensional structure information recovery method, so the embodiment of the present application will not be repeated here.

[0217] Embodiment 4:

[0218] The present application provides a spatial three-dimensional structure information recovery device, comprising:

[0219] An image acquisition module is used to acquire the target scene image to be analyzed taken by a color camera;

[0220] A spatial three-dimensional structure information recovery module is used to input the target scene image to be analyzed into a spatial three-dimensional structure information recovery model, and recover the spatial three-dimensional structure information of the target scene through the spatial three-dimensional structure information recovery model; wherein, the spatial three-dimensional structure information recovery model is trained using any of the spatial three-dimensional structure information recovery model training methods described above.

[0221] Reference Figure 4 The present application also provides a computer device, the internal structure of which can be as follows: Figure 4As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor designed for the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating device, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as a spatial three-dimensional structure information recovery model training method and / or a spatial three-dimensional structure information recovery method. The network interface of the computer device is used to communicate with an external terminal via a network connection. Furthermore, the above-mentioned computer device can also be provided with an input device and a display screen, etc. When the above-mentioned computer program is executed by the processor, a spatial three-dimensional structure information recovery model training method and / or a spatial three-dimensional structure information recovery method is implemented. The spatial three-dimensional structure recovery model includes an encoder, a decoder and a back-projection layer, and the spatial three-dimensional structure information recovery model training method includes: extracting features from the target scene image to be trained taken by a color camera through the encoder; estimating the association map obtained by associating the scaled depth map and the instance segmentation map from the extracted features through the decoder, back-projecting the association map to the point cloud space through the back-projection layer to obtain predicted spatial three-dimensional structure information; obtaining the spatial three-dimensional structure information as a supervisory signal; determining the value of the loss function of the spatial three-dimensional structure recovery model based on the predicted spatial three-dimensional structure information and the spatial three-dimensional structure information as a supervisory signal; and updating the model parameters of the spatial three-dimensional structure recovery model based on the value of the loss function. The spatial three-dimensional structure information recovery method includes: obtaining the target scene image to be analyzed taken by a color camera; inputting the target scene image to be analyzed into the spatial three-dimensional structure information recovery model, and recovering the spatial three-dimensional structure information of the target scene through the spatial three-dimensional structure information recovery model; wherein the spatial three-dimensional structure information recovery model is trained using any of the above-mentioned spatial three-dimensional structure information recovery model training methods. Those skilled in the art will understand that Figure 4 The structure shown in is merely a block diagram of a portion of the structure related to the present application solution and does not constitute a limitation on the computer device to which the present application solution is applied.

[0222] Embodiment 4:

[0223] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, a spatial three-dimensional structure information recovery model training method and / or a spatial three-dimensional structure information recovery method is implemented. The spatial three-dimensional structure recovery model includes an encoder, a decoder and a back-projection layer. The spatial three-dimensional structure information recovery model training method includes: extracting features from a target scene image to be trained taken by a color camera through the encoder; estimating a correlation map obtained by associating a scaled depth map and an instance segmentation map from the extracted features through the decoder, and back-projecting the correlation map to a point cloud space through the back-projection layer to obtain predicted spatial three-dimensional structure information; obtaining spatial three-dimensional structure information as a supervisory signal; determining a value of a loss function of the spatial three-dimensional structure recovery model based on the predicted spatial three-dimensional structure information and the spatial three-dimensional structure information as a supervisory signal; and updating model parameters of the spatial three-dimensional structure recovery model based on the value of the loss function. The spatial three-dimensional structure information recovery method comprises: obtaining a target scene image to be analyzed taken by a color camera; inputting the target scene image to be analyzed into a spatial three-dimensional structure information recovery model, and recovering the spatial three-dimensional structure information of the target scene by using the spatial three-dimensional structure information recovery model; wherein the spatial three-dimensional structure information recovery model is trained using any of the spatial three-dimensional structure information recovery model training methods described above.

[0224] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0225] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0226] The above description is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A spatial three-dimensional structure information recovery model training method, It is characterized in that The spatial three-dimensional structure recovery model includes an encoder, a decoder and a back-projection layer, and the method includes: Extracting features from the target scene image to be trained taken by the color camera by using the encoder; The decoder estimates the associated map obtained by associating the scaled depth map and the instance segmentation map from the extracted features. Back-projecting the association graph into the point cloud space through the back-projection layer to obtain predicted spatial three-dimensional structure information; Obtain spatial three-dimensional structure information as a supervisory signal; Determining a value of a loss function of the spatial three-dimensional structure restoration model based on the predicted spatial three-dimensional structure information and the spatial three-dimensional structure information as a supervisory signal; The model parameters of the spatial three-dimensional structure restoration model are updated based on the value of the loss function.

2. The spatial three-dimensional structure information recovery model training method according to claim 1, It is characterized in that The step of back-projecting the association graph into the point cloud space through the back-projection layer to obtain predicted spatial three-dimensional structure information includes: Obtaining internal parameters of the color camera; Performing distortion correction on the correlation graph using the internal parameters to obtain a distortion-corrected correlation graph; Based on the pixel coordinates of the distortion-corrected association map, the internal parameters and the depth information of the scaled depth map, the distortion-corrected association map is back-projected into the point cloud space using a back-projection formula to obtain predicted spatial three-dimensional structure information.

3. The spatial three-dimensional structure information recovery model training method according to claim 1, It is characterized in that The spatial three-dimensional structure information includes the three-dimensional coordinates of the instance point cloud and the instance label of the instance point cloud. The step of determining the value of the loss function of the spatial three-dimensional structure restoration model based on the predicted spatial three-dimensional structure information and the spatial three-dimensional structure information as a supervisory signal includes: Determine a first loss value based on the predicted three-dimensional coordinates of the instance point cloud and the three-dimensional coordinates of the instance point cloud as a supervisory signal; Determine a second loss value based on the instance label of the predicted instance point cloud and the instance label of the instance point cloud as the supervisory signal; A value of a loss function of the spatial three-dimensional structure restoration model is determined based on the first loss value and the second loss value.

4. The spatial three-dimensional structure information recovery model training method according to claim 1, It is characterized in that Before the step of obtaining the spatial three-dimensional structure information as a supervisory signal, the method further includes: Acquire the target scene image to be processed taken by the color camera and the internal parameters of the color camera; Predicting a scaled depth map corresponding to the target scene image to be processed; Performing instance segmentation on the target scene image to be processed to obtain an instance segmentation map corresponding to the target scene image to be processed; According to the pixel coordinate position, the scaled depth map corresponding to the target scene image to be processed and the instance segmentation map corresponding to the target scene image to be processed are associated to obtain an association map corresponding to the target scene image to be processed; Based on the internal parameters, the association graph corresponding to the target scene image to be processed is back-projected into the point cloud space to obtain preliminary spatial three-dimensional structure information; The preliminary spatial three-dimensional structure information is stored as the spatial three-dimensional structure information used as a supervisory signal.

5. The spatial three-dimensional structure information recovery model training method according to claim 1, It is characterized in that Before the step of obtaining the spatial three-dimensional structure information as a supervisory signal, the method includes: Acquire the target scene image to be processed taken by the color camera and the internal parameters of the color camera; Predicting a scaled depth map corresponding to the target scene image to be processed; Performing instance segmentation on the target scene image to be processed to obtain an instance segmentation map corresponding to the target scene image to be processed; According to the pixel coordinate position, the scaled depth map corresponding to the target scene image to be processed and the instance segmentation map corresponding to the target scene image to be processed are associated to obtain an association map corresponding to the target scene image to be processed; Based on the internal parameters, the association graph corresponding to the target scene image to be processed is back-projected into the point cloud space to obtain preliminary spatial three-dimensional structure information; According to the instance labels in the instance segmentation map corresponding to the target scene image to be processed, a point cloud plane fitting method is used to perform plane fitting on the point clouds of each plane category in the preliminary spatial three-dimensional structure information to obtain optimized spatial three-dimensional structure information; The optimized spatial three-dimensional structure information is stored as the spatial three-dimensional structure information used as a supervisory signal.

6. The spatial three-dimensional structure information recovery model training method according to claim 4 or 5, It is characterized in that The step of predicting the scaled depth map corresponding to the target scene image to be processed includes: Inputting the target scene image to be processed into a first depth estimation model, estimating scale-free depth information of the target scene image to be processed by the first depth estimation model, and obtaining a scale-free depth map; The scale of the scale-free depth map is restored by using a priori calibration objects to obtain a scaled depth map corresponding to the target scene image to be processed.

7. A method for restoring spatial three-dimensional structural information, It is characterized in that include: Acquire the target scene image to be analyzed captured by a color camera; The target scene image to be analyzed is input into a spatial three-dimensional structure information recovery model, and the spatial three-dimensional structure information of the target scene is recovered by the spatial three-dimensional structure information recovery model; wherein the spatial three-dimensional structure information recovery model is trained using the spatial three-dimensional structure information recovery model training method described in any one of claims 1 to 6.

8. A spatial three-dimensional structure information recovery model training device, It is characterized in that The spatial three-dimensional structure recovery model includes an encoder, a decoder and a back-projection layer, and the device includes: A feature extraction module, used for extracting features from the target scene image to be trained obtained by taking a color camera through the encoder; A correlation map estimation module, used to estimate the correlation map obtained by associating the scaled depth map and the instance segmentation map from the extracted features through the decoder A back-projection module, used for back-projecting the association map into a point cloud space through the back-projection layer to obtain predicted spatial three-dimensional structure information; A spatial three-dimensional structure information acquisition module, used to acquire spatial three-dimensional structure information as a supervisory signal; A loss value determination module, used to determine the value of the loss function of the spatial three-dimensional structure restoration model based on the predicted spatial three-dimensional structure information and the spatial three-dimensional structure information as a supervisory signal; A model parameter updating module is used to update the model parameters of the spatial three-dimensional structure restoration model based on the value of the loss function.

9. A spatial three-dimensional structure information recovery device, It is characterized in that include: An image acquisition module is used to acquire the target scene image to be analyzed taken by a color camera; A spatial three-dimensional structure information recovery module is used to input the target scene image to be analyzed into a spatial three-dimensional structure information recovery model, and recover the spatial three-dimensional structure information of the target scene through the spatial three-dimensional structure information recovery model; wherein, the spatial three-dimensional structure information recovery model is trained using the spatial three-dimensional structure information recovery model training method described in any one of claims 1-6.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images