Three-dimensional information determination method, device, electronic device and storage medium

By matching and fusion of the image frames of the target video, the problem of three-dimensional information prediction discontinuity in the prior art is solved, and continuous tracking and accurate acquisition of three-dimensional information on the object facade during movement is realized.

CN114898074BActive Publication Date: 2024-11-26BEIJING DAJIA INTERNET INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210466361.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-29
Publication Date
2024-11-26
Estimated Expiration
2042-04-29

AI Technical Summary

Technical Problem

When the prior art determines the three-dimensional information of the object in a video stream, the prediction results lack time and space continuity, and it is impossible to continuously obtain the three-dimensional information of the object's facade during movement.

Method used

By acquiring the first and second image frames of the target video, the facade information in the first image frame is determined, and the facade information of the second image frame is fused with the information of the first image frame based on feature matching, and the calculation is performed. The three-dimensional information in the second image frame is output to achieve tracking and continuous acquisition of the object facade.

Benefits of technology

It realizes continuous tracking and accurate acquisition of the three-dimensional information on the target object's facade during movement, and improves the accuracy of the three-dimensional information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114898074B_ABST
    Figure CN114898074B_ABST
Patent Text Reader

Abstract

The present disclosure is about a method, device and electronic device for determining three-dimensional information, and the method includes: acquiring a target video; determining the first plane information of the facade in the first image frame; determining the first three-dimensional information of the facade in the first image frame in the first coordinate system; performing feature matching between the second image frame and the first image frame to obtain the second three-dimensional information of the second image frame; determining the second plane information of the facade in the second image frame, fusing the second plane information with the first plane information to obtain the fused plane information, and determining the newly added three-dimensional information; determining the second three-dimensional information and the newly added three-dimensional information as the target three-dimensional information of the second image frame in the first coordinate system. By fusing the tracking result with the newly added three-dimensional information of the second image frame relative to the first image frame in the first coordinate system, the three-dimensional information of the target object in the second image frame is obtained, so that the three-dimensional information of the target object can be continuously obtained during the movement process.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a three-dimensional information determination method, device, electronic device and storage medium. Background Art

[0002] With the continuous development of science and technology and the popularization of the Internet, it is necessary to obtain the three-dimensional information of objects in some virtual reality scene tasks. For example, in the AR field, it is usually necessary to predict the position of the object's facade relative to itself and the shape of the object, etc., to restore the three-dimensional information of the object's facade, so as to better superimpose virtual effects on the object, such as spraying paint on the object surface.

[0003] However, when determining the three-dimensional information of an object, related technologies usually input single image frames into a neural network separately to predict the three-dimensional information of the object's facade in each image frame. The prediction in the video stream is unstable, and the prediction results do not have continuity in time and space, that is, it is impossible to continuously obtain the three-dimensional information of the object's facade during movement. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides a three-dimensional information determination method, device, electronic device and storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for determining three-dimensional information is provided, including:

[0006] Acquire a target video, the target video comprising a first image frame and a second image frame, the first image frame and the second image frame both comprising a target object, a first moment corresponding to the first image frame in the target video being earlier than a second moment corresponding to the second image frame in the target video;

[0007] Determining first plane information of each facade of the target object in the first image frame;

[0008] Based on first plane information of each facade of the target object in the first image frame, determine first three-dimensional information of each facade of the target object in the first image frame in a first coordinate system, where the first coordinate system is a coordinate system established with a shooting device that shoots the target video as an origin;

[0009] Performing feature matching on each facade in the second image frame and each facade in the first image frame, and determining second three-dimensional information of each facade in the second image frame in the first coordinate system based on the feature matching result and the first three-dimensional information;

[0010] Determine second plane information of each facade of the target object in the second image frame, fuse the second plane information with the first plane information to obtain fused plane information, and determine three-dimensional information newly added to the second image frame relative to the first image frame in the first coordinate system based on the fused plane information;

[0011] The second three-dimensional information and the newly added three-dimensional information are fused to obtain target three-dimensional information of each facade of the target object in the second image frame in the first coordinate system.

[0012] Optionally, determining first plane information of each facade of the target object in the first image frame includes:

[0013] Inputting the first image frame into a pre-trained first neural network to obtain a first image region where the target object is located in the first image frame;

[0014] The first image area is input into a pre-trained second neural network to obtain a first grayscale image corresponding to the first image area, the first coordinates of the endpoints of the intersection lines of the facades of the target object in the first image area in the second coordinate system, the first normal vectors of the facades of the target object in the first image area in the first coordinate system, and the second coordinate system is a coordinate system established with the target point in the image frame as the origin.

[0015] Optionally, determining first three-dimensional information of each facade of the target object in the first image frame in the first coordinate system based on first plane information of each facade of the target object in the first image frame includes:

[0016] Calculating the coordinates of a first center point of a first facade of a target object in the first image frame in the second coordinate system, where the first center point is a center point of a grayscale image corresponding to the first facade;

[0017] Determine the coordinates of the first center point in the first coordinate system by using a pinhole camera model based on the coordinates of the first center point in the second coordinate system and the distance from the first center point to the shooting device;

[0018] Determine the three-dimensional information of the second facade based on the coordinates of the endpoints of the intersection line of the first facade and the second facade in the second coordinate system and the three-dimensional information of the first facade;

[0019] Based on the normal vector of the first facade in the first coordinate system and the coordinates of the first center point in the first coordinate system, the coordinates of the feature point of the first facade in the first coordinate system are calculated; based on the normal vector of the second facade in the first coordinate system and the coordinates of the second center point in the first coordinate system, the coordinates of the feature point of the second facade in the first coordinate system are calculated.

[0020] Optionally, performing feature matching on each facade in the second image frame with each facade in the first image frame, and determining second three-dimensional information of each facade in the second image frame in the first coordinate system based on the feature matching result and the first three-dimensional information, includes:

[0021] Extracting feature points of each facade in the first image frame and feature points of each facade in the second image frame;

[0022] Perform feature matching on the feature points of each facade in the second image frame and the feature points of each facade in the first image frame; the feature points of each facade are the feature points in the grayscale image of the facade;

[0023] For any feature point of each facade in the second image frame, the first three-dimensional information corresponding to the target feature point matching the feature point in the first image frame is determined as the second three-dimensional information of the feature point in the first coordinate system.

[0024] Optionally, the method further includes:

[0025] For any feature point of each facade in the second image frame, project the target three-dimensional information corresponding to the feature point through the target camera model to obtain the coordinates of the corresponding projected feature point in the second coordinate system: and calculate the error between the coordinates of the projected feature point in the second coordinate system and the coordinates of the feature point in the second coordinate system;

[0026] Based on the error, position information generated by the shooting device in the process of shooting the second image frame and the first image frame is determined, and the position information includes rotation information and translation information.

[0027] Optionally, the fusing the second plane information with the first plane information to obtain fused plane information, and determining, based on the fused plane information, three-dimensional information newly added to the second image frame relative to the first image frame in the first coordinate system, includes:

[0028] Determine a target facade in the second image frame, and calculate three-dimensional information of feature points of the target facade in a first coordinate system corresponding to the first image frame; the target facade is the same facade in the second image frame as that in the first image frame;

[0029] Projecting the calculated three-dimensional information into a second coordinate system corresponding to the first image frame to obtain a plurality of projection points, determining a second grayscale image based on the plurality of projection points, fusing the second grayscale image with the first grayscale image, and calculating the three-dimensional information of the fused projection points in the first coordinate system corresponding to the second image frame;

[0030] Determine three-dimensional information of an endpoint of a target intersection line in a first coordinate system corresponding to the second image frame, the target intersection line being an intersection line included in second plane information corresponding to the target facade;

[0031] The three-dimensional information of the fused projection point in the first coordinate system corresponding to the second image frame and the three-dimensional information of the endpoint of the target intersection line in the first coordinate system corresponding to the second image frame are determined as the three-dimensional information newly added to the second image frame relative to the first image frame in the first coordinate system.

[0032] Optionally, determining the target facade in the second image frame and calculating three-dimensional information of feature points of the target facade in a first coordinate system corresponding to the first image frame includes:

[0033] Converting the second normal vector included in the second plane information into the first coordinate system corresponding to the first image frame by using the posture information to obtain a converted normal vector;

[0034] Calculating an angle between the converted normal vector and a first normal vector included in the first plane information, and determining the same facade in the second image frame and the first image frame based on the angle;

[0035] Extracting feature points in the grayscale image included in the second plane information of the target facade, and calculating the three-dimensional information of the feature points in the first coordinate system corresponding to the first image frame based on the coordinates of the first center point corresponding to the feature points and the target facade in the second coordinate system, and the first normal vector of the target facade in the first coordinate system;

[0036] The calculated three-dimensional information of the feature points is converted into a first coordinate system corresponding to the first image frame through the posture information to obtain the converted three-dimensional information as the three-dimensional information of the feature points of the target facade in the first coordinate system corresponding to the first image frame.

[0037] Optionally, the determining of three-dimensional information of the endpoint of the target intersection line in a first coordinate system corresponding to the second image frame includes:

[0038] Projecting the endpoint of the target intersection line onto the endpoint intersection line of the first image frame; the endpoint intersection line is determined based on the coordinates of the endpoint of the intersection line included in the first plane information in the second coordinate system;

[0039] The three-dimensional information of the endpoint of the target intersection line in the first coordinate system corresponding to the second image frame is determined by the coordinates of the first center point corresponding to the projection endpoint and the target facade in the second coordinate system, and the first normal vector of the target facade in the first coordinate system.

[0040] According to a second aspect of an embodiment of the present disclosure, there is provided a three-dimensional information determining device, including:

[0041] A target video acquisition module is configured to acquire a target video, wherein the target video includes a first image frame and a second image frame, wherein both the first image frame and the second image frame include a target object, and a first moment corresponding to the first image frame in the target video is earlier than a second moment corresponding to the second image frame in the target video;

[0042] a plane information determination module, configured to determine first plane information of each facade of the target object in the first image frame;

[0043] A first three-dimensional information determination module is configured to determine first three-dimensional information of each facade of the target object in the first image frame in the first coordinate system based on first plane information of each facade of the target object in the first image frame, wherein the first coordinate system is a coordinate system established with a shooting device shooting the target video as an origin;

[0044] a second three-dimensional information determination module, configured to perform feature matching between each facade in the second image frame and each facade in the first image frame, and determine second three-dimensional information of each facade in the second image frame in the first coordinate system based on the feature matching result and the first three-dimensional information;

[0045] a plane information fusion module, configured to determine second plane information of each facade of the target object in the second image frame, fuse the second plane information with the first plane information to obtain fused plane information, and determine three-dimensional information newly added to the second image frame relative to the first image frame in the first coordinate system based on the fused plane information;

[0046] The target three-dimensional information determination module is configured to perform fusion of the second three-dimensional information and the newly added three-dimensional information to obtain the target three-dimensional information of each facade of the target object in the second image frame in the first coordinate system.

[0047] Optionally, the first plane information determining module is specifically configured to execute:

[0048] Inputting the first image frame into a pre-trained first neural network to obtain a first image region where the target object is located in the first image frame;

[0049] The first image area is input into a pre-trained second neural network to obtain a first grayscale image corresponding to the first image area, the first coordinates of the endpoints of the intersection lines of the facades of the target object in the first image area in the second coordinate system, the first normal vectors of the facades of the target object in the first image area in the first coordinate system, and the second coordinate system is a coordinate system established with the target point in the image frame as the origin.

[0050] Optionally, the first three-dimensional information determining module is specifically configured to execute:

[0051] Calculating the coordinates of a first center point of a first facade of a target object in the first image frame in the second coordinate system, where the first center point is a center point of a grayscale image corresponding to the first facade;

[0052] Determine the coordinates of the first center point in the first coordinate system by using a pinhole camera model based on the coordinates of the first center point in the second coordinate system and the distance from the first center point to the shooting device;

[0053] Determine the three-dimensional information of the second facade based on the coordinates of the endpoints of the intersection line of the first facade and the second facade in the second coordinate system and the three-dimensional information of the first facade;

[0054] Based on the normal vector of the first facade in the first coordinate system and the coordinates of the first center point in the first coordinate system, the coordinates of the feature point of the first facade in the first coordinate system are calculated; based on the normal vector of the second facade in the first coordinate system and the coordinates of the second center point in the first coordinate system, the coordinates of the feature point of the second facade in the first coordinate system are calculated.

[0055] Optionally, the second three-dimensional information determining module is configured to execute, including:

[0056] Extracting feature points of each facade in the first image frame and feature points of each facade in the second image frame;

[0057] Perform feature matching on the feature points of each facade in the second image frame and the feature points of each facade in the first image frame; the feature points of the facade are the feature points in the grayscale image of the facade;

[0058] For any feature point of each facade in the second image frame, the first three-dimensional information corresponding to the target feature point matching the feature point in the first image frame is determined as the second three-dimensional information of the feature point in the first coordinate system.

[0059] Optionally, the device further comprises:

[0060] A feature point projection module is configured to perform, for any feature point of each facade in the second image frame, projecting the target three-dimensional information corresponding to the feature point through the target camera model to obtain the coordinates of the corresponding projected feature point in the second coordinate system: and calculating the error between the coordinates of the projected feature point in the second coordinate system and the coordinates of the feature point in the second coordinate system;

[0061] The posture information determination module is configured to determine the posture information generated in the process of the shooting device shooting the second image frame and the first image frame based on the error, and the posture information includes rotation information and translation information.

[0062] Optionally, the plane information fusion module includes:

[0063] A first three-dimensional information determining unit is configured to determine a target facade in the second image frame and calculate three-dimensional information of feature points of the target facade in a first coordinate system corresponding to the first image frame; the target facade is the same facade in the second image frame as that in the first image frame;

[0064] A second three-dimensional information determining unit is configured to project the calculated three-dimensional information into a second coordinate system corresponding to the first image frame to obtain a plurality of projection points, determine a second grayscale image based on the plurality of projection points, fuse the second grayscale image with the first grayscale image, and calculate the three-dimensional information of the fused projection points in the first coordinate system corresponding to the second image frame;

[0065] A third three-dimensional information determining unit is configured to determine three-dimensional information of an endpoint of a target intersection line in a first coordinate system corresponding to the second image frame, wherein the target intersection line is an intersection line included in the second plane information corresponding to the target facade;

[0066] The plane information fusion unit is configured to execute the three-dimensional information of the fused projection point in the first coordinate system corresponding to the second image frame, and the three-dimensional information of the endpoint of the target intersection line in the first coordinate system corresponding to the second image frame, to determine the three-dimensional information newly added to the second image frame relative to the first image frame in the first coordinate system.

[0067] Optionally, the first three-dimensional information determining unit is specifically configured to execute;

[0068] Converting the second normal vector included in the second plane information into the first coordinate system corresponding to the first image frame by using the posture information to obtain a converted normal vector;

[0069] Calculating an angle between the converted normal vector and a first normal vector included in the first plane information, and determining the same facade in the second image frame and the first image frame based on the angle;

[0070] Extracting feature points in the grayscale image included in the second plane information of the target facade, and calculating the three-dimensional information of the feature points in the first coordinate system corresponding to the first image frame based on the coordinates of the first center point corresponding to the feature points and the target facade in the second coordinate system, and the first normal vector of the target facade in the first coordinate system;

[0071] The calculated three-dimensional information of the feature points is converted into a first coordinate system corresponding to the first image frame through the posture information to obtain the converted three-dimensional information as the three-dimensional information of the feature points of the target facade in the first coordinate system corresponding to the first image frame.

[0072] Optionally, the third three-dimensional information determining unit is specifically configured to execute:

[0073] Projecting the endpoint of the target intersection line onto the endpoint intersection line of the first image frame; the endpoint intersection line is determined based on the coordinates of the endpoint of the intersection line included in the first plane information in the second coordinate system;

[0074] The three-dimensional information of the endpoint of the target intersection line in the first coordinate system corresponding to the second image frame is determined by the coordinates of the first center point corresponding to the projection endpoint and the target facade in the second coordinate system, and the first normal vector of the target facade in the first coordinate system.

[0075] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0076] processor;

[0077] a memory for storing processor-executable instructions;

[0078] The processor is configured to execute the method as described in the first aspect.

[0079] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method described in the first aspect.

[0080] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, which, when executed on a computer, enables the computer to execute the steps of the method described in the first aspect.

[0081] The technical solution provided by the embodiment of the present disclosure is to obtain a target video, where the target video includes a first image frame and a second image frame, both of which include a target object, and a first moment corresponding to the first image frame in the target video is earlier than a second moment corresponding to the second image frame in the target video; determine first plane information of each facade of the target object in the first image frame, and determine first three-dimensional information of each facade of the target object in the first image frame in a first coordinate system based on the first plane information of each facade of the target object in the first image frame; perform feature matching on each facade in the second image frame with each facade in the first image frame, and determine second three-dimensional information of each facade in the second image frame in the first coordinate system based on the feature matching result and the first three-dimensional information; determine second plane information of each facade of the target object in the second image frame, fuse the second plane information with the first plane information to obtain fused plane information, and determine new three-dimensional information of the second image frame relative to the first image frame in the first coordinate system based on the fused plane information; fuse the second three-dimensional information with the new three-dimensional information to obtain target three-dimensional information of each facade of the target object in the second image frame in the first coordinate system.

[0082] It can be seen that through the technical solution provided by the embodiment of the present disclosure, when determining the three-dimensional information of the target object in the second image frame, the features of each facade in the second image frame are matched with those of each facade in the first image frame, that is, the tracking of the three-dimensional information of the target object in the second image frame is realized, and the tracking result is obtained, that is, the second three-dimensional information of each facade in the second image frame in the first coordinate system is obtained; and by fusing the tracking result with the newly added three-dimensional information of the second image frame relative to the first image frame in the first coordinate system, the three-dimensional information of the target object in the second image frame is obtained, instead of re-determining the three-dimensional information of the second image frame separately, thereby realizing the continuous acquisition of the three-dimensional information of the target object during the movement, thereby improving the accuracy of the three-dimensional information of the target object.

[0083] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0085] Figure 1 is a flow chart of a method for determining three-dimensional information according to an exemplary embodiment;

[0086] Figure 2 is a schematic diagram of a scenario provided in an embodiment of the present disclosure;

[0087] Figure 3 yes Figure 1 Flow chart of step S120;

[0088] Figure 4 yes Figure 1 Flow chart of step S130;

[0089] Figure 5 yes Figure 1 Flow chart of step S150;

[0090] Figure 6 is a flowchart showing a specific implementation according to an exemplary embodiment;

[0091] Figure 7 is a block diagram of a three-dimensional information determination device according to an exemplary embodiment;

[0092] Figure 8 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0093] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0094] In the AR field, it is usually necessary to predict the position of an object's facade relative to itself and the shape of the object in order to restore the three-dimensional information of the object's facade, thereby better superimposing virtual effects on the object's surface, such as spraying paint on the object's surface.

[0095] When determining the three-dimensional information of an object, related technologies usually input single image frames into a neural network to predict the three-dimensional information of the object's facade in each image frame. The prediction in the video stream is unstable, and the prediction results do not have continuity in time and space, that is, it is impossible to continuously obtain the three-dimensional information of the object's facade during movement.

[0096] To this end, the embodiments of the present disclosure provide a three-dimensional information determination method, device, electronic device and storage medium.

[0097] In the first aspect, a three-dimensional information determination method provided by an embodiment of the present disclosure is first described in detail.

[0098] like Figure 1 As shown, a three-dimensional information determination method provided by an embodiment of the present disclosure may include the following steps:

[0099] S110, obtaining a target video.

[0100] The target video includes a first image frame and a second image frame, both the first image frame and the second image frame include a target object, and a first moment corresponding to the first image frame in the target video is earlier than a second moment corresponding to the second image frame in the target video.

[0101] Specifically, in practical applications, the target video may include multiple image frames. Among them, the first image frame may be the first image frame that includes the target object in the target video. The second image frame may be the next image frame of the first image frame, or the second image frame may also be an image frame that is located after the first image frame and is separated from the first image frame by a preset frame, which is all reasonable. Of course, if the three-dimensional information of the target object is determined from the Nth image frame of the target video, the first image frame may also be the Nth image frame of the target video, and the second image frame may be the N+nth image frame, where N and n are both positive integers. The embodiments of the present disclosure do not specifically limit this.

[0102] S120: Determine first plane information of each facade of the target object in the first image frame.

[0103] The first plane information of each facade may include: a grayscale image corresponding to the facade, the coordinates of the endpoints of the intersection lines of the facade with other facades in the second coordinate system, and the normal vector of the facade in the first coordinate system. The first coordinate system is a coordinate system established with the shooting device for shooting the target video as the origin, and the second coordinate system is a coordinate system established with the target point in the image frame as the origin.

[0104] In one embodiment, first plane information of each facade of the target object in the first image frame is determined, such as Figure 3 As shown, the following two steps may be included:

[0105] S121, inputting the first image frame into a pre-trained first neural network to obtain a first image region where a target object is located in the first image frame.

[0106] The first neural network may be an object detection neural network. The first image frame may be input into the object detection neural network to obtain a first image region where the target object is located in the first image frame, that is, to obtain the position of the target object in the first image frame (a rectangular frame of the target object).

[0107] S122, input the first image area into a pre-trained second neural network to obtain a first grayscale image corresponding to the first image area, the first coordinates of the endpoints of the intersection lines of the facades of the target object in the first image area in the second coordinate system, the first normal vectors of the facades of the target object in the first image area in the first coordinate system, and the second coordinate system is a coordinate system established with the target point in the image frame as the origin.

[0108] The second neural network may be an object pose estimation neural network.

[0109] For the first image frame, based on the result of the object detection neural network, the following three information are predicted by the object pose estimation neural network: the predicted two-dimensional mask result of each visible object facade, which is a first grayscale image with the same size as the original image of the visible object facade; the first coordinates P1(x, y) and P2(x, y) of the two endpoints of the intersection line of the object facade in the first coordinate system, i.e., the pixel coordinate system; and the first normal vector n of each visible object facade of the target object in the first image frame in the first coordinate system, i.e., the camera coordinate system. c1 , expressed as (n x1 ,n1 y ,n z1 ),like Figure 2 As shown; wherein, the camera coordinate system may be a coordinate system established with the camera photographing the target object as the center; the pixel coordinate system may refer to a two-dimensional coordinate system established with the upper left corner of the first image frame as the center.

[0110] It can be seen that in the present embodiment, two neural networks are pre-trained, namely, an object detection neural network and an object pose estimation neural network. When determining the first plane information of each facade of the target object in the first image frame, the first image frame is successively input into the object detection neural network and the object pose estimation neural network, so that the first plane information of each facade of the target object in the first image frame can be obtained more accurately.

[0111] S130: Determine first three-dimensional information of each external surface of the target object in the first image frame in a first coordinate system based on first plane information of each external surface of the target object in the first image frame.

[0112] In order to continuously track the three-dimensional information of the first image frame and the second image frame in the subsequent steps, the three-dimensional information of the feature points of each facade of the target object in the first image frame in the first coordinate system can be determined based on the plane information of each facade of the target object in the first image frame. That is, the three-dimensional information of the target object in the first image frame is initialized, and the three-dimensional information of the target object in the second image frame can be tracked based on the three-dimensional information of the target object in the first image frame.

[0113] For the sake of completeness and clear description, the specific implementation of S130 will be described in detail in the following embodiments.

[0114] S140, performing feature matching on each facade in the second image frame and each facade in the first image frame, and determining second three-dimensional information of each facade in the second image frame in the first coordinate system based on the feature matching result and the first three-dimensional information.

[0115] When the second position in the second image frame matches the first position in the first image frame, the second three-dimensional information corresponding to the second position is the same as the first three-dimensional information corresponding to the first position.

[0116] Specifically, the feature matching method may be an optical flow method, feature point matching, etc. The embodiment of the present disclosure does not specifically limit the feature matching method.

[0117] In one embodiment, S140, performing feature matching on each facade in the second image frame and each facade in the first image frame, and determining second three-dimensional information of each facade in the second image frame in the first coordinate system based on the feature matching result and the first three-dimensional information, may include the following steps:

[0118] Step a1, extracting feature points of each facade in the first image frame and feature points of each facade in the second image frame.

[0119] Among them, the feature point can be a corner point with prominent texture.

[0120] Step a2: performing feature matching on the feature points of each facade in the second image frame and the feature points of each facade in the first image frame.

[0121] Among them, a feature point of a facade is a feature point in the grayscale image of the facade.

[0122] Step a3: for any feature point of each facade in the second image frame, the first three-dimensional information corresponding to the target feature point matching the feature point in the first image frame is determined as the second three-dimensional information of the feature point in the first coordinate system.

[0123] Specifically, after matching the feature points of each facade in the second image frame with the features of each facade in the first image frame, the feature point of each facade in the second image frame corresponds to a feature point of the facade in the first image frame, and the three-dimensional information of the feature points of each facade in the first image frame in the first coordinate system can be calculated through the above steps. Therefore, the three-dimensional information of the feature points of each facade in the first image frame in the first coordinate system can be determined by the three-dimensional information of the feature points of each facade in the first image frame in the first coordinate system.

[0124] It can be seen that through the technical solution provided by the embodiment of the present disclosure, by matching the feature points of each facade in the second image frame with the feature points of each facade in the first image frame, and determining the feature points of each facade in the second image frame through the three-dimensional information of the matched feature points in the first image frame, the three-dimensional information of the target object is tracked, which is beneficial for the subsequent steps to merge the three-dimensional information obtained by tracking with the three-dimensional information newly added to the second image frame relative to the first image frame, so as to realize the continuous acquisition of the three-dimensional information of the target object during the movement process and improve the accuracy of the three-dimensional information of the target object.

[0125] S150, determining second plane information of each facade of the target object in the second image frame, fusing the second plane information with the first plane information to obtain fused plane information, and determining the three-dimensional information newly added to the second image frame relative to the first image frame in the first coordinate system based on the fused plane information.

[0126] From the above description, it can be seen that by performing feature matching on each facade in the second image frame with each facade in the first image frame, it is possible to track the three-dimensional information of each facade of the target object, that is, to obtain a tracking result. In order to more accurately and continuously complete the three-dimensional information of each facade of the target object in the second image frame, the second image frame can be input into a neural network to obtain second plane information of each facade of the target object in the second image frame, and the second plane information is fused with the first plane information to obtain fused plane information. Based on the fused plane information, the three-dimensional information newly added to the second image frame relative to the first image frame in the first coordinate system is determined.

[0127] The second plane information of each facade includes: a grayscale image corresponding to the facade, coordinates of the endpoints of the intersection lines of the facade with other facades in the second coordinate system, and a normal vector of the facade in the first coordinate system. Since the method for obtaining the first plane information of each facade of the target object in the first image frame is described in detail, the method for obtaining the second plane information of each facade of the target object in the second image frame will not be described in detail here.

[0128] For the sake of clarity, the specific implementation of S150 will be described in detail in the following embodiments.

[0129] The technical solution provided by the embodiment of the present disclosure is to obtain a target video, where the target video includes a first image frame and a second image frame, both of which include a target object, and a first moment corresponding to the first image frame in the target video is earlier than a second moment corresponding to the second image frame in the target video; determine first plane information of each facade of the target object in the first image frame, and determine first three-dimensional information of each facade of the target object in the first image frame in a first coordinate system based on the first plane information of each facade of the target object in the first image frame; perform feature matching on each facade in the second image frame with each facade in the first image frame, and determine second three-dimensional information of each facade in the second image frame in the first coordinate system based on the feature matching result and the first three-dimensional information; determine second plane information of each facade of the target object in the second image frame, fuse the second plane information with the first plane information to obtain fused plane information, and determine new three-dimensional information of the second image frame relative to the first image frame in the first coordinate system based on the fused plane information; fuse the second three-dimensional information with the new three-dimensional information to obtain target three-dimensional information of each facade of the target object in the second image frame in the first coordinate system.

[0130] It can be seen that through the technical solution provided by the embodiment of the present disclosure, when determining the three-dimensional information of the target object in the second image frame, the features of each facade in the second image frame are matched with those of each facade in the first image frame, that is, the tracking of the three-dimensional information of the target object in the second image frame is realized, and the tracking result is obtained, that is, the second three-dimensional information of each facade in the second image frame in the first coordinate system is obtained; and by fusing the tracking result with the newly added three-dimensional information of the second image frame relative to the first image frame in the first coordinate system, the three-dimensional information of the target object in the second image frame is obtained, instead of re-determining the three-dimensional information of the second image frame separately, thereby realizing the continuous acquisition of the three-dimensional information of the target object during the movement, thereby improving the accuracy of the three-dimensional information of the target object.

[0131] For the sake of completeness and clear description, the specific implementation of S130 will be described in detail in the following embodiments.

[0132] In one implementation, S130, based on the plane information of each facade of the target object in the first image frame, determine first three-dimensional information of feature points of each facade of the target object in the first image frame in the first coordinate system, such as Figure 4 As shown, the following steps may be included:

[0133] S131, calculating the coordinates of a first center point of a first facade of a target object in a first image frame in a second coordinate system.

[0134] Among them, the first center point is the center point of the grayscale image corresponding to the first facade.

[0135] Specifically, the first external facade may be one of the external facades of the target object. In practical applications, the first external facade may be any visible external facade of the target object, which is not specifically limited in the embodiments of the present disclosure.

[0136] After the first facade is selected, the first facade is used as the basic facade, and the two-dimensional coordinates of the center point of the grayscale image of the basic facade in the first coordinate system are calculated, which can be recorded as O1(u,v).

[0137] S132: Based on the coordinates of the first center point in the second coordinate system and the distance from the first center point to the shooting device, determine the coordinates of the first center point in the first coordinate system through a pinhole camera model.

[0138] After determining the coordinates of the first center point in the first coordinate system and the distance from the first center point to the shooting device, i.e., the camera, the coordinates of the first center point in the first coordinate system can be calculated using the pinhole camera model. The pinhole camera model is used to associate the three-dimensional coordinate points in the first coordinate system, i.e., the camera coordinate system, with the two-dimensional coordinate points in the first coordinate system, i.e., the screen coordinate system.

[0139] Assuming that the coordinates of the first center point in the first coordinate system are (u, v), assuming that the distance from the first center point to the camera is 1, and the coordinates in the first coordinate system are (X, Y, Z), then the three-dimensional coordinate point in the first coordinate system and the two-dimensional coordinate point in the first coordinate system can be associated by the following formula.

[0140]

[0141] Among them, f x , f y , c x , c y are the intrinsic parameters of the camera, f x Refers to the imaging focal length in the x-axis direction, f y Refers to the imaging focal length in the y-axis direction; c x is the image center offset in the x-axis direction, c y is the imaging center offset in the y-axis direction.

[0142] S133, determining the three-dimensional information of the second facade based on the coordinates of the endpoints of the intersection line of the first facade and the second facade in the second coordinate system and the three-dimensional information of the first facade.

[0143] Specifically, according to the coordinates of the endpoints of the intersection of the first facade and the second facade in the second coordinate system, and the three-dimensional information of the first facade, i.e., the coordinates of the plane normal vector n1 and the plane center point O in the camera coordinate system, and the center point O2 (u, v) of the second facade, the three-dimensional coordinates Oc2 (x, y, z) of the center point of the second facade in the first coordinate system can be calculated.

[0144] S134, based on the normal vector of the first facade in the first coordinate system and the coordinates of the first center point in the first coordinate system, calculate the coordinates of the feature point of the first facade in the first coordinate system. Based on the normal vector of the second facade in the first coordinate system and the coordinates of the second center point in the first coordinate system, calculate the coordinates of the feature point of the second facade in the first coordinate system.

[0145] Specifically, multiple feature points are extracted from the first facade, and the coordinates of each extracted feature point in the first coordinate system are calculated according to the coordinates of the first center point of the first facade in the first coordinate system and the first normal vector of the first facade in the first coordinate system. Multiple feature points are extracted from the second facade, and the coordinates of each extracted feature point in the first coordinate system are calculated according to the coordinates of the second center point of the second facade in the first coordinate system and the second normal vector of the second facade in the first coordinate system. Figure 2 shown.

[0146] It can be seen that through the technical solution provided by the embodiment of the present disclosure, the three-dimensional information of multiple feature points in each facade of the first image frame in the first coordinate system can be accurately determined, and the three-dimensional information of each facade in the first coordinate system can be obtained through the three-dimensional information of multiple feature points in each facade in the first coordinate system. The three-dimensional information of all facades in the first image frame in the first coordinate system constitutes the three-dimensional information of the target object in the first image frame, thereby accurately determining the three-dimensional information of the target object in the first image frame.

[0147] In order to more accurately determine the three-dimensional information of the target object in the second image frame, in one embodiment, the three-dimensional information determination method may further include the following steps, namely step b1 and step b2:

[0148] Step b1, for any feature point of each facade in the second image frame, project the three-dimensional information corresponding to the feature point through the target camera model to obtain the coordinates of the corresponding projected feature point in the first coordinate system: and calculate the error between the coordinates of the projected feature point in the second coordinate system and the coordinates of the feature point in the second coordinate system.

[0149] Step b2: determining, based on the error, the position and posture information generated by the shooting device during the process of shooting the second image frame and the first image frame, wherein the position and posture information includes rotation information and translation information.

[0150] Specifically, after determining the three-dimensional information of the second image frame based on the three-dimensional information of the first image frame, the three-dimensional information of the second image frame can be projected to the second coordinate system of the second image frame to obtain a two-dimensional feature point. The camera frame tracking also obtains the corresponding feature points. The positions of these two feature points have errors, and then the posture information generated by the shooting device in the process of shooting the second image frame and the first image frame can be determined based on the errors. The posture information includes rotation information and translation information. The posture information can be adjusted later through nonlinear optimization.

[0151] It can be seen that by determining the posture information generated by the shooting device in the process of shooting the second image frame and the first image frame, when the second plane information is fused with the first plane information in the subsequent steps, the influence of the posture information of the shooting device on the fusion result can be avoided, and the second plane information and the first plane information can be fused more accurately.

[0152] For the sake of clarity, the specific implementation of S150 will be described in detail in the following embodiments.

[0153] S150, fusing the second plane information with the first plane information to obtain fused plane information, and determining, based on the fused plane information, the three-dimensional information newly added to the second image frame relative to the first image frame in the first coordinate system, such as Figure 5 As shown, the following steps may be included:

[0154] S151, determining a target facade in a second image frame, and calculating three-dimensional information of feature points of the target facade in a first coordinate system corresponding to the first image frame.

[0155] The target facade is the same facade in the second image frame as that in the first image frame.

[0156] In order to accurately fuse the second plane information with the first plane information, it is necessary to determine the same facade in the first image frame and the second image frame, and for the same facade, it is necessary to determine the three-dimensional information of the feature points of the target facade in the second image frame in the first coordinate system corresponding to the first image frame.

[0157] In one implementation, S151 may include the following steps, namely steps c1 to c4:

[0158] Step c1, converting the second normal vector included in the second plane information into the first coordinate system corresponding to the first image frame through the posture information to obtain the converted normal vector.

[0159] Specifically, through the formula n cj =T cicj ·n ci , convert the second normal vector included in the second plane information of the newly predicted second image frame into the first coordinate system corresponding to the first image frame. Wherein, n cj is the transformed normal vector, T cicj is the pose information, n ci is the second normal vector.

[0160] Step c2, calculating the angle between the converted normal vector and the first normal vector included in the first plane information, and determining the same facade in the second image frame and the first image frame based on the angle.

[0161] Specifically, calculate n cj If the angle between the first plane information and the first normal vector is less than the threshold, the two facades are determined to be the same facade, otherwise, they are determined to be different facades. If they are the same facade, the first normal vector included in the first plane information is still used as the normal vector of the plane.

[0162] Step c3, extracting the feature points in the grayscale image included in the second plane information of the target facade, and based on the coordinates of the first center point corresponding to the feature point and the target facade in the second coordinate system, and the first normal vector of the target facade in the first coordinate system, calculating the three-dimensional information of the feature point in the first coordinate system corresponding to the first image frame.

[0163] Specifically, for the same facade, the contour points of the grayscale image of the facade are extracted to obtain a series of two-dimensional contour points P m j , using the existing plane center and plane normal vector of the plane, calculate and obtain a series of three-dimensional coordinates of contour points in the first coordinate system corresponding to the first image frame.

[0164] Step c4, converting the calculated three-dimensional information of the feature points into the first coordinate system corresponding to the first image frame through the posture information, and obtaining the converted three-dimensional information as the three-dimensional information of the feature points of the target facade in the first coordinate system corresponding to the first image frame.

[0165] Specifically, the contour point P of the second image frame is mj Through the following formula P mi =T cicj ·P mj Convert to P mi , and P mi Transform to the second coordinate system, where T cicjThe converted three-dimensional information is obtained as the three-dimensional information of the feature points of the target facade in the first coordinate system corresponding to the first image frame.

[0166] It can be seen that through the technical solution provided in this embodiment, when the second normal vector included in the second plane information is converted to the first coordinate system corresponding to the first image frame, the posture information generated by the shooting device in the process of shooting the second image frame and the first image frame is taken into consideration, thereby avoiding the influence caused by the movement of the shooting device, and accurately converting the second normal vector to the first coordinate system corresponding to the first image frame.

[0167] S152, projecting the calculated three-dimensional information into a second coordinate system corresponding to the first image frame to obtain a plurality of projection points, determining a second grayscale image based on the plurality of projection points, and fusing the second grayscale image with the first grayscale image, and calculating the three-dimensional information of the fused projection points in the first coordinate system corresponding to the second image frame.

[0168] Specifically, after calculating the three-dimensional information of the feature points of the target facade in the first coordinate system corresponding to the first image frame, the calculated three-dimensional information can be projected into the second coordinate system corresponding to the first image frame to obtain multiple projection points, which constitute the grayscale image under the first image frame. It is fused with the grayscale image obtained by tracking, that is, the image is ANDed at the pixel level. Finally, the contour points obtained after fusion are calculated in the world coordinate system. That is, the three-dimensional information of the fused projection points in the first coordinate system corresponding to the second image frame is calculated.

[0169] S153: Determine three-dimensional information of the endpoints of the target intersection line in the first coordinate system corresponding to the second image frame.

[0170] The target intersection line is the intersection line included in the second plane information corresponding to the target facade.

[0171] Specifically, in order to accurately realize the fusion of the second plane information and the first plane information, while projecting the feature points of the target facade onto the first image frame, it is also necessary to update the coordinates of the endpoints of the three-dimensional information intersection line in the first coordinate system corresponding to the second image frame.

[0172] In one implementation, S153 may include the following two steps:

[0173] Step d1, projecting the endpoints of the target intersection line onto the endpoint intersection line of the first image frame.

[0174] The endpoint intersection line is determined based on the coordinates of the endpoints of the intersection line included in the first plane information in the second coordinate system;

[0175] Step d2, determining the three-dimensional information of the endpoint of the target intersection line in the first coordinate system corresponding to the second image frame by using the coordinates of the first center point corresponding to the projection endpoint and the target facade in the second coordinate system, and the first normal vector of the target facade in the first coordinate system.

[0176] Specifically, the newly predicted intersection endpoint of the second image frame is vertically projected onto the intersection line of the tracked first image frame, and the coordinates of the intersection endpoint in the camera coordinate system corresponding to the second image frame are calculated using the plane center and the normal vector, thereby updating the coordinates of the intersection endpoint in the first coordinate system.

[0177] It can be seen that through the technical solution provided in this embodiment, by projecting the intersection endpoint of the second image frame onto the intersection line of the tracked first image frame, and calculating the coordinates of the projected intersection endpoint in the camera coordinate system corresponding to the second image frame, it is possible to accurately update the coordinates of the intersection endpoint in the first coordinate system, that is, to make the coordinates of the intersection endpoint in the first coordinate system more accurate.

[0178] S154, determining the three-dimensional information of the fused projection point in the first coordinate system corresponding to the second image frame and the three-dimensional information of the endpoint of the target intersection line in the first coordinate system corresponding to the second image frame as the three-dimensional information newly added to the second image frame in the first coordinate system relative to the first image frame.

[0179] Specifically, the three-dimensional information newly added to the second image frame in the first coordinate system relative to the first image frame may include two parts. The first part is the three-dimensional information of the fused projection point in the first coordinate system corresponding to the second image frame, and the other part is the three-dimensional information of the endpoint of the target intersection line in the first coordinate system corresponding to the second image frame.

[0180] It can be seen that through this embodiment, by fusing the plane information of the first image frame with the plane information of the second image frame, and determining the three-dimensional information of the fused projection point in the first coordinate system corresponding to the second image frame, and the three-dimensional information of the endpoint of the target intersection line in the first coordinate system corresponding to the second image frame as the three-dimensional information newly added to the second image frame in the first coordinate system relative to the first image frame, it is possible to continuously obtain the three-dimensional information of the newly added facade of the second image frame relative to the first image frame, thereby continuously increasing the three-dimensional information of the newly photographed facade.

[0181] For the sake of clarity, the technical solution of the present disclosure will be described in detail below in conjunction with a specific embodiment.

[0182] like Figure 6 As shown, another three-dimensional information determination method provided by an embodiment of the present disclosure may include the following steps:

[0183] S610, obtaining a target video.

[0184] The target video includes a first image frame and a second image frame, both the first image frame and the second image frame include a target object, and a first moment corresponding to the first image frame in the target video is earlier than a second moment corresponding to the second image frame in the target video.

[0185] S620, sequentially inputting the first image frame into a second neural network of a pre-trained first neural network to obtain first plane information of each facade of the target object in the first image frame.

[0186] Among them, the first plane information includes the first grayscale image corresponding to the first image area, the first coordinates of the endpoints of the intersection lines of the facades of the target object in the first image area in the second coordinate system, and the first normal vectors of each facade of the target object in the first image area in the first coordinate system.

[0187] S630: Determine first three-dimensional information of each external surface of the target object in the first image frame in the first coordinate system according to first plane information of each external surface of the target object in the first image frame.

[0188] The specific steps may be: calculating the coordinates of a first center point of a first facade of the target object in the first image frame in the second coordinate system, the first center point being the center point of a grayscale image corresponding to the first facade;

[0189] Based on the coordinates of the first center point in the second coordinate system and the distance from the first center point to the shooting device, determine the coordinates of the first center point in the first coordinate system through a pinhole camera model;

[0190] Determine the three-dimensional information of the second facade based on the coordinates of the endpoints of the intersection line of the first facade and the second facade of the target object in the second coordinate system and the three-dimensional information of the first facade;

[0191] Based on the normal vector of the first facade in the first coordinate system and the coordinates of the first center point in the first coordinate system, the coordinates of the feature point of the first facade in the first coordinate system are calculated; based on the normal vector of the second facade in the first coordinate system and the coordinates of the second center point in the first coordinate system, the coordinates of the feature point of the second facade in the first coordinate system are calculated, and the second center point is the center point of the grayscale image corresponding to the second facade.

[0192] S640, matching feature points of the first image frame and the second image frame, for each feature point in the second image frame, determining a target feature point in the first image frame that matches the feature point, and determining the three-dimensional information of the target feature point in the first coordinate system as the three-dimensional information of the feature point in the first coordinate system.

[0193] Specifically, this step can be divided into the following three sub-steps:

[0194] The first sub-step is to extract feature points of each facade in the first image frame and feature points of each facade in the second image frame, wherein the feature points may be corner points with prominent textures.

[0195] The second sub-step is to perform feature matching on the feature points of each facade in the second image frame and the feature points of each facade in the first image frame, wherein the feature points of each facade are feature points in the grayscale image of the facade.

[0196] The third sub-step is to determine, for any feature point of each facade in the second image frame, the first three-dimensional information corresponding to the target feature point matching the feature point in the first image frame as the second three-dimensional information of the feature point in the first coordinate system.

[0197] S650, projecting the three-dimensional information of each feature point of the second image frame in the first coordinate system to the second coordinate system, obtaining the coordinates of the corresponding projected feature point in the second coordinate system, and calculating the posture information of the shooting device based on the error between the coordinates of the projected feature point in the second coordinate system and the coordinates of the feature point in the second coordinate system.

[0198] Specifically, this step can be divided into the following two sub-steps:

[0199] The first sub-step is: for any feature point of each facade in the second image frame, the three-dimensional information corresponding to the feature point is projected through the target camera model to obtain the coordinates of the corresponding projected feature point in the first coordinate system: and the error between the coordinates of the projected feature point in the second coordinate system and the coordinates of the feature point in the second coordinate system is calculated.

[0200] The second sub-step is: determining, based on the error, the posture information generated in the process of the shooting device shooting the second image frame and the first image frame, where the posture information includes rotation information and translation information.

[0201] S660, sequentially input the second image frame into the second neural network of the pre-trained first neural network to obtain second plane information of each facade of the target object in the second image frame; based on the posture information calculated in the previous step, fuse the second plane information with the first plane information to obtain fused plane information; and determine the three-dimensional information newly added to the second image frame relative to the first image frame based on the fused plane information.

[0202] This step can be divided into the following four sub-steps:

[0203] The first sub-step is: determining the target facade in the second image frame, and calculating the three-dimensional information of the feature points of the target facade in the first coordinate system corresponding to the first image frame.

[0204] The second sub-step is: projecting the calculated three-dimensional information into a second coordinate system corresponding to the first image frame to obtain multiple projection points, determining a second grayscale image based on the multiple projection points, and fusing the second grayscale image with the first grayscale image, and calculating the three-dimensional information of the fused projection points in the first coordinate system corresponding to the second image frame.

[0205] The third sub-step is: determining the three-dimensional information of the endpoints of the target intersection line in the first coordinate system corresponding to the second image frame.

[0206] The fourth sub-step is to determine the three-dimensional information of the fused projection point in the first coordinate system corresponding to the second image frame and the three-dimensional information of the endpoint of the target intersection line in the first coordinate system corresponding to the second image frame as the three-dimensional information newly added to the second image frame in the first coordinate system relative to the first image frame.

[0207] S670: Determine the second three-dimensional information and the newly added three-dimensional information as target three-dimensional information of each facade of the target object in the second image frame in the first coordinate system.

[0208] Since the above S610 to S670 have been explained in detail in the above embodiments, they will not be repeated here.

[0209] It can be seen that through the technical solution provided by the embodiment of the present disclosure, when determining the three-dimensional information of the target object in the second image frame, the features of each facade in the second image frame are matched with those of each facade in the first image frame, that is, the tracking of the three-dimensional information of the target object in the second image frame is realized, and the tracking result is obtained, that is, the second three-dimensional information of each facade in the second image frame in the first coordinate system is obtained; and by fusing the tracking result with the newly added three-dimensional information of the second image frame relative to the first image frame in the first coordinate system, the three-dimensional information of the target object in the second image frame is obtained, instead of re-determining the three-dimensional information of the second image frame separately, thereby realizing the continuous acquisition of the three-dimensional information of the target object during the movement, thereby improving the accuracy of the three-dimensional information of the target object.

[0210] In the embodiment of the present disclosure, a three-dimensional information determination device is also provided. Figure 7 As shown, the device comprises:

[0211] The target video acquisition module 710 is configured to acquire a target video, wherein the target video includes a first image frame and a second image frame, wherein both the first image frame and the second image frame include a target object, and a first time corresponding to the first image frame in the target video is earlier than a second time corresponding to the second image frame in the target video;

[0212] A plane information determination module 720 is configured to determine first plane information of each facade of the target object in the first image frame;

[0213] The first three-dimensional information determining module 730 is configured to determine, based on the first plane information of the respective facades of the target object in the first image frame, the first three-dimensional information of each facade of the target object in the first image frame in the first coordinate system, wherein the first coordinate system is a coordinate system established with the shooting device shooting the target video as the origin;

[0214] The second three-dimensional information determination module 740 is configured to perform feature matching between each facade in the second image frame and each facade in the first image frame, and determine second three-dimensional information of each facade in the second image frame in the first coordinate system based on the feature matching result and the first three-dimensional information;

[0215] The plane information fusion module 750 is configured to determine the second plane information of each facade of the target object in the second image frame, fuse the second plane information with the first plane information to obtain fused plane information, and determine the three-dimensional information newly added to the second image frame relative to the first image frame in the first coordinate system based on the fused plane information;

[0216] The target three-dimensional information determination module 760 is configured to perform fusion of the second three-dimensional information and the newly added three-dimensional information to obtain target three-dimensional information of each facade of the target object in the second image frame in the first coordinate system.

[0217] It can be seen that through the technical solution provided by the embodiment of the present disclosure, when determining the three-dimensional information of the target object in the second image frame, the features of each facade in the second image frame are matched with those of each facade in the first image frame, that is, the tracking of the three-dimensional information of the target object in the second image frame is realized, and the tracking result is obtained, that is, the second three-dimensional information of each facade in the second image frame in the first coordinate system is obtained; and by fusing the tracking result with the newly added three-dimensional information of the second image frame relative to the first image frame in the first coordinate system, the three-dimensional information of the target object in the second image frame is obtained, instead of re-determining the three-dimensional information of the second image frame separately, thereby realizing the continuous acquisition of the three-dimensional information of the target object during the movement, thereby improving the accuracy of the three-dimensional information of the target object.

[0218] Optionally, the first plane information determining module is specifically configured to execute:

[0219] Inputting the first image frame into a pre-trained first neural network to obtain a first image region where the target object is located in the first image frame;

[0220] The first image area is input into a pre-trained second neural network to obtain a first grayscale image corresponding to the first image area, the first coordinates of the endpoints of the intersection lines of the facades of the target object in the first image area in the second coordinate system, the first normal vectors of the facades of the target object in the first image area in the first coordinate system, and the second coordinate system is a coordinate system established with the target point in the image frame as the origin.

[0221] Optionally, the first three-dimensional information determining module is specifically configured to execute:

[0222] Calculating the coordinates of a first center point of a first facade of a target object in the first image frame in the second coordinate system, where the first center point is a center point of a grayscale image corresponding to the first facade;

[0223] Determine the coordinates of the first center point in the first coordinate system by using a pinhole camera model based on the coordinates of the first center point in the second coordinate system and the distance from the first center point to the shooting device;

[0224] Determine the three-dimensional information of the second facade based on the coordinates of the endpoints of the intersection line of the first facade and the second facade in the second coordinate system and the three-dimensional information of the first facade;

[0225] Based on the normal vector of the first facade in the first coordinate system and the coordinates of the first center point in the first coordinate system, the coordinates of the feature point of the first facade in the first coordinate system are calculated; based on the normal vector of the second facade in the first coordinate system and the coordinates of the second center point in the first coordinate system, the coordinates of the feature point of the second facade in the first coordinate system are calculated.

[0226] Optionally, the second three-dimensional information determining module is configured to execute, including:

[0227] Extracting feature points of each facade in the first image frame and feature points of each facade in the second image frame;

[0228] Perform feature matching on the feature points of each facade in the second image frame and the feature points of each facade in the first image frame; the feature points of the facade are the feature points in the grayscale image of the facade;

[0229] For any feature point of each facade in the second image frame, the first three-dimensional information corresponding to the target feature point matching the feature point in the first image frame is determined as the second three-dimensional information of the feature point in the first coordinate system.

[0230] Optionally, the device further comprises:

[0231] A feature point projection module is configured to perform, for any feature point of each facade in the second image frame, projecting the target three-dimensional information corresponding to the feature point through the target camera model to obtain the coordinates of the corresponding projected feature point in the second coordinate system: and calculating the error between the coordinates of the projected feature point in the second coordinate system and the coordinates of the feature point in the second coordinate system;

[0232] The posture information determination module is configured to determine the posture information generated in the process of the shooting device shooting the second image frame and the first image frame based on the error, and the posture information includes rotation information and translation information.

[0233] Optionally, the plane information fusion module includes:

[0234] A first three-dimensional information determining unit is configured to determine a target facade in the second image frame and calculate three-dimensional information of feature points of the target facade in a first coordinate system corresponding to the first image frame; the target facade is the same facade in the second image frame as that in the first image frame;

[0235] A second three-dimensional information determining unit is configured to project the calculated three-dimensional information into a second coordinate system corresponding to the first image frame to obtain a plurality of projection points, determine a second grayscale image based on the plurality of projection points, fuse the second grayscale image with the first grayscale image, and calculate the three-dimensional information of the fused projection points in the first coordinate system corresponding to the second image frame;

[0236] A third three-dimensional information determining unit is configured to determine three-dimensional information of an endpoint of a target intersection line in a first coordinate system corresponding to the second image frame, wherein the target intersection line is an intersection line included in the second plane information corresponding to the target facade;

[0237] The plane information fusion unit is configured to execute the three-dimensional information of the fused projection point in the first coordinate system corresponding to the second image frame, and the three-dimensional information of the endpoint of the target intersection line in the first coordinate system corresponding to the second image frame, to determine the three-dimensional information newly added to the second image frame relative to the first image frame in the first coordinate system.

[0238] Optionally, the first three-dimensional information determining unit is specifically configured to execute;

[0239] Converting the second normal vector included in the second plane information into the first coordinate system corresponding to the first image frame by using the posture information to obtain a converted normal vector;

[0240] Calculating an angle between the converted normal vector and a first normal vector included in the first plane information, and determining the same facade in the second image frame and the first image frame based on the angle;

[0241] Extracting feature points in the grayscale image included in the second plane information of the target facade, and calculating the three-dimensional information of the feature points in the first coordinate system corresponding to the first image frame based on the coordinates of the first center point corresponding to the feature points and the target facade in the second coordinate system, and the first normal vector of the target facade in the first coordinate system;

[0242] The calculated three-dimensional information of the feature points is converted into a first coordinate system corresponding to the first image frame through the posture information to obtain the converted three-dimensional information as the three-dimensional information of the feature points of the target facade in the first coordinate system corresponding to the first image frame.

[0243] Optionally, the third three-dimensional information determining unit is specifically configured to execute:

[0244] Projecting the endpoint of the target intersection line onto the endpoint intersection line of the first image frame; the endpoint intersection line is determined based on the coordinates of the endpoint of the intersection line included in the first plane information in the second coordinate system;

[0245] The three-dimensional information of the endpoint of the target intersection line in the first coordinate system corresponding to the second image frame is determined by the coordinates of the first center point corresponding to the projection endpoint and the target facade in the second coordinate system, and the first normal vector of the target facade in the first coordinate system.

[0246] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0247] Figure 8 8 is a block diagram of a three-dimensional information determination device 800 according to an exemplary embodiment. For example, the device 800 is an electronic device, which may be a mobile phone, a computer, a digital broadcast terminal, a message transceiver device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0248] Reference Figure 8 , the device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 817 .

[0249] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0250] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0251] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.

[0252] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0253] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 817. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0254] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0255] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also detect the position change of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0256] The communication component 817 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 817 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 818 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0257] In an exemplary embodiment, the device 800 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned three-dimensional information determination method.

[0258] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the device 800 to perform the above method. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0259] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method described in the first aspect.

[0260] It can be seen that through the technical solution provided by the embodiment of the present disclosure, when determining the three-dimensional information of the target object in the second image frame, the features of each facade in the second image frame are matched with those of each facade in the first image frame, that is, the tracking of the three-dimensional information of the target object in the second image frame is realized, and the tracking result is obtained, that is, the second three-dimensional information of each facade in the second image frame in the first coordinate system is obtained; and by fusing the tracking result with the newly added three-dimensional information of the second image frame relative to the first image frame in the first coordinate system, the three-dimensional information of the target object in the second image frame is obtained, instead of re-determining the three-dimensional information of the second image frame separately, thereby realizing the continuous acquisition of the three-dimensional information of the target object during the movement, thereby improving the accuracy of the three-dimensional information of the target object.

[0261] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, which, when executed on a computer, enables the computer to execute the steps of the method described in the first aspect.

[0262] It can be seen that through the technical solution provided by the embodiment of the present disclosure, when determining the three-dimensional information of the target object in the second image frame, the features of each facade in the second image frame are matched with those of each facade in the first image frame, that is, the tracking of the three-dimensional information of the target object in the second image frame is realized, and the tracking result is obtained, that is, the second three-dimensional information of each facade in the second image frame in the first coordinate system is obtained; and by fusing the tracking result with the newly added three-dimensional information of the second image frame relative to the first image frame in the first coordinate system, the three-dimensional information of the target object in the second image frame is obtained, instead of re-determining the three-dimensional information of the second image frame separately, thereby realizing the continuous acquisition of the three-dimensional information of the target object during the movement, thereby improving the accuracy of the three-dimensional information of the target object.

[0263] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, DSL (Digital Subscriber Line, digital subscriber line)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD (Digital Versatile Disc)), or a semiconductor medium (eg, an SSD (Solid State Disk)), etc.

[0264] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0265] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0266] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for determining three-dimensional information, characterized in that: include: Get the target video; The target video includes a first image frame and a second image frame, the first image frame and the second image frame both include a target object, and a first time point corresponding to the first image frame in the target video is earlier than a second time point corresponding to the second image frame in the target video; Determine the first plane information of each facade of the target object in the first image frame; wherein the first plane information of each facade includes: a grayscale image corresponding to the facade, coordinates of the endpoints of the intersection lines of the facade with other facades in the second coordinate system, and a normal vector of the facade in the first coordinate system; Based on first plane information of each facade of the target object in the first image frame, determine first three-dimensional information of each facade of the target object in the first image frame in the first coordinate system, where the first coordinate system is a coordinate system established with a shooting device that shoots the target video as an origin; Performing feature matching on each facade in the second image frame and each facade in the first image frame, and determining second three-dimensional information of each facade in the second image frame in the first coordinate system based on the feature matching result and the first three-dimensional information; Determine the second plane information of each facade of the target object in the second image frame, fuse the second plane information with the first plane information to obtain fused plane information, and determine the three-dimensional information newly added to the second image frame relative to the first image frame in the first coordinate system based on the fused plane information; wherein the second plane information of each facade includes: a grayscale image corresponding to the facade, coordinates of the endpoints of the intersection line of the facade with other facades in the second coordinate system, and a normal vector of the facade in the first coordinate system; The second three-dimensional information and the newly added three-dimensional information are fused to obtain target three-dimensional information of each facade of the target object in the second image frame in the first coordinate system.

2. The method according to claim 1, characterized in that The determining first plane information of each facade of the target object in the first image frame includes: Inputting the first image frame into a pre-trained first neural network to obtain a first image region where the target object is located in the first image frame; The first image area is input into a pre-trained second neural network to obtain a first grayscale image corresponding to the first image area, the first coordinates of the endpoints of the intersection lines of the facades of the target object in the first image area in the second coordinate system, the first normal vectors of the facades of the target object in the first image area in the first coordinate system, and the second coordinate system is a coordinate system established with the target point in the first image frame as the origin.

3. The method according to claim 1, characterized in that: The determining, based on the first plane information of each facade of the target object in the first image frame, first three-dimensional information of each facade of the target object in the first image frame in the first coordinate system includes: Calculating the coordinates of a first center point of a first facade of a target object in the first image frame in the second coordinate system, where the first center point is a center point of a grayscale image corresponding to the first facade; Determine the coordinates of the first center point in the first coordinate system by using a pinhole camera model based on the coordinates of the first center point in the second coordinate system and the distance from the first center point to the shooting device; Determine the three-dimensional information of the second facade based on the coordinates of the endpoints of the intersection line of the first facade and the second facade of the target object in the second coordinate system and the three-dimensional information of the first facade; Based on the normal vector of the first facade in the first coordinate system and the coordinates of the first center point in the first coordinate system, the coordinates of the feature point of the first facade in the first coordinate system are calculated; based on the normal vector of the second facade in the first coordinate system and the coordinates of the second center point in the first coordinate system, the coordinates of the feature point of the second facade in the first coordinate system are calculated, and the second center point is the center point of the grayscale image corresponding to the second facade.

4. The method according to any one of claims 1 to 3, characterized in that: The step of performing feature matching on each facade in the second image frame and each facade in the first image frame, and determining second three-dimensional information of each facade in the second image frame in the first coordinate system based on the feature matching result and the first three-dimensional information, includes: Extracting feature points of each facade in the first image frame and feature points of each facade in the second image frame; Perform feature matching on the feature points of each facade in the second image frame and the feature points of each facade in the first image frame; the feature points of each facade are the feature points in the grayscale image of the facade; For any feature point of each facade in the second image frame, the first three-dimensional information corresponding to the target feature point matching the feature point in the first image frame is determined as the second three-dimensional information of the feature point in the first coordinate system.

5. The method according to claim 2, characterized in that: The method further comprises: For any feature point of each facade in the second image frame, project the target three-dimensional information corresponding to the feature point through the target camera model to obtain the coordinates of the corresponding projected feature point in the second coordinate system: and calculate the error between the coordinates of the projected feature point in the second coordinate system and the coordinates of the feature point in the second coordinate system; Based on the error, position information generated by the shooting device in the process of shooting the second image frame and the first image frame is determined, and the position information includes rotation information and translation information.

6. The method according to claim 5, characterized in that The fusing the second plane information with the first plane information to obtain fused plane information, and determining, based on the fused plane information, three-dimensional information newly added to the second image frame relative to the first image frame in the first coordinate system, comprises: Determine a target facade in the second image frame, and calculate three-dimensional information of feature points of the target facade in a first coordinate system corresponding to the first image frame; the target facade is the same facade in the second image frame as that in the first image frame; Projecting the calculated three-dimensional information into a second coordinate system corresponding to the first image frame to obtain a plurality of projection points, determining a second grayscale image based on the plurality of projection points, fusing the second grayscale image with the first grayscale image, and calculating the three-dimensional information of the fused projection points in the first coordinate system corresponding to the second image frame; Determine three-dimensional information of an endpoint of a target intersection line in a first coordinate system corresponding to the second image frame, the target intersection line being an intersection line included in second plane information corresponding to the target facade; The three-dimensional information of the fused projection point in the first coordinate system corresponding to the second image frame and the three-dimensional information of the endpoint of the target intersection line in the first coordinate system corresponding to the second image frame are determined as the three-dimensional information newly added to the second image frame relative to the first image frame in the first coordinate system.

7. The method according to claim 6, characterized in that The determining of the target facade in the second image frame and calculating the three-dimensional information of the feature points of the target facade in the first coordinate system corresponding to the first image frame includes: Converting the second normal vector included in the second plane information into the first coordinate system corresponding to the first image frame by using the posture information to obtain a converted normal vector; Calculating an angle between the converted normal vector and a first normal vector included in the first plane information, and determining the same facade in the second image frame and the first image frame based on the angle; Extracting feature points in the grayscale image included in the second plane information of the target facade, and calculating the three-dimensional information of the feature points in the first coordinate system corresponding to the first image frame based on the coordinates of the first center point corresponding to the feature points and the target facade in the second coordinate system, and the first normal vector of the target facade in the first coordinate system; The calculated three-dimensional information of the feature points is converted into a first coordinate system corresponding to the first image frame through the posture information to obtain the converted three-dimensional information as the three-dimensional information of the feature points of the target facade in the first coordinate system corresponding to the first image frame.

8. The method according to claim 6, characterized in that The determining of three-dimensional information of the endpoints of the target intersection line in the first coordinate system corresponding to the second image frame includes: Projecting the endpoint of the target intersection line onto the endpoint intersection line of the first image frame; the endpoint intersection line is determined based on the coordinates of the endpoint of the intersection line included in the first plane information in the second coordinate system; The three-dimensional information of the endpoint of the target intersection line in the first coordinate system corresponding to the second image frame is determined by the coordinates of the first center point corresponding to the projection endpoint and the target facade in the second coordinate system, and the first normal vector of the target facade in the first coordinate system.

9. A three-dimensional information determination device, characterized in that: include: A target video acquisition module is configured to acquire a target video, wherein the target video includes a first image frame and a second image frame, wherein both the first image frame and the second image frame include a target object, and a first moment corresponding to the first image frame in the target video is earlier than a second moment corresponding to the second image frame in the target video; A plane information determination module is configured to determine first plane information of each facade of the target object in the first image frame; wherein the first plane information of each facade includes: a grayscale image corresponding to the facade, coordinates of endpoints of intersection lines of the facade with other facades in the second coordinate system, and a normal vector of the facade in the first coordinate system; A first three-dimensional information determination module is configured to determine first three-dimensional information of each facade of the target object in the first image frame in the first coordinate system based on first plane information of each facade of the target object in the first image frame, wherein the first coordinate system is a coordinate system established with a shooting device shooting the target video as an origin; a second three-dimensional information determination module, configured to perform feature matching between each facade in the second image frame and each facade in the first image frame, and determine second three-dimensional information of each facade in the second image frame in the first coordinate system based on the feature matching result and the first three-dimensional information; a plane information fusion module, configured to determine the second plane information of each facade of the target object in the second image frame, fuse the second plane information with the first plane information to obtain fused plane information, and determine the three-dimensional information newly added to the second image frame relative to the first image frame in the first coordinate system based on the fused plane information; wherein the second plane information of each facade includes: a grayscale image corresponding to the facade, coordinates of the endpoints of the intersection line of the facade with other facades in the second coordinate system, and a normal vector of the facade in the first coordinate system; The target three-dimensional information determination module is configured to perform fusion of the second three-dimensional information and the newly added three-dimensional information to obtain the target three-dimensional information of each facade of the target object in the second image frame in the first coordinate system.

10. The device according to claim 9, characterized in that The first plane information determination module is specifically configured to execute: Inputting the first image frame into a pre-trained first neural network to obtain a first image region where the target object is located in the first image frame; The first image area is input into a pre-trained second neural network to obtain a first grayscale image corresponding to the first image area, the first coordinates of the endpoints of the intersection lines of the facades of the target object in the first image area in the second coordinate system, the first normal vectors of the facades of the target object in the first image area in the first coordinate system, and the second coordinate system is a coordinate system established with the target point in the image frame as the origin.

11. The device according to claim 9, characterized in that The first three-dimensional information determination module is specifically configured to execute: Calculating the coordinates of a first center point of a first facade of a target object in the first image frame in the second coordinate system, where the first center point is a center point of a grayscale image corresponding to the first facade; Determine the coordinates of the first center point in the first coordinate system by using a pinhole camera model based on the coordinates of the first center point in the second coordinate system and the distance from the first center point to the shooting device; Determine the three-dimensional information of the second facade based on the coordinates of the endpoints of the intersection line of the first facade and the second facade in the second coordinate system and the three-dimensional information of the first facade; Based on the normal vector of the first facade in the first coordinate system and the coordinates of the first center point in the first coordinate system, the coordinates of the feature point of the first facade in the first coordinate system are calculated; based on the normal vector of the second facade in the first coordinate system and the coordinates of the second center point in the first coordinate system, the coordinates of the feature point of the second facade in the first coordinate system are calculated, and the second center point is the center point of the grayscale image corresponding to the second facade.

12. The device according to any one of claims 9 to 11, characterized in that The second three-dimensional information determination module is configured to execute, including: Extracting feature points of each facade in the first image frame and feature points of each facade in the second image frame; Perform feature matching on the feature points of each facade in the second image frame and the feature points of each facade in the first image frame; the feature points of each facade are the feature points in the grayscale image of the facade; For any feature point of each facade in the second image frame, the first three-dimensional information corresponding to the target feature point matching the feature point in the first image frame is determined as the second three-dimensional information of the feature point in the first coordinate system.

13. The device according to claim 10, characterized in that The device also includes: A feature point projection module is configured to perform, for any feature point of each facade in the second image frame, projecting the target three-dimensional information corresponding to the feature point through the target camera model to obtain the coordinates of the corresponding projected feature point in the second coordinate system: and calculating the error between the coordinates of the projected feature point in the second coordinate system and the coordinates of the feature point in the second coordinate system; The posture information determination module is configured to determine the posture information generated in the process of the shooting device shooting the second image frame and the first image frame based on the error, and the posture information includes rotation information and translation information.

14. The device according to claim 13, characterized in that The plane information fusion module includes: A first three-dimensional information determining unit is configured to determine a target facade in the second image frame and calculate three-dimensional information of feature points of the target facade in a first coordinate system corresponding to the first image frame; the target facade is the same facade in the second image frame as that in the first image frame; A second three-dimensional information determining unit is configured to project the calculated three-dimensional information into a second coordinate system corresponding to the first image frame to obtain a plurality of projection points, determine a second grayscale image based on the plurality of projection points, fuse the second grayscale image with the first grayscale image, and calculate the three-dimensional information of the fused projection points in the first coordinate system corresponding to the second image frame; A third three-dimensional information determining unit is configured to determine three-dimensional information of an endpoint of a target intersection line in a first coordinate system corresponding to the second image frame, wherein the target intersection line is an intersection line included in the second plane information corresponding to the target facade; The plane information fusion unit is configured to execute the three-dimensional information of the fused projection point in the first coordinate system corresponding to the second image frame, and the three-dimensional information of the endpoint of the target intersection line in the first coordinate system corresponding to the second image frame, to determine the three-dimensional information newly added to the second image frame relative to the first image frame in the first coordinate system.

15. The device according to claim 14, characterized in that The first three-dimensional information determining unit is specifically configured to execute; Converting the second normal vector included in the second plane information into the first coordinate system corresponding to the first image frame by using the posture information to obtain a converted normal vector; Calculating an angle between the converted normal vector and a first normal vector included in the first plane information, and determining the same facade in the second image frame and the first image frame based on the angle; Extracting feature points in the grayscale image included in the second plane information of the target facade, and calculating the three-dimensional information of the feature points in the first coordinate system corresponding to the first image frame based on the coordinates of the first center point corresponding to the feature points and the target facade in the second coordinate system, and the first normal vector of the target facade in the first coordinate system; The calculated three-dimensional information of the feature points is converted into a first coordinate system corresponding to the first image frame through the posture information to obtain the converted three-dimensional information as the three-dimensional information of the feature points of the target facade in the first coordinate system corresponding to the first image frame.

16. The device according to claim 14, characterized in that The third three-dimensional information determining unit is specifically configured to execute: Projecting the endpoint of the target intersection line onto the endpoint intersection line of the first image frame; the endpoint intersection line is determined based on the coordinates of the endpoint of the intersection line included in the first plane information in the second coordinate system; The three-dimensional information of the endpoint of the target intersection line in the first coordinate system corresponding to the second image frame is determined by the coordinates of the first center point corresponding to the projection endpoint and the target facade in the second coordinate system, and the first normal vector of the target facade in the first coordinate system.

17. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the method according to any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the steps of the method according to any one of claims 1 to 8.

19. A computer program product, characterized in that When the method is executed on a computer, the computer is enabled to execute the steps of the method according to any one of claims 1 to 8.