Data acquisition method, electronic equipment, readable storage medium and program product

Abstract

The invention provides a data acquisition method, electronic equipment, a readable storage medium and a program product, and belongs to the technical field of augmented reality. The method comprises the following steps: acquiring image information; based on the image information, determining first pose information of a collection point corresponding to the image information; based on the first pose information and second pose information of a reference acquisition point of a supplementary acquisition area, determining an association relationship between the acquisition point corresponding to the image information and the reference acquisition point of the supplementary acquisition area, the supplementary acquisition area being an area with data missing in a three-dimensional model; and determining a supplementary collection image set based on the association relationship and the image information.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of augmented reality technology, and in particular to a data acquisition method, an electronic device, a readable storage medium, and a program product. Background Art

[0002] With the rapid development of image processing technology, augmented reality (AR) technology has been widely used. The steps of 3D reconstruction of AR scenes based on monocular vision include: data collection and processing, sparse point cloud reconstruction, dense point cloud reconstruction, gridding, texture mapping, and then a textured 3D model can be generated. However, due to the lack of data collected in the relevant area, the generated 3D model may have holes, missing areas, patchiness and other problems in some areas, resulting in an incomplete 3D model. This will cause problems such as errors in the position of AR special effects and inaccurate spatial positioning, affecting the application experience related to AR scenes, such as AR navigation, tour guides, special effects display and other application experiences.

[0003] At present, in order to solve the problem of incomplete 3D models, the relevant technology is to manually confirm the target area that needs to collect additional data based on the 3D model; the data collectors go to the field again to collect additional data; the supplemented data is merged with the original data used to build the 3D model into a new data set; a new 3D model is recreated based on the new data set; after that, if the new 3D model is still incomplete, it is necessary to collect additional data based on the new 3D model again until a complete 3D model is obtained. In this way, each time a 3D reconstruction is completed, the integrity of the 3D model needs to be manually confirmed, and the data collectors may need to collect additional data multiple times, resulting in high costs for creating 3D models. Summary of the invention

[0004] The embodiments of the present invention provide a data collection method, an electronic device, a readable storage medium and a program product to solve the problem in the prior art that a collection person may need to perform multiple data supplement collections, resulting in high costs for creating a three-dimensional model.

[0005] In a first aspect, an embodiment of the present invention provides a data collection method, the method comprising:

[0006] Get image information;

[0007] Based on the image information, determining the first position information of the acquisition point corresponding to the image information;

[0008] Determine, based on the first posture information and second posture information of a reference collection point in a supplementary sampling area, an association relationship between a collection point corresponding to the image information and a reference collection point in the supplementary sampling area, wherein the supplementary sampling area is an area in which data is missing in the three-dimensional model;

[0009] Based on the association relationship and the image information, a set of supplementary images is determined.

[0010] In a second aspect, an embodiment of the present invention provides a data acquisition device, the device comprising:

[0011] A first acquisition module, used for acquiring image information;

[0012] A first determination module, used to determine the first position information of the acquisition point corresponding to the image information based on the image information;

[0013] A second determination module is used to determine the association relationship between the acquisition point corresponding to the image information and the reference acquisition point of the supplementary sampling area based on the first posture information and the second posture information of the reference acquisition point of the supplementary sampling area, wherein the supplementary sampling area is an area in which data is missing in the three-dimensional model;

[0014] The third determination module is used to determine a set of supplementary images based on the association relationship and the image information.

[0015] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the above-mentioned data acquisition method when executed by the processor.

[0016] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned data acquisition method are implemented.

[0017] In a fifth aspect, an embodiment of the present invention provides a computer program product, comprising computer instructions, which implement the steps of the above-mentioned data acquisition method when executed by a processor.

[0018] In an embodiment of the present invention, by predetermining the supplementary sampling area where data is missing and the second pose information of the reference acquisition point of the supplementary sampling area in the three-dimensional model, the association relationship between the acquisition point corresponding to the image information and the reference acquisition point of the supplementary sampling area is determined based on the first pose information determined by the image information and the second pose information of the reference acquisition point of the supplementary sampling area, and the supplementary sampling image set is determined based on the association relationship and the image information. In this way, after confirming the supplementary sampling area in the three-dimensional model, the second pose information of the reference acquisition point for collecting data in the supplementary sampling area can be automatically planned. In this way, when the user performs an AR application experience in a real scene, the data in the supplementary sampling area is automatically collected, thereby reducing the workload of the collection personnel, reducing the cost of creating the three-dimensional model, and improving the efficiency of creating the three-dimensional model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0020] Figure 1 It is a flow chart of a data collection method provided by an embodiment of the present invention;

[0021] Figure 2 It is a schematic diagram of the principle of determining the regional scope of the supplementary mining area;

[0022] Figure 3 It is a schematic diagram of the determination principle of the first plane;

[0023] Figure 4 It is a schematic diagram of the principle of determining the first plane information;

[0024] Figure 5 It is a schematic diagram of the principle of determining the third plane and the second plane information of the third plane;

[0025] Figure 6 It is a definite schematic diagram of the set of reference collection points;

[0026] Figure 7 It is a schematic diagram for determining the orientation of the reference collection point;

[0027] Figure 8 It is a schematic diagram for determining the viewing angle range of the reference acquisition point;

[0028] Fig. 9 is a structural schematic diagram of a data acquisition device provided by an embodiment of the present invention;

[0029] Fig.10 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] The following first describes the data collection method provided by the embodiment of the present invention.

[0032] It should be noted that the data acquisition method involves technical fields such as AR, three-dimensional reconstruction, image processing, and artificial intelligence, and can be specifically applied to AR application experience based on real scenes to build a three-dimensional model of the real scene. The data acquisition method can be applied to a data acquisition device, which may include a camera or have a receiving function, and can collect or receive images in the real scene of the three-dimensional model.

[0033] See also Figure 1 , the figure shows a schematic flow chart of the data collection method provided by an embodiment of the present invention.

[0034] like Figure 1 As shown, the method may include the following steps:

[0035] Step 101, obtaining image information.

[0036] In some embodiments, image information sent by other devices may be received, such as image RGB information sent by other devices. In some embodiments, the user may also turn on the camera in the data acquisition device during the AR application experience in the actual scene, and the data acquisition device may use its camera to scan and capture images in the actual scene to obtain image information, such as image RGB information.

[0037] Step 102: Based on the image information, determine the first position information of the acquisition point corresponding to the image information.

[0038] In some embodiments, if the image information is collected by the user through a camera in a data collection device, the user's posture information can be calculated in real time using existing or new posture solving technologies such as Simultaneous Localization and Mapping (SLAM) algorithm to obtain the user's first posture information, which is the posture information of the collection point corresponding to the image information. The first posture information may include fourth coordinate information representing the user's position and second orientation information representing the user's orientation.

[0039] Step 103, based on the first posture information and the second posture information of the reference collection point of the supplementary sampling area, determine the association relationship between the collection point corresponding to the image information and the reference collection point of the supplementary sampling area, wherein the supplementary sampling area is an area in which data is missing in the three-dimensional model.

[0040] In some embodiments, based on the first posture information and the supplementary sampling reference information of a predetermined supplementary sampling area, the association relationship between the acquisition point corresponding to the image information and the reference acquisition point of the supplementary sampling area is determined, and the supplementary sampling reference information includes the second posture information of the reference acquisition point. At the position of the reference acquisition point, data in the supplementary sampling area can be collected based on the second posture information.

[0041] Before step 103, the supplementary sampling area in the three-dimensional model can be determined. The three-dimensional model can be reconstructed by collecting some images. Based on the constructed three-dimensional model, the area where data is missing in the three-dimensional model can be determined to determine the supplementary sampling area.

[0042] In some embodiments, manual participation can be made to confirm the supplementary sampling area that needs additional data collection. The three-dimensional model of the actual scene can be constructed using three-dimensional reconstruction technology based on the original image set T. The three-dimensional model can be manually observed to mark incomplete areas in the three-dimensional model, such as areas with holes, missing areas, or patchiness. Afterwards, the incomplete areas in the three-dimensional model can be marked to form an information set of the supplementary sampling area, using S = {S1, S2, S3, ..., S i} indicates that, among them, S i represents the i-th supplementary mining area in the three-dimensional model, S i The first coordinate information of the i-th supplementary sampling area may be included.

[0043] The supplementary sampling reference information of the supplementary sampling area can be determined based on the information set of the supplementary sampling area. For each supplementary sampling area, the supplementary sampling reference information of the supplementary sampling area can be determined based on the first coordinate information of the supplementary sampling area. The supplementary sampling reference information can include a reference sampling point set of the supplementary sampling area and second posture information of each reference sampling point in the reference sampling point set. In some embodiments, the supplementary sampling reference information can also include area range information of the supplementary sampling area.

[0044] In some embodiments, the second posture information may include only the second coordinate information of the reference acquisition point and the viewing angle range based on the reference acquisition point that can collect data in the supplementary acquisition area corresponding to the reference acquisition point, that is, the determination of the association relationship between the acquisition point corresponding to the image information and the reference acquisition point can be realized. Of course, the second posture information may also include the first orientation information of the reference acquisition point. In some embodiments, the second posture information may include only the second coordinate information and the first orientation information of the reference acquisition point, that is, the determination of the association relationship between the acquisition point corresponding to the image information and the reference acquisition point can be realized. Of course, the second posture information may also include the viewing angle range. In some embodiments, the supplementary acquisition reference information may also include the area range information of the supplementary area.

[0045] In some embodiments, the surrounding position points of the position corresponding to the first coordinate information can be determined as reference collection points, and the reference collection point set and the second posture information of each reference collection point can be determined accordingly to obtain the supplementary collection reference information of the supplementary collection area. In some embodiments, for each supplementary collection area, before step 103, the method further includes:

[0046] Acquire first coordinate information of the supplementary mining area;

[0047] Based on the first coordinate information, determining the area range information of the supplementary mining area;

[0048] Based on the area range information, determining first plane information;

[0049] Based on the first plane information, second position information of the reference acquisition point is determined.

[0050] Among them, the first plane indicated by the first plane information is parallel to the second plane in the three-dimensional area indicated by the area range information, and the plane range of the first plane and the second plane is the same, the first plane is parallel to the preset plane, and the first distance between the first plane and the preset plane is a first preset value.

[0051] It is possible to manually confirm the supplementary sampling areas in the 3D model that need additional data collection, and obtain the first coordinate information of each supplementary sampling area accordingly. i According to the first coordinate information of the supplementary mining area, an existing or new algorithm is used to confirm the regional range of the supplementary mining area to obtain regional range information. The regional range of the supplementary mining area can surround the coordinate position of the supplementary mining area, and the regional range of the supplementary mining area can be any three-dimensional shape, such as a three-dimensional sphere, a cone, and a three-dimensional rectangle, which is not specifically limited here.

[0052] Take a three-dimensional rectangle as an example to illustrate the supplementary mining area S i The area range is C i Indicates that it is a spatial rectangular box, such as Figure 2 As shown, according to the first coordinate information of the supplementary mining area, the circumscribed three-dimensional rectangular frame R of the supplementary mining area can be obtained, the center point of the circumscribed three-dimensional rectangular frame R is Ro, and its area range information includes the coordinate values ​​of the eight vertices of the spatial rectangular frame.

[0053] Among them, the preset plane can be any plane, for example, the preset plane can be the ground, and the first preset value can be set according to the actual situation, such as being set to be close to the distance of the camera from the ground during the user's AR experience, such as 1.5 meters. The first plane can be determined at a plane position higher than the ground h and parallel to the ground. In some embodiments, the first plane is the second plane in the three-dimensional rectangular frame R. In some embodiments, due to the higher or lower position of the three-dimensional rectangular frame R, the first plane is not in the three-dimensional rectangular frame R, and the first plane is different from the second plane. Accordingly, based on the determination of the first plane, the first plane information of the first plane can be obtained, and the first plane information may include the coordinates of the four vertices of the rectangular frame or the coordinates of the two vertices on the diagonal.

[0054] In some embodiments, the boundary line R_line of the first plane may be determined based on the first plane information, such as Figure 3 As shown, the first plane is located in the three-dimensional model, and at a plane position higher than the ground h and parallel to the ground, the first plane in the three-dimensional model is determined, and a rectangular line R_line in the three-dimensional space is determined, and the rectangular line R_line can be the boundary line of the first plane, and is located on the surface of the circumscribed three-dimensional rectangular frame R. The reference acquisition points of the supplementary sampling area can be determined on the boundary line R_line of the first plane to obtain a reference acquisition point set, and accordingly, the second posture information of the reference acquisition points can be obtained.

[0055] In order to prevent the camera from being too close to the supplementary acquisition area, resulting in an incomplete acquired image, in some embodiments, determining the second pose information of the reference acquisition point based on the first plane information includes:

[0056] Determining second plane information based on the first plane information;

[0057] Based on the second plane information, determine a reference acquisition point of the supplementary acquisition area, and obtain second coordinate information of the reference acquisition point;

[0058] Based on the second coordinate information and the area range information, second position information of the reference collection point is determined.

[0059] The first plane is parallel to a third plane indicated by the second plane information, and the center points of the first plane and the third plane are the same, and the plane range of the third plane is larger than the plane range of the first plane.

[0060] In some embodiments, based on the second plane information, a reference acquisition point of the supplementary acquisition area can be determined on the boundary line of the third plane to obtain second coordinate information of the reference acquisition point.

[0061] In some embodiments, Figure 4As shown, the first plane is in the three-dimensional model, that is, the first plane is the second plane in the regional range of the supplementary mining area. A two-dimensional coordinate system can be established in the first plane. The center point R_line_o of the first plane can be used as the coordinate origin (0,0), the length parallel to the rectangular line R_line is the x-axis, and the width parallel to the rectangular line R_line is the y-axis. The two-dimensional coordinate system is constructed, and the rectangular line R_line can be represented by the coordinates of the two diagonal points in the first plane, that is, the first plane information can include the coordinates of the two diagonal points in the first plane, which are (x_line_o,y_line_o, ... min ,y min ) and (x max ,y max ). Among them, x min and x max Respectively represent the minimum and maximum values ​​of the coordinates of each position point on the first plane on the x-axis, and y min and max They respectively represent the minimum and maximum coordinates of each position point on the first plane on the y-axis.

[0062] In some embodiments, the third plane can be obtained by expanding the rectangle center point R_line_o outward by a certain distance to the reference rectangle. The boundary line of the third plane is represented by RN_line. RN_line can be represented by the coordinates of two diagonal points in the third plane. That is, the second plane information may include the coordinates of two diagonal points in the third plane, which are (x min -L x ,y min -L y ) and (x max +L x ,y max +L y ).

[0063] Among them, L x , L y are the expansion distances of the x and y axes, i.e., L x and L y are the length and width of R_line respectively, such as Figure 5 As shown. x and L y can be any length. In some embodiments, L x =max((x max -x min ) / 2,L min );L y =max((y max -y min ) / 2,L min ), where the function max(A,B) means taking the larger value between A and B, L minThe minimum extension distance can be set, for example, it can be set to 3 meters by default. Afterwards, the reference acquisition point of the supplementary acquisition area can be determined on the boundary line of the third plane based on the second plane information. In this way, image acquisition can be performed at the determined reference acquisition point to prevent the camera from being too close to the supplementary acquisition area, resulting in incomplete acquisition images, thereby improving the integrity and accuracy of image acquisition.

[0064] In some embodiments, the reference collection points of the supplementary collection area can be randomly determined on the boundary line of the third plane. In some embodiments, several points can be selected on the rectangular line RN_line to form a reference collection point set M, and the reference collection points are selected as follows:

[0065] The reference acquisition point set includes four vertices of the rectangular line RN_line;

[0066] The line connecting the center point R_line_o of the third plane and one of the vertices of the rectangular line RN_line is used as a reference. With R_line_o as the center, a ray is emitted every 10 degrees. The intersection point of the ray and the rectangular line RN_line can be added to the reference collection point set M, such as Figure 6 shown.

[0067] Correspondingly, the second coordinate information of the reference acquisition point can be determined based on the position of the selected reference acquisition point and the second plane information.

[0068] In some embodiments, the second posture information may include the first orientation information of the reference acquisition point. The reference acquisition point may be set as p, and p is connected with the center point Ro of the three-dimensional rectangular frame R to obtain the vector That is, vector The indicated direction is the first orientation information of the reference collection point, such as Figure 7 shown.

[0069] In some embodiments, the second position information may include a viewing angle range of the reference acquisition point. Optionally, determining the second position information of the reference acquisition point based on the second coordinate information and the area range information includes:

[0070] Determine first orientation information of the reference acquisition point based on the second coordinate information and the third coordinate information, wherein the third coordinate information indicates a center point position of a target area, and the target area is a three-dimensional area indicated by the area range information;

[0071] Determining a viewing angle range of the reference acquisition point based on the first orientation information and the area range information;

[0072] The second position information of the reference acquisition point is determined based on the second coordinate information and the viewing angle range of the reference acquisition point.

[0073] In some embodiments, the second posture information includes second coordinate information and a viewing angle range of the reference acquisition point.

[0074] In some embodiments, according to the area range information, it is determined that in the three-dimensional rectangular frame R indicated by the area range information, the direction indicated by the first direction information is the direction The two-dimensional plane under mapping can determine the left and right and upper and lower boundary ranges of the two-dimensional plane, and accordingly the viewing angle range of the reference acquisition point can be determined based on the boundary ranges, such as Figure 8 shown.

[0075] In some embodiments, for the i-th supplementary mining area S i , supplementary mining area S i The supplementary procurement reference information may include the following information:

[0076] Area range information, indicating the area range C i It is a spatial rectangular frame, containing 8 vertex coordinate values;

[0077] Reference collection point set M = {p1, p2, p3, ..., p m}, p m represents the mth reference acquisition point;

[0078] The second posture information of each reference collection point includes second coordinate information, first orientation information and viewing angle range.

[0079] In this embodiment, after manually confirming the supplementary sampling area in the three-dimensional model, the second posture information of the reference sampling point in the supplementary sampling area where data needs to be collected can be automatically planned, including the reference sampling point position and the viewing angle range of the data to be collected. In this way, data collection in the supplementary sampling area can be performed with reference to the reference sampling point position and the viewing angle range, thereby realizing automatic data collection in the supplementary sampling area.

[0080] In some embodiments, the first pose information and the second pose information of each reference acquisition point may be compared respectively, and the association relationship between the acquisition point corresponding to the image information and the reference acquisition point may be determined based on the comparison result. For example, the position between the acquisition point and the reference acquisition point may be compared, and the orientation between the acquisition point and the reference acquisition point may be compared. When the distance between the acquisition point and the reference acquisition point is relatively close, and the orientation angle between the acquisition point and the reference acquisition point is relatively small, it is determined that the acquisition point corresponding to the image information is associated with the reference acquisition point. Otherwise, the acquisition point corresponding to the image information is not associated with the reference acquisition point.

[0081] Optionally, the first posture information includes fourth coordinate information and second orientation information, and the second posture information includes second coordinate information and a viewing angle range of data that can be collected in the supplementary sampling area based on the reference sampling point; the step 103 specifically includes:

[0082] Determine a second distance between the reference acquisition point and the acquisition point corresponding to the image information based on the fourth coordinate information and the second coordinate information;

[0083] When the second distance is less than or equal to a second preset value and the second orientation information is within the viewing angle range, determining that the collection point corresponding to the image information is associated with the reference collection point;

[0084] When the second distance is greater than a second preset value, or the second orientation information is not within the viewing angle range, it is determined that the acquisition point corresponding to the image information is not associated with the reference acquisition point.

[0085] The second preset value may be set according to actual conditions, and represents the maximum allowable distance associated with the collection point and the reference collection point.

[0086] In this embodiment, by determining the second distance between the reference acquisition point and the acquisition point corresponding to the image information, and determining the relationship between the second orientation information of the acquisition point and the viewing angle range of the reference acquisition point, and by using the second distance and the determined relationship, the association relationship between the acquisition point corresponding to the image information and the reference acquisition point is determined. In this way, it is possible to avoid the situation where, when the orientation of the acquisition point is compared with the reference acquisition point, the orientation of the acquisition point is not within the viewing angle range of the reference acquisition point due to the small viewing angle range of the reference acquisition point, resulting in the acquisition image being irrelevant to the supplementary acquisition area, thereby improving the image acquisition accuracy of the supplementary acquisition area.

[0087] In some embodiments, there may be multiple supplementary sampling areas of the three-dimensional model, and there may also be multiple reference sampling points in the supplementary sampling areas, which will make the workload of determining the association relationship between the sampling points and the reference sampling points relatively large. Optionally, determining the second distance between the reference sampling point and the sampling point corresponding to the image information based on the fourth coordinate information and the second coordinate information includes:

[0088] When the fourth coordinate information is located within the area range indicated by the area range information of the supplementary acquisition area, a second distance between the reference acquisition point and the acquisition point corresponding to the image information is determined based on the fourth coordinate information and the second coordinate information.

[0089] That is, if the user's three-dimensional coordinates are not within the area range of the supplementary sampling area, it means that the image information is not the data that needs to be supplemented for the three-dimensional model. If the user's three-dimensional coordinates are within the area range of one or more supplementary sampling areas, the user's first posture information can be compared with the second posture information of the reference acquisition point, and the association relationship between the acquisition point corresponding to the image information and the reference acquisition point of the supplementary sampling area can be determined based on the comparison result. This can reduce the workload of determining the association relationship between the acquisition point and the reference acquisition point, thereby improving the data acquisition efficiency of the three-dimensional model.

[0090] In some embodiments, the process of determining the association relationship between the collection point and the reference collection point is as follows:

[0091] Traverse the area range of the information set S of the supplementary sampling area to determine whether the three-dimensional coordinates of the user are within the area range; if both are not within the area range of the supplementary sampling area, it means that the image information is not the data that needs to be supplemented for the three-dimensional model;

[0092] If the user's 3D coordinates are within the range of one or more supplementary sampling areas, then the reference sampling point set M of the supplementary sampling area is traversed; suppose the traversal reaches the jth reference sampling point, and p j Indicates that according to the user's three-dimensional coordinates and the reference acquisition point p j The three-dimensional coordinates of the two are calculated, and the distance dis between the two, that is, the second distance, is calculated. If dis is greater than the second preset value, it is determined that the acquisition point is close to the reference acquisition point p. j Not related;

[0093] If dis is less than or equal to the second preset value, it can be further determined whether the camera is facing the reference acquisition point p j If it is not within the viewing angle range, determine the acquisition point and the reference acquisition point p j Not associated, if within the viewing angle, determine the acquisition point and the reference acquisition point p j association.

[0094] Step 104: Determine a set of supplementary images based on the association relationship and the image information.

[0095] The supplementary sampling image set is used to reconstruct the supplementary sampling area in the three-dimensional model.

[0096] When a user experiences an AR application in a real scene, after determining that the collection point is associated with the reference collection point, the image information and information related to the image information can be stored, such as uploaded to the background system. The information uploaded to the background system can include:

[0097] User ID information, device ID information;

[0098] Image information;

[0099] The user's first position information;

[0100] The supplementary acquisition area to which the image information belongs, the associated reference acquisition point, and the distance dis between the associated reference acquisition point.

[0101] By maintaining a table, the image status of each supplementary sampling area can be known, as shown in Table 1 below.

[0102] Table 1 The images of the supplementary sampling area

[0103]

[0104] The image information collected by the user is the supplementary image, such as supplementary image 1. When the collection point corresponding to the image information is associated with the reference collection point, the image information can be added to the candidate image set of the reference collection point, that is, if the collection point corresponding to the image information is associated with the reference collection point 1, the image information can be added as supplementary image 1 to the candidate image set of the reference collection point 1 to form a candidate image set of the supplementary area.

[0105] In some embodiments, the candidate image set may be determined as a supplementary image set. In some embodiments, some images may be selected from the candidate image set as a supplementary image set, and the supplementary image set and the original image set used to construct the three-dimensional model may be combined as a new data set to reconstruct the three-dimensional model.

[0106] It should be noted that in order to improve the data collection efficiency of the supplementary acquisition area, when collecting data, the user can be guided to collect data at the corresponding reference acquisition point according to the image situation of each supplementary acquisition area. For example, if more images are collected at the acquisition point associated with reference acquisition point 1, and images have not been collected at the positions associated with other reference acquisition points, the user can be reminded to change the acquisition position to collect images of other supplementary acquisition areas or other positions in the supplementary acquisition area.

[0107] When the acquisition point corresponding to the image information is associated with the reference acquisition point, the image information acquired by the user can be added to the candidate image set, and when the candidate image set meets the one-time acquisition condition, the supplementary acquisition image set is determined based on the candidate image set. That is, optionally, the step 104 specifically includes:

[0108] In a case where the association relationship indicates that the acquisition point corresponding to the image information is associated with the reference acquisition point, adding the image information to a candidate image set of the reference acquisition point;

[0109] When the number of target reference acquisition points in each supplementary acquisition area of ​​the three-dimensional model reaches a third preset value, determining the supplementary acquisition image set based on the candidate image set;

[0110] The target reference acquisition point is a reference acquisition point in which the number of images in the candidate image set reaches a fourth preset value.

[0111] The third preset value may be set according to actual conditions, for example, the third preset value may be set to 3. The fourth preset value may also be set according to actual conditions, which is not specifically limited here.

[0112] The one-time supplementary sampling condition refers to the ability to construct a relatively complete supplementary sampling image set at one time. The one-time supplementary sampling condition is that the number of target reference acquisition points in each supplementary sampling area of ​​the three-dimensional model reaches a third preset value. In other words, when the number of target reference acquisition points in each supplementary sampling area of ​​the three-dimensional model reaches the third preset value, a relatively complete supplementary sampling image set can be constructed at one time based on the candidate image set, so that the three-dimensional model constructed by merging the supplementary sampling image set and the original image set into a new data set can be relatively complete.

[0113] For each supplementary sampling area, let Qcount be the number of target reference acquisition points. When the acquisition point corresponding to the image information is associated with the reference acquisition point, the number of images in the candidate image set corresponding to the reference acquisition point can be increased by 1. When the number of images in the candidate image set corresponding to the reference acquisition point reaches the minimum limit P min That is, when it reaches the fourth preset value, Qcount can be increased by 1.

[0114] When the number of target reference acquisition points in each supplementary acquisition area of ​​the three-dimensional model reaches a third preset value, the candidate image set can be determined as the supplementary acquisition image set. In some embodiments, some images can also be selected from the candidate image set.

[0115] The images in the candidate image set corresponding to each reference acquisition point can be sorted and updated regularly, and the update rules are as follows:

[0116] The images in the candidate image set can be divided into two parts: the confirmation set T and the candidate set W. The upper limit of the number of confirmation sets is set to k. The images in the candidate image set are sorted from small to large according to the distance dis to obtain the image list List.

[0117] Traverse the image list List, and set the traversal to the current image i:

[0118] If the number of images in the confirmed set T reaches the upper limit k, the remaining images are added to the candidate set W; if image i is the first image, add it to the confirmed set T, and use image i as the reference image I; calculate the image similarity between image i and reference image I, if the similarity is higher than the threshold s_r, add image i to the candidate set W; otherwise, add image i to the confirmed set T, use image i as the reference image I, and continue to traverse the image list List;

[0119] Until the traversal of the image list List is completed, or the number of images in the confirmation set T reaches the upper limit k, at this time, a confirmation set T with different images can be selected from the candidate image set corresponding to the reference acquisition point.

[0120] When the Qcount of all supplementary mining areas exceeds the threshold Q min That is, when the third preset value is used, images are selected from the candidate image set as the supplementary image set.

[0121] For each reference acquisition point, let the number of images selected from the candidate image set be f_k, and traverse each reference acquisition point:

[0122] If f_k is less than the number of confirmed sets T, select f_k images in ascending order of distance dis to add to the supplementary image set;

[0123] If f_k is greater than the number of confirmed sets T, all images in the confirmed set T can be added to the supplementary image set, and the remaining f_k-len(T) images are randomly selected from the candidate set W corresponding to the reference acquisition point and added to the supplementary image set; where len(T) is the length of the confirmed set T.

[0124] In this way, a relatively complete set of supplementary images can be constructed at one time, so that the three-dimensional model constructed by merging the supplementary image set with the original image set into a new data set can be relatively complete.

[0125] In this embodiment, by predetermining the supplementary sampling area where data is missing in the three-dimensional model and the second pose information of the reference acquisition point of the supplementary sampling area, the association relationship between the acquisition point corresponding to the image information and the reference acquisition point of the supplementary sampling area is determined based on the first pose information determined by the image information and the second pose information of the reference acquisition point of the supplementary sampling area, and the supplementary sampling image set is determined based on the association relationship and the image information. In this way, after confirming the supplementary sampling area in the three-dimensional model, the second pose information of the reference acquisition point for collecting data in the supplementary sampling area can be automatically planned. In this way, when the user performs an AR application experience in a real scene, the data in the supplementary sampling area is automatically collected, thereby reducing the workload of the collection personnel, reducing the cost of creating the three-dimensional model, and improving the efficiency of creating the three-dimensional model.

[0126] The data acquisition device provided by the embodiment of the present invention is described below.

[0127] See also Fig. 9 , the figure shows a schematic diagram of the structure of the data acquisition device provided by an embodiment of the present invention.

[0128] like Fig. 9 As shown, the data acquisition device 900 includes:

[0129] A first acquisition module 901, used to acquire image information;

[0130] A first determination module 902 is used to determine the first position information of the acquisition point corresponding to the image information based on the image information;

[0131] A second determination module 903 is used to determine the association relationship between the acquisition point corresponding to the image information and the reference acquisition point of the supplementary acquisition area based on the first posture information and the second posture information of the reference acquisition point of the supplementary acquisition area, wherein the supplementary acquisition area is an area in which data is missing in the three-dimensional model;

[0132] The third determination module 904 is configured to determine a set of supplementary images based on the association relationship and the image information.

[0133] Optionally, the device further comprises:

[0134] A second acquisition module is used to acquire first coordinate information of the supplementary mining area;

[0135] A fourth determination module, configured to determine area range information of the supplementary mining area based on the first coordinate information;

[0136] A fifth determining module, configured to determine first plane information based on the area range information;

[0137] The sixth determination module is used to determine the second posture information of the reference acquisition point based on the first plane information.

[0138] Optionally, the sixth determining module is specifically configured to:

[0139] Determining second plane information based on the first plane information;

[0140] Based on the second plane information, determine a reference acquisition point of the supplementary acquisition area, and obtain second coordinate information of the reference acquisition point;

[0141] Based on the second coordinate information and the area range information, second position information of the reference collection point is determined.

[0142] Optionally, the sixth determining module is specifically configured to:

[0143] Determine first orientation information of the reference acquisition point based on the second coordinate information and the third coordinate information, wherein the third coordinate information indicates a center point position of a target area, and the target area is a three-dimensional area indicated by the area range information;

[0144] Determining a viewing angle range of the reference acquisition point based on the first orientation information and the area range information;

[0145] The second position information of the reference acquisition point is determined based on the second coordinate information and the viewing angle range of the reference acquisition point.

[0146] Optionally, the first posture information includes fourth coordinate information and second orientation information, and the second posture information includes second coordinate information and a viewing angle range of data that can be collected in the supplementary sampling area based on the reference sampling point; the second determination module 903 is specifically used to:

[0147] Determine a second distance between the reference acquisition point and the acquisition point corresponding to the image information based on the fourth coordinate information and the second coordinate information;

[0148] When the second distance is less than or equal to a second preset value and the second orientation information is within the viewing angle range, determining that the collection point corresponding to the image information is associated with the reference collection point;

[0149] When the second distance is greater than a second preset value, or the second orientation information is not within the viewing angle range, it is determined that the acquisition point corresponding to the image information is not associated with the reference acquisition point.

[0150] Optionally, the second determining module 903 is further configured to:

[0151] When the fourth coordinate information is located within the area range indicated by the area range information of the supplementary acquisition area, a second distance between the reference acquisition point and the acquisition point corresponding to the image information is determined based on the fourth coordinate information and the second coordinate information.

[0152] Optionally, the third determining module 904 is specifically configured to:

[0153] In a case where the association relationship indicates that the acquisition point corresponding to the image information is associated with the reference acquisition point, adding the image information to a candidate image set of the reference acquisition point;

[0154] When the number of target reference acquisition points in each supplementary acquisition area of ​​the three-dimensional model reaches a third preset value, determining the supplementary acquisition image set based on the candidate image set;

[0155] The target reference acquisition point is a reference acquisition point in which the number of images in the candidate image set reaches a fourth preset value.

[0156] The above-mentioned data acquisition device 900 can implement each process implemented in the above-mentioned data acquisition method embodiment and can achieve the same technical effect. In order to avoid repetition, it will not be described here.

[0157] See also Fig.10, the figure shows a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Fig.10 As shown, the electronic device 1000 further includes: a processor 1001 , a memory 1002 , a user interface 1003 and a bus interface 1004 .

[0158] The processor 1001 is used to read the program in the memory 1002 and execute the following process:

[0159] Get image information;

[0160] Based on the image information, determining the first position information of the acquisition point corresponding to the image information;

[0161] Determine, based on the first posture information and second posture information of a reference collection point in a supplementary sampling area, an association relationship between a collection point corresponding to the image information and a reference collection point in the supplementary sampling area, wherein the supplementary sampling area is an area in which data is missing in the three-dimensional model;

[0162] Based on the association relationship and the image information, a supplementary image set is determined.

[0163] exist Fig.10 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically one or more processors represented by processor 1001 and various circuits of memory represented by memory 1002 are linked together. The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface 1004 provides an interface. For different user devices, the user interface 1003 may also be an interface that can be connected to external or internal devices, and the connected devices include but are not limited to keypads, displays, speakers, microphones, joysticks, etc.

[0164] The processor 1001 is responsible for managing the bus architecture and general processing, and the memory 1002 can store data used by the processor 1001 when performing operations.

[0165] Optionally, the processor 1001 is further configured to:

[0166] Acquire first coordinate information of the supplementary mining area;

[0167] Based on the first coordinate information, determining the area range information of the supplementary mining area;

[0168] Based on the area range information, determining first plane information;

[0169] Based on the first plane information, second position information of the reference acquisition point is determined.

[0170] Optionally, the processor 1001 is further configured to:

[0171] Determining second plane information based on the first plane information;

[0172] Based on the second plane information, determine a reference acquisition point of the supplementary acquisition area, and obtain second coordinate information of the reference acquisition point;

[0173] Based on the second coordinate information and the area range information, second position information of the reference collection point is determined.

[0174] Optionally, the processor 1001 is further configured to:

[0175] Determine first orientation information of the reference acquisition point based on the second coordinate information and the third coordinate information, wherein the third coordinate information indicates a center point position of a target area, and the target area is a three-dimensional area indicated by the area range information;

[0176] Determining a viewing angle range of the reference acquisition point based on the first orientation information and the area range information;

[0177] The second position information of the reference acquisition point is determined based on the second coordinate information and the viewing angle range of the reference acquisition point.

[0178] Optionally, the first posture information includes fourth coordinate information and second orientation information, and the second posture information includes second coordinate information and a viewing angle range of data that can be collected in the supplementary sampling area based on the reference sampling point; the processor 1001 is further configured to:

[0179] Determine a second distance between the reference acquisition point and the acquisition point corresponding to the image information based on the fourth coordinate information and the second coordinate information;

[0180] When the second distance is less than or equal to a second preset value and the second orientation information is within the viewing angle range, determining that the collection point corresponding to the image information is associated with the reference collection point;

[0181] When the second distance is greater than a second preset value, or the second orientation information is not within the viewing angle range, it is determined that the acquisition point corresponding to the image information is not associated with the reference acquisition point.

[0182] Optionally, the processor 1001 is further configured to:

[0183] When the fourth coordinate information is located within the area range indicated by the area range information of the supplementary acquisition area, a second distance between the reference acquisition point and the acquisition point corresponding to the image information is determined based on the fourth coordinate information and the second coordinate information.

[0184] Optionally, the processor 1001 is further configured to:

[0185] In a case where the association relationship indicates that the acquisition point corresponding to the image information is associated with the reference acquisition point, adding the image information to a candidate image set of the reference acquisition point;

[0186] When the number of target reference acquisition points in each supplementary acquisition area of ​​the three-dimensional model reaches a third preset value, determining the supplementary acquisition image set based on the candidate image set;

[0187] The target reference acquisition point is a reference acquisition point in which the number of images in the candidate image set reaches a fourth preset value.

[0188] Optionally, an embodiment of the present invention further provides an electronic device 1000, including a processor 1001, a memory 1002, and a computer program stored in the memory 1002 and executable on the processor 1001. When the computer program is executed by the processor 1001, each process of the above-mentioned data acquisition method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0189] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned data acquisition method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0190] The embodiment of the present application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, the various processes of the above-mentioned data acquisition method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, they are not repeated here.

[0191] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0192] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0193] In the embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0194] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0195] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0196] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.

[0197] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A data collection method, characterized in that: The method comprises: Get image information; Based on the image information, determining the first position information of the acquisition point corresponding to the image information; Determine, based on the first posture information and second posture information of a reference collection point in a supplementary sampling area, an association relationship between a collection point corresponding to the image information and a reference collection point in the supplementary sampling area, wherein the supplementary sampling area is an area in which data is missing in the three-dimensional model; Based on the association relationship and the image information, a supplementary image set is determined.

2. The method according to claim 1, characterized in that Before determining the association relationship between the acquisition point corresponding to the image information and the reference acquisition point of the supplementary sampling area based on the first posture information and the second posture information of the reference acquisition point of the supplementary sampling area, the method further includes: Acquire first coordinate information of the supplementary mining area; Based on the first coordinate information, determining the area range information of the supplementary mining area; Based on the area range information, determining first plane information; Based on the first plane information, second position information of the reference acquisition point is determined.

3. The method according to claim 2, characterized in that The determining, based on the first plane information, second posture information of the reference acquisition point includes: Determining second plane information based on the first plane information; Based on the second plane information, determine a reference acquisition point of the supplementary acquisition area, and obtain second coordinate information of the reference acquisition point; Based on the second coordinate information and the area range information, second position information of the reference collection point is determined.

4. The method according to claim 3, characterized in that The determining, based on the second coordinate information and the area range information, second position information of the reference collection point includes: Determine first orientation information of the reference acquisition point based on the second coordinate information and the third coordinate information, wherein the third coordinate information indicates a center point position of a target area, and the target area is a three-dimensional area indicated by the area range information; Determining a viewing angle range of the reference acquisition point based on the first orientation information and the area range information; The second position information of the reference acquisition point is determined based on the second coordinate information and the viewing angle range of the reference acquisition point.

5. The method according to claim 1, characterized in that The first posture information includes fourth coordinate information and second orientation information, and the second posture information includes second coordinate information and a viewing angle range in which data in the supplementary acquisition area can be acquired based on the reference acquisition point; determining the association relationship between the acquisition point corresponding to the image information and the reference acquisition point of the supplementary acquisition area based on the first posture information and the second posture information of the reference acquisition point of the supplementary acquisition area includes: Determine a second distance between the reference acquisition point and the acquisition point corresponding to the image information based on the fourth coordinate information and the second coordinate information; When the second distance is less than or equal to a second preset value and the second orientation information is within the viewing angle range, determining that the collection point corresponding to the image information is associated with the reference collection point; When the second distance is greater than a second preset value, or the second orientation information is not within the viewing angle range, it is determined that the acquisition point corresponding to the image information is not associated with the reference acquisition point.

6. The method according to claim 5, characterized in that The determining, based on the fourth coordinate information and the second coordinate information, a second distance between the reference acquisition point and the acquisition point corresponding to the image information comprises: When the fourth coordinate information is located within the area range indicated by the area range information of the supplementary acquisition area, a second distance between the reference acquisition point and the acquisition point corresponding to the image information is determined based on the fourth coordinate information and the second coordinate information.

7. The method according to claim 1, characterized in that The step of determining a set of supplementary images based on the association relationship and the image information includes: In a case where the association relationship indicates that the acquisition point corresponding to the image information is associated with the reference acquisition point, adding the image information to a candidate image set of the reference acquisition point; When the number of target reference acquisition points in each supplementary acquisition area of ​​the three-dimensional model reaches a third preset value, determining the supplementary acquisition image set based on the candidate image set; The target reference acquisition point is a reference acquisition point in which the number of images in the candidate image set reaches a fourth preset value.

8. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the data acquisition method according to any one of claims 1 to 7 when executed by the processor.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data acquisition method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the data collection method according to any one of claims 1 to 7.

Images