Three-dimensional modeling method, device, network device and computer-readable storage medium
By obtaining the position and perspective data of the remote device, the front-end sensors are controlled to collect necessary point cloud data, which solves the problem of non-essential point cloud data calculation in three-dimensional real-time modeling, and improves user experience and performance.
Patent Information
- Application Number
- CN201910905028.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-09-24
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2039-09-24
AI Technical Summary
In the existing three-dimensional real-time modeling technology, due to the limited field of view angle and large calculation amount, the calculation of non-essential point cloud data will affect the user experience and cause delays and action delays.
By obtaining the position data and perspective data of the remote device, obtain the required point cloud data, and control the working status of the front-end sensor, only the necessary point cloud data are collected and synthesized, and three-dimensional modeling is performed.
It reduces the modeling of non-essential spaces, reduces the delay from point cloud acquisition to remote user imaging, improves the user experience, and improves the performance of three-dimensional real-time modeling.
Smart Images

Figure CN112634427B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of communication technology, and in particular to a three-dimensional modeling method, device, network equipment and computer-readable storage medium. Background Art
[0002] In the existing mainstream 3D real-time modeling technology solutions, multiple sensor devices are often used as front-end acquisition devices to achieve human body modeling, large-scale scene modeling, etc.; for example: three data acquisition sensors are used to collect 3D point cloud data of the human body, and the point cloud is synthesized and optimized on the server and transmitted to the remote end. The remote user can see the real-time portrait at the front end through smart glasses.
[0003] However, in the current 3D real-time modeling scheme, due to the limited field of view of human binoculars or the limited field of view of the device worn by the person (the field of view of current devices such as smart glasses is still limited by technology), point cloud data that is not within the observer's field of view is unnecessary;
[0004] The process of collecting, synthesizing, and optimizing point clouds involves a large amount of computation, and the resulting delay will affect the user experience. Specifically, from point cloud collection to remote user imaging, if the delay is too large, the remote user will experience motion delays and freezes. Summary of the invention
[0005] The purpose of the present invention is to provide a three-dimensional modeling method, apparatus, network device and computer-readable storage medium to solve the problem of calculating unnecessary point cloud data in the three-dimensional real-time modeling solution in the prior art.
[0006] In order to solve the above technical problems, an embodiment of the present invention provides a three-dimensional modeling method, including:
[0007] Obtain the location data and viewing angle data of the remote device;
[0008] Acquire required point cloud data according to the position data and the viewing angle data;
[0009] Acquire sensing data of a front-end sensor according to the point cloud data;
[0010] Three-dimensional modeling is performed based on the sensor data.
[0011] Optionally, the acquiring the position data and the viewing angle data of the remote device includes:
[0012] The position change data and orientation change data of the remote device are obtained to obtain the corresponding rotation matrix R and translation vector t.
[0013] Optionally, acquiring required point cloud data according to the position data and the viewing angle data includes:
[0014] The required point cloud data is obtained according to the R, t and the field of view angle data of the remote device.
[0015] Optionally, acquiring sensing data of a front-end sensor according to the point cloud data includes:
[0016] Determine first information of the target point cloud that the front-end sensor needs to sense based on the point cloud data;
[0017] Acquire second information of the point cloud that can be sensed by each front-end sensor;
[0018] Determine third information of the required front-end sensor according to the first information and the second information;
[0019] According to the third information, sensing data of the corresponding front-end sensor is acquired.
[0020] Optionally, after determining the third information of the required front-end sensor according to the first information and the second information, the method further includes:
[0021] The front-end sensor matching the third information is controlled to be in a working state, and the front-end sensor not matching the third information is controlled to be in a non-working state.
[0022] Optionally, determining third information of the required front-end sensor according to the first information and the second information includes:
[0023] Determining fourth information of the point cloud intersecting with the target point cloud according to the first information and the second information;
[0024] Determining third information of the required front-end sensor according to the fourth information;
[0025] The acquiring sensing data of the corresponding front-end sensor according to the third information includes:
[0026] According to the third information, acquiring sensing data of the corresponding front-end sensor as initial data;
[0027] According to the fourth information, the initial data is eliminated and the remaining data obtained is used as the target data;
[0028] The three-dimensional modeling is performed according to the sensor data, including:
[0029] Three-dimensional modeling is performed according to the target data.
[0030] Optionally, performing three-dimensional modeling according to the sensor data includes:
[0031] Determine a visual field concentration area and an edge area according to the viewing angle data and the viewing angle data of the remote device;
[0032] Determine, according to the sensing data, first point cloud data corresponding to the field of view concentration area and second point cloud data corresponding to the edge area;
[0033] According to a first threshold, performing a registration operation on the first point cloud data to obtain a first registration result;
[0034] According to a second threshold, performing a registration operation on the second point cloud data to obtain a second registration result;
[0035] Perform three-dimensional modeling according to the first registration result and the second registration result;
[0036] The first threshold is smaller than the second threshold.
[0037] The embodiment of the present invention further provides a three-dimensional modeling device, comprising:
[0038] A first acquisition module, used to acquire position data and viewing angle data of a remote device;
[0039] A second acquisition module, used to acquire required point cloud data according to the position data and the viewing angle data;
[0040] A third acquisition module is used to acquire sensing data of a front-end sensor according to the point cloud data;
[0041] The first modeling module is used to perform three-dimensional modeling according to the sensor data.
[0042] Optionally, the first acquisition module includes:
[0043] The first acquisition submodule is used to acquire the position change data and the orientation change data of the remote device to obtain the corresponding rotation matrix R and translation vector t.
[0044] Optionally, the second acquisition module includes:
[0045] The second acquisition submodule is used to acquire the required point cloud data according to the R, t and the field of view angle data of the remote device.
[0046] Optionally, the third acquisition module includes:
[0047] A first determination submodule, used to determine first information of a target point cloud that a front-end sensor needs to sense based on the point cloud data;
[0048] The third acquisition submodule is used to acquire second information of the point cloud that can be sensed by each front-end sensor;
[0049] A second determining submodule, configured to determine third information of the required front-end sensor according to the first information and the second information;
[0050] The fourth acquisition submodule is used to acquire the sensing data of the corresponding front-end sensor according to the third information.
[0051] Optionally, also include:
[0052] The first control module is used to control the front-end sensor that matches the third information to be in a working state, and control the front-end sensor that does not match the third information to be in a non-working state after determining the third information of the required front-end sensor based on the first information and the second information.
[0053] Optionally, the second determining submodule includes:
[0054] A first determining unit, configured to determine fourth information of the point cloud intersecting with the target point cloud according to the first information and the second information;
[0055] A second determining unit, configured to determine third information of a required front-end sensor according to the fourth information;
[0056] The fourth acquisition submodule includes:
[0057] A first acquisition unit, configured to acquire sensing data of a corresponding front-end sensor according to the third information as initial data;
[0058] A first processing unit, configured to perform a elimination process on the initial data according to the fourth information, and obtain remaining data as target data;
[0059] The first modeling module comprises:
[0060] The first modeling submodule is used to perform three-dimensional modeling according to the target data.
[0061] Optionally, the first modeling module includes:
[0062] A third determination submodule, configured to determine a visual field concentration area and an edge area according to the viewing angle data and the viewing angle data of the remote device;
[0063] A fourth determination submodule, configured to determine, based on the sensing data, first point cloud data corresponding to the field of view concentration area and second point cloud data corresponding to the edge area;
[0064] A first processing submodule, configured to perform a registration operation on the first point cloud data according to a first threshold value to obtain a first registration result;
[0065] A second processing submodule, configured to perform a registration operation on the second point cloud data according to a second threshold value to obtain a second registration result;
[0066] A second modeling submodule, used for performing three-dimensional modeling according to the first registration result and the second registration result;
[0067] The first threshold is smaller than the second threshold.
[0068] An embodiment of the present invention further provides a network device, comprising: a processor and a transceiver;
[0069] The processor is used to obtain the position data and viewing angle data of the remote device through the transceiver;
[0070] Acquire required point cloud data according to the position data and the viewing angle data;
[0071] Acquire sensing data of a front-end sensor according to the point cloud data;
[0072] Three-dimensional modeling is performed based on the sensing data.
[0073] Optionally, the processor is specifically configured to:
[0074] The transceiver is used to obtain the position change data and orientation change data of the remote device, and obtain the corresponding rotation matrix R and translation vector t.
[0075] Optionally, the processor is specifically configured to:
[0076] The required point cloud data is obtained according to the R, t and the field of view angle data of the remote device.
[0077] Optionally, the processor is specifically configured to:
[0078] Determine first information of the target point cloud that the front-end sensor needs to sense based on the point cloud data;
[0079] Acquire second information of the point cloud that can be sensed by each front-end sensor;
[0080] Determine third information of the required front-end sensor according to the first information and the second information;
[0081] According to the third information, sensing data of the corresponding front-end sensor is acquired.
[0082] Optionally, the processor is further configured to:
[0083] After determining the third information of the required front-end sensor according to the first information and the second information, the front-end sensor matching the third information is controlled to be in a working state, and the front-end sensor not matching the third information is controlled to be in a non-working state.
[0084] Optionally, the processor is specifically configured to:
[0085] Determining fourth information of the point cloud intersecting with the target point cloud according to the first information and the second information;
[0086] Determining third information of the required front-end sensor according to the fourth information;
[0087] According to the third information, acquiring sensing data of the corresponding front-end sensor as initial data;
[0088] According to the fourth information, the initial data is eliminated and the remaining data obtained is used as the target data;
[0089] Three-dimensional modeling is performed according to the target data.
[0090] Optionally, the processor is specifically configured to:
[0091] Determine a visual field concentration area and an edge area according to the viewing angle data and the viewing angle data of the remote device;
[0092] Determine, according to the sensing data, first point cloud data corresponding to the field of view concentration area and second point cloud data corresponding to the edge area;
[0093] According to a first threshold, performing a registration operation on the first point cloud data to obtain a first registration result;
[0094] According to a second threshold, performing a registration operation on the second point cloud data to obtain a second registration result;
[0095] Perform three-dimensional modeling according to the first registration result and the second registration result;
[0096] The first threshold is smaller than the second threshold.
[0097] An embodiment of the present invention further provides a network device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor; when the processor executes the program, the above-mentioned three-dimensional modeling method is implemented.
[0098] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the above-mentioned three-dimensional modeling method are implemented.
[0099] The beneficial effects of the above technical solution of the present invention are as follows:
[0100] In the above scheme, the 3D modeling method obtains the position data and viewing angle data of the remote device; obtains the required point cloud data based on the position data and viewing angle data; obtains the sensor data of the front-end sensor based on the point cloud data; and performs 3D modeling based on the sensor data; it can realize 3D real-time modeling through remote viewing angle control, reduce the modeling of unnecessary space (i.e., the calculation of unnecessary point clouds), so as to effectively improve the performance of 3D real-time modeling, and reduce the delay from point cloud acquisition to remote user imaging, improve user experience, and well solve the problem of calculation of unnecessary point cloud data in the 3D real-time modeling scheme in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS
[0101] Figure 1 A schematic diagram of a three-dimensional modeling method flow chart of an embodiment of the present invention;
[0102] Figure 2 A schematic diagram of data interaction according to an embodiment of the present invention;
[0103] Figure 3 A schematic diagram of a system for implementing a three-dimensional modeling method according to an embodiment of the present invention;
[0104] Figure 4 Schematic diagram of the field of view of an embodiment of the present invention;
[0105] Figure 5 A schematic diagram of a point cloud required for an embodiment of the present invention;
[0106] Figure 6 This is a schematic diagram of point cloud coverage of a front-end sensor according to an embodiment of the present invention;
[0107] Figure 7 A schematic diagram of the structure of a three-dimensional modeling device according to an embodiment of the present invention;
[0108] Figure 8 The figure is a schematic diagram of the network device structure according to an embodiment of the present invention. DETAILED DESCRIPTION
[0109] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.
[0110] The present invention aims to solve the problem of unnecessary point cloud data calculation in the existing 3D real-time modeling scheme, and provides a 3D modeling method which can be applied to network devices such as Figure 1 As shown, including:
[0111] Step 11: Obtain the location data and viewing angle data of the remote device;
[0112] Step 12: Obtain required point cloud data according to the position data and viewing angle data;
[0113] Step 13: Acquire sensing data of the front-end sensor according to the point cloud data;
[0114] Step 14: Perform three-dimensional modeling based on the sensor data.
[0115] The 3D modeling method provided in the embodiment of the present invention obtains the position data and viewing angle data of the remote device; obtains the required point cloud data based on the position data and viewing angle data; obtains the sensor data of the front-end sensor based on the point cloud data; and performs 3D modeling based on the sensor data. It can realize 3D real-time modeling through remote viewing angle control, reduce the modeling of unnecessary space (i.e., the calculation of unnecessary point clouds), so as to effectively improve the performance of 3D real-time modeling, and reduce the delay from point cloud acquisition to remote user imaging, improve user experience, and well solve the problem of calculation of unnecessary point cloud data in the 3D real-time modeling scheme in the prior art.
[0116] The obtaining of the position data and the viewing angle data of the remote device includes: obtaining the position change data and the orientation change data of the remote device, and obtaining the corresponding rotation matrix R and the translation vector t.
[0117] Specifically, acquiring the required point cloud data according to the position data and the viewing angle data includes: acquiring the required point cloud data according to R, t and the viewing angle data of the remote device.
[0118] In an embodiment of the present invention, acquiring the sensor data of the front-end sensor based on the point cloud data includes: determining first information of the target point cloud that the front-end sensor needs to perceive based on the point cloud data; acquiring second information of the point cloud that each front-end sensor can perceive; determining third information of the required front-end sensor based on the first information and the second information; and acquiring the sensor data of the corresponding front-end sensor based on the third information.
[0119] Furthermore, after determining the third information of the required front-end sensor according to the first information and the second information, it also includes: controlling the front-end sensor matching the third information to be in a working state, and controlling the front-end sensor not matching the third information to be in a non-working state.
[0120] This can avoid unnecessary waste of resources and save resources.
[0121] Wherein, determining the third information of the required front-end sensor according to the first information and the second information includes: determining the fourth information of the point cloud intersecting with the target point cloud according to the first information and the second information; determining the third information of the required front-end sensor according to the fourth information;
[0122] Correspondingly, the acquiring of the sensing data of the corresponding front-end sensor according to the third information includes: acquiring the sensing data of the corresponding front-end sensor according to the third information as initial data; and performing elimination processing on the initial data according to the fourth information, and obtaining the remaining data as target data;
[0123] The performing three-dimensional modeling according to the sensing data includes: performing three-dimensional modeling according to the target data.
[0124] Specifically, the three-dimensional modeling based on the sensor data includes: determining the field of view concentration area and the edge area based on the viewing angle data and the field of view angle data of the remote device; determining the first point cloud data corresponding to the field of view concentration area and the second point cloud data corresponding to the edge area based on the sensor data; performing a registration operation on the first point cloud data according to a first threshold to obtain a first registration result; performing a registration operation on the second point cloud data according to a second threshold to obtain a second registration result; performing three-dimensional modeling based on the first registration result and the second registration result; wherein the first threshold is less than the second threshold.
[0125] This makes it possible to accurately calculate the point cloud data of the focused area of the field of view and roughly calculate the point cloud data of the edge area of the field of view, so as to minimize the amount of point cloud synthesis calculation while ensuring that the image seen by the user is clear.
[0126] The following is a further explanation of the three-dimensional modeling method provided in the embodiment of the present invention by examples, wherein a server is taken as an example of a network device and a remote user device is taken as an example of a remote device.
[0127] In response to the above technical problems, and taking into account the large amount of calculation involved in the process of point cloud collection, synthesis, and optimization, the delay caused by which will affect the user experience, and reducing the calculation of unnecessary point cloud data can reduce the delay; an embodiment of the present invention provides a three-dimensional modeling method, which reduces the modeling of unnecessary spaces (i.e., the calculation of unnecessary point clouds) through perspective control, and further reduces the level of refinement of modeling in the edge areas of the field of view at the same time, so as to reduce the calculation delay and effectively improve the performance of three-dimensional real-time modeling.
[0128] The solution provided by the embodiment of the present invention can be specifically implemented as follows: real-time tracking of the position and viewing angle of the remote user (i.e., the remote user device), and real-time feedback of the position and viewing angle to the server; the server calculates which point cloud data of the front-end sensors are needed at the current position and viewing angle, controls the working and waiting states of the front-end sensors, synthesizes and optimizes the necessary point cloud data as needed, and forms a three-dimensional model; and then transmits the model to the remote user. For data interaction examples, see Figure 2 , Figure 2 The a in here represents the front-end sensor (also called data acquisition sensor).
[0129] The implementation of the solution provided in the embodiment of the present invention can be achieved by Figure 3 The implementation system shown in the figure can be divided into three parts: remote user equipment, server, and front-end sensor:
[0130] The remote user equipment includes:
[0131] Visual presentation module: can be used to present three-dimensional images in the user's field of view.
[0132] Positioning attitude module: It can be used to track the location and viewing direction of the user (remote user device). It can be achieved through a variety of methods such as motion inertial sensor (IMU) or visual odometry (VO) algorithm, or a combination of the two to form visual inertial odometry (VIO).
[0133] The communication module can be used to communicate with the server.
[0134] Servers, including:
[0135] The model generation module can be used to stitch, synthesize, and optimize the three-dimensional point cloud collected by the front-end sensor.
[0136] View control module: can be used to receive the position and view direction perceived by the remote user device in real time, determine the spatial range of the necessary point cloud data (i.e. the view range of the remote user device), and determine the sensor that must be used currently based on this spatial range.
[0137] The communication module can be used to communicate with the communication module of the front-end sensor and the communication module of the remote user equipment.
[0138] Front-end sensors, including:
[0139] Point cloud acquisition module can be used to collect point cloud information.
[0140] Equipment control module: can be used to receive the sensor information that must be used from the viewing angle control module, control the sensors that must be used to be in the data collection state, and set other sensors to other states (such as non-working state, specifically, suspended working state).
[0141] The communication module can be used to communicate with the server.
[0142] The following is a specific example of the implementation process of the three-dimensional modeling method provided in the embodiment of the present invention (the modeling object is the human body as an example).
[0143] a) System initialization
[0144] Fix the front-end sensor in space and complete the calibration (implemented on the server: unify the coordinate system and determine the relative positions); define the world coordinate system where the front-end sensor is located and the world coordinate system where the remote user device is located, and define the initial position of the image presented in the remote user's field of view.
[0145] For specific calibration and definition methods, please refer to the existing scheme, which will not be repeated here.
[0146] b) Real-time viewing angle tracking of remote users
[0147] Use the motion inertial sensor (IMU) or visual odometry (VO) algorithm, or a combination of the two to form a visual inertial odometry (VIO) to track the orientation and position changes of the remote user device in real time, represented by the rotation matrix R and the translation vector t. Each position in the trajectory of the remote user device in space corresponds to an R and t. The remote user device transmits R and t to the server in real time.
[0148] c) The server determines the point cloud required for the current remote user's perspective
[0149] The field of view of a remote user device (such as smart glasses) is represented by FOV. Figure 4 As shown, the field of view extends in the space, dividing the space into a space within the field of view and a space outside the field of view.
[0150] The server receives the real-time feedback of R and t from the remote user device, and calculates the current required point cloud P according to the user's position, orientation and field of view. remote , P remote Defined in the world coordinate system where the front-end sensor is located, assuming that the world coordinate system where the front-end sensor is located is consistent with the world coordinate system where the remote user device is located, that is:
[0151] P remote =f(R,t,fov);
[0152] Where f is a function with R, t, and fov as independent variables. Figure 5 As shown in the figure, the surface point cloud represented by the dotted line on the observed human body (including the side of the head) is the P currently sought. remote .
[0153] d) The server determines which front-end sensor data is currently needed to generate Premote
[0154] The point cloud perceived by the front-end sensor is denoted as P sensor ,make To meet the current point cloud requirements, P sensor According to the following rules, if the point cloud P collected by the sensor sensor With P remote If there is an intersection, the point cloud data of the intersection part is retained and the rest is discarded; if the point cloud P collected by the sensor sensor Completely with P remote If there is no intersection, all point clouds of the sensor are discarded; that is, the sensor set S currently used is:
[0155]
[0156] like Figure 6 As shown, the point clouds collected by sensors A, B, and D are all or partly in the point cloud P in the previous step. remote Therefore, the sensors to be used at the current moment are sensors A, B, and D.
[0157] e) Control the front-end sensor to turn off or on
[0158] According to the sensor set determined in the previous step, control signals are transmitted to the corresponding sensors to control their on and off. This is meaningful for using a large number of sensors to achieve real-time 3D construction of large scenes, so that idle sensors are in a non-working state to reduce consumption.
[0159] f) Point cloud synthesis calculation
[0160] Now process the point clouds of sensors A, B, and D. As described in step d), keep all the point clouds of sensor A, keep the point clouds of sensors B and D in P remote In the point cloud, remove the point that does not belong to P remote Point cloud.
[0161] In addition, considering the characteristics of human vision: the central focus area is clear and the edge area of the field of vision is blurred; in real-time 3D modeling, the modeling of the central area of the user's field of vision can be made more refined, while the modeling standards for the edge areas of the field of vision can be appropriately lowered to reduce the amount of calculation.
[0162] Therefore, in the embodiment of the present invention, the point clouds of sensors A, B, and D are registered. According to the user's real-time perspective, the point cloud data of the focus area of the field of view can be accurately calculated, and the point cloud data of the edge area of the field of view can be roughly calculated. There are many methods for point cloud registration, and here we only take the ICP (Iterative Closest Point) algorithm as an example to introduce:
[0163] The ICP algorithm can be understood as a continuous loop calculation, finding the nearest corresponding point between every two point clouds, and calculating an RT matrix (rotation and translation matrix) that minimizes the distance between the corresponding points after the transformation from these corresponding points, and transforming according to this RT matrix, repeating the above steps until some values reach the set threshold and then stopping the iteration. Therefore, for the field of view focus area that needs to be accurately calculated, a smaller threshold (such as the first threshold mentioned above) is selected when using the ICP algorithm to make the point cloud registration result more accurate; conversely, for the edge area that only needs to be roughly calculated, a larger threshold (such as the second threshold mentioned above) is selected, and the registration is ended only by roughly registering the point cloud.
[0164] Regarding the demarcation of the field of view concentration area and the edge area, the field of view center point can be determined according to the current viewing angle, and the range can be determined in combination with the field of view fov of the remote user device. The specific method is not limited.
[0165] It should be noted that the application scope of the solution provided by the embodiment of the present invention is not limited to three-dimensional real-time human body modeling, but also includes real-time object modeling, large-scale scene space modeling, etc.
[0166] As can be seen from the above, the solution provided by the embodiment of the present invention involves:
[0167] (1) Determine the point cloud range required by the remote user through field of view control;
[0168] (2) Determine which sensors to use to collect data based on the required point cloud and sensor parameters;
[0169] (3) Crop the point cloud collected by the working sensor so that unnecessary point clouds do not participate in the synthesis of the final image;
[0170] (4) Accurately calculate the point cloud data of the focal area of the field of view, and roughly calculate the point cloud data of the edge area of the field of view, so as to minimize the amount of point cloud synthesis calculation while ensuring that the image seen by the user is clear.
[0171] The embodiment of the present invention also provides a three-dimensional modeling device, such as Figure 7 As shown, including:
[0172] A first acquisition module 71, used to acquire position data and viewing angle data of a remote device;
[0173] A second acquisition module 72, used to acquire required point cloud data according to the position data and the viewing angle data;
[0174] A third acquisition module 73 is used to acquire sensing data of a front-end sensor according to the point cloud data;
[0175] The first modeling module 74 is used to perform three-dimensional modeling according to the sensing data.
[0176] The 3D modeling device provided in the embodiment of the present invention obtains the position data and viewing angle data of the remote device; obtains the required point cloud data based on the position data and viewing angle data; obtains the sensor data of the front-end sensor based on the point cloud data; and performs 3D modeling based on the sensor data. It is capable of controlling 3D real-time modeling through the remote viewing angle, reducing the modeling of unnecessary spaces (i.e., the calculation of unnecessary point clouds), so as to effectively improve the performance of 3D real-time modeling, and reduce the delay from point cloud acquisition to remote user imaging, thereby improving user experience, and well solving the problem of calculation of unnecessary point cloud data in the 3D real-time modeling scheme in the prior art.
[0177] The first acquisition module includes: a first acquisition submodule, which is used to acquire the position change data and orientation change data of the remote device to obtain the corresponding rotation matrix R and translation vector t.
[0178] Specifically, the second acquisition module includes: a second acquisition submodule, used to acquire the required point cloud data according to R, t and the field of view angle data of the remote device.
[0179] In an embodiment of the present invention, the third acquisition module includes: a first determination submodule, used to determine the first information of the target point cloud that the front-end sensor needs to perceive based on the point cloud data; a third acquisition submodule, used to acquire the second information of the point cloud that each front-end sensor can perceive; a second determination submodule, used to determine the third information of the required front-end sensor based on the first information and the second information; and a fourth acquisition submodule, used to acquire the sensor data of the corresponding front-end sensor based on the third information.
[0180] Furthermore, the three-dimensional modeling device also includes: a first control module, which is used to control the front-end sensor that matches the third information to be in a working state, and control the front-end sensor that does not match the third information to be in a non-working state after determining the third information of the required front-end sensor based on the first information and the second information.
[0181] The second determination submodule includes: a first determination unit, configured to determine fourth information of the point cloud intersecting with the target point cloud according to the first information and the second information; and a second determination unit, configured to determine third information of the required front-end sensor according to the fourth information;
[0182] Correspondingly, the fourth acquisition submodule includes: a first acquisition unit, used to acquire the sensor data of the corresponding front-end sensor according to the third information as initial data; a first processing unit, used to eliminate the initial data according to the fourth information, and obtain the remaining data as target data;
[0183] The first modeling module includes: a first modeling submodule, which is used to perform three-dimensional modeling according to the target data.
[0184] Specifically, the first modeling module includes: a third determination submodule, used to determine the field of view focusing area and the edge area according to the viewing angle data and the field of view angle data of the remote device; a fourth determination submodule, used to determine the first point cloud data corresponding to the field of view focusing area and the second point cloud data corresponding to the edge area according to the sensor data; a first processing submodule, used to perform a registration operation on the first point cloud data according to a first threshold to obtain a first registration result; a second processing submodule, used to perform a registration operation on the second point cloud data according to a second threshold to obtain a second registration result; a second modeling submodule, used to perform three-dimensional modeling based on the first registration result and the second registration result; wherein the first threshold is less than the second threshold.
[0185] Among them, the implementation embodiments of the above-mentioned three-dimensional modeling method are all applicable to the embodiments of the three-dimensional modeling device and can achieve the same technical effects.
[0186] The embodiment of the present invention also provides a network device, such as Figure 8 As shown, it includes: a processor 81 and a transceiver 82;
[0187] The processor 81 is used to obtain the position data and viewing angle data of the remote device through the transceiver 82;
[0188] Acquire required point cloud data according to the position data and the viewing angle data;
[0189] Acquire sensing data of a front-end sensor according to the point cloud data;
[0190] Three-dimensional modeling is performed based on the sensing data.
[0191] The network device provided in the embodiment of the present invention obtains the position data and viewing angle data of the remote device; obtains the required point cloud data based on the position data and viewing angle data; obtains the sensor data of the front-end sensor based on the point cloud data; and performs three-dimensional modeling based on the sensor data. It can realize three-dimensional real-time modeling through remote viewing angle control, reduce the modeling of unnecessary space (i.e., the calculation of unnecessary point clouds), so as to effectively improve the performance of three-dimensional real-time modeling, and reduce the delay from point cloud acquisition to remote user imaging, improve user experience, and well solve the problem of calculation of unnecessary point cloud data in the three-dimensional real-time modeling solution in the prior art.
[0192] The processor is specifically used to: obtain the position change data and orientation change data of the remote device through the transceiver to obtain the corresponding rotation matrix R and translation vector t.
[0193] Specifically, the processor is used to obtain the required point cloud data according to R, t and the field of view angle data of the remote device.
[0194] In an embodiment of the present invention, the processor is specifically used to: determine first information of the target point cloud that the front-end sensor needs to perceive based on the point cloud data; obtain second information of the point cloud that each front-end sensor can perceive; determine third information of the required front-end sensor based on the first information and the second information; and obtain sensor data of the corresponding front-end sensor based on the third information.
[0195] Furthermore, the processor is also used to: after determining the third information of the required front-end sensor based on the first information and the second information, control the front-end sensor that matches the third information to be in a working state, and control the front-end sensor that does not match the third information to be in a non-working state.
[0196] Among them, the processor is specifically used to: determine the fourth information of the point cloud that intersects with the target point cloud based on the first information and the second information; determine the third information of the required front-end sensor based on the fourth information; obtain the sensor data of the corresponding front-end sensor based on the third information as initial data; according to the fourth information, eliminate the initial data and obtain the remaining data as target data; and perform three-dimensional modeling based on the target data.
[0197] Specifically, the processor is used to: determine the field of view focusing area and the edge area according to the viewing angle data and the field of view angle data of the remote device; determine the first point cloud data corresponding to the field of view focusing area and the second point cloud data corresponding to the edge area according to the sensor data; perform a registration operation on the first point cloud data according to a first threshold to obtain a first registration result; perform a registration operation on the second point cloud data according to a second threshold to obtain a second registration result; perform three-dimensional modeling according to the first registration result and the second registration result; wherein the first threshold is less than the second threshold.
[0198] Among them, the implementation embodiments of the above-mentioned three-dimensional modeling method are all applicable to the embodiments of the network device and can achieve the same technical effects.
[0199] An embodiment of the present invention further provides a network device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor; when the processor executes the program, the above-mentioned three-dimensional modeling method is implemented.
[0200] Among them, the implementation embodiments of the above-mentioned three-dimensional modeling method are all applicable to the embodiments of the network device and can achieve the same technical effects.
[0201] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the above-mentioned three-dimensional modeling method are implemented.
[0202] Among them, the implementation embodiments of the above-mentioned three-dimensional modeling method are all applicable to the embodiments of the computer-readable storage medium and can achieve the same technical effects.
[0203] It should be noted that many functional components described in this specification are referred to as modules / sub-modules / units in order to more particularly emphasize the independence of their implementation methods.
[0204] In the embodiment of the present invention, module / submodule / unit can be implemented with software so that it can be executed by various types of processors. For example, an identified executable code module can include one or more physical or logical blocks of computer instructions, for example, it can be constructed as an object, process or function. Nevertheless, the executable code of the identified module does not need to be physically located together, but can include different instructions stored in different bits, which constitute the module and realize the specified purpose of the module when these instructions are logically combined together.
[0205] In fact, executable code module can be a single instruction or many instructions, and can even be distributed on a plurality of different code segments, distributed among different programs, and distributed across a plurality of memory devices. Similarly, operating data can be identified in the module, and can be implemented and organized in the data structure of any appropriate type according to any appropriate form. The operating data can be collected as a single data set, or can be distributed in different locations (including on different storage devices), and can only be present on a system or network as an electronic signal at least in part.
[0206] When a module can be implemented by software, considering the level of existing hardware technology, a person skilled in the art can build a corresponding hardware circuit to implement the corresponding function of the module that can be implemented by software without considering the cost. The hardware circuit includes a conventional very large scale integration (VLSI) circuit or gate array and existing semiconductors such as logic chips, transistors, or other discrete components. The module can also be implemented by a programmable hardware device, such as a field programmable gate array, a programmable array logic, a programmable logic device, etc.
[0207] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary personnel in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A three-dimensional modeling method, characterized in that: include: Obtain the location data and viewing angle data of the remote device; Acquire required point cloud data according to the position data and the viewing angle data; Acquire sensing data of a front-end sensor according to the point cloud data; Performing three-dimensional modeling based on the sensor data; Wherein, acquiring the sensing data of the front-end sensor according to the point cloud data includes: Determine first information of the target point cloud that the front-end sensor needs to sense based on the point cloud data; Acquire second information of the point cloud that can be sensed by each front-end sensor; Determine third information of the required front-end sensor according to the first information and the second information; According to the third information, acquiring sensing data of the corresponding front-end sensor; After determining the third information of the required front-end sensor according to the first information and the second information, the method further includes: Controlling the front-end sensor that matches the third information to be in a working state, and controlling the front-end sensor that does not match the third information to be in a non-working state; Wherein, determining the third information of the required front-end sensor according to the first information and the second information includes: Determining fourth information of the point cloud intersecting with the target point cloud according to the first information and the second information; Determining third information of the required front-end sensor according to the fourth information; The acquiring sensing data of the corresponding front-end sensor according to the third information includes: According to the third information, acquiring sensing data of the corresponding front-end sensor as initial data; According to the fourth information, the initial data is eliminated and the remaining data obtained is used as the target data; The three-dimensional modeling is performed according to the sensor data, including: Three-dimensional modeling is performed according to the target data.
2. The three-dimensional modeling method according to claim 1, characterized in that: The obtaining of the position data and the viewing angle data of the remote device includes: The position change data and orientation change data of the remote device are obtained to obtain the corresponding rotation matrix R and translation vector t.
3. The three-dimensional modeling method according to claim 2, characterized in that: The step of obtaining the required point cloud data according to the position data and the viewing angle data includes: The required point cloud data is obtained according to the R, t and the field of view angle data of the remote device.
4. The three-dimensional modeling method according to claim 1 or 2, characterized in that: The three-dimensional modeling is performed according to the sensor data, including: Determine a visual field concentration area and an edge area according to the viewing angle data and the viewing angle data of the remote device; Determine, according to the sensing data, first point cloud data corresponding to the field of view concentration area and second point cloud data corresponding to the edge area; According to a first threshold, performing a registration operation on the first point cloud data to obtain a first registration result; According to a second threshold, performing a registration operation on the second point cloud data to obtain a second registration result; Perform three-dimensional modeling according to the first registration result and the second registration result; The first threshold is smaller than the second threshold.
5. A three-dimensional modeling device, characterized in that: include: A first acquisition module, used to acquire position data and viewing angle data of a remote device; A second acquisition module, used to acquire required point cloud data according to the position data and the viewing angle data; A third acquisition module is used to acquire sensing data of a front-end sensor according to the point cloud data; A first modeling module, used for performing three-dimensional modeling according to the sensor data; Wherein, the third acquisition module includes: A first determination submodule, used to determine first information of a target point cloud that a front-end sensor needs to sense based on the point cloud data; The third acquisition submodule is used to acquire second information of the point cloud that can be sensed by each front-end sensor; A second determining submodule, configured to determine third information of the required front-end sensor according to the first information and the second information; A fourth acquisition submodule, used to acquire sensing data of the corresponding front-end sensor according to the third information; Wherein, the three-dimensional modeling device further comprises: A first control module is used to control the front-end sensor matching the third information to be in a working state, and control the front-end sensor not matching the third information to be in a non-working state after determining the third information of the required front-end sensor according to the first information and the second information; Wherein, the second determining submodule includes: A first determining unit, configured to determine fourth information of the point cloud intersecting with the target point cloud according to the first information and the second information; A second determining unit, configured to determine third information of a required front-end sensor according to the fourth information; The fourth acquisition submodule includes: A first acquisition unit, configured to acquire sensing data of a corresponding front-end sensor according to the third information as initial data; A first processing unit, configured to perform a elimination process on the initial data according to the fourth information, and obtain remaining data as target data; The first modeling module comprises: The first modeling submodule is used to perform three-dimensional modeling according to the target data.
6. The three-dimensional modeling device according to claim 5, characterized in that: The first acquisition module includes: The first acquisition submodule is used to acquire the position change data and the orientation change data of the remote device to obtain the corresponding rotation matrix R and translation vector t.
7. The three-dimensional modeling device according to claim 6, characterized in that: The second acquisition module includes: The second acquisition submodule is used to acquire the required point cloud data according to the R, t and the field of view angle data of the remote device.
8. The three-dimensional modeling device according to claim 5 or 6, characterized in that: The first modeling module comprises: A third determination submodule, configured to determine a visual field concentration area and an edge area according to the viewing angle data and the viewing angle data of the remote device; A fourth determination submodule, configured to determine, based on the sensing data, first point cloud data corresponding to the field of view concentration area and second point cloud data corresponding to the edge area; A first processing submodule, configured to perform a registration operation on the first point cloud data according to a first threshold value to obtain a first registration result; A second processing submodule, configured to perform a registration operation on the second point cloud data according to a second threshold value to obtain a second registration result; A second modeling submodule, used for performing three-dimensional modeling according to the first registration result and the second registration result; The first threshold is smaller than the second threshold.
9. A network device, characterized in that: include: processor and transceiver; The processor is used to obtain the position data and viewing angle data of the remote device through the transceiver; Acquire required point cloud data according to the position data and the viewing angle data; Acquire sensing data of a front-end sensor according to the point cloud data; Performing three-dimensional modeling based on the sensor data; The processor is specifically used to: determine first information of the target point cloud that the front-end sensor needs to sense according to the point cloud data; obtain second information of the point cloud that each front-end sensor can sense; determine third information of the required front-end sensor according to the first information and the second information; and obtain sensor data of the corresponding front-end sensor according to the third information; The processor is further configured to: after determining the third information of the required front-end sensor according to the first information and the second information, control the front-end sensor matching the third information to be in a working state, and control the front-end sensor not matching the third information to be in a non-working state; Among them, the processor is specifically used to: determine the fourth information of the point cloud that intersects with the target point cloud based on the first information and the second information; determine the third information of the required front-end sensor based on the fourth information; obtain the sensor data of the corresponding front-end sensor based on the third information as initial data; according to the fourth information, eliminate the initial data and obtain the remaining data as target data; and perform three-dimensional modeling based on the target data.
10. A network device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor; characterized in that: When the processor executes the program, the three-dimensional modeling method according to any one of claims 1 to 4 is implemented.
11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the three-dimensional modeling method according to any one of claims 1 to 4 are implemented.
