Realistic trajectory simulation and view capture content recommendation method for digital twin scenes

Through real three-dimensional real-life model and infrared motion capture system, the virtual experimental site of the motion robot is built, and the environmental images obtained by the camera are simulated, which solves the complexity of the installation position and angle optimization of the intelligent motion robot camera, and realizes more efficient experimental processes and cost savings .

CN115128965BActive Publication Date: 2024-11-22ZHEJIANG UNIV OF TECH
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202210144333.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-17
Publication Date
2024-11-22
Estimated Expiration
2042-02-17

AI Technical Summary

Technical Problem

When existing intelligent sports robots move in a large area, the optimization of camera installation position and angle requires repeated testing, resulting in many experiments, high difficulty and high cost, affecting the judgment results of machine vision.

Method used

Through real three-dimensional real-life model reconstruction and infrared three-dimensional motion capture system, a three-dimensional digital experimental site for motion robots in a virtual environment is built, environmental images obtained by cameras at different positions and angles are simulated, and machine vision evaluation is carried out, and the optimal installation location is recommended and angle.

Benefits of technology

This greatly reduces the complexity and cost of experiments, improves experimental efficiency, and quickly evaluates the camera installation position and angle through virtual simulation, reducing the number of actual installation and testing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115128965B_ABST
    Figure CN115128965B_ABST
Patent Text Reader

Abstract

A method for recommending content for realistic trajectory simulation and perspective capture of digital twin scenes, including: collecting image information of a real experimental site based on drone tilt photography technology to build a digital experimental environment; building an infrared three-dimensional motion capture system to record the motion data of a motion robot; building a digital model of a motion robot in a three-dimensional digital experimental environment to restore the motion trajectory and motion posture of the motion robot; collecting environmental images obtained by a virtual camera through virtual simulation, and performing certain machine vision evaluation on the environmental images, and preliminarily recommending more suitable content for machine vision analysis. The present invention collects the actual motion trajectory and motion posture of a real moving object, realizes its simulated motion in a three-dimensional digital experimental environment, reduces the complexity of the experiment, and also reduces the equipment cost and time cost of the experiment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of virtual simulation technology in three-dimensional scenes, involving technologies such as real-scene modeling, motion capture, and physical simulation, and specifically relates to a method for evaluating and recommending the camera position of a motion robot. Background Art

[0002] Existing intelligent robots rely on various sensors to acquire information about their surroundings, which serves as the basis for their movements. When moving within a large space, these robots use environmental images as input. These images are processed to extract useful environmental information, such as pathways, obstacles, and distances to objects. This information is then provided to the robot, which then controls its movements based on this environmental information. This process of replacing the human eye with measurement and judgment is known as machine vision. Environmental image information is captured by cameras. The camera's mounting position and angle on the robot's body determine the image quality, which in turn influences the machine vision's judgment. The precise location and angle of the camera on the robot's body ensure that the captured image is optimal for machine vision analysis and judgment, making it a crucial component of machine vision.

[0003] The development and testing of motion robots requires installing multiple cameras in different locations. The robot's motion is repeatedly tested throughout the robot's movements, and the images captured by different cameras are analyzed to determine which images are most consistent with machine vision analysis. However, the exact installation location and angle require repeated experimentation, so experiments often require repeated comparisons of environmental images to ensure optimal installation positions and angles. This significantly increases the number of experiments, difficulty, and cost.

[0004] Therefore, if we can improve the experimental process through virtual simulation, and easily simulate the environmental images captured by cameras installed at different positions on the robot's body, we can significantly reduce the number of experiments, reduce experimental difficulty, and save experimental costs. The captured environmental images are used for machine vision evaluation. Since the position and angle of the video capture will affect the video capture results, the capture results will affect the machine vision calculation results. Based on the evaluation results, we can judge the quality of the virtual camera installation positions and make preliminary recommendations for the most suitable content.

[0005] Among them, the technical background involved in the present invention includes a true three-dimensional real scene model reconstruction system and an infrared three-dimensional motion capture system.

[0006] A true 3D real-scene model reconstruction system is a computer system and its supporting equipment used in various fields such as earth science and water conservancy engineering. True 3D real-scene model reconstruction is achieved by capturing real scenes and performing reverse modeling to restore real 3D information such as buildings or landforms. Advantages of true 3D real-scene model reconstruction include high speed, low cost, high precision, and real-scene restoration. A common reconstruction process utilizes drone oblique photography technology. By tilting the gimbal and camera during flight, aerial images are simultaneously captured from multiple angles, including vertical and oblique angles, creating high-precision photographs. This captures complete and accurate information about the appearance of ground objects, as well as the camera's height and geographic coordinates at the time of image acquisition. 3D modeling software developed based on the principles of oblique photography is then used, using the continuous aerial images captured by the drone and a graphics processing unit, to quickly, accurately, and automatically generate realistic 3D real-scene models.

[0007] Motion capture involves placing trackers (markers) at key locations on a moving object. A motion capture system captures the tracker's position and then processes it with a computer to generate three-dimensional spatial coordinate data. This technology involves generating data that can be directly understood and processed by computers, such as dimensional measurement, positioning, and orientation of objects in physical space. Once computer-recognized, this data can be applied in animation, gait analysis, biomechanics, and ergonomics. An infrared three-dimensional motion capture system consists of an infrared motion capture lens, motion capture software, reflective markers, a Power over Ethernet (PoE) switch, and several accessories (such as a calibration frame and lens fixture). Essentially, it uses infrared light to obtain the three-dimensional coordinates of the tracker for positioning. Through computational analysis, various kinematic data related to the tracker, such as velocity and acceleration, are obtained. Using infrared light as a basis, multiple lenses observe and locate specific markers to determine their specific spatial positions. The LED on the optical motion capture lens panel emits infrared light of a specific wavelength, which illuminates the object being captured (the reflective marker). The reflective material on the marker's surface reflects the infrared light back to the lens. This reflected infrared light undergoes signal processing, image capture, and algorithm processing to determine the two-dimensional coordinates of the reflective marker in the lens. A motion capture system typically includes multiple motion capture lenses. Calibration is used to determine the specific position of each lens, and computer software is used to analyze the three-dimensional coordinates of each tracking point and its motion history.

[0008] The true three-dimensional real scene model reconstruction system and the infrared three-dimensional motion capture system are both relatively mature application systems at present, and are also the underlying supporting technologies that the present invention relies on. Summary of the Invention

[0009] The present invention aims to overcome the above-mentioned shortcomings of the existing technology and provides a method for simulating realistic trajectory and capturing perspectives for content recommendation in digital twin scenarios. This method simulates the environmental imagery of a moving robot through virtual simulation, as well as the robot's motion process. It then derives the environmental images that can be captured by virtual cameras at specific angles and at different positions on the robot's body. The method then performs a certain machine vision evaluation on the environmental images, preliminarily recommending more appropriate content and ensuring the optimal camera installation position and angle.

[0010] The concept of the present invention is as follows: the first step is to construct a three-dimensional real-scene model of the real experimental site of the motion robot through a true three-dimensional real-scene model reconstruction system. For example, the spatial dimensions of the real experimental site of the motion robot are set to X(m)*Y(m)*Z(m), where X represents length, Y represents width, and Z represents height. First, the image information of the X(m)*Y(m)*Z(m) real site is collected using drone oblique photography technology. Based on this, a 1:1 three-dimensional real-scene model of X(m)*Y(m)*Z(m) is generated in the oblique photography modeling software in a virtual environment. The three-dimensional real-scene model can completely replicate the three-dimensional structure and appearance of the real experimental site. The three-dimensional visualization platform is a platform for three-dimensional visualization applications that provides browsing of real-scene three-dimensional models. The three-dimensional real-scene model is imported into the three-dimensional visualization platform, and the three-dimensional visualization platform is used to restore the visual environment of the real scene, so as to build a three-dimensional digital experimental environment for the next step.

[0011] The second step is to use an infrared motion capture system, built in a real-world experimental field, to capture the robot's motion data within the experimental field. The system then recreates the robot's trajectory and posture within a virtual 3D reality model. Specifically, specific markers are attached to the robot's body. The infrared motion capture system uses multiple cameras to observe and locate these markers, acquiring their specific spatial locations. This large amount of coordinate data is then transmitted to the motion capture software via a POE connector, allowing it to calculate information such as the target's position, orientation, and trajectory in space. The infrared motion capture system is installed in the real-world experimental field, with specific markers attached to the robot's body. This allows the system to capture the robot's trajectory and posture within the X(m)*Y(m)*Z(m) real-world experimental field. The infrared motion capture system then captures the robot's trajectory and posture within the X(m)*Y(m)*Z(m) real-world field. In the three-dimensional visualization platform, a 1:1 virtual model of the motion robot consistent with the motion robot is also constructed, and the acquired marker point positioning data is subjected to certain calculations and transformations, including smoothing of the motion trajectory path, transformation of three-dimensional space coordinate data and longitude and latitude, calculation of turning angular velocity, etc., so as to simulate the digital motion trajectory and motion posture of the virtual model of the motion robot in the three-dimensional visualization platform, and virtually simulate the motion process of the real motion robot in the real experimental site.

[0012] The third step involves constructing a virtual camera at a fixed position and angle on the body of the virtual robot model. Because the first and second steps have already constructed a true 3D real-world model of the robot's experimental environment, as well as the digitized motion trajectory and posture of the virtual robot model within the true 3D real-world model, these two technical foundations can be used to reconstruct the image information of the true 3D real-world model's visual environment captured by the virtual camera as it follows the movement of the virtual robot model, thereby simulating the real-world environment image it might capture. Specifically, a virtual camera at a fixed position and angle is attached to the virtual robot, and the offset of the virtual camera's perspective caused by changes in the motion trajectory and posture is calculated in real time. This allows the virtual camera's perspective data to be updated in real time, simulating the environment image captured by the virtual camera at that position and angle. This method allows for the acquisition of simulated environmental images from any simulated perspective of the robot's body for machine learning and analysis, significantly improving experimental efficiency and reducing costs.

[0013] Step 4: The location and angle of the ambient imagery will affect the video capture results, which in turn will affect the machine vision calculations. Based on the evaluation results, the quality of the virtual camera placement is determined, and the optimal virtual camera shooting position is recommended as the actual machine installation location, along with preliminary recommendations for suitable content.

[0014] The present invention simulates a three-dimensional real-scene model of the real-scene environment in a digital way, providing a more convenient and flexible digital experimental environment for machine vision analysis of motion robots, and better judging the installation position of the robot camera in a lower-cost and more convenient experimental way.

[0015] The present invention's method for recommending content based on realistic trajectory simulation and perspective capture for digital twin scenarios is characterized by utilizing digital technology to digitize a motion robot, a real experimental site, and a motion trajectory, constructing a digital experimental environment, collecting environmental images captured by a virtual camera through virtual simulation, and performing certain machine vision evaluations on the environmental images to preliminarily recommend more appropriate content for machine vision analysis. The method comprises the following steps:

[0016] Step 1: Reconstruction of a true 3D real-scene model: Using drone oblique photography technology to collect image information of the X(m)*Y(m)*Z(m) real experimental site, then generate a 1:1 X(m)*Y(m)*Z(m) 3D real-scene model in the oblique photography modeling software. Finally, import it into the 3D visualization platform to establish a 1:1 X(m)*Y(m)*Z(m) 3D digital experimental environment. The specific steps include:

[0017] Step 11. Use drone oblique photography technology to collect X(m)*Y(m)*Z(m) image information of the actual experimental site. Set the drone's flight speed, shooting interval, camera tilt angle, side overlap, route overlap, and flight altitude.

[0018] Step 12. Generate a 1:1 X(m)*Y(m)*Z(m) 3D real scene model in the oblique photography modeling software;

[0019] Step 13. Import the 3D visualization platform to establish a 1:1 X(m)*Y(m)*Z(m) 3D digital experimental environment.

[0020] Step 2: Build an infrared 3D motion capture system: Attach n infrared motion capture lenses evenly around the top of the real experimental site, ensuring that the lens field of view can cover the capture area of ​​X(m)*Y(m)*Z(m). Attach k reflective markers to the surface of the robot to be motion captured. The number and placement of the reflective markers should be based on accurately capturing its movements. The specific steps include the following:

[0021] Step 21. Use brackets, clamps, and other lens-fixing devices around the top of the actual experimental site to evenly mount n infrared motion capture lenses, ensuring that the lens field of view covers a capture area of ​​X(m)*Y(m)*Z(m). Connect all lenses to a POE switch via network cables.

[0022] Step 22. Use the calibration tool to calibrate, allowing the motion capture software to calculate the accurate relative spatial position and angle of the camera, establish a three-dimensional XYZ coordinate system, and prepare for accurate motion capture;

[0023] Step 23. Attach k reflective markers to the body of the robot whose motion needs to be captured. The number and placement of the reflective markers should be such that the robot's motion can be accurately captured.

[0024] Step 24. Once the system is able to recognize the target object in real time, a complete optical motion capture system has been established. Next, proceed directly to motion capture and data recording.

[0025] Step 3: Record the motion data of the motion robot: The motion robot moves in the real experimental field, and the infrared three-dimensional motion capture system equipment records a series of three-dimensional spatial data of the object's motion {[X ab ,Y ab ,Z ab ],…,a∈[1,k],b∈[0,t]}, where k represents the number of reflective marking points and t represents the different motion capture time points during the recording process, which is a time series with a very short time interval.

[0026] Step 4: Build a 1:1 digital model of the motion robot in a three-dimensional digital experimental environment.

[0027] Step 5: Restore the robot's motion trajectory and posture: Based on the three-dimensional spatial data recorded by the infrared three-dimensional motion capture system, in a 1:1 X(m)*Y(m)*Z(m) three-dimensional digital experimental environment, through certain calculations and transformations, and combined with the digital model outline of the robot, simulate the robot's motion trajectory and posture. The result of this step can be expressed as a series of dynamic motion processes of the robot's digital model in the three-dimensional digital experimental environment. The motion process is highly consistent with the robot's motion process in the actual experimental field. Specifically, the following steps are included:

[0028] Step 51. According to the recorded three-dimensional space data {[X ab ,Y ab ,Z ab ],…, a∈[1, k], b∈[0, t]} in a 1:1 X(m)*Y(m)*Z(m) three-dimensional digital experimental environment, where k represents the number of reflective markers and t represents the different motion capture time points during the recording process, which is a time series with a very short time interval. Set the three-dimensional spatial data of a reflective marker point as the motion path calculation data, such as a=a0, that is Then convert it into the data type required by the 3D visual platform;

[0029] Step 52. Calculate the duration of the movement of the digital model of the moving robot through each of three adjacent recorded three-dimensional spatial data coordinate points, set the speed variable of the digital model of the moving robot to be the speed, calculate the distance between each two points of the three adjacent longitude and latitude using the haversine formula, and then calculate the duration of the movement through the three points using the speed formula;

[0030] The haversine formula is:

[0031]

[0032] Where r represents the equatorial radius of the earth; φ1 and φ5 represent the latitudes of the two points; λ1 and λ5 represent the longitudes of the two points.

[0033] The speed formula is:

[0034] t d =dis / speed........................................ (6)

[0035] where t d represents the duration of the movement, dis represents the distance obtained by Expression 5, and speed represents the movement speed of the digital model of the moving robot.

[0036] Step 53. Calculate the smoothing of the motion trajectory. According to the Bezier curve formula, set the three-dimensional spatial data of a certain reflective mark point as the motion path calculation data, such as a=a0, that is, t represents different capture points during the motion process;

[0037] The expression of the Bezier curve formula is:

[0038]

[0039] where t p Represents the motion time interval of the digital model of the moving robot, t a ∈[0,t d ], t d Calculated by expression 6 in step 52, p0, p1, p2 represent the coordinate points of adjacent three-dimensional space data of each three records. p represents the result of the calculation.

[0040] Step 54. Convert the result calculated by expression 7 in step 53 into the data type required by the three-dimensional visual platform, and then calculate the required orientation angle of two adjacent points in the three-dimensional visual platform to restore the motion posture.

[0041] Step 6, binding virtual camera: construct a virtual camera on the body contour of the digital model of the motion robot. The virtual camera can be fixed at a certain position of the body contour of the digital model and set at a fixed angle. When the digital model of the motion robot moves in the three-dimensional digital experimental environment, the virtual camera will move along with the digital model, and its orientation angle will remain unchanged with the digital model. That is, the digital model of the motion robot moves along the motion trajectory and motion posture in the three-dimensional digital experimental environment. According to the relative position and relative angle of the virtual camera and the digital model, the offset of the virtual camera's perspective change caused by the change of the motion trajectory and motion posture can be calculated in real time, thereby updating the perspective data of the virtual camera in real time, simulating the environmental image obtained by the virtual camera at this position and angle, and the environmental image comes from the three-dimensional digital experimental environment constructed in step 1. Specifically, the following steps are included:

[0042] Step 61. Move the digital model of the motion robot along the motion trajectory and motion posture in the three-dimensional digital experimental environment. The motion data is obtained from the data required by the three-dimensional visual platform converted in step 54. Then, the motion time between two adjacent points is calculated using equations (5) and (6) in step 52. The motion posture data is obtained from the calculation in step 54. Then, the three-dimensional visual platform provides a method to calculate the position of the digital model of the motion robot in real time and change the posture of the digital model of the motion robot in real time.

[0043] Step 62. Build a virtual camera on the contour of the digital model of the robot. For example, take a plane where a reflective marker is located and select a point to bind the virtual camera. This point has a fixed offset from a0 in step 51 in the X, Y, and Z directions. The corresponding offset and the real-time position of the virtual camera can be calculated.

[0044] Step 63. Calculate in real time the offset of the virtual camera's perspective change caused by the change in motion trajectory and motion posture, thereby updating the virtual camera's perspective data in real time. Calculate in real time the required orientation angle of the line connecting the virtual camera's plane and the position of the digitized model of the moving robot on the three-dimensional visual platform using the method of Step 54. Subtract the result from the previous angle to obtain the offset of the current virtual camera's perspective change, and update the perspective data.

[0045] Step 7: Obtain environmental images for content recommendation: Based on step 6, design N virtual cameras and record N video files, denoted as Vn, where n∈[1,N]. These N video files are used for machine vision evaluation. Since the position and angle of the video shooting will affect the video capture results, and the shooting results will affect the machine vision calculation results, the quality of the virtual camera installation position is judged based on the evaluation results, and the optimal virtual camera shooting position is recommended as the actual machine installation location. The specific steps include:

[0046] Step 71: Design N virtual cameras according to step 6, record N video files, denoted as Vn, where n∈[1,N], start recording digital image video data, and finally crop the image video data according to the resolution set according to actual needs to obtain the final video file;

[0047] In step 72, N video files are used for machine vision evaluation. The quality of the virtual camera installation position is judged based on the evaluation results, and a better virtual camera shooting position is recommended as the actual machine installation position. The evaluation process can be composed of multiple evaluation items.

[0048] Preferably, taking CesiumJs (a JavaScript library written in JavaScript using WebGL, which builds a three-dimensional visualization platform through a web browser) as an example, in step 13, the three-dimensional visualization platform is imported to establish a 1:1 X(m)*Y(m)*Z(m) three-dimensional digital experimental environment.

[0049] The three-dimensional real scene model data is imported into CesiumJs to establish a three-dimensional digital experimental environment, and according to the function method Cesium.Cesium3DTileset({url:}) provided by it, the url parameter represents the storage address of the model data, and a model object is created;

[0050] The expression is:

[0051] Cesium.Cesium3DTileset({url:model data storage address})..........................(1)

[0052] Preferably, in step 51, the Cesium.Cartesian3(x,y,z) method provided by CesiumJs, wherein x, y, z correspond to three-dimensional spatial data [X 1b ,Y 1b ,Z 1b ], first convert it into a Cartesian coordinate system (world coordinate) object;

[0053] The expression is:

[0054] Cesium.Cartesian3(X 1b ,Y 1b ,Z 1b )......................................(2)

[0055] Then, the Cesium.Cartographic.fromCartesian(cartesian) method provided by CesiumJs, where cartesian is the Cartesian coordinate system (world coordinate) object converted by formula (2), is converted into an radian representation object of longitude and latitude in the geographic coordinate system;

[0056] The expression is:

[0057] Cesium.Cartographic.fromCartesian(cartesian)........................(3)

[0058] Then, the Cesium.Math.toDegrees(cartographic) method provided by CesiumJs is used. Cartographic is the arc representation object of the longitude and latitude in the geographic coordinate system converted by formula (3), which contains three parameters: latitude, longitude, and height. They are converted into angle representation objects respectively. The expression is:

[0059] Cesium.Math.toDegrees(cartographic.latitude)........................(4)

[0060] Preferably, in step 54, the result calculated by formula (7) is converted into longitude and latitude data by expressions 2, 3, and 4 in step 51, and then the orientation angle of the two adjacent points to the north direction is calculated, and the plane angle (radian value) between the line segment from the origin (0,0) to the point (x,y) and the positive direction of the x-axis is calculated by Math.atan2(y,x), where (x,y) represents the point parallel to the line connecting it and (0,0) and the line connecting the two adjacent points, and the motion posture is restored.

[0061] The expression for calculating (x, y) is:

[0062] x=Math.cos(startLat)*Math.sin(endLat)-Math.sin(startLat)*

[0063] Math.cos(endLat)*Math.cos(endLon-startLng)................................................................(8)

[0064] y=Math.sin(endLon-startLon)*Math.cos(endLat)........................(9)

[0065] Where startLon represents the starting longitude; endLon represents the ending longitude; startLat represents the starting latitude; destLat represents the ending latitude;

[0066] The expression for calculating the plane angle (radian value) is:

[0067] Math.atan2(y,x)................................................(10)

[0068] Where x and y are the calculation results of equations (8) and (9).

[0069] Preferably, in step 61, the function method Cesium.Cartesian3.fromDegrees() provided by CesiumJs calculates the digital model position of the motion robot in real time, and Cesium.HeadingPitchRoll() changes the digital model posture of the motion robot in real time;

[0070] The digital model position expression of the computational motion robot is:

[0071] Cesium.Cartesian3.fromDegrees(currentLon,currentLat,currentHeight)............(11)

[0072] Where currentLon represents the real-time longitude; currentLat represents the real-time latitude; and currentHeight represents the real-time altitude.

[0073] The digital model posture expression of the motion robot is:

[0074] let hpRoll=new Cesium.HeadingPitchRoll();........................(12)

[0075] hpRoll.heading=Cesium.Math.toRadians(X0);........................(13)

[0076] Among them, hpRoll represents the object that changes its posture; heading represents the yaw angle parameter, and also includes pitch (pitch angle) and row (pitch angle) parameters that can be changed; Cesium.Math.toRadians() is also a function method provided by CesiumJs, which converts the posture variable X0 from radians to an angle representation object.

[0077] Preferably, in step 62, the real-time position of the virtual camera is calculated by the function method Cesium.Matrix4.multiplyByPoint() provided by CesiumJs;

[0078] The offset calculation is expressed as:

[0079] Cesium.Cartesian3.multiplyByScalar(Cesium.Cartesian3.UNIT_X,X s ,positionV).(14)

[0080] Among them, Cesium.Cartesian3.UNIT_X represents the change of velocity vector in the X direction, Cesium.Cartesian3.UNIT_Y represents the change of velocity vector in the Y direction, and Cesium.Cartesian3.UNIT_Z represents the change of velocity vector in the Z direction; s Indicates the offset in the current specified direction; positionV represents the offset vector object, which is created by the Cesium.Cartesian3() function method provided by CesiumJs.

[0081] The real-time position calculation expression is:

[0082] Cesium.Matrix4.multiplyByPoint(modelMatrix,positionV,position)............(15)

[0083] Where modelMatrix represents a 4x4 transformation matrix object, created by the function method Cesium.Transforms.headingPitchRollToFixedFrame() provided by CesiumJs; positionV is calculated by formula (14); position is calculated by formula (11).

[0084] The 4x4 conversion matrix expression is:

[0085]

[0086] Where position and hpRoll are calculated by equations (11), (12), and (13); Cesium.Ellipsoid.WGS84 represents the standard WGS84 ellipsoid; the Cesium.Transforms.localFrameToFixedFrameGenerator() method depends on the orientation of the model data, and has east, south, west, and north.

[0087] The function method viewer.scene.camera.setView() provided by CesiumJs binds the virtual camera.

[0088] The expression is:

[0089]

[0090]

[0091] The destination represents the location of the virtual camera; the orientation represents the posture of the virtual camera, including heading (yaw angle), pitch (pitch angle) and row (roll angle), which can be set as needed.

[0092] Preferably, in step 71, the window content is captured through the getDisplayMedia() method provided by the web interface, and the media stream recording object is generated using the MediaRecorder() method. Finally, the digital image video data is started to be recorded through the .start() method. Finally, the image video data is cropped according to the resolution set according to actual needs to obtain the final video file;

[0093] The capture window content expression is:

[0094] navigator.mediaDevices.getDisplayMedia(displayMediaOptions);.............(18)

[0095] displayMediaOptions indicates setting capture parameters.

[0096] The recording media stream expression is:

[0097] recorder=new MediaRecorder(captureStream);........................(19)

[0098] recorder.start();................................................(20)

[0099] Where captureStream represents the window content object generated by the previous expression; recorder represents the media stream object.

[0100] Preferably, in step 72, the obstacle avoidance function is used as an example to illustrate the evaluation process and evaluation method, and the video V is taken. n exist[ The video segments of the time period are used as input to the machine vision obstacle avoidance evaluation algorithm and are provided to the machine vision obstacle avoidance evaluation algorithm for calculation. The distance of the virtual obstacle obtained by the calculation of each video segment input is calculated. and calculate Distance from reality The smaller the error, the better the quality of the video (which is beneficial for machine vision analysis). The machine vision obstacle avoidance evaluation algorithm is expressed as:

[0101]

[0102] in Represents the video frame image of the video file shot by the nth virtual camera at time t, OBSTACLE AVOIDANCE is a machine vision obstacle avoidance evaluation algorithm formula.

[0103] The obstacle avoidance error rate is expressed as:

[0104]

[0105] V n The obstacle avoidance error is S of all video frames in the time period t mean

[0106] Finally, obstacle avoidance evaluation is performed on all video files in this time period. The smaller the video, the better the effect. Sort from small to large, and the first Y positions in the agreed sorting are the recommended positions.

[0107] By adopting the above-mentioned technical solution, the present invention collects the actual motion trajectory and motion posture of a real-life moving object, realizing its simulated motion in a three-dimensional digital experimental environment. Simultaneously, the present invention processes the collected points to ensure that they are accurately and smoothly connected into a trajectory line in a virtual three-dimensional space, and enables the virtual camera to accurately, smoothly, and controllably move the viewing angle along the trajectory line for shooting. Because the three-dimensional digital experimental environment reconstructed using a true three-dimensional real-scene model reconstruction system is highly realistic to the real environment, the digital image video captured by the virtual camera can highly simulate the image captured by the real camera. Furthermore, using a digital experimental method, theoretically, any number of virtual cameras can be installed on the surface of the digital model of the moving robot. Therefore, it is possible to simulate the environmental images that can be captured by cameras installed at any angle on the surface of the moving robot's body, without the need to actually install cameras. This greatly reduces the complexity of the experiment and reduces the equipment and time costs of the experiment. The acquired environmental images are used for machine vision evaluation. Based on the evaluation results, the quality of the virtual camera installation position is judged, and a certain machine vision evaluation is performed on the environmental images to initially recommend suitable content.

[0108] The advantages of the present invention are: reducing the complexity of the experiment, and also reducing the equipment cost and time cost of the experiment. BRIEF DESCRIPTION OF THE DRAWINGS

[0109] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments, illustrating the method for recommending content based on realistic trajectory simulation and perspective capture for digital twin scenarios.

[0110] Figure 1 It is a schematic diagram of a three-dimensional digital experimental environment for establishing a real experimental site according to the present invention;

[0111] Figure: 1. 8 infrared motion capture cameras mounted on the top, 2. Real experimental site, 3. Motion robot, 4.1, 4.2, 4.3. Reflective markers, 5. 3D digital experimental environment, 6. Digital model of the motion robot, 7. Virtual camera;

[0112] Figure 2 This is a flowchart of step 1 of the present invention for reconstructing a true three-dimensional real scene model;

[0113] Figure 3 This is a flowchart of step 2 of the present invention for building an infrared three-dimensional motion capture system;

[0114] Figure 4 This is a flow chart of restoring the motion trajectory and motion posture of the motion robot in step 5 of the present invention;

[0115] Figure 5 This is a flowchart of step 6 of the present invention for binding a virtual camera.

[0116] Figure 6 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0117] The technical solution of the present invention is further described in detail below through specific embodiments and drawings.

[0118] The method for recommending content for realistic trajectory simulation and perspective capture in digital twin scenarios described in the present invention is characterized by using digital technology to digitize a motion robot, a real experimental site, and a motion trajectory to construct a digital experimental environment. The method then uses virtual simulation to capture environmental images captured by a virtual camera, uses the captured environmental images for machine vision evaluation, and preliminarily recommends relatively suitable content for machine vision analysis. The method mainly includes the following steps:

[0119] Step 1, reconstruction of true 3D real scene model: Based on the UAV oblique photography technology, the image information of the X(m)*Y(m)*Z(m) real experimental site is collected, and then a 1:1 X(m)*Y(m)*Z(m) 3D real scene model is generated in the oblique photography modeling software. Finally, it is imported into the 3D visualization platform to establish a 1:1 X(m)*Y(m)*Z(m) 3D digital experimental environment. Figure 2 As shown, the following steps are included:

[0120] Step 11. Using drone oblique photography technology, collect image information of a 10m*10m*10m real-world experimental site. Set the flight speed to 8m / s, the capture interval to 2s, and the camera tilt angle to 45°. Because this will be used for 3D scene model reconstruction, set the lateral overlap to 75% and the flight path overlap to 80%. The flight altitude should be approximately 30 meters above the scene, or 40 meters.

[0121] Step 12. Generate a 1:1 10(m)*10(m)*10(m) 3D real-world model in the oblique photography modeling software. Using ContextCapture software as an example, the software automatically performs dodging, triangulation, calculation of triangulation encryption points, TIN model construction, and texture fitting to generate a 1:1 10(m)*10(m)*10(m) 3D real-world model.

[0122] Step 13. Import the 3D visualization platform to establish a 1:1 10m*10m*10m 3D digital experimental environment. Using CesiumJS (a JavaScript library written in JavaScript and using WebGL to build a 3D visualization platform through a web browser) as an example, export the 3D real-world model data in the 3DTiles format supported by CesiumJS from ContextCapture software and import it into CesiumJS to establish a 1:1 10m*10m*10m 3D digital experimental environment.

[0123] The three-dimensional real scene model data is imported into CesiumJs to establish a three-dimensional digital experimental environment, and according to the function method Cesium.Cesium3DTileset({url:}) provided by it, the url parameter represents the storage address of the model data, and a model object is created;

[0124] The expression is:

[0125] Cesium.Cesium3DTileset({url:model data storage address})..........................(1)

[0126] Step 2: Build an infrared 3D motion capture system: Evenly attach n infrared motion capture lenses around the top of the real experimental site to ensure that the lens field of view can cover the capture area of ​​X(m)*Y(m)*Z(m), and attach k reflective markers to the surface of the robot body that needs motion capture. The number and location of the reflective markers should be based on the ability to accurately capture its movements, such as Figure 3 As shown, the following steps are included:

[0127] Step 21. Use brackets, clamps and other lens fixing devices around the top of the actual experimental site, refer to Figure 1 As shown in Figure 1, bind an average of 8 infrared motion capture lenses to ensure that the lens field of view can cover a capture area of ​​10 (m) * 10 (m) * 10 (m), and connect all lenses to a POE switch via a network cable;

[0128] Step 22. Use the calibration tool to calibrate, allowing the motion capture software to calculate the accurate relative spatial position and angle of the camera, establish a three-dimensional XYZ coordinate system, and prepare for accurate motion capture;

[0129] Step 23. Stick five reflective marking points on the body of the robot that needs motion capture. The number and location of the reflective marking points should be based on the ability to accurately capture its movements. Figure 1As shown in 3, place a reflective mark in the middle of each of the five surfaces of the robot except the bottom surface. Figure 1 As shown in 4.1, 4.2, and 4.3;

[0130] Step 24. Once the system is able to recognize the target object in real time, a complete optical motion capture system has been established. Next, proceed directly to motion capture and data recording.

[0131] Step 3: Record the motion data of the motion robot: The motion robot moves in the real experimental field, and the infrared three-dimensional motion capture system equipment records a series of three-dimensional spatial data of the object's motion {[X ab ,Y ab ,Z ab ],…,a∈[1,k],b∈[0,t]}, where k represents the number of reflective marking points, and t represents the different action capture time points in the recording process, which is a time series with a very short time interval. In this specific implementation, k=5;

[0132] Step 4: In a three-dimensional digital experimental environment, a digital model of the motion robot is constructed at a 1:1 scale.

[0133] Step 5: Restore the motion trajectory and motion posture of the motion robot: Based on the three-dimensional spatial data recorded by the infrared three-dimensional motion capture system equipment, in a 1:1 X(m)*Y(m)*Z(m) three-dimensional digital experimental environment, through certain calculations and transformations, and combined with the digital model outline of the motion robot, simulate the motion trajectory and motion posture of the motion robot; the result of this step can be expressed as a series of dynamic motion processes of the digital model of the motion robot in the three-dimensional digital experimental environment, and its motion process highly matches the motion process of the motion robot in the real experimental field, such as Figure 4 As shown, the following steps are included:

[0134] Step 51. According to the recorded three-dimensional space data {[X ab ,Y ab ,Z ab ],…, a∈[1,5],b∈[0,t]} in a 1:1 10(m)*10(m)*10(m) three-dimensional digital experimental environment, refer to Figure 1 As shown in 4.1, the three-dimensional space data of the point is set as the motion path calculation data, a=1, that is, {[X 1b ,Y 1b ,Z 1b],…, b∈[0,t]}, t represents the different action capture time points in the recording process, which is a time series with a very short time interval. The Cesium.Cartesian3(x,y,z) method provided by CesiumJs, where x, y, and z correspond to three-dimensional spatial data [X 1b ,Y 1b ,Z 1b ], first convert it into a Cartesian coordinate system (world coordinate) object;

[0135] The expression is:

[0136] Cesium.Cartesian3(X 1b ,Y 1b ,Z 1b )......................................(2)

[0137] Then, the Cesium.Cartographic.fromCartesian(cartesian) method provided by CesiumJs, where cartesian is the Cartesian coordinate system (world coordinate) object converted by expression 2, is converted into an radian representation object of longitude and latitude in the geographic coordinate system;

[0138] The expression is:

[0139] Cesium.Cartographic.fromCartesian(cartesian)........................(3)

[0140] Then, the Cesium.Math.toDegrees(cartographic) method provided by CesiumJs is used, where cartographic is the arc representation object of the longitude and latitude in the geographic coordinate system converted by expression 3, which contains three parameters: latitude, longitude, and height, and converts them into angle representation objects respectively.

[0141] The expression is:

[0142] Cesium.Math.toDegrees(cartographic.latitude)........................(4)

[0143] Step 52. Calculate the duration of the motion of the digital model of the moving robot through each of three adjacent recorded three-dimensional spatial data coordinate points. Set the speed variable of the digital model of the moving robot as speed. Calculate the distance between each two of the three adjacent longitude and latitude coordinate points converted from Expression 4 in Step 51 using the haversine formula. Then, calculate the duration of the motion through the three points using the speed formula.

[0144] The haversine formula is expressed as:

[0145]

[0146] Where r represents the equatorial radius of the earth; φ1 and φ2 represent the latitudes of the two points; λ1 and λ2 represent the longitudes of the two points.

[0147] The speed formula is:

[0148] t d =dis / speed................................................(6)

[0149] where t d represents the duration of the movement, dis represents the distance obtained by Expression 5, and speed represents the movement speed of the digital model of the moving robot.

[0150] Step 53. Smoothing calculation of motion trajectory, calculated according to Bezier curve formula, data reference Figure 1 As shown in 4.1, the three-dimensional space data of the point is set as the motion path calculation data, a=1, that is, {[X 1b ,Y 1b ,Z 1b ],…,b∈[0,t]}, t represents different capture points during the motion process;

[0151] The Bezier curve formula is:

[0152]

[0153] where t p Represents the motion time interval of the digital model of the moving robot, t a ∈[0,t d ], t d Calculated by expression 6 in step 52, p0, p1, p2 represent the adjacent three-dimensional spatial data coordinate points {[X 1b ,Y 1b ,Z 1b ],…,b∈[0,t]},p represents the result of calculation.

[0154] Step 54. Convert the result of expression 7 calculated in step 53 into longitude and latitude data using expressions 2, 3, and 4 in step 51. Then calculate the orientation angle of the two adjacent points relative to the north direction. Use Math.atan2(y,x) to calculate the plane angle (in radians) between the line segment from the origin (0,0) to the point (x,y) and the positive direction of the x-axis. (x,y) represents the point parallel to the line connecting it and (0,0) and the line connecting the two adjacent points, and restore the motion posture.

[0155] The expression for calculating (x, y) is:

[0156] x=Math.cos(startLat)*Math.sin(endLat)-Math.sin(startLat)*

[0157] Math.cos(endLat)*Math.cos(endLon-startLng)................................................................(8)

[0158] y=Math.sin(endLon-startLon)*Math.cos(endLat)........................(9)

[0159] Where startLon represents the starting longitude; endLon represents the ending longitude; startLat represents the starting latitude; destLat represents the ending latitude;

[0160] The expression for calculating the plane angle (radian value) is:

[0161] Math.atan2(y,x)................................................(10)

[0162] Where x and y are the calculation results of expressions 8 and 9.

[0163] Step 6, bind the virtual camera: construct a virtual camera on the body contour of the digital model of the moving robot. The virtual camera can be fixed at a certain position of the body contour of the digital model and set at a fixed angle. When the digital model of the moving robot moves in the three-dimensional digital experimental environment, the virtual camera will move with the digital model, and its orientation angle will remain unchanged with the digital model. That is, the digital model of the moving robot moves along the motion trajectory and motion posture in the three-dimensional digital experimental environment. According to the relative position and relative angle of the virtual camera and the digital model, the offset of the virtual camera's perspective change caused by the change of the motion trajectory and motion posture can be calculated in real time, so as to update the perspective data of the virtual camera in real time, and simulate the environmental image obtained by the virtual camera at this position and angle. The environmental image comes from the three-dimensional digital experimental environment constructed in step 1, such as Figure 5 As shown, the following steps are included:

[0164] Step 61. Move the digital model of the robot along the motion trajectory and motion posture in the three-dimensional digital experimental environment. The motion data is taken from the longitude and latitude data in step 54. Then, the motion time between two adjacent points is calculated using expressions 5 and 6 in step 52. The motion posture data is taken from expressions 8, 9, and 10 in step 54. Then, the function method Cesium.Cartesian3.fromDegrees() provided by CesiumJs is used to calculate the position of the digital model of the robot in real time, and Cesium.HeadingPitchRoll() is used to change the posture of the digital model of the robot in real time.

[0165] The digital model position expression of the computational motion robot is:

[0166] Cesium.Cartesian3.fromDegrees(currentLon,currentLat,currentHeight)............(11)

[0167] Where currentLon represents the real-time longitude; currentLat represents the real-time latitude; and currentHeight represents the real-time altitude.

[0168] The digital model posture expression of the motion robot is:

[0169] let hpRoll=new Cesium.HeadingPitchRoll();........................(12)

[0170] hpRoll.heading=Cesium.Math.toRadians(X0);........................(13)

[0171] Among them, hpRoll represents the object that changes its posture; heading represents the yaw angle parameter, and also includes pitch (pitch angle) and row (pitch angle) parameters that can be changed; Cesium.Math.toRadians() is also a function method provided by CesiumJs, which converts the posture variable X0 from radians to an angle representation object.

[0172] Step 62. Build a virtual camera on the digital model body contour of the motion robot, referring to Figure 1 As shown in 4.2, take the plane where the point is located as an example, select a point to bind the virtual camera, and the point is Figure 1 4.1 in the above code has fixed offsets in the X, Y, and Z directions. The offset is calculated by the Cesium.Cartesian3.multiplyByScalar() function method provided by CesiumJs. Then, based on the offset and the longitude and latitude data in step 54, the real-time position of the virtual camera is calculated by the Cesium.Matrix4.multiplyByPoint() function method provided by CesiumJs.

[0173] The offset calculation expression is:

[0174] Cesium.Cartesian3.multiplyByScalar(Cesium.Cartesian3.UNIT_X,X s ,positionV).(14)

[0175] Among them, Cesium.Cartesian3.UNIT_X represents the change of velocity vector in the X direction, Cesium.Cartesian3.UNIT_Y represents the change of velocity vector in the Y direction, and Cesium.Cartesian3.UNIT_Z represents the change of velocity vector in the Z direction; s Indicates the offset in the current specified direction; positionV represents the offset vector object, which is created by the Cesium.Cartesian3() function method provided by CesiumJs.

[0176] The real-time position calculation expression is:

[0177] Cesium.Matrix4.multiplyByPoint(modelMatrix,positionV,position)............(15)

[0178] Where modelMatrix represents a 4x4 transformation matrix object, created by the function method Cesium.Transforms.headingPitchRollToFixedFrame() provided by CesiumJs; positionV is calculated by expression 14; position is calculated by expression 11.

[0179] The 4x4 conversion matrix expression is:

[0180]

[0181] Where position and hpRoll are calculated by expressions 11, 12, and 13 in step 61; Cesium.Ellipsoid.WGS84 represents the standard WGS84 ellipsoid; the Cesium.Transforms.localFrameToFixedFrameGenerator() method depends on the orientation of the model data, and has east, south, west, and north.

[0182] The function method viewer.scene.camera.setView() provided by CesiumJs binds the virtual camera.

[0183] The expression is:

[0184]

[0185] The destination represents the location of the virtual camera; the orientation represents the posture of the virtual camera, including heading (yaw angle), pitch (pitch angle) and row (roll angle), which can be set as needed.

[0186] Step 63. Calculate in real time the offset of the virtual camera's perspective change caused by the change in motion trajectory and motion posture, thereby updating the virtual camera's perspective data in real time, and calculate in real time the plane where the virtual camera is located by the method of step 54, i.e., refer to Figure 1 4.2 in the above equation, a=2, that is, {[X 2b ,Y 2b ,Z 2b],…,b∈[0,t]}, t represents the different capture points in the motion process, and the orientation angle of the line connecting the digital model position of the moving robot with respect to the true north direction, and the result of the difference with the previous angle is the offset of the current virtual camera perspective change. Based on the three parameters of orientation in expression 17 in step 62, the perspective data is updated;

[0187] Step 7: Obtain environmental images for content recommendation: Based on step 6, design N virtual cameras and record N video files, denoted as Vn, where n∈[1,N]. These N video files are used for machine vision evaluation. Since the position and angle of the video shooting will affect the video capture results, the shooting results will affect the machine vision calculation results. Based on the evaluation results, the quality of the virtual camera installation position is judged, and the optimal virtual camera shooting position is recommended as the actual machine installation position. The following steps are included:

[0188] Step 71: Design N virtual cameras according to step 6, record N video files, denoted as Vn, where n∈[1,N], capture the window content through the getDisplayMedia() method provided by the web interface, and use the MediaRecorder() method to generate a media stream recording object. Finally, use the .start() method to start recording digital image video data. Finally, crop the image video data according to the resolution set according to actual needs to obtain the final video file.

[0189] The capture window content expression is:

[0190] navigator.mediaDevices.getDisplayMedia(displayMediaOptions);.............(18)

[0191] displayMediaOptions indicates setting capture parameters.

[0192] The recording media stream expression is:

[0193] recorder = new MediaRecorder(captureStream);........................ (19)

[0194] recorder.start();................................................(20)

[0195] Where captureStream represents the window content object generated by the previous expression; recorder represents the media stream object.

[0196] In step 72, N video files are used for machine vision evaluation. The quality of the virtual camera installation position is judged based on the quality of the evaluation results, and a better virtual camera shooting position is recommended as the actual machine installation position. The evaluation process consists of multiple evaluation items. The following uses one of the evaluation items, obstacle avoidance function, as an example to illustrate the evaluation process and evaluation method.

[0197] The obstacle avoidance function means that the machine vision should be able to identify obstacles on the forward path and determine the distance of the obstacles based on the continuous video captured by the camera, so as to control the robot to avoid the obstacles. z When there is an obstacle Z, it should be able to give an obstacle warning in T time and calculate the distance X between the robot and Z m , calculated X m With X z The error is kept within w%, such as w is less than 10. z , calculation time T, and error w are all expected experimental data agreed before the experiment.

[0198] During the experiment, when the digital model of the moving robot moves in the three-dimensional digital experimental environment, the three-dimensional digital experimental environment is a mirror image of the real experimental environment. The movement process of the digital model of the moving robot in the digital experimental environment is also a mirror image and digital twin process of the real robot's movement process in the real environment. In the real experimental environment, when the moving robot moves in front of obstacle Z, its digital experimental process is synchronously manifested as a virtual obstacle Z of the digital experimental environment appearing in front of the digital model of the moving robot. d . Virtual obstacle Z in digital experimental environment d When the digital model of the moving robot faces the virtual obstacle Z d Move, distance X z ±w, the system will automatically give a prompt and time mark and

[0199] The experiment is over, take video V n exist The video segments of the time period are used as input to the machine vision obstacle avoidance evaluation algorithm and are provided to the machine vision obstacle avoidance evaluation algorithm for calculation. The distance of the virtual obstacle obtained by the calculation of each video segment input is calculated. and calculate Distance from reality The smaller the error, the better the quality of the video (which is beneficial for machine vision analysis). The machine vision obstacle avoidance evaluation algorithm is expressed as:

[0200]

[0201] in Represents the video frame image of the video file shot by the nth virtual camera at time t, OBSTACLE AVOIDANCE is a machine vision obstacle avoidance evaluation algorithm formula.

[0202] The obstacle avoidance error rate is expressed as:

[0203]

[0204] V n The obstacle avoidance error is S of all video frames in the time period t mean

[0205] Finally, obstacle avoidance evaluation is performed on all video files in this time period. The smaller the video, the better the effect. Sort from small to large, and the first Y positions in the agreed sorting are the recommended positions.

Claims

1. A method for real-world trajectory simulation and perspective capture content recommendation for digital twin scenarios, characterized by: Use digital technology to digitize the motion robot, real experimental site, and motion trajectory, build a digital experimental environment, collect environmental images obtained by virtual cameras through virtual simulation, and perform machine vision evaluation on the environmental images. Preliminary recommendations are made for machine vision analysis, including the following steps: Step 1, reconstruction of true 3D real scene model: Based on the UAV oblique photography technology, collect the image information of the X(m)*Y(m)*Z(m) real experimental site, and then generate a 1:1 X(m)*Y(m)*Z(m) 3D real scene model in the oblique photography modeling software, and finally import it into the 3D visualization platform to establish a 1:1 X(m)*Y(m)*Z(m) 3D digital experimental environment, including: Step 11. Collect image information of the X(m)*Y(m)*Z(m) real experimental site based on the drone oblique photography technology; set the drone flight speed, shooting interval, camera tilt angle, lateral overlap, route overlap and flight altitude; Step 12. Generate a 1:1 X(m)*Y(m)*Z(m) three-dimensional real scene model in the oblique photography modeling software; Step 13. Import the 3D visualization platform to establish a 1:1 X(m)*Y(m)*Z(m) 3D digital experimental environment; Step 2, build an infrared 3D motion capture system: bind u infrared motion capture lenses on the top and around the real experimental site on average, ensure that the lens field of view can cover the capture area of ​​X(m)*Y(m)*Z(m), and affix k reflective marking points on the body surface of the motion robot that needs motion capture. The number and position of the reflective marking points shall be based on the ability to accurately capture its movements, including: Step 21. Use brackets and clamp lens fixtures to evenly bind u infrared motion capture lenses around the top of the real experimental site, ensure that the lens field of view can cover the capture area of ​​X(m)*Y(m)*Z(m), and connect all lenses to the POE switch via network cables; Step 22. Use the calibration tool to calibrate, allowing the motion capture software to calculate the accurate relative spatial position and angle of the lens, establish a three-dimensional XYZ coordinate system, and prepare for accurate motion capture; Step 23. Paste k reflective marking points on the body surface of the sports robot that needs motion capture. The number and position of the reflective marking points should be based on the ability to accurately capture its motion. Step 24. When the system can recognize the target object in real time, a complete optical motion capture system has been established. Next, directly perform motion capture and record data; Step 3, record the motion data of the motion robot: the motion robot moves in the real experimental field, and the infrared 3D motion capture system equipment records a series of 3D spatial data of the object's motion [X ab ,Y ab ,Z ab ], a∈[1, k], b∈[0, t], where k represents the number of reflective marking points, and t represents the different action capture time points during the recording process, which is a time series with a very short time interval; Step 4, in a three-dimensional digital experimental environment, construct a digital model of the motion robot at a 1:1 ratio; Step 5, restore the motion trajectory and motion posture of the motion robot: according to the three-dimensional space data recorded by the infrared three-dimensional motion capture system equipment, in a 1:1 X(m)*Y(m)*Z(m) three-dimensional digital experimental environment, through calculation and transformation, and combined with the digital model outline of the motion robot, simulate the motion trajectory and motion posture of the motion robot; the result of this step is a series of dynamic motion processes of the digital model of the motion robot in the three-dimensional digital experimental environment, and its motion process highly fits the motion process of the motion robot in the real experimental field, which specifically includes the following steps: Step 51. According to the recorded three-dimensional space data [X ab ,Y ab ,Z ab ], a∈[1, k], b∈[0, t] in a 1:1 X(m)*Y(m)*Z(m) three-dimensional digital experimental environment, where k represents the number of reflective marking points, and t represents the different action capture time points in the recording process, which is a time series with a very short time interval; the three-dimensional spatial data of a certain reflective marking point is set as the motion path calculation data, a=a0, that is, b∈[0, t], and then converted into the data type required by the three-dimensional visual platform; Step 52. Calculate the movement time of the digital model of the moving robot through every three recorded adjacent three-dimensional space data coordinate points, set the movement speed variable speed of the digital model of the moving robot, calculate the distance between every two points of three adjacent longitudes and latitudes by the haversine formula, and then calculate the movement time through the three points according to the speed formula; The haversine formula is: Where r represents the equatorial radius of the Earth; and represents the latitude of two points; λ1 and λ2 represent the longitude of two points; The speed formula is: t d =dis / speed ...................................................... (6) where t d represents the movement duration, dis represents the distance obtained by expression (5), and speed represents the movement speed of the digital model of the moving robot; Step 53. Calculate the smoothing of the motion trajectory. According to the Bezier curve formula, set the three-dimensional space data of a certain reflective mark point as the motion path calculation data, a=a0, that is, b∈[0, t], t represents different capture points during the motion process; The expression of the Bezier curve formula is: where t p Represents the motion time interval of the digital model of the moving robot, t p ∈[0,t d ], t d Calculated by expression (6) in step 52, p0, p1, p2 represent the coordinate points of adjacent three-dimensional space data of every three records. b∈[0, t], p represents the result of calculation; Step 54. Convert the result calculated by expression (7) in step 53 into the data type required by the three-dimensional visual platform, and then calculate the orientation angle required by two adjacent points in the three-dimensional visual platform to restore the motion posture; Step 6, binding a virtual camera: construct a virtual camera on the body contour of the digital model of the motion robot, the virtual camera is fixed at a certain position of the body contour of the digital model, and is set at a fixed angle; when the digital model of the motion robot moves in the three-dimensional digital experimental environment, the virtual camera will move with the digital model, and its orientation angle will remain unchanged with the digital model; that is, the digital model of the motion robot moves along the motion trajectory and motion posture in the three-dimensional digital experimental environment, and according to the relative position and relative angle of the virtual camera and the digital model, the offset of the change of the virtual camera's perspective caused by the change of the motion trajectory and motion posture is calculated in real time, so as to update the perspective data of the virtual camera in real time, and simulate the environmental image obtained by the virtual camera at the position and angle, and the environmental image comes from the three-dimensional digital experimental environment constructed in step 1, which specifically includes the following steps: Step 61. The digital model of the motion robot is moved along the motion trajectory and motion posture in the three-dimensional digital experimental environment. The motion data is taken from the data required by the three-dimensional visual platform converted in step 54. Then, the motion time between two adjacent points is calculated by equations (5) and (6) in step 52. The motion posture data is taken from the calculation result in step 54. Then, the three-dimensional visual platform provides a method to calculate the position of the digital model of the motion robot in real time and change the posture of the digital model of the motion robot in real time. Step 62. Construct a virtual camera on the contour of the digital model body of the motion robot, and select a point on the plane where a certain reflective mark point is located to bind the virtual camera. This point has a fixed offset with a0 in step 51 in the X, Y, and Z directions, and the corresponding offset and the real-time position of the virtual camera can be calculated; Step 63. Calculate in real time the offset of the change in the virtual camera's viewing angle caused by the change in the motion trajectory and motion posture, so as to update the viewing angle data of the virtual camera in real time. Calculate in real time the orientation angle required by the plane where the virtual camera is located and the line connecting the position of the digital model of the motion robot in the three-dimensional visual platform through the method of step 54, and calculate the result of the difference with the previous angle as the offset of the current virtual camera's viewing angle change, and update the viewing angle data. Step 7, obtain environmental images for content recommendation: design N virtual cameras according to step 6, record N video files and record them as Vn, where n∈[1,N]; use N video files for machine vision evaluation, because the position and angle of video shooting will affect the video shooting results, and the shooting results will affect the machine vision calculation results; judge the advantages and disadvantages of the virtual camera installation position according to the evaluation results, and recommend the better virtual camera shooting position as the actual machine installation position, which specifically includes the following steps: Step 71, design N virtual cameras according to step 6, record N video files as Vn, where n∈[1,N], start recording digital image video data, and finally crop the image video data according to the resolution set according to actual needs to obtain the final video file; Step 72, use N video files for machine vision evaluation, judge the quality of the virtual camera installation position according to the evaluation results, and recommend a better virtual camera shooting position as the actual machine installation position. The evaluation process consists of multiple evaluation items; the evaluation process and evaluation method are explained with the obstacle avoidance function. Take video V n exist The video segments of the time period are used as input to the machine vision obstacle avoidance evaluation algorithm, and are provided to the machine vision obstacle avoidance evaluation algorithm for calculation. The virtual obstacle distance obtained by calculating the input of each video segment is calculated. And calculate Distance from reality The smaller the error, the better the quality of the captured video. The machine vision obstacle avoidance evaluation algorithm is expressed as: in It represents the video frame image of the video file shot by the nth virtual camera at time j. OBSTACLEAVOIDANCE is the machine vision obstacle avoidance evaluation algorithm formula; The obstacle avoidance error rate is expressed as: V n The obstacle avoidance error is S of all video frames in the time period j Mean Finally, obstacle avoidance test is performed on all video files in this time period. The smaller the video, the better the effect. Sort from small to large, and the first I positions in the agreed sorting are the recommended positions.

2. The method for recommending content based on realistic trajectory simulation and perspective capture for digital twin scenes as claimed in claim 1, characterized in that: In step 13, a 1:1 X(m)*Y(m)*Z(m) 3D digital experimental environment is established by importing a 3D visualization platform; The three-dimensional real scene model data is imported into CesiumJs to establish a three-dimensional digital experimental environment, and a model object is created according to the function method Cesium.Cesium3DTileset({url:}) provided by it, and the url parameter represents the storage address of the model data; The expression is: Cesium.Cesium3DTileset({url:model data storage address})........................(1).

3. The method for recommending content based on realistic trajectory simulation and perspective capture for digital twin scenes as claimed in claim 1, characterized in that: In step 51, the Cesium.Cartesian3(x,y,z) method provided by CesiumJs first converts to a Cartesian coordinate system object, where x, y, and z correspond to three-dimensional space data respectively. [X 1b ,Y 1b ,Z 1b ]; The expression is: Cesium.Cartesian3(X 1b ,Y 1b ,Z 1b ).........................................(2)Then the Cesium.Cartographic.fromCartesian(cartesian) method provided by CesiumJs, where cartesian is the Cartesian coordinate system object converted by formula (2), is converted into an radian representation object of longitude and latitude in the geographic coordinate system; The expression is: Cesium.Cartographic.fromCartesian(cartesian)..............................(3) Then the Cesium.Math.toDegrees(cartographic) method provided by CesiumJs is used, where cartographic is the radian representation object of longitude and latitude in the geographic coordinate system converted by formula (3), including three parameters: latitude, longitude, and height. They are converted into angle representation objects respectively, and the expressions are as follows: Cesium.Math.toDegrees(cartographic.latitude)........................(4).

4. The method for recommending content based on realistic trajectory simulation and perspective capture for digital twin scenes as claimed in claim 3, characterized in that: In step 54, the result of equation (7) is converted into equation (2) in step 51, (3) and (4) are converted into longitude and latitude data, and then the orientation angle of the two adjacent points to the north direction is calculated. Math.atan2(y,x) is used to calculate the plane angle between the line segment from the origin (0,0) to the point (x,y) and the positive direction of the x-axis, that is, the radian value, where (x,y) represents the point whose line connecting it and (0,0) is parallel to the line connecting the two adjacent points, and the motion posture is restored. state; The expression for calculating (x,y) is: x=Math.cos(startLat)*Math.sin(endLat)-Math.sin(startLat)* Math.cos(endLat)*Math.cos(endLon-startLon)................................................................(8) y = Math.sin(endLon - startLon) * Math.cos(endLat)........................ (9) Where startLon represents the starting longitude; endLon represents the ending longitude; startLat represents the starting latitude; endLat indicates the latitude of the end point; The expression for calculating the plane angle, i.e. the radian value, is: Math.atan2(y,x).....................................................(10)where x, y are the calculation results of equations (8) and (9).

5. The method for recommending content based on realistic trajectory simulation and perspective capture for digital twin scenes as claimed in claim 1, characterized in that: In step 61, the function method Cesium.Cartesian3.fromDegrees() provided by CesiumJs calculates the digital model position of the moving robot in real time. Cesium.HeadingPitchRoll() changes the digital model posture of the moving robot in real time; The digital model position expression of the computational motion robot is: Cesium.Cartesian3.fromDegrees(currentLon,currentLat,currentHeight)...........(11)where currentLon represents the real-time longitude; currentLat represents the real-time latitude; currentHeight represents the real-time height; The digital model posture expression of the motion robot is: let hpRoll = new Cesium.HeadingPitchRoll(); ............................. (12) hpRoll.heading = Cesium.Math.toRadians(X0); ............................. (13) Among them, hpRoll represents the object that changes the posture; heading represents the yaw angle parameter, and also includes the pitch angle and row pitch angle parameters that can be changed; Cesium.Math.toRadians() is also a function method provided by CesiumJs, which converts the posture variable X0 from radians to an angle representation object.

6. The method for recommending content based on realistic trajectory simulation and perspective capture for digital twin scenes as claimed in claim 5, characterized in that: In step 62, the function method Cesium.Matrix4.multiplyByPoint() provided by CesiumJs calculates the real-time position of the virtual camera; The offset calculation is expressed as: Cesium.Cartesian3.multiplyByScalar(Cesium.Cartesian3.UNIT_X,X s ,positionV)..(14) Among them, Cesium.Cartesian3.UNIT_X represents the change of velocity vector in the X direction, and Cesium.Cartesian3.UNIT_Y represents the change of velocity vector in the Y direction, and Cesium.Cartesian3.UNIT_Z represents the change of velocity vector in the Z direction; s Represents the offset in the current specified direction; positionV represents the offset vector object, which is created by the function method Cesium.Cartesian3() provided by CesiumJs; The real-time position calculation expression is: Cesium.Matrix4.multiplyByPoint(modelMatrix,positionV,position)............(15) Where modelMatrix represents a 4x4 transformation matrix object, which is created by the function method Cesium.Transforms.headingPitchRollToFixedFrame() provided by CesiumJs; positionV is calculated by formula (14); position is calculated by formula (11); The 4x4 transformation matrix expression is: Cesium.Transforms.headingPitchRollToFixedFrame( position, hpRoll, Cesium.Ellipsoid.WGS84, Cesium.Transforms.localFrameToFixedFrameGenerator('south','east')).................................................................(16)where position and hpRoll are calculated by equations (11), (12), and (13); Cesium.Ellipsoid.WGS84 represents the standard WGS84 ellipsoid; The Cesium.Transforms.localFrameToFixedFrameGenerator() method depends on the orientation of the model data, which can be east, south, west, or north. The function method viewer.scene.camera.setView() provided by CesiumJs binds the virtual camera. The expression is: The destination represents the position of the virtual camera; the orientation represents the posture of the virtual camera, including the heading angle, pitch angle and row angle, which can be set as needed.

7. The method for recommending content based on realistic trajectory simulation and perspective capture for digital twin scenes as claimed in claim 1, characterized in that: In step 71, getDisplayMedia() provided by the web interface Capture the window content and use the MediaRecorder() method to generate a media stream recording object, and finally use .start() The method starts recording digital video data, and finally cuts the video data according to the resolution set according to actual needs to obtain the final video file; The capture window content expression is: navigator.mediaDevices.getDisplayMedia(displayMediaOptions);................(18) displayMediaOptions indicates setting capture parameters; The recorded media stream expression is: recorder = new MediaRecorder(captureStream);........................ (19) recorder.start();......................................................(20)where captureStream represents the window content object generated by the previous expression; recorder represents the media stream object.

Citation Information

Patent Citations

  • Virtual scene visualization method and system

    CN108307183A

  • Program for generating and displaying third-visual-angle scene of unmanned aerial vehicle

    CN111443723A

  • Live-action three-dimensional reconstruction method and system

    CN111629193A

  • Virtual reality interaction method based on digital twinning

    CN113485392A

  • Water resources real-time monitoring system based on BIM and digital elevation model

    CN113570275A