A method and device for predicting trajectory of tower crane object

Abstract

The purpose of this application is to provide a method and device for predicting the trajectory of a tower crane object. This application identifies the multi-dimensional posture data of the tower crane object in the real scene, and collects the actual movement speed of the tower crane object through the sensor installed on the tower crane; the multi-dimensional posture data and the actual movement speed are transmitted to the simulation environment, and the current speed and direction of the tower crane object are displayed using the same posture; the running trajectory of the tower crane object is predicted according to the current speed and direction of the tower crane object through the physical engine built in the simulation environment; the predicted running trajectory is displayed in the simulation environment and fed back to the real scene. It can improve the accuracy and safety of the prediction, and can predict the future trajectory and visualize more tower crane information.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a method and device for predicting the trajectory of a tower crane object. Background Art

[0002] In the construction of high-rise buildings, tower cranes are essential machinery used to lift raw materials such as steel bars, wooden planks, and steel pipes. When lifting objects, tower cranes need to know the height of the objects that have been lifted, the inclination angle and weight of the objects, as well as the running trajectory of the objects on the tower crane, so as to complete the lifting work safely and accurately. At present, the prediction of the running trajectory of tower crane objects is only carried out by the operators of the tower crane based on observation, which has low accuracy and safety, and cannot predict the future trajectory and visualize more tower crane information. Summary of the invention

[0003] One purpose of the present application is to provide a method and device for predicting the trajectory of a tower crane object, so as to solve the problem in the prior art that the information of the tower crane can only be manually observed and the running trajectory of the tower crane object cannot be accurately predicted.

[0004] According to one aspect of the present application, a method for tower crane object trajectory prediction is provided, the method comprising:

[0005] Identify the multi-dimensional posture data of the tower crane in the real scene, and collect the actual movement speed of the tower crane through the sensors installed on the tower crane;

[0006] The multi-dimensional posture data and the actual movement speed are transmitted into the simulation environment, and the current speed and direction of the tower crane object are displayed using the same posture;

[0007] Predicting the running trajectory of the tower crane object according to the current speed and direction of the tower crane object through a physical engine built in the simulation environment;

[0008] The predicted running trajectory is displayed in the simulation environment and fed back to the real scene.

[0009] Optionally, identifying multi-dimensional posture data of a tower crane object in a real scene includes:

[0010] Perform semantic segmentation on the hook and object images captured by the camera installed on the tower crane in the real scene to obtain an RGB object image, and convert the RGB object image into a depth map;

[0011] Extracting point cloud data from the depth map, wherein the point cloud data includes color features of each RGB pixel and depth features of the point cloud points;

[0012] Fusing the point cloud data to obtain multiple groups of fusion features;

[0013] Training calculations are performed based on the multiple groups of fusion features to obtain multi-dimensional posture data of the tower crane object in the real scene.

[0014] Optionally, the actual movement speed of the tower crane object is collected by a sensor installed on the tower crane, including:

[0015] The speed and acceleration of the tower crane object are collected by the IMU sensor installed on the tower crane, the speed and acceleration are solved, and the geographic coordinates of the tower crane object collected by the positioning sensor are solved;

[0016] The actual movement speed of the tower crane object at each geographical coordinate is determined based on the calculated speed, acceleration and geographical coordinates.

[0017] Optionally, the current speed and direction of the crane object are displayed using the same gesture, including:

[0018] The real-time parameters of the tower crane object in the real scene are calculated and simulated by the physical engine built in the simulation environment;

[0019] The current speed and direction of the tower crane object and real-time parameters are displayed using the same posture to complete the simulation of the real scene.

[0020] Optionally, predicting the running trajectory of the tower crane object according to the current speed and direction of the tower crane object by using a physical engine constructed in the simulation environment includes:

[0021] Performing three-dimensional object space kinematic modeling according to the current speed and direction of the tower crane object through a physical engine built in the simulation environment to obtain a motion model;

[0022] Predicting the state of the tower crane object according to the motion model, matching the state with the observation result to obtain an observation model, and using the observation model to correct the state of the tower crane object;

[0023] The motion model and the observation model are processed based on unscented Kalman filtering to obtain a predicted running trajectory of the tower crane object.

[0024] Optionally, processing the motion model and the observation model based on unscented Kalman filtering includes:

[0025] Determine a new Gaussian distribution of the motion model and the observation model after the position and posture of the tower crane object are updated;

[0026] The new Gaussian distribution is processed based on unscented Kalman filtering.

[0027] Optionally, determining a new Gaussian distribution of the motion model and the observation model after the position and posture of the tower crane object are updated includes:

[0028] The physical engine built in the simulation environment calculates two frames before and after the preset time interval from the current moment, saves the coordinate sequence of the tower crane object in the historical n frames, and obtains the historical trajectory sequence;

[0029] Perform nonlinear processing on the historical trajectory sequence saved at each preset time interval;

[0030] The sigma point after state transfer is transformed untraceably through the sigma algorithm to obtain a new Gaussian distribution.

[0031] Optionally, after predicting the running trajectory of the tower crane object according to the current speed and direction of the tower crane object by using a physical engine constructed in the simulation environment, the method further comprises:

[0032] The predicted trajectory of the tower crane is calibrated using Euler angles.

[0033] Optionally, displaying the predicted running trajectory in the simulation environment and feeding it back to the real scene includes:

[0034] In the simulation environment, a world coordinate axis is displayed with the tower crane object as the center and changes along with the posture of the tower crane object;

[0035] The predicted running trajectory is displayed using marking lines, the combined rate is displayed in a preset font, and the predicted speed is marked in direction using direction marking lines, wherein the combined rate is determined by the multi-dimensional posture data of the tower crane object.

[0036] According to another aspect of the present application, a device for predicting the trajectory of a tower crane object is also provided, the device comprising:

[0037] one or more processors; and

[0038] A memory storing computer readable instructions, which, when executed, cause the processor to perform the operations of the method as described above.

[0039] According to another aspect of the present application, a computer-readable medium is provided, on which computer-readable instructions are stored. The computer-readable instructions can be executed by a processor to implement the method described above.

[0040] Compared with the prior art, this application identifies the multi-dimensional posture data of the tower crane in the real scene, and collects the actual movement speed of the tower crane through the sensor installed on the tower crane; transmits the multi-dimensional posture data and the actual movement speed into the simulation environment, and uses the same posture to display the current speed and direction of the tower crane; predicts the running trajectory of the tower crane according to the current speed and direction of the tower crane through the physical engine built in the simulation environment; displays the predicted running trajectory in the simulation environment and feeds it back to the real scene. It can improve the accuracy and safety of the prediction, and can predict the future trajectory and visualize more tower crane information. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0042] Figure 1 A schematic flow chart of a method for predicting the trajectory of a tower crane object provided according to one aspect of the present application is shown;

[0043] Figure 2 A schematic diagram showing the real-time display effect of a tower crane object in a simulation environment in one embodiment of the present application is shown;

[0044] Figure 3 A schematic diagram showing the display effect of the current speed and current distance of a tower crane object in a simulation environment in one embodiment of the present application is shown;

[0045] Figure 4 A schematic diagram of a method flow chart of tower crane object trajectory prediction based on real simulation in one embodiment of the present application is shown;

[0046] Figure 5 A schematic diagram showing a method of predicting speed based on historical frames and next frames in an embodiment of the present application is shown.

[0047] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0048] The present application is described in further detail below in conjunction with the accompanying drawings.

[0049] In a typical configuration of the present application, the terminal, the device of the service network and the trusted party all include one or more processors (eg, a central processing unit (CPU)), an input / output interface, a network interface and a memory.

[0050] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0051] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, Phase-Change RAM (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of random access memory (RAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0052] Figure 1A schematic flow chart of a method for predicting the trajectory of a tower crane object provided according to one aspect of the present application is shown, and the method includes: steps S11 to S14, wherein, in step S11, the multi-dimensional posture data of the tower crane object in the real scene is identified, and the actual movement speed of the tower crane object is collected through the sensor installed on the tower crane; in step S12, the multi-dimensional posture data and the actual movement speed are transmitted to the simulation environment, and the current speed and direction of the tower crane object are displayed using the same posture; in step S13, the running trajectory of the tower crane object is predicted according to the current speed and direction of the tower crane object through the physical engine built in the simulation environment; in step S14, the predicted running trajectory is displayed in the simulation environment and fed back to the real scene. Thus, the tower crane and the environment in the real scene are simulated into the simulation environment, so as to simulate the corresponding situation in the simulation environment, and the running trajectory within a period of time in the future is predicted in the simulation environment, and then the predicted running trajectory is fed back to the real scene, so as to realize the trajectory prediction of the tower crane object based on the simulation environment.

[0053] Specifically, in step S11, the multi-dimensional posture data of the tower crane object in the real scene is identified, and the actual movement speed of the tower crane object is collected through the sensor installed on the tower crane.

[0054] The real scene is the scene where the actual tower crane is located, such as a certain operation site. The tower crane object is an object loaded on the tower crane that needs to be lifted and carried, such as a certain cargo or building materials. The multi-dimensional posture data includes posture data of the tower crane object in multiple dimensions, so that the running trajectory can be predicted more accurately. In this application, six-dimensional font data is used. Based on the coordinate system of the camera on the tower crane, the tower crane object has a description of translation and rotation vectors, where translation includes three degrees of freedom: front and back, up and down, and left and right. The rotation includes three sets of Euler angles: roll, pitch, and yaw, thereby obtaining the six-dimensional posture data. Multiple types of sensors are installed on the tower crane, so that the actual movement speed of the tower crane object can be collected through the sensors.

[0055] Step S12, the multi-dimensional posture data and the actual movement speed are transmitted to the simulation environment, and the current speed and direction of the tower crane object are displayed using the same posture.

[0056] The obtained multi-dimensional posture data and actual movement speed are transmitted to the simulation environment, based on which the tower crane information in the real physical scene is simulated and the real-time parameters are calculated. During the simulation, the tower crane and tower crane object are displayed in the same posture as in the real scene, and the current speed and direction of the same tower crane object are displayed to achieve a complete simulation, so that the predictions made in the simulation environment can also be truly fed back to the real environment to obtain accurate trajectory prediction results.

[0057] Step S13, predicting the running trajectory of the tower crane object according to the current speed and direction of the tower crane object through the physical engine built in the simulation environment; Step S14, displaying the predicted running trajectory in the simulation environment and feeding it back to the real scene.

[0058] A physics engine is built in the simulation environment. The physics engine is a real-time parameter calculation system that simulates the real physical scene through physical modeling. The running trajectory of the object is predicted in the physics engine according to the current speed and direction of the object in the real environment that has been input. The trajectory is then displayed in real time through the built physics engine and fed back to the real scene, and the real tower crane and object conditions are fed back through simulation parameters.

[0059] In one embodiment of the present application, in step S11, semantic segmentation is performed on the hook and object images captured by the camera installed on the tower crane in the real scene to obtain an RGB object image, and the RGB object image is converted into a depth image; point cloud data is extracted from the depth image, wherein the point cloud data includes color features of each RGB pixel and depth features of the point cloud points; the point cloud data are fused to obtain multiple groups of fused features; training calculations are performed based on the multiple groups of fused features to obtain multi-dimensional posture data of the tower crane object in the real scene.

[0060] Based on the six-dimensional posture estimation algorithm of deep vision, the hook and object images collected by the tower crane camera can be semantically segmented, and the segmented RGB object image (such as the cargo image) can be converted into a depth image to obtain RGB-D point cloud data. The color features of each RGB pixel and the depth features of the point cloud point are densely fused to obtain fused features. The posture is predicted based on the fused features, and multiple groups of fused features are trained to calculate the confidence. After optimizing the loss function, the final result is obtained.

[0061] In one embodiment of the present application, in step S11, the speed and acceleration of the tower crane object are collected by the IMU sensor installed on the tower crane, the speed and acceleration are calculated, and the geographic coordinates of the tower crane object collected by the positioning sensor are calculated; the actual movement speed of the tower crane object at each geographic coordinate is determined based on the calculated speed, acceleration and geographic coordinates.

[0062] The IMU sensor is installed on the tower crane to calculate the speed and acceleration, and the positioning sensor (GPS) is used to calculate the absolute geographic coordinates. The calculated data is transmitted to the simulation environment in real time and displayed. The world coordinate axis is always displayed with the object as the center and changes with the object's posture. Among them, when using IMU for calculation, the navigation result is obtained from the original angular velocity collected by the IMU through the gyroscope and the acceleration data calculated by the accelerometer. The navigation result includes position, speed, posture, etc. GPS also calculates the longitude and latitude and other information through the internal algorithm. The directly collected data is acceleration and angular velocity. The required data can be calculated by integration and other methods, such as "angular velocity->attitude", "acceleration->speed", "speed->position". According to the calculated speed, acceleration and geographic coordinates, the actual movement speed of the tower crane object at each geographic coordinate is determined. After obtaining the actual movement speed, it is fused and displayed as a combined rate. For example, the six-dimensional posture is vector-fused to obtain the synthesized bus speed and total angular velocity, and they are modulo to obtain the combined rate.

[0063] Specifically, in the simulation environment, the world coordinate axis is displayed with the crane as the center and changes with the posture of the crane; the predicted running trajectory is displayed with a marking line, the combined speed is displayed with a preset font, and the predicted speed is marked with a direction marking line, wherein the combined speed is determined by the multi-dimensional posture data of the crane. Here, the world coordinate axis is always displayed with the object as the center and changes with the posture of the object, such as Figure 2 As shown, the historical motion trajectory can be displayed with a dotted line, the combined velocity is represented by a preset font (such as yellow font) and changes in real time, and the direction mark line (such as an arrow) indicates the predicted velocity direction. The object in the simulation environment is always consistent with the actual object posture.

[0064] In one embodiment of the present application, in step S12, the real-time parameters of the tower crane object in the simulated real scene are calculated by the physical engine built in the simulation environment; the current speed and direction of the tower crane object and the real-time parameters are displayed using the same posture to complete the simulation of the real scene.

[0065] When making predictions in a simulation environment, a physical engine is built in the simulation environment. The physical engine is a real-time parameter calculation system that simulates real physical scenes through physical modeling, such as the gravity of objects, air resistance, atmospheric pressure, the physical structure of the tower crane itself, and the tension of the lifting rope, etc., so that the physical engine can calculate the speed, height, distance, air humidity, temperature, wind force and other simulation parameters of the object at future moments according to the current speed and direction of the tower crane object, and obtain a simulated predicted running trajectory of the tower crane object, so as to feed back the predicted running trajectory to the real tower crane and display it in the real environment for the convenience of the operator; at the same time, the obtained simulation parameters and predicted running trajectory are also displayed in the physical engine. Figure 3 As shown in the figure, using the physics engine, the hook rope moves and causes the object to shake. After adding these factors, the physics engine continuously calculates the object's speed value and the direction of movement at the next moment, and introduces some parameters in the real scene into the simulation environment for calculation to obtain more accurate prediction results. Here, the object in the simulation environment is always consistent with the actual object posture.

[0066] In one embodiment of the present application, in step S13, a 3D object space kinematic model is performed according to the current speed and direction of the crane object through the physical engine built in the simulation environment to obtain a motion model; the state of the crane object is predicted according to the motion model, and the state is matched with the observation result to obtain an observation model, and the state of the crane object is corrected using the observation model; the motion model and the observation model are processed based on the unscented Kalman filter to obtain a predicted running trajectory of the crane object. Here, Figure 4 As shown, after the six-dimensional posture data is obtained by visual posture estimation, the three-dimensional object space kinematic modeling is performed based on the real scene simulation of the physical engine. After modeling, the motion equation and observation equation are obtained. The motion equation and observation equation are processed based on the trajectory of the unscented Kalman filter to obtain the trajectory prediction. Specifically, the three-dimensional object space kinematic modeling is performed in the physical engine according to the current speed and direction of the tower crane object to obtain the motion model, which is:

[0067] p(x t |u t , x t-1 )

[0068] The motion model describes the state x t-1 Execute motion control instructions t After that, the current state x t The posterior distribution of .

[0069] u t =(v t ωt ) T

[0070] That is, the speed control command of the motor, the linear velocity and the angular velocity of rotation. Since the actual speed in real motion is different from the given speed, it is represented as a random variable with a mean of 0 and a finite variance:

[0071]

[0072] Here, ε b 2 It means the mean is 0 and the variance is b 2 The random error can be a normal distribution or a triangular distribution. 1 ~α 4 is a specific error parameter. Therefore, the true state after Δt is the motion after superimposing the noise Instead of the given speed (υω) T , and a certain rotation error is generated when reaching the final posture The error is also a random variable with mean 0:

[0073]

[0074] Therefore, the final complete motion model (state transfer function) is:

[0075]

[0076] It represents the change from state (x, y, θ) to (x', y', θ'), which is represented by the state X at two moments. t , X t-1 Computing Real Noisy Motion The specific value of u is determined by the given control t (υω) T The probability distribution of motion can be obtained, so the posterior probability of a certain state of the target object can be calculated.

[0077] The object state is corrected based on the degree of match between the object state predicted by the motion model and the observed result. The complete definition is:

[0078] p(z t |x t , m)

[0079] Among them, Z t is the observed value at time t, and m is the planned path.

[0080] The state of the tower crane is predicted based on the obtained motion model, and the state of the tower crane is corrected through the observation model to perform trajectory prediction.

[0081] Continuing with the above embodiment, the new Gaussian distribution formed by the motion model and the observation model after the posture of the tower crane object is updated is determined; the new Gaussian distribution is processed based on the unscented Kalman filter. Here, the new Gaussian distribution (probability distribution) formed by the motion equation (state transfer function) and the observation equation of the object after the posture is updated is determined, and the new Gaussian distribution is calculated after the unscented Kalman filter to predict the new state of the object, thereby obtaining the predicted trajectory.

[0082] Specifically, the physical engine built in the simulation environment calculates two frames before and after the preset time interval from the current moment, saves the coordinate sequence of the tower crane object in the historical n frames, and obtains the historical trajectory sequence; the historical trajectory sequence saved at each preset time interval is nonlinearly processed; the sigma point after the state transfer is transformed without trace through the sigma algorithm to obtain a new Gaussian distribution.

[0083] Take two frames before and after the current moment with a time interval of Δt, and save the coordinate sequence of the object in the historical n frames (for example, if n=10, the historical traces will be automatically cleared after n*Δt); perform nonlinear processing on the historical trajectory sequence saved every Δt, and then use the sigma point algorithm to perform a traceless transformation on the sigma point after the state transfer, so as to predict the new variance and covariance. In addition, according to the kinematic discrete model equation of the IMU, the Euler method is used to calculate the posture at the next moment, where q is a quaternion, which is another form of Euler angle. It is a calculation method generated to solve the universal lock problem, V is the speed, P is the position, w represents the world coordinate system, and b represents the object carrier coordinate system, so as to calculate the predicted position for error correction, as shown in the following formula:

[0084]

[0085]

[0086]

[0087] Where a and ω are the measured values ​​of acceleration and angular velocity, is the direct product of the matrix, and the updated acceleration and angular velocity at time k+1 are as follows:

[0088]

[0089]

[0090] like Figure 5As shown, A is the historical trajectory point of the object obtained by the sensor (indicated by red in the actual display), B is the historical trajectory (indicated by orange in the actual display), and C is the predicted trajectory after unit Δt (indicated by green in the actual display). The vector of the next frame and the current frame is used as the predicted velocity vector and is represented by an arrow in the simulation environment.

[0091] Assume that there is a Northeast Celestial coordinate system and a carrier coordinate system. Rotate the Northeast Celestial coordinate system three times so that the final coordinate system coincides with the carrier coordinate system. The rotation is counterclockwise in the order of ZXY, corresponding to the Euler angle Yaw-Pitch-Roll. The rotation matrix is ​​obtained after three rotations:

[0092]

[0093] Where ψ, θ, and φ are the rotation angles around the z, x, and y axes, respectively, which can be obtained through monitoring in the simulation environment. The three-axis angular velocity ωb (yaw, pitch, roll) of the carrier coordinate system (body system) can be expressed as follows:

[0094]

[0095] The three-axis rotation angle can be measured by the IMU sensor, and the total three-axis attitude change rate dv / dt can be obtained by inverting the matrix:

[0096]

[0097] In one embodiment of the present application, after predicting the running trajectory of the tower crane object according to the current speed and direction of the tower crane object through the physical engine built in the simulation environment, the predicted running trajectory of the tower crane object is calibrated using Euler angles. Here, an unscented Kalman filter fitting system is used to obtain a Gaussian distribution, thereby predicting the trajectory, and using Euler angles to calibrate the calibration error. Specifically, the step of unscented transformation includes the following process:

[0098] Step 1: Sample the original Gaussian distribution that the motion model obeys. When the observation model is used to correct the predicted state of the motion model, the sigma point after the correction is sampled; Step 2: Perform nonlinear transformation on the sampling points; Step 3: Assign weights and weighted statistical results. That is, linearize the strongly nonlinear system, so that the predicted probability distribution can also satisfy Gaussian. Let the nonlinear transformation be g(). For the sampling point set χ [i] , and obtain the following statistical results:

[0099]

[0100]

[0101] That is, the mean and variance are updated, and then the Kalman filter is introduced to predict the final probability statistics.

[0102] Through the above method, the trajectory of the tower crane object can be predicted based on real simulation. The object in the simulation environment and the real environment maintains the same posture for real-time display, and the parameter algorithm provided by the built physical engine is used to calculate the running trajectory in the future in real time, and finally fed back to the real scene. The prediction result is more accurate and can display more actual scenes.

[0103] In addition, an embodiment of the present application also provides a computer-readable medium on which computer-readable instructions are stored. The computer-readable instructions can be executed by a processor to implement the aforementioned method for predicting the trajectory of a tower crane object.

[0104] In one embodiment of the present application, a device for predicting the trajectory of a tower crane object is also provided, the device comprising:

[0105] one or more processors; and

[0106] A memory storing computer readable instructions, which, when executed, cause the processor to perform the operations of the method as described above.

[0107] For example, the computer readable instructions, when executed, cause the one or more processors to:

[0108] Identify the multi-dimensional posture data of the tower crane in the real scene, and collect the actual movement speed of the tower crane through the sensors installed on the tower crane;

[0109] The multi-dimensional posture data and the actual movement speed are transmitted into the simulation environment, and the current speed and direction of the tower crane object are displayed using the same posture;

[0110] Predicting the running trajectory of the tower crane object according to the current speed and direction of the tower crane object through a physical engine built in the simulation environment;

[0111] The predicted running trajectory is displayed in the simulation environment and fed back to the real scene.

[0112] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

[0113] It should be noted that the present application can be implemented in software and / or a combination of software and hardware, for example, can be implemented using an application specific integrated circuit (ASIC), a general purpose computer or any other similar hardware device. In one embodiment, the software program of the present application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of the present application (including relevant data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the present application can be implemented using hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0114] In addition, a part of the present application may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present application through the operation of the computer. The program instruction for calling the method of the present application may be stored in a fixed or removable recording medium, and / or transmitted through a data stream in a broadcast or other signal-bearing medium, and / or stored in a working memory of a computer device that runs according to the program instruction. Here, according to an embodiment of the present application, a device is included, the device including a memory for storing computer program instructions and a processor for executing program instructions, wherein, when the computer program instruction is executed by the processor, the device is triggered to run the method and / or technical solution based on the aforementioned multiple embodiments according to the present application.

[0115] It is obvious to those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and that the present application can be implemented in other specific forms without departing from the spirit or essential features of the present application. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present application. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural.

Claims

1. A method for predicting the trajectory of a tower crane, characterized in that: The method comprises: Identify the multi-dimensional posture data of the tower crane object in the real scene, and collect the actual movement speed of the tower crane object through the sensor installed on the tower crane, wherein the tower crane object is an object loaded on the tower crane that needs to be lifted and carried, and the multi-dimensional posture data includes the translation and rotation posture data of the tower crane object, wherein the translation includes front and back, up and down, left and right, and the rotation includes roll angle, pitch angle, and yaw angle; The multi-dimensional posture data and the actual movement speed are transmitted into the simulation environment, and the current speed and direction of the tower crane object are displayed using the same posture; Predicting the running trajectory of the tower crane object according to the current speed and direction of the tower crane object through a physical engine built in the simulation environment; Displaying the predicted running trajectory in the simulation environment and feeding it back to the real scene; Among them, the multi-dimensional posture data of the tower crane object in the real scene is recognized, including: Perform semantic segmentation on the hook and object images captured by the camera installed on the tower crane in the real scene to obtain an RGB object image, and convert the RGB object image into a depth map; Extracting point cloud data from the depth map, wherein the point cloud data includes color features of each RGB pixel and depth features of the point cloud points; Fusing the point cloud data to obtain multiple groups of fusion features; Perform training calculations based on the multiple groups of fusion features to obtain multi-dimensional posture data of the tower crane object in the real scene; Among them, the actual movement speed of the tower crane object is collected by the sensor installed on the tower crane, including: The speed and acceleration of the tower crane object are collected by the IMU sensor installed on the tower crane, the speed and acceleration are solved, and the geographic coordinates of the tower crane object collected by the positioning sensor are solved; Determine the actual movement speed of the crane object at each geographic coordinate based on the calculated speed, acceleration and geographic coordinates; Wherein, using the same posture to display the current speed and direction of the tower crane object includes: The real-time parameters of the tower crane object in the real scene are calculated and simulated by the physical engine built in the simulation environment, wherein the real-time parameters include the gravity of the tower crane object, air resistance, atmospheric pressure, the physical structure of the tower crane itself, and the pulling force and tension of the lifting rope; The current speed and direction of the tower crane object and real-time parameters are displayed using the same posture to complete the simulation of the real scene; The predicted running trajectory is displayed in the simulation environment and fed back to the real scene, including: In the simulation environment, a world coordinate axis is displayed with the tower crane object as the center and changes along with the posture of the tower crane object; The predicted running trajectory is displayed using marking lines, the combined rate is displayed in a preset font, and the predicted speed is marked in direction using direction marking lines, wherein the combined rate is determined by the multi-dimensional posture data of the tower crane object.

2. The method according to claim 1, characterized in that Predicting the running trajectory of the tower crane object according to the current speed and direction of the tower crane object through the physical engine built in the simulation environment, including: The physical engine built in the simulation environment performs three-dimensional object space kinematic modeling according to the current speed and direction of the tower crane object to obtain a motion model, wherein the motion model is: , The motion model describes the state x t-1 Execute motion control instructions After that, the current state x t The posterior distribution of , where , Indicates the speed control instruction of the motor. Indicates the line speed, Represents the angular velocity of rotation. The mean of the error between the actual velocity and the given velocity in real motion is set to 0, and the mean of the variance is set to 0. Then the random variable with finite variance is: ， Among them, here The mean is 0 and the variance is The random error, is a specific error parameter, using The actual state after superimposing noise Instead of a given speed , and a certain rotation error is generated when reaching the final posture , the error is a random variable with mean 0: ， The final complete motion model is: ， It represents the change from state (x, y, θ) to (x', y', θ'), which is represented by the state X at two moments. t , X t-1 Computing Real Noisy Motion The specific value of The state of the tower crane is predicted according to the motion model, the state is matched with the observation result to obtain an observation model, and the state of the tower crane is corrected using the observation model, wherein the observation model is: , where Z t is the observed value at time t, and m is the planned path; Determine a new Gaussian distribution formed by the motion model and the observation model after the position and posture of the tower crane object are updated, and process the new Gaussian distribution based on an unscented Kalman filter to obtain a predicted running trajectory of the tower crane object.

3. The method according to claim 2, characterized in that Determining a new Gaussian distribution of the motion model and the observation model after the position and posture of the tower crane object are updated, comprising: The physical engine built in the simulation environment calculates two frames before and after the preset time interval from the current moment, saves the coordinate sequence of the tower crane object in the historical n frames, and obtains the historical trajectory sequence; Perform nonlinear processing on the historical trajectory sequence saved at each preset time interval; The sigma point after state transfer is transformed untraceably through the sigma algorithm to obtain a new Gaussian distribution; The steps of the traceless transformation include the following processes: step 1, sampling the original Gaussian distribution obeyed by the motion model, when the predicted state of the motion model is corrected using the observation model, the sigma point after the corrected point has passed the state transfer is sampled; step 2, nonlinear transformation of the sampling points; step 3, assigning weights and weighted statistical results; Among them, let the nonlinear transformation be g(), for the sampling point set , and obtain the following statistical results: ， Among them, ω is the angular velocity. After updating the mean and variance, the Kalman filter is then introduced to predict the final probability statistics.

4. The method according to any one of claims 1 to 3, characterized in that After predicting the running trajectory of the tower crane object according to the current speed and direction of the tower crane object through the physical engine built in the simulation environment, it includes: Use Euler angles to calibrate the predicted trajectory of the tower crane; The calibration steps include the following process: Get the time interval from the current time The two frames before and after save the coordinate sequence of the object in the historical n frames; Every Nonlinear processing of historical trajectory sequences saved in time; The sigma point algorithm is used to perform an untraceable transformation on the sigma point after state transfer to predict the new variance and covariance; According to the kinematic discrete model equation of the IMU, the Euler method is used to calculate the posture at the next moment, using the following formula: ， Among them, q is a quaternion, which is a form of Euler angle, V is velocity, P is position, w represents the world coordinate system, b represents the object carrier coordinate system, a and ω are the acceleration and angular velocity measurement values, ⊗ is the direct product of the matrix, and the updated acceleration and angular velocity at time k+1 are as follows: 。 5. A device for predicting the trajectory of a tower crane, characterized in that: The device comprises: one or more processors; and A memory storing computer readable instructions which, when executed, cause the processor to perform the operations of the method as claimed in any one of claims 1 to 4.

6. A computer-readable medium having computer-readable instructions stored thereon, wherein the computer-readable instructions can be executed by a processor to implement the method according to any one of claims 1 to 4.

Images