Boom equipment and operation method thereof

Abstract

The invention belongs to the technical field of boom operation machinery, and provides boom equipment and an operation method thereof, and the operation method comprises the steps: in the movement process of a boom, sequentially obtaining a plurality of pictures of the environment around the boom with the boom as a visual angle; establishing a scene three-dimensional model around the boom according to the plurality of environment pictures; performing space coordinate system association on the established scene three-dimensional model and the equipment three-dimensional model of the boom equipment to obtain a space model; and controlling the boom to act according to the space model. By establishing the space model of the equipment and the surrounding environment of the equipment, the relative position between the boom and the surrounding reference object can be easily and accurately judged, so that the accurate operation of the boom is facilitated.

Description

Technical Field

[0001] The invention belongs to the technical field of boom operation machinery, and in particular relates to a boom device and an operation method thereof. Background Art

[0002] When a fire truck with a boom is performing high-altitude rescue work, the movement of the boom mainly depends on the firefighters' long-distance observation on the ground, who estimate the relative position between the working bucket and the target area through their eyes to determine the next operation of the boom. However, due to the poor field of vision and the influence of subjective factors, the distance is easily misjudged, resulting in the difficulty of accurately and quickly moving the boom to the appropriate working position. Especially when the weather visibility is low or there are obstacles between the firefighters and the boom, it will increase the difficulty of the firefighters' operation. Summary of the invention

[0003] In view of the above-mentioned defects or shortcomings, the present invention provides a boom equipment operating method and boom equipment, aiming to solve the technical problem that when the existing boom equipment with a boom is performing high-altitude operations, the relative position between the boom and the reference object is easily inaccurately determined.

[0004] To achieve the above object, the present invention provides a boom equipment operation method, wherein the boom equipment operation method comprises:

[0005] S100: during the movement of the boom, sequentially obtain a plurality of environment images around the boom from the perspective of the boom;

[0006] S200: Establish a three-dimensional model of the scene around the boom based on multiple environmental images;

[0007] S300: Associating the established scene three-dimensional model with the equipment three-dimensional model of the boom equipment in a spatial coordinate system to obtain a spatial model;

[0008] S400: Control the boom motion according to the space model.

[0009] In an embodiment of the present invention, step S200: establishing a three-dimensional model of the scene around the boom according to a plurality of environmental images, specifically includes:

[0010] S210: extracting feature points of the environment image;

[0011] S220: performing feature matching on feature points between different environment images, and performing geometric verification on the matched feature points;

[0012] S230: Convert each feature point that passes geometric verification into a three-dimensional coordinate according to the camera parameters when the corresponding feature point is photographed;

[0013] S240: Establishing a three-dimensional point cloud model according to all three-dimensional coordinates;

[0014] S250: Convert the 3D point cloud model into a scene 3D model.

[0015] In an embodiment of the present invention, after step S240: establishing a three-dimensional point cloud model according to all three-dimensional coordinates, the boom equipment operation method further includes:

[0016] S241: Iterating the 3D point cloud model according to the newly input environment picture;

[0017] S242: When the iterated three-dimensional point cloud model meets the preset conditions, output the iterated three-dimensional point cloud model and the optimal camera parameters corresponding to the output three-dimensional point cloud model.

[0018] In an embodiment of the present invention, step S250: converting the 3D point cloud model into a scene 3D model specifically includes:

[0019] S251: establishing a set of three-dimensional Gaussian function groups based on the output three-dimensional point cloud model, where the three-dimensional Gaussian function group includes multiple three-dimensional Gaussian functions;

[0020] S252: optimizing the basic description features of the 3D Gaussian body corresponding to each 3D Gaussian function using a stochastic gradient descent algorithm;

[0021] S253: Render each optimized 3D Gaussian body to obtain a three-dimensional model of the scene.

[0022] In an embodiment of the present invention, step S253: rendering each optimized 3D Gaussian volume to obtain a three-dimensional model of the scene specifically includes:

[0023] S253-1: Divide the screen into multiple squares, each square represents a tile;

[0024] S253-2: retain the 3D Gaussian bodies with confidence higher than the preset value within the camera's viewing angle, and exclude the rest;

[0025] S253-3: assigning a keyword combining the view space depth and the tile ID to each 3D Gaussian body in the retained 3D Gaussian bodies according to the number of tiles overlapped by each 3D Gaussian body;

[0026] S253-4: Sort the retained 3D Gaussian volumes according to the keywords;

[0027] S253-5: Load each tile into memory for parallel rendering to obtain a three-dimensional model of the scene.

[0028] In the embodiment of the present invention, the basic description features of the 3D Gaussian volume include the center point position, opacity, and covariance.

[0029] In an embodiment of the present invention, step S400: controlling the boom motion according to the space model specifically includes:

[0030] S410: Planning an optimal motion path for the boom to move to the target area according to the space model;

[0031] S420: Control the arm motion according to the optimal motion path.

[0032] In an embodiment of the present invention, step S100: during the boom movement, sequentially acquiring a plurality of pictures of the environment around the boom from the perspective of the boom, specifically including:

[0033] S110: During the movement of the boom, the boom is used as a viewing angle to capture images of the area around the boom;

[0034] S120: extracting frames from the captured video to obtain picture frames;

[0035] S130: performing unified specification processing on the extracted picture frames to make the resolution and size of each picture frame uniform, and the processed picture frames are environment pictures.

[0036] To achieve the above-mentioned purpose, the present invention also provides a boom device, which is used to execute the boom device operation method described above, wherein the boom device includes a boom mechanism, an image acquisition mechanism, a graphics calculation mechanism and a display mechanism, the image acquisition mechanism is arranged on the boom mechanism and is used to photograph the surrounding environment of the boom, the graphics calculation mechanism is used to establish a spatial model based on the shooting data of the first shooting element and the second shooting element, and the display mechanism is used to display the spatial model.

[0037] In an embodiment of the present invention, the arm mechanism includes a first arm and a second arm, the first arm includes at least one arm section, and the second arm is the end arm section of the arm mechanism; the image acquisition mechanism includes a first shooting element arranged on the first arm and a second shooting element arranged on the second arm.

[0038] Through the above technical solution, the boom equipment operation method provided by the embodiment of the present invention has the following beneficial effects:

[0039] When the boom equipment is in operation, the boom needs to perform amplitude change, extension and rotation movements at high altitude. This method obtains the environmental picture around the boom from the perspective of the boom, so that the environmental picture can more clearly reflect the situation of the reference objects (buildings or obstructions) around the boom, and effectively avoids misjudgment by the human eye due to the buildings or obstructions around the boom being too far from the ground or being blocked by haze.

[0040] After obtaining a clear picture of the environment, the 2D-3D modeling technology can be used to establish a three-dimensional model of the scene around the boom. Combined with the camera parameters when taking the environmental picture, the coordinate relationship between the established three-dimensional model of the scene and the shooting element can be determined. Based on this relationship, it can be converted into a coordinate relationship between the three-dimensional model of the scene and the three-dimensional model of the equipment of the boom equipment, thereby obtaining a spatial model that includes the boom equipment and the environment around the boom equipment. Based on this model, the operator can easily judge the situation of the boom and the surrounding reference objects, and can accurately judge the position of the boom relative to the surrounding reference objects, thereby facilitating the operator to move the boom to the specified position accurately and quickly.

[0041] Other features and advantages of the present invention will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings are used to provide an understanding of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present invention, but do not constitute a limitation of the present invention. In the accompanying drawings:

[0043] Figure 1 is a general step diagram of a boom equipment operation method according to an embodiment of the present invention;

[0044] Figure 2 is a specific step diagram of step S200 according to an embodiment of the present invention;

[0045] Figure 3 is a specific flow chart of step S240 according to an embodiment of the present invention;

[0046] Figure 4 is a specific flow chart of step S250 according to an embodiment of the present invention;

[0047] Figure 5 is a specific flow chart of step S253 according to an embodiment of the present invention;

[0048] Figure 6 is a specific flow chart of step S400 according to an embodiment of the present invention;

[0049] Figure 7 is a specific flow chart of step S100 according to an embodiment of the present invention;

[0050] Figure 8 is a simplified schematic diagram of a boom device according to an embodiment of the present invention.

[0051] Description of Reference Numerals

[0052] 1. Boom equipment; 2. Boom mechanism; 21. First boom; 22. Second boom; 3. Image acquisition mechanism; 31. First shooting element; 32. Second shooting element; 4. Graphic calculation mechanism; 5. Display mechanism. DETAILED DESCRIPTION

[0053] The specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0054] The boom equipment operation method of the present invention will be described below with reference to the accompanying drawings.

[0055] The present invention provides a boom equipment operation method, such as Figure 1 As shown, the boom equipment operation method includes:

[0056] S100: during the movement of the boom, sequentially obtain a plurality of environment images around the boom from the perspective of the boom;

[0057] S200: Establish a three-dimensional model of the scene around the boom based on multiple environmental images;

[0058] S300: Associating the established scene three-dimensional model with the equipment three-dimensional model of the boom equipment 1 in a spatial coordinate system to obtain a spatial model;

[0059] S400: Control the boom motion according to the space model.

[0060] When the boom equipment 1 is in operation, the boom needs to perform movements such as amplitude change, extension and retraction, and rotation at high altitude. The method obtains the environmental picture around the boom from the perspective of the boom, so that the environmental picture can more clearly reflect the situation of the reference objects (buildings or obstructions) around the boom, and effectively avoids misjudgment by the human eye due to the buildings or obstructions around the boom being too far from the ground or being blocked by haze.

[0061] After obtaining a clear picture of the environment, the 2D-3D modeling technology can be used to establish a three-dimensional model of the scene around the boom. Combined with the camera parameters when taking the environmental picture, the coordinate relationship between the established three-dimensional model of the scene and the shooting element can be determined. Based on this relationship, it can be converted into a coordinate relationship between the three-dimensional model of the scene and the three-dimensional model of the equipment of the boom equipment 1, and then a spatial model including the boom equipment 1 and the surrounding environment of the boom equipment 1 is obtained. Based on this model, the operator can easily judge the situation of the boom and the surrounding reference objects, and can relatively accurately judge the position of the boom relative to the surrounding reference objects, thereby facilitating the operator to move the boom to the specified position accurately and quickly.

[0062] In the embodiment of the present invention, a three-dimensional reconstruction algorithm can be used to achieve modeling from 2D to 3D, such as Figure 2 As shown, step S200: establishing a three-dimensional model of the scene around the boom according to a plurality of environmental images may specifically include:

[0063] S210: extracting feature points of the environment image;

[0064] S220: performing feature matching on feature points between different environment images, and performing geometric verification on the matched feature points;

[0065] S230: Convert each feature point that passes geometric verification into a three-dimensional coordinate according to the camera parameters when the corresponding feature point is photographed;

[0066] S240: Establishing a three-dimensional point cloud model according to all three-dimensional coordinates;

[0067] S250: Convert the 3D point cloud model into a scene 3D model.

[0068] When the environment picture is input into the graphics calculation mechanism 4 for modeling, the graphics calculation mechanism 4 will first detect the feature points of the input environment picture and find the feature points. These feature points can be more prominent points such as corners, edges, textures, etc. in the picture. After finding the feature points, feature matching is performed on the feature points between different pictures according to the preset feature matching algorithm to determine the correspondence between different images. After feature matching, the matched feature points need to be geometrically verified to eliminate those matching points that do not meet the geometric constraints, thereby improving the accuracy and robustness of the matching. Commonly used geometric verification algorithms include the RANSAC algorithm (random sampling consistency algorithm) and the LMedS algorithm (minimum median method algorithm).

[0069] After enough matching points have passed the geometric verification, a 3D point cloud model can be established based on the verified feature points. When establishing a 3D point cloud model, the verified feature points must first be converted into 3D coordinates. This step can use the internal and external parameters of the camera corresponding to the current feature points to calculate the conversion through triangulation. Among them, camera parameters refer to some parameters of the camera when taking pictures of the environment, such as focal length, principal point, pixel size, distortion coefficient, position of the shooting element in the world coordinate system, shooting direction, etc.

[0070] After converting each verified matching feature point into three-dimensional coordinates, a three-dimensional point cloud model can be obtained by performing spatial modeling based on the set of coordinates. After obtaining the three-dimensional point cloud model, a contour is added to the surface of the three-dimensional point cloud model to obtain a specific three-dimensional model of the scene.

[0071] In an embodiment of the present invention, Figure 3As shown, step S240: after establishing a three-dimensional point cloud model according to all three-dimensional coordinates, the boom equipment operation method further includes:

[0072] S241: Iterating the 3D point cloud model according to the newly input environment picture;

[0073] S242: When the iterated three-dimensional point cloud model meets the preset conditions, output the iterated three-dimensional point cloud model and the optimal camera parameters corresponding to the output three-dimensional point cloud model.

[0074] During the movement of the arm, the image acquisition mechanism 3 will continuously generate the latest environmental pictures, that is, the graphics calculation mechanism 4 will continuously have new environmental pictures input. After obtaining the newly input environmental pictures, the graphics calculation mechanism 4 will successively perform feature point extraction, feature point matching, geometric verification, convert the feature points into three-dimensional coordinates, and add the three-dimensional coordinates to the three-dimensional point cloud model. That is, through steps S241 and S242, the three-dimensional point cloud model can be continuously iterated, thereby ensuring that the subsequently generated three-dimensional model of the environment is more in line with the actual situation.

[0075] In the embodiment of the present invention, the output 3D point cloud model corresponds to the optimal camera parameters, the purpose of which is to improve the consistency between the subsequently rendered model and the actual scene.

[0076] The essence of model rendering is to add a surface to the 3D point cloud model, that is, the rendering process from the 3D point cloud model to the scene 3D model is actually the process of adding a surface to the 3D point cloud model. Whether the surface contour is added appropriately directly affects the quality of the 3D model.

[0077] In an embodiment of the present invention, a Gaussian splash rendering method may be preferably used to render the surface contour of the three-dimensional point cloud model. Figure 4 As shown, the Gaussian splash rendering method specifically includes:

[0078] S251: establishing a set of three-dimensional Gaussian function groups based on the output three-dimensional point cloud model, where the three-dimensional Gaussian function group includes a plurality of three-dimensional Gaussian functions;

[0079] S252: optimizing the basic description features of the 3D Gaussian body corresponding to each 3D Gaussian function using a stochastic gradient descent algorithm;

[0080] S253: Render each optimized 3D Gaussian body to obtain a three-dimensional model of the scene.

[0081] In step S251, the three-dimensional Gaussian function group refers to the set of all three-dimensional Gaussian functions in this rendering. Each three-dimensional Gaussian function can generate a 3D Gaussian body, and a three-dimensional Gaussian function can correspond to one feature point or a group of feature points. The basic description features of the 3D Gaussian body include the center point position, opacity, covariance, etc.

[0082] After establishing a three-dimensional Gaussian function for each (or each group of) feature points in the three-dimensional point cloud model, the basic description features of the 3D Gaussian body corresponding to each three-dimensional Gaussian function are optimized by using the stochastic gradient descent algorithm, so that a high-quality three-dimensional model can be obtained in the rendering stage, which can more accurately reflect the characteristics and structure of the input data.

[0083] like Figure 5 As shown, in an embodiment of the present invention, step S253: rendering each optimized 3D Gaussian volume to obtain a three-dimensional model of the scene specifically includes:

[0084] S253-1: Divide the screen into multiple squares, each square represents a tile;

[0085] S253-2: retain the 3D Gaussian bodies with confidence higher than the preset value within the camera's viewing angle, and exclude the rest;

[0086] S253-3: assigning a keyword combining the view space depth and the tile ID to each 3D Gaussian body in the retained 3D Gaussian bodies according to the number of tiles overlapped by each 3D Gaussian body;

[0087] S253-4: Sort the retained 3D Gaussian volumes according to the keywords;

[0088] S253-5: Load each tile into memory for parallel rendering to obtain a three-dimensional model of the scene.

[0089] The purpose of rendering is to make the content displayed on the screen more realistic. An important step in the rendering process is called rasterization, which is the process of converting a 3D model into a 2D image that can be displayed on the screen. Through rasterization, the intuitiveness and ease of processing of a two-dimensional plane can be used to process data in three-dimensional space, while meeting the needs of display, interaction, storage and processing.

[0090] In order to achieve fast and differentiable rasterization, a tile-based rasterizer can be designed. The rasterizer divides the screen into multiple blocks, each of which is a tile. The 3D Gaussian bodies with confidence higher than the preset value within the camera's viewing angle are retained, and the rest of the 3D Gaussian bodies are excluded. In the screened three-dimensional Gaussian functions (3D Gaussian bodies), a keyword combining the view space depth and the tile ID is assigned to each 3D Gaussian body according to the number of tiles they overlap, and then the retained 3D Gaussian bodies are sorted according to the keyword using a quick sorting algorithm. After sorting, we start a thread block for each tile. Each thread block will collaboratively load the Gaussian function package into shared memory and render the loaded 3D Gaussian body in parallel on a per-tile basis, thereby achieving rapid model building. Parallel rendering refers to the simultaneous rendering of multiple tiles.

[0091] In an embodiment of the present invention, the expression of the three-dimensional Gaussian function established with point μ (the center point position of the 3D Gaussian body) as the center in the world space is:

[0092]

[0093] in, Describes the degree of deviation of point x from the center of the Gaussian distribution. Here, x is a three-dimensional vector, Σ is the covariance matrix, (x) T is the transpose of x.

[0094] When optimizing the basic descriptive features, The loss function of the optimization process is given by loss and structural similarity loss The value of λ is generally 0.2, and its calculation expression is as follows:

[0095]

[0096] like Figure 6 As shown, in an embodiment of the present invention, step S400: controlling the boom motion according to the space model specifically includes:

[0097] S410: Planning an optimal motion path for the boom to move to the target area according to the space model;

[0098] S420: Control the arm motion according to the optimal motion path.

[0099] This method can realize one-button automatic control of the boom to reach the target area. After the spatial model is established, the graphic calculation mechanism 4 can analyze the relative position between the target area and the boom in the model and the size of the target area. The graphic calculation mechanism 4 plans the path based on the motion parameters of the boom, so that the boom can move quickly and accurately to the target area.

[0100] like Figure 7 As shown, in an embodiment of the present invention, step S100: during the movement of the boom, a plurality of pictures of the environment around the boom from the perspective of the boom are sequentially acquired, specifically including:

[0101] S110: During the movement of the boom, the boom is used as a viewing angle to capture images of the area around the boom;

[0102] S120: extracting frames from the captured video to obtain picture frames;

[0103] S130: performing unified specification processing on the extracted picture frames to make the resolution and size of each picture frame uniform, and the processed picture frames are environment pictures.

[0104] The environmental image in this method comes from the video data taken by the shooting element. The video data can reflect the dynamic changes of the surrounding environment and the changes of the camera posture during the movement of the arm. The environmental image is extracted by frame extraction, which can ensure that the time interval between the image frames input sequentially is short, thereby ensuring a large overlap between adjacent environmental images, facilitating subsequent feature point matching, and ensuring the quality of reconstruction of the three-dimensional point cloud model.

[0105] like Figure 8 As shown, to achieve the above purpose, the present invention also provides a boom device 1, which is used to perform the boom device operation method described above, wherein the boom device 1 includes a boom mechanism 2, an image acquisition mechanism 3, a graphics calculation mechanism 4 and a display mechanism 5, the image acquisition mechanism 3 is arranged on the boom mechanism 2 and is used to shoot the surrounding environment of the boom, the graphics calculation mechanism 4 is used to establish a space model according to the shooting data of the first shooting element 31 and the second shooting element 32, and the display mechanism 5 is used to display the space model. The boom device 1 can be a fire truck with a boom, a crane, a boom-type aerial work vehicle, etc.

[0106] like Figure 8 As shown, in an embodiment of the present invention, the arm mechanism 2 includes a first arm 21 and a second arm 22, the first arm 21 includes at least one arm section, and the second arm 22 is the end arm section of the arm mechanism 2; the image acquisition mechanism 3 includes a first shooting element 31 arranged on the first arm 21 and a second shooting element 32 arranged on the second arm 22.

[0107] When the arm mechanism 2 of the fire truck is extended, the second arm 22 represents the last arm section, and the first arm 21 includes other arm sections except the last arm section. For example, when the arm mechanism 2 of the fire truck is composed of three arms, the first arm 21 includes the first arm and the second arm, and the second arm 22 is a three-arm.

[0108] In an embodiment of the present invention, when arranging the installation position of the image acquisition mechanism 3, it is necessary to set at least one first shooting element 31 on at least one of the first arm and the second arm, and set at least one second shooting element 32 on the third arm, so as to capture multi-perspective images of the environment around the arm as much as possible.

[0109] The graphic calculation mechanism 4 can calculate the corresponding three-dimensional point cloud model and render the model according to the input image, and can also calculate the best moving path of the boom according to the spatial model. The display mechanism 5 is used to display the spatial model and the position of the boom in the model in real time.

[0110] The embodiments of the present invention are described below in conjunction with the specific operation process of a fire truck with a boom.

[0111] When the fire truck is working, the image acquisition mechanism 3 will take pictures of the fire truck and the surrounding environment of the boom. For example, a rotatable camera is set on the boom section close to the ground to capture the video of the surrounding environment of the fire truck. Alternatively, a camera is set on the middle boom section to take a video of the surrounding environment near the middle height of the boom, and a camera is set at the end of the second boom 22 to take a video of the surrounding environment from the perspective of the top of the boom.

[0112] After capturing a video of the surrounding environment, the video is subjected to frame extraction processing to obtain image frames, and then the image frames are processed to a uniform size and resolution to obtain an environment image that can be used as input data.

[0113] After the environment image is input to the graphics computing unit 4, the content of the input environment image is converted into a 3D point cloud model using a 3D reconstruction algorithm, and then the 3D point cloud model is rendered to obtain a 3D model of the scene. The 3D point cloud model contains the color and spatial information of each point that constitutes the 3D object.

[0114] Combined with the pre-established three-dimensional model of the fire truck, the spatial coordinate system of the fire truck and the surrounding environment is unified in the digital space, so that the established three-dimensional model of the scene is associated with the three-dimensional model of the equipment of the boom equipment 1 in the spatial coordinate system to obtain a spatial model.

[0115] The optimal working position of the target area is automatically generated according to the established spatial model. The path is automatically planned by analyzing the relative position relationship between the optimal working position and the working bucket. Finally, the boom is controlled to extend, extend, change the amplitude and rotate according to the planned route to achieve rapid movement of the boom to the target position.

[0116] In order to make the 3D reconstruction result more similar to the real scene, the Gaussian splash rendering method can be used to render the 3D point cloud model. Specifically, this method uses the points of the point cloud models of each dimension in the point cloud as the center point, creates a series of 3D Gaussian functions, and then uses the random gradient descent algorithm to optimize the color, opacity and covariance parameters of the 3D Gaussian function. Finally, the rendered result is input to the display screen for display, and a 3D scene model that restores the real world is obtained.

[0117] In summary, the boom equipment operation method of the present invention uses a camera to capture the surrounding environment information of the boom in real time and establish a real-time 3D model, which can assist the high-altitude operation of the equipment, thereby reducing the operating errors caused by the operator due to the long observation distance and subjective judgment errors. At the same time, it can also greatly reduce the impact of adverse factors such as haze obstruction and obstacle obstruction on the operator.

[0118] At the same time, the established spatial model contains rich spatial features of the target area. By analyzing the relative position and size of the boom and the target area in the model and combining it with the pre-established fire truck model, it is possible to automatically control the boom to the optimal working position with one click, greatly improving work efficiency.

[0119] Furthermore, the 3D Gaussian function is introduced to achieve flexible and more vivid scene representation, restore the real three-dimensional world for operators, improve the readability of 3D models for operators, and thus optimize the operator's operating experience.

[0120] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0121] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection, an electrical connection, or communication with each other; it can be a direct connection, or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0122] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0123] Although the embodiments of the present invention have been described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A boom equipment operation method, characterized in that: The boom equipment operation method comprises: During the movement of the boom, a plurality of pictures of the environment around the boom from the perspective of the boom are sequentially acquired; Establishing a three-dimensional model of the scene around the boom according to the plurality of environmental images; Associating the established scene three-dimensional model with the equipment three-dimensional model of the boom equipment (1) in a spatial coordinate system to obtain a spatial model; The boom movement is controlled according to the space model.

2. The boom equipment operation method according to claim 1, characterized in that: The three-dimensional model of the scene around the boom is established according to the plurality of environmental images, specifically including: Extracting feature points of the environment picture; Performing feature matching on the feature points between different environment pictures, and performing geometric verification on the matched feature points; Convert each of the feature points that pass the geometric verification into three-dimensional coordinates according to the camera parameters when photographing the corresponding feature points; Establishing a three-dimensional point cloud model according to all the three-dimensional coordinates; The three-dimensional point cloud model is converted into the scene three-dimensional model.

3. The boom equipment operation method according to claim 2, characterized in that: After the three-dimensional point cloud model is established according to all the three-dimensional coordinates, the boom equipment operation method further includes: Iterating the three-dimensional point cloud model according to the newly input environment picture; When the iterated three-dimensional point cloud model meets the preset conditions, the iterated three-dimensional point cloud model and the optimal camera parameters corresponding to the output three-dimensional point cloud model are output.

4. The boom equipment operation method according to claim 2, characterized in that: Converting the three-dimensional point cloud model into the three-dimensional model of the scene specifically includes: Based on the output three-dimensional point cloud model, a set of three-dimensional Gaussian function groups is established, wherein the three-dimensional Gaussian function group includes a plurality of three-dimensional Gaussian functions; A stochastic gradient descent algorithm is used to optimize the basic description features of the 3D Gaussian body corresponding to each of the three-dimensional Gaussian functions; Rendering each of the optimized 3D Gaussian volumes to obtain the three-dimensional model of the scene; Each of the three-dimensional Gaussian functions corresponds to a feature point or a group of feature points.

5. The boom equipment operation method according to claim 4, characterized in that: Rendering each of the optimized 3D Gaussian volumes to obtain the three-dimensional model of the scene specifically includes: Dividing the screen into a plurality of squares, each of the squares being defined as a tile; The 3D Gaussian bodies with confidence values ​​higher than a preset value within the camera viewing angle are retained, and the other 3D Gaussian bodies are excluded; Among the retained 3D Gaussian bodies, assigning a keyword combining a view space depth and a tile ID to each of the 3D Gaussian bodies according to the number of tiles overlapped by each of the 3D Gaussian bodies; sorting the retained 3D Gaussian bodies according to the keywords; Each of the tiles is loaded into memory for parallel rendering to obtain the three-dimensional model of the scene.

6. The boom equipment operation method according to claim 4, characterized in that: The basic description features of the 3D Gaussian body include center point position, opacity, and covariance.

7. The boom equipment operation method according to any one of claims 1 to 6, characterized in that: Controlling the boom motion according to the space model specifically includes: Planning the optimal motion path for the arm to move to the target area according to the space model; The arm movement is controlled according to the optimal motion path.

8. The boom equipment operation method according to any one of claims 1 to 6, characterized in that: During the movement of the boom, a plurality of pictures of the environment around the boom from the perspective of the boom are sequentially obtained, specifically including: During the movement of the arm, taking the arm as a viewing angle, photographing the area around the arm; Perform frame extraction on the video to obtain picture frames; The extracted picture frames are processed according to uniform specifications so that the resolution and size of each picture frame are uniform, and the processed picture frames are the environment pictures.

9. A boom equipment, used to perform the boom equipment operation method according to any one of claims 1 to 8, characterized in that: The arm equipment comprises: A boom mechanism (2), comprising a boom consisting of at least one boom section; An image acquisition mechanism (3) is arranged on the arm mechanism (2) and is used to photograph the environment surrounding the arm; A graphics calculation mechanism (4) is used to establish the spatial model based on the data captured by the image acquisition mechanism (3); A display mechanism (5) is used to display the spatial model.

10. The boom equipment according to claim 9, characterized in that: The boom mechanism (2) comprises a first boom (21) and a second boom (22), the first boom (21) comprising at least one boom section, and the second boom (22) being the end boom section of the boom mechanism (2); The image acquisition mechanism (3) comprises a first shooting element (31) arranged on the first arm support (21) and a second shooting element (32) arranged on the second arm support (22).

Images