View angle image synthesis method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a visual angle image synthesis method and device, equipment, a storage medium and a program product, and relates to the technical field of Internet of Things. The method comprises the following steps: receiving a detection image shot by a monocular camera, correcting the detection image based on camera parameters of the monocular camera, inputting the corrected detection image into a depth estimation network, and determining a three-dimensional point cloud corresponding to the detection image based on an output result of the depth estimation network; based on the three-dimensional point cloud, determining the position of an obstacle within the driving range of the field bridge cart; and determining a driving strategy of the yard crane cart based on the obstacle position, and controlling the driving state of the yard crane cart based on the driving strategy. The method provided by the invention effectively solves the problems of insufficient obstacle recognition capability and poorer anti-collision effect of the anti-collision system of the field bridge cart.

Description

Technical Field

[0001] The present application relates to the field of Internet of Things technology, and in particular to a perspective image synthesis method, device, equipment, storage medium and program product. Background Art

[0002] In a container terminal or yard, a gantry crane is a major device for handling and stacking containers. The gantry crane moves horizontally on the yard through a yard crane to reach different operating positions and complete the loading, unloading and handling of containers. To ensure mobile safety, the yard crane will be equipped with an anti-collision system to slow down and stop in time when an obstacle is found to avoid collision.

[0003] In related technologies, the anti-collision system mainly determines obstacles through target detection or combined with lidar detection. However, the former has poor general performance and has difficulty identifying untrained obstacle types. The latter is more expensive and also has problems such as insufficient resolution and poor generality. It is easy to fail to identify smaller obstacles. Summary of the invention

[0004] The present application provides a perspective image synthesis method, device, equipment, storage medium and product to solve the problems of insufficient obstacle recognition capability and poor anti-collision effect of the midfield bridge large vehicle anti-collision system in the related art.

[0005] In a first aspect, the present application provides a perspective image synthesis method, comprising:

[0006] Receive the detection image taken by the monocular camera and perform correction processing on the detection image based on the camera parameters of the monocular camera. The monocular camera is set on the field bridge trolley;

[0007] The rectified detection image is input into the depth estimation network, and based on the output result of the depth estimation network, the 3D point cloud corresponding to the detection image is determined;

[0008] Based on the 3D point cloud, determine the location of obstacles within the travel range of the yard crane;

[0009] Based on the location of the obstacle, the driving strategy of the yard bridge trolley is determined, and based on the driving strategy, the driving state of the yard bridge trolley is controlled.

[0010] In a second aspect, the present application provides a perspective image synthesis device, comprising:

[0011] The receiving module is used to receive the detection image taken by the monocular camera and perform correction processing on the detection image based on the camera parameters of the monocular camera. The monocular camera is set on the field bridge trolley;

[0012] A conversion module, used to input the corrected detection image into the depth estimation network, and determine the three-dimensional point cloud corresponding to the detection image based on the output result of the depth estimation network;

[0013] The analysis module is used to determine the location of obstacles within the travel range of the crane based on the three-dimensional point cloud;

[0014] The output module is used to determine the driving strategy of the field bridge trolley based on the obstacle position, and control the driving state of the field bridge trolley based on the driving strategy.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0016] The memory stores computer-executable instructions;

[0017] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0019] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0020] The perspective image synthesis method, device, equipment, storage medium and program product provided by the embodiment of the present disclosure receive the detection image taken by a monocular camera, and correct the detection image based on the camera parameters of the monocular camera, input the corrected detection image into the depth estimation network, and determine the three-dimensional point cloud corresponding to the detection image based on the output result of the depth estimation network; determine the obstacle position within the driving range of the field bridge truck based on the three-dimensional point cloud; determine the driving strategy of the field bridge truck based on the obstacle position, and control the driving state of the field bridge truck based on the driving strategy. Thus, the detection image is captured and corrected by a monocular camera, ensuring the accuracy of the image data, while significantly saving costs compared to multi-camera shooting or laser radar-based detection solutions. Then, the three-dimensional point cloud is generated by the depth estimation network to effectively identify the obstacle position within the driving range. Compared with the traditional method, this scheme improves the recognition ability of untrained obstacle types, especially the detection accuracy of smaller obstacles. Based on the precise obstacle position, the system can formulate a safer and more effective driving strategy, thereby improving the safety and operation efficiency of the field bridge truck. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0022] Figure 1 A diagram of an application scenario of the perspective image synthesis method provided in an embodiment of the present disclosure;

[0023] Figure 2 A flowchart of a perspective image synthesis method provided by an embodiment of the present disclosure;

[0024] Figure 3 A flowchart of a perspective image synthesis method provided by yet another embodiment of the present disclosure;

[0025] Figure 4a A flowchart of a perspective image synthesis method provided by yet another embodiment of the present disclosure;

[0026] Figure 4b for Figure 4a A flowchart of a specific method for cluster analysis provided by the illustrated embodiment;

[0027] Figure 5 A schematic diagram of the structure of a perspective image synthesis device provided by yet another embodiment of the present disclosure;

[0028] Figure 6 A schematic diagram of the structure of an electronic device provided by one embodiment of the present disclosure.

[0029] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0030] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0031] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0032] In addition, this application involves conducting big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.), and using artificial intelligence technology to make automated decisions, and making technical solutions that have a significant impact on personal rights and interests based on the results of automated decisions. The application provides users with corresponding operation entrances for them to choose to agree or reject the results of automated decisions; if the user chooses to reject, the expert decision-making process will be entered.

[0033] It should be noted that the method, device, equipment, storage medium and product for preventing collisions between large vehicles and field bridges provided in this application can be used in the field of Internet of Things technology, and the application field of the method, device, equipment, storage medium and product for preventing collisions between large vehicles and field bridges provided in this application is not limited.

[0034] Figure 1 A schematic diagram of an application scenario of the perspective image synthesis method provided in this application, such as Figure 1 As shown, during the anti-collision process of the bridge trolley, the bridge trolley 100 will capture the front image through the monocular camera 110 when traveling along the track. When an obstacle 120 is identified, the bridge trolley 100 will be controlled to slow down or stop to prevent collision with the obstacle 120.

[0035] It should be noted that Figure 1 The scene shown includes only one or a specific number of cranes, monocular cameras, and obstacles for illustration, but the present disclosure is not limited to this, that is, the number of cranes, monocular cameras, and obstacles can be arbitrary.

[0036] In container terminals and yards, gantry cranes are the main handling and stacking equipment, and their safe and efficient operation is crucial. Gantry cranes move horizontally on the yard through the yard crane to reach different working positions and complete the loading, unloading and handling of containers. However, with the increasing complexity and density of port operations, the safety challenges faced by yard cranes during movement are becoming increasingly prominent. To ensure mobile safety, yard cranes are usually equipped with anti-collision systems to slow down and stop in time when obstacles are found to avoid collisions. However, existing anti-collision systems mainly rely on target detection or combined with laser radar detection to judge obstacles. These methods have exposed significant deficiencies in practical applications. For example, although target detection technology is mature, its general performance is poor, and it is difficult to identify untrained obstacle types, which leads to blind spots in the complex environment of practical applications. On the other hand, although laser radar provides high detection accuracy, it is expensive and still has deficiencies in resolution and versatility, especially when identifying smaller obstacles, it is easy to miss detection.

[0037] Due to the complex and changeable port environment and the wide variety and irregularity of obstacles, traditional detection methods are difficult to fully cover all possible situations. In addition, the efficiency of port operations requires that the anti-collision system must be not only accurate but also respond quickly, which places higher demands on existing technologies. The difficulty in solving this problem lies in the need to control system cost and complexity while improving recognition accuracy and versatility due to the requirements for cost control and efficiency in the logistics field. Although traditional multi-sensor fusion solutions can improve detection capabilities, their complexity and cost make them difficult to promote in practical applications. Therefore, the existing technology lacks a solution that can both improve recognition capabilities and keep costs and system complexity within an acceptable range.

[0038] The perspective image synthesis method provided in this application solves the problem of insufficient recognition of the field bridge truck collision avoidance system by innovatively combining a monocular camera and a depth estimation network. The monocular camera is installed on the field bridge truck to capture the detection image and perform image correction through camera parameters to ensure the accuracy of the image data. The corrected image is input into the depth estimation network to generate a three-dimensional point cloud, thereby accurately identifying the location of obstacles. Based on this, the system formulates a driving strategy and optimizes the driving status of the field bridge truck, which not only improves the recognition accuracy and versatility, but also effectively reduces costs.

[0039] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0040] Figure 2Schematic diagram of the process of the perspective image synthesis method provided in this application Figure 1 ,like Figure 2 As shown, the method includes:

[0041] S201, receiving a detection image taken by a monocular camera, and performing correction processing on the detection image based on camera parameters of the monocular camera.

[0042] Among them, the monocular camera is set on the field bridge trolley.

[0043] Specifically, this embodiment is used to provide an overall description of the main steps of the perspective image synthesis method.

[0044] The executing entity in the disclosed embodiment is a controller that controls the driving state of the bridge trolley and can obtain image data taken by a monocular camera installed on the bridge trolley. Specifically, it can be a control system installed in the bridge trolley, or it can be a cloud server to which the control system of the cloud bridge trolley communicates. For the convenience of explanation, it is collectively referred to as a control system here.

[0045] During the driving process of the yard crane, the control system will continuously receive the detection images taken by the monocular camera installed on the yard crane. The monocular camera is a camera with only one lens, which is used to capture two-dimensional images. The installation position and angle of the camera need to be carefully designed to ensure that the field of view covers the entire driving range.

[0046] The detection image taken by the monocular camera will be distorted due to the distortion of the camera lens, so correction processing is required. Correction processing usually uses camera calibration technology to correct the geometric distortion in the image by pre-measuring the internal and external parameters of the camera (such as focal length, aperture, distortion coefficient, etc., which are used to describe the optical and geometric characteristics of the camera). In this way, the accuracy of the image data can be ensured, laying the foundation for subsequent depth estimation.

[0047] In some embodiments, image preprocessing techniques, such as denoising and contrast enhancement, may be combined to improve image quality and the accuracy of subsequent processing.

[0048] S202: Input the corrected detection image into a depth estimation network, and determine a three-dimensional point cloud corresponding to the detection image based on an output result of the depth estimation network.

[0049] Specifically, the depth estimation network is a deep learning-based neural network model used to infer three-dimensional depth information from two-dimensional images. By learning a large amount of training data, the depth estimation network can identify depth clues in the image, such as shadows, textures, and perspective relationships, and thus infer the depth information of each pixel in the two-dimensional image.

[0050] The rectified image is input into a trained depth estimation network, and the output depth map is combined with the original image to generate a 3D point cloud, providing a stereoscopic view of the scene.

[0051] In some embodiments, other sensor data, such as an inertial measurement unit (IMU) or GPS information, may be combined to enhance the robustness and accuracy of depth estimation.

[0052] S203: Based on the three-dimensional point cloud, determine the position of obstacles within the driving range of the yard crane.

[0053] Specifically, the obstacle position is the specific position and distance of the obstacle relative to the field crane in three-dimensional space. Combined with the three-dimensional point cloud generated in the above steps, possible obstacles can be further identified and segmented, and then their corresponding positions can be determined.

[0054] To identify specific obstacles, clustering algorithms such as DBSCAN (density-based spatial clustering applied to noise) can be used to separate different objects in the point cloud. By analyzing the geometric features and spatial position of each cluster, potential obstacles can be determined. Then, the distance and orientation of these obstacles relative to the field crane are calculated to generate an accurate description of the obstacle location.

[0055] In some embodiments, obstacle recognition can be combined with machine learning algorithms to improve the accuracy and speed of obstacle recognition.

[0056] S204: Determine the driving strategy of the yard bridge trolley based on the obstacle position, and control the driving state of the yard bridge trolley based on the driving strategy.

[0057] Specifically, the driving strategy is the driving path and speed adjustment plan of the bridge truck based on the current environment and obstacle location. The driving state is the specific motion parameters of the bridge truck during driving, such as speed, direction and acceleration.

[0058] The control system evaluates different driving paths and strategies based on the identified obstacle locations to avoid collisions and ensure safety, and adjusts the motion parameters of the crane based on the driving strategy, and controls the crane in real time through the control system. That is, by adjusting the speed, direction and acceleration of the crane to ensure that it drives safely along the predetermined path.

[0059] The perspective image synthesis method provided in the embodiment of the present application receives the detection image taken by the monocular camera, and corrects the detection image based on the camera parameters of the monocular camera, inputs the corrected detection image into the depth estimation network, and determines the three-dimensional point cloud corresponding to the detection image based on the output result of the depth estimation network; determines the obstacle position within the driving range of the field bridge truck based on the three-dimensional point cloud; determines the driving strategy of the field bridge truck based on the obstacle position, and controls the driving state of the field bridge truck based on the driving strategy. Thus, the detection image is captured and corrected by the monocular camera, ensuring the accuracy of the image data, while significantly saving costs compared to multi-camera shooting or laser radar-based detection solutions. Then, the three-dimensional point cloud is generated by the depth estimation network to effectively identify the obstacle position within the driving range. Compared with the traditional method, this scheme improves the recognition ability of untrained obstacle types, especially the detection accuracy of smaller obstacles. Based on the precise obstacle position, the system can formulate a safer and more effective driving strategy, thereby improving the safety and operation efficiency of the field bridge truck.

[0060] Figure 3 The process diagram of the bridge truck collision avoidance provided for this application Figure 2 ,like Figure 3 As shown, in this embodiment Figure 2 Based on the embodiment, the specific process in the perspective image synthesis method is described in detail, and the method includes:

[0061] S301, obtaining a calibration plate image taken by a monocular camera.

[0062] There are at least two calibration plate images, and shooting angles of different calibration plate images are different.

[0063] Specifically, this embodiment is used to further illustrate the calibration of a monocular camera, image processing, and driving vehicle strategy judgment based on obstacle positions.

[0064] First, before taking images with a monocular camera, the camera needs to be calibrated to obtain the camera's internal parameters. At this time, the monocular camera to be calibrated can be used to take multiple images of the calibration plate from different angles, and ensure that the images cover the camera's field of view as much as possible. The calibration plate should have different plane orientations and tilt angles when shooting. The monocular camera has been calibrated based on these images.

[0065] To ensure the accuracy of the calibration, at least two or more calibration plate images need to be taken to cover different parts of the camera's field of view, thereby more comprehensively calibrating the camera's intrinsic and extrinsic parameters.

[0066] The calibration plate should be placed at different distances and angles to obtain diverse viewing angle information. During shooting, the lighting conditions should be as uniform as possible to reduce the impact of shadows and reflections on image quality.

[0067] Through this multi-angle and multi-distance shooting method, a set of calibration plate images with rich information can be obtained, providing a reliable data basis for subsequent camera calibration.

[0068] S302: Determine the image coordinates of the target point of the calibration plate in the calibration plate image.

[0069] Among them, the target points include corner points and feature points of the calibration plate.

[0070] Specifically, after capturing the calibration plate image, the coordinates of the corner points and feature points of the calibration plate in the image (i.e., image coordinates) can be extracted to determine the one-to-one correspondence between the feature points on the image plane and the corresponding feature points on the physical calibration plate, thereby determining the camera parameters.

[0071] The target point appears in the image as the intersection of a checkerboard or the center of a dot. The control system can use computer vision algorithms, such as the corner detection function in the OpenCV library, to automatically identify and extract the coordinates of the target point in the image. During the extraction process, the algorithm preprocesses the image, such as grayscale and edge detection, to improve the accuracy of corner detection. The detected target point coordinates will be used for subsequent camera parameter calculations, so their accuracy directly affects the quality of the calibration results.

[0072] In order to improve the robustness of detection, the detection results of multiple images can be averaged or weighted. In addition, if there are obvious errors in the detection results, they can be optimized through manual correction or using more complex algorithms.

[0073] S303 : Determine camera parameters based on the correspondence between the image coordinates of the target point and the physical coordinates of the target point on the calibration plate.

[0074] Specifically, the camera parameters can be calculated by pairing the known geometric information of the calibration plate (that is, the physical coordinates of the target point on the calibration plate) with the coordinates of the feature points on the image plane (that is, the image coordinates) and combining them with camera calibration algorithms, such as the Zhang Zhengyou calibration method and the least squares method.

[0075] Camera parameters include the camera's intrinsic and extrinsic parameters. The camera's intrinsic parameters include focal length, principal point coordinates, and distortion coefficients, while the extrinsic parameters describe the camera's position and posture in the world coordinate system.

[0076] The camera calibration algorithm is used to minimize the projection error between the image coordinates and the physical coordinates, and the camera parameters are iteratively optimized.

[0077] In specific implementation, you first need to establish the physical coordinate system of the target point based on the physical coordinates of the target point on the calibration plate. Usually, one corner of the calibration plate is used as the origin, and the unit can be millimeters or centimeters. Then, by inputting the image coordinates and physical coordinates into the calibration algorithm, the intrinsic and extrinsic parameters of the camera are calculated. The calculation results can be verified by the reprojection error. The smaller the error, the more accurate the calibration result. If the error is large, the calibration accuracy can be improved by increasing the number of calibration images or improving the image quality.

[0078] S304: receiving a detection image taken by a monocular camera, and performing correction processing on the detection image based on camera parameters of the monocular camera.

[0079] Among them, the monocular camera is set on the field bridge trolley.

[0080] Specifically, based on the camera parameters obtained from the calibration results, the input detection image is distorted and the rectified image data can be obtained. For related content, see Figure 2 The corresponding description in the illustrated embodiment will not be repeated here.

[0081] S305: Input the corrected detection image into a depth estimation network, and determine the output result of the depth estimation network as a depth map corresponding to the detection image.

[0082] Specifically, commonly used depth estimation networks include U-Net, DenseDepth, Depth Anything, etc. These networks learn the mapping relationship between image features and depth information through training data. After the input detection image is processed by the network's convolution layer, pooling layer, and deconvolution layer, a depth map with the same size as the input image is output. Each pixel value in the depth map represents the relative depth of the pixel in the scene.

[0083] S306 , combining the depth coordinates in the depth map with the image coordinates in the corrected detection image to obtain the three-dimensional coordinates of the pixels corresponding to the detection image.

[0084] Specifically, the depth map provides the depth value of each pixel, that is, the distance from the camera to the pixel in the scene, that is, the depth coordinate.

[0085] Combined with the image coordinates (the two-dimensional position of the pixel), they can be converted into three-dimensional coordinates through a geometric transformation.

[0086] By substituting the 2D image coordinates and depth values ​​into the camera's intrinsic parameter matrix, the 3D coordinates of the pixels are calculated. The generated 3D pixel coordinates provide basic data for subsequent 3D spatial coordinate calculations and point cloud generation.

[0087] S307 . Determine the spatial three-dimensional coordinates corresponding to the three-dimensional coordinates of the pixel based on the camera parameters of the monocular camera.

[0088] According to the principle of converting pixel coordinates to the world table:

[0089] ;

[0090] Where X w , Y w , Z w is the three-dimensional space coordinate of the spatial coordinate system, u and v are image coordinates, and Z c is the depth value, the right side of the equal sign is the rotation transformation matrix, which can be determined based on the external parameters obtained in the calibration process in the previous step. The first matrix on the right side of the equal sign is the intrinsic parameter matrix of the monocular camera, which can also be obtained through the calibration process.

[0091] Therefore, through the above formula transformation, the conversion between two-dimensional image and three-dimensional point cloud can be completed through the depth map and image coordinates.

[0092] S308 . Determine a three-dimensional point cloud corresponding to the detected image based on the three-dimensional spatial coordinates.

[0093] Specifically, the point cloud generation process is to collect the spatial three-dimensional coordinates of all pixels together to form a complete three-dimensional spatial representation for further geometric analysis and object recognition.

[0094] In some embodiments, in order to improve the accuracy and resolution of the point cloud, multiple frames of images may be combined or filtering techniques may be used to remove noise and redundant data.

[0095] S309: Based on the three-dimensional point cloud, determine the position of obstacles within the driving range of the yard crane.

[0096] Specifically, the three-dimensional point cloud obtained through the above steps can be combined with algorithms such as through-filtering and Euclidean clustering segmentation to determine obstacles within the driving range of the field bridge truck, and then determine the position of the obstacles. For the specific method of determining the position of the obstacle, please refer to the description of the subsequent embodiments, which will not be expanded here.

[0097] S310: Determine the correspondence between the obstacle position and the pre-set field bridge vehicle driving area.

[0098] Among them, the driving area includes a safety area, a deceleration area and a parking area.

[0099] Specifically, by determining the position of the obstacle, it is possible to determine which part of the defined area the obstacle falls on, and the corresponding driving strategy is different according to the different corresponding driving areas.

[0100] S311. If the obstacle is located in a safe area, the driving strategy of the yard crane is determined to be normal driving.

[0101] Specifically, when an obstacle is determined to be in a safe area, it means that the obstacle does not pose a threat to the travel of the crane. Therefore, in this case, the travel strategy can be set to normal travel, and there is no need to send a signal to the travel mechanism of the crane, that is, the crane can continue to travel along the predetermined path and speed.

[0102] S312: If the obstacle is located in the deceleration area, the driving strategy of the yard bridge truck is determined to be deceleration to stop.

[0103] Specifically, the deceleration area is usually a buffer zone, which is used to gradually reduce the vehicle speed when an obstacle approaches, so as to avoid the impact and potential danger caused by sudden stopping. When the obstacle is in the deceleration area, the control system needs to send a signal to the driving mechanism of the long bridge taxi to gradually reduce the throttle or apply the brakes, so as to achieve a smooth deceleration process. If the obstacle continues to approach or enter the parking area, the system should be prepared to execute further parking strategies.

[0104] S313: If the obstacle is located in the parking area, the driving strategy of the yard bridge truck is determined to be direct parking.

[0105] Specifically, when an obstacle enters the parking area, it means that the vehicle must stop immediately to avoid a collision. Implementing a direct parking strategy requires fast and accurate control of the braking system to stop the vehicle safely in the shortest distance.

[0106] S314. Based on the driving strategy, control the driving state of the yard crane.

[0107] Specifically, according to the driving strategy, the control system will send corresponding control signals to the driving mechanism to dynamically adjust the vehicle's power output and braking system.

[0108] For normal driving strategies, the system should maintain a stable speed and direction of the vehicle; for deceleration or parking strategies, the system should gradually reduce the speed or apply brakes to ensure that the vehicle stops safely at the predetermined location, thereby effectively achieving the purpose of large vehicle collision avoidance.

[0109] The perspective image synthesis method provided by the embodiment of the present disclosure effectively improves the obstacle recognition capability and anti-collision effect of the bridge truck by combining a monocular camera with a depth estimation network. The use of a monocular camera reduces hardware costs, and at the same time generates a high-precision three-dimensional point cloud through a depth estimation network, overcoming the shortcomings of traditional methods in identifying untrained obstacle types and small obstacles. The accuracy of depth information is ensured through precise camera calibration and image correction. Obstacle position judgment and driving strategy formulation based on three-dimensional point clouds realize dynamic control of the driving state of the bridge truck, prevent collisions with obstacles in the driving area, enhance the safety and adaptability of the system, and the overall solution has the advantages of high efficiency, economy and reliability.

[0110] Figure 4a The process diagram of the bridge truck collision avoidance provided for this application Figure 3 ,like Figure 4a As shown, in this embodiment Figure 2 and Figure 3 Based on the embodiment, the specific process in the perspective image synthesis method is further described in detail. The method includes:

[0111] S401, receiving a detection image taken by a monocular camera, and performing correction processing on the detection image based on camera parameters of the monocular camera.

[0112] Among them, the monocular camera is set on the field bridge trolley.

[0113] Specifically, this embodiment is used to further refine the description of the part related to determining the obstacle position based on the above-mentioned embodiment.

[0114] S402: Input the corrected detection image into a depth estimation network, and determine the three-dimensional point cloud corresponding to the detection image based on the output result of the depth estimation network.

[0115] Specifically, the contents of step S401 to step S402 are the same as the corresponding steps in the aforementioned embodiment, and will not be repeated here.

[0116] S403: Determine the driving range of the vehicle.

[0117] Specifically, the point cloud data obtained in the aforementioned steps contains the point clouds of all objects within the shooting range of the monocular camera. In order to evaluate whether there are obstacles within the driving range of the yard bridge truck, it is necessary to determine the point cloud data within the actual driving range of the yard bridge truck, so as to perform direct filtering on the above point cloud data according to the actual driving range to obtain the point cloud data within the actual driving range.

[0118] The actual driving range of the crane can be pre-configured according to the installation position of the monocular camera and represented by a three-dimensional bounding box, which defines the drivable area of ​​the crane in space.

[0119] By clarifying the driving range, irrelevant point cloud data can be effectively filtered out in subsequent steps, reducing the amount of calculation and improving the accuracy of obstacle detection.

[0120] S404: Perform rotation transformation processing on the three-dimensional point cloud based on the set rotation transformation matrix.

[0121] The rotation transformation matrix is ​​used to adjust the ground point cloud in the three-dimensional point cloud to be parallel to the horizontal plane of the space.

[0122] Specifically, since the point cloud data also contains a large number of ground corresponding point clouds, it is necessary to filter out the ground corresponding 3D points before determining the point cloud of the obstacle. Therefore, it is necessary to adjust the ground points in the point cloud to be parallel to the horizontal plane of the space.

[0123] The rotation transformation is implemented through a rotation matrix, which can be calculated based on the initial pose and the target pose of the point cloud. Specifically, the normal vectors of the point cloud need to be estimated, and then the rotation matrix required to rotate these normal vectors to be parallel to the Z axis is calculated. After applying this rotation matrix, the ground points in the point cloud will be adjusted to the horizontal plane, which simplifies the subsequent processing steps.

[0124] The accuracy of the rotation transformation directly affects the effect of obstacle detection. Therefore, the accuracy of the sensor and the influence of environmental factors need to be considered when calculating the rotation matrix.

[0125] S405: Filter the ground point cloud in the three-dimensional point cloud.

[0126] Specifically, by filtering the ground point cloud, unnecessary data can be removed, thereby more accurately identifying obstacles above the ground.

[0127] In some embodiments, the control system can determine the ground point cloud based on a plane fitting algorithm based on random sample consistency, based on the three-dimensional space points in the three-dimensional point cloud that are parallel to the spatial horizontal plane and have an angle tolerance range below a set value, and then filter the three-dimensional space points in the three-dimensional point cloud that belong to the ground point cloud.

[0128] Specifically, the filtering process is usually based on a plane fitting algorithm, such as the Random Sample Consensus (RANSAC) algorithm. The RANSAC algorithm randomly selects several points in the point cloud for plane fitting and evaluates the quality of the fitting results to determine which points belong to the ground. Specifically, the algorithm randomly selects three points in the point cloud, assumes that these three points form a plane, and then calculates the distance from all points to the plane. If the distance from a point to the plane is less than the set threshold (which can be achieved by setting the angle tolerance range to a set value, such as 5°), the point is considered to belong to the ground.

[0129] Through multiple iterations, the RANSAC algorithm can effectively identify ground points and remove them from the point cloud. The filtered point cloud will only contain objects above the ground, providing a more accurate data basis for subsequent obstacle detection.

[0130] S406: Perform cluster analysis on the filtered three-dimensional point cloud in the vehicle driving range area to determine the obstacle center coordinates and obstacle boundary box in the filtered three-dimensional point cloud.

[0131] Specifically, by performing cluster analysis on the three-dimensional point cloud after ground filtering, obstacles can be identified.

[0132] Specific clustering algorithms include DBSCAN (density-based clustering algorithm), K-means, Euclidean clustering, etc. Through cluster analysis, the center coordinates and bounding box of each obstacle can be determined.

[0133] In some embodiments, Figure 4b As shown in FIG. 1 , it is a flow chart of a specific method of cluster analysis. The specific process of cluster analysis includes the following steps:

[0134] S4061. Determine the filtered three-dimensional point cloud in the vehicle driving range as the target three-dimensional point cloud.

[0135] Specifically, the three-dimensional point clouds filtered in the above steps are determined as target three-dimensional point clouds for subsequent analysis, so as to perform cluster analysis based on these target three-dimensional point clouds.

[0136] S4062. Perform cluster analysis on the target three-dimensional point cloud, and determine the cluster center obtained by the cluster analysis as the obstacle center.

[0137] Specifically, cluster analysis can use clustering algorithms such as Euclidean clustering. Through cluster analysis, the center coordinates of each obstacle, that is, the cluster center, can be determined. The specific calculation method can be obtained by calculating the average coordinates of all points in each cluster.

[0138] S4063. Determine the coordinates of the obstacle center based on the coordinates of the cluster center.

[0139] Specifically, the coordinates of the obstacle center can be obtained by calculating the average coordinates of all points in the cluster.

[0140] S4064. Based on each cluster analysis, obtain the coordinate range of the three-dimensional point cloud corresponding to the cluster, and determine the boundary point coordinates of the obstacle.

[0141] Specifically, based on the above steps, the boundary point coordinates of each cluster can be determined to construct a three-dimensional bounding box of the obstacle. The boundary point coordinates refer to the vertex coordinates of the smallest rectangle or cube that contains all the points in the cluster. These coordinates define the range occupied by the obstacle in three-dimensional space, which is crucial for subsequent path planning and obstacle avoidance strategy formulation. In order to determine the boundary point coordinates, the maximum and minimum coordinate values ​​of all the points in the cluster can be calculated to obtain the size and position of the bounding box.

[0142] S4065: Determine a bounding box of the obstacle based on the boundary point coordinates.

[0143] Specifically, the bounding box of each obstacle is constructed using the boundary point coordinates determined in the previous step.

[0144] The bounding box can be represented by the coordinates of each vertex corresponding to the bounding box, or by the size of the bounding box (such as length, width and height).

[0145] S4066: Determine the type of obstacle based on the obstacle bounding box.

[0146] Specifically, by analyzing the bounding box information of the obstacle, the type of obstacle can be further determined. Identifying the type of obstacle is crucial to developing an appropriate driving strategy, because different types of obstacles may require different treatments. The type of obstacle is usually identified in combination with the size, shape, and other feature information of the bounding box. Machine learning or deep learning algorithms can be used to train models to identify different types of obstacles, such as vehicles, pedestrians, and fixed facilities.

[0147] The recognition results can be used to optimize the driving strategy of the large vehicle. For example, if the obstacle size is smaller than a certain threshold, there is no need to respond to the obstacle (for example, the obstacle is a plastic bag or other object that will not affect the driving of the large vehicle).

[0148] In this way, a more accurate decision-making basis can be provided for the driving strategy of large vehicles.

[0149] S407: Determine the position of the obstacle within the travel range of the yard crane based on the obstacle center coordinates and the obstacle boundary box.

[0150] Specifically, the obstacle center coordinates and bounding box information obtained through cluster analysis can be used to determine the location of the obstacle within the travel range of the field bridge vehicle. The location information of the obstacle includes its coordinates in three-dimensional space and the space range it occupies.

[0151] By comparing the center coordinates of the obstacle with the current position of the vehicle to determine the relative position, and combining the bounding box information of the obstacle, the shape and size of the obstacle can be further analyzed, thereby providing more detailed data support for the driving strategy.

[0152] S408. Determine the driving strategy of the yard bridge trolley based on the obstacle position, and control the driving state of the yard bridge trolley based on the driving strategy.

[0153] Specifically, the contents related to step S408 are the same as those of the corresponding steps in the aforementioned embodiment, and will not be repeated here.

[0154] The perspective image synthesis method provided in the embodiment of the present disclosure significantly improves the obstacle recognition capability of the field bridge truck anti-collision system by introducing a monocular camera combined with a depth estimation network. First, the low cost and flexibility of the monocular camera are used to solve the high cost problem of the traditional lidar system. Secondly, the three-dimensional point cloud is generated by the depth estimation network, so that the system can accurately identify and locate obstacles within the driving range of the truck, especially for small and untrained obstacle types, which improves the versatility and accuracy of recognition. In addition, combined with cluster analysis and bounding box construction, the system can effectively filter the ground point cloud and identify the center coordinates and boundaries of the obstacles, so as to formulate a more accurate driving strategy. This solution not only improves the safety and reliability of the anti-collision system, but also optimizes the operating efficiency of the truck, reduces operational risks, and has significant practical value.

[0155] Figure 5 This is a schematic diagram of the structure of the perspective image synthesis device provided in this application, such as Figure 5 As shown, the perspective image synthesis device 500 provided in this embodiment includes:

[0156] The receiving module 510 is used to receive the detection image taken by the monocular camera and perform correction processing on the detection image based on the camera parameters of the monocular camera. The monocular camera is set on the field bridge trolley;

[0157] A conversion module 520 is used to input the corrected detection image into the depth estimation network, and determine the three-dimensional point cloud corresponding to the detection image based on the output result of the depth estimation network;

[0158] The analysis module 530 is used to determine the location of obstacles within the travel range of the field crane based on the three-dimensional point cloud;

[0159] The output module 540 is used to determine the driving strategy of the field bridge truck based on the obstacle position, and control the driving state of the field bridge truck based on the driving strategy.

[0160] In a possible implementation, the receiving module 510 is further used to receive a detection image taken by a monocular camera, and before correcting the detection image based on the camera parameters of the monocular camera, obtain a calibration plate image taken by the monocular camera, wherein there are at least two calibration plate images, and shooting angles of different calibration plate images are different; determine the image coordinates of a target point of the calibration plate in the calibration plate image, the target point including a corner point and a feature point of the calibration plate; and determine the camera parameters based on the correspondence between the image coordinates of the target point and the physical coordinates of the target point on the calibration plate.

[0161] In one possible implementation, the conversion module 520 is specifically used to input the corrected detection image into a depth estimation network, and determine the output result of the depth estimation network as a depth map corresponding to the detection image; combine the depth coordinates in the depth map with the image coordinates in the corrected detection image to obtain the pixel three-dimensional coordinates corresponding to the detection image; determine the spatial three-dimensional coordinates corresponding to the pixel three-dimensional coordinates based on the camera parameters of the monocular camera; and determine the three-dimensional point cloud corresponding to the detection image based on the spatial three-dimensional coordinates.

[0162] In one possible implementation, the analysis module 530 is specifically used to determine the driving range of the large vehicle; based on a set rotation transformation matrix, perform rotation transformation processing on the three-dimensional point cloud, wherein the rotation transformation matrix is ​​used to adjust the ground point cloud in the three-dimensional point cloud to be parallel to the horizontal plane of the space; filter the ground point cloud in the three-dimensional point cloud; perform cluster analysis on the filtered three-dimensional point cloud in the driving range of the large vehicle to determine the center coordinates of the obstacles and the obstacle bounding box in the filtered three-dimensional point cloud; based on the center coordinates of the obstacles and the obstacle bounding box, determine the position of the obstacles within the driving range of the field bridge large vehicle.

[0163] In one possible implementation, the analysis module 530 is specifically used to determine the filtered three-dimensional point cloud in the driving range area of ​​the large vehicle as the target three-dimensional point cloud; perform cluster analysis on the target three-dimensional point cloud, and determine the cluster center obtained by the cluster analysis as the obstacle center; determine the coordinates of the obstacle center based on the coordinates of the cluster center; determine the boundary point coordinates of the obstacle based on the coordinate range of the three-dimensional point cloud corresponding to each cluster analysis; and determine the boundary box of the obstacle based on the boundary point coordinates.

[0164] In one possible implementation, the analysis module 530 is specifically used to determine the ground point cloud based on a plane fitting algorithm based on random sample consistency, based on three-dimensional space points in the three-dimensional point cloud that are parallel to the spatial horizontal plane and whose angle tolerance range is below a set value; and filter the three-dimensional space points in the three-dimensional point cloud that belong to the ground point cloud.

[0165] In one possible implementation, the analysis module 530 is also used to perform cluster analysis on the filtered three-dimensional point cloud in the vehicle driving range area, determine the obstacle center coordinates and obstacle bounding box in the filtered three-dimensional point cloud, and then determine the obstacle type based on the obstacle bounding box.

[0166] In one possible implementation, the output module 540 is specifically used to determine the correspondence between the obstacle position and the pre-set driving area of ​​the field bridge truck, the driving area including the safety area, the deceleration area and the parking area; if the obstacle position is located in the safety area, the driving strategy of the field bridge truck is determined to be normal driving; if the obstacle position is located in the deceleration area, the driving strategy of the field bridge truck is determined to decelerate to stop; if the obstacle position is located in the parking area, the driving strategy of the field bridge truck is determined to be direct parking; based on the driving strategy, the driving state of the field bridge truck is controlled.

[0167] The perspective image synthesis device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0168] Figure 6 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 6 As shown, the electronic device 60 provided in this embodiment includes: at least one processor 601 and a memory 602. Optionally, the device 60 also includes a communication component 603. The processor 601, the memory 602 and the communication component 603 are connected via a bus 604.

[0169] In a specific implementation process, at least one processor 601 executes the computer execution instructions stored in the memory 602, so that at least one processor 601 executes the above method.

[0170] The specific implementation process of the processor 601 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0171] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.

[0172] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0173] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0174] The present application also provides a computer program product, including a computer program, which implements the method in any of the above embodiments when executed by a processor.

[0175] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method in any of the above embodiments is implemented.

[0176] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0177] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0178] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0179] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0180] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0181] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0182] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0183] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0184] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0185] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0186] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. Unless otherwise specified, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. Unless otherwise specified, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory (RRAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), an enhanced dynamic random access memory (EDRAM), a high-bandwidth memory (HBM), a hybrid memory cube (HMC), etc.

[0187] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0188] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0189] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0190] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for synthesizing a perspective image, characterized in that: The steps include: Receiving a detection image taken by a monocular camera, and performing correction processing on the detection image based on camera parameters of the monocular camera, wherein the monocular camera is arranged on the field bridge trolley; Inputting the corrected detection image into a depth estimation network, and determining a three-dimensional point cloud corresponding to the detection image based on an output result of the depth estimation network; Based on the three-dimensional point cloud, determine the position of obstacles within the travel range of the field crane; Based on the position of the obstacle, the driving strategy of the bridge trolley is determined, and based on the driving strategy, the driving state of the bridge trolley is controlled.

2. The method according to claim 1, characterized in that The method further includes: receiving a detection image taken by a monocular camera, and before correcting the detection image based on camera parameters of the monocular camera, the method further includes: Acquire a calibration plate image captured by the monocular camera, wherein there are at least two calibration plate images, and shooting angles of different calibration plate images are different; Determine the image coordinates of the target points of the calibration plate in the calibration plate image, wherein the target points include corner points and feature points of the calibration plate; The camera parameters are determined based on the correspondence between the image coordinates of the target point and the physical coordinates of the target point on the calibration plate.

3. The method according to claim 2, characterized in that The step of inputting the corrected detection image into a depth estimation network, and determining a three-dimensional point cloud corresponding to the detection image based on an output result of the depth estimation network, comprises: Inputting the corrected detection image into a depth estimation network, and determining an output result of the depth estimation network as a depth map corresponding to the detection image; Combining the depth coordinates in the depth map with the image coordinates in the corrected detection image to obtain the three-dimensional coordinates of the pixels corresponding to the detection image; Determining the spatial three-dimensional coordinates corresponding to the three-dimensional coordinates of the pixel based on the camera parameters of the monocular camera; Based on the three-dimensional spatial coordinates, a three-dimensional point cloud corresponding to the detection image is determined.

4. The method according to claim 1, characterized in that: Determining the position of obstacles within the driving range of the field bridge vehicle based on the three-dimensional point cloud includes: Determine the driving range of the vehicle; Based on a set rotation transformation matrix, the three-dimensional point cloud is subjected to rotation transformation processing, wherein the rotation transformation matrix is ​​used to adjust the ground point cloud in the three-dimensional point cloud to be parallel to the horizontal plane of the space; Filtering the ground point cloud in the three-dimensional point cloud; Performing cluster analysis on the filtered three-dimensional point cloud in the driving range of the vehicle to determine the center coordinates of obstacles and the obstacle boundary box in the filtered three-dimensional point cloud; Based on the obstacle center coordinates and the obstacle boundary box, the obstacle position within the travel range of the field bridge trolley is determined.

5. The method according to claim 4, characterized in that The cluster analysis is performed on the filtered three-dimensional point cloud in the driving range of the large vehicle to determine the obstacle center coordinates and obstacle boundary box in the filtered three-dimensional point cloud, including: Determine the filtered three-dimensional point cloud in the vehicle driving range area as the target three-dimensional point cloud; Performing cluster analysis on the target three-dimensional point cloud, and determining the cluster center obtained by the cluster analysis as the obstacle center; Based on the coordinates of the cluster center, determining the coordinates of the obstacle center; Based on each cluster analysis, the coordinate range of the three-dimensional point cloud corresponding to the cluster is obtained, and the coordinates of the boundary points of the obstacle are determined; Based on the boundary point coordinates, a bounding box of the obstacle is determined.

6. The method according to claim 4, characterized in that The filtering of the ground point cloud in the three-dimensional point cloud comprises: A plane fitting algorithm based on random sample consistency is used to determine the ground point cloud based on three-dimensional space points in the three-dimensional point cloud that are parallel to the spatial horizontal plane and whose angle tolerance range is below a set value; The three-dimensional space points in the three-dimensional point cloud that belong to the ground point cloud are filtered.

7. The method according to claim 4, characterized in that After clustering analysis is performed on the filtered three-dimensional point cloud in the vehicle driving range area to determine the obstacle center coordinates and obstacle boundary box in the filtered three-dimensional point cloud, the method further includes: The obstacle type is determined based on the obstacle bounding box.

8. The method according to any one of claims 1 to 7, characterized in that The method of determining the driving strategy of the bridge trolley based on the obstacle position and controlling the driving state of the bridge trolley based on the driving strategy includes: Determine the correspondence between the obstacle position and the preset field bridge vehicle driving area, wherein the driving area includes a safety area, a deceleration area, and a parking area; If the obstacle is located in the safety area, the driving strategy of the bridge trolley is determined to be normal driving; If the obstacle is located in the deceleration area, the driving strategy of the bridge trolley is determined to be deceleration to parking; If the obstacle is located in the parking area, the driving strategy of the bridge trolley is determined to be direct parking; Based on the driving strategy, the driving state of the field bridge vehicle is controlled.

9. A perspective image synthesis device, characterized in that: include: A receiving module, used for receiving a detection image taken by a monocular camera, and performing correction processing on the detection image based on camera parameters of the monocular camera, wherein the monocular camera is arranged on the field bridge trolley; A conversion module, used to input the corrected detection image into a depth estimation network, and determine a three-dimensional point cloud corresponding to the detection image based on an output result of the depth estimation network; An analysis module, used to determine the position of obstacles within the travel range of the field crane based on the three-dimensional point cloud; The output module is used to determine the driving strategy of the field bridge trolley based on the position of the obstacle, and control the driving state of the field bridge trolley based on the driving strategy.

10. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 8 when executed by a processor.

Images