Vehicle end and remote driving control system for surface mine operation

Abstract

The invention relates to the technical field of intelligent driving, and particularly discloses a vehicle end and a remote driving control system for surface mine operation, and the vehicle end comprises 360-degree looking-around camera equipment, a vehicle chassis, a vehicle end controller and an intelligent gateway; the vehicle chassis is used for adjusting the running state of the vehicle according to the control instruction of the vehicle end controller, and the running state of the vehicle at least comprises advancing, retreating, course angle adjusting and acceleration adjusting; the vehicle end controller is used for generating a corresponding control instruction according to the control signal of the cabin end, obtaining a panoramic image of the surrounding environment of the vehicle end according to the remote driving control device for surface mine operation and sending the panoramic image of the surrounding environment of the vehicle end to the cabin end; and the intelligent gateway is used for realizing communication between the vehicle end controller and the cabin end. The vehicle end provided by the invention can realize completely unmanned operation of vehicles in a mining area.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and in particular to a vehicle terminal and a remote driving control system for open-pit mine operations. Background Art

[0002] With the continuous advancement of smart cars and connected vehicle technologies, the application of autonomous driving in specific environments, such as open-pit mining, is gradually becoming operational. However, in practice, autonomous driving technology still needs further refinement. Human intervention is still required in complex working conditions, such as precise parking in parking and operating areas, and when multiple vehicles are working together. Furthermore, due to the often remote locations of mines, establishing a stable and reliable network connection is costly. Furthermore, the speed of mining trucks within the operating area is also limited.

[0003] Therefore, how to achieve completely unmanned operation of intelligent driving in mining areas has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the Invention

[0004] The present invention provides a vehicle-side and remote driving control system for open-pit mine operations, which solves the problem in related technologies that it is impossible to achieve fully unmanned intelligent driving operation in mining areas.

[0005] As a first aspect of the present invention, a vehicle side is provided, comprising: a 360-degree surround view camera device, a vehicle chassis, a vehicle side controller, and an intelligent gateway, wherein the 360-degree surround view camera device, the vehicle chassis, and the intelligent gateway are all communicatively connected to the vehicle side controller, and the vehicle side controller includes a remote driving control device for open-pit mine operations;

[0006] The 360-degree surround view camera device is used to collect images of the vehicle's surrounding environment in real time;

[0007] The vehicle chassis is used to adjust the vehicle's operating state according to the control instructions of the vehicle-side controller, and the vehicle's operating state includes at least forward, backward, heading angle adjustment and acceleration adjustment;

[0008] The vehicle-side controller is used to generate corresponding control instructions according to the control signal of the cockpit side, and obtain a panoramic image of the vehicle-side surrounding environment according to the remote driving control device for open-pit mine operation and send the panoramic image of the vehicle-side surrounding environment to the cockpit side;

[0009] The intelligent gateway is used to realize the communication between the vehicle-side controller and the cockpit end;

[0010] The remote driving control device for open-pit mine operation includes:

[0011] A startup module, used to start the vehicle body and the 360-degree surround view camera equipment mounted on the vehicle body according to the vehicle operation control instructions on the cockpit end;

[0012] An acquisition module is used to acquire the vehicle surrounding environment image data information collected by the 360-degree surround view camera device;

[0013] A preprocessing module, configured to perform dedistortion preprocessing on the vehicle surrounding environment image data information to obtain a distortion-corrected image;

[0014] A stitching and fusion module is used to stitch and fuse the distortion-corrected images to obtain a panoramic image of the vehicle's surrounding environment;

[0015] The sending module is used to send the panoramic image of the vehicle's surrounding environment to the cockpit end, and the cockpit end can perform remote driving control based on the panoramic image of the vehicle's surrounding environment.

[0016] Furthermore, dedistortion preprocessing is performed on the vehicle surrounding environment image data information to obtain a distortion-corrected image, including:

[0017] determining a distortion radius according to an offset of distorted pixel coordinates relative to a center in the vehicle surrounding environment image data information;

[0018] The coordinates of the pixels without distortion in the vehicle surrounding environment image data are determined according to the correlation relationship between the distorted radius and the undistorted radius.

[0019] Furthermore, determining the undistorted pixel coordinates in the vehicle surrounding environment image data information according to the correlation relationship between the distorted radius and the undistorted radius includes:

[0020] The correlation between the distorted radius and the undistorted radius is determined, where the correlation between the two is expressed as follows:

[0021] ,

[0022] in, Indicates the distortion radius, 、 、 、 Represents the distortion coefficient of the camera device, where , , ;

[0023] The expression of the undistorted pixel coordinates is determined as follows:

[0024] ,

[0025] .

[0026] Furthermore, the distortion-corrected images are stitched and fused to obtain a panoramic image of the vehicle, including:

[0027] Performing coordinate transformation on the distortion-corrected image to obtain distortion-corrected image information in a world coordinate system;

[0028] Perform image stitching based on the distortion-corrected image in the world coordinate system to obtain a stitched image of the vehicle's surrounding environment;

[0029] The vehicle-side surrounding environment stitching images are fused to obtain a vehicle-side panoramic image.

[0030] Furthermore, performing image fusion on the vehicle-side surrounding environment stitching image to obtain a vehicle-side panoramic image includes:

[0031] For any two stitched images to be fused in the vehicle-side surrounding environment stitched image, determining that a weight value of a region where one stitched image occupies a dominant position is 1, and a weight value of a region where the other stitched image occupies a dominant position is 0;

[0032] For each pixel in the overlapping area, determining a fused pixel value as a weighted result of a pixel value in one of the stitched images and a pixel value in the other stitched image;

[0033] Repeat the above process to obtain the pairwise fusion results of the vehicle-side surrounding environment stitching images;

[0034] The vehicle-side panoramic image is obtained based on the pairwise fusion results of the vehicle-side surrounding environment stitching images.

[0035] As another aspect of the present invention, a remote driving control system for open-pit mine operations is provided, which includes: a cockpit end and a vehicle end, the cockpit end and the vehicle end are communicatively connected, and the vehicle end includes the vehicle end described above.

[0036] Furthermore, the cabin end includes a cabin end gateway device, a cabin end control unit, a cabin end industrial computer and a cabin end human-computer interaction unit, and the cabin end gateway device, the cabin end industrial computer and the cabin end human-computer interaction unit are all communicatively connected to the cabin end control unit.

[0037] The cabin-side gateway device is used to realize the communication connection between the cabin-side control unit and the vehicle side;

[0038] The cabin-end human-computer interaction unit is used to realize human-computer interaction between the cabin-end control unit and the cabin-end industrial computer;

[0039] The cabin-side control unit is used to receive and process the image information fed back by the vehicle side and process the image information to realize remote driving control of the vehicle side, and to control the cabin-side human-computer interaction unit to display the operating status of the vehicle side to the user;

[0040] The cabin-end industrial computer is used to receive user operation instructions according to the cabin-end human-computer interaction unit, and generate vehicle-end operation control signals according to the user operation instructions.

[0041] Furthermore, it also includes a cloud, which is communicated with the cockpit end and the vehicle end respectively, and is used to store and manage data information of the cockpit end and the vehicle end, and to provide data update services for the cockpit end and the vehicle end.

[0042] The vehicle side provided by the present invention is communicatively connected to the cockpit side. The cockpit side can send remote control commands to the vehicle side. The vehicle side can obtain panoramic image information of the vehicle side's surroundings through a 360-degree surround-view camera device and send this panoramic image information to the cockpit side so that the cockpit side can issue corresponding control commands and achieve remote control of the vehicle side. Because this vehicle side can obtain accurate and clear panoramic image information of the surrounding environment, the cockpit side can remotely control the vehicle side, thereby making it possible for the vehicle side in the mining area to be unmanned. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present invention, but do not constitute a limitation of the present invention.

[0044] Figure 1 This is a structural block diagram of the vehicle side provided by the present invention.

[0045] Figure 2 This is a structural block diagram of the remote driving control device for open-pit mine operations provided by the present invention.

[0046] Figure 3 This is a flowchart of the de-distortion preprocessing provided by the present invention.

[0047] Figure 4 This is a flowchart of image stitching and image fusion provided by the present invention.

[0048] Figure 5 This is a structural block diagram of the remote driving control system for open-pit mine operations provided by the present invention. DETAILED DESCRIPTION

[0049] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0050] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0051] It should be noted that the terms "first," "second," and the like in the specification and claims of the present invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate for the embodiments of the present invention described herein. In addition, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatuses.

[0052] In this embodiment, a vehicle end 10 is provided, wherein, as Figure 1 As shown, it includes: a 360-degree surround view camera device 11, a vehicle chassis 12, a vehicle-side controller 13 and an intelligent gateway 14. The 360-degree surround view camera device 11, the vehicle chassis 12 and the intelligent gateway 14 are all communicatively connected to the vehicle-side controller 13, and the vehicle-side controller 13 includes a remote driving control device 100 for open-pit mine operations;

[0053] The 360-degree surround view camera device 11 is used to collect images of the vehicle's surrounding environment in real time;

[0054] The vehicle chassis 12 is used to adjust the vehicle's operating state according to the control instructions of the vehicle-side controller, and the vehicle's operating state at least includes forward, backward, heading angle adjustment and acceleration adjustment;

[0055] The vehicle-side controller 13 is used to generate corresponding control instructions according to the control signal of the cockpit side, and obtain a panoramic image of the vehicle-side surrounding environment according to the remote driving control device for open-pit mine operation and send the panoramic image of the vehicle-side surrounding environment to the cockpit side;

[0056] The intelligent gateway 14 is used to realize the communication between the vehicle-side controller and the cockpit-side.

[0057] In this embodiment of the present invention, the vehicle side includes a 360-degree surround-view camera device, a vehicle chassis, a vehicle-side controller, and an intelligent gateway device. The vehicle-side controller, serving as an edge computing unit, integrates a video processing node, a chassis control node, and a routing communication node. The sensing device is connected to the vehicle-side controller via a wiring harness, and the vehicle chassis and the vehicle-side controller are connected via a CAN bus. The intelligent gateway device is responsible for enabling two-way communication between the vehicle side and the external environment.

[0058] The panoramic vision camera module, the core of the 360-degree surround-view camera system, consists primarily of a front-facing camera, a right-side camera, a left-side camera, and a rearview camera. The vehicle-side controller utilizes panoramic vision algorithms to dedistort, stitch, and stream the images from these four cameras, ensuring an immersive visual experience for the remote driver. Furthermore, the vehicle-side chassis (specifically, its control system) manages key information such as vehicle forward and reverse movements, heading angle adjustment, and acceleration. As the core component of the remote driving system, the vehicle-side controller primarily handles video encoding, command processing, and forwarding.

[0059] In the embodiment of the present invention, Figure 2 As shown, the remote driving control device 100 for open-pit mine operation includes:

[0060] The starting module 110 is used to start the vehicle body and the 360-degree surround view camera device mounted on the vehicle body according to the vehicle operation control instruction of the cockpit end;

[0061] In the embodiments of the present invention, it should be understood that the vehicle side is communicatively connected to the cockpit side, and the cockpit side in the embodiments of the present invention is different from the traditional concept of a driving cockpit. The cockpit side in the embodiments of the present invention is separated from the vehicle side, which is equivalent to separating the cockpit side (control mechanism) that issues instructions from the vehicle from the vehicle side (actuator). The remote cockpit side and the vehicle side can be in different geographical locations and, through a communication connection (for example, 5G communication), send the vehicle operation control instructions issued by the remote cockpit side in the dispatch room to the vehicle side operating in the mining area to control the vehicle side to execute functions such as forward, reverse, braking, and steering. After receiving the vehicle operation control instructions from the cockpit side, the vehicle side can activate the vehicle body and the 360-degree surround view camera equipment mounted on the vehicle body to detect the surrounding environment.

[0062] It should be noted that, in the embodiment of the present invention, since the vehicle body is equipped with a 360-degree surround-view camera device, it can perceive the vehicle's surrounding environment information in 360 degrees without blind spots.

[0063] An acquisition module 120 is configured to acquire image data information of the vehicle's surrounding environment collected by the 360-degree surround view camera device;

[0064] It should be understood that the image information of the vehicle's surrounding environment collected by the 360-degree surround view camera device is obtained in real time.

[0065] The preprocessing module 130 is used to perform dedistortion preprocessing on the vehicle surrounding environment image data information to obtain a distortion-corrected image;

[0066] The above-mentioned vehicle surrounding environment image data information is pre-processed to remove distortion and obtain a corrected image, so that accurate surrounding environment image information can be obtained later, so that the cockpit end can make accurate control instructions to the vehicle end.

[0067] A stitching and fusion module 140 is configured to stitch and fuse the distortion-corrected images to obtain a panoramic image of the vehicle's surrounding environment.

[0068] It should be understood that the image data information of the vehicle's surrounding environment collected by the 360-degree surround-view camera device needs to be stitched before a panoramic image of the vehicle's surroundings can be obtained. In order to avoid ghosting and color difference caused by image stitching, image fusion is required to obtain a panoramic image of the vehicle's surroundings.

[0069] The sending module 150 is used to send the panoramic image of the vehicle's surrounding environment to the cockpit end, and the cockpit end can perform remote driving control based on the panoramic image of the vehicle's surrounding environment.

[0070] In an embodiment of the present invention, the vehicle side processes the image information of the vehicle's surrounding environment collected by the 360-degree surround-view camera device to obtain a panoramic image of the surrounding environment, and then sends it to the cockpit side so that the cockpit side can make corresponding control instructions based on the panoramic image of the surrounding environment to achieve accurate remote control of the vehicle side.

[0071] In summary, the vehicle side provided by the present invention achieves a communication connection with the cockpit side, and the cockpit side can send remote control commands to the vehicle side. The vehicle side can obtain panoramic image information of the vehicle side's surroundings through a 360-degree surround-view camera device and send this panoramic image information to the cockpit side so that the cockpit side can issue corresponding control commands and realize remote control of the vehicle side. Because this vehicle side can obtain accurate and clear panoramic image information of the surrounding environment, the cockpit side can remotely control the vehicle side, thereby making it possible for the vehicle side in the mining area to be unmanned.

[0072] In an embodiment of the present invention, the vehicle surrounding environment image data information is subjected to dedistortion preprocessing to obtain a distortion-corrected image, such as Figure 3 As shown, including:

[0073] S310, determining a distortion radius according to an offset of distorted pixel coordinates relative to a center in the vehicle surrounding environment image data information;

[0074] It should be understood that the calculation of the offset of the distorted pixel coordinates relative to the center and , and the distortion radius :

[0075] ,

[0076] ,

[0077] ,

[0078] ;

[0079] Among them, u and v both represent the pixel coordinate values on the image plane. The inverse tangent function represents the inverse trigonometric function.

[0080] S320 : Determine the coordinates of undistorted pixels in the vehicle surrounding environment image data information according to the correlation relationship between the distorted radius and the undistorted radius.

[0081] In the embodiment of the present invention, the coordinates of the pixel without distortion are further determined based on the correlation between the distorted radius and the undistorted radius.

[0082] Specifically, determining the undistorted pixel coordinates in the vehicle surrounding environment image data information according to the correlation relationship between the distorted radius and the undistorted radius includes:

[0083] The correlation between the distorted radius and the undistorted radius is determined, where the correlation between the two is expressed as follows:

[0084] ,

[0085] in, Indicates the distortion radius, 、 、 、 Represents the distortion coefficient of the camera device, where , , ;

[0086] The expression of the undistorted pixel coordinates is determined as:

[0087] ,

[0088] .

[0089] In an embodiment of the present invention, the distortion-corrected images are stitched and fused to obtain a panoramic image of the vehicle side, such as Figure 4 As shown, including:

[0090] S410, performing coordinate transformation on the distortion-corrected image to obtain distortion-corrected image information in a world coordinate system;

[0091] In the embodiment of the present invention, the corrected image pixels are mapped one by one to the bird's-eye view plane ( ).

[0092] First, convert the pixel coordinates into normalized coordinates. For the coordinates of a point on the image ( ), transform it into the normalized plane coordinate point :

[0093] ,

[0094] ,

[0095] ,

[0096] Then normalize the coordinates Convert to world coordinates:

[0097] ,

[0098] Assuming the ground is a two-dimensional plane with Z = 0, we get .

[0099] in 、 For pre-calibrated offline parameters, are the rotation matrix and the translation matrix, is the internal parameter matrix.

[0100] S420: performing image stitching based on the distortion-corrected image in the world coordinate system to obtain a stitched image of the vehicle's surrounding environment;

[0101] In the embodiment of the present invention, the specific method for image stitching is to generate a mask image of a required resolution corresponding to the four stitching areas of front, back, left, and right, and then perform image fusion on the areas where two of them overlap.

[0102] S430: Perform image fusion on the vehicle-side surrounding environment stitched images to obtain a vehicle-side panoramic image.

[0103] It should be understood that there will be some color difference and ghosting between the four stitched images. To achieve a better visual effect, image fusion is required. Suppose there are two images, Image A and Image B. The purpose of image fusion is to achieve a smooth transition between the overlapping areas.

[0104] In an embodiment of the present invention, performing image fusion on the vehicle-side surrounding environment stitched images to obtain a vehicle-side panoramic image includes:

[0105] 1) For any two stitched images to be fused in the vehicle-side surrounding environment stitched image, determining that a weight value of a region where one stitched image occupies a dominant position is 1, and a weight value of a region where the other stitched image occupies a dominant position is 0;

[0106] It should be understood here that, for example, the left and front cameras have overlapping areas. After the distortion processing and the top-down transformation are completed, the two rectangular images are mapped to the left and front of the mask image. At this time, assuming that the left image is A and the weight value is set to 1, and the right image is B and the weight value is set to 0, there is an overlapping area in the upper left corner. Then this area has both A and B parts, so it will be between 0 and 1. At this time, the weight value can be weighted and averaged according to the distance from A, thereby producing a gradual transition effect, making the image smooth and natural. For some dislocation effects, it can be achieved by improving the accuracy of external parameter calibration.

[0107] 2) For each pixel in the overlapping area, determine the fused pixel value as the weighted result of the pixel value in one stitched image and the pixel value in the other stitched image;

[0108] First, define a weight map of the same size as the input image. Set the weight value for the area where Image A should dominate to 1, and the weight value for the area where Image B should dominate to 0. For each pixel in the overlapping area, the final fused pixel value Result is the weighted sum of the pixel value A of Image A and the pixel value B of Image B:

[0109] ,

[0110] 3) Repeat the above process to obtain the pairwise fusion results of the vehicle-side surrounding environment stitching images;

[0111] 4) Obtain a panoramic image of the vehicle side based on the pairwise fusion results of the vehicle side surrounding environment stitching images.

[0112] Thus, through dedistortion, splicing, and fusion operations, a complete panoramic image is finally stitched together. The overall stitching time is calculated to be approximately 2 milliseconds. After completing the image stitching node processing flow, the panoramic image is encoded and compressed by the video streaming node and pushed to the cabin display window. It should be noted that the 360-degree panoramic vision algorithm integrated in this invention has an overall delay of 160 milliseconds and an image pixel of 720P, which meets the basic requirements of scenes such as mines.

[0113] In summary, the 360-degree surround-view camera device of the embodiment of the present invention can be specifically composed of sub-cameras in four directions: front, rear, left, and right of the vehicle, achieving a measured delay of about 160 milliseconds for end-to-end video processing (at a speed of 10 kilometers per hour, the error is 0.50 meters, of which the camera hardware takes about 80 milliseconds to acquire the image, the fisheye image dedistortion algorithm takes about 20 milliseconds to process, the image splicing takes 2 milliseconds to form a frame of panoramic image, and the vehicle-side video is transmitted to the cabin-side decoding and display for about 50 milliseconds, for a total of 160 milliseconds). This process can meet the scenario requirements of remote control between the cabin and the vehicle, thereby making unmanned driving possible for vehicles in the mining area.

[0114] As another embodiment of the present invention, a remote driving control system 1 for open-pit mine operation is provided, wherein Figure 5 As shown, it includes: a cockpit end 20 and a vehicle end 10, the cockpit end 20 is communicatively connected with the vehicle end 10, and the vehicle end includes the vehicle end mentioned above.

[0115] In an embodiment of the present invention, the cabin end 20 includes a cabin end gateway device, a cabin end control unit, a cabin end industrial computer and a cabin end human-computer interaction unit. The cabin end gateway device, the cabin end industrial computer and the cabin end human-computer interaction unit are all communicatively connected to the cabin end control unit.

[0116] The cabin-side gateway device is used to realize the communication connection between the cabin-side control unit and the vehicle side;

[0117] The cabin-end human-computer interaction unit is used to realize human-computer interaction between the cabin-end control unit and the cabin-end industrial computer;

[0118] The cabin-side control unit is used to receive and process the image information fed back by the vehicle side and process the image information to realize remote driving control of the vehicle side, and to control the cabin-side human-computer interaction unit to display the operating status of the vehicle side to the user;

[0119] The cabin-end industrial computer is used to receive user operation instructions according to the cabin-end human-computer interaction unit, and generate vehicle-end operation control signals according to the user operation instructions.

[0120] It should be understood that the cabin-side human-machine interaction unit can specifically include a triple screen and a touch screen. The touch screen login and vehicle selection program is responsible for pairing vehicles with the remote cockpit, enabling a single cabin to match multiple vehicles, improving operational efficiency and the human-machine interaction experience. The front-end triple screen program is responsible for decoding and rendering vehicle-side video data, ensuring the remote driver has a 360-degree, comprehensive view of the vehicle's surroundings.

[0121] The cabin-side control unit is further divided into a video processing unit, a control command processing unit, and a CAN wired controller. The cabin-side gateway device connects to the video processing unit and the control command processing unit via Ethernet. The video processing unit is also connected to the main driver's large screen via Ethernet, and the main driver's large screen is also connected to the control command processing unit via Ethernet. The CAN wired controller is connected to the cabin-side control unit via the CAN bus. The cabin-side gateway device effectively acts as a router, facilitating communication between the cabin and the external environment. The cabin-side gateway device ensures the effective forwarding of control commands between the vehicle and the cabin, ensuring that the underlying data control status of the vehicle is promptly fed back to the cabin, providing the driver with necessary operational guidance information. Furthermore, control commands issued from the cabin can be smoothly transmitted to the corresponding vehicle to direct loading and unloading operations.

[0122] The cabin-side control unit is the core device in the remote cockpit, primarily responsible for video decoding, CAN command processing, and forwarding. During remote takeover, the triple-screen display system provides real-time information on the vehicle's operating status and a panoramic video of the surrounding area, providing efficient and safe loading and unloading guidance to the remote operator. The cabin-side industrial computer controls key information such as the vehicle's forward and reverse movements, steering, turn signals, and wipers.

[0123] In the embodiment of the present invention, Figure 5 As shown, the remote driving control system for open-pit mine operations also includes a cloud 30, which is communicated with the cockpit end 20 and the vehicle end 10 respectively. The cloud 30 is used to store and manage data information of the cockpit end and the vehicle end, and to provide data update services for the cockpit end and the vehicle end.

[0124] It's important to note that the cloud integrates a data update service module, a task scheduling module, and a platform data management and fault logging module, as well as a data storage service module. The task scheduling module is responsible for forwarding processed scheduling tasks to designated vehicles to switch driving modes. The platform manages and binds vehicle and cabin information, and supports the recording and analysis of big data from both vehicle and cabin operations, enabling more efficient guidance for remote takeover operations.

[0125] Specifically, the specific working process of the remote driving control system for open-pit mine operations is as follows: the cockpit computer device transmits the information of the vehicle's forward, reverse, steering wheel, lights, wipers and other information control to the cabin hardware data processing program; the cockpit computer device processes and forwards CAN instructions, and transmits the information to the cabin computer device; the triple-screen program installed on the cockpit computer device is responsible for decoding and rendering the video and displaying it on the triple-screen; the cloud updates the vehicle status information on the touch screen computing device for users to log in and select the vehicle; the vehicle controller collects the surrounding information collected by the 360-degree surround view camera device After processing the environmental information, a panoramic image of the vehicle's surroundings is obtained (the 360-degree surround-view camera perceives the information around the vehicle in 360 degrees without blind spots and encodes and transmits it in video form through the vehicle-side controller). Therefore, the vehicle-side controller can be responsible for video preprocessing and encoding, command processing and forwarding; the video processing node encodes the above video information and pushes it to the cabin-side receiving window for display; the vehicle-side controller decodes the video, processes and forwards the CAN commands, and displays it in real time through the cockpit triple screen; the vehicle-side control node controls the vehicle's forward, backward, heading angle, acceleration and other information by receiving the operation control commands from the cockpit.

[0126] In summary, the remote driving control system for open-pit mine operations provided by the present invention has successfully achieved the ability to perform remote control driving operations in a 5G network environment by integrating the collaborative integrated architecture process of the vehicle side, cloud side and cockpit side. Since the vehicle side integrates 360-degree panoramic vision, it is convenient for safe driving in special places such as meeting vehicles, while enhancing the driver's immersive experience. At the same time, thanks to the use of 5G networks, the remote driving control system for open-pit mine operations in the embodiment of the present invention has shorter latency and better performance. In addition, the remote driving control system for open-pit mine operations of the present invention supports a "multi-vehicle multi-cabin" mode, that is, each cockpit can choose to use the vehicle information provided by the cloud data.

[0127] It will be understood that the above embodiments are merely exemplary embodiments for illustrating the principles of the present invention, and the present invention is not limited thereto. Those skilled in the art will appreciate that various modifications and improvements can be made without departing from the spirit and substance of the present invention, and such modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A vehicle terminal, characterized in that: include: A 360-degree surround-view camera device, a vehicle chassis, a vehicle-side controller, and an intelligent gateway, wherein the 360-degree surround-view camera device, the vehicle chassis, and the intelligent gateway are all communicatively connected to the vehicle-side controller, and the vehicle-side controller includes a remote driving control device for open-pit mine operations; The 360-degree surround view camera device is used to collect images of the vehicle's surrounding environment in real time; The vehicle chassis is used to adjust the vehicle's operating state according to the control instructions of the vehicle-side controller, and the vehicle's operating state includes at least forward, backward, heading angle adjustment and acceleration adjustment; The vehicle-side controller is used to generate corresponding control instructions according to the control signal of the cockpit side, and obtain a panoramic image of the vehicle-side surrounding environment according to the remote driving control device for open-pit mine operation and send the panoramic image of the vehicle-side surrounding environment to the cockpit side; The intelligent gateway is used to realize the communication between the vehicle-side controller and the cockpit end; The remote driving control device for open-pit mine operation includes: A startup module, used to start the vehicle body and the 360-degree surround view camera equipment mounted on the vehicle body according to the vehicle operation control instructions on the cockpit end; An acquisition module is used to acquire the image data information of the vehicle's surrounding environment collected by the 360-degree surround view camera device; A preprocessing module, configured to perform dedistortion preprocessing on the vehicle surrounding environment image data information to obtain a distortion-corrected image; A stitching and fusion module is used to stitch and fuse the distortion-corrected images to obtain a panoramic image of the vehicle's surrounding environment; The sending module is used to send the panoramic image of the vehicle's surrounding environment to the cockpit end, and the cockpit end can perform remote driving control based on the panoramic image of the vehicle's surrounding environment.

2. The vehicle end according to claim 1, characterized in that: Performing dedistortion preprocessing on the vehicle surrounding environment image data to obtain a distortion-corrected image, including: determining a distortion radius according to an offset of distorted pixel coordinates relative to a center in the vehicle surrounding environment image data information; The coordinates of the pixels without distortion in the vehicle surrounding environment image data are determined according to the correlation relationship between the distorted radius and the undistorted radius.

3. The vehicle end according to claim 2, characterized in that: Determining the coordinates of undistorted pixels in the vehicle surrounding environment image data information according to the correlation relationship between the distorted radius and the undistorted radius includes: The correlation between the distorted radius and the undistorted radius is determined, where the correlation between the two is expressed as follows: ， in, Indicates the distortion radius, 、 、 、 Represents the distortion coefficient of the camera device, where , , ; The expression of the undistorted pixel coordinates is determined as: ， 。 4. The vehicle end according to claim 1, characterized in that Performing image stitching and image fusion on the distortion-corrected images to obtain a vehicle-side panoramic image, including: Performing coordinate transformation on the distortion-corrected image to obtain distortion-corrected image information in a world coordinate system; Perform image stitching based on the distortion-corrected image in the world coordinate system to obtain a stitched image of the vehicle's surrounding environment; The vehicle-side surrounding environment stitching images are fused to obtain a vehicle-side panoramic image.

5. The vehicle end according to claim 4, characterized in that: Performing image fusion on the vehicle-side surrounding environment stitched images to obtain a vehicle-side panoramic image, including: For any two stitched images to be fused in the vehicle-side surrounding environment stitched image, determining that a weight value of a region where one stitched image occupies a dominant position is 1, and a weight value of a region where the other stitched image occupies a dominant position is 0; For each pixel in the overlapping area, determining a fused pixel value as a weighted result of a pixel value in one of the stitched images and a pixel value in the other stitched image; Repeat the above process to obtain the pairwise fusion results of the vehicle-side surrounding environment stitching images; The vehicle-side panoramic image is obtained based on the pairwise fusion results of the vehicle-side surrounding environment stitching images.

6. A remote driving control system for open-pit mine operations, characterized in that: include: The cockpit end and the vehicle end are communicatively connected to each other, and the vehicle end includes the vehicle end described in any one of claims 1 to 5.

7. The remote driving control system for open-pit mine operation according to claim 6, characterized in that: The cabin end includes a cabin end gateway device, a cabin end control unit, a cabin end industrial computer and a cabin end human-computer interaction unit, and the cabin end gateway device, the cabin end industrial computer and the cabin end human-computer interaction unit are all communicatively connected to the cabin end control unit. The cabin-side gateway device is used to realize the communication connection between the cabin-side control unit and the vehicle side; The cabin-end human-computer interaction unit is used to realize human-computer interaction between the cabin-end control unit and the cabin-end industrial computer; The cabin-side control unit is used to receive and process the image information fed back by the vehicle side and process the image information to realize remote driving control of the vehicle side, and to control the cabin-side human-computer interaction unit to display the operating status of the vehicle side to the user; The cabin-end industrial computer is used to receive user operation instructions according to the cabin-end human-computer interaction unit, and generate vehicle-end operation control signals according to the user operation instructions.

8. The remote driving control system for open-pit mine operation according to claim 6, characterized in that: It also includes a cloud, which is communicated with the cockpit end and the vehicle end respectively, and is used to store and manage data information of the cockpit end and the vehicle end, and to provide data update services for the cockpit end and the vehicle end.

Images