Method and device for drawing model of panoramic imaging stitching image of remote control vehicle

Abstract

The invention discloses a method for drawing a model by stitching images for panoramic imaging of a remote-controlled vehicle, comprising: a remote control terminal receiving in real time a 2D panoramic image without a vehicle model and stitched by a built-in panoramic imaging system of the remote-controlled vehicle and transmitted from the remote-controlled vehicle via a wireless transmission device; or receiving independent images of cameras around the remote-controlled vehicle and internal and external parameter calibration data of each camera relative to the center of the vehicle body transmitted via the wireless transmission device, and calculating the timestamp difference of the image; calculating the relative displacement of the vehicle at the current moment relative to the time when the image is uploaded; if the image received in real time is an image without a vehicle model, the relative displacement is converted into a pixel offset to render and draw a vehicle model; if the image received in real time is an independent image of each camera, the external parameter calibration data of each camera relative to the center of the vehicle body is corrected according to the relative displacement, and then the image is stitched with the internal parameter data to render and draw the vehicle model.

Description

Technical Field

[0001] The present invention relates to the technical field of panoramic imaging stitching image drawing model of remote control vehicle, and in particular to a method and device for drawing a panoramic imaging stitching image model of remote control vehicle. Background Art

[0002] At present, for places such as remote mountainous areas or mines where mechanical operations are required, due to their special geographical environment, the operation of mechanical vehicles is all done by installing panoramic 360 cameras on the mechanical vehicles, transmitting the images through the 4G / 5G network to a remote control room in the distance, and the remote driver remotely controls the mechanical vehicles through the remote control terminal in the control room. For the remote control driver of the remote vehicle / vehicle, the perception of the vehicle environment is poor; at present, the 4G / 5G network is commonly used to transmit real-time images to the remote control terminal in the distance for presentation, which is used for real-time feedback of remote driving. However, if only the panoramic imaging system simply stitches the images and transmits them over long distances, the real-time feedback of remote driving will be poor due to the delay of long-distance image transmission and the inevitable delay of some other links; ultimately affecting the perception of the remote driver, the operation and driving are prone to deviation, and serious accidents will occur. Summary of the invention

[0003] In the face of relatively static closed scenes, the purpose of the present invention is to provide a method and device for obtaining a panoramic stitching image of the remote-controlled vehicle, ultimately improving the perception experience of the remote driver during driving, so as to prevent accidents.

[0004] The method for drawing a model for panoramic imaging stitching of a remote-controlled vehicle according to the present invention comprises:

[0005] The remote control terminal receives in real time the 2D panoramic image without vehicle model coverage, which is stitched by the built-in panoramic imaging system of the remote control vehicle and transmitted from the remote control vehicle via wireless transmission equipment; or receives the independent images of each camera around the remote control vehicle transmitted via wireless transmission equipment, as well as the internal and external parameter calibration data of each camera itself relative to the center of the vehicle body; the image data packet uploaded by the remote control vehicle shall additionally include the vehicle speed and acceleration at the time of image generation, and the timestamp obtained by calibration of the satellite timing system;

[0006] The remote control terminal calculates the timestamp difference between the current time and the time when the original image was generated based on the timestamp calibrated by the satellite timing system contained in the image data packet uploaded by the remote control vehicle and the system timestamp calibrated by the satellite timing system of the remote control terminal at the current time;

[0007] The remote control terminal calculates the relative displacement of the vehicle at the current moment relative to the time when the image is uploaded based on the above timestamp difference, the speed of the remotely controlled vehicle, and the acceleration of the remotely controlled vehicle;

[0008] If the image received in real time by the remote control terminal is an image without a vehicle model, the relative displacement is converted into a pixel offset, and the vehicle model is rendered on the received image according to the pixel offset;

[0009] If the images received in real time by the remote control terminal are independent images of each camera, the external parameter calibration data of each camera relative to the center of the vehicle body will be corrected according to the relative displacement; and the images uploaded by the remote control vehicle will be spliced ​​with the corrected external parameter calibration data and internal parameter data of each camera, and the vehicle model will be rendered on the spliced ​​image.

[0010] The device for drawing a model for panoramic imaging and stitching images of a remote-controlled vehicle described in the present invention comprises: installing a vehicle satellite timing system, a wireless transmission device, a plurality of cameras, and a motion sensing chip on the remote-controlled vehicle; obtaining the speed and acceleration of the vehicle when uploading the image at the current acquisition moment through satellite positioning and an inertial navigation module; calibrating the system time through the satellite timing system, and finally timestamping the video frame and the acceleration data; finally uploading the non-vehicle model image stitched by the panoramic imaging system or the independent image of each camera to a remote remote control terminal through the wireless transmission device; and additionally uploading the speed and acceleration of the vehicle at the moment of image generation, the timestamp calibrated by the satellite timing system, and the internal and external parameter calibration data of each camera relative to the center of the vehicle body to the remote control terminal;

[0011] A remote control terminal satellite timing system, a display device, and a wireless receiving device are installed on the remote control terminal, wherein the satellite timing system is used to calibrate the system time of the remote control terminal;

[0012] Afterwards, the remote control terminal receives in real time the 2D panoramic image without vehicle model coverage, which is stitched by the built-in panoramic imaging system of the remote control vehicle and transmitted from the remote control vehicle via the wireless transmission device, or the independent images of each camera and the internal and external calibration data of each camera itself relative to the center of the vehicle body; as well as the vehicle speed, acceleration, and satellite timing system timestamp at the time of image generation; and calculates the time difference based on the current timestamp of the remote control terminal and the timestamp uploaded by the remote control vehicle obtained above, and calculates the relative displacement of the vehicle at the current moment relative to the time when the image is uploaded based on the time difference, the uploaded speed, and the acceleration;

[0013] If the image received in real time is an image without a car model, the relative displacement of the vehicle at the current moment relative to the time when the image is uploaded is converted into a pixel offset, and the image is rendered and the vehicle model is drawn on the display device according to the pixel offset; if the image received in real time is an independent image of each camera, the external parameter calibration data of each camera relative to the center of the vehicle body is corrected according to the relative displacement of the system at the current moment relative to the time when the image is uploaded, the images of each camera are spliced ​​with the corrected external parameter calibration data and internal parameter data, and a 2D or 3D model of the vehicle is drawn, and finally the image is rendered on the display device.

[0014] As a preferred solution of the present invention, the multiple cameras are four cameras installed in the front, back, left and right of the vehicle that needs to be remotely controlled.

[0015] As a preferred solution of the present invention, the wireless transmission device is a 4G or 5G wireless transmission device; the wireless receiving device is a 4G or 5G wireless receiving device.

[0016] As a preferred solution of the present invention, the vehicle satellite timing system is a GPS or Beidou vehicle satellite timing system; the remote control terminal satellite timing system is a GPS or Beidou remote control terminal satellite timing system. Beneficial effects of the present invention:

[0017] The method and device for drawing a model for panoramic imaging and stitching images of a remote-controlled vehicle described in the present invention, the remote control terminal receives in real time a 2D panoramic image without vehicle model coverage, which is stitched by the built-in panoramic imaging system of the remote-controlled vehicle and transmitted by the remote-controlled vehicle through a wireless transmission device, or independent images of each camera, and internal and external parameter calibration data relative to the center of the vehicle body; and the speed, acceleration, and satellite timing system timestamp of the vehicle at the time of image acquisition. Since it is aimed at vehicles in remote mountainous areas or mines that need to perform mechanical operations and there are fewer vehicles and equipment around them (mostly slow-moving or static targets), the timestamp difference of the image is calculated based on the timestamp of the vehicle's satellite timing system and the timestamp of the satellite timing system of the remote-controlled terminal at the current moment, and then the displacement of the current vehicle relative to the time of image uploading is calculated based on the vehicle speed, acceleration and timestamp difference when uploading the image. If the real-time reception If the received image is an image without a vehicle model, the relative displacement of the vehicle at the current moment relative to the time when the image is uploaded is converted into a pixel offset, and the vehicle model is rendered on the image; if the image received in real time is an independent image of each camera, the external parameter calibration data of each camera relative to the center of the vehicle body is corrected according to the relative displacement of the vehicle at the current moment relative to the time when the image is uploaded, and the corrected external parameter calibration data of each camera and the internal parameter data are spliced ​​for the independent image of each camera, and then the vehicle model is rendered on the spliced ​​image. The remote driver can remotely control the vehicle in the remote mountainous area or mine according to the drawn vehicle model, which can greatly enhance the perception experience of the remote driver during the driving process, achieve the effect of efficiently transmitting video images, and solve the problem of time-consuming and delayed long-distance transmission of video images, which affects the remote driver's perception and causes accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of a method for drawing a model for panoramic imaging stitching of a remote-controlled vehicle provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0021] Figure 1 A flowchart of a method for drawing a model for panoramic imaging stitching of a remote-controlled vehicle provided by an embodiment of the present invention. Figure 1 As shown, the method includes:

[0022] The remote control terminal receives in real time the 2D panoramic image without vehicle model coverage, which is stitched by the built-in panoramic imaging system of the remote control vehicle and transmitted from the remote control vehicle via wireless transmission equipment; or receives the independent images of each camera around the remote control vehicle transmitted via wireless transmission equipment, as well as the internal and external parameter calibration data of each camera itself relative to the center of the vehicle body; the image data packet uploaded by the remote control vehicle shall additionally include the vehicle speed and acceleration at the time of image generation, and the timestamp obtained by calibration of the satellite timing system;

[0023] The remote control terminal calculates the timestamp difference between the current time and the time when the original image was generated based on the timestamp calibrated by the satellite timing system contained in the image data packet uploaded by the remote control vehicle and the system timestamp calibrated by the satellite timing system of the remote control terminal at the current time;

[0024] The remote control terminal calculates the relative displacement of the vehicle at the current moment relative to the time when the image is uploaded based on the above timestamp difference, the speed of the remotely controlled vehicle, and the acceleration of the remotely controlled vehicle;

[0025] If the image received in real time by the remote control terminal is an image without a vehicle model, the relative displacement is converted into a pixel offset, and the vehicle model is rendered on the received image according to the pixel offset;

[0026] If the images received in real time by the remote control terminal are independent images of each camera, the external parameter calibration data of each camera relative to the center of the vehicle body will be corrected according to the relative displacement; and the images uploaded by the remote control vehicle will be spliced ​​with the corrected external parameter calibration data and internal parameter data of each camera, and the vehicle model will be rendered on the spliced ​​image.

[0027] The remote driver can remotely control the vehicle in remote mountainous areas or mines according to the drawn vehicle model, which can greatly improve the remote driver's perceptual experience during driving, and can efficiently transmit video images, solving the problem of time-consuming and delayed long-distance transmission of video images, which affects the remote driver's perception and causes accidents.

[0028] If the images received in real time by the remote control terminal are independent images from each camera, the image stitching performed at the remote control terminal can adapt to the stitching of 3D or 2D panoramic images, making the visual effect of the stitched image better and further improving the perception experience of the remote driver during driving.

[0029] If the image received in real time by the remote control terminal is an image without a car model, it only needs to transmit one path of spliced ​​images, which puts less pressure on the bandwidth and further greatly improves the transmission efficiency of the video image.

[0030] An embodiment of the present invention provides a device for drawing a model for panoramic imaging and stitching images of a remote-controlled vehicle, comprising installing a vehicle satellite timing system, a wireless transmission device, a plurality of cameras, and a motion sensing chip on the remote-controlled vehicle; obtaining the speed and acceleration of the vehicle when uploading the image at the current acquisition moment through satellite positioning and an inertial navigation module; calibrating the system time through the satellite timing system, and finally timestamping the video frame and the acceleration data; finally uploading the non-vehicle model image stitched by the panoramic imaging system or the independent image of each camera to a remote remote control terminal through the wireless transmission device; and additionally uploading the vehicle speed, acceleration, and timestamp calibrated by the satellite timing system at the time of image generation, as well as the internal and external parameter calibration data of each camera relative to the center of the vehicle body to the remote control terminal;

[0031] A remote control terminal satellite timing system, a display device, and a wireless receiving device are installed on the remote control terminal, wherein the satellite timing system is used to calibrate the system time of the remote control terminal;

[0032] Afterwards, the remote control terminal receives in real time the 2D panoramic image without vehicle model coverage, which is stitched by the built-in panoramic imaging system of the remote control vehicle and transmitted from the remote control vehicle via the wireless transmission device, or the independent images of each camera and the internal and external calibration data of each camera itself relative to the center of the vehicle body; as well as the vehicle speed, acceleration, and satellite timing system timestamp at the time of image generation; and calculates the time difference based on the current timestamp of the remote control terminal and the timestamp uploaded by the remote control vehicle obtained above, and calculates the relative displacement of the vehicle at the current moment relative to the time when the image is uploaded based on the time difference, the uploaded speed, and the acceleration;

[0033] If the image received in real time is an image without a car model, the relative displacement of the vehicle at the current moment relative to the time when the image is uploaded is converted into a pixel offset, and the image is rendered and the vehicle model is drawn on the display device according to the pixel offset; if the image received in real time is an independent image of each camera, the external parameter calibration data of each camera relative to the center of the vehicle body is corrected according to the relative displacement of the system at the current moment relative to the time when the image is uploaded, the images of each camera are spliced ​​with the corrected external parameter calibration data and internal parameter data, and a 2D or 3D model of the vehicle is drawn, and finally the image is rendered on the display device.

[0034] Since the bottom of the vehicle is a blind spot that cannot be observed by the camera, a panoramic parking method with bottom-of-vehicle perspective can be used to infer the image of the bottom of the vehicle. This way, the overall spliced ​​image does not have any abrupt black areas, giving the driver a better sense and experience.

[0035] The multiple cameras are four cameras installed at the front, back, left and right of the remote-controlled vehicle, and the four cameras capture images around the vehicle.

[0036] The wireless transmission device is a 4G or 5G wireless transmission device; the wireless receiving device is a 4G or 5G wireless receiving device.

[0037] The vehicle satellite timing system is a GPS or Beidou vehicle satellite timing system; the remote control terminal satellite timing system is a GPS or Beidou remote control terminal satellite timing system.

[0038] When the remote control terminal receives in real time the independent images of each camera and the internal and external parameter calibration data of each camera relative to the center of the vehicle body transmitted by the wireless transmission equipment of the vehicle to be remotely controlled, the image stitching performed at the remote control terminal can adapt to the stitching of 3D or 2D panoramic images, making the visual effect of the stitching better and further improving the perception experience of the remote driver during driving.

[0039] The remote control terminal receives in real time the 2D panoramic image without vehicle model coverage, which is stitched by the built-in panoramic imaging system of the remote-controlled vehicle and transmitted by the vehicle to be remotely controlled through wireless transmission equipment. The remote control terminal only needs to receive and transmit the stitched picture, and the bandwidth pressure is relatively small, which can further enhance the perception experience of the remote driver during driving and achieve the effect of preventing accidents.

[0040] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the inventive concept of the present invention, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A method for drawing a model for panoramic imaging stitching of a remote-controlled vehicle, characterized in that: include: The remote control terminal receives in real time the 2D panoramic image without vehicle model coverage, which is stitched by the built-in panoramic imaging system of the remote control vehicle and transmitted from the remote control vehicle via wireless transmission equipment; or receives the independent images of each camera around the remote control vehicle transmitted via wireless transmission equipment, as well as the internal and external parameter calibration data of each camera itself relative to the center of the vehicle body; the image data packet uploaded by the remote control vehicle shall additionally include the speed of the remote control vehicle at the time of image generation, the acceleration of the remote control vehicle, and the timestamp obtained by calibration of the satellite timing system; The remote control terminal calculates the timestamp difference between the current time and the time when the original image was generated based on the timestamp calibrated by the satellite timing system contained in the image data packet uploaded by the remote control vehicle and the system timestamp calibrated by the satellite timing system of the remote control terminal at the current time; The remote control terminal calculates the relative displacement of the vehicle at the current moment relative to the time when the image is uploaded based on the above timestamp difference, the speed of the remotely controlled vehicle, and the acceleration of the remotely controlled vehicle; If the image received in real time by the remote control terminal is a 2D panoramic image without a vehicle model covering it, the relative displacement is converted into a pixel offset, and the vehicle model is rendered on the received image according to the pixel offset; If the images received in real time by the remote control terminal are independent images of each camera, the external parameter calibration data of each camera relative to the center of the vehicle body will be corrected according to the relative displacement; and the images uploaded by the remote control vehicle will be spliced ​​through the corrected external parameter calibration data and internal parameter calibration data of each camera, and the vehicle model will be rendered on the spliced ​​image.

2. A device for drawing a model by stitching images of panoramic imaging of a remote-controlled vehicle, characterized in that: include: The vehicle satellite timing system, wireless transmission equipment, multiple cameras, and motion sensing chips are installed on the remotely controlled vehicle; the vehicle speed and acceleration when uploading the image at the current acquisition moment are obtained through satellite positioning and inertial navigation modules; the satellite timing system is used to calibrate the system time, and finally the video frame and acceleration data are timestamped; finally, the 2D panoramic image without vehicle model coverage or the independent image of each camera spliced ​​by the panoramic imaging system is uploaded to the remote remote control terminal through the wireless transmission equipment; In addition, the speed and acceleration of the remote-controlled vehicle at the time of image generation, the timestamp calibrated by the satellite timing system, and the internal and external calibration data of each camera relative to the center of the vehicle body are uploaded to the remote control terminal; A remote control terminal satellite timing system, a display device, and a wireless receiving device are installed on the remote control terminal, wherein the satellite timing system is used to calibrate the system time of the remote control terminal; Afterwards, the remote control terminal receives in real time the 2D panoramic image without vehicle model coverage, which is stitched by the built-in panoramic imaging system of the remote control vehicle and transmitted from the remote control vehicle via the wireless transmission device, or the independent images of each camera and the internal and external parameter calibration data of each camera itself relative to the center of the vehicle body; as well as the speed, acceleration and timestamp of the remote control vehicle calibrated by the satellite timing system at the time of the above image generation; and calculates the time difference according to the timestamp of the remote control terminal at the current time and the timestamp calibrated by the satellite timing system uploaded by the remote control vehicle, and calculates the relative displacement of the remote control vehicle at the current time relative to the time when the image is uploaded according to the time difference and the uploaded speed and acceleration of the remote control vehicle; If the image received in real time is a 2D panoramic image without a vehicle model covering it, the relative displacement of the vehicle at the current moment relative to the time when the image is uploaded is converted into a pixel offset, and the image is rendered on the display device according to the pixel offset to draw the vehicle model; if the image received in real time is an independent image of each camera, the external parameter calibration data of each camera relative to the center of the vehicle body is corrected according to the relative displacement of the system at the current moment relative to the time when the image is uploaded, and the images of each camera are spliced ​​through the corrected external parameter calibration data and internal parameter calibration data of each camera, and a 2D or 3D model of the vehicle is drawn, and finally the image is rendered on the display device.

3. The device for drawing a model by stitching images of a panoramic imaging of a remote-controlled vehicle according to claim 2, characterized in that: The multiple cameras are four cameras installed on the front, back, left and right sides of the remotely controlled vehicle.

4. The device for drawing a model by stitching images of a panoramic imaging of a remote-controlled vehicle according to claim 2, characterized in that: The wireless transmission device is a 4G or 5G wireless transmission device; the wireless receiving device is a 4G or 5G wireless receiving device.

5. The device for drawing a model by stitching images of a panoramic imaging of a remote-controlled vehicle according to claim 2, characterized in that: The vehicle satellite timing system is a GPS or Beidou vehicle satellite timing system; the remote control terminal satellite timing system is a GPS or Beidou remote control terminal satellite timing system.

Images