Control methods for autonomous vehicles, vehicles and storage media

Abstract

This application discloses a control method, device, equipment, and storage medium for an autonomous vehicle, relating to the technical field of autonomous driving. The method of this application includes: acquiring vehicle driving monitoring data of the autonomous vehicle; determining the driving action to be performed based on the vehicle driving monitoring data, wherein the vehicle driving monitoring data is obtained by fusing and stitching together several video data collected by multiple cameras; if the driving action to be performed is a first driving action, receiving a driving operation instruction from the autonomous vehicle and performing autonomous driving operation of the autonomous vehicle according to the driving operation instruction, wherein the first driving action is a precise driving action; if the driving action to be performed is a second driving action, receiving a driving operation instruction sent by a remote control platform and performing remote driving control operation of the autonomous vehicle according to the driving operation instruction, wherein the second driving action is a non-precise driving action.

Description

[0001] This invention patent application is a divisional application of Chinese invention patent application filed on April 30, 2021, with application number 202110487435.3 and titled "Control Method, Apparatus, Device and Storage Medium for Unmanned Vehicles". Technical Field

[0002] This invention relates to the field of autonomous driving technology, and in particular to a control, device, equipment and storage medium for an autonomous vehicle. Background Technology

[0003] In autonomous driving technology, the environment in which vehicles operate is often highly complex. Surrounding vehicles, pedestrians, other obstacles, road markings, and lane lines are all crucial factors in the vehicle's environment. To better detect the environment, current autonomous driving systems use multiple cameras to collect data about the vehicle's surroundings. These cameras then use this data to make autonomous driving decisions. However, without processing the data collected from these multiple cameras, accurate data cannot be provided for autonomous driving decisions, leading to a high rate of decision-making errors.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide a control method, device, equipment, and storage medium for autonomous vehicles, aiming to solve the technical problem that multiple cameras in the prior art cannot provide accurate data for autonomous driving decisions, resulting in a high decision-making error rate.

[0006] To achieve the above objectives, the present invention provides a control method for an autonomous vehicle, the method comprising the following steps:

[0007] Acquire several video data points collected by multiple cameras, and fuse and stitch the video data to obtain vehicle driving monitoring data for the autonomous vehicle.

[0008] The driving action to be performed is determined based on the vehicle driving monitoring data.

[0009] If the current action to be performed is the first driving action, then the driver receives the driving operation instructions from the autonomous vehicle and performs the autonomous driving operation of the autonomous vehicle according to the driving operation instructions.

[0010] If the current action to be performed is the second driving action, then the driver receives the driving operation instructions sent by the remote control platform and performs the remote driving control operation of the driverless car according to the driving operation instructions.

[0011] Optionally, the step of acquiring video data from multiple cameras and fusing and stitching the video data to obtain vehicle driving monitoring data for the autonomous vehicle includes:

[0012] Acquire several video data points from multiple cameras, and transform these video data points into the target coordinate system based on the installation positions of the multiple cameras to obtain several video data points to be processed;

[0013] The aforementioned video data to be processed are preprocessed to obtain a number of video data to be fused.

[0014] The plurality of video data to be fused are divided into regions, and each region is assigned a corresponding weight value according to a preset assignment strategy.

[0015] The video data to be fused are spliced ​​and fused according to the weight values ​​to obtain the vehicle driving monitoring data of the autonomous vehicle.

[0016] Optionally, the preprocessing of the plurality of video data to be processed to obtain a plurality of video data to be fused includes:

[0017] Obtain the preset distortion center and preset distortion coefficients corresponding to multiple cameras, wherein the preset distortion center and preset distortion coefficients are obtained in advance by calibrating the multiple cameras according to their own parameters using the straight line calibration method;

[0018] Based on the preset distortion center and preset distortion coefficient corresponding to the multiple cameras, the distortion correction is performed on the several video data to be processed to obtain several corrected video data.

[0019] The corrected video data is cropped to obtain several video data to be merged.

[0020] Optionally, the step of dividing the plurality of video data to be fused into regions and assigning corresponding weight values ​​to each region according to a preset assignment strategy includes:

[0021] The grid is set according to the image resolution corresponding to the several corrected video data;

[0022] The plurality of video data to be fused are divided into regions according to the grid to obtain video data after region division;

[0023] The degree of deformation of the video data in each region is determined based on the plurality of video data to be processed and the plurality of video data to be fused.

[0024] Each region is assigned a corresponding weight value based on the degree of deformation.

[0025] Optionally, the step of stitching and fusing the plurality of video data to be fused according to the weight values ​​to obtain vehicle driving monitoring data of the autonomous vehicle includes:

[0026] Determine multiple initial overlapping regions corresponding to several video data to be fused, and determine multiple initial regions corresponding to each video data to be fused within each initial overlapping region;

[0027] Compare the weight values ​​of multiple initial regions corresponding to the video data to be fused within each initial overlapping region, and select the initial region with the largest weight value as the target region.

[0028] Multiple target regions are stitched together to obtain the target overlapping region;

[0029] The vehicle driving monitoring data of the autonomous vehicle is obtained based on the several video data to be fused and the target overlapping area.

[0030] Optionally, obtaining the vehicle driving monitoring data of the autonomous vehicle based on the plurality of video data to be fused and the target overlapping region includes:

[0031] The target image is obtained by stitching together the several video data to be fused and the target overlapping region.

[0032] Obtain the average grayscale value of each video data to be fused within each initial overlapping region;

[0033] The brightness value of the target image is adjusted based on the average gray value to obtain the adjusted target image;

[0034] The vehicle driving monitoring data of the autonomous vehicle is obtained based on the adjusted target image.

[0035] Optionally, after dividing the plurality of video data to be fused into regions and assigning corresponding weight values ​​to each region according to a preset assignment strategy, the method further includes:

[0036] Acquire jitter data from multiple cameras;

[0037] When the jitter data exceeds a preset value, the weight value is adjusted according to the jitter data to obtain the adjusted weight value;

[0038] The video data to be fused is spliced ​​and fused according to the adjusted weight values ​​to obtain the vehicle driving monitoring data of the autonomous vehicle.

[0039] Furthermore, to achieve the above objectives, the present invention also proposes a control device for an autonomous vehicle, the control device comprising:

[0040] The acquisition module is used to acquire several video data collected by multiple cameras, and to fuse and stitch the video data to obtain vehicle driving monitoring data of the autonomous vehicle.

[0041] The judgment module is used to determine the driving action to be performed based on the vehicle driving monitoring data.

[0042] The autonomous driving module is used to receive driving operation instructions from the autonomous vehicle if the current task is to perform the first driving action, and to perform the autonomous driving operation of the autonomous vehicle according to the driving operation instructions.

[0043] The remote driving module is used to receive driving operation instructions sent by the remote control platform if the current action to be performed is a second driving action, and to perform remote driving control operations of the driverless car according to the driving operation instructions.

[0044] Furthermore, to achieve the above objectives, the present invention also proposes a control device for an autonomous vehicle, the control device comprising: a memory, a processor, and a control program for the autonomous vehicle stored in the memory and executable on the processor, the control program being configured to implement the control method for the autonomous vehicle as described above.

[0045] In addition, to achieve the above objectives, the present invention also proposes a storage medium storing a control program for an autonomous vehicle, wherein the control program for the autonomous vehicle, when executed by a processor, implements the control method for the autonomous vehicle as described above.

[0046] This invention acquires and merges video data from multiple cameras to obtain vehicle driving monitoring data for an autonomous vehicle. Based on this data, it determines the driving action to be performed. If the desired action is a first driving action, it receives driving operation instructions from the autonomous vehicle and executes the autonomous driving operation accordingly. If the desired action is a second driving action, it receives driving operation instructions from a remote control platform and executes remote driving control operations for the autonomous vehicle. By merging and merging data from multiple cameras and transmitting the merged data to a decision-making system, the system can determine the driving action based on the vehicle driving monitoring data, providing more accurate data for autonomous driving decisions and avoiding high decision-making error rates caused by asynchronous video data. Attached Figure Description

[0047] Figure 1This is a schematic diagram of the structure of the control device for an autonomous vehicle in the hardware operating environment involved in the embodiments of the present invention;

[0048] Figure 2 This is a flowchart illustrating the first embodiment of the control method for an unmanned vehicle according to the present invention.

[0049] Figure 3 This is a flowchart illustrating the second embodiment of the control method for an unmanned vehicle of the present invention;

[0050] Figure 4 This is a flowchart illustrating the third embodiment of the control method for an unmanned vehicle of the present invention;

[0051] Figure 5 This is a structural block diagram of the first embodiment of the control device for an unmanned vehicle of the present invention.

[0052] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0053] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0054] Reference Figure 1 , Figure 1 This is a schematic diagram of the control device structure of an autonomous vehicle in the hardware operating environment involved in the embodiments of the present invention.

[0055] like Figure 1 As shown, the control equipment of the autonomous vehicle may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0056] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the control equipment of an autonomous vehicle and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0057] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a control program for an autonomous vehicle.

[0058] exist Figure 1 In the control device of the autonomous vehicle shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the control device of the autonomous vehicle of the present invention can be set in the control device of the autonomous vehicle. The control device of the autonomous vehicle calls the control program of the autonomous vehicle stored in the memory 1005 through the processor 1001 and executes the control method of the autonomous vehicle provided in the embodiment of the present invention.

[0059] This invention provides a control method for an autonomous vehicle, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of a control method for an unmanned vehicle according to the present invention.

[0060] In this embodiment, the control method for the autonomous vehicle includes the following steps:

[0061] Step S10: Acquire several video data points collected by multiple cameras, and perform fusion and splicing processing on the several video data points to obtain vehicle driving monitoring data of the autonomous vehicle.

[0062] It is understood that the execution subject in this embodiment is the control device of the autonomous vehicle. The control device of the autonomous vehicle can be a computer installed on the autonomous vehicle, or other devices that can achieve the same or similar functions. This embodiment does not limit this. This embodiment uses a computer installed on the autonomous vehicle as the execution subject for explanation.

[0063] It is easy to understand that step S10 of this embodiment includes: acquiring several video data collected by multiple cameras, performing fusion and splicing processing on the several video data to obtain road condition information; acquiring several sensor information collected by vehicle sensors, and obtaining vehicle driving monitoring data of the autonomous vehicle based on the road condition information and the several sensor information.

[0064] It should be noted that the vehicle driving monitoring data includes: communication conditions, road condition information, driving speed, etc.; the communication conditions include 5G communication, GPS or Beidou satellite signals, etc.; the road condition information includes lane lines, traffic signs, traffic participants and obstacles, etc.; the driving speed refers to the vehicle's set driving speed, such as the speed of the vehicle during automatic driving is no more than 10 km / h; the speed during turning is no more than 5 km / h.

[0065] It is understood that the computer installed on the autonomous vehicle in this embodiment includes at least an onboard perception system and an onboard positioning, planning and decision control system. After the autonomous driving function is activated, the onboard perception system obtains the current driving environment status of the autonomous vehicle and sends the obtained data to the onboard positioning, planning and decision control system through communication methods such as onboard Ethernet. In addition to onboard Ethernet, the communication methods between onboard units can also be LVDS, USB, CAN bus, WIFI, 5G and other communication methods.

[0066] It should be understood that, in this embodiment, step S10 further includes: acquiring several video data collected by multiple cameras, performing fusion and stitching processing on the several video data to obtain a target image, and obtaining a clear view of the front (Q-degree field of view), forward S-range, lateral W-range, and rearward L-range of the driving vehicle based on the target image; and using the clear view as road condition information in the vehicle driving monitoring data of the autonomous vehicle.

[0067] It should be noted that the vehicle-mounted perception system mainly consists of a visual perception processing system and an ultrasonic radar processing system. The visual perception processing system comprises a panoramic surround-view system consisting of N high-definition fisheye wide-angle cameras, M high-definition front-view cameras, and a visual processing controller. High-definition video images captured by the panoramic surround-view system and the high-definition front-view cameras are transmitted to the visual processing controller. The visual processor processes all images to form a clear view of the vehicle's front (Q-degree field of view), forward (S-degree), lateral (W-degree), and rearward (L-degree) views, and transmits this view to a remote backend via 5G. The visual processor processes the video images and outputs target-level information to the vehicle-mounted positioning and planning decision-making control system. The visual processor has functions such as lane line recognition, traffic sign recognition, and recognition of traffic participants and obstacles. The ultrasonic radar processing system consists of 12 ultrasonic radars and a radar controller. It collects obstacle distance information of the vehicle, processes it, and outputs the distance and position information of the target objects to the vehicle-mounted positioning and planning decision-making control system.

[0068] It should be understood that in this embodiment, the visual processing controller of the visual perception processing system stitches and fuses the data collected by N high-definition fisheye wide-angle cameras and M high-definition front-view cameras to obtain a panoramic image. Based on the panoramic image, a clear view of the front (Q-degree field of view), forward S-range, lateral W-range, and rearward L-range of the vehicle is formed, thereby providing synchronous and accurate video images for the vehicle positioning and planning decision control system.

[0069] It should be noted that in this embodiment, when performing several video image stitching and fusion processes, one of the coordinate systems corresponding to the N high-definition fisheye wide-angle cameras and the M high-definition front-view cameras is selected as the target coordinate system. The data from the other cameras is transformed into the target coordinate system. In specific implementation, since the forward visual information is more important, the coordinate system corresponding to the high-definition front-view cameras is selected as the target coordinate system. Distortion correction is performed on the data collected by multiple cameras, the image is divided into regions, and weights are assigned to each region according to the degree of distortion. Image stitching and fusion are performed according to the weight values ​​corresponding to each region to obtain the vehicle driving monitoring data of the autonomous vehicle. When assigning weight values, the impact of bumps and body shaking during vehicle movement on video data stitching is also considered. The weight values ​​are adjusted according to the camera shaking data to make the fused and stitched data more accurate. In order to make the video data clearer, the brightness value of the fused and stitched image is adjusted to obtain a clear, accurate, and synchronized video image, providing the basis data for autonomous driving decision-making.

[0070] Step S20: Determine the driving action to be performed based on the vehicle driving monitoring data.

[0071] It should be noted that the decision-making unit in the vehicle positioning, planning and decision-making control system makes decision-making logic judgments on the received visual target signals, radar signals, positioning signals, route planning, remote monitoring and control system commands, etc., to determine the driving action that the driverless car should perform. For example, based on the received information, it can determine whether the current action to be performed is to move forward, turn left, turn right, change lanes or stop.

[0072] Step S30: If the current action to be performed is the first driving action, then receive the driving operation instructions from the driverless car and execute the autonomous driving operation of the driverless car according to the driving operation instructions.

[0073] In this embodiment, the first driving action refers to a precise driving action, such as steering wheel, accelerator, and brake movements. When the decision unit determines that a precise driving action is to be performed, it automatically receives the driving operation command from the autonomous vehicle and executes the autonomous driving operation according to the command. For example, if the action to be performed is braking, the autonomous vehicle receives the braking driving operation command automatically sent by the vehicle positioning, planning, and decision control system, and executes the braking operation according to the command. The vehicle positioning, planning, and decision control system mainly consists of a positioning module and a planning and decision module. The positioning module receives high-definition map positioning signals as primary positioning information and receives positioning signals from 5G base stations and surrounding environmental signals from the visual processing system for comprehensive auxiliary positioning correction.

[0074] Furthermore, if the current action to be performed is a first driving action, receiving the driving operation command from the autonomous vehicle and executing the autonomous driving operation of the autonomous vehicle according to the driving operation command further includes: generating a corresponding control command based on the first driving action; responding to the control command and executing the autonomous driving operation of the autonomous vehicle. In this embodiment, the decision unit determines that the current action to be performed is a precise driving action, generates a corresponding control command based on the precise driving action, and the onboard execution system responds to the control command to execute the autonomous driving operation of the autonomous vehicle. Specifically, the execution system receives control commands such as target vehicle speed, target driving torque, target braking torque, target gear, target steering angle, and steering angular velocity issued by the onboard positioning, planning, and decision control system, responds to the control commands in real time, and sends back the relevant control results. For example, if the current action to be performed is a deceleration operation, the onboard positioning, planning, and decision control system issues a control command to reduce the vehicle speed to 9 km / h, so that the autonomous vehicle adjusts its current speed to 9 km / h. The execution system consists of the vehicle's power output and transmission control system, braking control system, steering control system, etc.

[0075] Step S40: If the current action to be performed is the second driving action, then receive the driving operation instruction sent by the remote control platform, and perform the remote driving control operation of the driverless car according to the driving operation instruction.

[0076] In this embodiment, the second driving action refers to a non-precise driving action, such as starting or stopping. These actions can be determined not only by the autonomous vehicle based on the driving monitoring data, but also by monitoring whether the autonomous vehicle is in a dangerous situation via a remote control platform client or mobile app. When a dangerous situation occurs, a corresponding non-precise driving action is generated. For example, if a user of the remote control platform detects a vehicle suddenly changing lanes in front of the autonomous vehicle via a mobile app, an emergency stop operation is required; or if a vehicle running a red light appears on the left side when passing through an intersection, an emergency stop operation is required.

[0077] Because performing non-precise driving actions entirely on the autonomous vehicle would be difficult and unsafe, and would require high-specification, high-precision sensors, increasing costs, a new approach is adopted. When non-precise driving actions like parking are required, the vehicle automatically receives a parking command sent via 5G from a remote control platform. The driver then uses their own visual observation through the remote control platform, saving on the use of high-specification, high-precision sensors like LiDAR and avoiding excessively high costs associated with fully autonomous driving. The autonomous vehicle then executes the parking command. The remote control platform's stop command has the highest priority.

[0078] Furthermore, if the current action to be performed is a second driving action, then receiving the driving operation instruction sent by the remote control platform and performing the remote driving control operation of the autonomous vehicle according to the driving operation instruction also includes: receiving the driving operation instruction for the second driving action sent by the remote control platform using 5G, performing the remote driving control operation of the autonomous vehicle according to the driving operation instruction; and sending the execution result feedback information of the second driving action to the remote control platform so that the remote control platform can determine whether the autonomous vehicle has completed the driving operation instruction based on the execution result feedback information.

[0079] In this embodiment, the remote control platform mainly consists of an in-vehicle 5G communication module, a 5G base station, a 5G core network and regional network, a remote monitoring and cloud computing platform, a mobile phone, and an APP. When the remote control platform detects that the current autonomous vehicle needs to perform a non-precise driving action, such as a parking operation, the driver on the remote control platform sends a parking driving operation command to the autonomous vehicle via 5G. The autonomous vehicle executes the parking operation according to the parking operation command and sends the feedback information of the execution result to the remote control platform. The driver on the remote control platform determines whether the current autonomous vehicle has completed the parking driving operation command based on the feedback information of the execution result. If it has not been completed, the parking command is resent to the autonomous vehicle.

[0080] This embodiment acquires and merges video data from multiple cameras to obtain vehicle driving monitoring data for the autonomous vehicle. Based on this data, it determines the driving action to be performed. If the desired action is a first driving action, it receives driving operation instructions from the autonomous vehicle and executes the autonomous driving operation accordingly. If the desired action is a second driving action, it receives driving operation instructions from a remote control platform and executes remote driving control operations for the autonomous vehicle. By merging and merging data from multiple cameras and transmitting the merged data to the decision-making system, the system can determine the driving action based on the vehicle driving monitoring data. This provides more accurate data for autonomous driving decisions and avoids high error rates caused by asynchronous video data.

[0081] refer to Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the control method for an unmanned vehicle according to the present invention.

[0082] Based on the first embodiment described above, step S10 of the control method for an autonomous vehicle in this embodiment includes:

[0083] Step S101: Acquire several video data points collected by multiple cameras, and convert the video data points to the target coordinate system based on the installation positions of the multiple cameras to obtain several video data points to be processed.

[0084] Understandably, the target coordinate system can be the vehicle's coordinate system, with the center of the vehicle's front as the origin, or it can be the coordinate system corresponding to one of the multiple cameras. For example, the multiple cameras consist of four fisheye cameras and one front-view camera. The four fisheye cameras are respectively arranged in the front bumper, rear bumper, and left and right rearview mirrors, while the one front-view camera is arranged in the center of the windshield. The coordinate system corresponding to the front-view camera is selected as the target coordinate system. The coordinate system is transformed according to the installation position of each camera and the internal and external parameters of the camera to obtain several video data to be processed.

[0085] In practical implementation, to make data stitching and fusion easier, the cameras can be installed coaxially. For example, a bracket can be set up in the center of the front of the top of the vehicle roof, and four fisheye cameras and one forward-looking camera can be arranged on the bracket. The top of the camera position is no more than 2.4m above the ground so that the cameras can capture images from various angles, optimize the visual range of the camera images, and select the coordinate system corresponding to the forward-looking camera as the target coordinate system to convert the video data collected by the four fisheye cameras into the target coordinate system.

[0086] Step S102: Preprocess the plurality of video data to be processed to obtain a plurality of video data to be fused.

[0087] It is understood that the preprocessing in this embodiment includes distortion correction. Distortion correction has two main elements: distortion center and distortion coefficient. In specific implementation, the distortion center and distortion coefficient are determined by calibrating the camera. After acquiring video data, the video data is preprocessed according to the distortion center and distortion coefficient stored in the preset storage area to obtain several video data to be fused.

[0088] Specifically, step S102 includes: obtaining preset distortion centers and preset distortion coefficients corresponding to multiple cameras, wherein the preset distortion centers and preset distortion coefficients are calibrated in advance using a straight line calibration method based on the parameters of the multiple cameras themselves; performing distortion correction on the plurality of video data to be processed based on the preset distortion centers and preset distortion coefficients corresponding to the multiple cameras to obtain a plurality of corrected video data; and cropping the plurality of corrected video data to obtain a plurality of video data to be fused.

[0089] It should be noted that each camera has a different preset distortion center and preset distortion coefficient. In the actual implementation, each camera is calibrated in advance, and its corresponding preset distortion center and preset distortion coefficient are recorded. When the camera acquires video data, distortion correction is performed on the video data using the preset distortion center and preset distortion coefficient to obtain the corrected video data.

[0090] In practical implementation, the process of calibrating the camera to determine the preset distortion center can be as follows: The camera is used to photograph a checkerboard-shaped planar board from the front. The checkerboard planar board is used as the base viewpoint, and the captured image is used as the shooting viewpoint. A fundamental matrix is ​​established between the base viewpoint and the shooting viewpoint. During the shooting process, the checkerboard planar board is directly facing the camera. Therefore, based on the base viewpoint and the shooting viewpoint, the lines connecting the corresponding light points are determined, and the intersections with their respective planes are the poles between the base viewpoint and the shooting viewpoint, which are the distortion center points. Multiple distortion center coordinates are determined by capturing multiple checkerboard images, and the average value is taken to obtain the preset distortion center.

[0091] The process of calibrating a camera to determine the preset distortion coefficients can be as follows: capture a checkerboard image, determine the coordinates of the checkerboard intersections in the image through feature detection, and then calculate the preset distortion coefficients corresponding to the camera using a linear calibration method.

[0092] Step S103: Divide the several video data to be fused into regions respectively, and assign corresponding weight values ​​to each region according to a preset assignment strategy.

[0093] It is understandable that the process of region division can be carried out using a grid with a preset density. The preset density of the grid can be determined in advance by the user or based on the image resolution. By assigning weights to each region, each region of the image carries depth information, providing a data foundation for image data fusion.

[0094] Specifically, step S103 includes: setting a grid according to the image resolution corresponding to the plurality of corrected video data; dividing the plurality of video data to be fused into regions according to the grid to obtain video data after region division; determining the degree of deformation of video data in each region according to the plurality of video data to be processed and the plurality of video data to be fused; and assigning a weight value to each region according to the degree of deformation.

[0095] It should be noted that different cameras capture images with different resolutions, resulting in different grid sizes. For example, an image captured by camera A has a resolution of 704×576, and after correction, the resulting image size is 1840×1570 (related to the preset distortion coefficient). The image obtained after cropping has a resolution of 704×576. The process of setting the grid according to the image resolution can be done by dividing the image into 6400 8.8×7.2 regions based on a preset number, such as 6400.

[0096] Understandably, since the original video data and the image obtained after correction and cropping have the same resolution, the data before and after correction in each region are compared to determine the degree of image deformation. When each grid region is divided finely enough, the region farther away from the image center has a greater degree of deformation. In grid regions near the edge, the image data is completely inconsistent. Each region is assigned a corresponding weight value according to the degree of deformation. For example, the weight value of a region without deformation is 1, and the preset minimum value is 0.3. The weight value of the four corner edges of the image is assigned to 0.3. At the boundary center, the degree of image deformation is not large, and the weight value is assigned to 0.75, so as to achieve the assignment of corresponding weight values ​​to each region according to the degree of deformation.

[0097] Specifically, after step S103, the method further includes: acquiring jitter data corresponding to multiple cameras; when the jitter data exceeds a preset value, adjusting the weight value according to the jitter data to obtain an adjusted weight value; and performing splicing and fusion processing on the plurality of video data to be fused according to the adjusted weight value to obtain vehicle driving monitoring data of the autonomous vehicle.

[0098] It is understandable that shake data is data that characterizes camera shake. It can be measured by an inertial measurement unit. Shake data can be either the amplitude or frequency of shake. The preset value can be set according to the actual situation. When the shake data exceeds the preset value, it indicates that the camera shake is too large, which may lead to errors in the collected data. Therefore, the weight values ​​are adjusted according to the shake data. The shake situation is different depending on the installation position of the camera. The camera with large shake data will have a larger decrease in weight value, and the camera with small shake data will have a smaller decrease in weight value. For example, if the shake amplitude of camera A is greater than that of camera B, the weight value of each region in the video data A collected by camera A will decrease by 0.5, and the weight value of each region in the video data B collected by camera B will decrease by 0.2.

[0099] Step S104: Perform splicing and fusion processing on the several video data to be fused according to the weight values ​​to obtain vehicle driving monitoring data of the autonomous vehicle.

[0100] It should be understood that during the process of stitching, the areas captured by each camera overlap. After converting all the video data into a coordinate system, the video images are initially fused. However, the data in the overlapping areas are not yet determined. Therefore, the area with the largest weight value is selected as a component of the final image based on the weight value, thus obtaining the stitched and fused video data. The stitched and fused video data is used as the vehicle driving monitoring data for autonomous vehicles.

[0101] This embodiment acquires several video data points from multiple cameras, transforms these data points to a target coordinate system based on the camera installation locations, resulting in several video data points to be processed. These video data points are then preprocessed to obtain several video data points to be fused. Each of these video data points is then divided into regions, and each region is assigned a corresponding weight value according to a preset assignment strategy. Finally, the video data points are stitched and fused according to their weight values ​​to obtain vehicle driving monitoring data for the autonomous vehicle. By stitching and fusing the video data using different region weight values, the accuracy of the fused data is ensured. Furthermore, a strategy based on jitter data adjustment of the weight values ​​is proposed to avoid inaccuracies caused by jitter. The data point with the highest weight value is selected as the final image data, which is then transmitted to the decision-making system. This allows the decision-making system to determine the driving action to be performed based on the vehicle driving monitoring data, providing more accurate data for autonomous driving decisions and avoiding high decision-making error rates due to low video data accuracy.

[0102] refer to Figure 4 , Figure 4 This is a flowchart illustrating the fourth embodiment of the control method for an unmanned vehicle according to the present invention.

[0103] Based on the first embodiment described above, step S104 of the control method for the autonomous vehicle in this embodiment includes:

[0104] Step S1041: Determine multiple initial overlapping regions corresponding to several video data to be fused, and determine multiple initial regions corresponding to each video data to be fused within each initial overlapping region.

[0105] It should be understood that the camera installation location in this embodiment has multiple overlapping areas. Within the overlapping areas, each video data to be fused that overlaps has several initial areas. The initial overlapping area refers to the area where the overlapping video data to be fused overlaps. The initial area refers to the area that each video data to be fused is pre-divided according to the grid within the overlapping area. For example, if video data A and video data B overlap, the corresponding initial overlapping area includes initial area A1, initial area A2, ..., initial area A20, initial area B1, initial area B2, ..., initial area B20.

[0106] Step S1042: Compare the weight values ​​of multiple initial regions corresponding to the video data to be fused within each initial overlapping region, and select the initial region with the largest weight value as the target region.

[0107] It should be noted that this embodiment considers that the initial regions of the video data to be fused that overlap within the overlapping region correspond one-to-one and overlap. In this case, the initial region with the largest weight value is selected as the target region. The target overlapping region is then formed based on multiple target regions.

[0108] It is understandable that there may be situations where the initial regions do not correspond completely. In this case, step S1042 includes determining multiple initial regions corresponding to the video data to be fused that overlap under each initial overlapping region, assigning weight values ​​to each pixel in the region according to the weight values ​​of the initial region, comparing the weight values ​​of each overlapping pixel, and selecting the pixel with the largest weight value as the target pixel, thereby forming the target overlapping region.

[0109] Step S1043: Stitch together multiple target regions to obtain the target overlapping region.

[0110] Step S1044: Obtain vehicle driving monitoring data of the autonomous vehicle based on the plurality of video data to be fused and the target overlapping area.

[0111] It is understandable that for areas where there is no overlap, the vehicle driving monitoring data is the video data to be fused. For areas where there is overlap, the vehicle driving monitoring data is the target overlapping area. By splicing together several video data to be fused and the target overlapping area, the vehicle driving monitoring data of the autonomous vehicle is obtained.

[0112] Specifically, step S1044 includes: stitching together the plurality of video data to be fused and the target overlapping region to obtain a target image; obtaining the average grayscale value corresponding to each video data to be fused in each initial overlapping region; adjusting the brightness value of the target image according to the average grayscale value to obtain an adjusted target image; and obtaining vehicle driving monitoring data of the autonomous vehicle based on the adjusted target image.

[0113] It is understood that this embodiment proposes a method to improve the display quality of video image data. By obtaining the average grayscale value of the video data to be fused that overlaps in the overlapping area, the brightness value of the target image is adjusted according to the average grayscale value, thereby obtaining a video image with improved display quality.

[0114] This embodiment determines multiple initial overlapping regions corresponding to several video data sets to be fused, and further determines multiple initial regions corresponding to each video data set to be fused within each initial overlapping region. The weight values ​​of these initial regions are compared, and the region with the highest weight value is selected as the target region. Multiple target regions are then stitched together to obtain the target overlapping region. Based on the video data sets to be fused and the target overlapping region, vehicle driving monitoring data for the autonomous vehicle is obtained. By stitching and fusing video data using the weight values ​​of different regions, the accuracy of the fused data is ensured. The data with the highest weight value is selected as the final image data. Furthermore, a method for adjusting the image brightness value is proposed to improve the display quality of the video data. The final image data is then transmitted to the decision-making system, enabling the system to determine the driving action to be performed based on the vehicle driving monitoring data. This provides more accurate data for autonomous driving decisions and avoids high decision-making error rates caused by low video data accuracy.

[0115] Furthermore, this embodiment of the invention also proposes a storage medium storing a control program for an autonomous vehicle, which, when executed by a processor, implements the control method for the autonomous vehicle as described above.

[0116] Reference Figure 5 , Figure 5 This is a structural block diagram of the first embodiment of the control device for an unmanned vehicle of the present invention.

[0117] like Figure 5 As shown, the control device for an autonomous vehicle proposed in this embodiment of the invention includes:

[0118] The acquisition module 10 is used to acquire several video data collected by multiple cameras, and to perform fusion and splicing processing on the several video data to obtain vehicle driving monitoring data of the autonomous vehicle.

[0119] The judgment module 20 is used to determine the driving action to be performed based on the vehicle driving monitoring data.

[0120] The autonomous driving module 30 is used to receive the driving operation instructions of the autonomous vehicle if the current action to be performed is the first driving action, and to perform the autonomous driving operation of the autonomous vehicle according to the driving operation instructions.

[0121] The remote driving module 40 is used to receive driving operation instructions sent by the remote control platform if the current action to be performed is a second driving action, and to perform remote driving control operations of the driverless car according to the driving operation instructions.

[0122] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solutions of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any restrictions on this.

[0123] This embodiment acquires and merges video data from multiple cameras to obtain vehicle driving monitoring data for the autonomous vehicle. Based on this data, it determines the driving action to be performed. If the desired action is a first driving action, it receives driving operation instructions from the autonomous vehicle and executes the autonomous driving operation accordingly. If the desired action is a second driving action, it receives driving operation instructions from a remote control platform and executes remote driving control operations for the autonomous vehicle. By merging and merging data from multiple cameras and transmitting the merged data to the decision-making system, the system can determine the driving action based on the vehicle driving monitoring data. This provides more accurate data for autonomous driving decisions and avoids high error rates caused by asynchronous video data.

[0124] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this invention. In practical applications, those skilled in the art can select some or all of the workflow to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0125] In addition, for technical details not described in detail in this embodiment, please refer to the control method of the autonomous vehicle provided in any embodiment of the present invention, which will not be repeated here.

[0126] In one embodiment, the acquisition module 10 is further configured to acquire several video data collected by multiple cameras, convert the several video data to a target coordinate system based on the installation position of the multiple cameras to obtain several video data to be processed, preprocess the several video data to obtain several video data to be fused, divide the several video data to be fused into regions respectively, assign corresponding weight values ​​to each region according to a preset assignment strategy, and perform splicing and fusion processing on the several video data to be fused according to the weight values ​​to obtain vehicle driving monitoring data of the autonomous vehicle.

[0127] In one embodiment, the acquisition module 10 is further configured to acquire preset distortion centers and preset distortion coefficients corresponding to multiple cameras, perform distortion correction on the plurality of video data to be processed according to the preset distortion centers and preset distortion coefficients corresponding to the multiple cameras, obtain a plurality of corrected video data, and crop the plurality of corrected video data to obtain a plurality of video data to be fused.

[0128] In one embodiment, the acquisition module 10 is further configured to set a grid according to the image resolution corresponding to the plurality of corrected video data, divide the plurality of video data to be fused into regions according to the grid to obtain video data after region division, determine the degree of deformation of video data in each region according to the plurality of video data to be processed and the plurality of video data to be fused, and assign a weight value to each region according to the degree of deformation.

[0129] In one embodiment, the acquisition module 10 is further configured to determine multiple initial overlapping regions corresponding to a plurality of video data to be fused, and to determine multiple initial regions corresponding to each video data to be fused within each initial overlapping region, compare the weight values ​​corresponding to the multiple initial regions corresponding to each video data to be fused within each initial overlapping region, select the initial region with the largest weight value as the target region, stitch the multiple target regions together to obtain the target overlapping region, and obtain vehicle driving monitoring data of the autonomous vehicle based on the plurality of video data to be fused and the target overlapping region.

[0130] In one embodiment, the acquisition module 10 is further configured to stitch together the plurality of video data to be fused and the target overlapping region to obtain a target image, acquire the average grayscale value corresponding to each video data to be fused in each initial overlapping region, adjust the brightness value of the target image according to the average grayscale value to obtain an adjusted target image, and obtain vehicle driving monitoring data of the autonomous vehicle based on the obtained adjusted target image.

[0131] In one embodiment, the acquisition module 10 is further configured to acquire jitter data corresponding to multiple cameras, and when the jitter data exceeds a preset value, adjust the weight value according to the jitter data to obtain an adjusted weight value, and perform splicing and fusion processing on the plurality of video data to be fused according to the adjusted weight value to obtain vehicle driving monitoring data of the autonomous vehicle.

[0132] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0133] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0134] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0135] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A control method of an unmanned vehicle, characterized by, The method, applicable to autonomous vehicles and remote monitoring, includes: Acquire several video data points from multiple cameras, and transform these video data points into the target coordinate system based on the installation positions of the multiple cameras to obtain several video data points to be processed; The aforementioned video data to be processed are preprocessed to obtain a number of video data to be fused. The plurality of video data to be fused are divided into regions, and each region is assigned a corresponding weight value according to a preset assignment strategy. Acquire jitter data from multiple cameras; When the jitter data exceeds a preset value, the weight value is adjusted according to the jitter data to obtain the adjusted weight value; The video data to be fused is spliced ​​and fused according to the adjusted weight values ​​to obtain the vehicle driving monitoring data of the autonomous vehicle. The video data to be fused is spliced ​​and fused according to the weight values ​​to obtain the vehicle driving monitoring data of the autonomous vehicle, and the driving action to be performed is determined according to the vehicle driving monitoring data. If the first driving action to be performed is to receive the driving operation instructions from the autonomous vehicle, and execute the autonomous driving operation of the autonomous vehicle according to the driving operation instructions, wherein the first driving action is a precision driving action, and the precision driving action is the steering wheel, accelerator and brake actions. If the current action to be performed is the second driving action, then the driving operation instruction sent by the remote control platform is received, and the remote driving control operation of the driverless car is executed according to the driving operation instruction. The second driving action is a non-precise driving action, which is an start or stop action.

2. The control method for an unmanned vehicle according to claim 1, characterized in that, The steps of receiving driving operation instructions from the autonomous vehicle and executing the autonomous driving operation of the autonomous vehicle according to the driving operation instructions if the current action to be performed is the first driving action include: Generate corresponding control commands based on the first driving action; Responding to control commands, it executes autonomous driving operations for the driverless car.

3. The control method for an unmanned vehicle according to claim 1, characterized in that, The step of receiving driving operation instructions sent by the remote control platform and executing remote driving control operations of the autonomous vehicle according to the driving operation instructions if the current action to be performed is the second driving action includes: Receive driving operation instructions for the second driving action sent by the remote control platform using 5G, and execute remote driving control operations of the driverless car according to the driving operation instructions; The execution result feedback information of the second driving action is sent to the remote control platform so that the remote control platform can determine whether the driverless car has completed the driving operation command based on the execution result feedback information.

4. A car, characterized in that, The method includes a memory, a processor, and a control program for an autonomous vehicle stored in the memory and running on the processor, wherein the processor, when executing the control program for the autonomous vehicle, implements the steps of the method according to any one of claims 1 to 3.

5. A computer-readable storage medium, characterized in that, It stores a control program for an autonomous vehicle, characterized in that, when the control program for the autonomous vehicle is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.

Images