Automatic driving remote control system based on artificial intelligence

Abstract

The invention belongs to the field of remote control, and provides an artificial intelligence-based automatic driving remote control system, which comprises a controlled vehicle, a cloud module and a remote driving module, the controlled vehicle is used for acquiring a road condition video around the controlled vehicle; the cloud module is used for forwarding the road condition video to the remote driving module and detecting a danger prompt event on the road condition video; during detection, the obtained basic sub-image and the supplementary sub-image are preprocessed to obtain a preprocessed frame, and then the preprocessed frame is subjected to danger prompt event detection. The remote driving module is used for displaying a road condition video and a danger prompt event and acquiring a vehicle control instruction input by a driver; the cloud module is also used for forwarding the vehicle control instruction to the controlled vehicle; the controlled vehicle is further used for executing the vehicle control instruction. According to the invention, the efficiency of preprocessing the frames in the road condition video is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of remote control, and in particular to an automatic driving remote control system based on artificial intelligence. Background Art

[0002] Remote automatic driving of a vehicle means that the driver remotely controls the vehicle, remotely sends the control instructions for the vehicle to the controlled vehicle, and the vehicle controller executes the control instructions. During the process of remote automatic driving, since the driver is not in the vehicle, how to timely detect and warn of dangerous situations around the vehicle becomes an important issue. In the existing technology, multiple cameras can be set up around the vehicle to obtain road condition videos around the vehicle, and then the road condition videos can be transmitted to the cloud, which is then forwarded to the remote driving control terminal for display. In addition, by setting up a corresponding artificial intelligence detection algorithm in the cloud to detect whether a preset type of dangerous warning event appears in the road condition video, such as rear-end collision, running a red light, etc., the detected dangerous warning events are also pushed to the remote driving control terminal for display, and the driver makes a decision, thereby increasing the probability of the remote driver discovering dangerous events in a timely manner and the timeliness of handling dangerous warning events.

[0003] Existing AI detection algorithms typically require pre-processing, including noise reduction, before identifying traffic video. Since the time between a vehicle detecting a hazard and responding is typically short, efficiently pre-processing the video frames to ensure rapid hazard detection becomes a critical challenge. Summary of the Invention

[0004] In view of the above problems, the object of the present invention is to provide an artificial intelligence-based autonomous driving remote control system, comprising a controlled vehicle, a cloud module and a remote driving module; The controlled vehicle is used to obtain the road condition video around the controlled vehicle and send the road condition video to the cloud module; The cloud module is used to forward the road condition video to the remote driving module, and is used to detect danger warning events in the road condition video and send the detected danger warning events to the remote driving module; The detection of dangerous warning events in road condition videos includes: Each frame of the traffic video is preprocessed in the following manner to obtain a preprocessed frame: Calculate the computational complexity coefficient of the frame that needs to be preprocessed; Calculate the number Q of basic sub-images according to the computational complexity coefficient; Divide the frame to be preprocessed into Q basic sub-images and obtain T supplementary sub-images; The Q basic sub-images and the T supplementary sub-images are distributed to Q+T computing nodes for parallel computing, and the Q basic sub-images and the T supplementary sub-images are pre-processed in parallel to obtain the pre-processed Q basic sub-images and the processed T supplementary sub-images; Merge the preprocessed Q basic sub-images and the processed T supplementary sub-images to obtain a preprocessed frame; Input the pre-processed frames into a preset detection model to detect danger warning events; The remote driving module is used to display road condition videos and hazard warning events, as well as to obtain vehicle control commands input by the driver and send the vehicle control commands to the cloud module; The cloud module is also used to forward vehicle control commands to the controlled vehicle; The controlled vehicle is also used to execute vehicle control instructions.

[0005] Optionally, the controlled vehicle includes a shooting unit, a first communication unit and a controller unit; The shooting unit is used to obtain the road condition video around the controlled vehicle; The first communication unit is used to send the road condition video to the cloud module and to receive the vehicle control instructions sent from the cloud module; The controller unit is used to execute vehicle control instructions.

[0006] Optionally, the shooting unit includes a plurality of cameras arranged around the controlled vehicle; Each camera has a different shooting angle; Each camera is used to obtain road condition videos from its own shooting angle.

[0007] Optionally, the cloud module includes a forwarding unit and a detection unit; The forwarding unit is used to receive the road condition video from the controlled vehicle and forward the road condition video to the remote driving module; The detection unit is used to detect dangerous warning events in road condition videos; The forwarding unit is also used to send the detected danger warning event to the remote driving module; The forwarding unit is also used to receive vehicle control instructions from the remote driving module and send the vehicle control instructions to the controlled vehicle.

[0008] Optionally, the remote driving module includes a display unit, a second communication unit and an input unit; The second communication unit is used to communicate with the cloud module and receive road condition videos and danger warning events from the cloud module; The display unit is used to display the road condition video and the danger warning event; The input unit is used for the driver to input vehicle control instructions; The second communication unit is also used to send vehicle control instructions to the cloud module.

[0009] Optionally, the computational complexity coefficient is calculated using the following formula: in, represents the computational complexity coefficient, Represents the set of pixels contained in the frame that needs to be preprocessed. express The number of pixels in express The pixel i in the brightness image The gradient value of the corresponding pixel point in Indicates the standard value of the preset gradient value variance, 、 represents the weight parameter, Indicates the brightness image In , the number of pixels with pixel value j, Indicates the preset standard value of detail information, brightness image The image of the luminance component of the frame that needs to be preprocessed in the Lab color space.

[0010] Optionally, calculating the number Q of basic sub-images according to the computational complexity coefficient includes: Q is calculated using the following formula:

[0011] in, Indicates the maximum value of the set computational complexity coefficient, Indicates the maximum number of base sub-images to be set.

[0012] Optionally, dividing the frame to be preprocessed into Q basic sub-images includes: Divide the frame to be preprocessed into Q non-overlapping basic sub-images of the same size; The basis sub-image is divided into M rows and N columns on the pre-processed frame, .

[0013] Optionally, obtaining T supplementary sub-images includes: The variation coefficient of each basis sub-image except the basis sub-image in the last column; Sort the coefficients of variation from large to small, and store the top T basic sub-images into a set ; Get them separately The complementary sub-image corresponding to each basic sub-image in .

[0014] Optionally, the coefficient of variation is calculated as follows: For the basic sub-image at row m and column n , calculated using the following formula Coefficient of variation:

[0015] in, express The coefficient of variation, express The grayscale value of pixel k in, Represents the basic sub-image of pixel k at row m and column n+1 The grayscale value of the corresponding pixel point, express The total number of pixels in , .

[0016] The artificial intelligence-based automatic driving remote control system of the present invention, in the process of preprocessing frames in road condition videos, first calculates the complexity coefficient, then calculates the number of basic sub-images based on the complexity coefficient, then obtains supplementary sub-images based on the basic sub-images, and finally selects a corresponding number of computing nodes based on the total number of basic sub-images and supplementary sub-images, and preprocesses the basic sub-images and supplementary sub-images in parallel, thereby greatly improving the efficiency of preprocessing frames in road condition videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The present invention is further described with reference to the accompanying drawings. However, the embodiments in the accompanying drawings do not constitute any limitation to the present invention. A person skilled in the art can obtain other drawings based on the following drawings without creative effort.

[0018] Figure 1 , which is a diagram of an exemplary embodiment of the artificial intelligence-based automatic driving remote control system of the present invention.

[0019] Figure 2 , which is a diagram of an exemplary embodiment of a controlled vehicle of the present invention.

[0020] Figure 3 , which is a diagram of an exemplary embodiment of the remote driving module of the present invention. DETAILED DESCRIPTION

[0021] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0022] like Figure 1 In one embodiment shown, the present invention provides an artificial intelligence-based autonomous driving remote control system, comprising a controlled vehicle, a cloud module, and a remote driving module; The controlled vehicle is used to obtain the road condition video around the controlled vehicle and send the road condition video to the cloud module; The cloud module is used to forward the road condition video to the remote driving module, and is used to detect danger warning events in the road condition video and send the detected danger warning events to the remote driving module; The detection of dangerous warning events in road condition videos includes: Each frame of the traffic video is preprocessed in the following manner to obtain a preprocessed frame: Calculate the computational complexity coefficient of the frame that needs to be preprocessed; Calculate the number Q of basic sub-images according to the computational complexity coefficient; Divide the frame to be preprocessed into Q basic sub-images and obtain T supplementary sub-images; The Q basic sub-images and the T supplementary sub-images are distributed to Q+T computing nodes for parallel computing, and the Q basic sub-images and the T supplementary sub-images are pre-processed in parallel to obtain the pre-processed Q basic sub-images and the processed T supplementary sub-images; Merge the preprocessed Q basic sub-images and the processed T supplementary sub-images to obtain a preprocessed frame; Input the pre-processed frames into a preset detection model to detect danger warning events; The remote driving module is used to display road condition videos and hazard warning events, as well as to obtain vehicle control commands input by the driver and send the vehicle control commands to the cloud module; The cloud module is also used to forward vehicle control commands to the controlled vehicle; The controlled vehicle is also used to execute vehicle control instructions.

[0023] The artificial intelligence-based automatic driving remote control system of the present invention, in the process of preprocessing frames in road condition videos, first calculates the complexity coefficient, then calculates the number of basic sub-images based on the complexity coefficient, then obtains supplementary sub-images based on the basic sub-images, and finally selects a corresponding number of computing nodes based on the total number of basic sub-images and supplementary sub-images, and preprocesses the basic sub-images and supplementary sub-images in parallel, thereby greatly improving the efficiency of preprocessing frames in road condition videos.

[0024] Existing image preprocessing methods typically use a single computing node to directly preprocess the entire frame. This approach clearly cannot meet the time requirements for remote vehicle control. Because the time between a vehicle detecting a hazard warning event and responding is typically short, the cloud module must be able to quickly detect whether a frame contains a pre-defined hazard warning event, thereby providing a timely warning to the driver and ensuring the safety of remote vehicle control.

[0025] The present invention obtains a computational complexity coefficient and then adaptively obtains the number of basic sub-images based on the computational complexity coefficient, thereby achieving adaptive tracking changes between the number of basic sub-images and the complexity of the frame. The higher the complexity of the details of the frame, the larger the number of basic sub-images.

[0026] For frames with less complex details, the number of basic sub-images will be relatively small, and accordingly, the number of supplementary sub-images will also be relatively small. Then, the number of computing nodes that need to participate in the preprocessing process will be relatively small, resulting in a larger number of idle computing nodes, so that a larger number of computing nodes can participate in other steps of detecting danger warning events, further improving the efficiency of detecting danger warning events.

[0027] For frames with more complex details, the number of basic sub-images will be relatively large, and accordingly, the number of supplementary sub-images will also be relatively large. At this time, more computing nodes are required to perform preprocessing steps to ensure the accuracy of the preprocessing results.

[0028] Such a design maximizes the utilization of the computing nodes in the cloud module of the present invention, thereby improving the efficiency of detecting danger warning events.

[0029] Optional, such as Figure 2 As shown, the controlled vehicle includes a shooting unit, a first communication unit and a controller unit; The shooting unit is used to obtain the road condition video around the controlled vehicle; The first communication unit is used to send the road condition video to the cloud module and to receive the vehicle control instructions sent from the cloud module; The controller unit is used to execute vehicle control instructions.

[0030] Optionally, the first communication unit includes a 5G communication chip, which communicates with the cloud module through the 5G communication network to achieve low-latency control of the controlled vehicle.

[0031] Optionally, the controller unit may include controllers for various devices on the vehicle, including controllers for controlling lights, windows, engines and other devices.

[0032] Optionally, the vehicle control instructions may include instructions for controlling the speed, instructions for controlling the forward direction, instructions for controlling various electronic devices on the controlled vehicle, etc.

[0033] Optionally, the shooting unit includes a plurality of cameras arranged around the controlled vehicle; Each camera has a different shooting angle; Each camera is used to obtain road condition videos from its own shooting angle.

[0034] In one configuration, cameras may be set up in the front, rear, left, and right directions of the controlled vehicle.

[0035] Optionally, the cloud module includes a forwarding unit and a detection unit; The forwarding unit is used to receive the road condition video from the controlled vehicle and forward the road condition video to the remote driving module; The detection unit is used to detect dangerous warning events in road condition videos; The forwarding unit is also used to send the detected danger warning event to the remote driving module; The forwarding unit is also used to receive vehicle control instructions from the remote driving module and send the vehicle control instructions to the controlled vehicle.

[0036] Specifically, the detection unit detects danger warning events by inputting the preprocessed frames into a preset detection model. When a danger warning event of a preset type is detected, the forwarding unit sends the danger warning event to the remote driving module.

[0037] Types of danger warning events may include vehicle lane change warning events, red light running warning events, and preceding vehicle rapid braking warning events.

[0038] Optionally, the preset detection model may include a CNN-based detection model.

[0039] Optional, such as Figure 3 As shown, the remote driving module includes a display unit, a second communication unit and an input unit; The second communication unit is used to communicate with the cloud module and receive road condition videos and danger warning events from the cloud module; The display unit is used to display the road condition video and the danger warning event; The input unit is used for the driver to input vehicle control instructions; The second communication unit is also used to send vehicle control instructions to the cloud module.

[0040] Specifically, when the driver sees the danger warning event displayed on the display unit, he can make defensive driving for the controlled vehicle according to his own driving experience and send relevant vehicle control instructions to the controlled vehicle.

[0041] Optionally, the computational complexity coefficient is calculated using the following formula: in, represents the computational complexity coefficient, Represents the set of pixels contained in the frame that needs to be preprocessed. express The number of pixels in express The pixel i in the brightness image The gradient value of the corresponding pixel point in Indicates the standard value of the preset gradient value variance, 、 represents the weight parameter, Indicates the brightness image In , the number of pixels with pixel value j, Indicates the preset standard value of detail information, brightness image The image of the luminance component of the frame that needs to be preprocessed in the Lab color space.

[0042] In the above implementation, the computational complexity coefficient is primarily derived from a comprehensive calculation of the gradient value and detail content. A larger variance in the gradient value and a greater detail content indicate a greater number of locations within the frame requiring recognition. Therefore, the frame needs to be divided into smaller basic sub-images, meaning the number of basic sub-images needs to be increased to ensure accurate detection. Conversely, a smaller variance in the gradient value and a lower detail content indicate a greater number of background pixels within the frame. In this case, the number of basic sub-images can be reduced, effectively improving the overall efficiency of the hazard warning event recognition process while maintaining preprocessing accuracy.

[0043] Optionally, calculating the number Q of basic sub-images according to the computational complexity coefficient includes: Q is calculated using the following formula:

[0044] in, Indicates the maximum value of the set computational complexity coefficient, Indicates the maximum number of base sub-images to be set.

[0045] Specifically, the setting of formula Q enables the number of computational basis sub-images to be adaptively changed as the computational complexity coefficient changes.

[0046] Optionally, dividing the frame to be preprocessed into Q basic sub-images includes: Divide the frame to be preprocessed into Q non-overlapping basic sub-images of the same size; The basis sub-image is divided into M rows and N columns on the pre-processed frame, .

[0047] Optionally, obtaining T supplementary sub-images includes: Calculate the coefficient of variation of each basis sub-image except the last two columns of basis sub-images respectively; Sort the coefficients of variation from large to small, and store the top T basic sub-images into a set ; Get them separately The complementary sub-image corresponding to each basic sub-image in .

[0048] Specifically, the pixel value changes between adjacent basic sub-images are obtained by calculating the variation coefficient. The greater the pixel value change, the greater the probability that the area where the basic sub-image is located belongs to the foreground area. Therefore, a supplementary sub-image is obtained in the area to effectively alleviate the situation where the pixel value in the preprocessed frame changes suddenly due to non-overlapping cutting.

[0049] If the frames that need to be preprocessed are directly cut in an overlapping manner, the number of sub-images cut out will be very large, the number of computing nodes involved in the preprocessing step will increase, while the number of computing nodes involved in other steps will decrease, thus affecting the processing efficiency of the entire hazard warning event recognition process; Therefore, the present invention can effectively alleviate pixel mutation while ensuring the efficiency of the entire recognition process of danger prompt events.

[0050] Specifically, the value of T is calculated using the following formula:

[0051] Represents the preset computational complexity coefficient comparison value. The larger the computational complexity coefficient, the larger the value of T, so that T can adaptively change with the computational complexity of the frame.

[0052] Optionally, the coefficient of variation is calculated as follows: For the basic sub-image at row m and column n , calculated using the following formula Coefficient of variation:

[0053] in, express The coefficient of variation, express The grayscale value of pixel k in, Represents the basic sub-image of pixel k at row m and column n+1 The grayscale value of the corresponding pixel point, express The total number of pixels in , .

[0054] The variation coefficient is mainly obtained by subtracting the grayscale values ​​of two pixels with the same relative position in two adjacent sub-images to obtain the change of pixel values. The greater the change in pixel value, the greater the variation coefficient.

[0055] Optionally, the respective The complementary sub-image corresponding to each basic sub-image in includes: for The basic sub-image in ,Will The basic sub-image on the right is denoted as ; The base sub-image and The value range of the horizontal axis is recorded as and ; and Represent the basic sub-images The minimum and maximum values ​​of the horizontal axis, and Represent the basic sub-images The minimum and maximum values ​​of the horizontal axis; The following method is used to calculate the complementary sub-images in and The overlap ratio in:

[0056]

[0057] in, Indicates that the supplementary sub-image is in The overlap ratio in Indicates that the supplementary sub-image is in The overlap ratio in and Respectively and The coefficient of variation of The minimum and maximum values ​​of the horizontal coordinates of the supplementary sub-image are calculated using the following formula:

[0058]

[0059] in, Indicates the minimum value of the horizontal coordinate of the supplementary sub-image, Indicates the maximum value of the horizontal coordinate of the supplementary sub-image; The vertical coordinate value of the supplementary sub-image is divided into the basic sub-image The value range is the same.

[0060] Specifically, the supplementary sub-image is mainly obtained from two adjacent basic sub-images, and then the overlapping ratio between the supplementary sub-image and the basic sub-image is obtained according to the variation coefficient. The larger the variation coefficient, the larger the overlapping ratio between the supplementary sub-image and the basic sub-image, thereby achieving coverage of more foreground effective information and effectively reducing the degree of mutation of the pixel value changes between two adjacent basic sub-images after preprocessing.

[0061] Optionally, merging the preprocessed Q basic sub-images and the processed T supplementary sub-images to obtain a preprocessed frame includes: First, the basic image is composed of Q preprocessed basic sub-images ; Then, each preprocessed supplementary sub-image is compared with the base image Perform image fusion to obtain preprocessed frames.

[0062] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. An artificial intelligence-based autonomous driving remote control system, characterized in that: Including controlled vehicles, cloud modules and remote driving modules; The controlled vehicle is used to obtain the road condition video around the controlled vehicle and send the road condition video to the cloud module; The cloud module is used to forward the road condition video to the remote driving module, and is used to detect danger warning events in the road condition video and send the detected danger warning events to the remote driving module; The detection of dangerous warning events in road condition videos includes: Each frame of the traffic video is preprocessed in the following manner to obtain a preprocessed frame: Calculate the computational complexity coefficient of the frame that needs to be preprocessed; Calculate the number Q of basic sub-images according to the computational complexity coefficient; Divide the frame to be preprocessed into Q basic sub-images and obtain T supplementary sub-images; The Q basic sub-images and the T supplementary sub-images are distributed to Q+T computing nodes for parallel computing, and the Q basic sub-images and the T supplementary sub-images are pre-processed in parallel to obtain the pre-processed Q basic sub-images and the processed T supplementary sub-images; Merge the preprocessed Q basic sub-images and the processed T supplementary sub-images to obtain a preprocessed frame; Input the pre-processed frames into a preset detection model to detect danger warning events; The remote driving module is used to display road condition videos and hazard warning events, as well as to obtain vehicle control commands input by the driver and send the vehicle control commands to the cloud module; The cloud module is also used to forward vehicle control commands to the controlled vehicle; The controlled vehicle is also used to execute vehicle control instructions.

2. The artificial intelligence-based automatic driving remote control system according to claim 1, characterized in that: The controlled vehicle includes a shooting unit, a first communication unit and a controller unit; The shooting unit is used to obtain the road condition video around the controlled vehicle; The first communication unit is used to send the road condition video to the cloud module and to receive the vehicle control instructions sent from the cloud module; The controller unit is used to execute vehicle control instructions.

3. The artificial intelligence-based automatic driving remote control system according to claim 2, characterized in that: The shooting unit includes a plurality of cameras arranged around the controlled vehicle; Each camera has a different shooting angle; Each camera is used to obtain road condition videos from its own shooting angle.

4. The artificial intelligence-based automatic driving remote control system according to claim 1, characterized in that: The cloud module includes a forwarding unit and a detection unit; The forwarding unit is used to receive the road condition video from the controlled vehicle and forward the road condition video to the remote driving module; The detection unit is used to detect dangerous warning events in road condition videos; The forwarding unit is also used to send the detected danger warning event to the remote driving module; The forwarding unit is also used to receive vehicle control instructions from the remote driving module and send the vehicle control instructions to the controlled vehicle.

5. The artificial intelligence-based automatic driving remote control system according to claim 1, characterized in that: The remote driving module includes a display unit, a second communication unit and an input unit; The second communication unit is used to communicate with the cloud module and receive road condition videos and danger warning events from the cloud module; The display unit is used to display the road condition video and the danger warning event; The input unit is used for the driver to input vehicle control instructions; The second communication unit is also used to send vehicle control instructions to the cloud module.

6. The artificial intelligence-based automatic driving remote control system according to claim 1, characterized in that: The computational complexity coefficient is calculated using the following formula: in, represents the computational complexity coefficient, Represents the set of pixels contained in the frame that needs to be preprocessed. express The number of pixels in express The pixel i in the brightness image The gradient value of the corresponding pixel point in Indicates the standard value of the preset gradient value variance, 、 represents the weight parameter, Indicates the brightness image In , the number of pixels with pixel value j, Indicates the preset standard value of detail information, brightness image The image of the luminance component of the frame that needs to be preprocessed in the Lab color space.

7. The artificial intelligence-based automatic driving remote control system according to claim 6, characterized in that: The step of calculating the number Q of basic sub-images according to the computational complexity coefficient includes: Q is calculated using the following formula:

8. Among them, Indicates the maximum value of the set computational complexity coefficient, Indicates the maximum number of base sub-images to be set.

9. The artificial intelligence-based automatic driving remote control system according to claim 1, characterized in that: The frame to be pre-processed is divided into Q basic sub-images, including: Divide the frame to be preprocessed into Q non-overlapping basic sub-images of the same size; The basis sub-image is divided into M rows and N columns on the pre-processed frame, .

10. The artificial intelligence-based automatic driving remote control system according to claim 8, characterized in that: The acquiring of T supplementary sub-images comprises: The variation coefficient of each basic sub-image except the basic sub-image in the last column; Sort the coefficients of variation from large to small, and store the top T basic sub-images into a set ; Get them separately The complementary sub-image corresponding to each basic sub-image in .

11. The artificial intelligence-based automatic driving remote control system according to claim 9, characterized in that: The coefficient of variation is calculated as follows: For the basic sub-image at row m and column n , calculated using the following formula Coefficient of variation:

12. Among them, express The coefficient of variation, express The grayscale value of pixel k in Represents the basic sub-image of pixel k at row m and column n+1 The grayscale value of the corresponding pixel point, express The total number of pixels in , .

Images