Method and apparatus for assisted driving of a vehicle

Abstract

This disclosure proposes a method and apparatus for assisted driving of a vehicle. The method includes: acquiring vehicle steering data and an environmental image along the vehicle's direction of travel in real time; predicting the vehicle's real-time trajectory at a remote location based at least on the steering data; combining the predicted real-time trajectory with the corresponding environmental image to generate an augmented image; and presenting the augmented image to the driver in real time. This method and apparatus can improve the reliability, speed, and safety of remote vehicle control through augmented reality technology.

Description

Technical Field

[0001] This disclosure relates to augmented reality (AR) and driver assistance, and more specifically, to methods, apparatus, computer program products, and computer-readable storage media for assisting driving of vehicles by using augmented reality technology to predict the driving trajectory of remotely controlled vehicles. Background Technology

[0002] During remote vehicle control, drivers typically observe the vehicle's status using real-time images transmitted from the vehicle's control unit on a display screen, and then control the vehicle by manipulating the steering wheel. Because the driver is far from the actual environment and there are transmission delays in the transmitted video streams or images, the driver needs more information to ensure the real-time nature, speed, and safety of the control commands. Furthermore, adding predictive information about the vehicle's driving path to the display screen further enhances the security requirements for remote vehicle control.

[0003] There are many problems to be solved in migrating display systems that integrate images and video streams of the vehicle's local environment with predicted driving path information to remote control scenarios, making it difficult to simply apply such AR-assisted navigation systems to remote vehicle control assistance.

[0004] Therefore, there is a need to develop solutions for real-time driving trajectory prediction and driver assistance for remotely controlled vehicles. Summary of the Invention

[0005] This disclosure aims to address the problems mentioned above and proposes a solution for AR-assisted driving in applications involving remote vehicle control.

[0006] According to one aspect of this disclosure, a method for assisted driving of a vehicle located at a remote position away from the driver of the vehicle is proposed. The method includes: acquiring vehicle steering data and environmental images in the vehicle's direction of travel in real time; predicting a real-time trajectory of the vehicle at the remote position based at least on the steering data; combining the predicted real-time trajectory and the environmental image corresponding to the real-time trajectory to generate an enhanced image; and presenting the enhanced image to the driver in real time.

[0007] According to another aspect of this disclosure, a device for assisting driving of a vehicle located at a remote location away from the driver is provided. The device includes: a data acquisition unit configured to acquire in real time steering data of the vehicle and an environmental image in the direction of travel of the vehicle; a trajectory prediction unit configured to predict the real-time trajectory of the vehicle at the remote location based at least on the steering data; and an image rendering and presentation unit configured to combine the predicted real-time trajectory and the environmental image corresponding to the real-time trajectory to generate an enhanced image and to present the enhanced image to the driver in real time.

[0008] According to another aspect of this disclosure, an electronic device is proposed, including a processor and a memory for storing executable instructions of the processor, wherein the processor is configured to execute the executable instructions to implement the method described above.

[0009] According to another aspect of this disclosure, a computer-readable storage medium is provided that stores a computer program thereon, the computer program including executable instructions that, when executed by a processor, implement the method described above.

[0010] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described above.

[0011] By employing the methods and apparatus proposed in this disclosure, drivers can directly view the vehicle's real-time driving trajectory in a remote scenario on the display screen of their local terminal device when remotely controlling the vehicle. This enhances driving prediction capabilities, thereby improving the reliability, speed, and safety of remote vehicle control. Based on real-time trajectory prediction and presentation, safety prompts for vehicle operation can also be provided to the driver, further improving the safety of remote vehicle control. Attached Figure Description

[0012] The above and other features and advantages of this disclosure will become more apparent from a detailed description of exemplary embodiments thereof with reference to the accompanying drawings.

[0013] Figure 1 This is a schematic block diagram illustrating the relevant settings in a mapping matrix calibration scenario according to embodiments of the present disclosure.

[0014] Figure 2 This is a schematic flowchart illustrating the mapping matrix calibration process according to an embodiment of the present disclosure.

[0015] Figure 3 This is a schematic diagram illustrating the determination of a vehicle trajectory based on a vehicle dynamics model according to an embodiment of the present disclosure.

[0016] Figure 4 This is a schematic block diagram illustrating the operational logic of a vehicle's assisted driving process according to an embodiment of the present disclosure.

[0017] Figure 5 This is a schematic diagram of a display interface on a terminal device screen according to an embodiment of the present disclosure.

[0018] Figure 6 This is a schematic flowchart of a method for assisting driving of a vehicle according to an embodiment of the present disclosure.

[0019] Figure 7This is a schematic block diagram of a device for assisting driving of a vehicle according to an embodiment of the present disclosure.

[0020] Figure 8 This is a schematic block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0021] Exemplary embodiments will now be described more fully in conjunction with the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided to make the disclosure comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art. In the drawings, the dimensions of some elements may be exaggerated or modified for clarity. The same reference numerals in the drawings denote the same or similar structures, and therefore their detailed description will be omitted.

[0022] Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to enhance a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure may be practiced without one or more of the specific details described, or other methods, elements, etc., may be employed. In other instances, basic or well-known structures, methods, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0023] When remotely controlling a vehicle, the driver obtains information about the vehicle's surroundings and driving status through images and / or video streams, as well as other parameter information displayed on a terminal device. Typically, cameras and / or other types of sensors installed on the vehicle capture two-dimensional (2D) or three-dimensional (3D) environmental information for the driver's reference. Negative factors affecting control effectiveness include the driver's inability to directly observe the real-world environment around the vehicle for reference, and the inevitable latency in real-time image and video transmission, leading to further lag in determining control commands based on delayed reference information. The real-time requirements of remote control increase proportionally with vehicle speed. In addition to minimizing the latency of image and video stream transmission, it is also necessary to minimize the lag in the entire feedback mechanism that determines the remote control control effect based on information from image and video stream feedback. Compared to the transmission of images and video streams, data types such as vehicle state information (e.g., turning radius and wheel steering angle involved in vehicle dynamics models) and driver input of control data (e.g., vehicle steering data) require less data volume and bandwidth for transmission. This makes their bidirectional transmission between the remote vehicle and the local terminal device less affected by network speed and bandwidth. Therefore, predicting the driving trajectory of a remote vehicle based on this data offers superior real-time performance and timeliness compared to judging the vehicle's trajectory solely by observing real-time transmitted images and video streams. Even if there is a delay in the images or video streams displayed on the terminal device, the real-time trajectory predicted by fusing data from the vehicle dynamics model is sufficient to clearly and accurately indicate the corresponding vehicle driving trajectory response to the current control input, creating better conditions for the driver to adjust vehicle control commands more quickly and effectively.

[0024] One prerequisite for providing enhanced displays by fusing real-time vehicle trajectories predicted from vehicle status information acquired by sensors other than cameras and driver input with real-time images and video streams displayed on a screen is ensuring that the real-time trajectory can be accurately mapped to the corresponding pixel positions in the image. Therefore, it is first necessary to calibrate the relationship between the actual trajectory of the remote vehicle on the ground and the virtual image trajectory in the image.

[0025] This paper uses image data acquired by an in-vehicle camera as an example to illustrate the calibration process. Those skilled in the art will understand that the calibration process is not limited to mapping the actual trajectory onto an image acquired by a camera-type image acquisition device. 2D or 3D image data acquired by other types of optical sensors (e.g., visible light sensors or invisible light sensors suitable for infrared, ultraviolet wavelengths, etc.), distance sensors (e.g., LiDAR or ultrasonic sensors), and / or the fusion of multiple types of sensors can also be mapped and calibrated to the vehicle's actual trajectory. In-vehicle cameras include, but are not limited to, front-facing cameras located at the front of the vehicle for capturing images of the environment ahead, and rear-facing cameras located at the rear of the vehicle for capturing images of the environment behind.

[0026] The mapping relationship between the actual trajectory on the ground and the image trajectory in the image can be represented by a mapping matrix. In the example of using an in-vehicle camera to capture images of the environment around the vehicle, the mapping matrix is ​​mainly determined by the 3D perspective principle and the camera's internal and external parameters. The calculation of the 3D perspective principle mainly involves the relative positional relationship between the object of interest in the surrounding environment and the camera in 3D space. The camera's internal parameters include the camera's focal length, optical center coordinates, distortion coefficients, etc., while the external parameters include the camera's position and attitude parameters in the ground coordinate system. The position is represented by position coordinates, and the attitude parameters can be represented, for example, by a rotation matrix and a translation vector. To simplify the calibration process, the position of the in-vehicle camera can be fixed and the camera's internal and external parameters kept constant, so that the parameters of the mapping matrix do not change due to adjustments in the camera settings.

[0027] It can be understood that the mapping matrix corresponds to a specific camera. When monitoring the real-time trajectory in front of a vehicle on a terminal device that provides vehicle-assisted driving functions based on augmented reality technology, a mapping matrix calibrated between the actual trajectory and the image trajectory in the image of the vehicle's front captured by the front-facing camera can be used. When monitoring the real-time trajectory behind a vehicle (e.g., remotely controlling a vehicle to reverse), a mapping matrix calibrated between the actual trajectory and the image trajectory in the image of the vehicle's rear captured by the rear-facing camera can be used.

[0028] In vehicle calibration, calibration objects or calibrated points can be set as reference points. The relationship between the two-dimensional ground coordinate system of the vehicle's actual surrounding environment and the image coordinate system of the environmental image (2D image) captured by the vehicle-mounted camera, i.e., the size and dimension of the mapping matrix, determines the number of calibration objects required. According to embodiments of this disclosure, the dimension of the mapping matrix can be 3*3. The number of calibration objects is generally four or more. The mapping matrix can be calculated by obtaining the pixel coordinate data of the reference points of these calibration objects in the ground coordinate system and the corresponding reference points in the image coordinate system of the environmental image during the calibration process.

[0029] Figure 1 and Figure 2 This schematically illustrates the process of calibrating a mapping matrix in a calibration scenario.

[0030] like Figure 1 As shown in the left side, the calibration process can begin in step S210 by setting four calibration objects 101-1 to 101-4 on the ground in front of the vehicle at the calibration site, serving as four reference points in the ground coordinate system. Considering the feasibility and standardization of the calibration, calibration objects 101-1 and 101-2 can be grouped together, and calibration objects 101-3 and 101-4 can be grouped together. Specifically, the calibration objects in the group containing calibration objects 101-1 and 101-2 are collinear with the vehicle's left front and left rear wheels (or located on the extension lines of the left front and left rear wheels) and are positioned in front of the left front wheel. Similarly, the calibration objects in the group containing calibration objects 101-3 and 101-4 are collinear with the vehicle's right front and right rear wheels (or located on the extension lines of the right front wheel 103 and right rear wheel) and are positioned in front of the right front wheel 103. Based on the distance of the calibration objects relative to the front axle of the vehicle, calibration objects 101-1 and 101-4 are closer to the front axle, while calibration objects 101-2 and 101-3 are farther away. According to embodiments of this disclosure, for ease of implementation and calculation, the lines connecting calibration objects 101-1 and 101-4, as well as calibration objects 101-2 and 101-3, can be further arranged parallel to the front axle of the vehicle; that is, these calibration objects are respectively positioned on at least two parallel lines parallel to the front axle. With this specific arrangement, the four calibration objects 101-1 to 101-4 are located at the four vertices of a rectangular shape. When using a rear-facing camera, the lines connecting calibration objects 101-1 and 101-4, as well as the lines connecting calibration objects 101-2 and 101-3, can be parallel to the rear axle of the vehicle. When the front and rear axles of the vehicle are parallel, the lines connecting these calibration objects are parallel to both axles. The front-facing camera 102 can be located at the center of the front of the vehicle, so that the calibration objects 101-1 and 101-4, as well as the calibration objects 101-2 and 101-3, are symmetrical to the camera 102 on the left and right sides, respectively.

[0031] Those skilled in the art will understand that although specific calibration object settings for ease of implementation and calculation are given above, the position of the calibration object can be arbitrarily set as long as the mapping matrix can be determined by its coordinates in the ground coordinate system and the image coordinate system.

[0032] The 2D ground coordinate system of the vehicle's surrounding environment can use the center of the vehicle's front axle (the center of the line connecting the two front wheels) as the origin, the front axle as the horizontal axis, and the horizontal direction perpendicular to the front axle as the vertical axis. In reality, the position of the front-facing camera may be offset relative to the center of the vehicle's front axle. During calibration, this distance can be incorporated into the camera's external parameters to simplify the calculation of the mapping matrix. Thus, in step S220, based on the relative position (including relative direction and relative distance) of each calibration object's reference point relative to the center of the vehicle's front axle (the origin), the ground position coordinates of each calibration object can be measured and determined. For example, the ground position coordinates of calibration objects 101-1 to 101-4 are P1, P2, P3, and P4, respectively. Each coordinate is represented by a two-dimensional coordinate array (x, y) formed by the horizontal coordinate value x and the vertical coordinate value y. The ground position coordinates of multiple calibration objects constitute the ground position data A1 = (P1, P2, P3, P4).

[0033] Next, in step S230, the vehicle-mounted camera is used to capture and acquire a 2D environmental image 104 of the area in front of the vehicle, including the markers 101-1 to 101-4, such as... Figure 1 The rectangular image on the right side of the image is shown. The three-dimensional perspective effect causes the calibration objects in the environmental image 104 to appear as if they are spatially compressed from near to far. The image coordinate system of the environmental image can be set with the pixel at the top left corner of the image as the origin, the horizontal axis as the horizontal direction, and the vertical axis as the vertical direction. In step S240, the image position coordinates p1, p2, p3, and p4 of each of the calibration objects 101-1 to 101-4 in the environmental image 104 are obtained by measuring the horizontal and vertical coordinates of the pixel at each image position relative to the origin. These image position coordinates constitute the image position data A2 = (p1, p2, p3, p4) of the calibration objects.

[0034] In step S250, a mapping matrix M from the ground coordinate system of the surrounding environment to the image coordinate system of the vehicle-mounted camera can be calculated based on ground position data A1 and image position data A2. According to embodiments of this disclosure, algorithms such as linear equations, curve fitting, or linear regression can be used to calculate the mapping matrix. If the vehicle has multiple cameras, the mapping matrix from the ground coordinate system of the surrounding environment to the image coordinate system of that camera can be calculated separately for each camera. According to embodiments of this disclosure, the camera mapping matrix can be pre-calculated and stored in the remote vehicle or in a local terminal device for later retrieval before remote control of the vehicle.

[0035] After calibration, the actual trajectory of a remote vehicle on the ground can be predicted in real time based on the vehicle's steering data, size, and driving status during remote vehicle control. Real-time trajectory prediction is based on vehicle kinematics principles. For example... Figure 3 As shown in the schematic diagram of the dynamic model, the actual driving trajectory of the vehicle on the ground is represented by the movement path of the vehicle's outline through a rectangular region centered at the vehicle's center C and with the four wheels as vertices. The center C of the rectangular region can be represented by the midpoint of the line connecting the midpoints of the front and rear axles, or by the vehicle's geometric center. The front and rear axles are respectively taken as the short sides of the rectangular region (assuming that the lengths of the front and rear axles are the same or approximately the same), and the length of the short side of the rectangular region is the wheelbase B. The lines connecting the left front wheel to the left rear wheel and the right front wheel to the right rear wheel are taken as the two long sides of the rectangular region, and the length of the long side of the rectangular region is the wheelbase L between the front and rear axles. The wheelbase B and wheelbase L constitute the vehicle's dimensional data.

[0036] During vehicle operation, the left front wheel 301 and right front wheel 303 rotate around point O as their center of rotation or the center of a circle, with the distance R from the vehicle center C to the center of rotation O as the turning radius. The instantaneous steering angle of the left front wheel 301 is δ. l This guides the left front vertices of the rectangular area to form the dashed curve 302's driving trajectory. The instantaneous steering angle of the right front wheel 302 is δ. r The right front apex of the rectangular area is guided to form the dashed curve 304, which represents the vehicle's trajectory. The vehicle's steering data comes from vehicle control data input by the driver on their local terminal device. This vehicle control data includes steering data represented by the steering wheel's rotation angle, and / or acceleration and deceleration commands. Based on the steering data, the instantaneous steering angles of the left front wheel 301 and right front wheel 303 in the vehicle dynamics model described above can be calculated as real-time steering angle data. The driving direction indicates whether the remote vehicle is moving forward or backward. The vehicle's driving direction and turning radius constitute the vehicle's driving state data. Combining the real-time steering angle data, turning radius, and driving direction, the direction of movement of the rectangular area representing the vehicle's outline can be determined. The direction of movement of the rectangular area, combined with the geometric data of the rectangular area's outline—that is, the vehicle's dimensions—can ultimately determine the remote vehicle's real-time driving trajectory on the ground. Figure 3 The parameters involved in the vehicle dynamics model shown are all measured and calculated in real time, so the actual driving trajectory of the vehicle determined can be used to predict the real-time trajectory of the vehicle.

[0037] Figure 4 The mapping matrix M and obtained based on the calibration above are shown. Figure 3 The diagram illustrates the process of using a vehicle kinematics model to assist driving a remote vehicle.

[0038] The vehicle's assisted driving system first obtains the vehicle's steering data from the driver's input vehicle control data in step S410 and calculates the steering angle of each front wheel in the vehicle dynamics model. Then, in step S420, the steering angle, vehicle size, and vehicle driving status are input into the vehicle dynamics model to calculate the real-time trajectory of the remote vehicle on the ground.

[0039] Next, in step S430, the predicted real-time trajectory is mapped to a virtual image trajectory, which can be represented by image pixels in the image coordinate system, using a mapping matrix M. Simultaneously, in step S440, a real-time environmental image of the vehicle's surroundings is acquired by the vehicle's front-facing camera. Step S450 receives the virtual image trajectory and real-time environmental image provided in steps S430 and S440, and uses augmented reality technology to draw or overlay the virtual image trajectory onto the environmental image in real time, generating an enhanced environmental image that combines real environmental image information with the virtual image trajectory. To facilitate remote interaction with the driver, step S460 renders the enhanced environmental image on the display screen of the driver's local terminal device.

[0040] According to embodiments of this disclosure, for example Figure 4 The data processing tasks shown in steps S410 to S430—acquiring vehicle steering data, calculating the vehicle's real-time trajectory based on the steering data and vehicle driving status information from a remote vehicle, and generating an image trajectory that can be superimposed on the environmental image based on a pre-calibrated mapping matrix—can be performed by a terminal device located at the driver's location. In this way, the data information transmitted and exchanged between the remote vehicle and the local terminal device, besides the real-time environmental image acquired by the camera, only involves relatively small amounts of vehicle manipulation data (sent from the terminal device to the vehicle) and vehicle driving status data (sent from the vehicle to the terminal device), including steering data. This minimizes or eliminates the negative impact of remote data transmission on the signal transmission link for remote vehicle control. Furthermore, the data processing task of drawing and superimposing virtual image trajectories onto the environmental image to generate enhanced images can also be performed by the terminal device, fully utilizing the powerful data processing capabilities of the driver's local terminal device.

[0041] Figure 5 This illustrates an exemplary display interface presented on a driver's terminal device screen. In the environmental image on the right side of the figure, a predicted real-time vehicle trajectory 501, filled in blue, is overlaid on the ground. This enhanced environmental image, presented to the driver in real time, allows the driver to observe the vehicle's actual trajectory in response to the input vehicle operation data, enabling driving path prediction and driver assistance functions. Furthermore, other areas of the display screen or interface can provide features such as... Figure 5The left-hand screen displays vehicle trajectory safety information, with the highlighted red dotted line 502 showing the outline of the predicted real-time vehicle trajectory, and 503 and 504 indicating obstacles such as pedestrians and roadside trees, respectively. These safety prompts further help drivers understand the relative position of the vehicle to other objects and factors in the environment, and assess the safety of the input vehicle handling data.

[0042] Figure 6 An exemplary flowchart illustrates a method for assisting a vehicle according to an embodiment of the present disclosure. Wherein, in conjunction with... Figures 1 to 5 Details identical or similar to those described in the section on mapping matrix calibration and vehicle-assisted driving processes will not be repeated.

[0043] In step S610, this method acquires the steering data of a remote vehicle in real time through the input unit or user interface of a terminal device, and acquires environmental images in the vehicle's driving direction in real time through an onboard camera or other image acquisition unit. The acquisition of steering data and environmental images can be performed independently or in parallel.

[0044] Then, in step S620, the real-time driving trajectory of the vehicle on the ground in the remote scene is predicted based at least on steering data. For example, when predicting the real-time trajectory of the vehicle based on the vehicle dynamics model, steering data, stored or pre-acquired vehicle size data, and the real-time acquired current driving state of the vehicle can be used. The vehicle size can be calculated from the wheelbase and track width of the vehicle, which are the long and wide sides of the rectangular region used to define the vehicle's outline. The vehicle's driving state can include, for example, the vehicle's center position, driving direction, and turning radius.

[0045] Step S630 combines the predicted real-time driving trajectory with the corresponding temporal environmental image to generate an augmented reality-based image. For each ground location in the real-time driving trajectory, a mapping matrix is ​​used to map the ground location to the corresponding pixel location in the environmental image, thus transforming the ground driving trajectory into a virtual image trajectory. The virtual image trajectory is then fused with the environmental image through image overlay or rendering to obtain the augmented reality-enhanced environmental image.

[0046] Finally, in step S640, an enhanced image, augmented with the virtual image trajectory, is presented to the driver in real time. To improve the assistance effect of remote control, further safety prompts can be provided to the driver based on the enhanced image, indicating the relative spatial and positional relationships between the predicted real-time vehicle trajectory and other objects and objects around the vehicle.

[0047] Figure 7A schematic block diagram of a device for assisting driving of a vehicle according to the present disclosure is shown. The device 700 may include a data acquisition unit 710, a trajectory prediction unit 710, and an image rendering and presentation unit 730.

[0048] The data acquisition unit 710 is used to acquire, in real time, steering data for manipulating a remote vehicle and environmental images of the vehicle in its driving direction. The data acquisition unit 710 may further include an input component that receives vehicle manipulation data (including steering data) from the driver and a receiving component that receives environmental images from the vehicle's camera and / or images of the vehicle's driving status. The receiving component can receive data from the remote vehicle via the Internet or a dedicated network, either wired or wirelessly.

[0049] The trajectory prediction unit 720 is used to predict the real-time driving trajectory of the vehicle at a remote location based at least on steering data. The image rendering and presentation unit 730 is used to combine the predicted real-time trajectory with environmental images captured by a camera corresponding to the real-time trajectory to generate an enhanced image and to present the enhanced image to the driver in real time.

[0050] At least one of the data acquisition unit 710, trajectory prediction unit 720, and image rendering and presentation unit can be located at the driver's local location. The data acquisition unit 710 may also include an onboard camera at a remote vehicle, enabling the data acquisition unit 710 to acquire all real-time and non-real-time data required for remote vehicle assisted driving and remote control using augmented reality technology.

[0051] By employing the methods and devices disclosed herein, drivers can directly view the vehicle's real-time driving trajectory in a remote scenario and determine the spatial relationship between the vehicle and its surrounding environment from the display screen of their local terminal device when remotely controlling the vehicle. This enhances driving prediction capabilities by providing more interactive information to the driver, thereby improving the reliability, speed, and safety of remote vehicle control. Based on real-time trajectory prediction and presentation, safety prompts for vehicle operation can also be provided to the driver, further improving the safety of remote vehicle control.

[0052] It should be noted that although several modules or units for vehicle driver assistance devices have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units. Components shown as modules or units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0053] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, having stored thereon a computer program including executable instructions that, when executed by, for example, a processor, can implement the steps of the method for assisted driving of a vehicle described in any of the above embodiments. In some possible implementations, various aspects of this disclosure can also be implemented as a program product including program code that, when run on a terminal device, causes the terminal device to perform the steps described in the various exemplary embodiments of this disclosure for the method for assisted driving of a vehicle.

[0054] The program product for implementing the above-described method according to embodiments of this disclosure may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of this disclosure is not limited thereto. In this document, a readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0055] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0056] The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0057] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0058] In exemplary embodiments of this disclosure, an electronic device is also provided, which may include a processor and a memory for storing executable instructions of the processor. The processor is configured to perform steps of the method for assisted driving of a vehicle according to any of the above embodiments by executing the executable instructions.

[0059] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0060] The following reference Figure 8 To describe an electronic device 800 according to such an embodiment of the present disclosure. Figure 8 The electronic device 800 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0061] like Figure 8As shown, the electronic device 800 is presented in the form of a general-purpose computing device. The components of the electronic device 800 may include, but are not limited to: at least one processing unit 810, at least one storage unit 820, a bus 830 connecting different system components (including storage unit 820 and processing unit 810), a display unit 840, etc.

[0062] The storage unit stores program code that can be executed by the processing unit 810, causing the processing unit 810 to perform the steps described in the method for assisted driving of a vehicle according to various exemplary embodiments of this disclosure. For example, the processing unit 810 can perform actions such as... Figure 2 , Figure 4 and Figure 6 The steps are shown in the figure.

[0063] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 8201 and / or a cache storage unit 8202, and may further include a read-only memory unit (ROM) 8203.

[0064] The storage unit 820 may also include a program / utility 8204 having a set (at least one) program module 8205, such program module 8205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0065] Bus 830 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0066] Electronic device 800 can also communicate with one or more external devices 900 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 800, and / or with any device that enables electronic device 800 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 850. Furthermore, electronic device 800 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 860. Network adapter 860 can communicate with other modules of electronic device 800 via bus 830. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0067] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the method for assisted driving of a vehicle according to the embodiments of this disclosure.

[0068] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A method for assisting driving a vehicle, the vehicle being located at a remote position away from a driver of the vehicle, comprising: Real-time acquisition of the vehicle's steering data and environmental images in the vehicle's driving direction; Predict the real-time trajectory of the vehicle at the remote location based at least on the steering data; The predicted real-time trajectory and the corresponding environmental image are combined to generate an enhanced image; The enhanced image is presented to the driver in real time.

2. The method of claim 1, wherein, Predicting the real-time trajectory of the vehicle at the remote location based at least on the steering data includes: The real-time trajectory is predicted using a vehicle dynamics model based on the steering data, the vehicle's dimensions, and its driving status, wherein the dimensions include the vehicle's wheelbase and track width, and the driving status includes the vehicle's center of gravity, direction of travel, and turning radius.

3. The method according to claim 2, characterized in that, Predicting the real-time trajectory using a vehicle dynamics model based on the steering data, the vehicle's dimensions, and its driving status further includes: The steering angles of the left and right front wheels of the vehicle are determined based on the steering data. The direction of movement of the vehicle is determined based on the steering angle, the turning radius, and the driving direction; The real-time trajectory is predicted based on the direction of movement, the center, and the size.

4. The method according to any one of claims 1 to 3, characterized in that, Combining the predicted real-time trajectory with the corresponding environmental image to generate an enhanced image includes: The real-time trajectory is mapped from the remote location to the environmental image based on the mapping matrix to generate an image trajectory; The image trajectory is superimposed on the environmental image corresponding to the real-time trajectory to generate the enhanced image.

5. The method according to claim 4, characterized in that, The mapping matrix is ​​determined through the following steps: Determine the ground position data of the calibration object set in the driving direction of the vehicle in the ground coordinate system; Acquire an environmental image along the vehicle's travel direction, wherein the environmental image includes the calibration object; Determine the image position data of the calibration object in the image coordinate system of the environmental image; The mapping matrix is ​​determined based on the ground location data and the image location data.

6. The method according to claim 5, characterized in that, The mapping matrix has a dimension of 3*3, and / or the number of the calibrated objects is at least 4.

7. The method according to claim 5 or 6, characterized in that, The calibration objects are respectively set on the extension lines of the left front wheel and the left rear wheel of the vehicle, and on the extension lines of the right front wheel and the right rear wheel of the vehicle.

8. The method according to claim 5 or 6, characterized in that, The ground position data of the calibration object is determined based on the relative distance between the calibration object and the center point of at least one of the front axle or rear axle of the vehicle.

9. The method according to any one of claims 1 to 3, characterized in that, The steering data is determined based on the driver's input.

10. The method according to any one of claims 1 to 3, characterized in that, The environmental images are acquired through the vehicle's camera.

11. The method according to any one of claims 1 to 3, characterized in that, Presenting the enhanced image to the driver in real time includes: The enhanced image is displayed in real time on a terminal device located at the driver's position.

12. The method according to claim 11, characterized in that, The terminal device predicts the real-time trajectory and / or overlays the predicted real-time trajectory onto the environmental image corresponding to the real-time trajectory acquired from the vehicle to generate the enhanced image.

13. The method according to any one of claims 1 to 3, characterized in that, Presenting the enhanced image to the driver in real time also includes: A security alert is generated based on the enhanced image.

14. A device for assisting driving of a vehicle, the vehicle being located at a remote location away from a driver of the vehicle, comprising: The data acquisition unit is configured to acquire the vehicle's steering data and environmental images in the vehicle's driving direction in real time. A trajectory prediction unit is configured to predict the real-time trajectory of the vehicle at the remote location based at least on the steering data; An image rendering and presentation unit is configured to combine the predicted real-time trajectory with the environmental image corresponding to the real-time trajectory to generate an enhanced image and to present the enhanced image to the driver in real time.

15. The device according to claim 14, characterized in that, The data acquisition unit includes at least one of the following: An input component is configured to receive at least steering data from the driver; A receiving component is configured to receive environmental images from the vehicle's camera and information about the vehicle's driving status.

16. The device according to claim 14, characterized in that, The device is configured to perform the method according to any one of claims 1 to 13.

17. The device according to claim 14, characterized in that, At least one of the data acquisition unit, the trajectory prediction unit, and the image rendering and presentation unit is included in the terminal device located at the driver's position.

18. An electronic device comprising: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the executable instructions to implement the method according to any one of claims 1 to 13.

19. A computer-readable storage medium having a computer program stored thereon, the computer program including executable instructions that, when executed by a processor, implement the method according to any one of claims 1 to 13.

20. A computer program product comprising a computer program that, when executed by a processor, performs the method according to any one of claims 1 to 13.

Images