Driving auxiliary line generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a driving assistance line generation method and device, electronic equipment and a storage medium, and relates to the technical field of remote driving. The driving assistance line generation method is applied to remote control equipment, and can comprise the following steps: receiving a target image acquired by image acquisition equipment on a vehicle in the driving process of the vehicle; calculating transmission time delay generated when the target image is received; calculating a first change speed of the vehicle according to the transmission time delay; wherein the first change speed is the change speed of the vehicle in the time period of the transmission time delay; predicting a wheel trajectory of the vehicle based on the first change speed and driving information of the vehicle; and drawing the wheel track line in the target image to obtain a driving auxiliary line. According to the scheme, driving auxiliary prompting can be carried out on the remote driving cabin side, and therefore the driving safety of the vehicle is improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of remote driving, and in particular to a method, device, electronic device and storage medium for generating a driving auxiliary line. Background Art

[0002] Nowadays, remote driving technology can be applied to multiple application scenarios, such as remote control of a fleet for cargo transportation. In addition, remote driving has some characteristics, such as the physical isolation of the remote cockpit and the vehicle, and network latency when the remote cockpit communicates with the vehicle. The network latency poses a safety hazard to the driving safety of the vehicle.

[0003] It can be seen that for remote driving, how to provide driving assistance prompts on the remote cockpit side to improve the driving safety of the vehicle is an urgent problem to be solved. Summary of the invention

[0004] The purpose of the embodiments of the present disclosure is to provide a method, device, electronic device and storage medium for generating driving auxiliary lines, so as to realize driving auxiliary prompts on the remote cockpit side. The specific technical solution is as follows:

[0005] According to one aspect of the present disclosure, an embodiment of the present disclosure provides a method for generating a driving auxiliary line, the method being applied to a remote control device, the method comprising:

[0006] Receiving a target image captured by an image acquisition device on the vehicle during the vehicle's driving;

[0007] Calculating the transmission delay generated when receiving the target image;

[0008] Calculating a first change speed of the vehicle according to the transmission delay; wherein the first change speed is a speed at which the vehicle changes within a period of the transmission delay;

[0009] Predicting a wheel trajectory of the vehicle based on the first change speed and driving information of the vehicle;

[0010] The wheel track line is drawn in the target image to obtain a driving auxiliary line.

[0011] According to one aspect of the present disclosure, an embodiment of the present disclosure provides a driving auxiliary line generation system, the system comprising: a remote control device, a vehicle control device;

[0012] The vehicle control device is used to obtain the target image captured by the image acquisition device and send the target image to the remote control device;

[0013] The remote control device is used to receive the target image captured by the image acquisition device during vehicle driving, calculate the transmission delay generated when the target image is received, calculate the first change speed according to the transmission delay and the driving information of the vehicle, predict the wheel trajectory line of the vehicle based on the first change speed and the driving information, draw the wheel trajectory line in the target image, and obtain a driving auxiliary line.

[0014] According to one aspect of the present disclosure, an embodiment of the present disclosure provides a driving auxiliary line generation device, the device is applied to a remote control device, and the device includes:

[0015] A receiving module, used for receiving a target image captured by an image acquisition device on the vehicle during the vehicle's driving process;

[0016] A first calculation module, used to calculate the transmission delay generated when the target image is received;

[0017] A second calculation module is used to calculate a first change speed according to the transmission delay and the driving information of the vehicle; wherein the first change speed is a speed at which the vehicle changes within a period of the transmission delay;

[0018] A prediction module, configured to predict a wheel trajectory of the vehicle based on the first change speed and the driving information;

[0019] A drawing module is used to draw the wheel track line in the target image to obtain a driving auxiliary line.

[0020] According to one aspect of the present disclosure, an embodiment of the present disclosure provides an electronic device, including:

[0021] one or more processors, and

[0022] A memory storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes a method according to any one of the above-mentioned driving assistance line generation methods.

[0023] According to one aspect of the present disclosure, an embodiment of the present disclosure provides a computer-readable storage medium storing a program, wherein the program includes instructions, which, when executed by one or more processors of a computing device, cause the computing device to execute a method according to any one of the above-mentioned driving assistance line generation methods.

[0024] Beneficial effects of the embodiments of the present disclosure:

[0025] The driving assistance line generation method provided by the embodiment of the present disclosure receives the target image captured by the image acquisition device on the vehicle during the driving process of the vehicle; then, calculates the transmission delay generated when the target image is received, calculates the first change speed according to the transmission delay, and predicts the wheel trajectory line of the vehicle based on the first change speed and the driving information of the vehicle, draws the wheel trajectory line in the target image, and obtains the driving assistance line. It can be seen that due to the transmission delay problem in the remote driving process, the present solution is not only based on the vehicle driving information, but also based on the transmission delay of the received target image during the driving assistance line generation process, so that a more accurate driving assistance line can be drawn in the target image, so that the present solution can realize driving assistance prompts on the remote cockpit side, thereby improving the driving safety of the vehicle.

[0026] Of course, it is not necessary to achieve all the advantages described above at the same time to implement any product or method of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0028] Figure 1 A schematic diagram of the effect of a driving auxiliary line provided by an embodiment of the present disclosure;

[0029] Figure 2 A schematic diagram of a flow chart of a method for generating auxiliary driving lines provided in an embodiment of the present disclosure;

[0030] Figure 3 A schematic diagram of a flow chart of another method for generating auxiliary driving lines provided in an embodiment of the present disclosure;

[0031] Figure 4 A schematic diagram of the effect of a driving trajectory point provided by an embodiment of the present disclosure;

[0032] Figure 5 A schematic diagram of the effect of a wheel track line provided by an embodiment of the present disclosure;

[0033] Figure 6 A schematic diagram of the position of an image acquisition device relative to a vehicle provided in an embodiment of the present disclosure;

[0034] FIG. 7( a ) is a three-dimensional schematic diagram of the conversion between a camera coordinate system and an image coordinate system provided by an embodiment of the present disclosure;

[0035] FIG7( b ) is a plan view schematically showing the transformation between a camera coordinate system and an image coordinate system provided by an embodiment of the present disclosure;

[0036] Figure 8 A flowchart of a data processing method for remote driving provided in an embodiment of the present disclosure;

[0037] Fig. 9 A schematic diagram of a process flow of a vehicle safety detection method provided by an embodiment of the present disclosure;

[0038] Fig.10 A schematic diagram of a flow chart of a safety detection method for a remote cockpit provided in an embodiment of the present disclosure;

[0039] Fig.11 A schematic diagram of a flow chart of a safety detection method for vehicle images provided in an embodiment of the present disclosure;

[0040] Fig.12 A schematic diagram of the structure of a driving auxiliary line generation system provided by an embodiment of the present disclosure;

[0041] FIG. 13( a ) is a schematic diagram of the structure of another driving auxiliary line generation system provided in an embodiment of the present disclosure;

[0042] FIG. 13( b ) is a schematic diagram of the structure of another driving auxiliary line generation system provided in an embodiment of the present disclosure;

[0043] Fig.14 A schematic diagram of the structure of a driving auxiliary line generating device provided by an embodiment of the present disclosure;

[0044] Fig.15 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field based on the present disclosure belong to the scope of protection of the present disclosure.

[0046] In order to better understand the present solution, before introducing the solution provided by the embodiment of the present disclosure, the driving auxiliary line and remote driving technology are briefly introduced in conjunction with the accompanying drawings:

[0047] Driving auxiliary lines are used as driving auxiliary prompts during vehicle driving, which can achieve the purpose of improving vehicle driving safety. For example, in order to facilitate understanding of the relationship between driving auxiliary lines and vehicle driving, you can refer to Figure 1 ,like Figure 1 As shown, the driving auxiliary line is located in front of the vehicle, specifically in the image screen about the front of the vehicle, wherein, Figure 1 The perspective is the driving auxiliary line under the top-down perspective; of course, there can also be driving auxiliary lines in the front perspective, driving auxiliary lines in the left perspective, driving auxiliary lines in the right perspective and driving auxiliary lines in the rear perspective, which are not specifically limited in the embodiments of the present disclosure.

[0048] It can be understood that remote driving technology is based on the interaction between a remote control console, a cloud server and a vehicle, thereby realizing remote driving. The remote control console can be a remote cockpit, which can also be called a remote end. The remote cockpit is a car-like cockpit console for the center console office, including brakes, accelerators, steering wheels, gears, wired / wireless network equipment and emergency stop controls, etc. The vehicle can also be called an autonomous driving vehicle or a vehicle end. The vehicle is equipped with 4 / 5G network equipment or wireless equipment. Of course, it can also be equipped with a GPS (Global Positioning System) device for at least positioning, time synchronization, etc. The remote cockpit can also include a screen for displaying sensor data transmitted back from the vehicle end; the driving auxiliary line can be displayed on the screen of the remote cockpit so that the driver can perform remote driving. The remote cockpit can also be called a remote end, and the vehicle can also be called a vehicle end. This disclosure does not limit this. In addition, the specific process of generating the driving auxiliary line will be described in detail in subsequent embodiments, so it will not be repeated here.

[0049] A method for generating auxiliary driving lines provided by an embodiment of the present disclosure is introduced below.

[0050] The driving auxiliary line generation method provided in the embodiment of the present disclosure can be applied to a remote control device, which is arranged in a remote cockpit. In a specific application, the remote control device can be a terminal device or a server. The present disclosure does not specifically limit the specific form of the remote control device.

[0051] Specifically, the execution subject of the driving auxiliary line generation may be a driving auxiliary line generation device. Exemplarily, when the driving auxiliary line generation device is applied to a terminal device, the driving auxiliary line generation device may be a client running in the terminal device, and the client may be used to generate driving auxiliary lines. When the driving auxiliary line generation method is applied to a server, the driving auxiliary line generation device may be a computer program running in the server, and the computer program may be used to generate driving auxiliary lines.

[0052] It can be understood that the method provided in the embodiments of the present disclosure can be applied to various scenarios suitable for remote driving technology, such as remotely controlling the driving of a fleet, controlling a remotely controlled fleet to transport goods, etc., so the embodiments of the present disclosure do not specifically limit the application scenarios.

[0053] In addition, it should be emphasized that the remote cockpit and the vehicle can be synchronized in time during the process of generating the auxiliary driving line, wherein the vehicle can be synchronized based on the time of the GPS device, and the remote cockpit can be synchronized based on the time of the GPS device through the wired network. The time synchronization method between the remote cockpit and the vehicle is only used as an example and should not constitute a limitation on the embodiments of the present disclosure. A method for generating auxiliary driving lines is applied to a remote control device, and the method includes:

[0054] Receiving a target image captured by an image acquisition device on the vehicle during the vehicle's driving;

[0055] Calculating the transmission delay generated when receiving the target image;

[0056] Calculating a first change speed of the vehicle according to the transmission delay; wherein the first change speed is a speed at which the vehicle changes within a period of the transmission delay;

[0057] Predicting a wheel trajectory of the vehicle based on the first change speed and driving information of the vehicle;

[0058] The wheel track line is drawn in the target image to obtain a driving auxiliary line.

[0059] It can be seen that due to the transmission delay problem in the remote driving process, this solution is not only based on the vehicle driving information, but also based on the transmission delay of the received target image during the generation of the driving assistance lines, so that more accurate driving assistance lines can be drawn in the target image. Therefore, this solution can realize driving assistance prompts on the remote cockpit side, thereby improving the driving safety of the vehicle.

[0060] A method for generating auxiliary driving lines provided by an embodiment of the present disclosure is described below in conjunction with the accompanying drawings.

[0061] Figure 2 A driving auxiliary line generation method provided in an embodiment of the present disclosure is applied to a remote control device, such as Figure 2 As shown, the method may include steps S201-S205:

[0062] S201, receiving a target image captured by an image capture device on the vehicle during the vehicle's driving;

[0063] It is understandable that the vehicle can also be called a remote vehicle, and the vehicle is provided with a vehicle control device, and the vehicle is also provided with an image acquisition device and / or a point cloud acquisition device. For example, the image acquisition device can be a camera, a webcam, etc. The target image acquired by the image acquisition device can be a real-time image acquired by the image acquisition device, and the point cloud acquisition device can be a laser radar, such as a single-line laser radar, a multi-line laser radar, etc. The target point cloud acquired by the point cloud acquisition device can be a real-time point cloud acquired by the point cloud acquisition device.

[0064] It should be emphasized that the number of image acquisition devices installed on the vehicle can be multiple. Due to the different placement positions of the image acquisition devices, the acquisition ranges of the image acquisition devices can be different. For example, camera A is located at the left front of the vehicle, so camera A can capture images of the left front of the vehicle, camera B is located at the left rear of the vehicle, so camera B can capture images of the left rear of the vehicle, camera C is located at the right front of the vehicle, so camera C can capture images of the right front of the vehicle, and camera D is located at the right rear of the vehicle, so camera D can capture images of the right rear of the vehicle. Of course, the above is only an example for the setting of the image acquisition device. The embodiment of the present disclosure does not specifically limit the image acquisition device, and the images of different cameras can be spliced ​​to obtain images covering the fields of view of multiple cameras.

[0065] In the present disclosure, the target image received by the remote control device may be an image captured by an image acquisition device on the vehicle; or it may be an image captured by multiple image acquisition devices, in which case the number of target images is multiple; or it may be an image obtained by stitching together images captured by multiple image acquisition devices, such as stitching together images of the left front and right front cameras, or stitching together images of various fields of view into a bird's-eye view; and the one image acquisition device or the multiple image acquisition devices are devices that use the captured images to generate driving auxiliary lines.

[0066] In addition, the remote control device may include a first domain controller, so that the target image can be received through the first domain controller; the vehicle is provided with a second domain controller, which can also be called an image domain controller; after the image acquisition device on the vehicle acquires the target image, the target image can be sent to the cloud server through the second domain controller; the cloud server is used to forward and record the target image, and then send the target image to the first domain controller, so that the remote control device receives the target image. Of course, in other implementations, the vehicle can also directly send the target image to the first domain controller through the second domain controller, so that the remote control device receives the target image. The embodiment of the present disclosure does not specifically limit this. Optionally, in order to ensure the transmission efficiency of the target image, the vehicle can dynamically adjust the resolution of the target image or point cloud to be forwarded according to the network status of the vehicle before sending the target image to the cloud server or the first domain controller, so as to quickly realize the sensor data transmission. For example, the resolution of the target image or point cloud to be forwarded can be dynamically adjusted by the second domain controller of the vehicle.

[0067] The first domain controller and the second domain controller mentioned above are both domain controllers. Since they are located in different devices and have different specific functions, they are distinguished by "first" and "second".

[0068] S202, calculating the transmission delay generated when the target image is received;

[0069] Since there is a transmission delay problem in the remote driving process, in order to generate accurate driving auxiliary lines, in the present disclosure, the transmission delay generated when the target image is received is incorporated into the generation process of the driving auxiliary lines. Exemplarily, in one implementation, the transmission delay can be the transmission delay generated in the process of sending the target image. In this case, the transmission delay can be calculated by subtracting the sending time of the target image from the time when the remote control device receives the target image, wherein the sending time can be determined by the timestamp of the target image regarding the sending time.

[0070] Exemplarily, in another implementation, the transmission delay may also be the transmission delay generated by the image acquisition device during the acquisition process and the process of sending the target image. In this case, the transmission delay may be calculated by subtracting the acquisition time of the target image from the time when the remote control device receives the target image, wherein the acquisition time may be determined by the time stamp of the target image.

[0071] In addition, the transmission delay is generally very short, and the unit can be milliseconds ms. For example, according to the timestamp corresponding to the target image collected and the timestamp corresponding to before the target image is collected, it can be calculated that the transmission delay is 100ms.

[0072] According to some embodiments, the transmission delay can be calculated in the following manner. At the data sending end (such as the vehicle end), for the image sequence to be transmitted, the key frames and difference frames in the image sequence are determined, and based on the reference time source, an absolute timestamp is generated for the key frames, and a relative timestamp is generated for the difference frames to obtain the target data corresponding to each frame of the image. In this way, the receiving end (such as the remote end) can obtain the data transmission delay by parsing the absolute timestamp or relative timestamp in each frame of image data and comparing it with the current moment of the reference time source. Among them, the key frame belongs to intra-frame compression, and the image data of the frame is completely retained during encoding. The difference frame is obtained by encoding the difference image information between the image data of the frame and the key frame image data. Determining the key frames and difference frames in the image sequence includes:

[0073] Determine the frame type of each frame based on preset image group parameters; wherein the preset image group parameters are used to group multiple frames of images to be encoded, the first frame in each group of images is a key frame, and the remaining frames are difference frames; and / or

[0074] Each frame of image data is encoded according to a preset encoding method to obtain a frame type identifier in the encoded data, and the frame type of each frame of image is determined.

[0075] Here, both the key frame and the difference frame have a fixed data transmission cycle, for example, the key frame is 1 frame per second and the difference frame is 30 frames per second, then the key frame and difference frame of the image sequence can be determined according to the two data transmission cycles. The frame between two adjacent key frames is the difference frame. Even if the image frame at a certain moment is not collected or transmitted successfully, the image frame still participates in the counting of key frames and difference frames.

[0076] Optionally, the absolute timestamp of the key frame includes: the moment of the reference time source corresponding to the key frame. The relative timestamp of the difference frame includes the difference between the moment of the reference time source corresponding to the difference frame and the moment of the reference time source corresponding to the difference frame. The absolute timestamp includes a first number of characters (such as 8 bytes), and the relative timestamp includes a second number of characters (such as 2 bytes); the first number of characters is greater than the second number of characters. The absolute timestamp or the relative timestamp can be added to the frame header or frame tail of the image data to obtain the target data. The vehicle end and the remote end are synchronized through the GPS device, and the reference time source is the global navigation satellite system time source, the network standard time, etc. In addition, the absolute timestamp and the relative timestamp can be any of the following time types: the moment when the current frame image data is generated from the image signal, the moment when the current frame image data is encoded, and the moment when the encoded data is obtained after encoding the current image data.

[0077] The above-mentioned method for calculating the transmission delay is only an example and should not constitute a limitation on the embodiments of the present disclosure.

[0078] S203, calculating a first change speed of the vehicle according to the transmission delay;

[0079] The first changing speed is the speed at which the vehicle changes during the transmission delay period.

[0080] There may be multiple implementations of calculating the first change speed of the vehicle based on the transmission delay. Exemplarily, in one implementation, the product of the transmission delay and the acceleration of the vehicle is calculated to obtain the first change speed of the vehicle. Exemplarily, in another implementation, the product of the transmission delay and the acceleration of the vehicle may be calculated, and then the product may be corrected according to a predetermined correction coefficient, thereby obtaining the first change speed of the vehicle. The acceleration of the vehicle may be reported by the vehicle end to the remote control device. Exemplarily, the vehicle end may simultaneously transmit the acceleration of the vehicle when transmitting the target image.

[0081] The above calculation of the first change speed of the vehicle based on the transmission delay is only an example and should not constitute a limitation to the present disclosure.

[0082] In addition, since the acceleration of the vehicle can be positive or negative, the first change speed can also be positive or negative. When the first change speed is positive, it indicates that the speed of the vehicle increases during the transmission delay period, and when the first change speed is negative, it indicates that the speed of the vehicle decreases during the transmission delay period. For example, the transmission delay is 100ms, that is, 0.1s, and the acceleration of vehicle A is 1m / s 2 , then, the first change speed of vehicle A is 0.1m / s; the transmission delay is 100ms, that is, 0.1s, and the acceleration of vehicle A is -1m / s 2 , then the first change speed of vehicle A is -0.1m / s.

[0083] It should be emphasized that, since the embodiment of the present disclosure calculates the changing speed of the vehicle during the transmission delay and uses the changing speed as consideration information for generating the auxiliary driving lines, the generated auxiliary driving lines are more accurate.

[0084] S204, predicting a wheel trajectory of the vehicle based on the first change speed and the driving information of the vehicle;

[0085] After obtaining the first change speed, since the driving information of the vehicle plays an important role in predicting the subsequent driving process of the vehicle, and the auxiliary driving line plays a prompting role for the wheel trajectory of the vehicle, the wheel trajectory of the vehicle can be predicted based on the first change speed and the driving information of the vehicle. Among them, the wheel trajectory is one or more, including but not limited to the trajectory of the wheel on one side and the trajectory of the wheel on both sides, wherein the trajectory of the wheel on one side includes the trajectory of the left wheel and the trajectory of the right wheel. Considering that the length of the truck is relatively long, the front wheel and the rear wheel are far apart. Therefore, when the wheel on one side includes multiple wheels, the trajectory of the wheel on one side can also include the trajectory of each wheel on one side, such as the trajectory of the front wheel on one side, the trajectory of the middle wheel on one side, and the trajectory of the rear wheel on one side.

[0086] It can be understood that, exemplarily, in one implementation, the wheel trajectory prediction process may include: predicting the vehicle's driving trajectory points at a future moment based on the first change speed and the vehicle's driving information, and determining the vehicle's wheel trajectory based on the driving trajectory points.

[0087] It is understandable that for the sake of clarity, the specific process of predicting the wheel trajectory of the vehicle will be introduced in other embodiments, so it will not be repeated here.

[0088] S205: Draw the wheel track line in the target image to obtain a driving auxiliary line.

[0089] It is understandable that displaying the target image including the driving auxiliary line on the screen of the remote cockpit can facilitate the driver to perform remote driving. In addition, if multiple image acquisition devices acquire the target image, that is, the number of target images is multiple, the target images can be spliced ​​and the wheel track line can be drawn in the spliced ​​image to obtain the driving auxiliary line, which is also reasonable.

[0090] In addition, for the sake of clarity, the process of drawing the wheel track line on the target image is introduced in other embodiments, so it will not be repeated here.

[0091] Optionally, in one implementation, the method further includes:

[0092] Whether to issue a collision warning is determined based on the positional relationship between the driving assistance line and the edge of the road, as well as the positional relationship between the driving assistance line and other vehicles.

[0093] Among them, the auxiliary driving lines may include auxiliary driving lines of the front wheels of the vehicle and / or auxiliary driving lines of the rear wheels of the vehicle. The embodiment of the present disclosure can not only determine whether the auxiliary driving lines of the front wheels have a collision risk, but also determine whether the auxiliary driving lines of the rear wheels have a collision risk.

[0094] It is understandable that there is a preset first safety distance between the auxiliary line and the edge of the road, and a preset second safety distance between the auxiliary line and other vehicles. Both the first safety distance and the second safety distance can be empirical values. When the distance between the auxiliary line and the edge of the road is less than the first safety distance, it is determined to issue a collision warning. When the distance between the auxiliary line and other vehicles is less than the second safety distance, it is determined to issue a collision warning. For example, the first safety distance is 1m, and the distance between vehicle A and the edge of the road is 0.8m. At this time, it is determined to issue a collision warning; the second safety distance is 1m, and the distance between vehicle A and vehicle B is 0.5m. At this time, it is determined to issue a collision warning.

[0095] It can be seen that based on the positional relationship between the driving assistance line and the edge of the road, and the positional relationship between the driving assistance line and other vehicles, it can be determined whether to issue a collision warning. Therefore, the embodiment of the present disclosure can improve the safety of remote driving and facilitate the driver to perform remote driving.

[0096] It can be seen that due to the transmission delay problem in the remote driving process, this solution is not only based on the vehicle driving information, but also based on the transmission delay of the received target image during the generation of the driving assistance lines, so that more accurate driving assistance lines can be drawn in the target image. Therefore, this solution can realize driving assistance prompts on the remote cockpit side, thereby improving the driving safety of the vehicle.

[0097] Optionally, in one implementation, the embodiment of the present disclosure also provides another method for generating a driving auxiliary line, such as Figure 3 As shown, the following steps may be included:

[0098] S301, receiving a target image captured by an image capture device on the vehicle during the vehicle's driving;

[0099] S302, calculating the transmission delay generated when the target image is received;

[0100] S303, calculating a first change speed of the vehicle according to the transmission delay;

[0101] It is understandable that steps S201-S203, S205 are the same as steps S301-S303, S306, so they are not described in detail here.

[0102] S304, predicting a plurality of driving trajectory points of the vehicle according to the first change speed and the driving information of the vehicle;

[0103] It is understood that the driving information may include at least one of the following:

[0104] The speed and acceleration of the vehicle, and the steering angle and steering angle coefficient of the steering wheel of the vehicle.

[0105] The steering angle of the vehicle's steering wheel represents the current rotation angle of the steering wheel, and the turning angle coefficient represents the corresponding relationship between the steering angle of the steering wheel and the rotation angle of the wheel. It can be understood that the speed, acceleration and steering angle of the vehicle's steering wheel can all be detected in real time, and the turning angle coefficient can be a constant. For example, the turning angle coefficient is 0.8, then when the steering angle of the steering wheel is 100 degrees, the rotation angle of the wheel is 80 degrees.

[0106] Optionally, in one implementation, step S304 may include steps A1-A2:

[0107] Step A1, for each future moment, predicting the distance traveled by the vehicle in a time period from a corresponding reference moment to the future moment according to the speed, acceleration and the first change speed of the vehicle, as the travel distance corresponding to the future moment; the reference moment corresponding to each future moment is the moment before the future moment;

[0108] It is understandable that the intervals between adjacent future moments can be the same, for example, the intervals between adjacent future moments are 100ms, and the reference moment corresponding to each future moment is the moment before the future moment, and the intervals between each future moment and the corresponding reference moment can be the same. It is understandable that the future moment is based on the current moment, and the current moment can be the moment of executing the generation method, or the moment of receiving the target image, etc., which are all reasonable.

[0109] In addition, for each future moment, the reference moment corresponding to the future moment can be the moment before the future moment, or the current moment, but it is not limited to this. For example, it can also be the first two moments of the future moment, and the disclosed embodiment does not specifically limit this. It should be emphasized that the shorter the interval between each future moment and the corresponding reference moment, the higher the accuracy of the predicted vehicle's driving trajectory point, and the higher the accuracy of the generated driving auxiliary line. Conversely, the longer the interval between each future moment and the corresponding reference moment, the lower the accuracy of the predicted driving trajectory point, and the lower the accuracy of the generated driving auxiliary line. For example, the reference moment is the moment before the future moment a, and the time period from the future moment a to the corresponding reference moment is 1s. The distance traveled by vehicle A within this 1s is the driving distance corresponding to the future moment a. It can be understood that the reference moment is the first two moments of the future moment, which is similar to the process of the previous moment, so it will not be repeated.

[0110] It should be noted that, in one implementation, the time interval between adjacent future moments can be set according to an empirical value; in another implementation, the time interval between adjacent future moments can be set according to the maximum safety distance. For example, the allowable speed for vehicle remote driving is 36km / h, that is, 10m / s, and the maximum allowable safety distance is 1m. Then, the time required for the vehicle to travel 1m is 0.1s, that is, 100ms, so this time can be used as the time interval between adjacent future moments.

[0111] Optionally, in one implementation, step A1, for each future moment, predicting the distance traveled by the vehicle in a time period from a corresponding reference moment to the future moment according to the speed, acceleration and the first change speed of the vehicle as the travel distance corresponding to the future moment, may include steps A11-A12:

[0112] Step A11, for each future moment, calculate the second change speed corresponding to the future moment according to the acceleration, the time difference between the future moment and the corresponding reference moment, and the target constant; wherein the target constant is related to the sorting sequence number of the future moment in the moment sequence, and the second change speed is the speed of the vehicle during the period from the current moment to the future moment;

[0113] It can be understood that the target constant is generally the sorting number of the future moment in the moment sequence minus one, and the initial moment of the moment sequence is the current moment. Exemplarily, the sorting number of the future moment a in the moment sequence is 3, then the target constant is 2. Of course, the target constant is related to the sorting number of the future moment in the moment sequence, and it may not be the sorting number minus one. The embodiment of the present disclosure does not specifically limit the target constant. In addition, the current moment can be understood as the starting moment, that is, the starting moment of starting to generate the auxiliary driving line. The current moment is also the moment before the future moment. The embodiment of the present disclosure does not specifically limit the starting moment.

[0114] It should be emphasized that, in one implementation, the product of the acceleration, the time difference between the future moment and the corresponding reference moment, and the target constant can be calculated to obtain the second change speed corresponding to the future moment, and the second change speed is the speed of the vehicle during the period from the current moment to the future moment. For example, the acceleration of vehicle A is 1m / s 2 , the time difference between the future moment a and the previous moment is 0.1, and the target constant is 2, then the second change speed corresponding to the future moment a is 0.2m / s.

[0115] Step A12, for each future moment, based on the speed, the first change speed, the second change speed corresponding to the future moment, and the time difference between the future moment and the corresponding reference moment, determine the driving distance corresponding to the future moment.

[0116] It can be understood that the travel distance corresponding to the future moment can be determined based on the sum of the vehicle speed, the first change speed, and the second speed corresponding to the future moment, and the product of the time difference between the future moment and the corresponding reference moment. For example, the speed of vehicle A is 5m / s, the first change speed is 1m / s, the second change speed corresponding to the future moment a is 2m / s, and the time difference between the future moment a and the previous moment is 0.1s. It can be determined that the travel distance corresponding to the future moment is 0.8m.

[0117] The process of determining the driving distance corresponding to the future time in step A12 can be expressed in the form of a formula, as shown in Formula 1:

[0118] Formula 1

[0119] dsN=(v1+va*t*N+va*dt)*t

[0120] Among them, dsN represents the driving distance corresponding to the future moment, v1 represents the speed of the vehicle, va represents the acceleration of the vehicle, N represents the target constant, dt represents the transmission delay, t represents the time difference between the future moment and the corresponding reference moment, va*t*N represents the second speed corresponding to the future moment, and va*dt represents the first speed.

[0121] Step A2, predicting the driving trajectory points of the vehicle at each future moment according to the driving distance corresponding to each future moment, as well as the steering angle and the steering angle coefficient, to obtain multiple driving trajectory points of the vehicle.

[0122] It can be understood that predicting the vehicle's driving trajectory points requires not only predicting the driving distance corresponding to each future moment, but also predicting the vehicle's driving direction. The vehicle's driving direction can be predicted based on the steering angle and steering coefficient of the steering wheel, thereby obtaining multiple driving trajectory points of the vehicle.

[0123] It can be seen that the embodiment of the present disclosure not only predicts the driving distance at future moments when predicting the driving trajectory points, but also predicts the driving direction at each future moment, thereby determining the driving trajectory points of the vehicle. In addition, the transmission delay is taken into account during the prediction process, so that the predicted driving trajectory points are more accurate, thereby making the generated driving auxiliary lines more accurate, thereby improving the driving safety of the vehicle.

[0124] Optionally, in one implementation, step A2, predicting the driving trajectory points of the vehicle at each future moment according to the driving distance corresponding to each future moment, as well as the steering angle and the steering angle coefficient, to obtain multiple driving trajectory points of the vehicle, may include steps A21-A23:

[0125] Step A21, determining a wheel rotation angle of the vehicle based on the steering angle and the steering angle coefficient;

[0126] It is understandable that the driver can turn the steering wheel in the remote cockpit, and the vehicle's wheel rotation angle can be determined based on the product of the steering angle of the steering wheel and the turning angle coefficient. For example, the turning angle coefficient is 0.8, so when the steering angle of the steering wheel is 100 degrees, the wheel rotation angle is 80 degrees. In addition, the wheel end can also be provided with a steering wheel and a corresponding automatic driving control system, which can control the steering wheel to rotate automatically, and the vehicle's wheel rotation angle can be determined according to the steering angle of the steering wheel and the turning angle sparseness.

[0127] The process of determining the wheel rotation angle in step A21 can be expressed in the form of a formula, as shown in Formula 2:

[0128] Formula 2

[0129] vd=vs*vu

[0130] Among them, vd represents the vehicle's wheel rotation angle, vs represents the steering angle coefficient, and vu represents the steering angle of the steering wheel.

[0131] Step A22, for each future moment, calculating the coordinates of the driving trajectory point of the vehicle at the future moment according to the driving distance corresponding to the future moment, the wheel rotation angle, and the coordinates of the driving trajectory point at the reference moment;

[0132] It can be understood that the driving distance corresponding to the future moment is the distance traveled by the vehicle in the time period from the corresponding reference moment to the future moment. According to the driving distance corresponding to the future moment and the rotation angle of the wheel, the horizontal and vertical coordinates in the time period from the reference time period to the future moment can be calculated. By adding the horizontal and vertical coordinates in the time period from the reference time period to the future moment and the coordinates of the driving trajectory points in the reference time period, the coordinates of the driving trajectory points of the vehicle in the future moment can be obtained. It should be noted that the calculated coordinates of the driving trajectory points of the vehicle at each future moment are: the coordinates in the vehicle coordinate system.

[0133] Optionally, in one implementation, step A22, for each future moment, calculating the coordinates of the driving trajectory point of the vehicle at the future moment according to the driving distance corresponding to the future moment, the wheel rotation angle, and the coordinates of the driving trajectory point at the reference moment, may include steps A221-A222:

[0134] Step A221, for each future moment, calculating the lateral travel distance corresponding to the future moment according to the travel distance corresponding to the future moment and the wheel rotation angle, and calculating the abscissa of the travel trajectory point of the vehicle at the future moment based on the lateral travel distance and the abscissa of the travel trajectory point at the reference moment;

[0135] It can be understood that, in one implementation, the lateral travel distance corresponding to the future moment can be calculated based on the product of the travel distance corresponding to the future moment and the cosine value of the wheel rotation angle, and the sum of the lateral travel distance and the horizontal coordinate of the travel trajectory point at the reference moment is calculated to obtain the horizontal coordinate of the vehicle's travel trajectory point at the future moment. Exemplarily, the travel distance corresponding to the future moment b is 10, the wheel rotation angle is 60 degrees, and the cosine value is 1 / 2. It can be calculated that the lateral coordinate distance corresponding to the future moment b is 5, and the horizontal coordinate of the travel trajectory point at the previous moment is 5. Then, the horizontal coordinate of the travel trajectory point of vehicle A at the future moment b can be calculated to be 10.

[0136] The process of calculating the horizontal coordinate of the driving trajectory point at the future time in step A221 can be embodied in the form of a formula. In this case, the reference time is the previous time, as shown in Formula 3:

[0137] Formula 3

[0138] xN=x(N-1)+dsN*cos(vd)

[0139] Among them, xN represents the horizontal coordinate of the driving trajectory point, x(N-1) represents the horizontal coordinate of the driving trajectory point at the previous moment, dsN represents the driving distance corresponding to the future moment, and vd represents the wheel rotation angle of the vehicle.

[0140] Step A222, for each future moment, calculate the longitudinal driving distance corresponding to the future moment according to the driving distance corresponding to the future moment and the wheel rotation angle, and calculate the longitudinal coordinate of the driving trajectory point of the vehicle at the future moment based on the longitudinal driving distance and the longitudinal coordinate of the driving trajectory point at the reference moment.

[0141] It can be understood that, in one implementation, the longitudinal travel distance corresponding to the future moment can be calculated based on the product of the travel distance corresponding to the future moment and the sine value of the wheel rotation angle, and the sum of the longitudinal travel distance and the ordinate of the travel trajectory point at the reference moment is calculated to obtain the ordinate of the vehicle's travel trajectory point at the future moment. Exemplarily, the travel distance corresponding to the future moment b is 10, the wheel rotation angle is 60 degrees, and the sine value is about 0.866. The longitudinal coordinate distance corresponding to the future moment b can be calculated to be 8.66, and the ordinate of the travel trajectory point at the previous moment is 6. Then, the abscissa of the travel trajectory point of vehicle A at the future moment b can be calculated to be 14.66.

[0142] The process of calculating the ordinate of the driving trajectory point at the future time in step A222 can be expressed in the form of a formula, in which the reference time is the previous time, as shown in Formula 4:

[0143] Formula 4

[0144] yN=y(N-1)+dsN*sin(vd)

[0145] Among them, yN represents the ordinate of the driving trajectory point, y(N-1) represents the ordinate of the driving trajectory point at the previous moment, dsN represents the driving distance corresponding to the future moment, and vd represents the wheel rotation angle of the vehicle.

[0146] Step A23, based on the obtained coordinates of the driving trajectory points of the vehicle, determine the driving trajectory points of the vehicle at each future moment, and obtain multiple driving trajectory points of the vehicle.

[0147] It is understandable that, for each future moment, based on the calculated horizontal coordinate and vertical coordinate of the vehicle's driving trajectory point at the future moment, the vehicle's driving trajectory point at the future moment can be accurately determined. At this time, the vehicle's driving trajectory points at each future moment are counted to obtain multiple vehicle driving trajectory points.

[0148] In order to better understand the part about the driving trajectory points, the following will be introduced in conjunction with the accompanying drawings, such as Figure 4 Shown:

[0149] There are four driving trajectory points in the figure, namely P0, P1, P2 and P3, among which the horizontal coordinates of P0 are x0, y0, the horizontal coordinates of P1 are x1, y1, the horizontal coordinates of P2 are x2, y2, and the horizontal coordinates of P3 are x3, y3.

[0150] It can be understood that the vehicle travels from the reference point to P0, and the corresponding driving distance of P0 is ds. Therefore, since P0 is the first point, the product of the driving distance corresponding to P0 and the cosine value of the wheel rotation angle is calculated to obtain the horizontal coordinate x0 of P0. The product of the driving distance corresponding to P0 and the sine value of the wheel rotation angle is calculated to obtain the vertical coordinate y0 of P0. The other driving trajectory points are calculated in this way.

[0151] It can be seen that the embodiment of the present disclosure can obtain multiple driving trajectory points of the vehicle by calculating the horizontal and vertical coordinates of the driving trajectory points. The driving trajectory points at each future moment are calculated, which facilitates the subsequent steps to draw more accurate driving auxiliary lines. Therefore, this solution can realize driving assistance prompts on the remote cockpit side, thereby improving the driving safety of the vehicle.

[0152] It should be emphasized that the above-mentioned specific implementation method of predicting multiple driving trajectory points of the vehicle based on the first change speed and the driving information of the vehicle is only an example and should not constitute a limitation of the present disclosure.

[0153] S305, determining a wheel track line of the vehicle according to a plurality of driving track points;

[0154] It can be understood that by connecting multiple driving trajectory points, a driving trajectory line can be obtained. Since the vehicle's wheel trajectory lines can be two, namely the trajectory line of the left wheel and the trajectory line of the right wheel, the driving trajectory line can be offset into two to obtain the vehicle's wheel trajectory line.

[0155] Optionally, in one implementation, the multiple driving trajectory points are trajectory points of the vehicle in a vehicle coordinate system; wherein the vehicle coordinate system takes a reference point of the vehicle as an origin;

[0156] Accordingly, step S305 may include steps B1-B3:

[0157] Step B1, determining a driving trajectory line formed by the plurality of driving trajectory points;

[0158] It is understandable that a plurality of driving trajectory points may be connected to form a driving trajectory line. For example, the driving trajectory points P0, P1, P2 and P3 of vehicle A may be connected to obtain a driving trajectory line of vehicle A. The connection line may be a smooth connection line.

[0159] Step B2, performing an offset process on the driving trajectory line according to the lateral offset information of the left wheel of the vehicle relative to the reference point, to obtain a first trajectory line of the left wheel;

[0160] It is understandable that the lateral offset information of the left wheel of the vehicle relative to the reference point can be determined in the vehicle coordinate system. Based on the lateral offset information, the driving trajectory line can be offset to the left to obtain the first trajectory line of the left wheel. It should be emphasized that the first trajectory line of the left wheel is displayed in the vehicle coordinate system. For example, the lateral offset information of the left wheel of vehicle A relative to the reference point is 4m, and the driving trajectory line is offset to the left by 4m to obtain the first trajectory line of the left wheel.

[0161] Step B3: performing an offset process on the driving trajectory line according to the lateral offset information of the right wheel of the vehicle relative to the reference point to obtain a second trajectory line of the right wheel.

[0162] It is understandable that the lateral offset information of the right wheel of the vehicle relative to the reference point can be determined in the vehicle coordinate system. Based on the lateral offset information, the driving trajectory line can be offset to the right to obtain the second trajectory line of the right wheel. It should be emphasized that the second trajectory line of the right wheel is displayed in the vehicle coordinate system. For example, the lateral offset information of the right wheel of vehicle A relative to the reference point is 4m, and the driving trajectory line is offset to the right by 4m to obtain the second trajectory line of the right wheel.

[0163] In order to better understand the content about wheel track line, the following will be introduced with reference to the accompanying drawings, such as Figure 5 As shown:

[0164] The vehicle coordinate system takes the vehicle's reference point as its origin, olx is the lateral offset information of the left wheel relative to the reference point, orx is the lateral offset information of the right wheel relative to the reference point, and oy is the longitudinal offset information of the image acquisition device on the vehicle compared to the reference point.

[0165] It can be seen that the embodiment of the present disclosure offsets the driving trajectory line based on the lateral offset information of the left and right wheels of the vehicle relative to the reference point in the vehicle coordinate system, and can obtain the first trajectory line of the left wheel and the second trajectory line of the right wheel, thereby facilitating the subsequent steps of drawing more accurate driving auxiliary lines. Therefore, this solution can realize driving assistance prompts on the remote cockpit side, thereby improving the driving safety of the vehicle.

[0166] S306, drawing the wheel track line in the target image to obtain a driving auxiliary line.

[0167] It can be seen that due to the transmission delay problem in the remote driving process, this solution also takes the transmission delay into account when predicting the driving trajectory points, so that the predicted driving trajectory points are more accurate, and then more accurate driving auxiliary lines can be drawn in the target image, so as to realize driving auxiliary prompts on the remote cockpit side and improve the driving safety of the vehicle.

[0168] Optionally, based on the above embodiment, in another embodiment of the present disclosure, the step of drawing the wheel track line in the target image to obtain a driving auxiliary line includes step C1:

[0169] Step C1, drawing the wheel track line in the target image according to the position parameters of the image acquisition device relative to the vehicle to obtain a driving auxiliary line.

[0170] It is understandable that the position parameters of the image acquisition device relative to the vehicle can be the installation height of the image acquisition device relative to the vehicle reference point, and the pitch angle of the image acquisition device relative to the vertical direction of the space, wherein the vehicle reference point includes but is not limited to the center point of the vehicle, the center point of the front of the vehicle, the center point of the rear of the vehicle, etc., of course, not limited to this, and the pitch angle can also be called the Pitch angle. It should be emphasized that the wheel track line is located in the vehicle coordinate system, and the target image is located in the image coordinate system. In the process of converting the vehicle coordinate system to the image coordinate system, it is necessary to first convert it to the camera coordinate system, wherein the camera coordinate system represents the coordinate system with the reference point of the image acquisition device as the origin.

[0171] Optionally, in one implementation, step C1, drawing the wheel track line in the target image according to the position parameter of the image acquisition device relative to the vehicle to obtain the driving auxiliary line, may include steps C11-C13:

[0172] Step C11, based on the installation height and installation pitch angle of the image acquisition device, converting the wheel trajectory line into a third trajectory line in the camera coordinate system of the image acquisition device;

[0173] It can be understood that the wheel trajectory line is translated in the vertical direction based on the installation height of the image acquisition device, and then the translated wheel trajectory line is rotated according to the installation pitch angle, so that the wheel trajectory line can be converted into a third trajectory line in the camera coordinate system. It should be emphasized that the wheel trajectory line can also be rotated according to the installation pitch angle first, and then translated according to the installation height, so as to obtain the third trajectory line in the camera coordinate system. The embodiments of the present disclosure are not specifically limited. For example, the installation height of the camera is 1.6m and the installation pitch angle is 60 degrees, so the wheel trajectory line can be translated 1.6m in the vertical direction and then rotated 60 degrees to obtain the third trajectory line in the camera coordinate system.

[0174] In order to facilitate the understanding of the above part, a brief description is given below in conjunction with the accompanying drawings:

[0175] like Figure 6As shown, the camera is installed on the top of the vehicle, oz is the installation height of the camera, oa is the installation pitch angle, and the vertical direction can be understood as the z-axis of the camera coordinate system.

[0176] In addition, based on the installation height and installation pitch angle of the image acquisition device, the wheel trajectory line is converted into a third trajectory line in the camera coordinate system of the image acquisition device, that is, the wheel trajectory line in the vehicle coordinate system is converted to the camera coordinate system, wherein the vehicle coordinate system and the camera coordinate system can be pre-calibrated with a conversion matrix, and the parameters involved in the conversion matrix may include the installation height and the installation pitch angle. The present disclosure does not limit the method for determining the conversion matrix between the vehicle coordinate system and the camera coordinate system.

[0177] Step C12, converting the third trajectory line into a fourth trajectory line in the image coordinate system of the image acquisition device;

[0178] It should be emphasized that the process of converting the third trajectory line in the camera coordinate system into the fourth trajectory line in the image coordinate system can be understood as the conversion from a three-dimensional coordinate system to a two-dimensional coordinate system. The camera coordinate system takes the camera as the origin, while the image coordinate system takes the image plane as the origin. The vertical distance between the camera coordinate system and the image coordinate system is the focal length f. Then, according to the principle of similar triangles, the third trajectory line can be converted into the fourth trajectory line.

[0179] In order to better understand the content of converting from the camera coordinate system to the image coordinate system, the following will be explained with the accompanying drawings:

[0180] As shown in Figure 7(a), Figure 7(a) is a stereogram of the transformation between the camera coordinate system and the image coordinate system, wherein point C is the camera center, the rectangle is the image screen, and the Z axis is the principal axis. It can be understood that the coordinate system with point C as the origin is the camera coordinate system, and the camera coordinate system has an X-axis, a Y-axis, and a Z-axis; and the coordinate system with point P in the image plane as the origin is the image coordinate system, and the image coordinate system has an x-axis and a y-axis. A straight line passing through point C intersects with the image screen at point N1, and there is also point N2 on the straight line.

[0181] As shown in FIG7(b), FIG7(b) is a plane diagram of the transformation between the camera coordinate system and the image coordinate system, wherein the transformation between the camera coordinate system and the image coordinate system is similar to the principle of similar triangles. In FIG7(b), the distance between point N1 and point P is: fY / Z, where f represents the focal length.

[0182] It is understandable that this part can also be embodied in the form of a formula, as shown in Formula 5:

[0183] Formula 5

[0184] Xc / xi=Zc / f→xi=Xc*f / Zc

[0185] Yc / yi=Zc / f→yi=Yc*f / Zc

[0186] Among them, (Xc, Yc, Zc) represents the x-axis, Y-axis and Z-axis coordinates of any point in the camera coordinate system, (xi, yi) represents the horizontal coordinate of any point in the image coordinate system, and f represents the focal length.

[0187] Step C13: drawing the fourth trajectory line in the target image to obtain the driving auxiliary line.

[0188] It can be understood that the fourth trajectory line is a trajectory line in the image coordinate system, and the driving auxiliary line can be obtained by drawing the fourth trajectory line in the target image.

[0189] It can be seen that the embodiment of the present disclosure converts the wheel trajectory line in the vehicle coordinate system into the image coordinate system through two coordinate system conversions, thereby obtaining a driving auxiliary line, realizing driving auxiliary prompts on the remote cockpit side, and improving the driving safety of the vehicle. By displaying the wheel trajectory line on the screen after coordinate conversion, the amount of calculation can be reduced, the calculation time can be reduced, and real-time performance can be ensured. Of course, in another embodiment of the present disclosure, the calculated driving trajectory points can also be converted to obtain the coordinates of each driving trajectory point in the image coordinate system, obtain the mapped points, and connect the mapped points to obtain the driving auxiliary line.

[0190] Optionally, based on the above embodiment, in another implementation of the present disclosure, the method further includes:

[0191] In response to receiving a first heartbeat message periodically sent by the vehicle, detecting whether the first heartbeat message meets a risk condition set for remote driving;

[0192] In response to the first heartbeat information meeting the risk condition, a remote-end coping operation corresponding to the risk condition is performed.

[0193] Optionally, the method further comprises:

[0194] A second heartbeat message is periodically sent to the vehicle so that the vehicle, in response to receiving the second heartbeat message, detects whether the second heartbeat information meets the risk conditions set for remote driving, and if so, executes a vehicle-side response operation corresponding to the risk condition.

[0195] Optionally, any heartbeat message includes: a timestamp and a message count value;

[0196] The risk condition includes one or more of a first risk condition, a second risk condition, and a third risk condition;

[0197] The first risk condition includes: a difference between a message count value in a currently received heartbeat message and a message count value in a previously received heartbeat message is greater than a first threshold;

[0198] The second risk condition includes: for a plurality of consecutive heartbeat messages received, the difference between the message count values ​​in each two adjacent heartbeat messages is greater than a second threshold; wherein the number of the plurality of consecutive heartbeat messages is at least three;

[0199] The third risk condition includes: a difference between a timestamp in a currently received heartbeat message and a timestamp in a previously received heartbeat message is greater than a third threshold.

[0200] Optionally, the remote-end response operation corresponding to the first risk condition and the remote-end response operation corresponding to the third risk condition include: outputting a prompt message indicating that the vehicle is out of control; the vehicle-end response operation corresponding to the first risk condition and the vehicle-end response operation corresponding to the third risk condition include: triggering a parking operation;

[0201] The remote end response operation corresponding to the second risk condition includes: outputting a prompt message for prompting network instability;

[0202] The vehicle-side response operation corresponding to the second risk condition includes: sending a prompt message to the remote control device to indicate that the network is unstable.

[0203] It can be understood that the above content is consistent with a data processing method for remote driving provided in an embodiment of the present disclosure, so the specific introduction of the above content will be introduced in subsequent embodiments.

[0204] In addition, the present disclosure also provides a data processing method for remote driving. Figure 8 As shown, this may include:

[0205] S801, in response to receiving a first heartbeat message periodically sent by the vehicle, detecting whether the first heartbeat message meets a risk condition set for remote driving;

[0206] It is understandable that the first domain controller of the remote control device has been introduced above, so the remote control device can also receive the first heartbeat message sent by the vehicle through the first domain controller, and detect whether the first heartbeat message meets the risk conditions set for remote driving.

[0207] Optionally, any heartbeat message includes: a timestamp and a message count value;

[0208] The risk condition includes one or more of a first risk condition, a second risk condition, and a third risk condition;

[0209] Correspondingly, the first risk condition includes: a difference between a message count value in a currently received heartbeat message and a message count value in a previously received heartbeat message is greater than a first threshold;

[0210] The second risk condition includes: for a plurality of consecutive heartbeat messages received, the difference between the message count values ​​in each two adjacent heartbeat messages is greater than a second threshold; wherein the number of the plurality of consecutive heartbeat messages is at least three;

[0211] The third risk condition includes: a difference between a timestamp in a currently received heartbeat message and a timestamp in a previously received heartbeat message is greater than a third threshold.

[0212] It should be emphasized that the message count value in the heartbeat message is 0-255 in a cycle. For example, if the message count value in the previous heartbeat message received is 100 and the message count value in the currently received heartbeat message is 105, then the difference is 5.

[0213] It is understandable that the risk conditions can be pre-set by the user. Only when the first risk condition, the second risk condition and the third risk condition are met at the same time, it is considered that the risk conditions set for remote driving are met. Alternatively, only one risk condition may be met to be considered that the risk conditions set for remote driving are met. The present disclosed embodiment does not make specific limitations on this.

[0214] It can be understood that, for the first risk condition, when the difference between the message count values ​​in the two received heartbeat messages is greater than the first threshold, the first risk condition is considered to be met. Exemplarily, the message count value in the previous heartbeat message is 20, the message count value in the currently received heartbeat message e is 35, the difference is 15, and the first threshold is 10. At this time, it can be considered that the currently received heartbeat message e meets the first risk condition.

[0215] It can be understood that for the second risk condition, it is required that the difference between the message count values ​​in each of the two adjacent heartbeat messages of a continuous plurality of heartbeat messages is greater than the second threshold value, then it is considered that the second risk condition is met. In this way, if the difference between the message count values ​​in one adjacent heartbeat message is greater than the second threshold value, then it is considered that the second risk condition is not met, wherein the continuous plurality of heartbeat messages can be 5, 6, etc., which are not specifically limited in the embodiment of the present disclosure. Exemplarily, three heartbeat messages are received continuously, namely, heartbeat message f, heartbeat message g, and heartbeat message h, wherein the message count value in heartbeat message f is 248, the message count value in heartbeat message g is 7, and the message count value in heartbeat message h is 20. Then, it can be considered that the difference between heartbeat message f and heartbeat message g is 14, the difference between heartbeat message g and heartbeat message h is 13, and the second threshold value is 10, which are all greater than the second threshold value. It can be considered that the multiple heartbeat messages received continuously meet the second risk condition.

[0216] It can be understood that, for the third risk condition, when the difference between the timestamps in the two received heartbeat messages is greater than the third threshold, the third risk condition is considered to be met. Exemplarily, the timestamp in the previous heartbeat message is 0706112037, that is, 11:20:37 on July 6, and the timestamp in the currently received heartbeat message g is 0706112105, that is, 11:21:01 on July 6, the difference is 24 seconds, and the third threshold is 20 seconds. At this time, it can be considered that the currently received heartbeat message g meets the third risk condition.

[0217] Of course, other risk conditions may also be set, such as requiring that the difference between the timestamps of multiple consecutive heartbeat messages between each two adjacent heartbeat messages is greater than a fourth threshold. The disclosed embodiment does not limit the risk conditions set for remote driving.

[0218] It can be seen that the embodiment of the present disclosure satisfies different situations by setting three risk conditions. It can detect the timestamp in the heartbeat message, and can also detect the message count value in the heartbeat message, so as to detect the risk conditions set for remote driving. Therefore, this scheme detects heartbeat messages at multiple levels, which can improve the safety of remote driving.

[0219] S802: In response to the first heartbeat information meeting the risk condition, executing a remote-end coping operation corresponding to the risk condition.

[0220] It can be understood that, in response to the first heartbeat message meeting the risk condition, the remote end response operations corresponding to the first risk condition, the second risk condition and the third risk condition are executed.

[0221] Optionally, in one implementation, the remote-end response operation corresponding to the first risk condition and the remote-end response operation corresponding to the third risk condition include: outputting a prompt message indicating that the vehicle is out of control; the vehicle-end response operation corresponding to the first risk condition and the vehicle-end response operation corresponding to the third risk condition include: triggering a parking operation, which may include pull-over parking and / or direct parking;

[0222] The remote end response operation corresponding to the second risk condition includes: outputting a prompt message for prompting network instability;

[0223] The vehicle-side response operation corresponding to the second risk condition includes: sending a prompt message to the remote control device to indicate that the network is unstable.

[0224] It should be noted that the first risk condition and the second risk condition are to detect the message count value or timestamp in the currently received heartbeat message and the previous heartbeat message, so the risk level of the first risk condition and the third risk condition is the same, so the corresponding remote end and vehicle end response operations are also the same. In addition, the vehicle end response operation is the vehicle side detection that the second heartbeat message meets the risk condition set for remote driving, and the vehicle end response operation that has been matched with the risk condition is executed. The safety detection of the vehicle will be introduced later.

[0225] It can be understood that the remote-side response operation can be outputting a prompt message, while the vehicle-side response operation can be parking or outputting a prompt message. The remote-side response operation corresponding to the first risk condition and the third risk condition is to output a prompt message of vehicle out of control, thereby reminding the driver that the vehicle is out of control, so that the driver can stop the car to ensure safe driving. The vehicle-side response operation corresponding to the first risk condition and the third risk condition is to trigger a parking operation. Then, when the first risk condition and the third risk condition are met, it is a more serious situation for remote driving, so it is necessary to stop the car directly. The remote-side response operation corresponding to the second risk condition is to output a prompt message for prompting network instability, thereby reminding the driver that the current network is unstable and please drive carefully. The vehicle-side response operation corresponding to the second risk condition is to send a prompt message for prompting network instability to the remote control device. It can be seen that when the second risk condition is met, it is not as serious as when the first risk condition and the third risk condition are met, so only a prompt message for prompting network instability is output.

[0226] It can be seen that the embodiment of the present disclosure presets corresponding remote-end response operations and vehicle-end response operations for different risk conditions, so that when network delays or network instability occur, prompt messages can be output to the driver or parking operations can be directly triggered, thereby improving the safety of remote driving.

[0227] Optionally, in one implementation, the method further includes:

[0228] A second heartbeat message is periodically sent to the vehicle so that the vehicle, in response to receiving the second heartbeat message, detects whether the second heartbeat information meets the risk conditions set for remote driving, and if so, executes a vehicle-side response operation corresponding to the risk condition.

[0229] It is understandable that if it is detected that the first heartbeat message does not meet the risk condition set for remote driving, a second heartbeat message can be periodically sent to the vehicle. It should be emphasized that the heartbeat message periodically sent by the remote cockpit is called the first heartbeat message, and the heartbeat message periodically sent by the vehicle can be called the second heartbeat message. The first heartbeat message and the second heartbeat message are only used to distinguish the sender, and do not limit this solution.

[0230] It is understandable that the vehicle detects whether the second heartbeat message meets the risk conditions set for remote driving in response to the second heartbeat message. If it meets the risk conditions set for remote driving, the vehicle-side response operation corresponding to the risk conditions is performed. If it does not meet the risk conditions set for remote driving, the first heartbeat message continues to be sent to the remote cockpit. In addition, the risk conditions set for remote driving of the vehicle are also one or more of the first risk conditions, the second risk conditions and the third risk conditions, which have been introduced above and will not be repeated here.

[0231] Exemplarily, in response to receiving the second heartbeat message, vehicle A detects whether the difference between the message count value in the second heartbeat message and the message count value in the previous second heartbeat message received is greater than the first threshold. At this time, the difference is 20 and the first threshold is 10, so the first risk condition is met, triggering the parking operation.

[0232] It can be seen that the embodiment of the present disclosure adopts a method of cyclically sending heartbeat messages to ensure the safety of remote driving. The safety of remote driving can be improved by continuously sending heartbeat messages between the vehicle and the remote cockpit and detecting risk conditions set for remote driving.

[0233] The following will introduce the content of safety detection from both the vehicle and the remote cockpit, with the help of the attached pictures, such as Fig. 9 , Fig.10 As shown:

[0234] Fig. 9 To introduce the safety inspection of the vehicle, the risk conditions set for remote driving are the first risk condition, the second risk condition and the third risk condition.

[0235] S901, receiving a second heartbeat message forwarded by a cloud server;

[0236] It can be understood that the cloud server is used to forward various messages between the vehicle and the remote cockpit, which will be specifically introduced in subsequent embodiments and will not be elaborated here.

[0237] S902, determining the counter value of the current second heartbeat message;

[0238] S903, detecting whether a difference between a message count value in a currently received second heartbeat message and a message count value in a previously received second heartbeat message is greater than a first threshold;

[0239] It can be understood that if the difference is greater than the first threshold, step S907 is executed, and if the difference is less than the first threshold, step S904 is executed;

[0240] S904, detecting whether the difference between the message count values ​​in each two adjacent second heartbeat messages received is greater than a second threshold;

[0241] It can be understood that if the difference values ​​are greater than the second threshold, step S907 is executed, and if the difference values ​​are less than the second threshold, step S905 is executed;

[0242] S905, detecting whether a difference between a timestamp in a currently received second heartbeat message and a timestamp in a previously received second heartbeat message is greater than a third threshold;

[0243] It can be understood that if the difference is greater than the third threshold, step S907 is executed, and if the difference is less than the third threshold, step S906 is executed;

[0244] S906, sending a first heartbeat message to the remote cockpit;

[0245] S907, controlling the vehicle to make an emergency stop.

[0246] It can be understood that the above content is consistent with the content introduced in the above embodiment and will not be repeated here.

[0247] Fig.10 To introduce the safety detection of the remote cockpit, the risk conditions set for remote driving are the first risk condition, the second risk condition and the third risk condition.

[0248] S1001, receiving a first heartbeat message forwarded by a cloud server;

[0249] S1002, determining the counter value of the current first heartbeat message;

[0250] S1003, detecting whether a difference between a message count value in a currently received first heartbeat message and a message count value in a previously received first heartbeat message is greater than a first threshold;

[0251] It can be understood that if the difference is greater than the first threshold, step S1007 is executed, and if the difference is less than the first threshold, step S1004 is executed;

[0252] S1004, detecting whether the difference between the message count values ​​in each two adjacent first heartbeat messages received is greater than a second threshold;

[0253] It can be understood that if the difference values ​​are greater than the second threshold, step S1007 is executed, and if the difference values ​​are less than the second threshold, step S1005 is executed;

[0254] S1005, detecting whether a difference between a timestamp in a currently received first heartbeat message and a timestamp in a previously received first heartbeat message is greater than a third threshold;

[0255] It can be understood that if the difference is greater than the third threshold, step S1007 is executed, and if the difference is less than the third threshold, step S1006 is executed;

[0256] S1006, sending a second heartbeat message to the vehicle;

[0257] S1007, stop remote driving.

[0258] It can be understood that the above content is consistent with the content introduced in the above embodiment and will not be repeated here.

[0259] In the disclosed embodiment, the vehicle and the remote cockpit can send heartbeat messages to each other to detect whether the heartbeat message meets the risk condition. If it meets the risk condition, the corresponding response operation is performed. It can be seen that the disclosed embodiment has multiple risk detection methods, and can detect heartbeat messages from different levels, thereby improving the safety of remote driving. For problems such as network delay and network instability, corresponding processing can also be given to ensure the safety of remote driving.

[0260] Moreover, it should be understood that the autonomous driving vehicle may encounter a vehicle autonomous driving system failure (such as a software failure such as a control algorithm failure, or a hardware failure such as a sensor failure) during driving, or encounter a situation where an obstacle cannot be identified. At this time, it may be necessary to actively seek help from the remote control device, wherein the vehicle autonomous driving system is located in the autonomous driving vehicle and assists the vehicle in autonomous driving. Therefore, the first heartbeat message sent by the vehicle end to the remote end can also carry the request sent by the vehicle end to the remote end, such as sending a vehicle takeover request to request the remote driving end to control the vehicle driving, requesting the remote end to transmit the surrounding environment information or the judgment result of the surrounding environment to the vehicle end, such as the identification result of the obstacle, etc. The second heartbeat message can also carry the instruction sent by the remote end to the vehicle end, which includes but is not limited to: a turn instruction, a lane change instruction, an acceleration instruction, a deceleration instruction, a pull-over instruction, an immediate stop instruction, a U-turn instruction, a loading and unloading instruction, etc., and the embodiment of the present disclosure does not specifically limit the instruction.

[0261] Further, the instructions, requests and collected data to be sent by the present disclosure can be set in the heartbeat information and sent together with the heartbeat message. The first heartbeat message and the second heartbeat message can both include a high-frequency heartbeat message and a low-frequency heartbeat message, and the sending frequency of the high-frequency heartbeat message is faster than the sending frequency of the low-frequency heartbeat message. In some embodiments, the high-frequency heartbeat message is a heartbeat message with a sending frequency greater than or equal to the first frequency threshold, and the low-frequency heartbeat message is a heartbeat message with a sending frequency less than the first frequency threshold. Among them, the high-frequency heartbeat message can carry some relatively more important information (such as vehicle control instructions) to ensure the real-time nature of the instructions, including but not limited to: instructions to turn on remote driving, instructions to turn off remote driving, and vehicle control instructions sent by the remote cockpit during the process of controlling the vehicle's driving; prompt information of vehicle faults sent by the vehicle end, vehicle speed, vehicle steering angle, and vehicle acceleration. The low-frequency heartbeat can carry relatively less important information, including but not limited to: vehicle location information, vehicle voice broadcast information, and vehicle safety verification information sent by the vehicle end. The embodiments of the present disclosure do not specifically limit the content contained in the high-frequency and low-frequency heartbeat messages.

[0262] Among them, the vehicle voice broadcast information includes but is not limited to the broadcast information of the vehicle turning on remote driving, the broadcast information of the vehicle turning off remote driving, the broadcast information of the vehicle turning, and the broadcast information of the vehicle risk warning. Vehicle safety verification information can be sent by the vehicle end or by the remote end. In this way, whether it is information that the remote end wants to send to the vehicle end, or information that the vehicle end wants to send to the remote end, the attribute of the information will be judged first, and the information will be assigned to the heartbeat message of the corresponding frequency for sending according to the attribute. For example, the speed and position information collected by the vehicle end at the same time, the speed information will be sent to the remote control device with the high-frequency heartbeat message, and the position information will be sent to the remote control device with the low-frequency heartbeat message.

[0263] Optionally, the present disclosure can monitor the command response delay between the remote control device and the vehicle control device in real time or periodically, and adjust the sending time of the heartbeat message in real time or periodically. For example, the command response delay between the remote control device and the vehicle control device is calculated, and the subsequent heartbeat message to be sent is sent in advance according to the command response delay. Among them, the specified response delay includes the delay after the remote control device sends the control command to the vehicle control device and the vehicle control device starts to execute the command. For example, the remote control device sends a stop command to the vehicle control device at time t1, and the vehicle control device starts to execute the stop command at time t2, then the command response delay is t2-t1. In this way, the subsequent heartbeat message to be sent can be sent in advance (t2-t1) time.

[0264] In some real-time examples, the remote control device of the present disclosure may be multiple remote control devices, each of which may be used to control the travel of one or more vehicles, and the multiple remote control devices may reside in the same space or be distributed in different spaces. Then, when a vehicle needs to seek help from a remote control device, it may send a communication connection request to the remote control device that is currently closest to it, so that the remote control device may send a control instruction to it or transmit the required data to it.

[0265] Optionally, the cloud server is connected to multiple remote control devices for communication, and the request sent by the vehicle is first sent to the remote control device, and the cloud server allocates the request and sends the request to the allocated remote control device, for example: analyzing and determining the remote control device that is currently idle and in working state to process the request. Among them, the remote control device in working state indicates that there is a driver in the corresponding remote cockpit. In addition, the vehicle control device also has an identification algorithm to identify the current fault type and fault level. The request sent by the vehicle control device also carries a fault type identifier and a fault level identifier, so that the cloud server allocates the request according to the fault type identifier and the fault level identifier, and the critical fault level will be allocated to the remote control device with the most idle resources or the fastest average response speed for processing.

[0266] Of course, the vehicle can also directly establish a communication connection with the remote control device before starting to run, so that the remote control device can monitor the running status of the vehicle at any time. At this time, the vehicle can rely on its own automatic driving system to control its own driving, or it can be directly controlled by the remote control device. When the vehicle encounters danger or failure, it can actively request the remote control device to take over; the remote control device can also monitor the driving status of the vehicle in real time, and when it is judged that the vehicle is in danger based on the information on the screen, it will actively send control instructions to control the vehicle.

[0267] Optionally, based on the above embodiments, the present disclosure provides a data processing method for remote driving, such as Fig.11 As shown, the method also includes:

[0268] S1101, acquiring image data collected by a camera;

[0269] It can be understood that the image data collected by the camera is the target image in the above embodiment.

[0270] S1102, analyzing the image data;

[0271] It is understandable that analysis is performed to determine whether there is an obstacle in the image data.

[0272] S1103, determining whether there is a dangerous obstacle 50 meters ahead, and whether the driving auxiliary line intersects with the dangerous obstacle ahead;

[0273] Among them, dangerous obstacles can be pedestrians, vehicles or road edges, etc. For example, if there is a vehicle within 50m ahead and the vehicle intersects with the driving auxiliary line, a dangerous information can be prompted.

[0274] It is understandable that the determination of whether there is a dangerous obstacle 50 m ahead and whether the driving auxiliary line intersects with the dangerous obstacle ahead can be obtained through the analysis results of the image data.

[0275] It can be understood that if there is a dangerous obstacle and the driving auxiliary line intersects with the dangerous obstacle ahead, step S1104 is executed; if there is no dangerous obstacle and the driving auxiliary line does not intersect with the dangerous obstacle ahead, step S1101 is executed.

[0276] S1104, controlling the vehicle to make an emergency stop.

[0277] It can be seen that the embodiment of the present disclosure can analyze whether there are obstacles 50 meters ahead based on the image data collected by the camera, thereby improving the safety of remote driving.

[0278] Based on the above method embodiment, Fig.12 As shown, an embodiment of the present disclosure provides a driving auxiliary line generation system, the system comprising: a remote control device 1210, a vehicle control device 1220;

[0279] The vehicle control device 1220 is used to obtain the target image captured by the image acquisition device and send the target image to the remote control device 1210;

[0280] The remote control device 1210 is used to receive the target image captured by the image acquisition device during vehicle driving, calculate the transmission delay generated when the target image is received, calculate the first change speed according to the transmission delay and the driving information of the vehicle, predict the wheel trajectory line of the vehicle based on the first change speed and the driving information, draw the wheel trajectory line in the target image, and obtain a driving auxiliary line.

[0281] Optionally, the remote control device is a control device of a remote cockpit, and the vehicle control device sends the image to the remote control device, including:

[0282] The vehicle control device sends the image to a cloud server, so that the cloud server sends the image to the control device of the remote cockpit.

[0283] It can be understood that the vehicle control device can send the image to the cloud server, or it can send the image to the cloud server through the second domain controller, so that the cloud server sends the image to the control device of the remote cockpit, or the cloud server sends the image to the first domain controller of the remote cockpit. The present disclosed embodiment does not make specific limitations.

[0284] It should be emphasized that the cloud server is mainly responsible for forwarding and recording messages and images. Of course, the cloud server does not need to be involved in the entire system. The first domain controller and the second domain controller are mainly responsible for receiving and sending messages and images. The first domain controller is located in the remote cockpit, and the second domain controller is located in the vehicle.

[0285] Optionally, the remote control device predicts a wheel trajectory of the vehicle based on the first change speed and the driving information of the vehicle, including:

[0286] The remote control device predicts a plurality of driving trajectory points of the vehicle according to the first change speed and the driving information of the vehicle;

[0287] A wheel trajectory line of the vehicle is determined according to a plurality of driving trajectory points.

[0288] Optionally, the driving information includes at least one of the following: the speed and acceleration of the vehicle, and the steering angle and steering angle coefficient of the steering wheel of the vehicle.

[0289] Optionally, the remote control device predicts a plurality of driving trajectory points of the vehicle according to the first change speed and the driving information of the vehicle, including:

[0290] The remote control device predicts, for each future moment, the distance traveled by the vehicle in a time period from a corresponding reference moment to the future moment according to the speed, acceleration and the first change speed of the vehicle, as the travel distance corresponding to the future moment; the reference moment corresponding to each future moment is the moment before the future moment;

[0291] According to the driving distance corresponding to each future moment, as well as the steering angle and the steering angle coefficient, the driving trajectory points of the vehicle at each future moment are predicted to obtain a plurality of driving trajectory points of the vehicle.

[0292] Optionally, the remote control device predicts, for each future moment, a distance traveled by the vehicle in a time period from a corresponding reference moment to the future moment according to the speed, acceleration and the first change speed of the vehicle, as the travel distance corresponding to the future moment, including:

[0293] The remote control device calculates, for each future moment, a second change speed corresponding to the future moment according to the acceleration, the time difference between the future moment and the corresponding reference moment, and the target constant; wherein the target constant is related to the sorting sequence number of the future moment in the moment sequence, and the second change speed is the speed of the vehicle during the period from the current moment to the future moment;

[0294] For each future moment, the driving distance corresponding to the future moment is determined based on the speed, the first change speed, the second change speed corresponding to the future moment, and the time difference between the future moment and the corresponding reference moment.

[0295] Optionally, the remote control device predicts the driving trajectory points of the vehicle at each future moment according to the driving distance corresponding to each future moment, as well as the steering angle and the steering angle coefficient, to obtain multiple driving trajectory points of the vehicle, including:

[0296] Determining a wheel rotation angle of the vehicle based on the steering angle and the steering angle coefficient;

[0297] For each future moment, the coordinates of the driving trajectory point of the vehicle at the future moment are calculated according to the driving distance corresponding to the future moment, the wheel rotation angle, and the coordinates of the driving trajectory point at the reference moment;

[0298] Based on the obtained coordinates of the driving trajectory points of the vehicle, the driving trajectory points of the vehicle at each future moment are determined to obtain a plurality of driving trajectory points of the vehicle.

[0299] Optionally, the remote control device calculates the coordinates of the driving trajectory point of the vehicle at each future moment according to the driving distance corresponding to the future moment, the wheel rotation angle, and the coordinates of the driving trajectory point at the reference moment, including:

[0300] The remote control device calculates, for each future moment, a lateral travel distance corresponding to the future moment according to the travel distance corresponding to the future moment and the wheel rotation angle, and calculates the abscissa of the travel trajectory point of the vehicle at the future moment based on the lateral travel distance and the abscissa of the travel trajectory point at the reference moment;

[0301] For each future moment, the longitudinal driving distance corresponding to the future moment is calculated based on the driving distance corresponding to the future moment and the wheel rotation angle, and the longitudinal coordinate of the driving trajectory point of the vehicle at the future moment is calculated based on the longitudinal driving distance and the longitudinal coordinate of the driving trajectory point at the reference moment.

[0302] Optionally, the multiple driving trajectory points are trajectory points of the vehicle in a vehicle coordinate system; wherein the vehicle coordinate system takes the reference point of the vehicle as the origin;

[0303] The remote control device determines the wheel trajectory of the vehicle according to a plurality of driving trajectory points, including:

[0304] The remote control device determines a driving trajectory line formed by the plurality of driving trajectory points;

[0305] According to the lateral offset information of the left wheel of the vehicle relative to the reference point, the driving trajectory line is offset to obtain a first trajectory line of the left wheel;

[0306] The driving trajectory is offset according to the lateral offset information of the right wheel of the vehicle relative to a reference point to obtain a second trajectory of the right wheel.

[0307] Optionally, the remote control device draws the wheel track line in the target image to obtain the driving auxiliary line, including:

[0308] The remote control device draws the wheel track line in the target image according to the position parameters of the image acquisition device relative to the vehicle to obtain a driving auxiliary line.

[0309] Optionally, the remote control device draws the wheel track line in the target image according to the position parameter of the image acquisition device relative to the vehicle to obtain the driving auxiliary line, including:

[0310] The remote control device converts the wheel trajectory into a third trajectory in the camera coordinate system of the image acquisition device based on the installation height and the installation pitch angle of the image acquisition device;

[0311] Converting the third trajectory line into a fourth trajectory line in the image coordinate system of the image acquisition device;

[0312] The fourth trajectory line is drawn in the target image to obtain the driving auxiliary line.

[0313] Optionally, the remote control device is also used for:

[0314] Whether to issue a collision warning is determined based on the positional relationship between the driving assistance line and the edge of the road, as well as the positional relationship between the driving assistance line and other vehicles.

[0315] Optionally, the remote control device is also used for:

[0316] In response to receiving a first heartbeat message periodically sent by the vehicle, detecting whether the first heartbeat message meets a risk condition set for remote driving;

[0317] In response to the first heartbeat information meeting the risk condition, a remote-end coping operation corresponding to the risk condition is performed.

[0318] Optionally, the remote control device is also used for:

[0319] A second heartbeat message is periodically sent to the vehicle so that the vehicle, in response to receiving the second heartbeat message, detects whether the second heartbeat information meets the risk conditions set for remote driving, and if so, executes a vehicle-side response operation corresponding to the risk condition.

[0320] Optionally, any heartbeat message includes: a timestamp and a message count value;

[0321] The risk condition includes one or more of a first risk condition, a second risk condition, and a third risk condition;

[0322] The first risk condition includes: a difference between a message count value in a currently received heartbeat message and a message count value in a previously received heartbeat message is greater than a first threshold;

[0323] The second risk condition includes: for a plurality of consecutive heartbeat messages received, the difference between the message count values ​​in each two adjacent heartbeat messages is greater than a second threshold; wherein the number of the plurality of consecutive heartbeat messages is at least three;

[0324] The third risk condition includes: a difference between a timestamp in a currently received heartbeat message and a timestamp in a previously received heartbeat message is greater than a third threshold.

[0325] Optionally, the remote-end response operation corresponding to the first risk condition and the remote-end response operation corresponding to the third risk condition include: outputting a prompt message indicating that the vehicle is out of control; the vehicle-end response operation corresponding to the first risk condition and the vehicle-end response operation corresponding to the third risk condition include: triggering a parking operation;

[0326] The remote end response operation corresponding to the second risk condition includes: outputting a prompt message for prompting network instability;

[0327] The vehicle-side response operation corresponding to the second risk condition includes: sending a prompt message to the remote control device to indicate that the network is unstable.

[0328] Among them, the specific implementation method of the functions realized by the above-mentioned driving auxiliary line generation system can refer to the corresponding content of the above-mentioned method embodiment, which will not be repeated here.

[0329] The driving assistance line generation system provided by the embodiment of the present disclosure can, during the driving assistance line generation process, be based not only on vehicle driving information but also on the transmission delay of the received target image, so that relatively accurate driving assistance lines can be drawn in the target image. Therefore, this solution can implement driving assistance prompts on the remote cockpit side, thereby improving the driving safety of the vehicle.

[0330] Optionally, the embodiment of the present disclosure also provides another driving auxiliary line generation system, as shown in FIG. 13( a ) and FIG. 13( b ), which may include a cloud server 1310 , a vehicle 1320 , and a remote cockpit 1330 ;

[0331] FIG. 13( a ) introduces the function of generating auxiliary driving lines.

[0332] Among them, the steps executed by the internal device of the vehicle are: the image acquisition device performs image acquisition, and the second domain controller sends the acquired target image to the cloud server. The second domain controller can also be called an image transmission controller or a communication device; the steps executed by the internal device of the remote cockpit are: the first domain controller receives the target image, the remote control device reads the steering angle of the steering wheel in the cockpit and other driving information, the remote control device generates a wheel trajectory line based on the driving information, and the remote control device draws the wheel trajectory line in the target image to obtain a driving auxiliary line.

[0333] FIG13( b ) is about the safety detection function.

[0334] Among them, the steps executed by the internal device of the vehicle are: the second domain controller receives the second heartbeat information and sends the first heartbeat message, the second safety detection device performs a safety detection on the second heartbeat information, the image acquisition device performs image acquisition, the image detection device detects the acquired image, and the VCU unit (Vehicle Control Unit) performs corresponding response operations; the steps executed by the internal device of the remote cockpit are: the first domain controller receives the first heartbeat information and sends the second heartbeat message, the first safety detection device performs a safety detection on the first heartbeat information, and the remote control device controls the screen to display the reminder message.

[0335] Based on the above method embodiment, Fig.14 As shown, a driving auxiliary line generating device is applied to a remote control device, and the device comprises:

[0336] The receiving module 1410 is used to receive the target image captured by the image acquisition device on the vehicle during the vehicle's driving process;

[0337] A first calculation module 1420, configured to calculate a transmission delay generated when the target image is received;

[0338] The second calculation module 1430 is used to calculate a first change speed according to the transmission delay and the driving information of the vehicle; wherein the first change speed is the speed of the vehicle changing within the period of the transmission delay;

[0339] A prediction module 1440, configured to predict a wheel trajectory of the vehicle based on the first change speed and the driving information;

[0340] The drawing module 1450 is used to draw the wheel track line in the target image to obtain the driving auxiliary line.

[0341] Optionally, the prediction module includes:

[0342] A prediction submodule, configured to predict a plurality of driving trajectory points of the vehicle according to the first change speed and the driving information of the vehicle;

[0343] The determination submodule is used to determine the wheel trajectory line of the vehicle according to a plurality of driving trajectory points.

[0344] Optionally, the driving information includes at least one of the following: the speed and acceleration of the vehicle, and the steering angle and steering angle coefficient of the steering wheel of the vehicle.

[0345] Optionally, the prediction submodule includes:

[0346] a first prediction unit, configured to predict, for each future moment, a distance traveled by the vehicle in a time period from a corresponding reference moment to the future moment according to the speed, acceleration and the first change speed of the vehicle, as a travel distance corresponding to the future moment; the reference moment corresponding to each future moment is a moment before the future moment;

[0347] The second prediction unit is used to predict the driving trajectory points of the vehicle at each future moment according to the driving distance corresponding to each future moment, as well as the steering angle and the steering angle coefficient, to obtain multiple driving trajectory points of the vehicle.

[0348] Optionally, the first prediction unit includes:

[0349] a first calculation subunit, for calculating, for each future moment, a second change speed corresponding to the future moment according to the acceleration, a time difference between the future moment and the corresponding reference moment, and a target constant; wherein the target constant is related to a sorting sequence number of the future moment in the moment sequence, and the second change speed is a speed of the vehicle changed in a period from the current moment to the future moment;

[0350] The first determination subunit is used to determine, for each future moment, a driving distance corresponding to the future moment based on the speed, the first change speed, the second change speed corresponding to the future moment, and the time difference between the future moment and the corresponding reference moment.

[0351] Optionally, the second prediction unit includes:

[0352] A second determining subunit, configured to determine a wheel rotation angle of the vehicle based on the steering angle and the turning angle coefficient;

[0353] A second calculation subunit is used to calculate the coordinates of the driving trajectory point of the vehicle at each future moment according to the driving distance corresponding to the future moment, the wheel rotation angle, and the coordinates of the driving trajectory point at the reference moment;

[0354] The third determining subunit is used to determine the driving trajectory points of the vehicle at each future moment based on the obtained coordinates of the driving trajectory points of the vehicle, so as to obtain a plurality of driving trajectory points of the vehicle.

[0355] Optionally, the second computing subunit is specifically configured to:

[0356] For each future moment, the lateral travel distance corresponding to the future moment is calculated according to the travel distance corresponding to the future moment and the wheel rotation angle, and the abscissa of the travel trajectory point of the vehicle at the future moment is calculated based on the lateral travel distance and the abscissa of the travel trajectory point at the reference moment;

[0357] For each future moment, the longitudinal driving distance corresponding to the future moment is calculated based on the driving distance corresponding to the future moment and the wheel rotation angle, and the longitudinal coordinate of the driving trajectory point of the vehicle at the future moment is calculated based on the longitudinal driving distance and the longitudinal coordinate of the driving trajectory point at the reference moment.

[0358] Optionally, the multiple driving trajectory points are trajectory points of the vehicle in a vehicle coordinate system; wherein the vehicle coordinate system takes the reference point of the vehicle as the origin;

[0359] The determination submodule comprises:

[0360] A determination unit, used to determine a driving trajectory line formed by the plurality of driving trajectory points;

[0361] A first offset processing unit, configured to perform an offset processing on the driving trajectory line according to the lateral offset information of the left wheel of the vehicle relative to the reference point, so as to obtain a first trajectory line of the left wheel;

[0362] The second offset processing unit is used to perform offset processing on the driving trajectory line according to the lateral offset information of the right wheel of the vehicle relative to the reference point to obtain a second trajectory line of the right wheel.

[0363] Optionally, the drawing module includes:

[0364] The drawing submodule is used to draw the wheel track line in the target image according to the position parameters of the image acquisition device relative to the vehicle to obtain a driving auxiliary line.

[0365] Optionally, draw submodules, including:

[0366] A first conversion unit, configured to convert the wheel trajectory line into a third trajectory line in a camera coordinate system of the image acquisition device based on an installation height and an installation pitch angle of the image acquisition device;

[0367] A second conversion unit, used for converting the third trajectory line into a fourth trajectory line in the image coordinate system of the image acquisition device;

[0368] A drawing unit is used to draw the fourth trajectory line in the target image to obtain the driving auxiliary line.

[0369] Optionally, the device further comprises:

[0370] The determination module is used to determine whether to issue a collision warning based on the position relationship between the driving auxiliary line and the road edge, and the position relationship between the driving auxiliary line and other vehicles.

[0371] Optionally, the device further comprises:

[0372] a detection module, configured to detect, in response to receiving a first heartbeat message periodically sent by the vehicle, whether the first heartbeat message meets a risk condition set for remote driving;

[0373] An execution module is used to execute a remote-end coping operation corresponding to the risk condition in response to the first heartbeat information meeting the risk condition.

[0374] Optionally, the device further comprises:

[0375] A sending module is used to periodically send a second heartbeat message to the vehicle, so that the vehicle responds to receiving the second heartbeat message by detecting whether the second heartbeat information meets the risk conditions set for remote driving, and if so, executes a vehicle-side response operation corresponding to the risk condition.

[0376] Optionally, any heartbeat message includes: a timestamp and a message count value;

[0377] The risk condition includes one or more of a first risk condition, a second risk condition, and a third risk condition;

[0378] The first risk condition includes: a difference between a message count value in a currently received heartbeat message and a message count value in a previously received heartbeat message is greater than a first threshold;

[0379] The second risk condition includes: for a plurality of consecutive heartbeat messages received, the difference between the message count values ​​in each two adjacent heartbeat messages is greater than a second threshold; wherein the number of the plurality of consecutive heartbeat messages is at least three;

[0380] The third risk condition includes: a difference between a timestamp in a currently received heartbeat message and a timestamp in a previously received heartbeat message is greater than a third threshold.

[0381] Optionally, the remote-end response operation corresponding to the first risk condition and the remote-end response operation corresponding to the third risk condition include: outputting a prompt message indicating that the vehicle is out of control; the vehicle-end response operation corresponding to the first risk condition and the vehicle-end response operation corresponding to the third risk condition include: triggering a parking operation;

[0382] The remote end response operation corresponding to the second risk condition includes: outputting a prompt message for prompting network instability;

[0383] The vehicle-side response operation corresponding to the second risk condition includes: sending a prompt message to the remote control device to indicate that the network is unstable.

[0384] The present disclosure also provides an electronic device, such as Fig.15 As shown, it includes a processor 1501, a communication interface 1502, a memory 1503 and a communication bus 1504, wherein the processor 1501, the communication interface 1502, and the memory 1503 communicate with each other through the communication bus 1504.

[0385] Memory 1503, used for storing computer programs;

[0386] The processor 1501 is used to implement the above-mentioned driving auxiliary line generation method when executing the program stored in the memory 1503.

[0387] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0388] The communication interface is used for communication between the above electronic device and other devices.

[0389] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0390] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0391] In another embodiment provided by the present disclosure, a computer-readable storage medium is further provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the above-mentioned driving assistance line generation methods is implemented.

[0392] In another embodiment provided by the present disclosure, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the driving assistance line generation methods in the above embodiments.

[0393] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0394] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0395] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0396] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the protection scope of the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure are included in the protection scope of the present disclosure.

Claims

1. A method for generating a driving auxiliary line, characterized in that: The method is applied to a remote control device, and the method comprises: Receiving a target image captured by an image acquisition device on the vehicle during the vehicle's driving; Calculating the transmission delay generated when receiving the target image; Calculating a first change speed of the vehicle according to the transmission delay; wherein the first change speed is a speed at which the vehicle changes within a period of the transmission delay; Predicting a wheel trajectory of the vehicle based on the first change speed and driving information of the vehicle; The wheel track line is drawn in the target image to obtain a driving auxiliary line.

2. The method according to claim 1, characterized in that The predicting of the wheel trajectory of the vehicle based on the first change speed and the driving information of the vehicle includes: Predicting a plurality of driving trajectory points of the vehicle according to the first change speed and driving information of the vehicle; A wheel trajectory line of the vehicle is determined according to a plurality of driving trajectory points.

3. The method according to claim 2, characterized in that The driving information includes at least one of the following: the speed and acceleration of the vehicle, and the steering angle and steering angle coefficient of the steering wheel of the vehicle.

4. The method according to claim 3, characterized in that The step of predicting a plurality of driving trajectory points of the vehicle according to the first change speed and the driving information of the vehicle includes: For each future moment, based on the speed, acceleration and first change speed of the vehicle, predict the distance traveled by the vehicle in a time period from the corresponding reference moment to the future moment as the travel distance corresponding to the future moment; the reference moment corresponding to each future moment is the moment before the future moment; According to the driving distance corresponding to each future moment, as well as the steering angle and the steering angle coefficient, the driving trajectory points of the vehicle at each future moment are predicted to obtain a plurality of driving trajectory points of the vehicle.

5. The method according to claim 4, characterized in that The predicting, for each future moment, based on the speed, acceleration and first change speed of the vehicle, a distance traveled by the vehicle in a time period from a corresponding reference moment to the future moment as the travel distance corresponding to the future moment includes: For each future moment, a second change speed corresponding to the future moment is calculated based on the acceleration, the time difference between the future moment and the corresponding reference moment, and the target constant; wherein the target constant is related to the sorting sequence number of the future moment in the moment sequence, and the second change speed is the speed of the vehicle during the period from the current moment to the future moment; For each future moment, the driving distance corresponding to the future moment is determined based on the speed, the first change speed, the second change speed corresponding to the future moment, and the time difference between the future moment and the corresponding reference moment.

6. The method according to claim 4, characterized in that The method predicts the driving trajectory points of the vehicle at each future moment according to the driving distance corresponding to each future moment, as well as the steering angle and the steering angle coefficient, to obtain multiple driving trajectory points of the vehicle, including: Determining a wheel rotation angle of the vehicle based on the steering angle and the steering angle coefficient; For each future moment, the coordinates of the driving trajectory point of the vehicle at the future moment are calculated according to the driving distance corresponding to the future moment, the wheel rotation angle, and the coordinates of the driving trajectory point at the reference moment; Based on the obtained coordinates of the driving trajectory points of the vehicle, the driving trajectory points of the vehicle at each future moment are determined to obtain a plurality of driving trajectory points of the vehicle.

7. The method according to claim 6, characterized in that The step of calculating the coordinates of the driving trajectory point of the vehicle at each future moment according to the driving distance corresponding to the future moment, the wheel rotation angle, and the coordinates of the driving trajectory point at the reference moment includes: For each future moment, the lateral travel distance corresponding to the future moment is calculated according to the travel distance corresponding to the future moment and the wheel rotation angle, and the abscissa of the travel trajectory point of the vehicle at the future moment is calculated based on the lateral travel distance and the abscissa of the travel trajectory point at the reference moment; For each future moment, the longitudinal driving distance corresponding to the future moment is calculated based on the driving distance corresponding to the future moment and the wheel rotation angle, and the longitudinal coordinate of the driving trajectory point of the vehicle at the future moment is calculated based on the longitudinal driving distance and the longitudinal coordinate of the driving trajectory point at the reference moment.

8. The method according to claim 2, characterized in that: The multiple driving trajectory points are trajectory points of the vehicle in a vehicle coordinate system; wherein the vehicle coordinate system takes the reference point of the vehicle as the origin; The step of determining the wheel track line of the vehicle according to the plurality of driving track points comprises: Determining a driving trajectory line formed by the plurality of driving trajectory points; According to the lateral offset information of the left wheel of the vehicle relative to the reference point, the driving trajectory line is offset to obtain a first trajectory line of the left wheel; The driving trajectory is offset according to the lateral offset information of the right wheel of the vehicle relative to a reference point to obtain a second trajectory of the right wheel.

9. The method according to claim 1, characterized in that: The step of drawing the wheel track line in the target image to obtain a driving auxiliary line includes: The wheel track line is drawn in the target image according to the position parameters of the image acquisition device relative to the vehicle to obtain a driving auxiliary line.

10. The method according to claim 9, characterized in that The step of drawing the wheel track line in the target image according to the position parameter of the image acquisition device relative to the vehicle to obtain a driving auxiliary line comprises: Based on the installation height and installation pitch angle of the image acquisition device, converting the wheel trajectory line into a third trajectory line in the camera coordinate system of the image acquisition device; Converting the third trajectory line into a fourth trajectory line in the image coordinate system of the image acquisition device; The fourth trajectory line is drawn in the target image to obtain a driving auxiliary line.

11. The method according to claim 1, characterized in that The method further comprises: Whether to issue a collision warning is determined based on the positional relationship between the driving assistance line and the edge of the road, as well as the positional relationship between the driving assistance line and other vehicles.

12. The method according to claim 1, characterized in that The method further comprises: In response to receiving a first heartbeat message periodically sent by the vehicle, detecting whether the first heartbeat message meets a risk condition set for remote driving; In response to the first heartbeat information meeting the risk condition, a remote-end coping operation corresponding to the risk condition is performed.

13. The method according to claim 12, characterized in that The method further comprises: A second heartbeat message is periodically sent to the vehicle so that the vehicle, in response to receiving the second heartbeat message, detects whether the second heartbeat information meets the risk conditions set for remote driving, and if so, executes a vehicle-side response operation corresponding to the risk condition.

14. The method according to claim 12 or 13, characterized in that Any heartbeat message includes: a timestamp and a message count value; The risk condition includes one or more of a first risk condition, a second risk condition, and a third risk condition; The first risk condition includes: a difference between a message count value in a currently received heartbeat message and a message count value in a previously received heartbeat message is greater than a first threshold; The second risk condition includes: for a plurality of consecutive heartbeat messages received, the difference between the message count values ​​in each two adjacent heartbeat messages is greater than a second threshold; wherein the number of the plurality of consecutive heartbeat messages is at least three; The third risk condition includes: a difference between a timestamp in a currently received heartbeat message and a timestamp in a previously received heartbeat message is greater than a third threshold.

15. The method according to claim 14, characterized in that The remote-side response operation corresponding to the first risk condition and the remote-side response operation corresponding to the third risk condition include: outputting a prompt message indicating that the vehicle is out of control; the vehicle-side response operation corresponding to the first risk condition and the vehicle-side response operation corresponding to the third risk condition include: triggering a parking operation; The remote end response operation corresponding to the second risk condition includes: outputting a prompt message for prompting network instability; The vehicle-side response operation corresponding to the second risk condition includes: sending a prompt message to the remote control device to indicate that the network is unstable.

16. A driving auxiliary line generation system, characterized in that: The system comprises: a remote control device, a vehicle control device; The vehicle control device is used to obtain the target image captured by the image acquisition device and send the target image to the remote control device; The remote control device is used to receive the target image captured by the image acquisition device during vehicle driving, calculate the transmission delay generated when the target image is received, calculate the first change speed according to the transmission delay and the driving information of the vehicle, predict the wheel trajectory line of the vehicle based on the first change speed and the driving information, draw the wheel trajectory line in the target image, and obtain a driving auxiliary line.

17. The system according to claim 16, characterized in that The remote control device is a control device of a remote cockpit, and the vehicle control device sends the image to the remote control device, including: The vehicle control device sends the image to a cloud server, so that the cloud server sends the image to the control device of the remote cockpit.

18. A driving auxiliary line generating device, characterized in that: The device is applied to a remote control device, and comprises: A receiving module, used for receiving a target image captured by an image acquisition device on the vehicle during the vehicle's driving process; A first calculation module, used to calculate the transmission delay generated when the target image is received; A second calculation module is used to calculate a first change speed according to the transmission delay and the driving information of the vehicle; wherein the first change speed is a speed at which the vehicle changes within a period of the transmission delay; A prediction module, configured to predict a wheel trajectory of the vehicle based on the first change speed and the driving information; A drawing module is used to draw the wheel track line in the target image to obtain a driving auxiliary line.

19. An electronic device, characterized in that: include: one or more processors, and A memory storing a program, the program comprising instructions which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 15.

20. A computer-readable storage medium storing a program, characterized in that: The program comprises instructions which, when executed by one or more processors of a computing device, cause the computing device to perform a method according to any one of claims 1-15.

Images