Early warning method and device for remote control driving, electronic equipment and storage medium

Abstract

The invention discloses an early warning method and device for remote control driving, electronic equipment and a storage medium, and the method comprises the steps: determining tactile feedback information based on a risk condition that an obstacle in front of a first vehicle collides with the first vehicle; wherein the tactile feedback information is used for indicating the tactile feedback equipment to perform tactile feedback operation; performing layered rendering on the first video data based on the tactile feedback information to obtain second video data; the first video data is shot video data of the forward driving condition of the first vehicle; and displaying the second video data, and controlling a tactile feedback device to execute tactile feedback operation based on the tactile feedback information. According to the scheme, double prompts can be provided for the user in the aspects of touch and vision, so that the user can pay attention to the existing risk, and the effect and purpose of early warning in the remote control driving process are further achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of remote control driving, and in particular to a remote control driving early warning method, device, electronic equipment and storage medium. Background Art

[0002] In the field of remote control technology, camera video streams are typically fed back to the driver, allowing them to monitor road conditions and control the vehicle remotely. However, in certain scenarios, such as dangerous outdoor scenes, muddy roads, at night, or in low-light conditions, detailed images in the video stream are often lost. This makes it difficult for the driver to accurately judge the depth of potholes, identify the risk of the vehicle skidding, and provide timely visual warnings. This can lead to delayed driver control and potentially cause the vehicle to lose control. Summary of the Invention

[0003] The present invention provides a remote control driving warning method, device, electronic device and storage medium, which can provide dual prompts to users in terms of touch and vision, so that users can pay attention to existing risks, thereby achieving the effect and purpose of warning during remote control driving.

[0004] In a first aspect, the present invention provides a remote control driving warning method, comprising:

[0005] determining tactile feedback information based on a risk of collision between an obstacle in front of the first vehicle and the first vehicle; the tactile feedback information being used to instruct a tactile feedback device to perform a tactile feedback operation, the tactile feedback device being associated with the first vehicle, and the tactile feedback device being used to remotely control the first vehicle;

[0006] Performing layered rendering on the first video data based on the tactile feedback information to obtain second video data; the first video data is video data of the first vehicle moving forward;

[0007] The second video data is displayed, and a tactile feedback device is controlled to perform a tactile feedback operation based on the tactile feedback information, so as to provide an early warning during the remote control driving process.

[0008] In a second aspect, the present invention further provides a remote control driving warning device, comprising:

[0009] a tactile feedback information determination module, configured to determine tactile feedback information based on a risk of collision between an obstacle in front of the first vehicle and the first vehicle; the tactile feedback information being used to instruct a tactile feedback device to perform a tactile feedback operation, the tactile feedback device being associated with the first vehicle, and the tactile feedback device being used to remotely control the first vehicle;

[0010] a second video data determination module, configured to perform layered rendering on the first video data based on the tactile feedback information to obtain second video data; the first video data being video data of the first vehicle moving forward;

[0011] The warning execution module is used to display the second video data and control the tactile feedback device to perform a tactile feedback operation based on the tactile feedback information to issue a warning during the remote control driving process.

[0012] In a third aspect, an embodiment of the present invention further provides an electronic device, including:

[0013] one or more processors;

[0014] a storage device for storing one or more programs,

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the remote control driving warning method provided in any embodiment of the present invention.

[0016] In a fourth aspect, an embodiment of the present invention further provides a storage medium comprising computer-executable instructions, which, when executed by a computer processor, are used to execute a remote control driving warning method as provided in any embodiment of the present invention.

[0017] The technical solution of the embodiment of the present invention determines tactile feedback information based on the risk of collision between the obstacle in front of the first vehicle and the first vehicle; wherein the tactile feedback information is used to instruct the tactile feedback device to perform a tactile feedback operation; based on the tactile feedback information, the first video data is layered rendered to obtain the second video data; the first video data is the video data of the moving situation in front of the first vehicle captured; the second video data is displayed, and the tactile feedback device is controlled to perform the tactile feedback operation based on the tactile feedback information. Through this solution, the user can not only clearly feel the tactile situation brought by the tactile feedback device, but also see the rendered and highlighted video data, thereby achieving dual prompts to the user in terms of touch and vision, so that the user can pay attention to the existing risks, or the user can become more focused, thus achieving the effect and purpose of early warning during remote control driving.

[0018] The above content of the invention is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other features, advantages, and aspects of the various embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

[0020] Figure 1 A flowchart of a remote control driving warning method provided by an embodiment of the present invention;

[0021] Figure 2 A schematic flow chart of another remote control driving warning method provided by an embodiment of the present invention;

[0022] Figure 3 A schematic structural diagram of a remote control driving warning device provided by an embodiment of the present invention;

[0023] Figure 4 A schematic structural diagram of an electronic device for implementing a warning method for remote control driving provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0025] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0026] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0027] It should be noted that the concepts of "first" and "second" mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0028] It should be noted that the modifications of "one" and "multiple" mentioned in the present invention are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0029] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0030] Figure 1 This is a flow chart of a remote control driving warning method provided by an embodiment of the present invention. The embodiment of the present invention is applicable to situations where warning prompts are provided during remote control driving of a vehicle. The method can be executed by a remote control driving warning device, which can be implemented in the form of software and / or hardware and is generally integrated into any electronic device with network communication capabilities, such as a mobile terminal, PC, or server. Figure 1 As shown, the remote control driving warning method according to the embodiment of the present invention may include the following process:

[0031] S110. Determine tactile feedback information based on a risk of collision between an obstacle in front of the first vehicle and the first vehicle; the tactile feedback information is used to instruct a tactile feedback device to perform a tactile feedback operation, the tactile feedback device is associated with the first vehicle, and the tactile feedback device is used to remotely control the first vehicle.

[0032] The first vehicle is a vehicle that can be driven based on remote control commands. It is understood that when remotely controlling a vehicle, the user is not directly on the vehicle. Instead, the user can receive video footage captured by the vehicle as it moves in real time through a relevant terminal device. Furthermore, based on the video footage, the user can remotely send remote control commands to the vehicle, thereby driving the vehicle to perform the relevant driving operations. In other words, the first vehicle can be a real vehicle to be remotely controlled in a real-world scenario.

[0033] The obstacle in front of the first vehicle can be any object that can hinder the normal travel of the first vehicle, such as pedestrians, road construction fences, warning signs, median strips, etc. When an obstacle appears in front of the first vehicle, the first vehicle needs to be controlled accordingly to avoid the obstacle. If the relevant control is not applied, the first vehicle may collide with the obstacle, causing damage to the vehicle.

[0034] The tactile feedback information is used to instruct the tactile feedback device to perform a tactile feedback operation. Tactile feedback operation refers to providing a physical perception or touch to the user through technical means such as force and vibration. For example, the tactile feedback operation may be vibration feedback. The tactile feedback device is associated with the first vehicle, and the tactile feedback device can be used to remotely control the first vehicle. The tactile feedback device can send a remote control instruction to the first vehicle, thereby driving the first vehicle. For example, the tactile feedback device can be a driving simulator, a simulated steering wheel, a simulated seat, etc. Of course, the tactile feedback device can be differentiated based on actual needs, which is not further defined here.

[0035] Specifically, for an obstacle appearing in front of the first vehicle, the risk of collision between the obstacle and the first vehicle can be determined. For example, the likelihood of a collision risk can be determined based on the distance between the obstacle and the first vehicle and the first vehicle's current speed. Tactile feedback information can then be determined based on the collision risk, and the specific tactile feedback operation to be applied can be determined based on the tactile feedback information. It is easy to understand that different levels and types of tactile feedback operations can be performed to provide a prompt for the risk situation.

[0036] S120 . Perform layered rendering on the first video data based on the tactile feedback information to obtain second video data; the first video data is video data of the first vehicle moving forward, which is captured.

[0037] The first video data may be captured by an onboard camera device installed on the first vehicle. For example, the first video data may be captured by a driving recorder on the first vehicle, capturing the scene in front of the vehicle. The first video data records various detailed information about the scene in front of the first vehicle, including but not limited to objects, obstacles, and the locations of obstacles appearing in the scene.

[0038] Specifically, based on the tactile feedback information, it is equivalent to determining that tactile feedback is needed, and the tactile feedback performed is inevitably associated with the corresponding obstacle. At the same time, the first video data records the situation in front of the first vehicle. That is to say, these obstacles that may cause collision risks are still located in the picture of the first video data, and the solution of this embodiment is precisely aimed at this. The first video data is layered and rendered based on the tactile feedback information to obtain the second video data. In other words, the second video data is equivalent to the rendering of the first video data. Compared with the first video data, the second video data has a more prominent differentiated display effect. For example, it can highlight the obstacles in the picture.

[0039] S130: Display the second video data, and control the tactile feedback device to perform a tactile feedback operation based on the tactile feedback information, so as to provide an early warning during the remote control driving process.

[0040] Specifically, after determining the tactile feedback information and the second video data, the second video data can be displayed on the user's terminal device, and the tactile feedback device can be controlled to perform tactile feedback operations based on the tactile feedback information. This allows the user to not only see the rendered and highlighted video data, but also clearly feel the tactile conditions brought by the tactile feedback device, achieving dual tactile and visual prompts to the user, thereby allowing the user to pay attention to the existing risks or become more focused, thus achieving the effect and purpose of early warning during remote control driving.

[0041] It should be noted that the terminal device used to display the second video data and the tactile feedback device can be an integrated device or independent devices, which is not further specified here. However, both devices are located at the user's location, and the user can achieve remote control of the vehicle only by coordinating these two devices.

[0042] The technical solution of the embodiment of the present invention determines tactile feedback information based on the risk of collision between the obstacle in front of the first vehicle and the first vehicle; wherein the tactile feedback information is used to instruct the tactile feedback device to perform a tactile feedback operation; based on the tactile feedback information, the first video data is layered rendered to obtain the second video data; the first video data is the video data of the moving situation in front of the first vehicle captured; the second video data is displayed, and the tactile feedback device is controlled to perform the tactile feedback operation based on the tactile feedback information. Through this solution, the user can not only clearly feel the tactile situation brought by the tactile feedback device, but also see the rendered and highlighted video data, thereby achieving dual prompts to the user in terms of touch and vision, so that the user can pay attention to the existing risks, or the user can become more focused, thus achieving the effect and purpose of early warning during remote control driving.

[0043] Figure 2 This is a flow chart of another remote control driving warning method provided by an embodiment of the present invention. The technical solution of this embodiment further optimizes the process of layered rendering of the first video data based on tactile feedback information in the above embodiment to obtain the second video data. This embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 2 As shown, the remote control driving warning method according to the embodiment of the present invention may include the following process:

[0044] S210. Determine tactile feedback information based on a risk of collision between an obstacle in front of the first vehicle and the first vehicle; the tactile feedback information is used to instruct a tactile feedback device to perform a tactile feedback operation, the tactile feedback device is associated with the first vehicle, and the tactile feedback device is used to remotely control the first vehicle.

[0045] As an optional but non-limiting implementation method, tactile feedback information is determined based on the risk of a collision between an obstacle in front of the first vehicle and the first vehicle, including: determining risk level information based on the risk of a collision between the obstacle in front of the first vehicle and the first vehicle, and judging whether the risk level information exceeds a risk threshold; the risk level information is used to reflect the possibility of a collision between the obstacle and the first vehicle; when the risk level information exceeds the risk threshold, tactile feedback information is determined according to a first mapping rule; the first mapping rule is used to indicate the type of tactile feedback information corresponding to different risk levels and the detailed content of the tactile feedback information; when the risk level information does not exceed the risk threshold, tactile feedback information is determined based on a second mapping rule and the risk level information; the second mapping rule is used to indicate the type of tactile feedback information triggered by each risk factor, and the risk factor is used to describe the dangerous factors existing in the risk level information, and the risk factor is determined based on the risk level information. By adopting this optional solution, different degrees of tactile feedback information can be set for different levels of collision risk situations. After perceiving different degrees of tactile feedback information, the user can clearly understand whether the current risk situation is urgent and what specific detailed factors cause the risk, which makes it easier for the user to quickly identify the specific circumstances of the dangerous information.

[0046] The risk level information reflects the likelihood of a collision between the obstacle and the first vehicle. For example, the risk level information may include low risk, medium risk, and high risk. Of course, the risk level information can also be represented by a numerical value, and the specific setting can be differentiated based on actual needs. It is understood that the greater the likelihood of a collision between the obstacle and the first vehicle, the greater the risk level, and the more dangerous it is; the less likely the obstacle is to collide with the first vehicle, the lower the risk level, and the safer it is. There is a positive correlation between the two.

[0047] The risk threshold can be understood as a risk critical value determined based on actual needs. With the help of this risk threshold, various risk levels can be roughly classified to determine the extent to which the corresponding risk level should be handled.

[0048] Specifically, based on the risk of the obstacle in front of the first vehicle colliding with the first vehicle, corresponding risk level information can be determined, and it can be determined whether the risk level information exceeds the risk threshold. In the case that the risk level information exceeds the risk threshold, it indicates that the current risk situation is more severe, and the tactile feedback information can be determined according to the first mapping rule; wherein the first mapping rule is used to indicate the type of tactile feedback information corresponding to different risk levels and the detailed content of the tactile feedback information. The type of tactile feedback information refers to the type of tactile feedback information, and the types of tactile feedback information include: the vibration frequency of the simulated steering wheel, the vibration duration, and the pulse intensity of the simulated seat. The detailed content of the tactile feedback information is the detailed situation corresponding to a specific type of tactile feedback information. In other words, the first mapping rule can directly indicate the detailed tactile feedback information corresponding to each different risk level. For example, when it is determined that the risk level information is "high risk" and the "high risk" has exceeded the risk threshold, the tactile feedback information corresponding to the "high risk" can be directly searched in the first mapping rule. For example, the corresponding information may be: the simulated steering wheel is subjected to the highest intensity tactile vibration, the vibration duration exceeds 500 milliseconds, and a warning box is added to the displayed video and the full-screen flashes, etc.

[0049] If the risk level information does not exceed the risk threshold, indicating that the current risk situation is not particularly urgent or severe, tactile feedback information can be determined based on a second mapping rule and the risk level information. The second mapping rule indicates the type of tactile feedback information triggered by each risk factor, where the risk factor describes the risk factor present in the risk level information. It is easy to understand that when determining the risk level, the risk level must correspond to certain key parameters. Therefore, these key parameters can be used to determine the risk level. These key parameters can then serve as risk factors, which essentially describe the risk factors present in the risk level information. For example, the risk level information can be determined based on the obstacle's approach speed and the terrain slope. Therefore, both the obstacle's approach speed and the terrain slope can serve as risk factors. The second mapping rule records the type of tactile feedback information triggered by each risk factor. For example, in the second mapping rule, the obstacle's approach speed corresponds to the vibration frequency of the simulated steering wheel, and the terrain slope corresponds to the pulse intensity of the simulated seat. Of course, the second mapping rule can also include more such mapping relationships, which can be configured based on actual needs.

[0050] It should be noted that these mapping relationships do not have just one piece of mapping data, but rather multiple pieces of mapping data. For example, the mapping relationship between "obstacle approach speed" and simulated steering wheel vibration frequency may have multiple mapping data. Because different obstacle approach speeds will result in different mapped simulated steering wheel vibration frequencies, these multiple mapping data are essentially all related to the same mapping relationship. For example, if the obstacle approach speed is 10m / s, the corresponding simulated steering wheel vibration frequency is 10Hz; if the obstacle approach speed is 15m / s, the corresponding simulated steering wheel vibration frequency is 15Hz, and so on.

[0051] Furthermore, based on the risk level information and the second mapping rule, detailed tactile feedback information can be determined. In short, the second mapping rule essentially determines the type of tactile feedback information triggered by each risk factor. After perceiving the corresponding tactile feedback information, the user can identify which risk factor is problematic and requires attention. For example, if the user senses a violent vibration in the simulated steering wheel, the user will know that the obstacle is approaching at a high speed and is bound to approach soon. This configuration not only provides users with tactile feedback to prompt them, but also facilitates their quick identification of the specific danger information.

[0052] As an optional but non-limiting implementation, risk level information is determined by obtaining three-dimensional point cloud data, video image data, and vehicle posture data of the first vehicle; analyzing and extracting the three-dimensional point cloud data, video image data, and vehicle posture data to obtain multiple physical characteristic parameters; the multiple physical characteristic parameters are used to reflect the physical environment encountered by the first vehicle during its travel, as well as the physical characteristics of the first vehicle itself; and determining the risk level information based on the multiple physical characteristic parameters. This optional solution, by obtaining more comprehensive information about the first vehicle's travel conditions, can determine more accurate risk level information.

[0053] The three-dimensional point cloud data can be obtained by measuring the environment around the vehicle using a laser radar installed on the first vehicle. The video image data can be obtained by capturing images using a camera (e.g., a driving recorder) installed on the first vehicle. The vehicle posture data can be obtained by measuring using an IMU sensor installed on the first vehicle.

[0054] Optionally, before analyzing and extracting the three-dimensional point cloud data, video image data and vehicle posture data, the method further includes: performing time axis alignment and coordinate system conversion on the three-dimensional point cloud data, video image data and vehicle posture data.

[0055] Specifically, since the three types of data, namely three-dimensional point cloud data, video image data and vehicle posture data, are measured by means of different sensors or devices, and the sensors are not connected to each other, there may be inconsistencies and inconsistencies in the time and space positions between the various types of data. Therefore, it is necessary to align the time axis and convert the coordinate system of these three types of data to ensure that they are aligned consistently in time and space. Specifically, the three-dimensional point cloud data, video image data and vehicle posture data can be added with corresponding timestamp information respectively, and these asynchronous data can be aligned to a unified time axis by using downsampling or interpolation methods, thereby achieving the alignment and unification of the three types of data in time. Based on the same principle, the three types of data can also be converted from their respective local coordinate systems to the global coordinate system of the first vehicle to perform coordinate system conversion to achieve alignment and unification in space.

[0056] Furthermore, after the three types of data have been processed and aligned in time and space as described above, they can be analyzed and extracted to obtain multiple physical characteristic parameters. These multiple physical characteristic parameters are used to reflect the physical environment characteristics encountered by the first vehicle during its travel, as well as the physical characteristics of the first vehicle itself. Physical environment characteristics include obstacle distance, terrain roughness, terrain slope, obstacle approach speed, obstacle size, road width, etc. The physical characteristics of the first vehicle itself include vehicle speed, vehicle speed change rate, vehicle tilt angle, and vehicle posture information. It should be noted that the physical environment characteristics encountered by the first vehicle during its travel and the physical characteristics of the first vehicle itself can be differentiated based on actual needs and are not further specified here. After obtaining the multiple physical characteristic parameters, analysis and judgment can be performed based on the multiple physical characteristic parameters to determine the current risk level of the obstacle to the vehicle. For example, the multiple physical characteristic parameters obtained can be input into a pre-trained model, which, after processing, outputs the corresponding risk level information.

[0057] As an optional but non-limiting implementation, the risk threshold is determined by: determining the user's proficiency in remote-driving the first vehicle; determining a critical condition for triggering a risk based on the proficiency information; and determining the corresponding risk threshold based on the critical condition. This optional solution allows for different risk thresholds to be set based on the user's different proficiency in remote-driving, thereby better tailoring the user's specific circumstances and enabling appropriate warnings at the right time.

[0058] The proficiency information can be used to reflect the user's proficiency in operating the remote control. The proficiency information can include beginner, intermediate, advanced, expert, etc., and of course, more detailed divisions and settings can be made based on actual conditions. This is not further specified here.

[0059] Optionally, determining the user's proficiency information for remotely controlling the first vehicle includes: determining the proficiency information based on identification information of the user's login account; or determining the proficiency information based on remote control driving operation test results. The operation test results include: operation smoothness, steering range, braking lead, and emergency operation frequency.

[0060] Specifically, during the user's remote driving process, proficiency information can be determined based on the identification information of the user's login account. For example, if the identification information of the user's login account is "novice," then the user's proficiency information for remote driving of the first vehicle can be determined to be "beginner." Alternatively, the user's operation during remote driving can be tested, and the proficiency information can be determined based on the remote driving operation test results. For example, if the remote driving operation test results determine that the user is proficient in operation, then the user's proficiency information for remote driving of the first vehicle can be determined to be "expert."

[0061] Furthermore, the critical conditions for triggering risks can be determined based on the proficiency information, and the corresponding risk thresholds can be determined based on the critical conditions. For example, if the proficiency level is "beginner," the corresponding critical condition for triggering risks could be: a stationary obstacle is less than 10 meters from the first vehicle, triggering a risk. The corresponding risk thresholds based on the proficiency information could then be the specific parameters corresponding to "low risk." As long as the actual detected risk exceeds the specific parameters corresponding to "low risk," a corresponding warning alert could be triggered. For another example, when the proficiency is "expert", the critical condition for triggering the corresponding risk can be: the risk is triggered when the moving obstacle is less than 5 meters away from the first vehicle; then the corresponding risk threshold under the proficiency information can be the specific parameter corresponding to "high risk", so as long as the actual detected risk situation exceeds the specific parameter corresponding to "high risk", the corresponding early warning prompt can be triggered, etc. If it does not exceed the specific parameter corresponding to "high risk", then the early warning prompt will not be triggered. In other words, for users at the "expert" level, the early warning prompt will only be triggered when the risk is high, and no early warning will be issued at other risk levels to avoid disturbing the user's attention. This implementation plan sets different risk thresholds according to the different proficiency of users in remote control driving, so that it can better fit the user's actual situation and can provide reasonable and appropriate early warning prompts at the right time.

[0062] S220. Determine first rendering layer data and second rendering layer data based on the tactile feedback information; the first rendering layer data is used to render a tactile association layer, which is a layer used to highlight at least part of the screen content; and the second rendering layer data is used to render an emergency warning layer, which is a layer used to convey emergency information.

[0063] Specifically, based on the tactile feedback information, it is possible to determine what the current risk situation is and what tactile feedback operation is required. This is equivalent to further determining what specific rendering needs to be applied to the first video data, for example, what kind of rendering needs to be performed at which position in the first video data, etc., that is, based on the tactile feedback information, the first rendering layer data and the second rendering layer data can be determined. The first rendering layer data is used to render the tactile association layer, which is a layer used to highlight at least part of the screen content. Highlighting at least part of the screen content can be displaying a vibration icon in the screen, or adding a pseudo-color image to the screen, etc., so as to achieve a visual effect of highlighting at least part of the screen content. The second rendering layer data is used to render the emergency warning layer, which is a layer used to convey emergency information. That is, the content displayed in the emergency warning layer can be used to convey emergency information. For example, the emergency warning layer can flash full screen or add a 3D warning box to the screen to achieve the effect of an emergency prompt.

[0064] S230: Use the first video data as basic visual layer data, and perform layered rendering on the first video data based on the first rendering layer data and the second rendering layer data to obtain second video data.

[0065] Specifically, after determining the first rendering layer data and the second rendering layer data, the first video data can be used as the basic visual layer data, the tactile association layer can be rendered based on the first rendering layer data, and the emergency warning layer can be rendered based on the second rendering layer data. After the rendering is completed, the relevant rendering results can be superimposed on the first video data to obtain the second video data.

[0066] It should be noted that, during the process of determining the first and second rendering layer data based on tactile feedback information, if only one of the rendering layer data is determined, then only the corresponding layer needs to be rendered, and the other layer does not need to be rendered. For example, if only the first rendering layer data is determined based on tactile feedback information, and no second rendering layer data is available, then only the tactile-related layer corresponding to the first rendering layer data needs to be rendered, and the emergency warning layer does not need to be rendered.

[0067] As an optional but non-limiting implementation, before determining the first and second rendering layer data based on the tactile feedback information, the method further includes: determining first trigger information based on the risk of an obstacle in front of the first vehicle; the first trigger information includes a trigger time, an obstacle mapping location, and a risk level; and generating a synchronization trigger instruction based on the first trigger information; the synchronization trigger instruction is used to instruct the tactile feedback device to perform a tactile feedback operation synchronously with the display of the second video data. With this optional solution, tactile feedback can be performed on the tactile feedback device in response to a risk situation, synchronously with the display of the second video data.

[0068] Specifically, first trigger information can be determined based on the risk of an obstacle in front of the first vehicle. The first trigger information includes the trigger time, obstacle mapping location, and risk level. In other words, the first trigger information contains key information that requires an early warning, and a synchronous trigger instruction can be generated based on the first trigger information. The synchronous trigger instruction is used to instruct the tactile feedback device to perform a tactile feedback operation synchronously with the display of the second video data. In other words, based on the synchronous trigger instruction, not only can the tactile feedback device be controlled to perform the tactile feedback operation, but the second video data can also be controlled to be displayed, and the two are performed synchronously.

[0069] As an optional but non-limiting implementation, after obtaining the second video data, the method further includes: determining whether the time at which the rendered content in the second video data appears is consistent with the triggering time of the tactile feedback operation to perform a temporal consistency check; determining whether the position at which the rendered content in the second video data appears in the image is consistent with the obstacle mapping position to perform a spatial consistency check; and displaying the second video data if both the spatial consistency check and the temporal consistency check pass. This optional solution can determine whether the tactile feedback operation performed by the tactile feedback device corresponds to the displayed content of the second video data, thereby ensuring consistency between the tactile and visual signals.

[0070] The triggering time of the tactile feedback operation is the time when the triggering operation starts to be implemented.

[0071] Specifically, after obtaining the second video data, it is possible to further determine whether the time at which the rendering content in the second video data appears is consistent with the triggering time of the tactile feedback operation to perform a time consistency check; if the two times are consistent, or the time difference is less than the preset time difference, it can be considered that the time consistency check has passed, otherwise it will fail. It is also possible to determine whether the position at which the rendering content in the second video data appears in the picture is consistent with the obstacle mapping position to perform a spatial consistency check to determine whether the position information of the obstacles in the two is consistent; for example, when the tactile feedback prompts that there are continuous potholes 3 meters to the right (specifically corresponding to the steering wheel vibrating at 200ms intervals for 3 consecutive times), a yellow warning box is synchronously marked for the 3 meters to the right in the visual picture, and a road roughness diagram is superimposed (such as blue for smooth and red for rough), which indicates a spatial consistency check, otherwise it will fail. If both the spatial consistency check and the temporal consistency check are passed, it means that the tactile feedback device performs a tactile feedback operation that corresponds to the display content of the second video data and there is no error, then the second video data can be displayed.

[0072] S240: Display the second video data, and control the tactile feedback device to perform a tactile feedback operation based on the tactile feedback information, so as to provide an early warning during the remote control driving process.

[0073] The technical solution of the embodiment of the present invention determines tactile feedback information based on the risk of collision between the obstacle in front of the first vehicle and the first vehicle; wherein the tactile feedback information is used to instruct the tactile feedback device to perform a tactile feedback operation; based on the tactile feedback information, the first video data is layered rendered to obtain the second video data; the first video data is the video data of the moving situation in front of the first vehicle captured; the second video data is displayed, and the tactile feedback device is controlled to perform the tactile feedback operation based on the tactile feedback information. Through this solution, the user can not only clearly feel the tactile situation brought by the tactile feedback device, but also see the rendered and highlighted video data, thereby achieving dual prompts to the user in terms of touch and vision, so that the user can pay attention to the existing risks, or the user can become more focused, thus achieving the effect and purpose of early warning during remote control driving.

[0074] Figure 3 This is a schematic diagram of the structure of a remote control driving warning device provided by an embodiment of the present invention. The embodiment of the present invention is applicable to the situation where a warning prompt is given during the remote control driving of a vehicle. The remote control driving warning device can be implemented in the form of software and / or hardware and is generally integrated into any electronic device with network communication function, such as a mobile terminal, PC or server. Figure 3As shown, the remote control driving warning device according to the embodiment of the present invention may include a tactile feedback information determination module 310, a second video data determination module 320, and a warning execution module 330. Among them:

[0075] a tactile feedback information determining module 310 for determining tactile feedback information based on a risk of collision between an obstacle in front of the first vehicle and the first vehicle; the tactile feedback information being used to instruct a tactile feedback device to perform a tactile feedback operation, the tactile feedback device being associated with the first vehicle, and the tactile feedback device being used to remotely control the first vehicle;

[0076] The second video data determination module 320 is configured to perform layered rendering on the first video data based on the tactile feedback information to obtain second video data; the first video data is video data of the first vehicle moving forward;

[0077] The warning execution module 330 is used to display the second video data and control the tactile feedback device to perform a tactile feedback operation based on the tactile feedback information to issue a warning during the remote control driving process.

[0078] The technical solution of the embodiment of the present invention is to determine tactile feedback information based on the risk of collision between the obstacle in front of the first vehicle and the first vehicle through a tactile feedback information determination module; wherein the tactile feedback information is used to instruct the tactile feedback device to perform a tactile feedback operation; the first video data is layered rendered based on the tactile feedback information by a second video data determination module to obtain second video data; the first video data is video data of the moving situation in front of the first vehicle; the second video data is displayed by an early warning execution module, and the tactile feedback device is controlled to perform a tactile feedback operation based on the tactile feedback information. Through this solution, the user can not only clearly feel the tactile situation brought by the tactile feedback device, but also see the rendered and highlighted video data, thereby achieving dual prompts to the user in terms of touch and vision, so that the user can pay attention to the existing risks, or the user can become more focused, thus achieving the effect and purpose of early warning during remote control driving.

[0079] As an optional but non-limiting implementation, the tactile feedback information determination module 310 includes a risk level determination unit, a first determination unit, and a second determination unit.

[0080] a risk level determination unit, configured to determine risk level information based on a risk of collision between an obstacle in front of the first vehicle and the first vehicle, and to determine whether the risk level information exceeds a risk threshold; the risk level information is configured to reflect a likelihood of collision between the obstacle and the first vehicle;

[0081] a first determining unit configured to determine the tactile feedback information according to a first mapping rule when the risk level information exceeds a risk threshold; the first mapping rule being configured to indicate a type of tactile feedback information corresponding to different risk levels and detailed content of the tactile feedback information;

[0082] The second determination unit is used to determine the tactile feedback information based on the second mapping rule and the risk level information when the risk level information does not exceed the risk threshold; the second mapping rule is used to indicate the type of tactile feedback information triggered by each risk factor, and the risk factor is used to describe the dangerous factors existing in the risk level information, and the risk factor is determined based on the risk level information.

[0083] As an optional but non-limiting implementation, the risk level determination unit includes a data acquisition subunit, a data analysis subunit, and a risk level determination subunit.

[0084] a data acquisition subunit, configured to acquire three-dimensional point cloud data, video image data, and vehicle posture data of the first vehicle;

[0085] a data analysis subunit, configured to analyze and extract the three-dimensional point cloud data, the video image data, and the vehicle posture data to obtain a plurality of physical characteristic parameters; the plurality of physical characteristic parameters are configured to reflect the physical environment characteristics encountered by the first vehicle during its travel and the physical characteristics of the first vehicle itself;

[0086] The risk level determination subunit is configured to determine the risk level information based on a plurality of the physical characteristic parameters.

[0087] As an optional but non-limiting implementation method, the risk threshold is determined in the following manner: determining the user's proficiency information in remotely controlling the first vehicle; based on the proficiency information, determining the critical conditions for risk triggering; and based on the critical conditions, determining the risk threshold corresponding to the proficiency information.

[0088] As an optional but non-limiting implementation, the second video data determination module 320 includes a rendering layer data determination unit and a second video data determination unit.

[0089] a rendering layer data determination unit, configured to determine first rendering layer data and second rendering layer data based on tactile feedback information; the first rendering layer data being used to render a tactile association layer, which is a layer used to highlight at least a portion of screen content; and the second rendering layer data being used to render an emergency warning layer, which is a layer used to convey emergency information.

[0090] The second video data determining unit is configured to use the first video data as basic visual layer data, and perform layered rendering on the first video data based on the first rendering layer data and the second rendering layer data to obtain the second video data.

[0091] As an optional but non-limiting implementation, the remote control driving warning device further includes a first trigger information determination module and a synchronous trigger instruction determination module.

[0092] A first trigger information determination module, configured to determine first trigger information based on a risk condition of an obstacle in front of the first vehicle; the first trigger information including a trigger time, an obstacle mapping location, and a risk level;

[0093] A synchronous trigger instruction determination module generates a synchronous trigger instruction based on the first trigger information; the synchronous trigger instruction is used to instruct the tactile feedback device to perform a tactile feedback operation, which is synchronized with the display of the second video data.

[0094] As an optional but non-limiting implementation, the second video data determination module 320 further includes a first check subunit, a second check subunit, and a video display subunit.

[0095] a first checking subunit, configured to determine whether a time when the rendered content in the second video data appears is consistent with a triggering time of the tactile feedback operation, so as to perform a time consistency check;

[0096] A second check subunit is used to determine whether the position where the rendered content in the second video data appears in the picture is consistent with the obstacle mapping position, so as to perform a spatial consistency check;

[0097] The video display subunit is used to display the second video data when both the spatial consistency check and the temporal consistency check are passed.

[0098] The remote control driving warning device provided in the embodiment of the present invention can be used to execute the remote control driving warning method, and has corresponding functional modules and beneficial effects for executing the remote control driving warning method.

[0099] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the embodiments of the present invention.

[0100] Figure 4This is a schematic diagram of the structure of an electronic device for implementing a warning method for remote control driving provided by an embodiment of the present invention. Figure 4 , which shows a schematic structural diagram of an electronic device 410 suitable for implementing an embodiment of the present invention. The terminal device in the embodiment of the present invention may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0101] like Figure 4 As shown, the electronic device 410 includes at least one processor 411, and a memory connected to the at least one processor 411 in communication, such as a read-only memory (ROM) 412, a random access memory (RAM) 413, etc., wherein the memory stores a computer program that can be executed by the at least one processor, and the processor 411 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 412 or the computer program loaded from the storage unit 518 into the random access memory (RAM) 413. Various programs and data required for the operation of the electronic device 410 can also be stored in the (RAM) 413. The processor 411, (ROM) 412 and (RAM) 413 are connected to each other via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0102] Multiple components in electronic device 410 are connected to I / O interface 415, including an input unit 416, such as a keyboard, mouse, etc.; an output unit 417, such as various types of displays, speakers, etc.; a storage unit 418, such as a magnetic disk, optical disk, etc.; and a communication unit 419, such as a network card, modem, wireless communication transceiver, etc. The communication unit 419 allows electronic device 410 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0103] Processor 411 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any other suitable processor, controller, microcontroller, etc. Processor 411 executes the remote control driving warning method provided in any embodiment of the present invention.

[0104] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the remote control driving warning method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication unit 419, or installed from the storage unit 418, or installed from the ROM 412. When the computer program is executed by the processor 411, the above-mentioned functions defined in the remote control driving warning method of the embodiment of the present invention are performed.

[0105] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0106] The electronic device provided in the embodiment of the present invention and the remote control driving warning method provided in the above embodiment belong to the same inventive concept. For technical details not described in detail in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0107] An embodiment of the present invention provides a computer storage medium having a computer program stored thereon. When the program is executed by a processor, the remote control driving warning method provided in the above embodiment is implemented.

[0108] It should be noted that the computer-readable medium described above in the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present invention, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0109] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0110] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0111] The flow charts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0112] The units involved in the embodiments of the present invention may be implemented in software or hardware, wherein the name of a unit does not necessarily limit the unit itself.

[0113] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0114] In the context of the present invention, machine-readable medium can be a tangible medium that can contain or store a program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0115] The above description is merely a preferred embodiment of the present invention and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present invention is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in the present invention.

[0116] In addition, although adopting specific order to describe each operation, this should not be interpreted as requiring these operations to be executed in the specific order shown or in sequential order.Under certain environment, multitasking and parallel processing may be advantageous.Similarly, although comprising some specific implementation details in the above discussion, these should not be interpreted as limiting the scope of the present invention.Some features described in the context of independent embodiment can also be implemented in single embodiment in combination.On the contrary, the various features described in the context of independent embodiment also can be implemented in multiple embodiments individually or in the mode of any suitable subcombination.

[0117] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A remote control driving warning method, characterized in that: The method comprises: determining tactile feedback information based on a risk of collision between an obstacle in front of the first vehicle and the first vehicle; the tactile feedback information being used to instruct a tactile feedback device to perform a tactile feedback operation, the tactile feedback device being associated with the first vehicle, and the tactile feedback device being used to remotely control the first vehicle; Performing layered rendering on the first video data based on the tactile feedback information to obtain second video data; the first video data is video data of the first vehicle moving forward; The second video data is displayed, and a tactile feedback device is controlled to perform a tactile feedback operation based on the tactile feedback information, so as to provide an early warning during the remote control driving process.

2. The method according to claim 1, characterized in that The determining of tactile feedback information based on a risk of collision between an obstacle in front of the first vehicle and the first vehicle includes: Determining risk level information based on a risk of collision between an obstacle in front of the first vehicle and the first vehicle, and determining whether the risk level information exceeds a risk threshold; the risk level information is used to reflect the likelihood of collision between the obstacle and the first vehicle; When the risk level information exceeds the risk threshold, the tactile feedback information is determined according to a first mapping rule; the first mapping rule is used to indicate the type of tactile feedback information corresponding to different risk levels and the detailed content of the tactile feedback information; When the risk level information does not exceed the risk threshold, the tactile feedback information is determined based on the second mapping rule and the risk level information; the second mapping rule is used to indicate the type of tactile feedback information triggered by each risk factor, and the risk factor is used to describe the dangerous factors existing in the risk level information, and the risk factor is determined based on the risk level information.

3. The method according to claim 2, characterized in that The risk level information is determined in the following manner: Acquiring three-dimensional point cloud data, video image data, and vehicle posture data of a first vehicle; Analyzing and extracting the three-dimensional point cloud data, the video image data, and the vehicle posture data to obtain a plurality of physical characteristic parameters; the plurality of physical characteristic parameters are used to reflect the physical environment characteristics encountered by the first vehicle during its movement and the physical characteristics of the first vehicle itself; The risk level information is determined based on the plurality of physical characteristic parameters.

4. The method according to claim 2, characterized in that The risk threshold is determined as follows: Determining user proficiency information for remotely controlling the first vehicle; Determining a critical condition for risk triggering based on the proficiency information; The risk threshold corresponding to the proficiency information is determined based on the critical condition.

5. The method according to claim 1, wherein The step of performing layered rendering on the first video data based on the tactile feedback information to obtain the second video data includes: Determining first rendering layer data and second rendering layer data based on the tactile feedback information; the first rendering layer data is used to render a tactile association layer, which is a layer used to highlight at least part of the screen content; and the second rendering layer data is used to render an emergency warning layer, which is a layer used to convey emergency information. The first video data is used as basic visual layer data, and the first video data is rendered in layers based on the first rendering layer data and the second rendering layer data to obtain the second video data.

6. The method according to claim 5, characterized in that Before determining the first rendering layer data and the second rendering layer data based on the tactile feedback information, the method further includes: Determining first trigger information based on a risk condition of an obstacle in front of the first vehicle; the first trigger information includes a trigger time, an obstacle mapping location, and a risk level; A synchronization trigger instruction is generated based on the first trigger information; the synchronization trigger instruction is used to instruct the tactile feedback device to perform a tactile feedback operation, which is synchronized with the display of the second video data.

7. The method according to claim 6, characterized in that After obtaining the second video data, the method further includes: Determining whether a time at which the rendered content in the second video data appears is consistent with a triggering time of the haptic feedback operation, so as to perform a time consistency check; Determining whether a position where the rendered content in the second video data appears in the picture is consistent with the obstacle mapping position to perform a spatial consistency check; When both the spatial consistency check and the temporal consistency check pass, the second video data is displayed.

8. A remote control driving warning device, characterized in that: The device comprises: a tactile feedback information determination module, configured to determine tactile feedback information based on a risk of collision between an obstacle in front of the first vehicle and the first vehicle; the tactile feedback information being used to instruct a tactile feedback device to perform a tactile feedback operation, the tactile feedback device being associated with the first vehicle, and the tactile feedback device being used to remotely control the first vehicle; a second video data determination module, configured to perform layered rendering on the first video data based on the tactile feedback information to obtain second video data; the first video data being video data of the first vehicle moving forward; The warning execution module is used to display the second video data and control the tactile feedback device to perform a tactile feedback operation based on the tactile feedback information to issue a warning during the remote control driving process.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the remote control driving warning method as described in any one of claims 1 to 7.

10. A storage medium containing computer-executable instructions, characterized in that: When the computer executable instructions are executed by a computer processor, they are used to execute the remote control driving warning method as described in any one of claims 1 to 7.

Images