Video generation method and device, electronic equipment and vehicle

Abstract

The invention relates to the technical field of vehicle control, and provides a video generation method and device, electronic equipment and a vehicle. The method comprises the following steps: acquiring target vehicle type information and a target driving scene of a target vehicle; and generating a target driving operation video of the target vehicle in the target driving scene by using a pre-trained target video generation model according to the target vehicle type information and the target driving scene. Therefore, the vehicle model information of different vehicles is input into the target video generation model, the driving operation videos of different vehicles can be generated, the target video generation model is reused, and the manufacturing cost of the driving operation videos is reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of vehicle control technology, and in particular to a video generation method, device, electronic equipment and vehicle. Background Art

[0002] With the development of the vehicle industry, driving operation videos can help users understand the operating details of the driving process. Since driving operations involve many scenarios and operating details, detailed explanations of driving operations through videos can help users understand driving services. However, shooting real cars or making driving operation videos through animation simulations will result in long video production cycles, high costs, and the inability to reuse driving operation videos of different models.

[0003] In view of this, how to avoid the long production cycle and non-reusability of driving operation videos has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] In view of this, the purpose of the present disclosure is to propose a video generation method, device, electronic device and vehicle to solve the problem in the prior art that the driving operation video production cycle is long and cannot be reused.

[0005] Based on the above purpose, the first aspect of the present disclosure proposes a video generation method, the method comprising:

[0006] Obtain target vehicle model information and target driving scenario of the target vehicle;

[0007] A target driving operation video of the target vehicle in the target driving scenario is generated by using a pre-trained target video generation model according to the target vehicle model information and the target driving scenario.

[0008] Based on the same inventive concept, the second aspect of the present disclosure proposes a video generating device, comprising:

[0009] An acquisition module is configured to acquire target vehicle model information and a target driving scene of a target vehicle;

[0010] The generation module is configured to use a pre-trained target video generation model to generate a target driving operation video of the target vehicle in the target driving scenario according to the target vehicle model information and the target driving scenario.

[0011] Based on the same inventive concept, the third aspect of the present disclosure proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method as described above when executing the computer program.

[0012] Based on the same inventive concept, a fourth aspect of the present disclosure proposes a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method as described above.

[0013] Based on the same inventive concept, the fifth aspect of the present disclosure proposes a vehicle, which includes the video generating device described in the second aspect or the electronic device described in the third aspect or the storage medium described in the fourth aspect.

[0014] As can be seen from the above, the video generation method, device, electronic device and vehicle provided by the present disclosure. Target vehicle model information and target driving scene of the target vehicle are obtained. Target driving operation video of the target vehicle in the target driving scene is generated using a pre-trained target video generation model according to the target vehicle model information and the target driving scene. In this way, the target driving operation video of the target vehicle can be generated using the target video generation model, and there is no need to shoot or animate the target vehicle to produce the target driving operation video, thereby reducing the production cycle of the target driving operation video. The vehicle model information of different vehicles is input into the target video generation model, and driving operation videos of different vehicles can be generated, so that the target video generation model can be reused, thereby reducing the production cost of the driving operation video. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 is a flowchart of a video generation method according to an embodiment of the present disclosure;

[0017] Figure 2 Flow chart of a method for generating an intelligent driving operation video based on AIGC according to an embodiment of the present disclosure;

[0018] Figure 3 is a schematic structural diagram of a video generating device according to an embodiment of the present disclosure;

[0019] Figure 4 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0021] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0022] Based on the description of background technology, the current vehicle intelligent driving technology is in a rapid development stage, mainly realizing L2 level intelligent driving capabilities, and some models already have L3 level intelligent driving functions. Intelligent driving technology has greatly reduced traffic accidents, improved traffic efficiency and provided a more comfortable driving experience. Because intelligent driving involves many scenarios and operational details, vehicle OEMs need to produce intelligent driving operation videos for specific models equipped with different intelligent driving functions, and use videos to explain in detail, so as to facilitate users to understand intelligent driving services.

[0023] There are currently two ways to produce smart driving operation videos: one is for the filming team to shoot the actual car, and the other is for the modeling video team to produce it through animation simulation. However, both video production methods involve long production cycles, high capital investment, and each model must be produced separately and cannot be reused.

[0024] The terms used in this disclosure are explained as follows:

[0025] Artificial Intelligence Generated Content (AIGC) refers to the use of artificial intelligence technology to generate various forms of content and data, including text, images, audio, video, etc.

[0026] Navigate on Autopilot (NOA) is an advanced driver assistance function that allows vehicles to achieve automatic driving on highways, including lane keeping, automatic lane changing, autonomous cruise control and navigation functions.

[0027] Adaptive Cruise Control (ACC) is an intelligent automatic control system that is mainly used to assist the driver in controlling the vehicle's speed and maintaining a safe distance from the vehicle in front.

[0028] The Highway Assist (HWA) system is an advanced system based on L2 autonomous driving technology. It integrates front and rear corner radars, front millimeter-wave radars, front-view cameras, domain controllers, panoramic surround-view cameras, ultrasonic radars and other equipment to achieve accurate detection of the surrounding environment, thereby providing assisted driving functions on highways.

[0029] Human Machine Interface (HMI) is the medium for interaction and information exchange between the system and the user, realizing the conversion between the internal form of information and the form acceptable to humans.

[0030] Electronic Park Brake (EPB) is a parking brake system controlled by electronic circuits. The electronic control unit receives the driver's instructions and controls the motor to drive the compression nut to achieve braking.

[0031] The Autonomous Emergency Braking system (AEB) is a vehicle active safety technology, mainly composed of three core parts: control module, ranging module and braking module.

[0032] The Electronic Stability Program (ESP) is mainly used to improve the safety and handling of the vehicle. It monitors the vehicle's driving status and helps the vehicle maintain dynamic balance through braking intervention in emergency situations to prevent the vehicle from oversteering or understeering when turning or avoiding obstacles in an emergency.

[0033] As mentioned above, how to avoid the long production cycle and non-reusability of driving operation videos has become an important research issue.

[0034] Based on the above description, if Figure 1 As shown, the video generation method proposed in this embodiment includes:

[0035] Step 101, obtaining target vehicle model information and target driving scene of a target vehicle.

[0036] In a specific implementation, the target vehicle is the vehicle for which a driving operation video needs to be produced. The target vehicle model information is the vehicle model information of the target vehicle. The target driving scene is the intelligent driving scene configured for the target vehicle.

[0037] The target vehicle model information may be a target vehicle model picture or a target vehicle model video, and in the embodiment of the present disclosure, the target vehicle model picture is preferred. The target vehicle model information includes the body information, interior information, exterior information, instrument information, and host information of the target vehicle.

[0038] For example, the model information of the target vehicle is model C, and is configured with smart driving scene a and smart driving scene b. Then it is determined that the target model information is model C, and the target driving scenes are smart driving scene a and smart driving scene b.

[0039] Step 102, using a pre-trained target video generation model, based on the target vehicle model information and the target driving scene, generates a target driving operation video of the target vehicle in the target driving scene.

[0040] In specific implementation, the target vehicle model information and the target driving scene are input into the pre-trained target video generation model to generate a target driving operation video of the target vehicle in the target driving scene. In this way, the target driving operation video is an intelligent driving operation video of the target vehicle, and the target driving operation video contains the target vehicle model information (body information, interior information, exterior information, instrument information and host information) and the configured target driving scene.

[0041] For example, the target vehicle model information of the target vehicle is model C, and the configured target driving scenes are intelligent driving scene a and intelligent driving scene b. Model C, intelligent driving scene a and intelligent driving scene b are input into the target video generation model to generate the target driving operation video of model C in intelligent driving scene a and the target driving operation video of model C in intelligent driving scene b.

[0042] In this way, by simply inputting the target vehicle model information and target driving scene, the target video generation model can be used to generate a target driving operation video to guide vehicle users to understand the relevant services of intelligent driving operation. The target video generation model can quickly generate a target driving operation video of the target vehicle, without the need to shoot or animate the target vehicle to produce the target driving operation video. It has a short production cycle, low cost, and can realize the reuse of the target video generation model. It can be used to batch produce intelligent driving operation videos of different shapes and different intelligent driving scenes, which can meet the increasing demand for intelligent driving videos.

[0043] Through the above embodiment, the target vehicle model information and the target driving scene of the target vehicle are obtained. The target driving operation video of the target vehicle in the target driving scene is generated according to the target vehicle model information and the target driving scene using the pre-trained target video generation model. In this way, the target driving operation video of the target vehicle can be generated using the target video generation model, and there is no need to shoot or animate the target vehicle to produce the target driving operation video, thereby reducing the production cycle of the target driving operation video. The vehicle model information of different vehicles is input into the target video generation model, and the driving operation videos of different vehicles can be generated, so that the target video generation model can be reused, thereby reducing the production cost of the driving operation video.

[0044] When generating a target driving operation video of a target vehicle, in order to make the target driving operation video more vivid, the target driving operation video includes the target vehicle model information of the target vehicle. The target driving operation video is generated according to the target vehicle model information and the target driving scene. The specific process is as follows:

[0045] In some embodiments, the target vehicle model information includes: target appearance information and target interior information; step 102 includes:

[0046] Step 1021: Generate an initial driving operation video under the target driving scenario according to the target driving scenario by using the target video generation model.

[0047] Step 1022, using the target video generation model, adding the target appearance information and the target interior information to the initial driving operation video to obtain a target driving operation video of the target vehicle in the target driving scene.

[0048] In a specific implementation, the target video generation model is used to generate an initial driving operation video under the target driving scene according to the target driving scene, wherein the initial driving operation video includes the target driving scene of the target vehicle.

[0049] For example, the target vehicle is configured with target driving scenarios of smart driving scenario a and smart driving scenario b. Smart driving scenario a and smart driving scenario b are input into the target video generation model to generate an initial driving operation video under smart driving scenario a and an initial driving operation video under smart driving scenario b.

[0050] The target appearance information (body information and exterior information) and target interior information (interior information, instrument information and host information) of the target vehicle are added to the initial driving operation video to obtain the target driving operation video of the target vehicle in the target driving scene.

[0051] For example, the target vehicle model information of the target vehicle is model C (appearance information C and interior information C). The appearance information C and interior information C are added to the initial driving operation video to obtain the target driving operation video of model C under intelligent driving scenario a and the target driving operation video of model C under intelligent driving scenario b.

[0052] Through the above scheme, the target video generation model can be used to generate an initial driving operation video in a target driving scene. The target appearance information and target interior information of the target vehicle are added to the initial driving operation video, so that the target driving operation video contains the target vehicle model information and the target driving scene.

[0053] In order to more accurately generate the initial driving operation video containing the target driving scene, the target driving scene of the target vehicle is judged. The specific process is as follows:

[0054] In some embodiments, the target driving scenario includes at least one of: an on-ramp or off-ramp scenario, a lane change protection scenario, and a distraction detection scenario; step 1021 includes:

[0055] Step 1021A, determining and processing the target driving scene of the target vehicle.

[0056] Step 1021B, in response to determining that the target driving scene is an on-ramp or off-ramp scene, determining an on-ramp or off-ramp video generation sub-model from the target video generation model, and using the on-ramp or off-ramp video generation sub-model to generate an initial driving operation video of the on-ramp or off-ramp.

[0057] Step 1021C, in response to determining that the target driving scene is a lane change protection scene, determining a lane change protection video generation sub-model from the target video generation model, and using the lane change protection video generation sub-model to generate an initial driving operation video for lane change protection.

[0058] Step 1021D, in response to determining that the target driving scene is a distraction detection scene, determining a distraction detection video generation sub-model from the target video generation model, and using the distraction detection video generation sub-model to generate an initial driving operation video for distraction detection.

[0059] During specific implementation, the target video generation model includes: an entry and exit ramp video generation sub-model, a lane change protection video generation sub-model, and a distraction detection video generation sub-model.

[0060] When the target driving scene configured on the target vehicle is an on-ramp or off-ramp scene, the on-ramp or off-ramp video generation sub-model is used to generate the initial driving operation video of the on-ramp or off-ramp. When the target driving scene configured on the target vehicle is a lane change protection scene, the lane change protection video generation sub-model is used to generate the initial driving operation video of lane change protection. When the target driving scene configured on the target vehicle is a distraction detection scene, the distraction detection video generation sub-model is used to generate the initial driving operation video of distraction detection.

[0061] Through the above scheme, the target driving scene of the target vehicle is judged, and the initial driving operation video is generated using the sub-model corresponding to the target driving scene, thereby avoiding the problem of waste of resources by directly using the target video generation model to generate the initial driving operation video.

[0062] In order to generate the target driving operation video more accurately, the target video generation model is pre-trained. The training process of the target video generation model is as follows:

[0063] In some embodiments, the training process of the target video generation model includes:

[0064] Step 201 : obtaining a sample video of a driving operation, extracting sample driving information of the driving operation from the sample video, and using the sample driving information to train a general video generation model to obtain an initial video generation model.

[0065] In a specific implementation, the sample video of the driving operation is sample data of the intelligent driving operation video. There are multiple sample videos of the driving operation. The general video generation model is a general large model for generating videos.

[0066] The general video generation model is trained using the sample driving information in the sample video to obtain an initial video generation model. In this way, the initial video generation model is trained to be a model specifically for producing intelligent driving operation videos, and the initial video generation model is used to generate intelligent driving operation videos.

[0067] Step 202: Obtain sample vehicle type information and sample intelligent driving scenes, and use the sample vehicle type information and the sample intelligent driving scenes to train the initial video generation model to obtain a target video generation model.

[0068] In specific implementation, the sample vehicle model information is sample data of the vehicle model, and the sample intelligent driving scenario is sample data of the intelligent driving scenario.

[0069] The initial video generation model is trained using sample vehicle model information and sample intelligent driving scenarios to obtain a target video generation model. In this way, the target video generation model is trained to be a model capable of producing intelligent driving operation videos of different vehicles, and the target video generation model is used to generate intelligent driving operation videos of the target vehicle in the target intelligent driving scenario.

[0070] For example, the sample vehicle model information includes: vehicle model A, vehicle model B and vehicle model C, and the sample intelligent driving scenes include: intelligent driving scene a, intelligent driving scene b and intelligent driving scene C. The initial video generation model is trained using the sample vehicle model information and the sample intelligent driving scene, and the target video generation model is obtained to generate intelligent driving operation videos of the sample vehicle model information and the sample intelligent driving scene.

[0071] When the target vehicle model information of the target vehicle is model C, and the target intelligent driving scene includes intelligent driving scene a and intelligent driving scene b, model C, intelligent driving scene a and intelligent driving scene b are input into the target video generation model to generate an intelligent driving operation video of the target vehicle. Among them, the intelligent driving operation video of the target vehicle includes: the first intelligent driving operation video of the target vehicle (including model C) under intelligent driving scene a, and the second intelligent driving operation video of the target vehicle (including model C) under intelligent driving scene b.

[0072] Through the above scheme, sample driving information of driving operation can be accurately extracted from sample videos of driving operation. The general video generation model is trained using sample driving information of driving operation, so that the obtained initial video generation model can be specially used to produce intelligent driving operation videos. The initial video generation model is trained using sample vehicle model information and sample intelligent driving scenes, so that the obtained target video generation model can produce intelligent driving operation videos of different vehicles.

[0073] In some embodiments, step 201 includes:

[0074] Step 2011, extracting sample start conditions, sample scenes, sample operations and sample exit conditions of intelligent driving from the sample video.

[0075] Step 2012: train the universal video generation model using the sample start condition, the sample scene, the sample operation and the sample exit condition to obtain an initial video generation model.

[0076] In specific implementation, the sample video of intelligent driving includes: sample start-up conditions, sample scenarios, sample operations and sample exit conditions of intelligent driving.

[0077] The universal video generation model is trained using sample start-up conditions, sample scenes, sample operations, and sample exit conditions of intelligent driving, so that the universal video generation model learns the start-up, scenes, operations, and exit of intelligent driving to obtain an initial video generation model.

[0078] In this way, the initial video generation model is trained as a model specifically for producing intelligent driving operation videos, and the initial video generation model is used to generate intelligent driving operation videos including startup, scenes, operations, and exits.

[0079] Through the above scheme, the general video generation model is trained using sample start-up conditions, sample scenes, sample operations, and sample exit conditions of intelligent driving, and the general video generation model learns the start-up, scenes, operations, and exit of intelligent driving. In this way, the trained initial video generation model can generate intelligent driving operation videos containing the start-up conditions, scenes, operations, and exit conditions of intelligent driving.

[0080] In some embodiments, step 2012 includes:

[0081] Step 2012A, sending the sample start condition, the sample scene, the sample operation and the sample exit condition to the general video generation model to obtain a first simulation video of intelligent driving.

[0082] Step 2012B: compare the first simulation video with the pre-stored first test video to obtain a first comparison result.

[0083] Step 2012C: based on the first comparison result, update the parameters of the general video generation model to obtain an initial video generation model.

[0084] In specific implementation, the sample start condition, sample scene, sample operation and sample exit condition are input into the general video generation model, and the general video generation model outputs the first simulation video of intelligent driving. The first simulation video is the intelligent driving operation video generated by the general video generation model based on the sample video during the training process.

[0085] The first test video is a standard video for testing the universal video generation model. The first simulation video and the first test video are compared to obtain a first comparison result, and a first loss function is determined based on the first comparison result. According to the first loss function, the parameters of the universal video generation model are updated. When the first loss function approaches a first preset value, the updated universal video generation model is used as the initial video generation model.

[0086] Through the above scheme, the universal video generation model is trained using the sample start-up conditions, sample scenes, sample operations and sample exit conditions of intelligent driving, so that the obtained initial video generation model can be used to specifically produce intelligent driving operation videos. In addition, based on the first comparison result between the first simulation video output by the universal video generation model and the first pre-stored test video, the parameters of the universal video generation model can be accurately adjusted and updated, so that the trained initial video generation model is more accurate.

[0087] In some embodiments, step 2012 includes:

[0088] Step 2012a, using the sample starting conditions to train the universal video generation model to obtain a first video generation model.

[0089] Step 2012b: Use the sample scene to train the first video generation model to obtain a second video generation model.

[0090] Step 2012c: Use the sample operation to train the second video generation model to obtain a third video generation model.

[0091] Step 2012d, using the sample exit condition to train the third video generation model to obtain an initial video generation model.

[0092] In a specific implementation, the sample start-up conditions are used to train the general video generation model to obtain the first video generation model. The trained first video generation model is a general video generation model after learning the intelligent driving start-up conditions, and can generate a video of intelligent driving start-up.

[0093] The first video generation model is trained using the sample scene to obtain a second video generation model. The trained second video generation model is a general video generation model after learning the intelligent driving scene, and can generate videos of intelligent driving startup and different scenes.

[0094] The second video generation model is trained using the sample operation to obtain a third video generation model. The trained third video generation model is a general video generation model after learning the intelligent driving operation, and can generate videos of intelligent driving startup, different scenes, and corresponding operations.

[0095] The sample exit condition is used to train the third video generation model to obtain an initial video generation model. The trained initial video generation model is a general video generation model after learning the intelligent driving scene, and can generate videos of intelligent driving startup, different scenes, corresponding operations, and exit.

[0096] In addition, step 2012 also includes: using the sample start condition to train the universal video generation model to obtain a first video generation model; using the sample scene to train the universal video generation model to obtain a second video generation model; using the sample operation to train the universal video generation model to obtain a third video generation model; using the sample exit condition to train the universal video generation model to obtain a fourth video generation model; combining the first video generation model, the second video generation model, the third video generation model and the fourth video generation model to obtain an initial video generation model.

[0097] Through the above scheme, the general video generation model is trained using sample start-up conditions, sample scenes, sample operations, and sample exit conditions in sequence. In this way, the general video generation model can sequentially learn the start-up conditions, scenes, operations, and exit conditions of intelligent driving, so that the trained initial video generation model can generate videos containing intelligent driving start-up, different scenes, and corresponding operations and exits, so that the initial video generation model can specifically produce intelligent driving operation videos.

[0098] In some embodiments, the process of extracting the sample start condition includes:

[0099] Step 20111, extracting the gear status, electronic parking brake system status, vehicle navigation status, driver's seat belt status, door status, hood status, trunk status, automatic emergency brake system status, vehicle electronic stability system status and vehicle speed status from the sample video.

[0100] Step 20112, based on the gear status, the electronic parking brake system status, the vehicle navigation status, the main driver's seat belt status, the door status, the hood status, the trunk status, the automatic emergency brake system status, the body electronic stability system status and the vehicle speed status, determine the sample starting conditions for intelligent driving.

[0101] In specific implementation, sample startup conditions for intelligent driving include: the gear state is forward gear, the EPB state is inactive, the vehicle navigation state is navigation on (that is, a starting point and an end point are set), the main driver's seat belt state is fastened, the door state is closed, the hood state and the trunk state are closed, the AEB state and the ESP state are on, and the vehicle speed state is within the preset driving speed range.

[0102] The preset driving speed range is the speed range of the vehicle under normal driving conditions, for example, the preset driving speed range may be 10 km / h-120 km / h.

[0103] Through the above scheme, based on the gear status, electronic parking brake system status, vehicle navigation status, main driver's seat belt status, door status, hood status, trunk status, automatic emergency braking system status, body electronic stability system status and vehicle speed status, the sample startup conditions of intelligent driving can be comprehensively and accurately determined.

[0104] In some embodiments, the sample scene extraction process includes:

[0105] Step 20113, extracting the on-ramp and off-ramp scenes, lane change protection scenes, distraction detection scenes, avoidance entrance scenes, vehicle avoidance scenes and intersection identification scenes from the sample video.

[0106] Step 20114, taking the ramp entry and exit scenario, the lane change protection scenario, the distraction detection scenario, the avoidance entrance scenario, the vehicle avoidance scenario and the intersection identification scenario as sample scenarios for intelligent driving.

[0107] In specific implementation, the ramp-in and ramp-out scenarios are to set the route according to the high-precision map navigation to achieve intelligent ramp-in and ramp-out. Specifically, the ramp features are extracted from the sample video, and the ramp-in and ramp-out scenarios are determined based on the ramp features. Among them, the ramp-in and ramp-out scenarios include the ramp-in and ramp-out scenarios.

[0108] The lane change protection scenario is to monitor the status of the vehicle behind in real time, and terminate the lane change in time when a fast-approaching vehicle is approaching from behind during the lane change, thereby reducing the risk of rear-end collision. Specifically, the lane change features are extracted from the sample video, and the lane change protection scenario is determined based on the lane change features.

[0109] The distraction detection scenario monitors the driver's status in real time. When the driver is distracted, fatigued, or the current operating environment does not support the system to continue to control the vehicle, different levels of prompts and alarms are given. Specifically, the driver's features are extracted from the sample video, and the distraction detection scenario is determined based on the driver's features.

[0110] The avoidance merge entrance scenario is to accurately identify the merging lane ahead based on navigation information and change lanes in advance to avoid the merge entrance. Specifically, the merge entrance features are extracted from the sample video, and the avoidance merge entrance scenario is determined based on the merge entrance features.

[0111] The vehicle avoidance scenario is to monitor surrounding vehicles in real time, and when a large truck is driving in the adjacent lane, intelligent avoidance is performed to ensure safety in multiple dimensions. Specifically, the surrounding vehicle features are extracted from the sample video, and the vehicle avoidance scenario is determined based on the surrounding vehicle features.

[0112] The intersection scene is to monitor in real time whether there is an intersection ahead, and change lanes before driving to the intersection to ensure efficient and safe driving. Specifically, the intersection features are extracted from the sample video, and the intersection scene is determined based on the intersection features.

[0113] Through the above scheme, the entry and exit ramp scenes, lane change protection scenes, distraction detection scenes, avoidance entrance scenes, vehicle avoidance scenes and intersection identification scenes are extracted from the sample videos, which can comprehensively and accurately determine the sample scenes of intelligent driving.

[0114] In order to train the general video generation model into a video generation model that can generate intelligent driving operation videos, the general video generation model is trained using sample driving information. The sample operation extraction process in the sample driving information is as follows:

[0115] In some embodiments, the sample driving information includes: a sample scene and a sample operation; the sample scene includes: an on-ramp and off-ramp scene; the extraction process of the sample operation includes:

[0116] Step 2011A, in response to determining that the sample scene is the on-ramp and off-ramp scene, extracting the on-ramp and off-ramp operation corresponding to the on-ramp and off-ramp scene from the sample video, and determining that the on-ramp and off-ramp operation is: reducing the vehicle speed to a preset speed, turning on the turn signal and giving a first prompt.

[0117] In specific implementation, when the sample scenario is an on-ramp or off-ramp scenario, the on-ramp or off-ramp operations are as follows: reducing the vehicle speed to a preset speed, turning on the turn signal, and giving the first prompt.

[0118] The preset vehicle speed is a pre-set vehicle speed for entering or exiting the ramp. After turning on the turn signal, the vehicle is controlled to enter or exit the ramp. The first prompt is the prompt for entering or exiting the ramp. The prompting methods of the first prompt include: voice broadcast prompt, screen display prompt and in-vehicle light prompt.

[0119] For example, the preset vehicle speed is 20 km / h, and the first prompt is a voice broadcast prompt. The specific operation of entering or exiting the ramp is: reduce the vehicle speed to 20 km / h, turn on the turn signal, control the vehicle to enter or exit the ramp, and perform the first voice broadcast prompt to remind the driver that the operation of entering or exiting the ramp is in progress.

[0120] Through the above scheme, by extracting the on-ramp and off-ramp operations corresponding to the on-ramp and off-ramp scenes, the general video generation model is trained using the on-ramp and off-ramp scenes and the corresponding on-ramp and off-ramp operations, so that the obtained initial video generation model can generate videos containing on-ramp and off-ramp scenes and on-ramp and off-ramp operations.

[0121] In order to train the general video generation model into a video generation model that can generate intelligent driving operation videos, the general video generation model is trained using sample driving information. The sample operation extraction process in the sample driving information is as follows:

[0122] In some embodiments, the sample driving information includes: a sample scene and a sample operation; the sample scene includes: a lane change protection scene; the extraction process of the sample operation includes:

[0123] Step 2011B, in response to determining that the sample scene is the lane change protection scene, extracting the lane change protection operation corresponding to the lane change protection scene from the sample video, and determining that the lane change protection operation is: turning on the turn signal before changing lanes and giving a second prompt, determining whether to stop changing lanes by monitoring the status of the vehicle behind, and turning off the turn signal and controlling the steering wheel to return to a preset angle after determining to stop changing lanes.

[0124] In specific implementation, when the sample scenario is a lane change protection scenario, the lane change protection operation is specifically: turn on the turn signal and give a second prompt before changing lanes, determine whether to stop changing lanes by monitoring the status of the vehicle behind, and after determining to stop changing lanes, turn off the turn signal and control the steering wheel to return to the preset angle.

[0125] The second prompt is a lane change protection prompt, and the second prompt includes: voice broadcast prompt, screen display prompt and in-vehicle light prompt. The preset angle is a preset steering wheel angle for normal driving of the vehicle after stopping the lane change.

[0126] For example, the second prompt is a voice announcement. The lane change protection operation is as follows: before changing lanes, turn on the turn signal and give a second voice announcement to remind the driver that the lane change protection operation is in progress. Before changing lanes, monitor the status of the vehicle behind you in real time. When the status of the vehicle behind you indicates that there is a vehicle behind you approaching quickly, stop changing lanes, turn off the turn signal, and control the steering wheel back to the preset angle for normal driving.

[0127] Through the above scheme, by extracting the lane change protection operation corresponding to the lane change protection scene, the general video generation model is trained using the lane change protection scene and the corresponding lane change protection operation, so that the obtained initial video generation model can generate a video containing the lane change protection scene and the lane change protection operation.

[0128] In order to train the general video generation model into a video generation model that can generate intelligent driving operation videos, the general video generation model is trained using sample driving information. The sample operation extraction process in the sample driving information is as follows:

[0129] In some embodiments, the sample driving information includes: a sample scene and a sample operation; the sample scene includes: a distraction detection scene; the extraction process of the sample operation includes:

[0130] Step 2011C, in response to determining that the sample scene is the distraction detection scene, extracting the distraction detection operation corresponding to the distraction detection scene from the sample video, and determining that the distraction detection operation is: after monitoring that the driver is in a distracted state, making a third prompt, after monitoring that the driver is continuously in a distracted state for a preset time period, controlling the vehicle to stop by reducing the speed and turning on the hazard lights.

[0131] In specific implementation, when the sample scenario is a distraction detection scenario, the distraction detection operation is specifically: after detecting that the driver is in a distracted state, a third prompt is given; after detecting that the driver continues to be distracted for a preset period of time, the vehicle is controlled to stop by reducing the speed and turning on the double flashes.

[0132] Among them, the third prompt is a lane change protection prompt, and the prompt methods of the third prompt include: voice broadcast prompt, screen display prompt and in-vehicle light prompt.

[0133] For example, the third prompt is in the form of voice broadcast prompt and in-car light prompt. The specific operation of distraction detection is: real-time monitoring of the driver's state, when the driver is distracted (distracted, fatigued or the current operating environment does not support the automatic driving system to continue to control the vehicle), the third voice broadcast prompt and the third in-car light prompt are performed. When the driver is distracted for a preset period of time, after reducing the speed and controlling the vehicle to park on the roadside, the double flash is turned on to remind the vehicles behind to pay attention.

[0134] Through the above scheme, by extracting the distraction detection operation corresponding to the distraction detection scene, the general video generation model is trained using the distraction detection scene and the corresponding distraction detection operation, so that the obtained initial video generation model can generate a video containing the distraction detection scene and the distraction detection operation.

[0135] In some embodiments, the process of extracting the sample exit condition includes:

[0136] Step 2011a, extracting the electronic parking brake system status, vehicle navigation status, driver's seat belt status, door status, automatic emergency brake system status, vehicle electronic stability system status and driver's hands on the steering wheel status from the sample video.

[0137] Step 2011b, based on the electronic parking brake system status, the main driver's seat belt status, the door status, the automatic emergency brake system status, the body electronic stability system status and the driver's hands on the steering wheel status, determine the first sample exit condition of intelligent driving.

[0138] Step 2011c, based on the vehicle navigation status, determine the second sample exit condition of the intelligent driving.

[0139] In specific implementation, the first sample exit condition of intelligent driving includes at least one of the following: the main driver's seat belt status is not fastened, the door status is open, the EPB status is activated, the AEB status and the ESP status are off, and the driver's hand-holding steering wheel status is not in a holding state.

[0140] When the vehicle meets at least one of the first sample exit conditions, the vehicle is controlled to exit the automatic driving mode, that is, the NOA function is completely exited, the adaptive cruise system ACC and the highway driving assistance system HWA are also exited, and the driver is prompted through the human-machine interface HMI (for example, the on-board screen).

[0141] In addition, when the driver pulls out the cruise control handle once or presses the brake pedal, the NOA function can also be exited, and the driver is prompted through the human-machine interface HMI (for example, the vehicle-mounted screen).

[0142] The second sample exit condition of intelligent driving includes: the vehicle navigation state is the navigation exit state.

[0143] When the vehicle meets the second sample exit condition, the vehicle is controlled to exit the automatic driving mode and switch to the highway assisted driving mode, that is, the NOA function exits and switches to the highway assisted driving system HWA.

[0144] Through the above scheme, based on the status of the electronic parking brake system, the main driver's seat belt status, the door status, the automatic emergency brake system status, the body electronic stability system status and the driver's hand-held steering wheel status, the first sample exit condition of intelligent driving can be fully and accurately determined, and the NOA function will be completely exited when the first sample exit condition is met. Based on the vehicle navigation status, the second sample exit condition of intelligent driving can be accurately determined. When the second sample exit condition is met, the NOA function is exited and switched to the highway assisted driving system HWA.

[0145] In some embodiments, step 202 includes:

[0146] Step 2021: Use the sample vehicle model information and the sample intelligent driving scene to train the initial video generation model to obtain a second simulation video of intelligent driving.

[0147] Step 2022: compare the second simulation video with the pre-stored second test video to obtain a second comparison result.

[0148] Step 2023: Based on the second comparison result, update the parameters of the initial video generation model to obtain a target video generation model.

[0149] In specific implementation, the sample vehicle model information and the sample intelligent driving scene are input into the initial video generation model, and the initial video generation model outputs a second simulated video of intelligent driving. The second simulated video is the intelligent driving operation video generated by the initial video generation model based on the sample vehicle model information and the sample intelligent driving scene during the training process.

[0150] The second test video is a standard video for testing the initial video generation model. The second simulation video and the second test video are compared to obtain a second comparison result, and a second loss function is determined based on the second comparison result. According to the second loss function, the parameters of the initial video generation model are updated. When the second loss function approaches a second preset value, the updated initial video generation model is used as the target video generation model.

[0151] Through the above scheme, the initial video generation model is trained using sample vehicle model information and sample intelligent driving scenarios, so that the target video generation model can produce intelligent driving operation videos of different vehicles. In addition, based on the second comparison result between the second simulation video output by the initial video generation model and the pre-stored second test video, the parameters of the initial video generation model can be accurately adjusted and updated, so that the trained target video generation model is more accurate.

[0152] Through the above embodiment, the target vehicle model information and the target driving scene of the target vehicle are obtained. The target driving operation video of the target vehicle in the target driving scene is generated according to the target vehicle model information and the target driving scene using the pre-trained target video generation model. In this way, the target driving operation video of the target vehicle can be generated using the target video generation model, and there is no need to shoot or animate the target vehicle to produce the target driving operation video, thereby reducing the production cycle of the target driving operation video. The vehicle model information of different vehicles is input into the target video generation model, and the driving operation videos of different vehicles can be generated, so that the target video generation model can be reused, thereby reducing the production cost of the driving operation video.

[0153] It should be noted that the embodiments of the present disclosure may be further described in the following manner:

[0154] Figure 2 FIG. 1 is a flow chart of a method for generating an intelligent driving operation video based on AIGC according to an embodiment of the present disclosure. Figure 2 As shown, the method for generating an intelligent driving operation video based on AIGC includes:

[0155] 1. AIGC general large model training stage

[0156] The intelligent driving operation video data (i.e., sample video) is input into the AIGC general large model (i.e., general video generation model), so that the general large model can learn to determine the intelligent driving service startup conditions, decompose the intelligent driving scenarios, learn the operations of different intelligent driving scenarios, and learn to determine the intelligent driving service exit conditions based on the input video data.

[0157] Through the above learning, the AIGC general big model generates neural networks and algorithms for the intelligent driving service startup conditions, intelligent driving scenarios, different intelligent driving scenario operations, and intelligent driving service exit conditions. The AIGC general big model is trained as the first version of the AIGC intelligent driving video dedicated big model (i.e., the initial video generation model) specifically for producing intelligent driving videos.

[0158] (1) Obtaining intelligent driving video

[0159] The intelligent driving platform feeds a large amount of intelligent driving video data into the AIGC general large model.

[0160] (2) Learning to determine the conditions for starting intelligent driving services

[0161] The AIGC general large model learns and determines the conditions for starting the intelligent driving service based on the input video data. For example, after learning, AIGC uses the large model to determine that the following conditions must be met to start the intelligent driving service:

[0162] ① The gear is in forward gear, EPB is not activated, and the driver sets the starting and ending points through the vehicle navigation;

[0163] ② The driver's seat belt is fastened;

[0164] ③The doors, hood and trunk are closed;

[0165] ④The vehicle's AEB and ESP function switches are turned on;

[0166] ⑤The vehicle speed is between 10-120km / h, etc.

[0167] (3) Decomposition of intelligent driving scenarios

[0168] The AIGC general large model decomposes intelligent driving scenarios based on the input video. For example, the intelligent driving scenarios decomposed by the AIGC general large model include:

[0169] ① Intelligent on / off ramp assistance: Set the route based on high-precision map navigation to achieve intelligent on / off ramp;

[0170] ② Intelligent lane change protection: Real-time monitoring of the status of the vehicle behind, when there is a fast-moving vehicle from behind during the lane change, the lane change is terminated in time to reduce the risk of rear-end collision;

[0171] ③ Distraction / fatigue detection: Real-time monitoring of the driver's status. When the driver is distracted, fatigued, or the current operating environment does not support the system to continue to control the vehicle, different levels of prompts and alarms are issued;

[0172] ④ Intelligent merging entrance avoidance: Based on navigation information, the system can accurately identify the merging lane ahead and change lanes in advance to avoid the merging entrance;

[0173] ⑤ Intelligent avoidance of large trucks: Real-time monitoring of surrounding conditions, intelligent avoidance when encountering large trucks in adjacent lanes, multi-dimensional safety assurance;

[0174] ⑥ Intelligent identification of easily confused intersections: Real-time monitoring of surrounding conditions, and changing lanes in advance when driving to easily confused intersections to ensure efficient and safe driving.

[0175] (4) Learn different intelligent driving scenarios

[0176] The AIGC general large model learns operations in different intelligent driving scenarios based on the input video, such as:

[0177] ① Intelligent on / off ramp assistance: Before entering / exiting the ramp, the vehicle speed is reduced to 20km / h, the turn signal is turned on simultaneously, and a voice broadcast is made to remind the driver that the intelligent on / off ramp is in progress;

[0178] ② Intelligent lane change protection: Turn on the turn signal and give a voice reminder to the driver before changing lanes. Monitor the status of the vehicle behind in real time before changing lanes. When there is a fast-moving vehicle coming from behind during the lane change, stop the lane change in time, turn off the turn signal, return the steering wheel to the center position and continue driving;

[0179] ③ Distraction / fatigue detection: Real-time monitoring of the driver's status. When the driver is distracted, fatigued, or the current operating environment does not support the system to continue to control the vehicle, the driver can be reminded through the lights or voice in the car. If necessary, the vehicle will turn on the turn signal, reduce the speed and park by the roadside, and turn on the double flash to remind the vehicles behind to pay attention.

[0180] (5) Learning to determine the exit conditions of intelligent driving services

[0181] The AIGC general large model determines the exit conditions of the intelligent driving service based on the input video learning. For example, after learning, AIGC uses the large model to determine that the intelligent driving service will be exited if the following conditions are met:

[0182] ① When the driver's seat belt is not fastened, the door is open, EPB is activated, and AEB and ESP functions are not turned on, the NOA function will be completely exited and will not return to adaptive cruise control (ACC) and highway driving assistance (HWA), and the user will be prompted through the HMI;

[0183] ② If the conditions for starting the intelligent driving service are met and the NOA function is activated, if the customer exits the set in-vehicle navigation, NOA will exit and switch to Highway Driving Assist (HWA); the customer can also exit the NOA function by pulling the cruise control handle outward once or stepping on the brake pedal, and the user will be prompted through the HMI;

[0184] ③ When the driver does not hold the steering wheel, the NOA function may also exit and the driver will be reminded.

[0185] (6) The first version of AIGC intelligent driving video dedicated large model

[0186] Through the above learning, the AIGC general big model generates neural networks and algorithms for the intelligent driving service startup conditions, intelligent driving scenarios, different intelligent driving scenario operations, and intelligent driving service exit conditions. The AIGC general big model is trained as the first version of the AIGC intelligent driving video dedicated big model specifically for producing intelligent driving videos.

[0187] 2. AIGC generates a large model for intelligent driving video reasoning experiment phase

[0188] The specific vehicle model (including body, interior and exterior trim, instruments, host computer and other information) and the desired intelligent driving scenario are input into the first edition AIGC intelligent driving operation video dedicated large model. The first edition dedicated large model uses the neural network and algorithms of the intelligent driving service startup conditions, intelligent driving scenarios, different intelligent driving scenario operations and intelligent driving service exit conditions in the model to generate an intelligent driving operation video of an intelligent driving vehicle model with a specific shape and specific driving scenario.

[0189] Then, the parameters of the initial version of the dedicated large model are corrected according to the video problems, the neural network and learning algorithm of the initial version of the dedicated large model are optimized, and the dedicated large model is tuned to finally form the final version of the AIGC intelligent driving operation video dedicated large model (i.e. the target video generation model).

[0190] (1) The first version of AIGC intelligent driving video dedicated large model obtains vehicle model images and intelligent driving scenarios

[0191] The specific vehicle model design (including body, interior and exterior decoration, instruments, host and other information) and the desired intelligent driving scenario are input into the first version of AIGC intelligent driving operation video dedicated large model for reasoning experiments.

[0192] (2) Generate a video to explain how to operate smart driving

[0193] The first version of the AIGC smart driving video dedicated large model generates a smart driving operation instruction video based on the input styling pictures and smart driving scenarios, combined with the neural network and algorithms of the smart driving service startup conditions, smart driving scenarios, different smart driving scenario operations and smart driving service exit conditions in the model.

[0194] (3) Optimize the parameters of the initial version of the large model

[0195] In response to the problems with the generated videos, the parameters of the first version of the dedicated large model were corrected, the neural network and learning algorithm of the first version of the dedicated large model were optimized, and the dedicated large model was tuned.

[0196] (4) AIGC generates a large model for intelligent driving videos

[0197] After repeated reasoning experiments, the large model neural network and learning algorithm were deeply optimized, and finally the final version of AIGC was formed to generate a large model dedicated to intelligent driving operation videos.

[0198] 3. Intelligent Driving Operation Video Generation Stage

[0199] The final version of the AIGC smart driving video dedicated large model generates a smart driving operation instruction video based on the input styling pictures and smart driving scenarios, combined with the neural network and algorithms of the smart driving service startup conditions, smart driving scenarios, different smart driving scenario operations and smart driving service exit conditions in the dedicated model, to guide vehicle users to understand the operation of smart driving related services.

[0200] (1) AIGC intelligent driving video dedicated large model to obtain vehicle model images and intelligent driving scenarios

[0201] The specific vehicle model styling diagram (i.e. target vehicle model information, including body, interior and exterior trim, instrumentation, host, etc.) and the desired intelligent driving scenario (i.e. target intelligent driving scenario) are input into the final version of the AIGC intelligent driving operation video dedicated large model.

[0202] (2) Generate a video to explain how to operate smart driving

[0203] The final version of the AIGC smart driving video dedicated large model generates a smart driving operation instruction video based on the input styling pictures and smart driving scenarios, combined with the neural network and algorithms of the smart driving service startup conditions, smart driving scenarios, different smart driving scenario operations and smart driving service exit conditions in the dedicated model, to guide vehicle users to understand the operation of smart driving related services.

[0204] Through the above embodiment, only the target vehicle model information and target intelligent driving scene of the target vehicle need to be input, and the target video generation model can be used to generate an intelligent driving operation video to guide vehicle users to understand and operate intelligent driving related services. It has a short production cycle, low capital, and can realize technology reuse. It can be used to batch produce intelligent driving operation videos of different shapes and different intelligent driving scenes, which can meet the increasing demand for intelligent driving videos.

[0205] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0206] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0207] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a video generating device.

[0208] refer to Figure 3 , the video generating device comprises:

[0209] An acquisition module 301 is configured to acquire target vehicle model information and target driving scene of a target vehicle;

[0210] The generation module 302 is configured to generate a target driving operation video of the target vehicle in the target driving scenario based on the target vehicle model information and the target driving scenario using a pre-trained target video generation model.

[0211] In some embodiments, the target vehicle model information includes: target appearance information and target interior information;

[0212] The generating module 302 includes:

[0213] an initial video generating unit, configured to generate an initial driving operation video under the target driving scenario according to the target driving scenario by using the target video generating model;

[0214] The target video generation unit is configured to use the target video generation model to add the target appearance information and the target interior information to the initial driving operation video to obtain a target driving operation video of the target vehicle in the target driving scene.

[0215] In some embodiments, the target driving scenario includes at least one of: an on-ramp or off-ramp scenario, a lane change protection scenario, and a distraction detection scenario;

[0216] The initial video generating unit comprises:

[0217] A judgment processing subunit is configured to perform judgment processing on a target driving scene of the target vehicle;

[0218] A first initial video sub-unit is configured to, in response to determining that the target driving scene is an on-ramp or off-ramp scene, determine an on-ramp or off-ramp video generation sub-model from the target video generation model, and generate an initial driving operation video of the on-ramp or off-ramp using the on-ramp or off-ramp video generation sub-model;

[0219] A second initial video sub-unit is configured to, in response to determining that the target driving scene is a lane change protection scene, determine a lane change protection video generation sub-model from the target video generation model, and generate an initial driving operation video of lane change protection using the lane change protection video generation sub-model;

[0220] The third initial video sub-unit is configured to determine a distraction detection video generation sub-model from the target video generation model in response to determining that the target driving scene is a distraction detection scene, and use the distraction detection video generation sub-model to generate an initial driving operation video for distraction detection.

[0221] In some embodiments, the apparatus further comprises:

[0222] A first training module is configured to obtain a sample video of a driving operation, extract sample driving information of the driving operation from the sample video, and train a general video generation model using the sample driving information to obtain an initial video generation model;

[0223] The second training module is configured to obtain sample vehicle model information and sample intelligent driving scenes, and use the sample vehicle model information and the sample intelligent driving scenes to train the initial video generation model to obtain a target video generation model.

[0224] In some embodiments, the sample driving information includes: sample scenes and sample operations; the sample scenes include: on-ramp and off-ramp scenes;

[0225] The first training module includes:

[0226] The on-ramp and off-ramp operation extraction unit is configured to extract the on-ramp and off-ramp operation corresponding to the on-ramp and off-ramp scene from the sample video in response to determining that the sample scene is the on-ramp and off-ramp scene, and determine that the on-ramp and off-ramp operation is: reducing the vehicle speed to a preset speed, turning on the turn signal and giving a first prompt.

[0227] In some embodiments, the sample driving information includes: sample scenarios and sample operations; the sample scenarios include: lane change protection scenarios;

[0228] The first training module includes:

[0229] The lane change protection operation extraction unit is configured to, in response to determining that the sample scene is the lane change protection scene, extract the lane change protection operation corresponding to the lane change protection scene from the sample video, and determine that the lane change protection operation is: turning on the turn signal and giving a second prompt before changing lanes, determining whether to stop changing lanes by monitoring the status of the vehicle behind, and turning off the turn signal and controlling the steering wheel to return to a preset angle after determining that the lane change is to be stopped.

[0230] In some embodiments, the sample driving information includes: sample scenarios and sample operations; the sample scenarios include: distraction detection scenarios;

[0231] The first training module includes:

[0232] The distraction detection operation extraction unit is configured to, in response to determining that the sample scene is the distraction detection scene, extract the distraction detection operation corresponding to the distraction detection scene from the sample video, and determine that the distraction detection operation is: after monitoring that the driver is in a distracted state, make a third prompt, and after monitoring that the driver is continuously in a distracted state for a preset time, control the vehicle to stop by reducing the speed and turn on the hazard lights.

[0233] For the convenience of description, the above device is described by dividing it into various modules according to its functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0234] The device of the above embodiment is used to implement the corresponding video generation method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0235] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the video generation method described in any of the above embodiments is implemented.

[0236] Figure 4A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0237] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0238] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0239] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0240] The communication interface 1040 is used to connect a communication module (not shown in the figure) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB (Universal Serial Bus), network cable, etc.), or through a wireless mode (such as mobile network, WIFI (Wireless Fidelity), Bluetooth, etc.).

[0241] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0242] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0243] The electronic device of the above embodiment is used to implement the corresponding video generation method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0244] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the video generation method described in any of the above embodiments.

[0245] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0246] The computer instructions stored in the storage medium of the above embodiments are used to enable the computer to execute the video generation method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0247] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments, the present application also provides a vehicle, including a video generating device, or an electronic device, or a storage medium in the above-mentioned embodiments, and the vehicle equipment implements the video generating method described in any of the above embodiments.

[0248] The vehicle of the above-mentioned embodiment is used to implement the video generation method described in any of the above-mentioned embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0249] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a computer program product, including computer program instructions. When the computer program instructions are run on a computer, the computer executes the video generation method described in any of the above embodiments, which has the beneficial effects of the corresponding method embodiments and will not be repeated here.

[0250] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0251] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0252] As an optional but non-limiting implementation, in response to receiving the user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0253] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0254] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0255] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present disclosure difficult to understand, the known power / ground connections to the integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it is apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0256] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0257] The embodiments of the present disclosure are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the present disclosure. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A video generation method, characterized in that: The method comprises: Obtain target vehicle model information and target driving scenario of the target vehicle; A target driving operation video of the target vehicle in the target driving scenario is generated by using a pre-trained target video generation model according to the target vehicle model information and the target driving scenario.

2. The method according to claim 1, characterized in that The target vehicle model information includes: target appearance information and target interior information; The method of using a pre-trained target video generation model to generate a target driving operation video of the target vehicle in the target driving scene according to the target vehicle model information and the target driving scene includes: Generate an initial driving operation video under the target driving scenario according to the target driving scenario by using the target video generation model; The target video generation model is used to add the target appearance information and the target interior information to the initial driving operation video to obtain a target driving operation video of the target vehicle in the target driving scene.

3. The method according to claim 2, characterized in that The target driving scenario includes at least one of: an on-ramp or off-ramp scenario, a lane change protection scenario, and a distraction detection scenario; The generating model using the target video to generate an initial driving operation video in the target driving scenario according to the target driving scenario includes: Determining and processing a target driving scene of the target vehicle; In response to determining that the target driving scene is an on-ramp or off-ramp scene, determining an on-ramp or off-ramp video generation sub-model from the target video generation model, and generating an initial driving operation video of the on-ramp or off-ramp using the on-ramp or off-ramp video generation sub-model; In response to determining that the target driving scene is a lane change protection scene, determining a lane change protection video generation sub-model from the target video generation model, and generating an initial driving operation video of lane change protection using the lane change protection video generation sub-model; In response to determining that the target driving scene is a distraction detection scene, a distraction detection video generation sub-model is determined from the target video generation model, and an initial driving operation video for distraction detection is generated using the distraction detection video generation sub-model.

4. The method according to claim 1, characterized in that The training process of the target video generation model includes: Acquire a sample video of a driving operation, extract sample driving information of the driving operation from the sample video, and use the sample driving information to train a general video generation model to obtain an initial video generation model; Sample vehicle type information and sample intelligent driving scenes are obtained, and the initial video generation model is trained using the sample vehicle type information and the sample intelligent driving scenes to obtain a target video generation model.

5. The method according to claim 4, characterized in that The sample driving information includes: sample scenes and sample operations; the sample scenes include: on-ramp and off-ramp scenes; The extraction process of the sample operation includes: In response to determining that the sample scene is the on-ramp and off-ramp scene, extracting the on-ramp and off-ramp operations corresponding to the on-ramp and off-ramp scene from the sample video, and determining that the on-ramp and off-ramp operations are: reducing the vehicle speed to a preset speed, turning on the turn signal, and giving a first prompt.

6. The method according to claim 4, characterized in that The sample driving information includes: sample scenarios and sample operations; the sample scenarios include: lane change protection scenarios; The extraction process of the sample operation includes: In response to determining that the sample scene is the lane change protection scene, a lane change protection operation corresponding to the lane change protection scene is extracted from the sample video, and the lane change protection operation is determined to be: turning on the turn signal and giving a second prompt before changing lanes, determining whether to stop changing lanes by monitoring the status of the vehicle behind, and turning off the turn signal and controlling the steering wheel to return to a preset angle after determining to stop changing lanes.

7. The method according to claim 4, characterized in that The sample driving information includes: sample scenes and sample operations; the sample scenes include: distraction detection scenes; The extraction process of the sample operation includes: In response to determining that the sample scene is the distraction detection scene, a distraction detection operation corresponding to the distraction detection scene is extracted from the sample video, and the distraction detection operation is determined to be: after monitoring that the driver is in a distracted state, a third prompt is given; after monitoring that the driver continues to be distracted for a preset time period, the vehicle is controlled to stop by reducing the speed and turning on the double flashes.

8. A video generating device, characterized in that: include: An acquisition module is configured to acquire target vehicle model information and a target driving scene of a target vehicle; The generation module is configured to use a pre-trained target video generation model to generate a target driving operation video of the target vehicle in the target driving scenario according to the target vehicle model information and the target driving scenario.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and running on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A vehicle, characterized in that: Includes the video generating device according to claim 8 or the electronic device according to claim 9.

Images