Driving video generation method, fatigue driving monitoring model training method and fatigue driving monitoring model monitoring method

Abstract

The invention discloses a driving video generation method, a fatigue driving monitoring model training method and a monitoring method.The driving video generation method comprises the steps that driver expression state change information is obtained according to a driver human body area corresponding to multiple frames of images included in a first driving video, according to driver human body area depth information and driver human body area angle information corresponding to multiple frames of images included in the first driving video, obtaining driver human body area position change information; and generating a plurality of target driving videos according to the target frame image, the driver expression state change information and the driver body area position change information included in the second driving video. Thus, based on the first driving video and the second driving video, multiple target driving videos can be automatically generated, manpower needed by driver driving image acquisition is saved, and the image acquisition efficiency is improved.

Description

Technical Field

[0001] The present application relates to the technical field of fatigue driving monitoring, and in particular to a driving video generation method, a fatigue driving monitoring model training method, a fatigue driving monitoring method, a vehicle control method and an electronic device. Background Art

[0002] Fatigue driving monitoring is a very important monitoring technology in the automotive field involving vehicle driving safety. By monitoring the driver's fatigue driving, traffic accidents can be effectively avoided.

[0003] In the prior art, by acquiring the driver's driving image such as driver image data or video data, the driver's driving image is input into the fatigue driving monitoring model, so that the fatigue driving monitoring model outputs the result information of whether the driver is driving fatigued. In order to ensure the output accuracy of the fatigue driving monitoring model, a large number of driver driving image samples are required to train the fatigue driving monitoring model. At present, the driver's driving image samples are collected manually, so it is necessary for multiple people to participate in the sample construction work such as image collection, which has the problems of wasting manpower and low sample construction efficiency. Summary of the invention

[0004] The present application provides a driving video generation method, a fatigue driving monitoring model training method, a fatigue driving monitoring method, a vehicle control method and an electronic device, which are used to solve the problems existing in the prior art of sample construction work such as multiple people participating in image acquisition, wasting manpower and low sample construction efficiency.

[0005] Among them, the driver's facial expression change information and the driver's body area position change information are obtained according to the driver's body area corresponding to multiple frame images of the first driving video, the driver's body area depth information and the driver's body area angle information, so as to generate multiple target driving videos according to the target frame image of the second driving video and the driver's facial expression change information and the driver's body area position change information. In this way, multiple driver driving videos can be generated according to the combination of the captured frame images of the driver's driving video and the driver's facial expression change information and the driver's body area position change information in other driving videos. Compared with the method of manually collecting each driver's driving video in the prior art, a lot of manpower is saved and the efficiency of generating driver driving videos is accelerated.

[0006] To solve the above technical problems, in the first aspect, the implementation method of the present application discloses a driving video generation method, the method comprising: determining the driver's body area, the driver's body area depth information and the driver's body area angle information respectively corresponding to multiple frame images included in a first driving video; obtaining the driver's facial expression state change information based on the driver's body area corresponding to the multiple frame images, and obtaining the driver's body area position change information based on the driver's body area depth information and the driver's body area angle information corresponding to the multiple frame images; determining the target frame image included in the second driving video, and generating multiple target driving videos based on the target frame image, the driver's facial expression state change information and the driver's body area position change information.

[0007] The driving video generation method provided by the implementation of the present application determines the driver's body area, the driver's body area depth information, and the driver's body area angle information corresponding to multiple frames of images in the first driving video, obtains the driver's facial expression change information according to the driver's body area corresponding to each image, and obtains the driver's body area position change information according to the driver's body area depth information and the driver's body area angle information corresponding to each image; and then generates multiple target driving videos according to the target frame image of the second driving video, the driver's facial expression change information, and the driver's body area position change information. In this way, multiple driving videos can be generated based on the combination of the driver's facial expression change information and the driver's body area position change information corresponding to the target frame image of the collected driving video and the multiple frames of images of other driving videos. Compared with the method of manually collecting each driving video in the prior art, a lot of manpower is saved and the generation efficiency of driving videos is accelerated.

[0008] According to another specific implementation of the present application, the driving video generation method disclosed in the implementation of the present application determines the driver's body area, the driver's body area depth information and the driver's body area angle information respectively corresponding to multiple frames of images included in the first driving video, including: performing image segmentation processing on the first driving video based on a portrait segmentation model to obtain multiple frames of images, performing image segmentation processing on each image to obtain the driver's body area corresponding to each image; and determining the driver's body area depth information corresponding to each image of the image acquisition camera, and obtaining the driver's head posture information corresponding to each image as the driver's body area angle information based on the driver's head posture estimation model.

[0009] By adopting the above technical solution, the driver's body area and the driver's body area angle information corresponding to each image are obtained based on the portrait segmentation model and the driver's head posture estimation model; and the driver's body area depth information corresponding to each image is obtained. In this way, the driver's facial expression state change information is obtained based on the driver's body area, and the driver's body area position change information is obtained based on the driver's body area depth information and the driver's body area angle information. Driving videos based on different driver's facial expression state change information and driver's body area position change information can be generated, without the need to manually collect multiple driving videos, making the driver's driving video generation more efficient.

[0010] According to another specific implementation of the present application, the driving video generation method disclosed in the implementation of the present application obtains the driver's facial expression state change information based on the driver's body area corresponding to multiple frames of images, including: performing feature encoding processing on the driver's body area corresponding to each image based on an encoder to obtain a feature group of the driver's facial expression state change; sorting the driver's facial expression state change feature group according to a time series to obtain the driver's facial expression state change information.

[0011] By adopting the above technical solution, encoding is performed based on the driver's body area, and the information on the change in the driver's facial expression is obtained by sorting in time sequence. Driving videos based on the change in the facial expression of different drivers can be generated without the need for manual collection of multiple driving videos, making the generation of driver driving videos more efficient.

[0012] According to another specific implementation of the present application, the driving video generation method disclosed in the implementation of the present application obtains the driver's body area position change information based on the driver's body area depth information and the driver's body area angle information corresponding to multiple frames of images, including: sorting the driver's body area depth information and the driver's body area angle information corresponding to each image in time series to obtain a driver's body area change feature group as the driver's body area position change information corresponding to the driver; converting the driver's body area position change information corresponding to the driver to obtain the driver's body area position change information corresponding to the image acquisition camera.

[0013] By adopting the above technical scheme, time sequence sorting is performed based on the depth information and the angle information of the driver's body area, and conversion processing is performed to obtain the position change information of the driver's body area corresponding to the image acquisition camera. Driving videos based on the position change information of the body areas of different drivers can be generated without the need for manual collection of multiple driving videos, making the generation of driver driving videos more efficient.

[0014] According to another specific implementation of the present application, the driving video generation method disclosed in the implementation of the present application determines the target frame image included in the second driving video, and generates multiple target driving videos based on the target frame image, information on changes in the driver's facial expression state, and information on changes in the driver's body area position, including: determining an image corresponding to any frame of the second driving video as the target frame image; inputting the target frame image, information on changes in the driver's facial expression state, and information on changes in the driver's body area position into a driving video generation model to obtain multiple target driving videos.

[0015] By adopting the above technical solution, multiple target driving videos are obtained based on the driving video generation model according to any target frame image of the second driving video, the driver's facial expression change information and the driver's body area position change information, so that the generated driver driving image can better conform to the driver's driving logic, and is more realistic and effective.

[0016] According to another specific implementation of the present application, the implementation of the present application discloses a driving video generation method, wherein a driving video generation model includes an encoder, an image identifier extractor, and a decoder, and a target frame image, information on changes in the driver's facial expression state, and information on changes in the position of the driver's body area are input into the driving video generation model to obtain multiple target driving videos, including: determining text identifier information corresponding to the target frame image, inputting the target frame image and the text identifier information into the encoder for encoding processing to obtain image encoding features; inputting the image encoding features, information on changes in the driver's facial expression state, and information on changes in the position of the driver's body area into the image identifier extractor for feature fusion processing to obtain fused image features; and inputting the fused image features into the decoder for decoding processing to obtain multiple target driving videos.

[0017] By adopting the above technical solution, multiple target video images are generated by combining and splicing the driver's facial expression change information and the driver's body area position change information with any target frame images of other driving videos, so that the generated driver driving image is more in line with the driver's driving logic, more realistic and effective.

[0018] According to another specific implementation of the present application, in the driving video generation method disclosed in the implementation of the present application, the driving video generation model is a stable diffusion model.

[0019] In the second aspect, the implementation of the present application also discloses a fatigue driving monitoring model training method, the method comprising: determining a training data set, the training data set comprising multiple sample videos, the sample videos comprising a target driving video generated by a driving video generation method provided based on any one of the implementations of the first aspect above; performing model training on an initial fatigue driving monitoring model according to the training data set to obtain a fatigue driving monitoring model.

[0020] The fatigue driving monitoring model training method provided in the implementation of the present application obtains a target driver's driving video to train the initial fatigue driving monitoring model, thereby obtaining a fatigue driving monitoring model. In this way, a large number of driver driving videos are used for model training, so that the generated fatigue driving monitoring model can be more accurate in driver fatigue monitoring.

[0021] On the third aspect, the implementation method of the present application also discloses a fatigue driving monitoring method, which includes: obtaining a driving video of the driver; inputting the driver's driving video into a fatigue driving monitoring model to obtain the driver's fatigue monitoring results, and the fatigue driving monitoring model is obtained by model training based on the fatigue driving monitoring model training method provided in the second aspect above.

[0022] The fatigue driving monitoring method provided by the implementation of the present application obtains the driver's driving video and inputs the driver's driving video into the fatigue driving monitoring model to obtain the driver's fatigue monitoring result. In this way, the fatigue driving monitoring model trained with a large number of driver driving videos is used to make the obtained fatigue monitoring result more accurate.

[0023] In a fourth aspect, the implementation method of the present application also discloses a vehicle control method, which includes: obtaining a driver's fatigue driving monitoring result, the fatigue driving monitoring result is obtained based on the fatigue driving monitoring method provided in the third aspect above; and performing corresponding control according to the fatigue driving monitoring result.

[0024] The vehicle control method provided by the implementation of the present application obtains the driver's fatigue driving monitoring results and controls the vehicle according to the fatigue driving monitoring results. In this way, a fatigue driving monitoring model trained with a large number of driver driving videos is used to obtain more accurate fatigue driving monitoring results, and then corresponding vehicle control is performed according to the fatigue driving monitoring results to improve user driving safety.

[0025] In the fifth aspect, the implementation method of the present application also discloses an electronic device, including: a processor, and a memory connected to the processor in communication, the processor including a central processing unit and a graphics processing unit; the memory stores a computer program, and the processor executes the computer program stored in the memory, so that the electronic device implements at least one of the driving video generation method provided in any one of the implementation methods of the first aspect above, the fatigue driving monitoring model training method provided in the second aspect, the fatigue driving monitoring method provided in the third aspect, and the vehicle control method provided in the fourth aspect.

[0026] In a sixth aspect, the implementation method of the present application further discloses a vehicle for implementing the fatigue driving monitoring method provided in the third aspect above, and / or implementing the vehicle control method provided in the fourth aspect above.

[0027] In the seventh aspect, the implementation method of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement at least one of the driving video generation method provided in any one of the implementation methods of the first aspect, the fatigue driving monitoring model training method provided in the second aspect, the fatigue driving monitoring method provided in the third aspect, and the vehicle control method provided in the fourth aspect.

[0028] In an eighth aspect, an implementation method of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements at least one of the driving video generation method provided in any one of the implementation methods of the first aspect, the fatigue driving monitoring model training method provided in the second aspect, the fatigue driving monitoring method provided in the third aspect, and the vehicle control method provided in the fourth aspect.

[0029] It can be understood that the beneficial effects of the fifth to eighth aspects mentioned above can also be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flowchart of a driving video generation method provided in an embodiment of the present application;

[0031] Figure 2 is a schematic diagram of a driver's body area position change information provided by an embodiment of the present application;

[0032] Figure 3 It is a schematic diagram of a principle of generating multiple target driving videos provided by an embodiment of the present application;

[0033] Figure 4 It is another schematic diagram of the principle of generating multiple target driving videos provided by an embodiment of the present application;

[0034] Figure 5 It is a flowchart of another driving video generation method provided in an embodiment of the present application;

[0035] Figure 6 It is a flowchart of a fatigue driving monitoring model training method provided in an embodiment of the present application;

[0036] Figure 7 It is a flowchart of a fatigue driving monitoring method provided in an embodiment of the present application;

[0037] Figure 8 It is a flow chart of a vehicle control method provided in an embodiment of the present application;

[0038] Fig. 9It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] As mentioned above, fatigue driving monitoring is a very important monitoring technology related to vehicle driving safety in the automotive field. By monitoring the driver's fatigue driving, traffic accidents can be effectively avoided.

[0040] In the prior art, by acquiring a driver's driving image such as driver image data or video data, the driver's driving image is input into a fatigue driving monitoring model, so that the fatigue driving monitoring model outputs the result information of whether the driver is driving fatigued. In order to ensure the output accuracy of the fatigue driving monitoring model, a large number of driver driving image samples are required to train the fatigue driving monitoring model. At present, driver driving image samples are all collected by manual collection of driver driving images, so the driver driving image requires multiple people to participate in sample construction work such as image collection, which has problems such as wasting manpower and low sample construction efficiency.

[0041] For example, with the widespread application of intelligent algorithms (such as AI algorithms) in automobile smart cockpits, the use of AI algorithms (as an example of fatigue driving monitoring models) to monitor driver fatigue is becoming more and more popular. Driver fatigue monitoring is one of the key algorithms, and driver fatigue monitoring is mainly based on driver driving image or driver driving video recognition. However, the large amount of driving data required for algorithm training is difficult to collect and annotate effectively and in large quantities. Therefore, a fast and effective driving video generation method is needed.

[0042] Based on this, the present application proposes a driving video generation method, which obtains the driver's facial expression state change information and the driver's body area position change information according to the driver's body area, the driver's body area depth information and the driver's body area angle information corresponding to multiple frame images of a first driving video, so as to generate multiple target driving videos according to the target frame image of a second driving video and the driver's facial expression state change information and the driver's body area position change information. In this way, multiple different driver driving videos can be generated based on the captured frame images of the driver's driving video and the combination of the driver's facial expression state change information and the driver's body area position change information in other driving videos. Compared with the method of manually collecting each driver's driving video in the prior art, a lot of manpower is saved and the efficiency of generating driver driving videos is accelerated.

[0043] The driving video generation method provided in the implementation of the present application is applied to a first electronic device, such as Figure 1 As shown, the method includes the following steps.

[0044] S110, determining the driver's body area, the driver's body area depth information, and the driver's body area angle information respectively corresponding to the multiple frames of images included in the first driving video.

[0045] S120, obtaining information on changes in the driver's facial expression based on the driver's body area corresponding to the multiple frames of images, and obtaining information on changes in the driver's body area position based on the driver's body area depth information and the driver's body area angle information corresponding to the multiple frames of images.

[0046] S130, determining a target frame image included in the second driving video, and generating a plurality of target driving videos according to the target frame image, information on changes in the driver's facial expression, and information on changes in the position of a region of the driver's body.

[0047] For step S110, determining the driver's body area, the driver's body area depth information and the driver's body area angle information respectively corresponding to the multiple frames of images included in the first driving video, including: performing image segmentation processing on the first driving video based on a portrait segmentation model to obtain multiple frames of images, performing image segmentation processing on each image to obtain the driver's body area corresponding to each image; and determining the driver's body area depth information corresponding to each image of the image acquisition camera, and obtaining the driver's head posture information corresponding to each image based on the driver's head posture estimation model as the driver's body area angle information.

[0048] Exemplarily, a video of a driver driving in fatigue (as an example of the first driving video) is collected through a binocular camera in the car, and a portrait segmentation and head posture estimation neural network model (as an example of a portrait segmentation model) is deployed to perform image segmentation processing on each frame of the collected driving video to obtain multiple frames of images, and then perform image segmentation and recognition processing on the driver's body area in each frame of the image to obtain the driver's body area corresponding to each frame of the image. At the same time, the depth information of the driver's area in each frame of the image is obtained through a calibrated binocular camera (as an example of an image acquisition camera) (that is, the depth information of the driver's body area), and the head posture of the driver in each frame of the image (that is, the driver's head posture information) is obtained through a head posture estimation model to obtain the pitch angle change information of the driver's head (that is, the angle change in the pitch direction, that is, the angle information of the driver's body area).

[0049] Furthermore, in the implementation of the present application, a short video of the driver's fatigue driving is collected by the in-car binocular camera as the first driving video. That is, the first driving video can be a video of the driver's fatigue driving, and of course it can also be a driving video of the driver when the driver is driving normally. In this way, in the implementation of the present application, the generated target driving video can be a driving video that represents the driver in a fatigue driving state, or it can be a driving video that represents the driver is not in a fatigue driving state. The implementation of the present application specifically takes the fatigue driving video as an example to illustrate the method for generating the driver's driving video.

[0050] Furthermore, in step S120, information on changes in the driver's facial expression state is obtained based on the driver's body areas corresponding to multiple frames of images, including: performing feature encoding processing on the driver's body areas corresponding to each image based on an encoder to obtain a feature group of changes in the driver's facial expression state; and sorting the feature group of changes in the driver's facial expression state according to a time series to obtain information on changes in the driver's facial expression state.

[0051] Exemplarily, the human body area corresponding to each frame image segmented from the first fatigue driving video is encoded with an AE autoencoder (as an example of an encoder) to obtain a driver's overall facial expression change feature group, and the driver's facial expression change feature group is sorted and saved according to the time sequence of the frame images (i.e., time series) to obtain the driver's facial expression change information, which is recorded as T1, T2, ..., Tn. That is, the driver's facial expression change information is the driver's facial expression change feature during driving in the first fatigue driving video, such as the change process from having eyes full of spirit to yawning to squinting eyes to closing eyes tightly to opening eyes slightly.

[0052] Furthermore, in the implementation method of the present application, for step S120, the driver's body area position change information is obtained based on the driver's body area depth information and the driver's body area angle information corresponding to multiple frames of images, including: sorting the driver's body area depth information and the driver's body area angle information corresponding to each image according to a time series to obtain a driver's body area change feature group; converting the driver's body area position change information corresponding to the driver to obtain the driver's body area position change information corresponding to the image acquisition camera.

[0053] Exemplarily, the change of the human body region in the depth direction (that is, the depth information of the driver's human body region) and the change of the human body region angle at the pitch angle (that is, the driver's human body region angle change information) in each frame of the first driving video are saved in a time series, recorded as D1, D2, ..., Dn (as an example of the driver's human body region change feature group). Because of the restraint of the seat belt during driving and the forward tilt of the head and body during fatigue driving, there is a lack of changes in other directions. Therefore, the change information of the driver's human body region in the depth direction and the change of the human body region at the pitch angle can be directly used to approximate the change of the driver in space during fatigue driving (that is, the driver's corresponding driver's human body region position change information). Further, the driver's corresponding driver's human body region position change information containing the driver's human body region angle change information and the driver's human body region depth change information is converted into human body region virtual camera trajectory change information, that is, C1, C2, ..., Cn (as an example of the driver's human body region position change information corresponding to the image acquisition camera).

[0054] For example, Figure 2 As shown in the figure, when the driver looks straight ahead, his pitch angle is 0°, and the angle of the virtual camera in the pitch direction is also 0°. When the driver's pitch angle changes by an angle γ, it can be equivalently converted into a virtual camera rotation by γ in the pitch direction, while the virtual camera is still facing the face. In this way, the changes in the position space of the human body area in the car during different fatigue driving processes can be preserved.

[0055] It should be noted that in the implementation method of the present application, the driver's body area depth information can be calculated based on the binocular camera using phase information to obtain the body area depth information. The driver's body area depth information reflects the distance of the body area relative to the camera plane, and the driver's body area angle information is estimated by a preset head posture estimation model. The driver's body area angle information is the deflection angle of the driver's head. The generated driver's body area change feature group (that is, the driver's body area position change information corresponding to the driver) is the coordinate information of the driver's head relative to the seat corresponding to each frame of the image, and the driver's body area position change information corresponding to the converted image acquisition camera is the coordinate information corresponding to the virtual camera obtained after performing three-dimensional coordinate conversion on the coordinate information of the driver's head relative to the seat.

[0056] In step S130, the target frame image included in the second driving video is determined, and multiple target driving videos are generated according to the target frame image, information on changes in the driver's facial expression and information on changes in the position of the driver's body area, including: determining an image corresponding to any frame of the second driving video as the target frame image; inputting the target frame image, information on changes in the driver's facial expression and information on changes in the position of the driver's body area into a driving video generation model to obtain multiple target driving videos.

[0057] Exemplarily, each driver corresponds to an ID number. In one implementation of the present application, the first driving video and the second driving video may be multiple different driving videos corresponding to the driver with the same ID. For the first driving video of a driver corresponding to a certain ID, each frame of the image is processed to obtain information on changes in the driver's facial expression and the position of the driver's body area. For the second driving video of the driver corresponding to the ID, any frame of the video image is intercepted as the target frame image to serve as the initial frame of the target driving video to be output.

[0058] Among them, any frame of video image (ie, target frame image) may be an initial frame image of the second driving video, or may be a previous frame image of the second driving video relative to each frame image of the first driving video.

[0059] Furthermore, in another implementation of the present application, the first driving video and the second driving video may be driving videos corresponding to drivers with different IDs. For the first driving video of the driver with ID 1, each frame of the image is processed to obtain information on changes in the driver's facial expression and position of the driver's body region. For the second driving video of the driver with ID 2, any frame of the video image is intercepted as the target frame image to serve as the initial frame of the target driving video to be output.

[0060] Furthermore, in the implementation of the present application, the driving video generation model is a stable diffusion model (also known as a stable diffusion model). The purpose of the stable diffusion model is to eliminate the continuous application of Gaussian noise to the training image, which can be regarded as a series of denoising autoencoders to obtain a clear image. The stable diffusion model consists of three parts: a variational autoencoder (VAE, Variational Autoencoder, also known as a VAE encoder) (as an example of an encoder), U-Net (also known as UNET) (as an example of an image identifier extractor) and a decoder (also known as a VAE decoder) (as an example of a decoder).

[0061] Therefore, in the implementation method of the present application, the driving video generation model includes an encoder, an image identifier extractor and a decoder, and the target frame image, the driver's facial expression state change information and the driver's body area position change information are input into the driving video generation model to obtain multiple target driving videos, including: determining the text identifier information corresponding to the target frame image, inputting the target frame image and the text identifier information into the encoder for encoding processing to obtain image encoding features; inputting the image encoding features, the driver's facial expression state change information and the driver's body area position change information into the image identifier extractor for feature fusion processing to obtain fused image features; inputting the fused image features into the decoder for decoding processing to obtain multiple target driving videos.

[0062] For example, Figure 3 As shown, the previous frame image of the frame image corresponding to any frame of the first driving video in the second driving video of a certain ID collected is used as the target frame image, and the target frame image is used as the initial input frame (that is, the initial frame image) of the target driving video to be generated, and the same specific text prompt word (that is, text) is added, for example, the text prompt word is "the driver is driving fatigued", and after VAE automatic encoding (that is, VAE encoding), the image coding features are generated, and then the image coding features, the features extracted from the encoded overall fatigue expression change (that is, the driver's expression state change information T1, T2, ..., Tn) and the fitted virtual camera trajectory change (that is, the driver's body area position change information C1, C2, ..., Cn) are inserted into the attention mechanism calculation layer of the stable diffusion model (that is, the image identifier extractor (UNET)) to obtain the fused image features. Among them, the driver's expression state change information T1, T2, ..., Tn and the driver's body area position change information C1, C2, ..., Cn are first encoded based on the encoding formula, and the encoding formula is

[0063] w k =AE(C k +T k )

[0064] Where AE is the encoder function, C k is the position change information of the driver’s body area, T k The driver's facial expression change information is used to obtain the coding features, and then the coding features are used to generate video frames. The generation formula of the video frame X is:

[0065]

[0066] in, is the initial input frame, that is, the target frame image, Δt is the time interval between the target frame image of the second driving video and the next frame image of the first driving video, DDIM_Backward is the denoising function, and SD is the stable diffusion model.

[0067] The DDIM_Backward denoising function can perform specific algorithm processing, starting from an initial frame image, and gradually generating subsequent frame images based on time intervals, thereby achieving the generation of a video or image sequence.

[0068] For example, Figure 3 As shown, the previous frame image 1 (as an example of the target frame image) and text corresponding to the frame image 1 of the first driving video corresponding to the second driving video are input, the previous frame image 2 (as another example of the target frame image) and text corresponding to the frame image 2 of the first driving video corresponding to the second driving video are input, and the previous frame image 3 (as an example of the target frame image) and text corresponding to the frame image 3 of the first driving video corresponding to the second driving video are input to VAE encoding (i.e., encoder) to obtain multiple image coding features, and the image coding features, the driver's facial expression change information, and the driver's body area position change information are sent to UNET (i.e., image identifier extractor) to obtain the inter-frame feature w2 (i.e., W2) between the 1st frame and the 2nd frame (as an example of fused image features), the inter-frame feature w3 (i.e., W3) between the 2nd frame and the 3rd frame (as another example of fused image features), ..., the inter-frame feature w of the n-1th frame and the nth frame n (ie, Wn) (as another example of fusing image features).

[0069] In this way, each frame image of the target driving video is generated based on the previous frame image of the second driving video in the time sequence combined with the driver's facial expression state change information and the driver's body area position change information corresponding to the next frame image of the first driving video in the time sequence, rather than random noise, making the generated target driving video more realistic.

[0070] Furthermore, the fused image features are input into the decoder (i.e., VAE decoding) to obtain multiple frames of fatigue driving images (for example, the generated second frame of fatigue driving image, the generated third frame of fatigue driving image, ..., the nth frame of fatigue driving image), and the multiple frames of fatigue driving images are combined and spliced ​​to generate the target driving video (i.e., the generated fatigue driving video).

[0071] Furthermore, if Figure 4As shown, the first frame image 1 of the second driving video (as an example of the target frame image) and the text are input to the VAE code (i.e., the encoder) to obtain multiple image coding features, and the image coding features, the driver's facial expression change information, and the driver's body area position change information are input to the UNET (i.e., the image identifier extractor) to obtain the inter-frame feature w2 (i.e., W2) between the first frame and the second frame (as an example of fused image features), the inter-frame feature w3 (i.e., W3) between the first frame and the third frame (as another example of fused image features), ..., the inter-frame feature w1 between the first frame and the nth frame n (ie, Wn) (as another example of fusing image features).

[0072] In this way, each frame image of the target driving video is generated based on the first frame image of the second driving video combined with the driver's facial expression change information and the driver's body area position change information corresponding to each frame image of the first driving video in time sequence, rather than random noise, making the generated target driving video more realistic.

[0073] Furthermore, the fused image features are input into the decoder (i.e., VAE decoding) to obtain multiple frames of fatigue driving images (for example, the generated second frame of fatigue driving image, the generated third frame of fatigue driving image, ..., the nth frame of fatigue driving image), and the multiple frames of fatigue driving images are combined and spliced ​​to generate the target driving video (i.e., the generated fatigue driving video).

[0074] It should be noted that, in the implementation of the present application, generating multiple target driving videos based on the target frame image, the information on the change in the driver's facial expression, and the information on the change in the position of the driver's body region is to sequentially perform a facial expression change combination and a position change combination on the first frame image of the second driving video (i.e., the target frame image) and each frame image of the first driving video, so that a target driving video can be generated in which the driver is in a different facial expression and position than each frame image of the second driving video. Alternatively, each frame image of the second driving video (i.e., the target frame image) and the previous frame image of the first driving video corresponding to each frame image of the second driving video are sequentially performed a facial expression change combination and a position change combination on the first driving video, so that a target driving video can be generated in which the driver is in a different facial expression and position than each frame image of the second driving video.

[0075] Thus, in the implementation method of the present application, by extracting the driver's overall facial expression change characteristics (that is, the driver's facial expression change information) and the camera trajectory change characteristics (that is, the driver's body area position change information) in the first driving video of drivers corresponding to different IDs, and combining any target frame image of a driver with any ID (same ID or different IDs), a driver fatigue driving video is generated, so that the driver's facial expression changes and position changes during driving are combined with frame images of other driving videos, so that driving videos of multiple drivers with different expressions and different driving positions can be generated.

[0076] In this way, by using the driving video data collection of drivers with different IDs, the information on the changes in the driver's facial expression and the changes in the position of the driver's body area in the driving video of each driver with each ID is extracted, and then combined with other driving videos, and different frame images are selected as target frame images. Each driving video can generate a large number of different driving videos that meet the characteristics of fatigue driving. Compared with traditional driver fatigue data collection or generation methods, the driving video generation method provided in this application fully considers the fatigue driving monitoring model's demand for the amount of fatigue driving data and the demand for the availability of driving videos. By combining the spatial changes in the driver's body posture during driving fatigue and the changes in the driver's overall facial expression, the generative model stable diffusion is controlled to effectively generate batches of driver fatigue image data or video data to obtain multiple target driving videos.

[0077] like Figure 5 As shown, the driver driving video generation method provided by the implementation of the present application, taking the generated driver driving video as a driver fatigue driving video as an example, specifically includes the following steps.

[0078] S10, collect a video of the driver's fatigue driving (i.e., the first driving video) through the binocular camera in the car, and deploy a portrait segmentation and head posture estimation neural network model (i.e., the portrait segmentation model) on the car computer, perform image segmentation processing on each frame of the collected driving video image to obtain multiple frames of images, and use the portrait segmentation algorithm to identify the driver's body area in each frame of the image (i.e., image segmentation and recognition processing) to obtain the driver's body area corresponding to each frame of the image, and at the same time obtain the depth information of the driver area in each frame of the image (i.e., the driver's body area depth information) through a calibrated binocular camera (i.e., the image acquisition camera), obtain the driver's head posture in each frame of the image (i.e., the driver's head posture information) through the head posture estimation model, and record its angle change in the pitch direction (i.e., obtain the pitch angle change information of the driver's head, i.e., the driver's body area angle information).

[0079] S20, the human body area segmented from the collected fatigue driving video of a certain ID (i.e., the first driving video) is encoded with the AE autoencoder to obtain the driver's overall expression state change feature group of the driver's expression state change, and is saved in time sequence, recorded as T1, T2, ..., Tn (i.e., the driver's expression change information). In this way, the overall expression change state of the driver in the driving videos with different IDs during the fatigue driving process can be saved. The changes in the depth direction (i.e., the driver's human body area depth information) and the pitch angle (i.e., the driver's human body area angle change information) of the human body area in each frame image of the first driving video are saved in time sequence, recorded as D1, D2, ..., Dn (i.e., the driver's human body area change feature group), and further converted into the human body area virtual camera trajectory change information, i.e., C1, C2, ..., Cn (i.e., the driver's human body area position change information). In this way, the changes in the human body position space of the driver in the driving videos with different IDs during the fatigue driving process can be saved.

[0080] S30, taking a certain frame (i.e., target frame image) of the collected fatigue driving data of a certain ID (i.e., the second driving video) as the initial input frame, inserting the encoded fatigue time series features T1, T2, ..., Tn (i.e., information on changes in the driver's facial expression) and the camera trajectory changes C1, C2, ..., Cn (i.e., information on changes in the position of the driver's body area) into the attention mechanism calculation layer (i.e., image identifier extractor (UNET)) of the stable diffusion model in time sequence, and then each frame image in the driving video generated is changed on the basis of the previous frame image or each frame image is changed on the basis of the first frame image instead of being generated by random noise, so that fatigue video data (i.e., target driving video) corresponding to a certain ID can be generated using the fatigue overall facial expression change features and camera trajectory changes of different IDs.

[0081] The driver driving video generation method provided by the implementation of the present application is actually a low-cost and high-diversity driver fatigue data generation method, that is, a driver fatigue driving video data generation method that integrates the driver's expression and posture characteristics. A short segment of driver fatigue driving data of drivers with different IDs is collected through an in-vehicle binocular camera (that is, a binocular camera) to obtain a first driving video, and the human body areas of all frames in the first driving video are segmented and extracted to obtain the driver's human body area. The depth information (that is, the driver's human body area depth information) and angle information (that is, the driver's human body area angle information) of the human body area extracted from each frame image are saved to obtain the driver's corresponding driver body area position change information, and the driver's human body area position change information is further processed into a virtual camera posture (that is, a virtual The driver's body area position change information corresponding to the virtual camera (image acquisition camera) is obtained, and the overall expression state change characteristics of the human body area corresponding to each frame image are extracted to obtain the driver's expression state change information. Based on the stable diffusion model, a driving video of a driver with an id (that is, the second driving video) is selected as an initial input frame image as the target frame image, and the expression action state encoding (that is, the driver's expression state change information) and the virtual camera posture (that is, the driver's body area position change information) of other driving videos (that is, the first driving video) of drivers with other ids (same id or different ids) are inserted into the attention mechanism calculation layer (that is, UNET), thereby generating a large amount of driver fatigue video data (that is, target driving video) close to actual driving behavior. In this way, by collecting short segments of driver fatigue driving or simulated fatigue driving videos of multiple different drivers, the angle information and expression change characteristics of each driver's body area and the driver in the driving video are extracted, so that the extracted different fatigue driving features can be expanded with different starting frames of any fatigue driving video (that is, the second driving video) to generate a large amount of fatigue driving video data.

[0082] Furthermore, the implementation of the present application also discloses a fatigue driving monitoring model training method, which is applied to a second electronic device, such as Figure 6 As shown, the specific steps include the following steps.

[0083] S210, determining a training data set, where the training data set includes a plurality of sample videos.

[0084] The sample video includes a target driving video generated by the driving video generation method provided in the above implementation manner. The target driving video generated by the driving video generation method provided in the above implementation manner can refer to the specific process of the above-mentioned steps S100 to S150.

[0085] S220: Perform model training on the initial fatigue driving monitoring model according to the training data set to obtain a fatigue driving monitoring model.

[0086] The fatigue driving monitoring model training method provided in the implementation of the present application obtains the target driving videos provided by the above implementation method and the fatigue driving monitoring results corresponding to each target driving video (for example, being in a fatigue driving state or not in a fatigue driving state) to perform model training on the initial fatigue driving monitoring model to obtain a fatigue driving monitoring model. In this way, a large number of driver driving videos are used to train the model, so that the generated fatigue driving monitoring model can be more accurate when performing driver fatigue monitoring.

[0087] Furthermore, the implementation of the present application also discloses a fatigue driving monitoring method, which is applied to electronic equipment or vehicles, such as Figure 7 As shown, the specific steps include the following steps.

[0088] S310, obtaining a driving video of the driver.

[0089] S320: Input the driver's driving video into a fatigue driving monitoring model to obtain a driver's fatigue driving monitoring result.

[0090] The fatigue driving monitoring model is obtained by model training based on the fatigue driving monitoring model training method provided in the above implementation.

[0091] The fatigue driving monitoring method provided by the implementation of the present application obtains the driver's driving video and inputs the driver's driving video into the fatigue driving monitoring model to obtain the driver's fatigue driving monitoring result. In this way, the fatigue driving monitoring model trained with a large number of driver driving videos is used to make the obtained fatigue driving monitoring result more accurate.

[0092] Furthermore, the implementation of the present application also discloses a vehicle control method, which is applied to an electronic device or a vehicle, such as Figure 8 As shown, the specific steps include the following steps.

[0093] S410, obtaining the driver's fatigue driving monitoring result.

[0094] The fatigue driving monitoring result is obtained based on the fatigue driving monitoring method provided by the above implementation.

[0095] S420, controlling the vehicle according to the fatigue driving monitoring result.

[0096] The vehicle control method provided by the implementation of the present application obtains the driver's fatigue driving monitoring results and performs corresponding control according to the fatigue driving monitoring results. In this way, a fatigue driving monitoring model trained with a large number of driver driving videos is used to obtain more accurate fatigue driving monitoring results, and then corresponding control is performed according to the fatigue driving monitoring results to improve the user's driving safety.

[0097] Among them, corresponding vehicle control based on fatigue driving monitoring results can specifically include decelerating the vehicle until the vehicle stops in a safe area, or stimulating the driver to keep the driver awake, etc.

[0098] In the implementation of the present application, the first electronic device and the second electronic device can be the same electronic device or different electronic devices, specifically, they can be electronic devices such as computers, mobile phones, remote computers, vehicles, and of course, cloud servers.

[0099] See also Fig. 9 , Fig. 9 The figure shows a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig. 9 As shown, the electronic device may include: a transceiver 121 , a processor 122 , and a memory 123 .

[0100] The processor 122 executes the computer execution instructions stored in the memory, so that the processor 122 executes the technical solution of at least one of the driving video generation method, fatigue driving monitoring model training method, fatigue driving monitoring method, and vehicle control method provided in the above embodiments. The processor 122 can be a general-purpose processor, including a central processing unit CPU, a network processor (NP), etc.; it can also be a digital data processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components.

[0101] The memory 123 is connected to the processor 122 via a system bus and completes communication between them. The memory 123 is used to store computer program instructions.

[0102] By way of example and not limitation, the memory 123 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 123 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 123 may be inside or outside the integrated gateway device. In a particular embodiment, the memory 123 is a non-volatile solid-state memory. In a particular embodiment, the memory 123 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0103] The transceiver 121 may be used to obtain tasks to be executed and configuration information of the tasks to be executed.

[0104] The system bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The transceiver is used to realize the communication between the database access device and other computers (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM) and may also include non-volatile memory.

[0105] The implementation of the present application also provides a vehicle, which is used to implement the technical solution of at least one of the driving video generation method, fatigue driving monitoring model training method, fatigue driving monitoring method, and vehicle control method provided in the above embodiments. Or it is used to implement the fatigue driving monitoring method and / or vehicle control method provided in the above embodiments.

[0106] An embodiment of the present application also provides a chip for running instructions, which is used to execute the technical solutions of at least one of the driving video generation method, fatigue driving monitoring model training method, fatigue driving monitoring method, and vehicle control method provided in the above embodiments.

[0107] An embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a processor of an electronic device, the processor of the electronic device executes a technical solution of at least one of the driving video generation method, fatigue driving monitoring model training method, fatigue driving monitoring method, and vehicle control method provided in the above embodiments.

[0108] In some possible implementations, various aspects of the method provided in the present application can also be implemented in the form of a program product, which includes program code. When the program product is run on a processor of an electronic device, the program code is used to enable the processor of the electronic device to execute the steps of the method described above in this specification according to various exemplary implementations of the present application. For example, the electronic device can execute the technical solution of at least one of the driving video generation method, fatigue driving monitoring model training method, fatigue driving monitoring method, and vehicle control method provided in the embodiments of the present application.

[0109] The program product may employ any combination of one or more readable media. The readable medium may be a readable data medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0110] The implementation method of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, it can implement the technical solution of at least one of the driving video generation method, fatigue driving monitoring model training method, fatigue driving monitoring method, and vehicle control method provided in the above embodiments.

[0111] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable information processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable information processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0112] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable information processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0113] These computer program instructions can also be loaded onto a computer or other programmable information processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0114] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary technical means in the art that are not disclosed in the present application.

[0115] It should be noted that, in addition to the implementation of the present application described in the above-mentioned specific embodiments, those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Although the description of the present application is introduced in conjunction with the preferred embodiment, this does not mean that the features of this invention are limited to the implementation. On the contrary, the purpose of introducing the invention in conjunction with the implementation is to cover other options or modifications that may extend based on the claims of the present application. In order to provide an in-depth understanding of the present application, the above description contains many specific details, and the present application may also be implemented without using these details. In addition, in order to avoid confusion or blurring the focus of the present application, some specific details will be omitted in the description. It should be noted that, in the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0116] It should be noted that in this specification, similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0117] Although the present application has been illustrated and described by referring to certain preferred implementations of the present application, it should be understood by those skilled in the art that the above contents are further detailed descriptions of the present application in combination with specific implementations, and it cannot be determined that the specific implementation of the present application is limited to these descriptions. Those skilled in the art may make various changes in form and details, including making several simple deductions or substitutions, without departing from the spirit and scope of the present application.

Claims

1. A driving video generation method, characterized in that: The method comprises: Determine the driver's body area, the driver's body area depth information, and the driver's body area angle information respectively corresponding to the multiple frames of images included in the first driving video; Obtaining information about changes in the driver's facial expression based on the driver's body region corresponding to the multiple frames of images, and obtaining information about changes in the driver's body region position based on the depth information and angle information of the driver's body region corresponding to the multiple frames of images; A target frame image included in the second driving video is determined, and a plurality of target driving videos are generated according to the target frame image, the information on changes in the driver's facial expression, and the information on changes in the position of a region of the driver's body.

2. The driving video generation method according to claim 1, characterized in that: Determining the driver's body area, the driver's body area depth information, and the driver's body area angle information respectively corresponding to the multiple frames of images included in the first driving video, including: Performing image segmentation processing on the first driving video based on the portrait segmentation model to obtain multiple frames of images, performing image segmentation processing on each of the images to obtain the driver's body region corresponding to each of the images; and Determine the depth information of the driver's human body area corresponding to each of the images of the image acquisition camera, and obtain the driver's head posture information corresponding to each of the images based on the driver's head posture estimation model as the driver's human body area angle information.

3. The driving video generation method according to claim 2, characterized in that: Obtaining information on changes in the driver's facial expression state according to the driver's body region corresponding to the multiple frames of images includes: Based on the encoder, feature encoding is performed on the driver's body area corresponding to each of the images to obtain a feature group of changes in the driver's facial expression; The driver's facial expression state change feature group is sorted according to time series to obtain the driver's facial expression state change information.

4. The driving video generation method according to claim 2 or 3, characterized in that: According to the depth information of the driver's human body region and the angle information of the driver's human body region corresponding to the multiple frames of images, the position change information of the driver's human body region is obtained, including: Sorting the driver's body region depth information and the driver's body region angle information corresponding to each of the images in time series to obtain a driver's body region change feature group as the driver's body region position change information corresponding to the driver; The position change information of the driver's body area corresponding to the driver is converted and processed to obtain the position change information of the driver's body area corresponding to the image acquisition camera.

5. The driving video generation method according to any one of claims 1 to 4, characterized in that: Determining a target frame image included in the second driving video, and generating a plurality of target driving videos according to the target frame image, the information on changes in the driver's facial expression, and the information on changes in the position of the driver's body region, including: Determine any frame image in the second driving video as the target frame image; The target frame image, the driver's facial expression change information, and the driver's body area position change information are input into a driving video generation model to obtain a plurality of the target driving videos.

6. The driving video generation method according to claim 5, characterized in that: The driving video generation model includes an encoder, an image identifier extractor, and a decoder. The target frame image, the driver's facial expression change information, and the driver's body area position change information are input into the driving video generation model to obtain a plurality of the target driving videos, including: Determine text identification information corresponding to the target frame image, input the target frame image and the text identification information into the encoder for encoding processing, and obtain image encoding features; Inputting the image coding features, the information on changes in the driver's facial expression and the information on changes in the driver's body area position into the image identification extractor for feature fusion processing to obtain fused image features; The fused image features are input into the decoder for decoding processing to obtain a plurality of the target driving videos.

7. The driving video generation method according to claim 6, characterized in that: The driving video generation model is a stable diffusion model.

8. A fatigue driving monitoring model training method, characterized in that: The method comprises: Determine a training data set, wherein the training data set includes a plurality of sample videos, and the sample videos include a target driving video generated based on the driving video generation method according to any one of claims 1 to 7; The initial fatigue driving monitoring model is trained according to the training data set to obtain the fatigue driving monitoring model.

9. A method for monitoring fatigue driving, characterized in that: The method comprises: Obtain driving video of the driver; The driver video is input into a fatigue driving monitoring model to obtain a fatigue driving monitoring result of the driver, and the fatigue driving monitoring model is obtained by model training based on the fatigue driving monitoring model training method described in claim 8.

10. A vehicle control method, characterized in that: The method comprises: Obtaining a driver's fatigue driving monitoring result, wherein the fatigue driving monitoring result is obtained based on the driver fatigue driving monitoring method according to claim 9; The vehicle is controlled according to the fatigue driving monitoring result.

11. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor, wherein the processor includes a central processing unit and a graphics processing unit; The memory stores a computer program, and the processor executes the computer program stored in the memory so that the electronic device implements at least one of the driving video generation method described in any one of claims 1 to 7, the fatigue driving monitoring model training method described in claim 8, the fatigue driving monitoring method described in claim 9, and the vehicle control method described in claim 10.

Images