Driving image generation method, fatigue driving monitoring model training method and fatigue driving monitoring model monitoring method

Abstract

The invention discloses a driver driving image generation and fatigue driving monitoring model training and monitoring method, and the method comprises the steps: determining at least one in-vehicle image when a target vehicle is in a driver-free state; performing image segmentation processing on each in-vehicle image to obtain a driver position area; determining a driver movable range in the driver position area, determining a driver fatigue degree and a driver moving behavior habit, and generating a driver position coding matrix according to the driver movable range, the driver fatigue degree and the driver moving behavior habit; and determining a first driver driving image, and generating a plurality of second driver driving images according to the first driver driving image, the driver position coding matrix and the at least one in-vehicle image. A plurality of second driver driving images can be generated according to one first driver driving image, so that a large amount of manpower for acquiring the driver driving images is saved, and the generation efficiency is improved.

Description

Technical Field

[0001] The present application relates to the technical field of driver fatigue monitoring, and in particular to a driver driving image generation method, a fatigue driving monitoring model training method, a fatigue driving monitoring method, a vehicle control method, an electronic device and a vehicle. Background Art

[0002] Fatigue driving monitoring is a very important monitoring technology in the automotive field involving vehicle driving safety. By monitoring the driver's fatigue driving, traffic accidents can be effectively avoided.

[0003] In the prior art, by acquiring a driver's driving image such as driver image data or video data, the driver's driving image is input into a fatigue driving monitoring model, so that the fatigue driving monitoring model outputs the result information of whether the driver is driving fatigued. In order to ensure the output accuracy of the fatigue driving monitoring model, a large number of driver driving image samples are required to train the fatigue driving monitoring model. At present, driver driving image samples are all collected by manual collection of driver driving images, so the driver driving image requires multiple people to participate in sample construction work such as image collection, which has problems such as wasting manpower and low sample construction efficiency. Summary of the invention

[0004] The present application provides a driver driving image generation method, a fatigue driving monitoring model training method, a fatigue driving monitoring method, a vehicle control method, an electronic device and a vehicle, which are used to solve the problems existing in the prior art of sample construction work such as multiple people participating in image collection, wasting manpower and low sample construction efficiency.

[0005] Among them, multiple driver driving images of the driver in different driving areas in the car are generated according to the driver driving image, the driver position coding matrix of the driver in different movable ranges, different driver fatigue levels, and different driver movement behavior habits, and at least one in-car image of the target vehicle. In this way, only one driver driving image needs to be collected to generate multiple driver driving images in different states. Compared with the method of manually collecting each driver driving image in the prior art, a lot of manpower is saved and the efficiency of collecting driver driving images is accelerated.

[0006] To solve the above technical problems, in the first aspect, the implementation method of the present application discloses a method for generating a driver driving image, which method includes: determining at least one in-vehicle image when the target vehicle is in a driverless state; performing image segmentation processing on each in-vehicle image to obtain a driver position area; determining the driver's movable range in the driver position area, as well as determining the driver's fatigue level information and the driver's movement behavior habit information, and generating a driver position coding matrix according to the driver's movable range, the driver's fatigue level information and the driver's movement behavior habit information; determining a first driver driving image, and generating multiple second driver driving images according to the first driver driving image, the driver position coding matrix, and at least one in-vehicle image.

[0007] The driver driving image generation method provided by the implementation of the present application determines the driver driving image (i.e., the first driver driving image) and determines at least one in-vehicle image of the corresponding vehicle model when the vehicle is in a driverless state; and performs image segmentation processing on each in-vehicle image to obtain the driver position area. Then, the driver's movable range in the driver position area is determined, as well as the driver's fatigue level information and the driver's mobile behavior habit information are determined, and a driver position coding matrix is ​​generated according to the driver's movable range, the driver's fatigue level information, and the driver's mobile behavior habit information. Finally, multiple driver driving images (i.e., the second driver driving image) are generated according to the driver driving image, the driver position coding matrix when the driver is in different position states, and each in-vehicle image of the target vehicle. In this way, multiple driver driving images in different states can be generated based on one driver driving image, which saves a lot of manpower and speeds up the efficiency of driver driving image generation.

[0008] According to another specific implementation of the present application, the driver driving image generation method disclosed in the implementation of the present application determines the driver's movable range in the driver's position area, including: determining the angle between the bottom edge of the seat included in the driver's position area and the image edge of the in-vehicle image, and determining the target area in the driver's position area according to the angle as the driver's movable range.

[0009] By adopting the above technical solution, the movable range of the driver is determined, and then a driving image of any driver within the corresponding movable range of the driver is generated. In this way, multiple real and effective driving images of the driver can be generated.

[0010] According to another specific implementation of the present application, the driver driving image generation method disclosed in the implementation of the present application generates a position coding matrix based on the driver's movable range, driver fatigue level information and driver movement behavior habit information, including: inputting the driver's movable range, driver fatigue level information and driver movement behavior habit information into a conditional probability model to perform probability calculation processing on the driver appearing in the corresponding driver's movable range, obtain the corresponding probability, and generate a driver position coding matrix based on the probability.

[0011] By adopting the above technical scheme, for the interior images of the vehicle of the corresponding model when there is no driver and the vehicle seat is in different front and rear positions, the probability of the driver appearing in the driver's movable range corresponding to each interior image of the target vehicle can be generated based on the selected driver fatigue level information and driver movement behavior habit information, and then the driver position coding matrix corresponding to all interior images of the target vehicle is obtained. Driver driving images of any driver at different positions in different vehicles can be generated, and a sufficient number of driver driving images can be obtained, making the generation of driver driving images more efficient.

[0012] According to another specific implementation of the present application, the driver driving image generation method disclosed in the implementation of the present application generates multiple second driver driving images based on a first driver driving image, a driver position coding matrix, and at least one in-vehicle image, including: performing image segmentation processing on the first driver driving image to obtain the driver's body area; inputting the driver's body area, the driver position coding matrix, and at least one in-vehicle image into a driver driving image generation model to generate multiple second driver driving images.

[0013] According to another specific implementation of the present application, the implementation of the present application discloses a driver driving image generation method, wherein the driver driving image generation model includes an encoder, an image identification extractor and a decoder, and the driver's body area, the driver's position coding matrix, and at least one in-vehicle image are input into the driver driving image generation model to generate multiple second driver driving images, including: inputting the driver's body area, the driver's position coding matrix, and at least one in-vehicle image into the encoder to obtain image coding features; determining text identification information of each in-vehicle image, and inputting the image coding features and the text identification information into the image identification extractor to obtain fused image features; and inputting the fused image features into the decoder to obtain multiple second driver driving images.

[0014] By adopting the above technical scheme, the human body area of ​​the first driver driving image is obtained according to the driver position coding matrix when the seat of the vehicle of the corresponding model is located in different positions and the driver is in different states. This makes the generated driver driving image more consistent with the driver's driving logic and more realistic and effective.

[0015] According to another specific implementation of the present application, in the driver driving image generation method disclosed in the implementation of the present application, the driver driving image generation model is a stable diffusion model.

[0016] According to another specific implementation of the present application, in the driver driving image generation method disclosed in the implementation of the present application, each in-vehicle image corresponding to the target vehicle is an in-vehicle image when the seat of the target vehicle is in different positions.

[0017] By adopting the above technical solution, by collecting pictures of different vehicles without drivers and when the seats of the vehicles are adjusted to different front-to-back distances, at least one in-vehicle image is obtained, and a driving video of any driver in the corresponding in-vehicle image can be generated. In this way, based on one driver driving image, multiple driver driving images of drivers in different in-vehicle positions of different vehicles can be generated, so that driver driving images can be generated in batches.

[0018] In the second aspect, the implementation of the present application also discloses a fatigue driving monitoring model training method, the method comprising: determining a training data set, the training data set comprising multiple sample images, the sample images comprising a second driver driving image generated by a driver driving image generation method provided based on any one of the implementations of the first aspect above; performing model training on an initial fatigue driving monitoring model according to the training data set to obtain a fatigue driving monitoring model.

[0019] The fatigue driving monitoring model training method provided in the implementation of the present application obtains a second driver driving image to train the initial fatigue driving monitoring model, thereby obtaining a fatigue driving monitoring model. In this way, a large number of driver driving images are used for model training, so that the generated fatigue driving monitoring model can be more accurate when performing driver fatigue monitoring.

[0020] On the third aspect, the implementation method of the present application also discloses a fatigue driving monitoring method, which includes: obtaining a driver's driving image; inputting the driver's driving image into a fatigue driving monitoring model to obtain the driver's fatigue driving monitoring results, and the fatigue driving monitoring model is obtained by model training based on the fatigue driving monitoring model training method provided in the second aspect above.

[0021] The fatigue driving monitoring method provided by the implementation of the present application obtains the driver's driving image (such as the driver's driving picture or the driver's driving video), inputs the driver's driving image into the fatigue driving monitoring model, and obtains the driver's fatigue driving monitoring result. In this way, the fatigue driving monitoring model trained with a large number of driver driving images is used, so that the fatigue driving monitoring result obtained is more accurate.

[0022] In a fourth aspect, the implementation method of the present application also discloses a vehicle control method, which includes: obtaining a driver's fatigue driving monitoring result, the fatigue driving monitoring result is obtained based on the fatigue driving monitoring method provided in the third aspect above; and performing corresponding control according to the fatigue driving monitoring result.

[0023] The vehicle control method provided by the implementation of the present application obtains the driver's fatigue driving monitoring results and performs corresponding control according to the fatigue driving monitoring results. In this way, a fatigue driving monitoring model trained with a large number of driver driving images is used to obtain more accurate fatigue driving monitoring results, and then corresponding control is performed according to the fatigue driving monitoring results to improve the user's driving safety.

[0024] In the fifth aspect, the implementation method of the present application also discloses an electronic device, including: a processor, and a memory connected to the processor for communication, the processor including a central processing unit and a graphics processing unit; the memory stores a computer program, and the processor executes the computer program stored in the memory, so that the electronic device implements the driver driving image generation method provided by any one of the implementation methods of the first aspect above, and / or the fatigue driving monitoring model training method provided by the second aspect.

[0025] In a sixth aspect, the implementation method of the present application further discloses a vehicle for implementing the fatigue driving monitoring method provided in the third aspect above, and / or implementing the vehicle control method provided in the fourth aspect above.

[0026] In the seventh aspect, the implementation method of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the driver driving image generation method provided by any one of the implementation methods of the first aspect above, and / or the fatigue driving monitoring model training method provided by the second aspect.

[0027] In an eighth aspect, an implementation method of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the driver driving image generation method provided in any one of the implementation methods of the first aspect above, and / or the fatigue driving monitoring model training method provided in the second aspect.

[0028] It can be understood that the beneficial effects of the fifth to eighth aspects mentioned above can also be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flowchart of a method for generating a driver driving image provided by an embodiment of the present application;

[0030] Figure 2is a schematic diagram of a driver's movable range provided in an embodiment of the present application;

[0031] Figure 3 It is a schematic diagram of the principle of generating a driver position coding matrix provided by an embodiment of the present application;

[0032] Figure 4 is a schematic diagram of a driver position coding matrix provided in an embodiment of the present application;

[0033] Figure 5 is a schematic diagram of the principle of generating a second driver driving image based on a driver driving image generation model provided in an embodiment of the present application;

[0034] Figure 6 It is a flowchart of another method for generating a driver driving image provided by an embodiment of the present application;

[0035] Figure 7 It is a flowchart of a fatigue driving monitoring model training method provided in an embodiment of the present application;

[0036] Figure 8 It is a flowchart of a fatigue driving monitoring method provided in an embodiment of the present application;

[0037] Fig. 9 is a flow chart of a vehicle control method provided in an embodiment of the present application;

[0038] Fig.10 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] As mentioned above, fatigue driving monitoring is a very important monitoring technology related to vehicle driving safety in the automotive field. By monitoring the driver's fatigue driving, traffic accidents can be effectively avoided.

[0040] In the prior art, by acquiring a driver's driving image such as driver image data or video data, the driver's driving image is input into a fatigue driving monitoring model, so that the fatigue driving monitoring model outputs the result information of whether the driver is driving fatigued. In order to ensure the output accuracy of the fatigue driving monitoring model, a large number of driver driving image samples are required to train the fatigue driving monitoring model. At present, driver driving image samples are all collected by manual collection of driver driving images, so the driver driving image requires multiple people to participate in sample construction work such as image collection, which has problems such as wasting manpower and low sample construction efficiency.

[0041] For example, with the widespread application of intelligent algorithms (such as AI algorithms) in automobile smart cockpits, the use of AI algorithms (as an example of fatigue driving monitoring models) to monitor driver fatigue is becoming more and more popular. Driver fatigue monitoring is one of the key algorithms, and driver fatigue monitoring is mainly based on driver driving images or driver driving video recognition. However, the large amount of driving data required for algorithm training is difficult to collect and annotate effectively and in large quantities. Therefore, a fast and effective method for generating driver driving images is needed.

[0042] Based on this, the present application proposes a method for generating a driver driving image, which generates multiple driver driving images of the driver in different driving areas in the vehicle according to the driver driving image, the driver position coding matrix of the driver in different movable ranges, different fatigue levels, and different moving behavior habits, and at least one in-vehicle image of the target vehicle. In this way, only one driver driving image needs to be collected to generate multiple driver driving images of the driver in different states (for example, different driving positions, different fatigue levels, and different driving behavior habits). Compared with the method of manually collecting each driver driving image in the prior art, a lot of manpower is saved and the efficiency of collecting driver driving images is accelerated.

[0043] The driver driving image generation method provided in the implementation mode of the present application is applied to a first electronic device, such as Figure 1 As shown, the method includes the following steps.

[0044] S110, determining at least one in-vehicle image of a target vehicle when the target vehicle is in a driverless state.

[0045] S120, performing image segmentation processing on each in-vehicle image to obtain a driver position area.

[0046] S130, determining the movable range of the driver in the driver position area, as well as the driver fatigue level information and the driver movement behavior habit information, and generating a driver position coding matrix according to the movable range of the driver, the driver fatigue level information and the driver movement behavior habit information.

[0047] S140, determining a first driver driving image, and generating a plurality of second driver driving images according to the first driver driving image, a driver position coding matrix, and at least one in-vehicle image.

[0048] The driver driving image generation method provided by the implementation of the present application determines the driver driving image (i.e., the first driver driving image) and determines at least one in-vehicle image of the corresponding vehicle model when the vehicle is in a driverless state; and performs image segmentation processing on each in-vehicle image to obtain the driver position area. Then, the driver's movable range in the driver position area is determined, as well as the driver's fatigue level information and the driver's mobile behavior habit information are determined, and a driver position coding matrix is ​​generated according to the driver's movable range, the driver's fatigue level information, and the driver's mobile behavior habit information. Finally, multiple driver driving images (i.e., the second driver driving image) are generated according to the driver driving image, the driver position coding matrix when the driver is in different position states, and each in-vehicle image of the target vehicle. In this way, multiple driver driving images in different states can be generated based on one driver driving image, which saves a lot of manpower and speeds up the efficiency of driver driving image generation.

[0049] In the implementation of the present application, in step S110, each in-vehicle image corresponding to the target vehicle is an in-vehicle image when the seat of the target vehicle is in different positions.

[0050] Exemplarily, pictures are collected when vehicles of different models (as examples of target vehicles) are in a driverless state and when the seats of the target vehicles are adjusted to different front-to-rear distances, to obtain at least one in-vehicle image.

[0051] For example, when the vehicle model 1 is in a driverless state and the vehicle seat is adjusted to different front-to-back distances, the vehicle interior images 1, 2, and 3 are collected; when the vehicle model 2 is in a driverless state and the vehicle seat is adjusted to different front-to-back distances, the vehicle interior images 1, 2, and 3 are collected. In this way, for each vehicle model, multiple vehicle interior images with the vehicle seat in different positions can be obtained, which is convenient for the subsequent generation of the driver's driving image.

[0052] Furthermore, in step S120, image segmentation processing is performed on all the obtained in-vehicle images to obtain the driver position area (ie, the main driving position area) corresponding to each in-vehicle image.

[0053] In the implementation of the present application, the image segmentation processing of the in-vehicle image can be specifically based on the image segmentation processing of the main driving position area based on the in-vehicle full-vehicle monitoring camera, or it can be based on the image segmentation algorithm.

[0054] Furthermore, in step S130, determining the movable range of the driver in the driver position area includes: determining the angle between the bottom edge of the seat included in the driver position area and the image edge of the in-vehicle image, and determining the target area in the driver position area according to the angle as the movable range of the driver.

[0055] For example, Figure 2 As shown, for each in-car image collected when there is no one in the car (i.e., no driver in the car), the bottom edges of the left and right bottoms of the seat (i.e., the driver's seat area) and the bottom edge of the image (i.e., the in-car image) form an angle θ, and a trapezoidal area (as an example of the target area) where the driver may appear in the driver's position area is constructed, and the driver's movable range in the driver's position area (i.e., the driver's possible movable range) is obtained. In this way, the driver's movable range corresponding to each in-car image can be obtained.

[0056] Furthermore, in step S130, a position coding matrix is ​​generated according to the driver's movable range, the driver's fatigue level information and the driver's movement behavior habit information, including: inputting the driver's movable range, the driver's fatigue level information and the driver's movement behavior habit information into a conditional probability model to perform probability calculation processing on the driver appearing in the corresponding driver's movable range, obtain the corresponding probability, and generate the driver's position coding matrix according to the probability.

[0057] Exemplarily, the driver fatigue level information and the driver movement behavior habit information that need to be generated are selected based on the display interface of the electronic device to determine the driver fatigue level information and the driver behavior habit information, wherein the driver fatigue level information includes mild fatigue and severe fatigue, and the driver movement behavior habit information is also the driver's driving habits including driving with the back rested in the seat, driving with the body leaning forward, automatic driving, etc.

[0058] In the implementation of the present application, mild fatigue and severe fatigue generally account for 50% each. Therefore, for multiple in-car images, half of the in-car images can be selected to generate driving images of the driver with mild fatigue information, and half of the in-car images can be selected to generate driving images of the driver with severe fatigue information.

[0059] like Figure 3 As shown, the driver's movable range, fatigue level information (i.e., action conditions, including mild fatigue or severe fatigue), and driver's movement behavior habit information corresponding to all in-car images of target vehicles of all vehicle models are input into the conditional probability model, so that the conditional probability model performs probability calculation processing on the probability of the driver appearing in the corresponding driver's movable range, and obtains the probability corresponding to each driver's movable range--driver's fatigue level information--driver's movement behavior habit information, and generates a driver position coding matrix (i.e., position restriction coding matrix) according to each probability. The driver position coding matrix can reflect the probability of the driver appearing in the corresponding driver's movable range under the corresponding driver's fatigue level information and the corresponding driver's movement behavior habit information.

[0060] For example, generate Figure 4 The driver position coding matrix with a probability between [0-1] is shown, where 0 means that the probability of the driver appearing in the corresponding driver movable range under the corresponding driver fatigue level and the corresponding driver movement behavior habit is 0, or if the corresponding driver movable range is the co-pilot area, then the probability of the driver appearing in the corresponding driver movable range under the corresponding driver fatigue level and the corresponding driver movement behavior habit is also 0, and 1 means that the probability of the driver appearing in the corresponding driver movable range under the corresponding driver fatigue level and the corresponding driver movement behavior habit is 100%. In this way, for multiple in-vehicle images of target vehicles of different models when there is no driver and the vehicle seat is in different front and rear positions, the probability of the driver appearing in the driver movable range corresponding to each in-vehicle image of the target vehicle can be generated based on the selected driver fatigue level information and driver movement behavior habit information, and then the driver position coding matrix corresponding to all in-vehicle images of the target vehicles of each model is obtained.

[0061] It should be noted that the size of the generated driver position encoding matrix is ​​consistent with the pixel size of the in-vehicle image of each vehicle model that needs to be input, so as to facilitate the subsequent generation of the second driver driving image.

[0062] Furthermore, in the implementation of the present application, in step S140, a driver fatigue driving image or a short driver fatigue driving video is collected from the driver through the vehicle-wide monitoring camera as the first driver driving image. That is, the first driver driving image can be a driver fatigue driving image (i.e., a driver fatigue driving picture), or a driver fatigue driving video, and of course, it can also be a driver driving image or driver driving video when the driver is driving normally. In this way, in the implementation of the present application, the generated second driver driving image can be a driving image that represents the driver in a fatigue driving state, or it can be a driving image that represents the driver is not in a fatigue driving state.

[0063] Furthermore, image segmentation processing is performed on the driver's body region in the driver's driving image or the driver's driving video to obtain the driver's body region.

[0064] Furthermore, in step S140, a plurality of second driver driving images are generated according to the first driver driving image, the driver position coding matrix, and at least one in-vehicle image, including: performing image segmentation processing on the first driver driving image to obtain the driver's body area; inputting the driver's body area, the driver position coding matrix, and at least one in-vehicle image into a driver driving image generation model to generate a plurality of second driver driving images.

[0065] Exemplarily, each driver corresponds to an ID number, and the driver's body area is segmented from the driver's driving image corresponding to a certain ID. The driver's body area, the driver position coding matrix generated above, and the in-vehicle image of the corresponding target vehicle are input into the driver image generation model to generate multiple second driver driving images.

[0066] In the implementation of the present application, the driver driving image generation model is a stable diffusion model (also known as a stable diffusion model). The purpose of the stable diffusion model is to eliminate the continuous application of Gaussian noise to the training image, which can be regarded as a series of denoising autoencoders to obtain a clear image. The stable diffusion model consists of three parts: a variational autoencoder (VAE, Variational Autoencoder, also known as a VAE encoder) (as an example of an encoder), a U-Net (also known as UNET) (as an example of an image identifier extractor) and a decoder (also known as a VAE decoder) (as an example of a decoder).

[0067] Therefore, in the implementation method of the present application, the driver driving image generation model includes an encoder, an image identification extractor and a decoder, and the driver's body area, the driver's position coding matrix, and at least one in-vehicle image are input into the driver driving image generation model to generate multiple second driver driving images, including: inputting the driver's body area, the driver's position coding matrix, and at least one in-vehicle image into the encoder to obtain image coding features; determining the text identification information of each in-vehicle image, and inputting the image coding features and the text identification information into the image identification extractor to obtain fused image features; inputting the fused image features into the decoder to obtain multiple second driver driving images.

[0068] For example, Figure 5As shown in FIG. 1 , the collected driving image of a driver with a certain ID (the segmented driver human body region is also the portrait image segmented by a certain ID), combined with the driver position coding matrix and the in-car images corresponding to different seat positions collected when there is no one in different vehicle models (also the image of a certain vehicle model), are automatically encoded by VAE (also VAE encoding), and the image coding features are generated, which are input into the stable diffusion model latent space (also the image identifier extractor (UNET)), and then the same specific text prompt word (also the text) is added. For example, the text prompt word is "the driver is driving fatigued", and the image coding features and the text prompt word are input into UNET to obtain the fused image features, and then the fused image features are input into the decoder (also VAE decoding), and the driver driving image (also the generated fatigue driving image) is obtained in which the human body region corresponding to the ID is located in each in-car image of the vehicle of the vehicle model, and the generated driver driving image is determined by the probability value corresponding to each in-car image in the driver position coding matrix corresponding to the vehicle model. That is, if the probability value is 0, the human body region corresponding to the ID corresponding to the in-car image of the vehicle of the vehicle model cannot generate the corresponding driver driving image. In this way, the human body area of ​​a driver with a certain ID is determined according to the driver position coding matrix, and a corresponding driver driving image is generated relative to the in-car image of the vehicle model with the seat located at different positions, so that the generated driver driving image is more in line with the driver's driving logic and more realistic and effective.

[0069] Furthermore, through multiple in-car images of vehicles of different models and different generation conditions (driver fatigue level information, driver movement behavior habit information), a driver position coding matrix generated based on vehicle backgrounds of different models and different seat positions can be selected, and then by obtaining the driver's body area corresponding to the driver driving image of any ID, the generation of the second driver driving image is repeated based on the aforementioned driver driving image generation method, and a sufficient number of driver fatigue driving images or driver fatigue driving videos can be generated as the second driver driving image.

[0070] For example, a driving image of a driver with ID 1 driving a vehicle in each vehicle interior image of different vehicle models is generated, and then a driving image of a driver with ID 2 driving a vehicle in each vehicle interior image of different vehicle models is generated, until driving images of drivers with all IDs driving vehicles in each vehicle interior image of different vehicle models are generated. In this way, it is only necessary to collect several driving images of drivers, so as to generate sufficient driving images of drivers in different seat positions and different driver states (i.e., different driver fatigue levels and different driver driving behavior habits) in vehicles of different vehicle models.

[0071] Therefore, if Figure 6As shown, the driver driving image generation method provided by the implementation of the present application, taking the generated driver driving image as a driver fatigue driving image as an example, specifically includes the following steps.

[0072] S10, collect a short video clip of the driver's fatigue driving (as an example of the first driver's driving image) through the in-vehicle camera, perform image segmentation processing on the driver's body area in the video to obtain the driver's body area, collect a small number of pictures of different car models when the seats are adjusted to different front and rear distances when there is no one in the car (that is, collect in-vehicle images of the corresponding vehicle model when the seats are in different positions), perform image segmentation processing on the main driving position area in the picture to obtain the driver position area.

[0073] S20, for the picture of no one in the car collected in the previous step, use the angle θ between the left and right bottom edges of the seat and the edge of the picture to construct a trapezoidal area where the driver may appear (that is, determine the angle between the bottom edge of the seat bottom in the driver's position area and the image edge of the image in the car, and determine the trapezoidal area in the driver's position area according to the angle as the driver's movable range), and select the fatigue level to be generated as mild fatigue or severe fatigue, as well as the driver's movement behavior habits. These three are used as input, and the conditional probability model is used to generate a position restriction coding matrix (that is, the driver position coding matrix). Through VAE automatic encoding, it is input into the stable diffusion model latent space, so that the generated fatigue driving data can be controlled to allow the driver to be in any main driving position in the car with randomness.

[0074] S30, the human body area segmented from the collected fatigue driving data of a certain ID (as an example of the driving image of the first driver) (that is, the image segmentation processing of the driving image of the first driver is performed to obtain the driver's human body area) is combined with the pictures of different seat positions collected when no one is in different car models (that is, the in-car images of the vehicle of the corresponding car model when the seats are in different positions) and the position coding matrix of the driver in the previous step, and the same specific text prompt words are added to generate driver fatigue data of different car models and different seat distances for this ID (that is, the human body area corresponding to the ID is located in the driver driving image of each in-car image of the vehicle of the model).

[0075] The driver driving image generation method provided by the implementation of the present application can be a method for generating a driver fatigue image or a driver fatigue video, that is, a method for generating driver fatigue driving image data based on a generative position coding matrix, firstly collects a small batch of driver fatigue data through the in-vehicle passenger monitoring system (OMS) camera (that is, obtains the first driver driving image), and collects in-vehicle images of different models when no one is present (that is, determines at least one in-vehicle image of the vehicle of the corresponding model when there is no driver), obtains the possible position information of the driver and drivers of different models (that is, determines the movable range of the driver in the driver position area), and then generates a position restriction coding matrix (that is, the driver position coding matrix) through a conditional probability model in combination with the driver's mobile behavior habits and action conditions, and then inserts the position restriction coding matrix into the latent space of the Stable Diffusion model, and can generate driver fatigue image data of different models (that is, the second driver driving image). In this way, compared with the traditional driver driving image collection or generation method, the present application fully considers the authenticity and availability of driving data generation. And by combining different models and different fatigue levels, a position restriction coding matrix is ​​generated, so as to control the stable diffusion model to effectively generate batches of driver fatigue image data or video data (that is, driver driving images). In this way, when generating driver driving images in the field of automotive intelligent cockpits, a large amount of in-vehicle fatigue driving image data can be generated under the condition of collecting only a small amount of fatigue driving data. This can reduce the collection and standard costs of specific data and speed up the generation efficiency of driver driving images.

[0076] Furthermore, the implementation of the present application also discloses a fatigue driving monitoring model training method, which is applied to a second electronic device, such as Figure 7 As shown, the specific steps include the following steps.

[0077] S210, determining a training data set, where the training data set includes a plurality of sample images.

[0078] The sample image includes a second driver driving image generated based on the driver driving image generation method provided by the above implementation method. The second driver driving image generated based on the driver driving image generation method provided by the above implementation method can refer to the specific process of the above-mentioned steps S100 to S150.

[0079] S220: Perform model training on the initial fatigue driving monitoring model according to the training data set to obtain a fatigue driving monitoring model.

[0080] The fatigue driving monitoring model training method provided in the implementation of the present application obtains the second driver driving images provided by the above implementation method and the fatigue driving monitoring results corresponding to each second driver driving image (for example, being in a fatigue driving state or not in a fatigue driving state) to perform model training on the initial fatigue driving monitoring model to obtain a fatigue driving monitoring model. In this way, a large number of driver driving images are used to train the model, so that the generated fatigue driving monitoring model can be more accurate in driver fatigue monitoring.

[0081] Furthermore, the implementation of the present application also discloses a fatigue driving monitoring method, which is applied to a vehicle, such as Figure 8 As shown, the specific steps include the following steps.

[0082] S310, obtaining a driving image of the driver.

[0083] S320, inputting the driver's driving image into a fatigue driving monitoring model to obtain a driver's fatigue driving monitoring result.

[0084] The fatigue driving monitoring model is obtained by model training based on the fatigue driving monitoring model training method provided in the above implementation.

[0085] The fatigue driving monitoring method provided by the implementation of the present application obtains the driver's driving image (such as the driver's driving picture or the driver's driving video), inputs the driver's driving image into the fatigue driving monitoring model, and obtains the driver's fatigue driving monitoring result. In this way, the fatigue driving monitoring model trained with a large number of driver driving images is used, so that the fatigue driving monitoring result obtained is more accurate.

[0086] Furthermore, the implementation of the present application also discloses a vehicle control method, which is applied to a vehicle, such as Fig. 9 As shown, the specific steps include the following steps.

[0087] S410, obtaining the driver's fatigue driving monitoring result.

[0088] The fatigue driving monitoring result is obtained based on the fatigue driving monitoring method provided by the above implementation.

[0089] S420, performing corresponding control according to the fatigue driving monitoring result.

[0090] The vehicle control method provided by the implementation of the present application obtains the driver's fatigue driving monitoring result and performs corresponding control according to the fatigue driving monitoring result. In this way, a fatigue driving monitoring model trained with a large number of driver driving images is used to obtain more accurate fatigue driving monitoring results, and then corresponding control is performed according to the fatigue driving monitoring results to improve the user's driving safety.

[0091] Among them, corresponding control based on the fatigue driving monitoring result can specifically be to slow down the vehicle until the vehicle stops in a safe area, or to stimulate the driver to keep the driver awake, etc.

[0092] In the implementation of the present application, the first electronic device and the second electronic device can be the same electronic device or different electronic devices, specifically, they can be electronic devices such as computers, mobile phones, remote computers, vehicles, and of course, cloud servers.

[0093] See also Fig.10 , Fig.10 The figure is a schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Fig.10 As shown, the electronic device may include: a transceiver 121 , a processor 122 , and a memory 123 .

[0094] The processor 122 executes the computer execution instructions stored in the memory, so that the processor 122 executes the technical solution of at least one of the driver driving image generation method, fatigue driving monitoring model training method, fatigue driving monitoring method, and vehicle control method provided in the above embodiments. The processor 122 can be a general-purpose processor, including a central processing unit CPU, a network processor (network processor, NP), etc.; it can also be a digital data processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components.

[0095] The memory 123 is connected to the processor 122 via a system bus and completes communication between them. The memory 123 is used to store computer program instructions.

[0096] By way of example and not limitation, the memory 123 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 123 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 123 may be inside or outside the integrated gateway device. In a particular embodiment, the memory 123 is a non-volatile solid-state memory. In a particular embodiment, the memory 123 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0097] The transceiver 121 may be used to obtain tasks to be executed and configuration information of the tasks to be executed.

[0098] The system bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The transceiver is used to realize the communication between the database access device and other computers (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM) and may also include non-volatile memory.

[0099] The implementation of the present application also provides a vehicle, which is used to implement the technical solution of at least one of the driver driving image generation method, fatigue driving monitoring model training method, fatigue driving monitoring method, and vehicle control method provided in the above embodiments. Or it is used to implement the fatigue driving monitoring method and / or vehicle control method provided in the above embodiments.

[0100] An embodiment of the present application also provides a chip for running instructions, which is used to execute the technical solutions of at least one of the driver driving image generation method, fatigue driving monitoring model training method, fatigue driving monitoring method, and vehicle control method provided in the above embodiments.

[0101] An embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a processor of an electronic device, the processor of the electronic device executes a technical solution of at least one of the driver driving image generation method, fatigue driving monitoring model training method, fatigue driving monitoring method, and vehicle control method provided in the above embodiments.

[0102] In some possible implementations, various aspects of the method provided in the present application can also be implemented in the form of a program product, which includes program code. When the program product is run on a processor of an electronic device, the program code is used to enable the processor of the electronic device to execute the steps of the method described above in this specification according to various exemplary implementations of the present application. For example, the electronic device can execute the technical solutions of at least one of the driver driving image generation method, fatigue driving monitoring model training method, fatigue driving monitoring method, and vehicle control method provided in the embodiments of the present application.

[0103] The program product may employ any combination of one or more readable media. The readable medium may be a readable data medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0104] The implementation method of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, it can implement the technical solution of at least one of the driver driving image generation method, fatigue driving monitoring model training method, fatigue driving monitoring method, and vehicle control method provided in the above embodiments.

[0105] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable information processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable information processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0106] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable information processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0107] These computer program instructions can also be loaded onto a computer or other programmable information processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0108] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary technical means in the art that are not disclosed in the present application.

[0109] It should be noted that, in addition to the implementation of the present application described in the above-mentioned specific embodiments, those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Although the description of the present application is introduced in conjunction with the preferred embodiment, this does not mean that the features of this invention are limited to the implementation. On the contrary, the purpose of introducing the invention in conjunction with the implementation is to cover other options or modifications that may extend based on the claims of the present application. In order to provide an in-depth understanding of the present application, the above description contains many specific details, and the present application may also be implemented without using these details. In addition, in order to avoid confusion or blurring the focus of the present application, some specific details will be omitted in the description. It should be noted that, in the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0110] It should be noted that in this specification, similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0111] Although the present application has been illustrated and described by referring to certain preferred implementations of the present application, it should be understood by those skilled in the art that the above contents are further detailed descriptions of the present application in combination with specific implementations, and it cannot be determined that the specific implementation of the present application is limited to these descriptions. Those skilled in the art may make various changes in form and details, including making several simple deductions or substitutions, without departing from the spirit and scope of the present application.

Claims

1. A method for generating a driver driving image, characterized in that: The method comprises: determining at least one in-vehicle image of the target vehicle when the target vehicle is in a driverless state; Performing image segmentation processing on each of the in-vehicle images to obtain a driver position area; Determining a movable range of a driver in the driver position area, as well as driver fatigue information and driver movement behavior habit information, and generating a driver position coding matrix according to the movable range of the driver, the driver fatigue information and the driver movement behavior habit information; A first driver driving image is determined, and a plurality of second driver driving images are generated according to the first driver driving image, the driver position encoding matrix, and the at least one in-vehicle image.

2. The method for generating a driver driving image according to claim 1, characterized in that: Determining a movable range of a driver in the driver position area includes: The angle between the bottom edge of the seat included in the driver position area and the image edge of the in-vehicle image is determined, and a target area in the driver position area is determined according to the angle as the movable range of the driver.

3. The method for generating a driver driving image according to claim 2, characterized in that: Generating a driver position coding matrix according to the movable range of the driver, the driver fatigue level information and the driver movement behavior habit information includes: The driver's movable range, the driver's fatigue level information and the driver's movement behavior habit information are input into the conditional probability model to calculate the probability of the driver appearing in the corresponding driver's movable range, obtain the corresponding probability, and generate the driver position coding matrix according to the probability.

4. The method for generating a driver driving image according to any one of claims 1 to 3, characterized in that: Generating a plurality of second driver driving images according to the first driver driving image, the driver position encoding matrix, and the at least one in-vehicle image, comprising: Performing image segmentation processing on the first driver driving image to obtain a driver body region; The driver's body area, the driver's position coding matrix, and the at least one in-vehicle image are input into a driver driving image generation model to generate a plurality of the second driver driving images.

5. The method for generating a driver driving image according to claim 4, characterized in that: The driver driving image generation model includes an encoder, an image identifier extractor, and a decoder, and the driver body region, the driver position encoding matrix, and the at least one in-vehicle image are input into the driver driving image generation model to generate a plurality of the second driver driving images, including: Inputting the driver's body area, the driver's position coding matrix, and the at least one in-vehicle image into the encoder for coding processing to obtain image coding features; Determine the text identification information of each of the in-vehicle images, input the image coding features and the text identification information into the image identification extractor for feature fusion processing to obtain fused image features; The fused image features are input into the decoder for decoding processing to obtain a plurality of driving images of the second driver.

6. The method for generating a driver driving image according to claim 5, characterized in that: The driver driving image generation model is a stable diffusion model.

7. The method for generating a driver driving image according to any one of claims 1 to 6, characterized in that: The in-vehicle images corresponding to the target vehicle are in-vehicle images when the seats of the target vehicle are in different positions.

8. A fatigue driving monitoring model training method, characterized in that: The method comprises: Determine a training data set, wherein the training data set includes a plurality of sample images, wherein the sample images include a second driver driving image generated based on the driver driving image generation method according to any one of claims 1 to 7; The initial fatigue driving monitoring model is trained according to the training data set to obtain a fatigue driving monitoring model.

9. A method for monitoring fatigue driving, characterized in that: The method comprises: Acquire a driving image of the driver; The driver's driving image is input into a fatigue driving monitoring model to obtain a fatigue driving monitoring result of the driver. The fatigue driving monitoring model is obtained by model training based on the fatigue driving monitoring model training method according to claim 8.

10. A vehicle control method, characterized in that: The method comprises: Obtaining a driver's fatigue driving monitoring result, wherein the fatigue driving monitoring result is obtained based on the fatigue driving monitoring method according to claim 9; Corresponding control is performed according to the fatigue driving monitoring result.

11. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor, wherein the processor includes a central processing unit and a graphics processing unit; The memory stores a computer program, and the processor executes the computer program stored in the memory so that the electronic device implements the driver driving image generation method as described in any one of claims 1 to 7, and / or the fatigue driving monitoring model training method as described in claim 8.

12. A vehicle, characterized in that: The vehicle is used to implement the fatigue driving monitoring method as described in claim 9 and / or the vehicle control method as described in claim 10.

Images