Vehicle Video Stabilization Method, Device, Vehicle and Storage Medium

Abstract

This application is applicable to the field of image processing technology, and provides a vehicle video stabilization method, device, vehicle and storage medium. The method includes: obtaining the vehicle video of the vehicle; inputting the vehicle video into a first neural network to obtain a plurality of grid-shaped first motion vector maps, where the first motion vector map is used to represent the offset vector corresponding to each grid of the subsequent frame image relative to the previous frame image in each adjacent two-frame image of the vehicle video; performing smoothing processing on the first motion vector map to obtain a processed second motion vector map; calculating a first motion optical flow map according to the second motion vector map; performing motion compensation on the frame images in the vehicle video according to the first motion optical flow map to obtain a stabilized video. The above method improves the accuracy of the motion estimation of the neural network for video images, thereby improving the processing effect of video stabilization.

Description

Technical Field

[0001] This application belongs to the technical field of image processing, and particularly relates to a vehicle-mounted video image stabilization method, device, vehicle and storage medium. Background Art

[0002] Currently, the video recording function in vehicles is very common, and relevant devices are available in various vehicle models. During the video recording process, it is inevitable to encounter bumpy roads, and at this time, an image stabilization function is required. Image stabilization is divided into two paths, one is a hardware image stabilization method mainly based on a pan-tilt, and the other is a pure software solution idea.

[0003] In the prior art, neural networks are usually used to extract key points of images, so as to estimate a motion matrix for the entire image. Its motion estimation accuracy is low, resulting in a poor processing effect of video image stabilization. Summary of the Invention

[0004] Embodiments of this application provide a vehicle-mounted video image stabilization method, device, vehicle and storage medium, which can improve the accuracy of motion estimation of neural networks for video images, thereby enhancing the processing effect of video image stabilization.

[0005] In a first aspect, embodiments of this application provide a vehicle-mounted video image stabilization method, including:

[0006] Obtain the vehicle-mounted video of the vehicle;

[0007] Input the vehicle-mounted video into a first neural network to obtain multiple grid-shaped first motion vector maps, where the first motion vector map is used to represent the offset vector corresponding to each grid of the subsequent frame image relative to the previous frame image in each adjacent two-frame image of the vehicle-mounted video;

[0008] Perform smoothing processing on the first motion vector map to obtain a processed second motion vector map;

[0009] Calculate a first motion optical flow map according to the second motion vector map, where the first motion optical flow map is used to represent the motion displacement corresponding to each pixel point of the subsequent frame image relative to the previous frame image in each adjacent two-frame image of the vehicle-mounted video;

[0010] Perform motion compensation on the frame images in the vehicle-mounted video according to the first motion optical flow map to obtain a stabilized video.

[0011] In a second aspect, embodiments of this application provide a vehicle-mounted video image stabilization device, including:

[0012] A first acquisition module, configured to obtain the vehicle-mounted video of the vehicle;

[0013] A network processing module, configured to input the vehicle-mounted video into a first neural network to obtain multiple grid-shaped first motion vector maps, where the first motion vector map is used to represent the offset vector corresponding to each grid of the subsequent frame image relative to the previous frame image in every two adjacent frames of the vehicle-mounted video;

[0014] A smoothing processing module, configured to perform smoothing processing on the first motion vector map to obtain a processed second motion vector map;

[0015] A calculation module, configured to calculate a first motion optical flow map according to the second motion vector map, where the first motion optical flow map is used to represent the motion displacement corresponding to each pixel point of the subsequent frame image relative to the previous frame image in every two adjacent frames of the vehicle-mounted video;

[0016] A motion compensation module, configured to perform motion compensation on the frame images in the vehicle-mounted video according to the first motion optical flow map to obtain a stabilized video.

[0017] In a third aspect, an embodiment of the present application provides a vehicle, including a terminal device, and the terminal device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and is characterized in that when the processor executes the computer program, the vehicle-mounted video stabilization method described in any one of the above first aspects is implemented.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program, and is characterized in that when the computer program is executed by a processor, the vehicle-mounted video stabilization method described in any one of the above first aspects is implemented.

[0019] In a fifth aspect, an embodiment of the present application provides a computer program product, and when the computer program product runs on a terminal device, the terminal device is enabled to execute the vehicle-mounted video stabilization method described in any one of the above first aspects.

[0020] The beneficial effects of the first aspect of the present application compared with the prior art are as follows: by acquiring the vehicle-mounted video; inputting the vehicle-mounted video into the first neural network to obtain multiple grid-like first motion vector maps, wherein the first motion vector map is used to represent the offset vector corresponding to each grid of the next frame image of each adjacent two frames in the vehicle-mounted video relative to the previous frame image; smoothing the first motion vector map to obtain a processed second motion vector map; based on the second motion vector map, calculating a first motion optical flow map, wherein the first motion optical flow map is used to represent the motion displacement corresponding to each pixel point of the next frame image of each adjacent two frames in the vehicle-mounted video relative to the previous frame image; performing motion compensation on the frame images in the vehicle-mounted video according to the first motion optical flow map to obtain a stabilized video. The embodiment of the present application utilizes the first neural network to estimate the grid-like motion vector map of the entire image in a grid manner, and performs image stabilization processing of the vehicle-mounted video image through the network-like motion vector map. On the basis of achieving end-to-end motion estimation, the original single motion matrix estimation method is extended to a plane grid, thereby improving the accuracy of the neural network in motion estimation of the video image, thereby improving the processing effect of video stabilization.

[0021] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0023] Figure 1 1 is a flow chart of a vehicle-mounted video stabilization method provided by an embodiment of the present application;

[0024] Figure 2 yes Figure 1 A schematic diagram showing the smoothing process of step S203 in FIG.

[0025] Figure 3 yes Figure 2 Schematic diagram of the specific implementation steps of step S204;

[0026] Figure 4 1 is a schematic structural diagram of a vehicle video stabilization device provided in one embodiment of the present application;

[0027] Figure 5 It is a structural schematic diagram of a vehicle provided in one embodiment of the present application. DETAILED DESCRIPTION

[0028] The in-vehicle video stabilization method of the present embodiment requires the use of a first neural network, which is a pre-trained network. Therefore, before describing the in-vehicle video stabilization method of the present embodiment in detail, the training process of the first neural network is first described. The training process of the first neural network is as follows:

[0029] S101 , obtaining a training data set, where the training data set includes a plurality of data pairs, each data pair including a first image and a second image, wherein the second image is jittered relative to the first image.

[0030] It should be noted that the image content of the first image and the second image is continuous. Optionally, the first image and the second image may be two adjacent frames of images in the sample video, wherein the second image is the latter frame of the two adjacent frames, and the first image is the former frame of the two adjacent frames.

[0031] S102: Train the unsupervised neural network based on the training data set to obtain a trained first neural network.

[0032] Here, an unsupervised neural network is a neural network that does not require a category (label), but only needs to be given input.

[0033] In a possible implementation, the implementation process of step S102 may include:

[0034] Step S1021: Input the first image and the second image in the data pair into an unsupervised neural network to obtain a grid-shaped third motion vector map, where the third motion vector map is used to represent each grid offset vector of the second image relative to the first image.

[0035] Here, the first image and the second image are input into the unsupervised neural network, and a grid with a total number of grids of S×S is constructed between the first image and the second image, where S represents the number of rows of the grid and S represents the number of columns of the grid. Each grid will regress 4 motion vectors Δ through the unsupervised neural network. i,j , where i∈[0,S), j∈[0,S), where Δ i,j Represents the motion vector corresponding to the grid corner point with coordinate position (i, j) in the grid map, i represents the row number of the grid map, j represents the column number of the grid map, and finally outputs a grid-shaped third motion vector map. The third motion vector map is a three-dimensional motion matrix of S×S×3, where each dimension in the three-dimensional motion matrix of S×S×3 represents the motion offset in the x-direction, y-direction, and z-direction.

[0036] Step S1022: Calculate the second motion optical flow map based on the third motion vector map. The second motion optical flow map is used to characterize the motion displacement corresponding to each pixel point of the second image relative to the first image.

[0037] In a possible implementation, the implementation process of step S1022 may include:

[0038] S1: Perform a linear transformation on the third motion vector map to calculate the homography matrix of each grid in the third motion vector map.

[0039] Here, perform a linear transformation on the 4 motion vectors corresponding to each grid in the third motion vector map to calculate the homography matrix of each grid in the third motion vector map.

[0040] S2: Calculate the first motion optical flow map based on the homography matrix of each grid in the third motion vector map.

[0041] Specifically, for each grid in the third motion vector map, obtain the homogeneous coordinates corresponding to the multiple pixel points included in the grid projected onto the image; calculate the second motion optical flow map based on the homography matrix of the grid and the homogeneous coordinates corresponding to the multiple pixel points included in the grid projected onto the image.

[0042] Through the above processing process, using the homography matrix of the sparse grid to approximate the global optical flow map, that is, the second motion optical flow map, can reduce the computational amount and improve the running speed.

[0043] Step S1023: Perform a bilinear interpolation operation on the second image according to the second motion optical flow map to obtain the third image after the second image moves.

[0044] It should be noted that based on the second motion optical flow map, performing a bilinear interpolation operation on the second image to obtain the third image after the second image moves is the motion compensation of the second image. That is, using the neural network to obtain the estimated motion displacement of the second image relative to the first image (the second motion optical flow map), and using the motion displacement of the second image relative to the first image to correct the second image to obtain the third image. Here, the third image needs to be as similar as possible to the first image to achieve the effect of image stabilization. Therefore, it is necessary to calculate the loss value between the first image and the third image and continuously optimize the unsupervised neural network to obtain an accurate motion estimation.

[0045] Step S1024: Calculate the loss value between the first image and the third image.

[0046] Here, the following formula (1) can be used to calculate the loss value between the first image and the third image.

[0047]

[0048] where A i,j represents the pixel value of the i-th row and j-th column in the first image, and B’ i,j represents the pixel value of the i-th row and j-th column in the third image, i represents the row number of the image pixels, j represents the column number of the image pixels, M represents the total number of rows of the image pixels, N represents the total number of columns of the image pixels, and L1 represents the loss value between the first image and the third image.

[0049] Step S1025: Update the network parameters of the unsupervised network based on the loss value to obtain the first neural network.

[0050] In this step, after updating the network parameters of the unsupervised network based on the loss value, if the loss value of the unsupervised network after this update is less than the preset threshold and the loss value of the unsupervised network after this update is equal to the loss value of the unsupervised network after its previous update, then determine the unsupervised network after this update as the first neural network; otherwise, return to execute step S1024 and subsequent steps until the loss value of the unsupervised network after this update is less than the preset threshold and the loss value of the unsupervised network after this update is equal to the loss value of the unsupervised network after its previous update. Or, after updating the network parameters of the unsupervised network based on the loss value, judge the number of times of network parameter update. If the number of times of network parameter update reaches the preset number of times, then determine the unsupervised network after this update as the first neural network; otherwise, return to execute step S1024 and subsequent steps until the number of times of network parameter update corresponding to the unsupervised network after this update reaches the preset number of times, and determine the unsupervised network after this update as the first neural network.

[0051] Figure 1 shows a schematic flowchart of the vehicle video stabilization method provided by the present application. Refer to Figure 1 , and the details of this vehicle video stabilization method are as follows:

[0052] S201: Obtain the vehicle video of the vehicle.

[0053] Here, the vehicle video of the vehicle can be obtained through a video recording device installed on the vehicle.

[0054] S202: Input the vehicle video into the first neural network to obtain multiple grid-shaped first motion vector maps, where the first motion vector map is used to represent the offset vector corresponding to each grid of the subsequent frame image relative to the previous frame image in every two adjacent frames of the vehicle video.

[0055] Here, the first neural network is a pre-trained neural network. The specific training process is as described above and will not be elaborated here.

[0056] It should be noted that every two adjacent frames of the vehicle video are fed through the first neural network to output a grid-like first motion vector map. For example, if the vehicle video includes N frames of images, and these N frames are fed into the first neural network, the first and second frames will correspond to a grid-like first motion vector map, the second and third frames will correspond to a grid-like first motion vector map, and so on, resulting in N-1 grid-like first motion vector maps.

[0057] S203: Perform smoothing processing on the first motion vector map to obtain a processed second motion vector map.

[0058] It should be noted that the purpose of smoothing the first motion vector graph is to remove obvious jitter encountered in the frame image and ensure the consistency of the motion trend of the entire in-vehicle video.

[0059] In a possible implementation, the implementation process of step S203 may include:

[0060] S2031 , for each frame of the vehicle-mounted video, obtain first motion vector maps corresponding to K frames before and after the current frame, where K is a positive integer.

[0061] S2032: Perform weighted fusion calculation on the first motion vector maps corresponding to the K frames before and after the current frame image to obtain a second motion vector map.

[0062] Specifically, the second motion vector map can be calculated using the following formula (2):

[0063]

[0064] in, is the weight that conforms to the Gaussian distribution, Indicates the influence of the previous and next frames on the current frame. The farther away the frame is from the current frame, the smaller the influence on the current frame. -K represents the Kth frame before the current frame, and K represents the Kth frame after the current frame. Represents the current frame image t k The motion vectors, that is, the motion vectors corresponding to the first motion vector map, Represents frame image t k + motion vector, The second motion vector map representing the current frame image.

[0065] After the above processing, the smoothed second motion vector graph corresponding to all frame images of the vehicle-mounted video is obtained.

[0066] For ease of understanding, refer to Figure 2Show the smoothing process. Assume that the motion trend of the entire in-vehicle video is upward. Here, only the motion vector of the upper left corner point in the grid of the first motion vector map is shown. It can be seen from the figure that the image of frame t2 encounters obvious jitter and the motion vector deviates. The deviation of the motion vector is corrected by smoothing multiple frames before and after.

[0067] S204. According to the second motion vector map, calculate the first motion optical flow map, where the first motion optical flow map is used to represent the motion displacement corresponding to each pixel point of the latter frame image relative to the former frame image in every two adjacent frame images of the in-vehicle video.

[0068] Refer to Figure 3 , in a possible implementation, the implementation process of step 204 may include:

[0069] Step 2041, perform a linear transformation on the second motion vector map to obtain the homography matrix of each grid in the second motion vector map.

[0070] Step 2042, calculate the first motion optical flow map according to the homography matrix of each grid in the second motion vector map.

[0071] Optionally, step 2042 may specifically include:

[0072] For each grid in the second motion vector map, obtain the homogeneous coordinates corresponding to multiple pixel points included in the projection of the grid onto the image; calculate the first motion optical flow map according to the homography matrix of the grid and the homogeneous coordinates corresponding to multiple pixel points included in the projection of the grid onto the image.

[0073] Through the above processing process, using the homography matrix of the sparse grid to approximate the global optical flow map, that is, the first motion optical flow map, can reduce the calculation amount and improve the running speed.

[0074] S205. Perform motion compensation on the frame images in the in-vehicle video according to the first motion optical flow map to obtain a stabilized video.

[0075] Specifically, for each frame image in the in-vehicle video, perform motion compensation on the frame image according to its corresponding first motion optical flow map to obtain a stabilized video composed of the compensated frame images.

[0076] The inventors further found that training all different motion blurs with a unified model is not applicable to all vehicle driving conditions. For example, the inventors found that when the vehicle speed is different, the vehicle jitter is different, and the jitter degree of the in-vehicle video obtained by shooting is also different. At this time, when still using the above-mentioned trained model for processing, it is easy to cause poor processing effects on in-vehicle videos with different jitter degrees. To solve this technical problem, the present application proposes to train different neural networks for different moving speed ranges; that is, different moving ranges correspond to different neural networks. Therefore, the in-vehicle video obtained by shooting the camera device at different moving speeds can be input into the neural network corresponding to the moving speed for processing, that is, in-vehicle videos with different jitter degrees can be input into different neural networks for processing, improving the video stabilization effect. For this idea, the in-vehicle video stabilization method provided by an embodiment of the present application may further include steps S601 and S602, which are introduced in detail as follows:

[0077] In S601, obtain the target moving speed.

[0078] Among them, the target moving speed is the moving speed of the camera device that shoots the in-vehicle video when shooting the in-vehicle video. In practical applications, when a user sits at a position inside a moving vehicle such as a train, car, or bus, the user can take pictures of the scenery outside. Since the vehicle is in a moving state, the video obtained by shooting may be jittery. Therefore, the user can send a request for video de-jittering to the terminal device.

[0079] It should be noted that the terminal device detecting that the user has sent the above video de-jittering request may be: detecting that the user has opened a preset APP or detecting that the user has performed a preset operation within the preset APP. The preset operation can be determined according to actual needs and is not limited here. Exemplarily, the preset operation can be clicking a preset control. That is, if the terminal device detects that the user clicks the preset control within the preset APP, it is considered that the preset operation has been detected, that is, it is considered that the user has sent a video de-jittering request.

[0080] In some possible embodiments, the above preset operation can also be a time-triggered operation. The terminal device can be configured with a corresponding video processing process during operation. This video processing process includes trigger nodes for multiple key events. The above key events include requests for video de-jittering. In this case, if the terminal device detects that it has reached the trigger node associated with the video de-jittering event, it executes the operations of this solution to obtain a stabilized video.

[0081] In the embodiments of the present application, after the terminal device detects a request for video de-jittering, it can obtain the in-vehicle video and the moving speed of the camera device when shooting the in-vehicle video.

[0082] In an embodiment of the present application, when the user is shooting a video in a vehicle in a driving state, the imaging device for shooting the in-vehicle video can be a camera installed on the vehicle. Based on this, the terminal device can specifically obtain the moving speed of the camera corresponding to the in-vehicle video at the time of shooting according to the following steps, which are described in detail as follows:

[0083] When the camera shoots the in-vehicle video, obtain the driving speed of the vehicle; determine the driving speed of the vehicle as the target moving speed.

[0084] In this embodiment, the terminal device can be wirelessly communicatively connected to the camera installed on the vehicle. Therefore, when the terminal device detects that the camera is shooting, it can obtain the driving speed of the vehicle at this time in real time through the in-vehicle system wirelessly communicatively connected to the terminal device. It should be noted that the in-vehicle system includes a wheel speed sensor, and the wheel speed sensor can be a magneto-electric wheel speed sensor or a Hall wheel speed sensor, and there is no limitation here.

[0085] In S602, find the neural network that matches the target moving speed from multiple neural networks, and use the matched neural network as the first neural network.

[0086] Among them, the neural network is obtained by training with the target sample image as the training set. When the target sample image is shot, the range interval of the moving speed of the corresponding camera covers the target moving speed.

[0087] In an embodiment of the present application, the terminal device can pre-store the matching relationship between different moving speed ranges and different neural networks. Therefore, after the terminal device obtains the target moving speed of the camera corresponding to the in-vehicle video at the time of shooting, it can obtain the neural network that matches the target moving speed according to the moving speed range where the target moving speed is located and the pre-stored matching relationship between different moving speed ranges and different neural networks, and use this neural network as the first neural network for subsequent processing.

[0088] Specifically, the matching relationship between different moving speed ranges and different neural networks is constructed through the following steps S701 to S7O2, which are described in detail as follows:

[0089] In S701, obtain the original sample images corresponding to each of the multiple preset speed range intervals.

[0090] In an implementation manner of this embodiment, the terminal device can pre-obtain multiple images shot by the camera at different moving speeds, and classify the above-mentioned multiple images according to the multiple preset speed range intervals set in advance and the moving speed of each image at the time of shooting, so as to obtain the original sample images corresponding to each of the multiple preset speed range intervals.

[0091] In S702, for each preset speed range interval, the corresponding original sample image is used as a training set to train the corresponding neural network.

[0092] In this embodiment, after the terminal device obtains the original sample images respectively corresponding to multiple preset speed range intervals, in order to improve the video stabilization effect of the in-vehicle video in the subsequent process, for any one preset speed range interval, only the original sample image corresponding to this preset speed range interval is used as a training set to train the preset original neural network, so as to obtain the trained neural network corresponding to this preset speed range interval. Among them, the original image deblurring model can be a pre-constructed first deep learning model, such as a generative adversarial network (DeblurGAN) model.

[0093] In the in-vehicle video stabilization method provided in this embodiment, before inputting the in-vehicle video into the first neural network, it may further include: inputting the in-vehicle video into the trained non-stabilized video classification model for processing to obtain the probability value that the in-vehicle video is a non-stabilized video; if this probability value is greater than the set threshold, then execute the step of inputting the in-vehicle video into the first neural network and subsequent steps. In this embodiment, in order to determine that the in-vehicle video is indeed a non-stabilized video, so as to avoid still executing steps S202-S205 for the stabilized video, resulting in the corresponding processing device doing useless work.

[0094] It should be noted that the non-stabilized video classification model is used to detect the probability that the in-vehicle video is a non-stabilized video. The non-stabilized video classification model can be obtained by training a pre-constructed second deep learning model based on a preset sample set. Taking the second deep learning model as the ResNet18 classification model as an example, each sample data in the preset sample set includes a first sample image and a second sample image. Among them, the first sample image is a historical non-stabilized video, and the second sample image is the corresponding historical clear image of this historical non-stabilized video. Through training, the second deep learning model can learn the probability value that all sample images are non-stabilized videos. At this time, the trained second deep learning model can be used as the non-stabilized video classification model.

[0095] In this embodiment, after obtaining the probability value that the in-vehicle video is a non-stabilized video, this probability value can be compared with the set threshold. Among them, the set threshold can be determined according to actual needs and is not limited here. Exemplarily, the set threshold can be 0.9. When it is detected that the probability value that the in-vehicle video is a non-stabilized video is greater than the set threshold, it can be determined that this in-vehicle video is a non-stabilized video. Therefore, the subsequent video stabilization steps are executed. When it is detected that the probability value that the in-vehicle video is a non-stabilized video is less than or equal to the set threshold, it can be determined that this in-vehicle video is not a non-stabilized video. Therefore, there is no need to perform video stabilization processing on this in-vehicle video, that is, execute the subsequent video stabilization steps.

[0096] As can be seen from the above, by inputting the obtained vehicle-mounted video into the trained non-steady video classification model for processing, the probability value that the vehicle-mounted video is a non-steady video is obtained, and this probability value is compared with the set threshold. Only when it is detected that the probability value is greater than the set threshold can it be determined that the vehicle-mounted video is a non-steady video. At this time, the subsequent video stabilization steps are executed, which reduces the working pressure of the relevant processing equipment and also improves the working efficiency of the relevant processing equipment.

[0097] When the image of the vehicle-mounted video is a color image, before inputting the vehicle-mounted video into the trained non-steady video classification model for processing, it may further include converting the image of the vehicle-mounted video from a color image to a grayscale image with three channels; using the first edge detection algorithm to process the grayscale image with three channels to obtain the edge image corresponding to each of the grayscale images with three channels; splicing the edge image and the image in the vehicle-mounted video to obtain an input image with six channels; inputting the input image with six channels into the non-steady video classification model for processing to obtain the probability value that the vehicle-mounted video is a non-steady video.

[0098] Since the color image is an RGB three-channel image, therefore, according to the image conversion method, the RGB three-channel image can be converted into grayscale images corresponding to each of the three channels, that is, the grayscale image corresponding to the R channel, the grayscale image corresponding to the G channel, and the grayscale image corresponding to the B channel. Among them, the image conversion method can specifically be: sorting all the pixel values corresponding to each channel in the color image. It can be understood that all the pixel values corresponding to the R channel in the vehicle-mounted video can be sorted in sequence to obtain the grayscale image corresponding to the R channel, all the pixel values corresponding to the G channel in the vehicle-mounted video can be sorted in sequence to obtain the grayscale image corresponding to the G channel, and all the pixel values corresponding to the B channel in the vehicle-mounted video can be sorted in sequence to obtain the grayscale image corresponding to the B channel.

[0099] After obtaining the grayscale images with three channels corresponding to the vehicle-mounted video, in order to obtain rich edge information of the vehicle-mounted video, the first edge detection algorithm can be used to process the grayscale images with three channels, so as to obtain the edge image corresponding to each of the grayscale images with three channels. Among them, the first edge detection algorithm can be the Sobel algorithm.

[0100] It should be noted that since the input of the deep learning model is usually in data format, in this embodiment, the pixel values of each color channel of the image in the vehicle-mounted video and the pixel values of a single channel corresponding to each of the three edge images corresponding to the image in the vehicle-mounted video can be spliced simultaneously, so as to obtain data with six channels, that is, the image in the vehicle-mounted video and the three-channel edge images are spliced to obtain an input image with six channels.

[0101] Exemplarily, assume that the pixel value of a certain pixel point in a certain image of the in-vehicle video is (25, 17, 24). The pixel value of the pixel point corresponding to this pixel point of the in-vehicle video in the edge image corresponding to the R channel is 20, the pixel value of the pixel point corresponding to this pixel point of the in-vehicle video in the edge image corresponding to the G channel is 15, and the pixel value of the pixel point corresponding to this pixel point of the in-vehicle video in the edge image corresponding to the B channel is 22. Then the data corresponding to the pixel point of the six channels after splicing is [25, 17, 24, 20, 15, 22].

[0102] The in-vehicle video stabilization method provided in this embodiment can obtain the edge information of the in-vehicle video through the first edge detection algorithm, thereby improving the accuracy of determining whether the in-vehicle video is an unstable video.

[0103] The inventor further found that the image quality in the in-vehicle video taken under weak light conditions is poor, resulting in poor subsequent in-vehicle video stabilization processing effects. To solve this problem, this application performs denoising processing on the in-vehicle video based on the trained denoising network model to obtain the denoised in-vehicle video, improve the picture quality of the image, and further improve the in-vehicle video stabilization processing effect.

[0104] In a possible implementation, performing denoising processing on the in-vehicle video based on the trained denoising network model to obtain the denoised in-vehicle video may include:

[0105] S801, for each frame of image in the in-vehicle video, based on the brightness value of each pixel point in this frame of image, calculate the average light brightness value corresponding to this frame of image.

[0106] Specifically, for each frame of image in the in-vehicle video, the corresponding average light brightness value is calculated using the following formula (3):

[0107]

[0108] where (x, y) represents the position coordinates of the pixel point in this frame of image, Lum(x, y) represents the brightness value of the pixel point in this frame of image, N represents the total number of pixel points in this frame of image, and δ is a preset parameter value.

[0109] S802, based on the trained denoising network model, perform denoising processing on each frame of image in the in-vehicle video whose average light brightness value is less than the first preset threshold to obtain the denoised in-vehicle video.

[0110] Here, if the average light brightness value is less than the first preset threshold, it indicates that the light condition corresponding to this frame of image is poor, and this frame of image is a noisy video image that needs to be denoised. That is, the frame of image is input into the trained denoising network model for denoising processing, and the denoised image is output. The denoised image is a clear image after removing noise, thereby improving the picture quality of the images during driving.

[0111] For the training process of the denoising network model, it can be specifically as follows:

[0112] S901, Obtain a training data set. Among them, the training data set includes multiple data pairs, and each data pair includes N first video images and N second video images. The light brightness value inside the vehicle corresponding to the first video image is less than the light brightness value inside the vehicle corresponding to the second video image. The first video image and the second video image respectively correspond to different shooting objects inside the same vehicle, and N is a positive integer.

[0113] Here, the training data set can be obtained through a collection device. Exemplarily, the collection device may include a DMS camera and an Occupancy Monitoring System (OMS) camera, and both the DMS camera and the OMS camera are used to collect video images inside the vehicle.

[0114] In a possible implementation manner, the implementation process of step S901 may include:

[0115] S9011, Collect the video images inside the vehicle when the first condition is met, and use the video images inside the vehicle when the first condition is met as the first video images. The first condition is that the light brightness value is less than the second preset threshold.

[0116] Here, the first condition that the light brightness value is less than the second preset threshold indicates that the light condition corresponding to the shooting environment inside the vehicle is poor. For example, the vehicle is driving at night, in a tunnel, or in an underground garage. Among them, the first video image is a noisy image.

[0117] S9012, Collect the video images inside the vehicle when the second condition is met, and use the video images inside the vehicle when the second condition is met as the second video images. The second condition is that the light brightness value is greater than or equal to the second preset threshold.

[0118] Here, the second condition that the light brightness value is greater than or equal to the second preset threshold indicates that the light condition corresponding to the shooting environment inside the vehicle is good. For example, the vehicle is driving during the day. Among them, the second video image is a noise-free image.

[0119] S9013, Construct a training data set based on the first video images and the second video images.

[0120] It should be noted that each first video image captured by the camera has a corresponding second video image, that is, the first video image and the second video image respectively correspond to different shooting objects in the same vehicle.

[0121] That is to say, the image acquisition environments of the first video image and the second video image corresponding to the first video image are related but not paired. That is, the shooting contents corresponding to the two video images do not need to be exactly the same.

[0122] As can be seen from the above description, the training data set in the embodiments of the present application is an unpaired data set, and the unpaired data set is easy to collect and obtain, which reduces the difficulty of training the denoising network model to a certain extent.

[0123] This step may specifically include: dividing the collected first video images and second video images into multiple data pairs, and each data pair includes N first video images and N second video images. It should be noted that each first video image in the N first video images corresponds to one second video image in the data pair. Different first video images may be video images in the same vehicle or video images in different vehicles.

[0124] Here, in order to enrich the diversity of the collected data, the placement positions of the cameras can be multiple positions, such as areas of the A-pillar, the center console, the steering tube, and the rearview mirror. In addition, the ages and genders of the people photographed in the vehicle, such as the driver and passengers in the vehicle, also need to have diversity.

[0125] S902. Perform iterative training on the denoising network model to be trained based on the training data set, and obtain the trained denoising network model.

[0126] The denoising network model to be trained can be a Generative Adversarial Networks (GAN) model. The GAN model includes a generator network and a discriminator network.

[0127] The general process of performing iterative training on the denoising network model to be trained based on the training data set is as follows: jointly perform adversarial training on the untrained generator network and discriminator network, and input the first video image in the data pair into the generator network to generate a random video image to deceive the discriminator network; then, the discriminator network judges the authenticity of this random video image. Finally, during the training process of these two types of networks, the capabilities of the two types of networks become stronger and stronger, and finally reach a steady state. The generator network that reaches the steady state is determined as the trained denoising network model. The specific process is shown in the following steps S9021 to S9026.

[0128] S9021. Input the first video image in the data pair into the denoising network model to obtain a third video image.

[0129] Step S9021 may specifically include: extracting the Y component of the YUV space image corresponding to the first video image in the data pair; inputting the Y component corresponding to the first video image into the denoising network model to obtain a third video image. It should be noted that the main structural information of the image is all on the Y component. Using the Y component corresponding to the image for model training can, on the one hand, reduce the amount of computation, and on the other hand, improve the robustness of the denoising network model so that it is not affected by the color space of the image.

[0130] It should be noted that the first video image is a noisy image. When starting training, the network parameters of the generation network are random network parameters, and the N third video images obtained after inputting the N first video images in the data pair into the generation network are N random video images.

[0131] The denoising network model can be a generation network. The network structure of the generation network adopts the network structure of U-Net, which includes M convolutional modules, and each convolutional module contains a convolutional layer, an activation function, and a normalization layer, where the convolutional layer is multiple. Among them, each convolutional module is a residual structure, divided into a downsampling stage and an upsampling stage. Among them, in the upsampling stage, bilinear upsampling is combined with the convolutional layer to replace the traditional deconvolution operation.

[0132] For example, the network structure of U-Net may include 8 convolutional modules, and each convolutional module contains 2 3×3 convolutional layers, a LeakyReLu activation function, and a normalization layer.

[0133] S9022, input the third video image into the global discriminant network to obtain a first discriminant result.

[0134] In this step, inputting the third video image into the global discriminant network aims to discriminate the authenticity of the third video image, and the output discriminant result is generally represented by 0 or 1. Among them, 0 indicates that the discriminant result is false, and 1 indicates that the discriminant result is true.

[0135] S9023, input the second video image in the data pair into the global discriminant network to obtain a second discriminant result.

[0136] Step S9023 may specifically include: extracting the Y component of the YUV space image corresponding to the second video image in the data pair; inputting the Y component corresponding to the second video image into the global discriminant network to obtain a second discriminant result. Here, inputting the second video image, that is, the real noiseless image, into the global discriminant network aims to discriminate the authenticity of the second video image.

[0137] S9024, input the local image of the third video image into the local discriminant network to obtain a third discriminant result.

[0138] Here, each third video image is divided into multiple local images and input into the local discrimination network to discriminate the authenticity of the local images in the third video image, so as to more finely guide the training of the subsequent generation network.

[0139] It should be noted that for the discrimination network, in the embodiments of the present application, it specifically refers to the global discrimination network and the local discrimination network. For the second video image in the data pair, that is, the real noiseless image, the discrimination network should try its best to correctly discriminate and output the value 1. For the third video image, that is, the third video image (fake denoised image) obtained by the generation network from the first video image (noisy image) in the data pair, the discrimination network should try its best to wrongly discriminate and output the value 0. And for the generation network, it should try its best to deceive the discrimination network. The two networks confront each other and continuously adjust the parameters. The ultimate goal is to make the discrimination network (global discrimination network and local discrimination network) unable to judge whether the output result of the generation network is real.

[0140] In the embodiments of the present application, the two networks confront each other and continuously adjust the parameters, specifically referring to continuously adjusting the network parameters of the global discrimination network, the local discrimination network and the generation network.

[0141] First, fix the network parameters of the generation network unchanged, and update the network parameters of the global discrimination network and the local discrimination network multiple times to make the global discrimination network and the local discrimination network have better discrimination capabilities; then, fix the network parameters of the global discrimination network and the local discrimination network unchanged, and update the network parameters of the generation network multiple times. [[ID=##**##]]

[0142] Before this implementation method, the network parameters of the global discrimination network and the local discrimination network in this implementation method have been updated multiple times by traversing some data pairs in the training dataset. This implementation method specifically elaborates on the process of fixing the network parameters of the global discrimination network and the local discrimination network unchanged and updating the network parameters of the generation network multiple times based on some unused data pairs.

[0143] Next, taking the global discrimination network as an example, the update process of the network parameters of the global discrimination network before this implementation method is elaborated.

[0144] a1: Input N first video images in the first data pair into the generation network G to obtain N random video images x f .

[0145] a2: Input the N random video images x f into the global discrimination network D1 to obtain N discrimination results D1(x f ).

[0146] a3: Input the N second video images x in the first data pairr Input the global discriminative network D1 to obtain N discriminative results D1(x r ).

[0147] a4: Calculate the loss value of the global discriminative network D1 based on the N discriminative results D1(x f ) and the N discriminative results D1(x r ).

[0148] Specifically, the loss value of the global discriminative network D1 is calculated using the following formulas (2), (3), and (4):

[0149]

[0150]

[0151]

[0152] where D c (x r , x f ) is used to represent the discrimination ability of the global discriminator for the second video image. The larger the value of D c (x r , x f ), the greater the probability that the second video image is discriminated as a real image. D c (x f , x r ) is used to represent the discrimination ability of the global discriminator for the random video image. The larger the value of D c (x f , x r [[ID=5~1]]), the greater the probability that the random video image is discriminated as a real image.

[0153] represents the expected value of the N discriminative results D1(x f ). σ(D1(x r )) represents the activation function value of the N discriminative results D1(x r ); represents the expected value of the N discriminative results D1(x r ). σ(D1(x f )) represents the activation function value of the N discriminative results D1(x f ).

[0154] a5: Update the network parameters of the global discriminative network D1 based on the loss value of the global discriminative network D1 to obtain the globally discriminative network D1 with updated parameters.

[0155] The above process is a process for updating the network parameters of the global discriminant network D1 once. After that, replace the first data pair in step a1 with the second data pair, and repeat the above steps to achieve another update of the network parameters of the global discriminant network D1 after the parameter update, until the network parameters are updated for the first preset number of times.

[0156] It should be noted that the process of updating the network parameters of the local discriminant network D2 is roughly the same as the process of updating the network parameters of the above global discriminant network. The differences are that the data pairs used for parameter update are different, the input to the local discriminant network D2 is the local image of the video image, and the loss function used to calculate the loss value of the local discriminant network D2 is different. Specifically, the loss function used to calculate the loss value of the local discriminant network D2 is as shown in the following formula (7):

[0157]

[0158] where D2(x r ) represents the discriminant result obtained after the local image of the second video image in the data pair is input into the local discriminant network D2, and D2(x f ) represents the discriminant result obtained after the local image of a random video image is input into the local discriminant network D2. The random video image is the video image obtained after the first video image in the data pair passes through the generation network.

[0159] S9025. Calculate the target loss value of the denoising network model according to the first discriminant result, the second discriminant result, and the third discriminant result.

[0160] Here, the training of the denoising network model, that is, the generation network, is guided by the discriminant results of two discriminant networks (i.e., the global discriminant network D1 and the local discriminant network D2). For the specific implementation process, please refer to Embodiment 2.

[0161] S9026. Update the network parameters of the denoising network model according to the target loss value to obtain the trained denoising network model.

[0162] Step S9026 may specifically include: after updating the network parameters of the denoising network model according to the target loss value, determine whether the updated denoising network model meets the preset conditions. If it meets the preset conditions, determine the updated denoising network model as the trained denoising network model; otherwise, update the updated denoising network model again until the updated denoising network model meets the preset conditions.

[0163] Among them, updating the updated denoising network model again specifically means: after updating the updated denoising network model to the denoising network model, return to execute step S9021 and subsequent steps. At this time, the data pair in step S9021 is replaced with the next data pair in the training data set.

[0164] In a specific embodiment, the preset condition is that the target loss value of the denoising network model after this update is equal to the target loss value of the denoising network model after its previous update, or the preset condition is that the cumulative number of network parameter updates of the updated denoising network model reaches a second preset number of times.

[0165] In an example, if the target loss value of the denoising network model after this update is not equal to the target loss value of the denoising network model after its previous update, and the number of network parameter updates of the denoising network model in this round has not reached a third preset number of times, then after updating the updated denoising network model to the denoising network model, return to execute step S9021 and subsequent steps until the target loss value of the denoising network model after this update is equal to the target loss value of the denoising network model after its previous update, or the number of network parameter updates of the denoising network model in this round reaches a third preset number of times.

[0166] Here, completing one round of network parameter update includes multiple network parameter updates for the global discriminant network, multiple network parameter updates for the local discriminant network, and multiple network parameter updates for the denoising network model.

[0167] Among them, the number of network parameter updates of the two discriminant networks is greater than the number of network parameter updates of the denoising network model. The data used for updating the network parameters of different networks is different. It should be noted that during the process of updating the network parameters of one network, the network parameters of the other two networks remain fixed.

[0168] When the number of network updates of the denoising network model in this round reaches a third preset number of times and the target loss value of the denoising network model after this update is not equal to the target loss value of the denoising network model after its previous update, then the next round of network parameter update process will be executed until the target loss value of the denoising network model after this update is equal to the target loss value of the denoising network model after its previous update. It should be noted that at this time, the loss values of the global discriminant network and the local discriminant network no longer decrease.

[0169] In a possible implementation manner, the implementation process of step S9025 may specifically include:

[0170] b1: Calculate a first loss value, where the first loss value represents the feature distance between the first video image and the fifth video image.

[0171] It should be noted that since the data pairs in the training data are unpaired and there is no real image (the second video image) corresponding to the first video image, in order to ensure the consistency of the video image content before and after the output of the generation network, the embodiment of the present application uses the perceptual loss function to calculate the first loss value representing the feature distance between the first video image and the third video image. The specific implementation process includes:

[0172] b11: Extract the image features of the first video image and the image features of the fifth video image respectively.

[0173] Specifically, use the pre-trained feature extraction model to extract the image features of the first video image and the image features of the fifth video image respectively. The feature extraction model can be the VGG-16 model.

[0174] b12: Calculate the feature distance between the first video image and the third video image according to the image features of the first video image and the image features of the third video image.

[0175] Specifically, use the following formula (8) to calculate the feature

[0176] distance:

[0177]

[0178] Among them, represents the image feature extracted by the j-th convolutional layer after the i-th max-pooling layer of the first video image through the feature extraction model. I represents the first video image; represents the image feature extracted by the j-th convolutional layer after the i-th max-pooling layer of the third video image through the feature extraction model. G(I) represents the third video image obtained after the first video image passes through the generation network. y represents the height of the image, and the value range is from 1 to H i,j , x represents the width of the image, and the value range is from 1 to w i,j , w i,j H i,j is used to represent the area of the image, and L p represents the first loss value, that is, the feature distance between the first video image and the third video image.

[0179] b13: Determine the feature distance between the first video image and the third video image as the first loss value.

[0180] b2: Calculate the second loss value according to the first discrimination result and the second discrimination result.

[0181] In a possible implementation manner, the implementation process of step b2 may include:

[0182] b21: Calculate the first expected value of the first discrimination result and the second expected value of the second discrimination result respectively.

[0183] b22: Based on a preset activation function, calculate the first function value of the first discrimination result and the second function value of the second discrimination result respectively.

[0184] b23: Calculate the second loss value according to the first expected value, the second expected value, the first function value and the second function value.

[0185] It should be noted that the above steps b21 to b23 can be represented by formula (9), that is, the following formula (9) is used to calculate the second loss value:

[0186]

[0187] Here, D c (x f , x r ) can be represented by the above formula (4), and D c (x r , x f ) can be represented by the above formula (5) and formula (6), that is:

[0188]

[0189]

[0190] At this time, D1(x f ) represents the first discrimination result, and D1(x r ) represents the second discrimination result. represents the second loss value.

[0191] b3: Calculate the third loss value according to the third discrimination result.

[0192] Specifically, the following formula (10) is used to calculate the third loss value:

[0193]

[0194] Among them, D2(x f ) represents the third discrimination result. represents the third loss value.

[0195] b4: Sum the first loss value, the second loss value and the third loss value to obtain the target loss value of the denoising network model.

[0196] Here, the target loss value

[0197] It should be understood that the sequence numbers of the steps in the above embodiments do not indicate the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0198] Corresponding to the method described in the above embodiments, Figure 4 The structural block diagram of the vehicle-mounted video stabilization device provided by the embodiment of the present application is shown. For the convenience of description, only the parts related to the embodiment of the present application are shown.

[0199] Referring to Figure 4 , the vehicle-mounted video stabilization device 400 may include: a first acquisition module 410, a network processing module 420, a smoothing processing module 430, a calculation module 440, and a motion compensation module 450.

[0200] Among them, the acquisition module 410 is used to acquire the vehicle-mounted video of the vehicle.

[0201] The network processing module 420 is used to input the vehicle-mounted video into the first neural network to obtain multiple grid-shaped first motion vector maps, where the first motion vector map is used to represent the offset vector corresponding to each grid of the subsequent frame image relative to the previous frame image in each adjacent two-frame image of the vehicle-mounted video.

[0202] The smoothing processing module 430 is used to perform smoothing processing on the first motion vector map to obtain a processed second motion vector map.

[0203] The calculation module 440 is used to calculate a first motion optical flow map according to the second motion vector map, where the first motion optical flow map is used to represent the motion displacement corresponding to each pixel point of the subsequent frame image relative to the previous frame image in each adjacent two-frame image of the vehicle-mounted video.

[0204] The motion compensation module 450 is used to perform motion compensation on the frame images in the vehicle-mounted video according to the first motion optical flow map to obtain a stabilized video.

[0205] In a possible implementation manner, the smoothing processing module 430 may specifically be used for:

[0206] For each frame image of the vehicle-mounted video, obtain the first motion vector maps corresponding to the previous K frame images and the subsequent K frame images of the current frame image, where K is a positive integer.

[0207] Perform weighted fusion calculation on the first motion vector maps corresponding to the previous K frame images and the subsequent K frame images of the current frame image to obtain a second motion vector map.

[0208] In a possible implementation manner, the vehicle-mounted video stabilization device 400 further includes: a speed acquisition module and a matching module.

[0209] A speed acquisition module, configured to acquire a target moving speed, where the target moving speed is the moving speed of a camera device that captures an in-vehicle video during the capture of the in-vehicle video;

[0210] A matching module, configured to find a neural network that matches the target moving speed from multiple neural networks, and use the matched neural network as the first neural network.

[0211] In a possible implementation, the camera device is a camera installed on a vehicle, and the speed acquisition module is specifically configured to: when the camera device captures the in-vehicle video, acquire the driving speed of the vehicle and determine the driving speed as the target moving speed.

[0212] In a possible implementation, the in-vehicle video stabilization device 400 further includes: a preprocessing module, configured to input the in-vehicle video into a trained non-stabilized video classification model for processing to obtain a probability value that the in-vehicle video is a non-stabilized video; and when the probability value is greater than a set threshold, transfer to the network processing module 420.

[0213] In a possible implementation, the in-vehicle video stabilization device 400 further includes: a denoising module, a first training module, and a second training module.

[0214] The denoising module is configured to perform denoising processing on the in-vehicle video based on a trained denoising network model to obtain a denoised in-vehicle video;

[0215] The first training module is configured to obtain a training data set, where the training data set includes multiple data pairs, each data pair includes N first video images and N second video images, the light brightness value corresponding to the first video image is less than the light brightness value corresponding to the second video image, the first video image and the second video image each correspond to a different shooting object, and N is a positive integer;

[0216] The second training module is configured to perform iterative training on a to-be-trained denoising network model based on the training data set to obtain a trained denoising network model.

[0217] In a possible implementation, the second training module is specifically configured to: input the first video image in the data pair into the denoising network model to obtain a third video image; input the third video image into a global discriminant network to obtain a first discriminant result; input the second video image in the data pair into the global discriminant network to obtain a second discriminant result; input a local image of the third video image into a local discriminant network to obtain a third discriminant result; calculate a target loss value of the denoising network model according to the first discriminant result, the second discriminant result, and the third discriminant result; update the network parameters of the denoising network model according to the target loss value to obtain a trained denoising network model.

[0218] In a possible implementation, the denoising module is specifically configured to: for each frame image of the vehicle-mounted video, calculate the average light brightness value corresponding to the frame image based on the brightness value of each pixel point in the frame image; based on the trained denoising network model, perform denoising processing on each frame image in the vehicle-mounted video whose average light brightness value is less than the first preset threshold, and obtain the vehicle-mounted video after denoising processing.

[0219] In a possible implementation, the calculation module 440 includes:

[0220] The first calculation unit is configured to perform a linear transformation on the second motion vector map to obtain the homography matrix of each grid in the second motion vector map.

[0221] The second calculation unit is configured to calculate the first motion optical flow map according to the homography matrix of each grid in the second motion vector map.

[0222] In a possible implementation, the second calculation unit may specifically be configured to:

[0223] For each grid in the second motion vector map, obtain the homogeneous coordinates corresponding to the multiple pixel points included in the projection of the grid onto the image.

[0224] Calculate the first motion optical flow map according to the homography matrix of the grid and the homogeneous coordinates corresponding to the multiple pixel points included in the projection of the grid onto the image.

[0225] In a possible implementation, the vehicle-mounted video stabilization device 400 according to the embodiment of the present application further includes:

[0226] The second acquisition module is configured to acquire a training data set, the training data set includes multiple data pairs, and each data pair includes a first image and a second image, wherein the second image has jitter relative to the first image;

[0227] The network training module is configured to train an unsupervised neural network based on the training data set to obtain a trained first neural network.

[0228] In a possible implementation, the network training module may specifically be configured to:

[0229] Input the first image and the second image in the data pair into the unsupervised neural network to obtain a grid-shaped third motion vector map, and the third motion vector map is used to represent the offset vector of each grid of the second image relative to the first image.

[0230] Calculate the second motion optical flow map according to the third motion vector map, and the second motion optical flow map is used to represent the motion displacement corresponding to each pixel point of the second image relative to the first image.

[0231] Perform bilinear interpolation on the second image according to the second motion optical flow map to obtain the third image after the second image moves.

[0232] Calculate the loss value between the first image and the third image.

[0233] Update the network parameters of the unsupervised network based on the loss value to obtain the first neural network.

[0234] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units, due to being based on the same concept as the method embodiment of this application, for their specific functions and the technical effects brought, please refer to the method embodiment part specifically, and will not be elaborated here.

[0235] Those skilled in the art can clearly understand that for the convenience and conciseness of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiment and will not be elaborated here.

[0236] The embodiment of this application also provides a vehicle. Refer to Figure 5 , in which the vehicle includes a terminal device 500, and the terminal device 500 may include: at least one processor 510, a memory 520, and a computer program stored in the memory 520 and executable on the at least one processor 510. When the processor 510 executes the computer program, it implements the steps in any of the above method embodiments, such as Figure 1 the steps S201 to S205 in the embodiment shown. Or, when the processor 510 executes the computer program, it implements the functions of each module / unit in the above device embodiments, such as Figure 4 the functions of the modules 410 to 450 shown.

[0237] Exemplarily, a computer program can be divided into one or more modules / units. One or more modules / units are stored in the memory 520 and executed by the processor 510 to complete this application. The one or more modules / units can be a series of computer program segments capable of performing specific functions, and these program segments are used to describe the execution process of the computer program in the terminal device 500.

[0238] Those skilled in the art can understand that Figure 5 merely examples of the terminal device, which do not constitute a limitation on the terminal device. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, buses, etc.

[0239] The processor 510 can be a central processing unit (CPU), or can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0240] The memory 520 can be an internal storage unit of the terminal device or an external storage device of the terminal device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. The memory 520 is used to store the computer program and other programs and data required by the terminal device. The memory 520 can also be used to temporarily store data that has been output or will be output.

[0241] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, the buses in the drawings of this application are not limited to only one bus or one type of bus.

[0242] The vehicle video image stabilization method provided by the embodiments of this application can be applied to terminal devices such as computers, tablet computers, laptop computers, netbooks, personal digital assistants (PDAs), etc. The embodiments of this application do not impose any restrictions on the specific types of terminal devices.

[0243] In the above embodiments, the descriptions of each embodiment have their own focuses. For parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0244] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0245] In the embodiments provided by this application, it should be understood that the disclosed terminal devices, apparatuses, and methods can be implemented in other ways. For example, the terminal device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0246] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units. That is, they can be located in one place, or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0247] In addition, the functional units in each embodiment of this application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0248] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-described embodiment methods of this application, it can also be completed by a computer program instructing relevant hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by one or more processors, the steps of the above-described various method embodiments can be implemented.

[0249] Similarly, as a computer program product, when the computer program product runs on a terminal device, it enables the terminal device to implement the steps in the above-described various method embodiments when executed.

[0250] Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice within the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0251] The above-described embodiments are only used to illustrate the technical solutions of this application, rather than to limit it; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A vehicle-mounted video stabilization method, characterized in that: include: Obtain vehicle-mounted video; Inputting the vehicle video into a first neural network to obtain a plurality of first motion vector maps in a grid format, wherein the first motion vector map is used to represent an offset vector corresponding to each grid of a subsequent frame image relative to a previous frame image between two adjacent frames in the vehicle video, and outputting a first motion vector map in a grid format for each adjacent frame image after the first neural network passes through the first two frames; performing smoothing processing on the first motion vector map to obtain a processed second motion vector map; Calculating a first motion optical flow map based on the second motion vector map, wherein the first motion optical flow map is used to represent the motion displacement corresponding to each pixel point of the latter frame image relative to the previous frame image of each two adjacent frames in the in-vehicle video; performing motion compensation on the frame images in the vehicle-mounted video according to the first motion optical flow map to obtain a stabilized video; The vehicle-mounted video stabilization method further includes: Obtaining a target moving speed, where the target moving speed is the moving speed of a camera device that shoots the vehicle-mounted video when shooting the vehicle-mounted video; A neural network that matches the target moving speed is searched from multiple neural networks, and the matched neural network is used as the first neural network.

2. The vehicle-mounted video stabilization method according to claim 1, wherein: The step of smoothing the first motion vector map to obtain a processed second motion vector map includes: For each frame of the vehicle-mounted video, obtain the first motion vector map corresponding to each of the K frames before and after the current frame, where K is a positive integer; The first motion vector maps corresponding to the K frames before and after the current frame image are weightedly fused to obtain the second motion vector map.

3. The vehicle-mounted video stabilization method according to claim 1, wherein: The camera device is a camera installed on a vehicle, and obtaining the target moving speed includes: When the camera device captures the vehicle-mounted video, the driving speed of the vehicle is acquired, and the driving speed is determined as the target moving speed.

4. The vehicle-mounted video stabilization method according to claim 1, wherein: Before inputting the vehicle video into the first neural network, the method further includes: Inputting the vehicle-mounted video into a trained non-stabilized video classification model for processing to obtain a probability value of the vehicle-mounted video being a non-stabilized video; If the probability value is greater than the set threshold, the step of inputting the vehicle video into the first neural network and subsequent steps are executed.

5. The vehicle-mounted video stabilization method according to claim 1, wherein: Before the vehicle-mounted video is input into the first neural network, the method further includes performing denoising processing on the vehicle-mounted video based on the trained denoising network model to obtain a denoised vehicle-mounted video; The training process of the denoising network model includes: Obtaining a training data set, wherein the training data set includes a plurality of data pairs, each of the data pairs includes N first video images and N second video images, the light brightness value corresponding to the first video image is smaller than the light brightness value corresponding to the second video image, the first video image and the second video image each correspond to a different photographed object, and N is a positive integer; The denoising network model to be trained is iteratively trained based on the training data set to obtain a trained denoising network model.

6. The vehicle-mounted video stabilization method according to claim 5, wherein: The iterative training of the denoising network model to be trained based on the training data set to obtain the trained denoising network model includes: Inputting the first video image in the data pair into the denoising network model to obtain a third video image; Inputting the third video image into a global discriminant network to obtain a first discriminant result; Inputting the second video image in the data pair into the global discriminant network to obtain a second discrimination result; Inputting the partial image of the third video image into the local discrimination network to obtain a third discrimination result; Calculating a target loss value of the denoising network model according to the first discrimination result, the second discrimination result, and the third discrimination result; The network parameters of the denoising network model are updated according to the target loss value to obtain a trained denoising network model.

7. The vehicle-mounted video stabilization method according to claim 5, wherein: The denoising process of the vehicle-mounted video based on the trained denoising network model includes: For each frame of the vehicle-mounted video, based on the brightness value of each pixel in the frame, calculate the average light brightness value corresponding to the frame; Based on the trained denoising network model, denoising is performed on each frame image in the in-vehicle video whose average light brightness value is less than a first preset threshold to obtain the in-vehicle video after denoising.

8. The vehicle-mounted video stabilization method according to claim 1, wherein: The step of calculating the first motion optical flow map according to the second motion vector map includes: Performing a straight-line linear transformation on the second motion vector map to obtain a homography matrix for each grid in the second motion vector map; The first motion optical flow map is obtained by calculation according to the homography matrix of each grid in the second motion vector map.

9. The vehicle-mounted video stabilization method according to claim 8, wherein: The calculating the first motion optical flow map according to the homography matrix of each grid in the second motion vector map includes: For each grid in the second motion vector map, obtaining homogeneous coordinates corresponding to a plurality of pixel points included in the grid projected onto the image; The first motion optical flow map is calculated based on the homography matrix of the grid and homogeneous coordinates corresponding to a plurality of pixel points included in the projection of the grid onto the image.

10. The vehicle-mounted video stabilization method according to claim 1, wherein: Before obtaining the vehicle-mounted video, the method further includes: Acquire a training data set, where the training data set includes a plurality of data pairs, each data pair includes a first image and a second image, wherein the second image is jittered relative to the first image; An unsupervised neural network is trained based on the training data set to obtain the trained first neural network.

11. The vehicle-mounted video stabilization method according to claim 10, wherein: The step of training the unsupervised neural network based on the training data set to obtain the trained first neural network includes: Inputting the first image and the second image in the data pair into an unsupervised neural network to obtain a third motion vector map in a grid shape, wherein the third motion vector map is used to represent an offset vector of each grid of the second image relative to the first image; Calculating a second motion optical flow map according to the third motion vector map, where the second motion optical flow map is used to represent the motion displacement corresponding to each pixel point of the second image relative to the first image; performing a bilinear interpolation operation on the second image according to the second motion optical flow map to obtain a third image after the second image moves; calculating a loss value between the first image and the third image; The network parameters of the unsupervised neural network are updated based on the loss value to obtain the first neural network.

12. A vehicle-mounted video stabilization device, characterized in that: include: A first acquisition module is used to acquire the vehicle's onboard video; a network processing module, configured to input the vehicle-mounted video into a first neural network to obtain a plurality of first motion vector maps in a grid format, wherein the first motion vector map is used to represent an offset vector corresponding to each grid of a subsequent frame image relative to a previous frame image between two adjacent frames of the vehicle-mounted video, and output a first motion vector map in a grid format for each adjacent frame of the video after the first neural network passes through the first two frames; a smoothing processing module, configured to perform smoothing on the first motion vector map to obtain a processed second motion vector map; a calculation module, configured to calculate a first motion optical flow map based on the second motion vector map, wherein the first motion optical flow map is used to represent the motion displacement corresponding to each pixel point of the latter frame image relative to the previous frame image of each two adjacent frames in the in-vehicle video; a motion compensation module, configured to perform motion compensation on the frame images in the vehicle-mounted video according to the first motion optical flow map to obtain a stabilized video; A speed acquisition module is used to acquire a target moving speed, where the target moving speed is the moving speed of the camera device that shoots the vehicle-mounted video when shooting the vehicle-mounted video; The matching module is used to search for a neural network that matches the target moving speed from multiple neural networks, and use the matched neural network as the first neural network.

13. A vehicle comprising a terminal device, the terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the vehicle-mounted video stabilization method according to any one of claims 1 to 11 is implemented.

14. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the vehicle-mounted video stabilization method according to any one of claims 1 to 11 is implemented.

Images