Video coding method and device, equipment and storage medium

Abstract

The invention discloses a video coding method and device, equipment and a storage medium. The method comprises the following steps: acquiring an external environment image of a vehicle; according to the current driving state of the vehicle, carrying out region division on the vehicle exterior environment image; compensating the image content complexity of each area to obtain the total image content complexity of the external environment image; according to the total image content complexity, obtaining a target code rate of the vehicle exterior environment image; and coding the external environment image according to the target code rate. By adopting the embodiment of the invention, the phenomenon that the network bandwidth occupied by the video is relatively large can be effectively avoided while the video quality is guaranteed, the video transmission speed is improved, the control experience and judgment accuracy of a remote control driver are further improved, and potential safety hazards are reduced.

Description

Technical Field

[0001] The present invention relates to the field of video technology, and in particular, to a video encoding method, apparatus, device, and storage medium. Background Art

[0002] In 5G remote driving, vehicle videos need to be re-encoded and then transmitted over the network to the remote control cockpit. Currently, when re-encoding vehicle videos, a target bitrate needs to be determined, and the vehicle videos are re-encoded according to the target bitrate. However, if the target bitrate is too large, it will cause the network bandwidth occupied by the video to be large, affecting the video transmission speed. If the target bitrate is too small, the video quality will be reduced. The reduction of both the video transmission speed and the video quality will affect the control experience and judgment accuracy of the remote control driver, and is likely to cause potential safety hazards. Summary of the Invention

[0003] The present invention provides a video encoding method, apparatus, device, and storage medium to solve the problem in the prior art that a reasonable target bitrate cannot be selected to encode vehicle videos.

[0004] To achieve the above object, an embodiment of the present invention provides a video encoding method, including:

[0005] Obtain an external environment image of the vehicle;

[0006] Divide the external environment image according to the current driving state of the vehicle;

[0007] Compensate the image content complexity of each region to obtain the total image content complexity of the external environment image;

[0008] Obtain the target bitrate of the external environment image according to the total image content complexity;

[0009] Encode the external environment image according to the target bitrate.

[0010] As an improvement of the above solution, the compensating the image content complexity of each region to obtain the total image content complexity of the external environment image includes:

[0011] Obtain the weighted compensation value of each region; wherein, the more interested the driver is in the region, the larger the weighted compensation value is;

[0012] Divide each region into multiple encoding blocks, and obtain the image content complexity of each encoding block;

[0013] Use the weighted compensation value to perform weighted compensation on the image content complexity of each encoding block to obtain the image content complexity of each encoding block after weighted compensation;

[0014] Add up the complexity of each weighted-compensated image content to obtain the total image content complexity of the external environment image of the vehicle.

[0015] As an improvement to the above solution, the regional division of the external environment image of the vehicle according to the current driving state of the vehicle includes:

[0016] When the vehicle is in a stationary state, divide the external environment image evenly into first regions of no interest.

[0017] When the vehicle is in a driving state, divide the region in the driving direction of the vehicle in the external environment image into regions to be divided, and divide the remaining regions in the external environment image into second regions of no interest.

[0018] According to the preset correspondence between the vehicle speed and the driver's field of view focusing range, calculate the current driver's field of view focusing range at the current vehicle speed; wherein, in the correspondence, the vehicle speed and the driver's field of view focusing range are inversely correlated.

[0019] According to the current driver's field of view focusing range, obtain the current driver's field of view focusing region in the region to be divided, divide the current driver's field of view focusing region into regions of interest, and divide the remaining regions in the region to be divided into third regions of no interest. The third regions of no interest include or do not include multiple sub-regions of no interest, and each sub-region of no interest is composed of a partial region on the left side outside the region of interest and a partial region on the right side outside the region of interest.

[0020] As an improvement to the above solution, after obtaining the current driver's field of view focusing region in the region to be divided, if the vehicle is in a turning state, the video encoding method further includes:

[0021] Based on the left-wheel turning motion trajectory and the right-wheel turning motion trajectory of the vehicle, obtain the central turning motion trajectory of the vehicle.

[0022] According to the central turning motion trajectory, obtain the driver's visual center of gravity of the vehicle in the turning state.

[0023] According to the driver's visual center of gravity of the vehicle in the turning state, calculate the visual center of gravity offset value of the driver's visual center of gravity of the vehicle in the straight-ahead state compared to the driver's visual center of gravity of the vehicle in the turning state.

[0024] Adjust the current driver's field of view focus area according to the visual center of gravity offset value, divide the adjusted current driver's field of view focus area into the regions of interest, and divide the remaining areas of the area to be divided into the third non - interested areas.

[0025] As an improvement to the above - mentioned solution, the encoding of the out - of - vehicle environment image according to the target bitrate includes:

[0026] Obtain the compensation method and compensation quantization parameter for each region;

[0027] Compensate the reference quantization parameter of each region by using the compensation method and the compensation quantization parameter to obtain each initial quantization parameter; among them, the smaller the initial quantization parameter of the region that the driver is more interested in.

[0028] Calculate the initial bitrate of the out - of - vehicle environment image under the initial quantization parameter;

[0029] When the difference between the initial bitrate and the target bitrate is within a preset difference range, encode the out - of - vehicle environment image by using the initial quantization parameter.

[0030] As an improvement to the above - mentioned solution, the video encoding method further includes:

[0031] When the difference between the initial bitrate and the target bitrate is not within the preset difference range, update each reference quantization parameter according to the relationship between the initial bitrate and the target bitrate to obtain each target reference quantization parameter;

[0032] Compensate the target reference quantization parameter of each region by using the compensation method and the compensation quantization parameter to obtain optimized quantization parameters;

[0033] Calculate the optimized bitrate of the out - of - vehicle environment image under the optimized quantization parameter;

[0034] If the difference between the optimized bitrate and the target bitrate is within the preset difference range, or the area of the first region is equal to the area of the second region, then encode the out - of - vehicle environment image by using the optimized quantization parameter;

[0035] Among them, the first region is the region in the current driving direction area where the compensation method of reduction is used with the compensation quantization parameter, the second region is the region in the current driving direction area where the compensation method of increase is used with the compensation quantization parameter, and the current driving direction area is the area in the out - of - vehicle environment image that is in the vehicle driving direction.

[0036] As an improvement to the above - mentioned solution, the video encoding method further includes:

[0037] If the difference between the optimized bitrate obtained after the latest compensation and the target bitrate is not within the preset difference range, and the area of the first region is not equal to the area of the second region, then adjust the compensation quantization parameter according to the relationship between the optimized bitrate obtained after the latest compensation and the target bitrate and the relationship between the area of the first region and the area of the second region;

[0038] After each adjustment of the compensation quantization parameter, use the latest adjusted compensation quantization parameter to compensate the target reference quantization parameter of each region until the difference between the optimized bitrate obtained after the latest compensation and the target bitrate is within the preset difference range, and then stop adjusting the compensation quantization parameter;

[0039] After stopping adjusting the compensation quantization parameter, encode the vehicle exterior environment image using the latest compensated optimized quantization parameter.

[0040] As an improvement to the above solution, the adjusting the compensation quantization parameter according to the relationship between the optimized bitrate obtained after the latest compensation and the target bitrate and the relationship between the area of the first region and the area of the second region includes:

[0041] When the optimized bitrate obtained after the latest compensation is less than the target bitrate, if the area of the first region is greater than the area of the second region, then increase the compensation quantization parameter, and if the area of the first region is less than the area of the second region, then decrease the compensation quantization parameter;

[0042] When the optimized bitrate obtained after the latest compensation is greater than the target bitrate, if the area of the first region is greater than the area of the second region, then decrease the compensation quantization parameter, and if the area of the first region is less than the area of the second region, then increase the compensation quantization parameter.

[0043] As an improvement to the above solution, the obtaining the vehicle exterior environment image includes:

[0044] Obtain multiple original vehicle exterior environment images collected by the vehicle's acquisition device;

[0045] Calibrate the acquisition device to obtain the internal parameter data and external parameter data of the acquisition device;

[0046] According to the internal parameter data and the internal parameter data, project all the original vehicle exterior environment images onto the same coordinate system;

[0047] Fuse and stitch the original vehicle exterior environment images belonging to the same perspective in the same coordinate system to obtain the vehicle exterior environment images under different perspectives.

[0048] To achieve the above object, an embodiment of the present invention further provides a video encoding device, including:

[0049] An out-vehicle environment image acquisition module, configured to acquire an out-vehicle environment image of the vehicle;

[0050] A region division module, configured to divide the out-vehicle environment image according to the current driving state of the vehicle;

[0051] An image content complexity compensation module, configured to compensate the image content complexity of each region to obtain the total image content complexity of the out-vehicle environment image;

[0052] A target bit rate acquisition module, configured to acquire a target bit rate of the out-vehicle environment image according to the total image content complexity;

[0053] An encoding module, configured to encode the out-vehicle environment image according to the target bit rate.

[0054] To achieve the above object, an embodiment of the present invention further provides a video encoding device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the above video encoding method is implemented.

[0055] To achieve the above object, an embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program runs, it controls the device where the computer-readable storage medium is located to execute the above video encoding method.

[0056] Compared with the prior art, a video encoding method, device, equipment, and storage medium provided by an embodiment of the present invention acquire an out-vehicle environment image of the vehicle; divide the out-vehicle environment image according to the current driving state of the vehicle; compensate the image content complexity of each region to obtain the total image content complexity of the out-vehicle environment image; acquire a target bit rate of the out-vehicle environment image according to the total image content complexity; and encode the out-vehicle environment image according to the target bit rate. It can be seen that by compensating the image content complexity of each region of the out-vehicle environment image, the embodiment of the present invention can assign different image content complexities to each region, so that the total image content complexity of the out-vehicle environment image better meets the video transmission requirements and video quality requirements in remote driving, thereby making the target bit rate determined according to the total image content complexity more reasonable, effectively avoiding the phenomenon that the network bandwidth occupied by the video is large while ensuring the video quality, improving the video transmission speed, further improving the control experience and judgment accuracy of remote control drivers, and reducing potential safety hazards. Description of the Drawings

[0057] Figure 1 is a flowchart of a video encoding method provided by an embodiment of the present invention;

[0058] Figure 2 is a schematic diagram of area division when the vehicle is in a stationary state provided by an embodiment of the present invention;

[0059] Figure 3 is a schematic diagram of area division when the vehicle is in a driving state provided by an embodiment of the present invention;

[0060] Figure 4 is another schematic diagram of area division when the vehicle is in a driving state provided by an embodiment of the present invention;

[0061] Figure 5 is a schematic diagram of the field of view focusing range at different vehicle speeds provided by an embodiment of the present invention;

[0062] Figure 6 is a schematic diagram of the division of the area to be divided provided by an embodiment of the present invention;

[0063] Figure 7 is another schematic diagram of the division of the area to be divided provided by an embodiment of the present invention;

[0064] Figure 8 is a schematic diagram of the central turning motion trajectory provided by an embodiment of the present invention;

[0065] Figure 9 is a structural block diagram of a video encoding device provided by an embodiment of the present invention;

[0066] Figure 10 is a structural block diagram of a video encoding device provided by an embodiment of the present invention. Detailed Embodiments

[0067] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0068] See Figure 1 , Figure 1 is a flowchart of a video encoding method provided by an embodiment of the present invention. The video encoding method includes:

[0069] S1. Obtain an external environment image of the vehicle;

[0070] S2. Divide the external environment image of the vehicle according to the current driving state of the vehicle;

[0071] S3. Compensate the image content complexity of each region to obtain the total image content complexity of the external environment image of the vehicle;

[0072] S4. Obtain the target bit rate of the external environment image of the vehicle according to the total image content complexity;

[0073] S5. Encode the external environment image of the vehicle according to the target bit rate.

[0074] In the embodiment of the present invention, by dividing the external environment image of the vehicle according to the current driving state of the vehicle, the external environment image can be reasonably divided into regions, and then the image content complexity of each region can be compensated, so as to improve the image content complexity of the region that the driver is more interested in and reduce the image content complexity of the remaining regions. That is to say, the greater the image content complexity obtained after compensation for the region that the driver is more interested in, and then the total image content complexity of the external environment image can better meet the video transmission requirements and video quality requirements in remote driving, effectively avoid the phenomenon that the network bandwidth occupied by the video is relatively large while ensuring the video quality, improve the video transmission speed, and then improve the control experience and judgment accuracy of the remote control driver, and reduce potential safety hazards.

[0075] In an optional embodiment, the obtaining of the external environment image of the vehicle includes:

[0076] Obtain multiple original external environment images collected by the acquisition device of the vehicle;

[0077] Calibrate the acquisition device to obtain the internal parameter data and external parameter data of the acquisition device;

[0078] Project all the original external environment images onto the same coordinate system according to the internal parameter data and the internal parameter data;

[0079] Fuse and splice the original external environment images belonging to the same perspective in the same coordinate system to obtain the external environment images under different perspectives.

[0080] It can be understood that the acquisition devices (cameras) are distributed around the vehicle and can acquire the original environmental images from various perspectives around the vehicle. For example, they can be distributed in the front, rear, and sides of the vehicle. The acquisition device set in the front of the vehicle acquires the front-view original environmental image in front of the vehicle, the acquisition device set in the rear of the vehicle acquires the rear-view original environmental image behind the vehicle, and the acquisition device set in the side of the vehicle acquires the side-view original environmental image on the side of the vehicle. Then, the original external environmental image of the vehicle includes any one or a combination thereof: the front-view original environmental image, the rear-view original environmental image, and the side-view original environmental image.

[0081] After acquiring the original external environmental images from various perspectives, calibrate the acquisition devices to obtain the internal parameter data and external parameter data of the acquisition devices. According to the internal parameter data and the internal parameter data, project all the original external environmental images onto the same coordinate system, and fuse and splice the original external environmental images belonging to the same perspective in the same coordinate system to obtain the external environmental images from different perspectives, that is, obtain a front-view environmental image and / or a rear-view environmental image with high resolution and wide field of view. The external environmental image of the vehicle includes: the front-view environmental image and / or the rear-view environmental image.

[0082] Exemplarily, in the case of 6 cameras (including the left front camera, the front camera, the right front camera, the left rear camera, the rear camera, and the right rear camera), splice the three original environmental images acquired by the left front camera, the front camera, and the right front camera at the same moment into a front-view environmental image, and splice the three original environmental images acquired by the left rear camera, the rear camera, and the right rear camera at the same moment into a rear-view environmental image.

[0083] In the case of 4 cameras (including the left side camera, the front camera, the right side camera, and the rear camera), splice the right half of the original environmental image acquired by the left side camera, the original environmental image acquired by the front camera, and the left half of the original environmental image acquired by the right side camera at the same moment into a front-view environmental image. It can be understood that the parallax between the original environmental image acquired by the left / right side camera and the original environmental image acquired by the front camera is too large. If forced to splice, the original environmental image acquired by the left / right side camera will be severely distorted. Therefore, in the embodiment of the present invention, the part of the original environmental image acquired by the left / right side camera close to the original environmental image acquired by the front camera (the right half of the original environmental image acquired by the left side camera, the left half of the original environmental image acquired by the right side camera) is taken and spliced with the original environmental image acquired by the front camera.

[0084] In the case of four cameras, the left half of the original environmental image captured by the left-side camera, the original environmental image captured by the rear camera, and the right half of the original environmental image captured by the right-side camera at the same moment are stitched together to form a rear-view environmental image. It can be understood that the parallax between the original environmental image captured by the left / right-side camera and the original environmental image captured by the rear camera is too large. If forced to stitch, the original environmental image captured by the left / right-side camera will be severely distorted. Therefore, in the embodiments of the present invention, the parts of the original environmental image captured by the left / right-side camera that are close to the original environmental image captured by the rear camera (the left half of the original environmental image captured by the left-side camera, the right half of the original environmental image captured by the right-side camera) are taken and stitched with the original environmental image captured by the rear camera.

[0085] After stitching is completed, there is:

[0086]

[0087] Among them,

[0088] - is the resolution of the vehicle exterior environmental image at the same viewing angle after stitching N original vehicle exterior environmental images;

[0089] - is the sum of the resolutions of N original vehicle exterior environmental images added together;

[0090] The vehicle exterior environmental image stitched in the embodiments of the present invention removes the overlapping area of the image, reduces the image resolution without substantially losing the image clarity, thereby further saving the network bandwidth occupied by the video during the video transmission process and further improving the video transmission speed.

[0091] In an alternative embodiment, the dividing the vehicle exterior environmental image into regions according to the current driving state of the vehicle includes:

[0092] When the vehicle is in a stationary state, the vehicle exterior environmental image is evenly divided into first regions of no interest;

[0093] When the vehicle is in a driving state, the region in the vehicle exterior environmental image that is in the vehicle driving direction is divided into regions to be divided, and the remaining regions in the vehicle exterior environmental image are divided into second regions of no interest;

[0094] According to the preset correspondence between the vehicle speed and the driver's field of view focus range, calculate the current driver's field of view focus range at the current vehicle speed of the vehicle; wherein, in the correspondence, the vehicle speed and the driver's field of view focus range are inversely correlated;

[0095] According to the current driver's field of view focusing range, the current driver's field of view focusing area in the area to be divided is obtained, the current driver's field of view focusing area is divided into areas of interest, and the remaining area of the area to be divided is divided into a third area of non-interest. The third area of non-interest includes or does not include a plurality of sub-areas of non-interest, and each sub-area of non-interest is composed of a partial area located on the left side outside the area of interest and a partial area located on the right side outside the area of interest.

[0096] It can be understood that during the vehicle movement, the change in the field of view felt by the human eye from the image is different from the physical feeling in the real world. On the image, the human eye will feel that the vehicle speed is significantly "accelerated". For example, at the same vehicle speed of 10 km / h, the environmental change felt by the human eye in the real world is small, while the environmental change felt on the image is large. At this time, people will think that the vehicle speed is very fast. Especially under the rapid change of the scenes on both sides, it will cause a sense of oppression and dizziness to the remote driver. Therefore, in the embodiments of the present invention, the field of view focusing area that the driver can pay more attention to is adjusted according to the current driving state of the vehicle, which is more in line with the human eye visual law, makes the divided area more reasonable, and then can allocate different image content complexities and quantization parameters to different areas, and then adjust the clarity of different areas, which can not only optimize the network bandwidth occupied by the video during the video transmission process, but also relieve the discomfort of the remote driver.

[0097] In the embodiments of the present invention, when the vehicle is in a stationary state, the driver's attention (degree of interest) in the image of the external environment of the vehicle is the same, and there is no need to divide different areas. At this time, the driver may not be very interested in the image of the external environment of the vehicle either. Therefore, the image of the external environment of the vehicle is divided into a first area of non-interest, such as Figure 2 both the forward view environment image and the rear view environment image are divided into the first area of non-interest. Optionally, when the current gear of the vehicle is in the P gear or N gear, it is determined that the vehicle is in a stationary state.

[0098] When the vehicle is in a driving state, the driver pays more attention to the image of the external environment in the vehicle driving direction. Therefore, the area in the image of the external environment of the vehicle that is in the vehicle driving direction is divided into the area to be divided, and the remaining area in the image of the external environment of the vehicle is divided into a second area of non-interest; for example, when the current gear of the vehicle is in the D gear, it means that the vehicle is in a forward driving state, and the driver pays more attention to the image of the external environment in the vehicle forward direction. At this time, the forward view environment image is divided into the area to be divided, and the remaining area of the image of the external environment of the vehicle, that is, the rear view environment image, is divided into the second area of non-interest, such as Figure 3, divide the forward view environment image into areas to be divided, and divide the rear view environment image into second non - interested areas; when the current gear of the vehicle is in the R gear, it indicates that the vehicle is in reverse driving state, and the driver pays more attention to the out - of - vehicle environment image in the reverse direction. At this time, divide the rear view environment image into areas to be divided, and divide the remaining area of the out - of - vehicle environment image, that is, the forward view environment image, into second non - interested areas, such as Figure 4 , divide the rear view environment image into areas to be divided, and divide the forward view environment image into second non - interested areas.

[0099] It can be understood that the faster the vehicle speed, the farther the focus point of the human eye's vision is and the narrower the vision focus range is. In order to make the areas of interest to the driver more in line with the human eye's visual law, the areas to be divided are further divided according to the corresponding relationship between the vehicle speed and the driver's vision focus range. To determine the current driver's vision focus area in the areas to be divided, it is necessary to calculate the current driver's vision focus range at the current vehicle speed of the vehicle. However, due to the particularity of remote driving, the conventional vehicle speed - vision focus range model is not applicable. Therefore, in the embodiments of the present invention, the corresponding relationship between the vehicle speed and the driver's vision focus range needs to be analyzed based on the actual vehicle speed and the driver's vision focus range.

[0100] The corresponding relationship between the vehicle speed and the driver's vision focus range is obtained through the following steps: In the driverless test field, under the conditions of the same road conditions, road types, and the vision range of the out - of - vehicle environment image, collect the driver's vision focus range at different vehicle speeds and conduct quantitative analysis on it. Assume that the road is straight, there are few or no interference objects around, and the vision range of the out - of - vehicle environment image is 240°. According to the roadside marks noticed by the driver's feedback, obtain the vision focus range at different vehicle speeds, such as Figure 5 shown, and the recorded data is as shown in Table 1 below:

[0101] Table 1 Vehicle speed - vision focus range

[0102] Vehicle speed Field of view focusing range Stationary 96.52° 5 km / h 88.70° 10 km / h 80.87° 15 km / h 73.04° 20 km / h 62.61° 25 km / h 46.96°

[0103] According to the inverse - correlation relationship between the vehicle speed and the vision focus range, establish a non - linear model of the vehicle speed and the vision focus range, that is, the corresponding relationship between the vehicle speed and the vision focus range:

[0104]

[0105] Among them,

[0106] A0 - is the vision focus range when the vehicle is in a stationary state;

[0107] A v - is the vision focus range at the vehicle speed V;

[0108] V is the current vehicle speed;

[0109] α is the preset vehicle speed influence coefficient;

[0110] K A - is the preset correction coefficient;

[0111] Using the visual focus ranges at different vehicle speeds obtained from the previous unmanned test site tests, fitting a non-linear model of vehicle speed and visual focus range to obtain the preset vehicle speed influence coefficient and the preset correction coefficient, and then determining the non-linear model of vehicle speed and visual focus range, that is, the corresponding relationship between vehicle speed and visual focus range, according to the preset vehicle speed influence coefficient and the preset correction coefficient; substituting the current vehicle speed of the vehicle into the corresponding relationship, the current driver's visual focus range at the current vehicle speed of the vehicle can be calculated;

[0112] Then, according to the current driver's visual focus range, the current driver's visual focus area in the area to be divided is obtained; generally, the driver's visual center of gravity is defaulted to be at the center of the vehicle exterior environment image, so the center axis of the vehicle exterior environment image can be used as the central reference, and expand to the left and right to reach the current driver's visual focus area, that is, the current driver's visual focus area is obtained.

[0113] Specifically, taking the ratio of the visual focus range of the current driver at the vehicle stationary state to the visual focus range as the ratio of the current driver's visual focus area in the vehicle exterior environment image, obtaining the pixel width of the current driver's visual focus area according to the ratio, using the center axis of the vehicle exterior environment image as the central reference and expanding the same pixel width to both the left and right sides. When the total pixel width of the expansion is the pixel width of the current driver's visual focus area, the current driver's visual focus area is obtained, and then the current driver's visual focus area is divided into the area of interest, and the remaining area of the area to be divided is divided into the third non-interest area. As Figure 6 shown, the third non-interest area does not include multiple sub-non-interest areas, and the third non-interest area is composed of the area located on the left outside the area of interest and the area located on the right outside the area of interest.

[0114] Exemplarily, the current driver's visual focus area is calculated according to the following formula:

[0115]

[0116]

[0117]

[0118] ROI 第三不感兴趣区域

[0119] = [0, 0, ROI 第三不感兴趣区域|左 , H] ∪ [W - ROI 第三不感兴趣区域|右 , 0, ROI 第三不感兴趣区域|右 , H]

[0120] Wherein,

[0121] A - is the total field of view of the external environment image of the vehicle;

[0122] W - is the total pixel width of the external environment image of the vehicle;

[0123] H - is the total pixel height of the external environment image of the vehicle;

[0124] ROI 感兴趣区域|w - represents the pixel width of the third non - interested region located outside the left side of the region of interest;

[0125] ROI 感兴趣区域 - represents the range of the region of interest, which are respectively the x - coordinate of the upper - left corner of the region of interest, the y - coordinate of the upper - left corner of the region of interest, the width of the region of interest, and the height of the region of interest;

[0126] ROI 第三不感兴趣区域|左 - represents the pixel width of the third non - interested region located outside the right side of the region of interest;

[0127] ROI 第三不感兴趣区域|右 - represents the pixel width of the third non - interested region located outside the right side of the region of interest;

[0128] ROI 第三不感兴趣区域 - includes two parts: the range of the third non - interested region located outside the left side of the region of interest [0, 0, ROI 第三不感兴趣区域|左 , H] and the range of the third non - interested region located outside the right side of the region of interest [W - ROI 第三不感兴趣区域|右 , 0, ROI 第三不感兴趣区域|右 , H], wherein, [0, 0, ROI 第三不感兴趣区域|左 , H] are respectively the x - coordinate of the upper - left corner of the third non - interested region on the left, the y - coordinate of the upper - left corner of the third non - interested region on the left, the width of the third non - interested region on the left, and the height of the third non - interested region on the left; [W - ROI 第三不感兴趣区域|右 , 0, ROI 第三不感兴趣区域|右 , H] are respectively the x - coordinate of the upper - left corner of the third non - interested region on the right, the y - coordinate of the upper - left corner of the third non - interested region on the right, the width of the third non - interested region on the right, and the height of the third non - interested region on the right.

[0129] It can be understood that the expression form of each divided region is [x, y, w, h], and each region is a rectangle, where x is the x - coordinate of the upper - left corner of the region, y is the y - coordinate of the upper - left corner of the region, w is the width of the region, and h is the height of the region.

[0130] Of course, in order to make the complexity of the image content and the distribution of quantization parameters in each region more reasonable, the third region of no interest can be further divided so that the third region of no interest is composed of multiple sub - regions of no interest, and each sub - region of no interest is composed of a partial region on the left side outside the region of interest and a partial region on the right side outside the region of interest; among them, the sub - region of no interest that is farther away from the region of interest is less interesting to the driver.

[0131] Exemplarily, as Figure 7 shown, the third region of no interest is divided into a first sub - region of no interest and a second sub - region of no interest, where the first sub - region of no interest is the region adjacent to the region of interest, and the second sub - region of no interest is the region not adjacent to the region of interest.

[0132] Exemplarily, the third region of no interest is divided according to the following formula:

[0133]

[0134]

[0135] ROI 第一子不感兴趣区域

[0136] =[ROI 第二子不感兴趣区域|左 ,0,ROI 第一子不感兴趣区域|左 ,H]∪[W - ROI 第一子不感兴趣区域|右 - ROI 第二子不感兴趣区域|右 ,0,ROI 第一不感兴趣区域|右 ,H]

[0137] ROI 第二子不感兴趣区域

[0138] =[0,0,ROI 第二子不感兴趣区域|左 ,H]∪[W - ROI 第二子不感兴趣区域|右 ,0,ROI 第二子不感兴趣区域|右 ,H]

[0139] Among them,

[0140] W - is the total pixel width of the out - of - vehicle environment image;

[0141] H - is the total pixel height of the out - of - vehicle environment image;

[0142] ROI 感兴趣区域|w - Represents the pixel width of the region of interest;

[0143] ROI 第一子不感兴趣区域|左 - Represents the pixel width of the first sub - non - region of interest located to the left outside the region of interest;

[0144] ROI 第一子不感兴趣区域|右 - Represents the pixel width of the first sub - non - region of interest located to the right outside the region of interest;

[0145] ROI 第二子不感兴趣区域|左 - Represents the pixel width of the second sub - non - region of interest located to the left outside the region of interest;

[0146] ROI 第二子不感兴趣区域|右 - Represents the pixel width of the second sub - non - region of interest located to the right outside the region of interest;

[0147] ROI 第一子不感兴趣区域 - Consists of two parts: the range of the first sub - non - region of interest located to the left outside the region of interest [ROI 第二子不感兴趣区域|左 ,0,ROI 第一子不感兴趣区域|左 ,H] and the range of the first sub - non - region of interest located to the right outside the region of interest

[0148] [W - ROI 第一子不感兴趣区域|右 - ROI 第二子不感兴趣区域|右 ,0,ROI 第一不感兴趣区域|右 ,H], where, [ROI 第二子不感兴趣区域|左 ,0,ROI 第一子不感兴趣区域|左 ,H] are respectively the x - coordinate of the upper - left corner of the first sub - non - region of interest on the left, the y - coordinate of the upper - left corner of the first sub - non - region of interest on the left, the width of the first sub - non - region of interest on the left, and the height of the first sub - non - region of interest on the left; [W - ROI 第一子不感兴趣区域|右 - ROI 第二子不感兴趣区域|右 ,0,ROI 第一不感兴趣区域|右 ,H] are respectively the x - coordinate of the upper - left corner of the first sub - non - region of interest on the right, the y - coordinate of the upper - left corner of the first sub - non - region of interest on the right, the width of the first sub - non - region of interest on the right, and the height of the first sub - non - region of interest on the right.

[0149] ROI 第二子不感兴趣区域 - Consists of two parts: the range of the second sub - non - region of interest located to the left outside the region of interest [0,0,ROI 第二子不感兴趣区域|左 ,H] and the range of the second sub - non - region of interest located to the right outside the region of interest [W - ROI 第二子不感兴趣区域|右 ,0,ROI 第二子不感兴趣区域|右 ,H], where, [0,0,ROI 第二子不感兴趣区域|左, H] are respectively the x - coordinate of the upper - left corner of the second sub - non - interested region on the left, the y - coordinate of the upper - left corner of the second sub - interested region on the left, the width of the second sub - interested region on the left, and the height of the second sub - interested region on the left.

[0150] [W - ROI 第二子不感兴趣区域|右 , 0, ROI 第二子不感兴趣区域|右 , H] are respectively the x - coordinate of the upper - left corner of the second sub - non - interested region on the right, the y - coordinate of the upper - left corner of the second sub - interested region on the right, the width of the second sub - interested region on the right, and the height of the second sub - interested region on the right.

[0151] In an optional embodiment, after obtaining the current driver's field - of - view focus region in the region to be divided, if the vehicle is in a turning state, the video encoding method further includes:

[0152] Based on the left - wheel turning motion trajectory and the right - wheel turning motion trajectory of the vehicle, obtain the central turning motion trajectory of the vehicle;

[0153] According to the central turning motion trajectory, obtain the driver's visual center of gravity of the vehicle in the turning state;

[0154] According to the driver's visual center of gravity of the vehicle in the turning state, calculate the visual center - of - gravity offset value of the driver's visual center of gravity of the vehicle in the straight - running state compared to the driver's visual center of gravity of the vehicle in the turning state;

[0155] According to the visual center - of - gravity offset value, adjust the current driver's field - of - view focus region, divide the adjusted current driver's field - of - view focus region into the interested region, and divide the remaining region of the region to be divided into the third non - interested region.

[0156] It can be understood that when the vehicle turns, the driver's visual center of gravity will change along with the turning motion trajectory, and it shifts left / right from the central axis of the out - of - vehicle environment image. The region of interest of the driver will also shift accordingly. For example, when the vehicle turns left, the driver will pay more attention to the left side, and the driver's visual center of gravity will shift to the left. When the vehicle turns right, the driver will pay more attention to the right side, and the driver's visual center of gravity will shift to the right. When the vehicle turns, the left tire will form a left - wheel turning motion trajectory, and the right tire will form a right - wheel turning motion trajectory. In the embodiment of the present invention, the central turning motion trajectory of the vehicle is confirmed based on the left - wheel turning motion trajectory and the right - wheel turning motion trajectory. The central turning motion trajectory is the turning motion trajectory formed by the centers of the left and right tires.

[0157] Specifically, in combination with Figure 8 , the central turning motion trajectory is obtained through the following steps:

[0158] The projection of the center point between the left front tire and the right front tire onto the ground is taken as the coordinate origin O. This coordinate origin O is the starting point coordinate of the circle formed by the central turning motion trajectory. The forward direction is the positive direction of the Y-axis, and the rightward direction is the positive direction of the X-axis. There is:

[0159]

[0160]

[0161] p c = (-Lcotα, -L)

[0162] p s = (0, 0)

[0163] Among them,

[0164] L - is the wheelbase of the front and rear tires;

[0165] α Left - is the left wheel turning angle;

[0166] α Right - is the right wheel turning angle. When turning left, α Right < α Left When turning right, α Right > α Left ;

[0167] α - is the vehicle turning angle;

[0168] R c - is the radius of the circle formed by the central turning motion trajectory;

[0169] p c - is the center coordinate of the circle formed by the central turning motion trajectory;

[0170] p s - is the starting point coordinate of the circle formed by the central turning motion trajectory;

[0171] After obtaining the radius and center coordinate of the circle formed by the central turning motion trajectory according to the above formula, the curve of the circle formed by the central turning motion trajectory can be obtained Take the intersection point coordinate (x d , T d ) of this curve and the straight line Y = T d that is close to the starting point coordinate as the driver's visual center of gravity of the vehicle at the vehicle turning angle α. The value of T d is an empirical value. For example Figure 8 , the straight line Y = T d has intersection points (x d1 , T d ) and (x d2 , Td ),the intersection coordinate (x s , T d2 ) closest to the starting coordinate p d is used as the driver's visual center of gravity when the vehicle is at the vehicle turning angle α. It can be understood that T d represents the farthest distance that the driver focuses on, and the intersection coordinate (x d , T d ) can be understood as the driver's visual center of gravity at the farthest distance that the driver focuses on at the vehicle turning angle α;

[0172] According to the external parameters of camera calibration, the intersection coordinate (x d , T d ) is transformed into the pixel coordinate (x p , y p ) in the image pixel coordinate system, and the visual center of gravity offset value Offset x of the driver's visual center of gravity of the vehicle in the straight state compared to the driver's visual center of gravity of the vehicle in the turning state is calculated. Then, according to Offset x , the region of interest and the third non - interested region are adjusted, and there is:

[0173]

[0174]

[0175]

[0176]

[0177] ROI 第三不感兴趣区域|α

[0178] = [0, 0, ROI 第三不感兴趣区域|左|α , H] ∪ [W - ROI 第三不感兴趣区域|右|α , 0, ROI 第三不感兴趣区域|右|α , H]

[0179] Among them,

[0180] Offset x - is the visual center of gravity offset value;

[0181] W - is the total pixel width of the vehicle - external environment image;

[0182] x p - is the x - coordinate of the intersection coordinate of the central turning motion track and the curve with Y = T d in the image pixel coordinate system;

[0183] ROI 感兴趣区域|w|α- is the range of the region of interest (ROI) when the vehicle turning angle is α. are respectively the x coordinate of the upper left corner of the region of interest when the vehicle turning angle is α, the y coordinate of the upper left corner of the region of interest when the vehicle turning angle is α, the width of the region of interest when the vehicle turning angle is α, and the height of the region of interest when the vehicle turning angle is α.

[0184] ROI 第三不感兴趣区域|左|α - is the third non - interested region [0, 0, ROI 第三不感兴趣区域|左|α , H] on the left outside the region of interest and the third non - interested region [W - ROI 第三不感兴趣区域|右|α , 0, ROI 第三不感兴趣区域|右|α , H] on the right outside the region of interest when the vehicle turning angle is α, where [0, 0, ROI 第三不感兴趣区域|左|α , H] are respectively the x coordinate of the upper left corner of the third non - interested region on the left when the vehicle turning angle is α, the y coordinate of the upper left corner of the third non - interested region on the left when the vehicle turning angle is α, the width of the third non - interested region on the left when the vehicle turning angle is α, and the height of the third non - interested region on the left when the vehicle turning angle is α; [W - ROI 第三不感兴趣区域|右|α , 0, ROI 第三不感兴趣区域|右|α , H] are respectively the x coordinate of the upper left corner of the third non - interested region on the right when the vehicle turning angle is α, the y coordinate of the upper left corner of the third non - interested region on the right when the vehicle turning angle is α, the width of the third non - interested region on the right when the vehicle turning angle is α, and the height of the third non - interested region on the right when the vehicle turning angle is α.

[0185] It can be understood that when the vehicle turns, each of the divided regions needs to be adjusted, that is, when there are the first sub - non - interested region and the second sub - non - interested region, if the vehicle is in a turning state, the first sub - non - interested region and the second sub - non - interested region are adjusted according to the visual center of gravity offset value.

[0186] Exemplarily, the first sub - non - interested region and the second sub - non - interested region are adjusted according to the following formula:

[0187]

[0188]

[0189]

[0190]

[0191] ROI 第一子不感兴趣区域|α

[0192] = [ROI第二子不感兴趣区域|左|α ,0,ROI 第一子不感兴趣区域|左|α ,H]∪[W -ROI 第一子不感兴趣区域|右|α -ROI 第二子不感兴趣区域|右|α ,0,ROI 第一不感兴趣区域|右|α ,H]

[0193] ROI 第二子不感兴趣区域|α

[0194] =[0,0,ROI 第二子不感兴趣区域|左|α ,H]∪[W -ROI 第二子不感兴趣区域|右|α ,0,ROI 第二子不感兴趣区域|右|α ,H]

[0195] Among them,

[0196] Offset x - is the visual center of gravity offset value;

[0197] W - is the total pixel width of the external vehicle environment image;

[0198] ROI 感兴趣区域|w - represents the pixel width of the region of interest;

[0199] ROI 第一子不感兴趣区域|左|α - represents the pixel width of the first sub - non - interested region on the left outside the region of interest when the vehicle turning angle is α;

[0200] ROI 第一子不感兴趣区域|右|α - represents the pixel width of the first sub - non - interested region on the right outside the region of interest when the vehicle turning angle is α;

[0201] ROI 第二子不感兴趣区域|左|α - represents the pixel width of the second sub - non - interested region on the left outside the region of interest when the vehicle turning angle is α;

[0202] ROI 第二子不感兴趣区域|右|α - represents the pixel width of the second sub - non - interested region on the right outside the region of interest when the vehicle turning angle is α;

[0203] ROI 第一子不感兴趣区域|α - includes two parts: the range of the first sub - non - interested region on the left outside the region of interest when the vehicle turning angle is α [ROI 第二子不感兴趣区域|左|α, 0,ROI 第一子不感兴趣区域|左|α ,H] and the range of the first sub - non - interested region on the right outside the region of interest when the vehicle turning angle is α [W - ROI 第一子不感兴趣区域|右|α -ROI 第二子不感兴趣区域|右|α ,0,ROI 第一不感兴趣区域|右|α ,H], among which, [ROI 第二子不感兴趣区域|左|α,0,ROI 第一子不感兴趣区域|左|α ,H] are respectively the x - coordinate of the upper - left corner of the first sub - region of interest not on the left side under the vehicle turning angle α, the y - coordinate of the upper - left corner of the first sub - region of interest not on the left side under the vehicle turning angle α, the width of the first sub - region of interest not on the left side under the vehicle turning angle α, and the height of the first sub - region of interest not on the left side under the vehicle turning angle α; [W - ROI 第一子不感兴趣区域|右|α - ROI 第二子不感兴趣区域|右|α ,0,ROI 第一不感兴趣区域|右|α ,H] are respectively the x - coordinate of the upper - left corner of the first sub - region of interest not on the right side under the vehicle turning angle α, the y - coordinate of the upper - left corner of the first sub - region of interest not on the right side under the vehicle turning angle α, the width of the first sub - region of interest not on the right side under the vehicle turning angle α, and the height of the first sub - region of interest not on the right side under the vehicle turning angle α.

[0204] ROI 第二子不感兴趣区域|α - It includes two parts: the range [0,0,ROI of the second sub - region of interest not on the left side outside the region of interest under the vehicle turning angle α 第二子不感兴趣区域|左|α ,H] and the range [W - ROI of the second sub - region of interest not on the right side outside the region of interest under the vehicle turning angle α 第二子不感兴趣区域|右|α ,0,ROI 第二子不感兴趣区域|右|α ,H], where, [0,0,ROI 第二子不感兴趣区域|左|α ,H] are respectively the x - coordinate of the upper - left corner of the second sub - region of interest not on the left side under the vehicle turning angle α, the y - coordinate of the upper - left corner of the second sub - region of interest on the left side under the vehicle turning angle α, the width of the second sub - region of interest on the left side under the vehicle turning angle α, and the height of the second sub - region of interest on the left side under the vehicle turning angle α, [W - ROI 第二子不感兴趣区域|右|α ,0,ROI 第二子不感兴趣区域|右|α ,H] are respectively the x - coordinate of the upper - left corner of the second sub - region of interest not on the right side under the vehicle turning angle α, the y - coordinate of the upper - left corner of the second sub - region of interest on the right side under the vehicle turning angle α, the width of the second sub - region of interest on the right side under the vehicle turning angle α, and the height of the second sub - region of interest on the right side under the vehicle turning angle α.

[0205] In an optional embodiment, compensating the image content complexity of each region to obtain the total image content complexity of the external vehicle environment includes:

[0206] Obtaining the weighted compensation value of each region; among them, the more interested the driver is in the region, the greater its weighted compensation value;

[0207] Dividing each of the regions into a plurality of coding blocks, and obtaining the image content complexity of each coding block;

[0208] Using the weighted compensation value, perform weighted compensation on the image content complexity of each coding block to obtain the image content complexity of each coding block after weighted compensation;

[0209] Add up the image content complexity of each coding block after weighted compensation to obtain the total image content complexity of the out-of-vehicle environment image.

[0210] It can be understood that each region is divided into M*N coding blocks, intra-frame prediction and inter-frame prediction are performed on each coding block to obtain the image content complexity of each coding block, and then the image content complexity of each coding block is weighted and compensated according to the weighted compensation value of each region to obtain the image content complexity of each coding block after weighted compensation. Finally, add up the image content complexity of each coding block after weighted compensation to obtain the total image content complexity of the out-of-vehicle environment image. Among them, the greater the weighted compensation value of the region that the driver is more interested in.

[0211] Exemplarily, taking the division of the out-of-vehicle environment image into the region of interest, the first sub-region of disinterest, and the second region of disinterest as an example, calculate the total image content complexity of the out-of-vehicle environment image according to the following formula:

[0212]

[0213]

[0214]

[0215] Among them,

[0216] Comp i - is the image content complexity of coding block i;

[0217] w i - is the weighted compensation value of coding block i;

[0218] w b - is the basic weighted weight.

[0219] In an optional embodiment, the encoding of the out-of-vehicle environment image according to the target bitrate includes:

[0220] Obtain the compensation method and compensation quantization parameter of each region;

[0221] Use the compensation method and the compensation quantization parameter to compensate the reference quantization parameter of each region to obtain each initial quantization parameter; among them, the smaller the initial quantization parameter of the region that the driver is more interested in;

[0222] Calculate the initial bit rate of the out-of-vehicle environment image under the initial quantization parameter;

[0223] When the difference between the initial bit rate and the target bit rate is within a preset difference range, encode the out-of-vehicle environment image using the initial quantization parameter.

[0224] It can be understood that at a certain bit rate, by hierarchically adjusting the quantization parameters of each region, limited bit rate resources can be more allocated to the regions of interest, thereby improving the image quality of the regions of interest and achieving a smooth transition in image clarity, enhancing the visual experience.

[0225] In the embodiment of the present invention, at the target bit rate, by compensating the reference quantization parameters of each region in different ways, the initial quantization parameters of each region are adjusted so that the initial bit rate of the out-of-vehicle environment image is within the preset difference range from the target bit rate, improving the clarity of the key regions that the driver pays more attention to and reducing the clarity of the remaining regions, which can ensure the clarity of the key regions while meeting the requirements for video fluency (video transmission requirements) of remote driving, and thus ensure the video quality.

[0226] It can be understood that in the embodiment of the present invention, by compensating the reference quantization parameters of each region in different ways, the smaller the initial quantization parameter of the region that the driver is more interested in, the higher the clarity of the region that the driver is interested in. The compensation methods at least include any two of the following: additive compensation, non-compensation, and subtractive compensation, so as to ensure different compensation methods for different regions.

[0227] Exemplarily, through the following formula, perform reduction compensation on the region of interest, non-compensation on the first sub-region of non-interest, and increase compensation on the second sub-region of non-interest to obtain the initial quantization parameter:

[0228]

[0229]

[0230] Wherein,

[0231] - is the initial quantization parameter of coding block i;

[0232] - is the reference quantization parameter;

[0233] - is the compensation quantization parameter, preferably,

[0234] - is the maximum value of the preset quantization parameter;

[0235] - is the minimum value of the preset quantization parameter.

[0236] Calculate the initial bitrate Bitrate1 according to the following formula:

[0237]

[0238] where Comp i - is the image content complexity of the coding block i;

[0239] - is the initial quantization parameter of the coding block i;

[0240] γ - is a preset coefficient, adjusted according to experience and test conditions. Generally, an encoder will provide an empirical value;

[0241] When the difference between the target bitrate and the target bitrate is within the preset difference range, encode the out-of-vehicle environment image using the initial quantization parameter.

[0242] In an optional implementation manner, the video encoding method further includes:

[0243] When the difference between the initial bitrate and the target bitrate is not within the preset difference range, update each of the reference quantization parameters according to the relationship between the initial bitrate and the target bitrate to obtain each target reference quantization parameter;

[0244] Compensate each target reference quantization parameter of each region using the compensation method and the compensation quantization parameter to obtain an optimized quantization parameter;

[0245] Calculate the optimized bitrate of the out-of-vehicle environment image under the optimized quantization parameter;

[0246] If the difference between the optimized bitrate and the target bitrate is within the preset difference range, or the area of the first region is equal to the area of the second region, encode the out-of-vehicle environment image using the optimized quantization parameter;

[0247] where the first region is the region in the current driving direction region where the reduced compensation method is used with the compensation quantization parameter, the second region is the region in the current driving direction region where the increased compensation method is used with the compensation quantization parameter, and the current driving direction region is the region in the out-of-vehicle environment image that is in the vehicle driving direction.

[0248] It can be understood that when the difference between the initial bitrate and the target bitrate is not within the preset difference range, it indicates that the difference between the initial bitrate and the target bitrate is too large. At this time, the reference quantization parameter is first updated to optimize the initial bitrate, so as to narrow the gap between the optimized bitrate and the target bitrate. Subsequently, the compensation quantization parameter is adjusted within a small range according to the specific situation, and the difference between the optimized bitrate and the target bitrate is continuously narrowed, so that the difference between the optimized bitrate obtained after the latest compensation and the target bitrate is within the preset difference range, or even equal to the target bitrate.

[0249] Exemplarily, when the difference between the initial bitrate and the target bitrate is not within the preset difference range, each reference quantization parameter is updated according to the following formula to obtain each target reference quantization parameter;

[0250]

[0251] where Bitrate b is the target bitrate, Bitrate1 is the initial bitrate, is the reference quantization parameter, is the target reference quantization parameter.

[0252] Using the compensation method and the compensation quantization parameter to compensate each target reference quantization parameter of each region to obtain an optimized quantization parameter;

[0253] At this time, calculate the optimized bitrate of the external environment image under the optimized quantization parameter, the area of the first region (the region in the current driving direction region that uses the compensation quantization parameter for the decreasing compensation method) and the area of the second region (the region in the current driving direction region that uses the compensation quantization parameter for the increasing compensation method); for example, in the embodiment of the present invention, the first region is the region of interest, and the second region is the second sub-region of non-interest.

[0254] Exemplarily, through the following formula, the region of interest is compensated by decreasing, the first sub-region of non-interest is not compensated, and the second sub-region of non-interest is compensated by increasing to obtain an optimized quantization parameter:

[0255]

[0256]

[0257] where

[0258] - is the optimized quantization parameter of coding block i;

[0259] - is the target reference quantization parameter;

[0260] - is the compensation quantization parameter, preferably,

[0261] - is the preset maximum value of the quantization parameter;

[0262] - is the preset minimum value of the quantization parameter.

[0263] Calculate the optimized bitrate Bitrate2 according to the following formula:

[0264]

[0265] where Comp i - is the image content complexity of the coding block i;

[0266] - is the optimized quantization parameter of the coding block i;

[0267] γ - is a preset coefficient, adjusted according to experience and test conditions. Generally, an encoder will provide an empirical value;

[0268] If the difference between the optimized bitrate obtained after the latest compensation and the target bitrate is within the preset difference range, it indicates that the optimized bitrate obtained after the latest compensation meets the video transmission requirements and video quality requirements for remote driving. At this time, there is no need to adjust the compensation quantization parameter, and the optimized quantization parameter can be directly used to encode the external environment image of the vehicle;

[0269] If the area of the first region is equal to the area of the second region, it indicates that if the compensation quantization parameter of the current driving direction region in the external environment image of the vehicle is adjusted further at this time, it will not affect the current driving mode region in the external environment image of the vehicle. Therefore, there is no need to adjust the compensation quantization parameter at this time, and the optimized quantization parameter can be directly used to encode the external environment image of the vehicle.

[0270] In an optional embodiment, the video encoding method further includes:

[0271] If the difference between the optimized bitrate obtained after the latest compensation and the target bitrate is not within the preset difference range, and the area of the first region is not equal to the area of the second region, then adjust the compensation quantization parameter according to the relationship between the optimized bitrate obtained after the latest compensation and the target bitrate and the relationship between the area of the first region and the area of the second region;

[0272] After each adjustment of the compensation quantization parameter, use the latest adjusted compensation quantization parameter to compensate the target reference quantization parameter of each region until the difference between the optimized bitrate obtained after the latest compensation and the target bitrate is within the preset difference range, and then stop adjusting the compensation quantization parameter;

[0273] After stopping adjusting the compensation quantization parameter, use the optimized quantization parameter obtained after the latest compensation to encode the external environment image of the vehicle.

[0274] It can be understood that when the difference between the optimized bitrate obtained after the latest compensation and the target bitrate is not within the preset difference range, and the areas of the first region and the second region are not equal, it means that there is still a slight gap between the current bitrate and the target bitrate at this time. Small-range adjustment of the compensation quantization parameter is required. After each adjustment of the compensation quantization parameter, calculate the optimized bitrate obtained after the latest compensation until the difference between the optimized bitrate obtained after the latest compensation and the target bitrate is within the preset difference range or even equal to the target bitrate. This indicates that the optimized bitrate obtained after the latest compensation meets the video transmission requirements and video quality requirements for remote driving. At this time, there is no need to adjust the compensation quantization parameter anymore, and the optimized quantization parameter obtained after the latest compensation can be directly used to encode the external environment image of the vehicle.

[0275] In an optional implementation manner, the adjusting the compensation quantization parameter according to the relationship between the optimized bitrate obtained after the latest compensation and the target bitrate and the relationship between the areas of the first region and the second region includes:

[0276] When the optimized bitrate obtained after the latest compensation is less than the target bitrate, if the area of the first region is greater than the area of the second region, increase the compensation quantization parameter; if the area of the first region is less than the area of the second region, decrease the compensation quantization parameter;

[0277] When the optimized bitrate obtained after the latest compensation is greater than the target bitrate, if the area of the first region is greater than the area of the second region, decrease the compensation quantization parameter; if the area of the first region is less than the area of the second region, increase the compensation quantization parameter.

[0278] It can be understood that when the optimized bitrate obtained after the latest compensation is less than the target bitrate, if the area of the first region is greater than the area of the second region, the compensation quantization parameter is increased. At this time, the image quality difference between the region of interest and the non - region of interest becomes larger. If the area of the first region is less than the area of the second region, the compensation quantization parameter is decreased. At this time, the image quality difference between the region of interest and the non - region of interest becomes smaller. All of the above steps can increase the optimized bitrate obtained after the latest compensation and gradually approach the target bitrate.

[0279] When the optimized bitrate obtained after the latest compensation is greater than the target bitrate, if the area of the first region is greater than the area of the second region, the compensation quantization parameter is decreased. At this time, the image quality difference between the region of interest and the non - region of interest becomes smaller. If the area of the first region is less than the area of the second region, the compensation quantization parameter is increased. At this time, the image quality difference between the region of interest and the non - region of interest becomes larger. All of the above steps can decrease the optimized bitrate obtained after the latest compensation and gradually approach the target bitrate.

[0280] Exemplarily, when the following situations occur, the compensation quantization parameter is adjusted or not adjusted according to the relationship between the optimized bitrate obtained after the latest compensation and the target bitrate and the relationship between the area of the first region and the area of the second region:

[0281] When , do not adjust the Q pT value; where Bitrate 最新 represents the optimized bitrate obtained after the latest compensation, Bitrate b represents the target bitrate, and Q pT represents the compensation quantization parameter.

[0282] When do not adjust the Q pT value;

[0283] When and , increase the Q pT value. When and , decrease the Q pT value; where S 第一区域 represents the area of the first region, and S 第二区域 represents the area of the second region;

[0284] When and , decrease the Q pT value; when and , increase the Q pT value.

[0285] A video encoding method provided by an embodiment of the present invention can assign different image content complexities to each area by compensating for the image content complexity of each area of the external environment image of the vehicle, so that the total image content complexity of the external environment image of the vehicle better meets the video transmission requirements and video quality requirements in remote driving, thereby making the target bit rate determined according to the total image content complexity more reasonable. It can effectively avoid the phenomenon that the network bandwidth occupied by the video is large while ensuring the video quality, improve the video transmission speed, and further improve the control experience and judgment accuracy of the remote control driver, and reduce potential safety hazards.

[0286] See Figure 9 , Figure 9 is a structural block diagram of a video encoding device 10 provided by an embodiment of the present invention. The video encoding device 10 includes:

[0287] An external environment image acquisition module 11 for acquiring an external environment image of the vehicle;

[0288] A region division module 12 for dividing the external environment image according to the current driving state of the vehicle;

[0289] An image content complexity compensation module 13 for compensating the image content complexity of each region to obtain the total image content complexity of the external environment image;

[0290] A target bit rate acquisition module 14 for acquiring the target bit rate of the external environment image according to the total image content complexity;

[0291] An encoding module 15 for encoding the external environment image according to the target bit rate.

[0292] Optionally, the image content complexity compensation module 13 is further configured to:

[0293] Obtain a weighted compensation value for each region; wherein, the more interested the driver is in the region, the greater the weighted compensation value;

[0294] Divide each region into a plurality of encoding blocks, and obtain the image content complexity of each encoding block;

[0295] Use the weighted compensation value to perform weighted compensation on the image content complexity of each encoding block to obtain the image content complexity of each encoding block after weighted compensation;

[0296] Add up the image content complexities of each encoding block after weighted compensation to obtain the total image content complexity of the external environment image.

[0297] Optionally, the area division module 12 is further configured to:

[0298] When the vehicle is in a stationary state, divide the external environment image of the vehicle into first regions of no interest;

[0299] When the vehicle is in a driving state, divide the region in the driving direction of the vehicle in the external environment image into a region to be divided, and divide the remaining regions in the external environment image of the vehicle into second regions of no interest;

[0300] According to the preset correspondence between the vehicle speed and the driver's visual focus range, calculate the current driver's visual focus range at the current vehicle speed of the vehicle; wherein, in the correspondence, the vehicle speed and the driver's visual focus range are in an inverse correlation relationship;

[0301] According to the current driver's visual focus range, obtain the current driver's visual focus region in the region to be divided, divide the current driver's visual focus region into a region of interest, and divide the remaining regions in the region to be divided into third regions of no interest. The third regions of no interest include or do not include a plurality of sub-regions of no interest, and each sub-region of no interest is composed of a partial region located on the left side outside the region of interest and a partial region located on the right side outside the region of interest.

[0302] Optionally, the area division module 12 is further configured to:

[0303] After obtaining the current driver's visual focus region in the region to be divided, if the vehicle is in a turning state, based on the left-wheel turning motion trajectory and the right-wheel turning motion trajectory of the vehicle, obtain the central turning motion trajectory of the vehicle;

[0304] According to the central turning motion trajectory, obtain the driver's visual center of gravity of the vehicle in the turning state;

[0305] According to the driver's visual center of gravity of the vehicle in the turning state, calculate the visual center of gravity offset value of the driver's visual center of gravity of the vehicle in the straight-ahead state compared to the driver's visual center of gravity of the vehicle in the turning state;

[0306] According to the visual center of gravity offset value, adjust the current driver's visual focus region, divide the adjusted current driver's visual focus region into the region of interest, and divide the remaining regions in the region to be divided into the third regions of no interest.

[0307] Optionally, the encoding module 15 is further configured to:

[0308] Obtain the compensation method and compensation quantization parameter of each region;

[0309] Compensate the reference quantization parameter of each of the regions by using the compensation method and the compensation quantization parameter to obtain each initial quantization parameter; wherein, for the region that the driver is more interested in, its initial quantization parameter is smaller;

[0310] Calculate the initial bit rate of the out-of-vehicle environment image under the initial quantization parameter;

[0311] When the difference between the initial bit rate and the target bit rate is within a preset difference range, encode the out-of-vehicle environment image by using the initial quantization parameter.

[0312] Optionally, the encoding module 15 is further configured to:

[0313] When the difference between the initial bit rate and the target bit rate is not within the preset difference range, update each of the reference quantization parameters according to the relationship between the initial bit rate and the target bit rate to obtain each target reference quantization parameter;

[0314] Compensate the target reference quantization parameter of each of the regions by using the compensation method and the compensation quantization parameter to obtain an optimized quantization parameter;

[0315] Calculate the optimized bit rate of the out-of-vehicle environment image under the optimized quantization parameter;

[0316] If the difference between the optimized bit rate and the target bit rate is within the preset difference range, or the area of the first region is equal to the area of the second region, encode the out-of-vehicle environment image by using the optimized quantization parameter;

[0317] Wherein, the first region is the region in the current driving direction region that uses the compensation quantization parameter for reducing compensation, the second region is the region in the current driving direction region that uses the compensation quantization parameter for increasing compensation, and the current driving direction region is the region in the out-of-vehicle environment image that is in the vehicle driving direction.

[0318] Optionally, the encoding module 15 is further configured to:

[0319] If the difference between the optimized bit rate obtained after the latest compensation and the target bit rate is not within the preset difference range, and the area of the first region is not equal to the area of the second region, adjust the compensation quantization parameter according to the relationship between the optimized bit rate obtained after the latest compensation and the target bit rate and the relationship between the area of the first region and the area of the second region;

[0320] After each adjustment of the compensation quantization parameter, use the latest adjusted compensation quantization parameter to compensate the target reference quantization parameter of each region until the difference between the optimized bitrate obtained after the latest compensation and the target bitrate is within the preset difference range, and then stop adjusting the compensation quantization parameter;

[0321] After stopping adjusting the compensation quantization parameter, use the optimized quantization parameter after the latest compensation to encode the out-of-vehicle environment image.

[0322] Optionally, the encoding module 15 is further configured to:

[0323] When the optimized bitrate obtained after the latest compensation is less than the target bitrate, if the area of the first region is greater than the area of the second region, increase the compensation quantization parameter; if the area of the first region is less than the area of the second region, decrease the compensation quantization parameter;

[0324] When the optimized bitrate obtained after the latest compensation is greater than the target bitrate, if the area of the first region is greater than the area of the second region, decrease the compensation quantization parameter; if the area of the first region is less than the area of the second region, increase the compensation quantization parameter.

[0325] Optionally, the out-of-vehicle environment image acquisition module 11 is further configured to:

[0326] Obtain multiple original out-of-vehicle environment images collected by a collection device of the vehicle;

[0327] Calibrate the collection device to obtain the internal parameter data and external parameter data of the collection device;

[0328] According to the internal parameter data and the internal parameter data, project all the original out-of-vehicle environment images onto the same coordinate system;

[0329] Fuse and splice the original out-of-vehicle environment images belonging to the same perspective in the same coordinate system to obtain out-of-vehicle environment images of different perspectives.

[0330] It should be noted that the working processes of the various modules in the video encoding device 10 according to the embodiments of the present invention may refer to the working processes of the video encoding method described in the above embodiments, and will not be elaborated herein.

[0331] A video encoding device 10 provided by an embodiment of the present invention can assign different image content complexities to each area by compensating the image content complexity of each area of the external environment image, so that the total image content complexity of the external environment image better meets the video transmission requirements and video quality requirements in remote driving. As a result, the target bitrate determined according to the total image content complexity is more reasonable, which can effectively avoid the phenomenon of large network bandwidth occupied by the video while ensuring the video quality, improve the video transmission speed, and further improve the control experience and judgment accuracy of remote driving personnel, reducing potential safety hazards.

[0332] An embodiment of the present invention also provides a computer-readable storage medium, which includes a stored computer program. When the computer program runs, it controls the device where the computer-readable storage medium is located to execute the video encoding method described in any of the above embodiments.

[0333] See Figure 10 , Figure 10 FIG. is a structural block diagram of a video encoding device 20 provided by an embodiment of the present invention. The video encoding device 20 includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, it implements the steps in the above video encoding method embodiment. Alternatively, when the processor 21 executes the computer program, it implements the functions of each module / unit in the above device embodiments.

[0334] Exemplarily, the computer program can be divided into one or more modules / units. The one or more modules / units are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the computer program in the video encoding device 20.

[0335] The video encoding device 20 may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art can understand that the schematic diagram is only an example of the video encoding device 20, and does not constitute a limitation on the video encoding device 20. It may include more or fewer components than shown, or combine certain components, or different components. For example, the video encoding device 20 may further include input / output devices, network access devices, a bus, etc.

[0336] The processor 21 may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The processor 21 is the control center of the video encoding device 20, and connects various parts of the entire video encoding device 20 through various interfaces and lines.

[0337] The memory 22 can be used to store the computer programs and / or modules. The processor 21 realizes various functions of the video encoding device 20 by running or executing the computer programs and / or modules stored in the memory 22, and by calling the data stored in the memory 22. The memory 22 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the mobile phone (such as audio data, phone book, etc.). In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0338] Among them, if the modules / units integrated in the video encoding device 20 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above-described method embodiments of the present invention, it can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor 21, the steps of the above-described method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0339] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the attached drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0340] The above is the preferred embodiment of the present invention. It should be pointed out that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements are also regarded as the protection scope of the present invention.

Claims

1. A video encoding method, characterized in that, Including: Obtaining an external environment image of the vehicle; Performing region division on the external environment image according to the current driving state of the vehicle; Compensating the image content complexity of each region to obtain the total image content complexity of the external environment image; Obtaining the target bit rate of the external environment image according to the total image content complexity; Encoding the external environment image according to the target bit rate.

2. The video encoding method according to claim 1, wherein The compensating the image content complexity of each region to obtain the total image content complexity of the external environment image includes: Obtaining a weighted compensation value for each region; wherein, the more interested the driver is in the region, the larger the weighted compensation value; Dividing each region into a plurality of coding blocks, and obtaining the image content complexity of each coding block; Using the weighted compensation value to perform weighted compensation on the image content complexity of each coding block to obtain the image content complexity of each coding block after weighted compensation; Adding the image content complexity of each coding block after weighted compensation to obtain the total image content complexity of the external environment image.

3. The video encoding method according to claim 1, wherein The performing region division on the external environment image according to the current driving state of the vehicle includes: When the vehicle is in a stationary state, evenly dividing the external environment image into first uninterested regions; When the vehicle is in a driving state, dividing the region in the external environment image in the vehicle driving direction into a region to be divided, and dividing the remaining regions in the external environment image into second uninterested regions; Calculating the current driver's visual focus range at the current vehicle speed according to the corresponding relationship between the preset vehicle speed and the driver's visual focus range; wherein, in the corresponding relationship, the vehicle speed and the driver's visual focus range are inversely correlated; According to the current driver's visual focus range, obtaining the current driver's visual focus region in the region to be divided, dividing the current driver's visual focus region into an interested region, and dividing the remaining regions in the region to be divided into third uninterested regions, and the third uninterested regions include or do not include a plurality of sub-uninterested regions, and each sub-uninterested region is composed of a partial region located outside the left side of the interested region and a partial region located outside the right side of the interested region.

4. The video encoding method according to claim 3, wherein After obtaining the current driver's visual focus region in the region to be divided, if the vehicle is in a turning state, the video encoding method further includes: Based on the left-wheel turning motion trajectory and the right-wheel turning motion trajectory of the vehicle, obtaining the central turning motion trajectory of the vehicle; According to the central turning motion trajectory, obtaining the driver's visual center of gravity of the vehicle in the turning state; According to the driver's visual center of gravity of the vehicle in the turning state, calculating the visual center of gravity offset value of the driver's visual center of gravity of the vehicle in the straight-ahead state compared with the driver's visual center of gravity of the vehicle in the turning state; Adjust the current driver's field of view focusing area according to the visual center of gravity offset value, divide the adjusted current driver's field of view focusing area into the regions of interest, and divide the remaining areas of the area to be divided into the third non - interested regions.

5. The video encoding method according to any one of claims 1-4, characterized in that, The encoding of the external environment image according to the target bit rate includes: Obtain the compensation method and compensation quantization parameter for each of the regions; Compensate the reference quantization parameter for each of the regions by using the compensation method and the compensation quantization parameter to obtain each initial quantization parameter; among them, for the regions that the driver is more interested in, the initial quantization parameter is smaller; Calculate the initial bit rate of the external environment image under the initial quantization parameter; When the difference between the initial bit rate and the target bit rate is within a preset difference range, encode the external environment image by using the initial quantization parameter.

6. The video encoding method according to claim 5, wherein The video encoding method further includes: When the difference between the initial bit rate and the target bit rate is not within the preset difference range, update each reference quantization parameter according to the relationship between the initial bit rate and the target bit rate to obtain each target reference quantization parameter; Compensate the target reference quantization parameter for each of the regions by using the compensation method and the compensation quantization parameter to obtain optimized quantization parameters; Calculate the optimized bit rate of the external environment image under the optimized quantization parameter; If the difference between the optimized bit rate and the target bit rate is within the preset difference range, or the area of the first region is equal to the area of the second region, encode the external environment image by using the optimized quantization parameter; Among them, the first region is the region in the current driving direction region that uses the compensation quantization parameter for the decreasing compensation method, the second region is the region in the current driving direction region that uses the compensation quantization parameter for the increasing compensation method, and the current driving direction region is the region in the external environment image that is in the vehicle driving direction.

7. The video encoding method according to claim 6, wherein The video encoding method further includes: If the difference between the optimized bit rate obtained after the latest compensation and the target bit rate is not within the preset difference range, and the area of the first region is not equal to the area of the second region, adjust the compensation quantization parameter according to the relationship between the optimized bit rate obtained after the latest compensation and the target bit rate and the relationship between the area of the first region and the area of the second region; After each adjustment of the compensation quantization parameter, compensate the target reference quantization parameter for each of the regions by using the latest adjusted compensation quantization parameter until the difference between the optimized bit rate obtained after the latest compensation and the target bit rate is within the preset difference range, and stop adjusting the compensation quantization parameter; After stopping adjusting the compensation quantization parameter, encode the external environment image by using the optimized quantization parameter obtained after the latest compensation.

8. The video encoding method according to claim 7, wherein The adjusting the compensation quantization parameter according to the relationship between the optimized bit rate obtained after the latest compensation and the target bit rate and the relationship between the area of the first region and the area of the second region includes: When the optimized bitrate obtained after the latest compensation is less than the target bitrate, if the area of the first region is greater than the area of the second region, increase the compensation quantization parameter; if the area of the first region is less than the area of the second region, decrease the compensation quantization parameter; When the optimized bitrate obtained after the latest compensation is greater than the target bitrate, if the area of the first region is greater than the area of the second region, decrease the compensation quantization parameter; if the area of the first region is less than the area of the second region, increase the compensation quantization parameter.

9. The video encoding method according to claim 1, wherein The obtaining of the vehicle's external environment image includes: Obtaining multiple original external environment images of the vehicle collected by the vehicle's acquisition device; Calibrating the acquisition device to obtain the internal parameter data and external parameter data of the acquisition device; Projecting all the original external environment images onto the same coordinate system according to the internal parameter data and the internal parameter data; Fusing and stitching the original external environment images belonging to the same perspective in the same coordinate system to obtain the external environment images from different perspectives.

10. A video encoding device, characterized in that, Including: An external environment image acquisition module for acquiring the vehicle's external environment image; A region division module for dividing the external environment image according to the vehicle's current driving state; An image content complexity compensation module for compensating the image content complexity of each region to obtain the total image content complexity of the external environment image; A target bitrate acquisition module for acquiring the target bitrate of the external environment image according to the total image content complexity; An encoding module for encoding the external environment image according to the target bitrate.

11. A video encoding device, characterized in that, Including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the video encoding method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program; wherein, when the computer program runs, it controls the device where the computer-readable storage medium is located to execute the video encoding method according to any one of claims 1 to 9.

Images