Display apparatus, display method, display system, and electronic device

Abstract

A display apparatus, a display method, a display system, and an electronic device, pertaining to the technical field of display, the display apparatus being applied to in-vehicle display and comprising: a display screen, which is configured to display a video picture acquired by an image acquisition apparatus in real time; and a processor, which is configured to adjust, in response to a target operation performed on a first video picture currently displayed in the display screen, the image acquisition apparatus and / or a video frame to be displayed, such that at least one of the following differences is present between a second video picture displayed after adjustment and the first video picture: picture size, field of view of the picture, the capturing angle corresponding to the picture, picture quality, and display position.

Description

Display device, display method, display system, and electronic equipment Technical Field

[0001] The present disclosure relates to the field of display technology, and in particular to a display device, a display method, a display system, and an electronic device. Background Art

[0002] With the development of display technology, in-vehicle displays have emerged. These display screens are installed in vehicles and can display real-time video feeds from multiple cameras located both inside and outside the vehicle. In some driving scenarios, the images displayed on the in-vehicle display screens can aid driving safety.

[0003] Overview

[0004] Based on the background technology, the present disclosure proposes a display device, a display method, a display system, and an electronic device.

[0005] Among them, a display device is used in a vehicle display, comprising:

[0006] A display screen configured to display video images captured in real time by the image capture device;

[0007] The processor is configured to adjust the image acquisition device and / or the video frame to be displayed in response to a target operation performed on the first video picture currently displayed in the display screen, so that there is at least one of the following differences between the second video picture displayed after the adjustment and the first video picture: picture size, picture field size, picture corresponding shooting angle, picture quality and display position.

[0008] Exemplarily, the display screen is further configured to display an adjustment icon;

[0009] The processor is also configured to adjust the posture and / or focal length of the image acquisition device in response to a target operation performed on the adjustment icon so that there is a difference in at least one of the field of view size and the shooting angle between the first video screen and the second video screen.

[0010] Exemplarily, the adjustment icon includes a first frame, and the target operation includes a scaling operation on the first frame and a size change operation on at least one side; wherein,

[0011] The processor is specifically configured to adjust the posture of the image acquisition device in response to the size change operation so that there is a change in shooting angle between the first video picture and the second video picture; and

[0012] In response to the zooming operation, the focal length of the image acquisition device is adjusted so that there is a change in the field of view size between the first video frame and the second video frame.

[0013] Exemplarily, adjusting the posture of the image acquisition device in response to the size change operation includes:

[0014] Based on the type of the image acquisition device, obtaining size parameters of the adjusted first frame;

[0015] Substituting the size parameter into a preset function corresponding to the image acquisition device to obtain a rotation angle of the image acquisition device;

[0016] The image acquisition device is rotated according to the rotation angle.

[0017] Exemplarily, when the type of the image acquisition device is the first type, the first border is a rectangular frame, the size parameter includes the length of each side of the first border, and the preset function includes a first preset function, and the first preset function is used to characterize the correspondence between the length of the side of the first border and the rotation angle;

[0018] When the type of the image acquisition device is the second type, the first border is a trapezoidal frame, the size parameters include the angles of each inner corner of the first border, and the preset function includes a second preset function, which is used to characterize the correspondence between the difference between the inner angles of the first border and the rotation angle.

[0019] Exemplarily, the display screen is configured to display video images captured by multiple different image capture devices in a split screen;

[0020] After the size change operation, the first border exceeds the split screen where the first video picture is located, and occupies a portion of at least one first split screen adjacent to the split screen;

[0021] Before and after the target operation, the processor is configured to keep the sizes of the split screen and the first split screen unchanged.

[0022] Exemplarily, the display screen is configured to display video images captured by multiple different image capture devices in a split screen, and the adjustment icon includes a second frame;

[0023] The processor is further configured to: upon detecting that the second border, after being adjusted, exceeds the split screen where the first video image is located and occupies a portion of at least one second split screen adjacent to the first split screen, synchronously adjust the size of the split screen and the size of the second split screen so that the adjusted second border is entirely located in the adjusted split screen;

[0024] And, the screen size of the video frame to be displayed is adjusted so that the video screen of the image acquisition device fills the adjusted split screen.

[0025] Exemplarily, the processor is further configured to, when the target operation indicates adjusting the picture quality, display the second video picture at a first display position where the first video picture is located, or display the second video picture at a second display position different from the first display position;

[0026] The duration between the display time of the second video image at the second display position and the target operation is greater than the duration between the display time of the second video image at the first display position and the target operation.

[0027] Exemplarily, the processor is further configured to display a third video picture captured by the image capture device in real time at the first display position when the second video picture is displayed at the second display position;

[0028] The interval between the video frames displayed in the first display position is shorter than the preset delay time.

[0029] Exemplarily, the processor is specifically configured to determine the adjustment time for adjusting the picture quality of the video frame when the target operation indicates to adjust the picture quality, and to schedule the second video picture to the first display position or the second display position based on the adjustment time.

[0030] Exemplarily, the processor is specifically configured to: determine multiple processes to be performed based on a first picture quality of the video frame and a second picture quality indicated by the target operation; and

[0031] Compensating for a preset adjustment time corresponding to each of the processing steps based on the data volume of the video frame and / or the current performance parameters of the processor;

[0032] The adjustment time is determined based on the compensated preset adjustment time.

[0033] Exemplarily, determining the multiple processes to be performed based on the first picture quality of the video frame and the second picture quality indicated by the first preset operation includes:

[0034] Determining an environment type corresponding to the video frame, where the environment type is used to characterize the type of interference factors that affect picture quality in the environment where the image acquisition device is located;

[0035] determining a plurality of the processing based on a difference between the environment type and a preset environment type, and the second picture quality;

[0036] The preset environment type indicates that there are no interference factors that affect the picture quality.

[0037] Exemplarily, the display screen is further configured to display a plurality of menu keys; wherein,

[0038] The processor is further configured to adjust the picture quality of the video frame in response to a triggering operation on at least one menu key among the plurality of menu keys, so that the picture quality of the second video picture is higher than the picture quality of the first video picture.

[0039] Exemplarily, the processor is also configured to determine the environment type corresponding to the video frame when the picture quality of the video frame is adjusted, and based on the environment type, determine multiple processing to be performed on the video frame, and schedule the video frame to the target image processing model corresponding to each processing in turn to improve the picture quality of the video frame through the target image processing model.

[0040] Exemplarily, the picture quality includes picture clarity;

[0041] The processor is configured to divide the video frame into a plurality of image blocks, and perform mask reconstruction on contents of some of the image blocks, so that the second video picture contains no or less interference factors that affect picture quality:

[0042] The interference factors include at least one of dim light, dust, thick fog, and rain and snow.

[0043] Exemplarily, dividing the video frame into a plurality of image blocks and performing mask reconstruction on contents of some of the image blocks includes:

[0044] Dividing the video frame to obtain a plurality of first image blocks;

[0045] Dividing the first image block into at least one second image block according to the definition corresponding to each of the first image blocks; wherein the first image block with lower definition is divided into a smaller number of second image blocks;

[0046] masking a portion of the second image block;

[0047] Based on the unmasked second image block, the content of the masked second image block is reconstructed.

[0048] Exemplarily, the processor is configured with a target model, and when the target model is called by the processor, is used to execute the steps of dividing the video frame into a plurality of image blocks and performing mask reconstruction on the contents of some of the image blocks;

[0049] The target model is generated using a first image sample of a first type and a second image sample of a second type of the same image as training samples, and the clarity of the first image sample is higher than that of the second image sample.

[0050] Exemplarily, during the training process, the target model is used to generate a mask image based on the first image sample and the second image sample, and reconstruct the mask image to obtain a reconstructed predicted image; wherein the masked third image block in the mask image comes from the second image sample, and the unmasked image block comes from the first image sample;

[0051] Furthermore, during the training of the target model, the parameters of the target model are updated based on the difference between the fourth image block corresponding to the position of the third image block in the predicted image and the fifth image block corresponding to the position of the third image block in the first image sample.

[0052] Exemplarily, the processor is configured to, in response to the target operation, denoise the video frame and perform super-resolution processing on the denoised video frame so that the resolution of the second video picture is higher than the resolution of the first video picture; and / or,

[0053] The processor is configured to, in response to the target operation, denoise the video frame and perform color enhancement processing on the denoised video frame so that the color saturation of the second video picture is higher than the color saturation of the first video picture.

[0054] Exemplarily, the processor is configured with a thread queue, and the thread queue includes a plurality of units connected in series; the plurality of units include a decoding unit, a data scheduling unit, a convolution unit, a super-resolution unit, and a rendering unit;

[0055] The processor is specifically configured to input the video frame into the thread queue so as to perform pipeline super-resolution processing on the video frame through a plurality of the units.

[0056] Exemplarily, the display screen is further configured to display an undo control;

[0057] The processor is further configured to, in response to a triggering operation of the cancel control, display the video frame currently captured in real time by the image capture device according to the display parameters of the first video picture, so that the display screen switches from the second video picture back to a fourth video picture having the same display parameters as the first video picture;

[0058] The display parameters include at least one of the screen size, the field of view, the shooting angle, and the picture quality.

[0059] Among them, a display method is applied to an in-vehicle display, the method comprising:

[0060] To display the video images captured in real time by the image acquisition device;

[0061] In response to a target operation performed on the first video frame currently displayed in the display screen, the image acquisition device and / or the video frame to be displayed are adjusted so that there is at least one of the following differences between the second video frame displayed after adjustment and the first video frame: picture size, picture field size, shooting angle corresponding to the picture, picture quality and display position.

[0062] Among them, a vehicle-mounted display system includes multiple image acquisition devices and a display device described in any example, wherein the multiple image acquisition devices are respectively connected to the display device, and the display device is configured to display video images captured in real time by the multiple image acquisition devices in a split-screen manner.

[0063] Among them, an electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is used to support the display of the display device as described in any one of the examples.

[0064] The display device provided by the present disclosure is applied to an in-vehicle display and includes a display screen and a processor, wherein the display screen is configured to display a video image captured in real time by an image capture device; the processor is configured to adjust the image capture device and / or the video frame to be displayed in response to a target operation performed on a first video image currently displayed on the display screen, so that there is at least one of the following differences between the second video image displayed after the adjustment and the first video image: image size, image field size, image corresponding shooting angle, image quality, and display position. Since the display device can adjust the video frame captured by the image capture device in response to the target operation on the first video image displayed in real time, it can achieve the adjustment of at least one of the image size, image field size, image corresponding shooting angle, image quality, and display position. In this way, the user can make the second video image displayed on the display screen present a variety of changes through direct operation on the display screen to meet their diverse observation needs, thereby improving the flexibility of the in-vehicle display in assisting safe driving.

[0065] The above description is only an overview of the technical solution of the present disclosure. In order to more clearly understand the technical means of the present disclosure, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present disclosure more obvious and easy to understand, the specific implementation methods of the present disclosure are listed below.

[0066] BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or related technologies, the following is a brief introduction to the drawings required for the description of the embodiments or related technologies. Obviously, the drawings described below are some embodiments of the present disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without inventive efforts. It should be noted that the scales in the drawings are for illustration only and do not represent the actual scale.

[0068] FIG1 shows a schematic diagram of a layout environment of a display device proposed in the present disclosure;

[0069] FIG2a and FIG2b respectively show schematic diagrams of a display device of the present disclosure during use;

[0070] FIG3 shows a schematic diagram of an interface for displaying a video image on a split screen in an embodiment of the present disclosure;

[0071] FIG4 shows a schematic diagram of an interface of a display screen in an embodiment of the present disclosure;

[0072] FIG5 a shows a schematic diagram of an interface of a display screen under a first type of image acquisition device according to an embodiment of the present disclosure;

[0073] FIG5 b shows a schematic diagram of an interface of a display screen under a second type of image acquisition device in an embodiment of the present disclosure;

[0074] FIG6 a shows a schematic diagram showing changes in a display interface with a first frame in an embodiment of the present disclosure;

[0075] FIG6 b shows a schematic diagram showing changes in the display interface with a second frame in an embodiment of the present disclosure;

[0076] FIG7a and FIG7b are schematic diagrams showing changes of two display interfaces in an embodiment of the present disclosure;

[0077] FIG8 is a schematic diagram showing a flow chart of steps of a processor in determining multiple processes to be executed in an embodiment of the present disclosure;

[0078] FIG9 is a schematic diagram showing a processing flow of processing video frames of different environment types according to an embodiment of the present disclosure;

[0079] FIG10 shows a schematic diagram of a display interface of a display screen in an embodiment of the present disclosure;

[0080] FIG11 is a schematic diagram showing a flow chart of steps for mask reconstruction of the contents of an image block in an embodiment of the present disclosure;

[0081] FIG12 is a schematic diagram showing a process of mask reconstruction of a video frame in an embodiment of the present disclosure;

[0082] FIG13 is a schematic diagram showing the process of training a target model in an embodiment of the present disclosure;

[0083] FIG14 is a schematic diagram showing loss calculation of a target model in an embodiment of the present disclosure;

[0084] FIG15 is a schematic diagram showing a flow chart of super-resolution processing performed in a processor according to an embodiment of the present disclosure;

[0085] FIG16 is a schematic diagram showing a flow chart of performing super-resolution processing on the image shown in FIG15 in an embodiment of the present disclosure;

[0086] FIG17 is a schematic diagram showing an undo control displayed on a video screen in an embodiment of the present disclosure;

[0087] FIG18 shows a schematic diagram of a split-screen interface of a display screen in an embodiment of the present disclosure;

[0088] FIG19 shows a schematic diagram of an interface of a display screen in an embodiment of the present disclosure;

[0089] FIG20 shows a schematic diagram of an interface when a display screen displays a first video image in an embodiment of the present disclosure;

[0090] FIG21 shows a schematic diagram of the interface when the display screen displays the second video image in an embodiment of the present disclosure.

[0091] FIG22 shows a schematic flow chart of the steps of the display method in an embodiment of the present disclosure.

[0092] Detailed description

[0093] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0094] In-vehicle displays are now widely used in vehicles. Among the more common in-vehicle displays, the central control screen typically displays not only the operating interface but also images captured by the vehicle's cameras. For example, the central control screen's operating interface includes various controls related to vehicle operation, allowing operators to control the vehicle's air conditioning, seats, media players, and other functions. When the vehicle enters parking mode, the central control screen switches to displaying video feeds from the front, rear, and rear cameras to assist the driver in parking safely.

[0095] With the rise of autonomous driving technology, in-vehicle displays have undergone adaptive improvements. Autonomous driving relies on cameras and radars distributed around the vehicle to judge obstacles to assist in autonomous driving. Correspondingly, the in-vehicle display will show the video images taken by each camera to assist the driver in manually observing surrounding obstacles to assist in safe driving.

[0096] Because the visibility of obstacles on the side and rear of the vehicle is required to be high during driving, the display delay of the side and rear display images is required to be high in the vehicle display, generally not exceeding 100ms. The display delay refers to the time interval from a video frame captured by the camera to the display of the video frame on the display screen. Generally speaking, the smaller the display delay, the better. However, when assisting safe driving, the visibility of obstacles on the side and rear of the vehicle also has certain requirements. For example, from the perspective of driver experience, it generally requires the necessary low delay of the image, while the key images around the vehicle require high definition.

[0097] However, the current in-vehicle display switches the display screen after the vehicle operating conditions change. For example, when entering parking conditions, it switches to displaying the video images captured by the rear and side cameras. It cannot balance the diverse needs of vehicle users.

[0098] In view of this, the inventors of the present disclosure provide a display device that can be used in vehicle-mounted displays. Specifically, the display device can adjust at least one of the picture size, field of view size, shooting angle corresponding to the picture, picture quality and display position of the video frame captured by the image acquisition device in response to operations on the video picture on the display screen, thereby changing the displayed video picture in terms of picture size, focal length of the camera, shooting angle and picture quality to meet the diverse picture observation needs of vehicle users.

[0099] Referring to FIG1 , FIG1 shows a schematic diagram of the deployment environment of the display device proposed in the present disclosure, wherein the display device can be deployed in a vehicle, which can be a passenger car, bus, truck, or tractor. The passenger car can be a micro-passenger car, an ordinary passenger car, or a mid-size passenger car; the bus can be a micro-bus, a light bus, a medium bus, a large bus, an extra-large bus, or a double-decker bus; and the truck can be a micro-truck, a light truck, a medium truck, or a heavy truck. In other words, the display device of the present disclosure can be used for in-vehicle display in any of the above-mentioned vehicles. FIG1 exemplifies the case where the vehicle is a large truck. In a large truck, the rearview mirror content needs to be displayed in real time with low latency, and the vehicle's blind spot vision requires ultra-high definition to ensure driving safety. As autonomous driving technology becomes increasingly mature, drivers have more time to select the field of view they want to observe. Therefore, different types of fields of view (i.e., different image acquisition devices) can be displayed in partitions on the main screen. When the driver / vehicle user needs to focus on a certain field of view (a certain image acquisition device), they can select that partition and select functions such as ultra-definition, denoising, and color enhancement for it.

[0100] The display device may be located in a central control position of the vehicle, such as on a center console near the windshield in the cockpit, or may be located in other positions of the vehicle, such as to the left or right of the driver's seat, or to the right or left of the passenger seat. FIG1 shows a case where the display device is located in a central control position.

[0101] A display device may be provided at multiple locations on the vehicle, and the display devices at different locations may be used to display video images captured by image capture devices located at different positions on the vehicle. For example, a display device may be provided on the left and right sides of the driver's seat, as well as on the center console. The display device on the left side may be used to display video images captured by the image capture device located at the left rearview mirror, the display device on the right side may be used to display video images captured by the image capture device located at the right rearview mirror, and the display devices located on the center console may be used to display video images captured by the image capture device located at the rear and front of the vehicle.

[0102] It should be noted that the vehicle on which the display device of the present disclosure is installed can be a vehicle that supports autonomous driving. Thus, the display device can be used in autonomous driving scenarios to help improve the driving safety of the vehicle during autonomous driving.

[0103] 2a and 2b , schematic diagrams of a display device of the present disclosure during use are shown respectively. As shown in FIG2a and 2b , the display device can be applied to an in-vehicle display, and includes a display screen 301 and a processor 302. The processor is connected to the display screen, and the processor can be regarded as a system-level chip of the display screen, which is used to control the image to be displayed on the display screen and respond to control operations from the display screen, such as touch operation, remote control operation, voice operation, gesture operation, etc. on the display screen.

[0104] Wherein, the display screen is configured to display the video images captured by the image acquisition device in real time;

[0105] In which, the processor is configured to adjust the image acquisition device and / or the video frame to be displayed in response to a target operation performed on the first video screen currently displayed on the display screen, so that there is at least one of the following differences between the second video screen displayed after adjustment and the first video screen: screen size, screen field size, shooting angle corresponding to the screen, screen quality and display position.

[0106] In this embodiment, the display screen can be an LCD (Liquid Crystal Display) display screen or an OLED (Organic Electroluminescence Display) display screen. In some examples, the display screen has a touch function to receive the user's touch operation. Among them, the display screen can be a single-screen display, that is, the display screen can be connected to an image acquisition device to display the video images captured by the image acquisition device in full screen. In this case, multiple display devices can be arranged in the vehicle. As described in the above example, a display device can be configured on the driver's seat, the co-driver's seat, and the center console. Alternatively, the display screen can be a multi-screen display, such as the display screen can be connected to multiple image acquisition devices, and the video images captured by multiple image acquisition devices can be displayed in split screens. For example, if there are three image acquisition devices as shown in Figure 1, the display screen can include four display areas, and different display areas are used to display video images captured by different image acquisition devices.

[0107] In some examples, when a single display screen displays a single video image, the display screen may be connected to multiple image acquisition devices. The video images captured by the multiple image acquisition devices may be synthesized and the synthesized image displayed full screen on the display screen. For example, two-dimensional images captured by multiple image acquisition devices may be synthesized into a three-dimensional image, thereby displaying the three-dimensional image on the display screen.

[0108] Among them, the image acquisition device can be a camera installed inside or outside the vehicle, such as a panoramic camera or an ordinary camera. When the image acquisition device is installed on the outside of the vehicle, as described above, it can be installed on the position of the left and right rearview mirrors. When the image acquisition device is installed on the inside of the vehicle, it can be installed on the front windshield to capture images directly in front of the vehicle.

[0109] Among them, the image acquisition device acquires video images in real time, which can also be understood as the image acquisition device can record video, and each video frame in the video stream can be displayed on the display screen. The video frame displayed on the display screen is called a video image.

[0110] Among them, when the display screen displays the video picture, as described above, the video picture can be displayed on a single screen or on a split screen; when the video picture is displayed on a single screen, the video picture can be displayed at the optimal position of the display screen, and the optimal position can be the middle display area of ​​the display screen, or can be an area biased towards the driver's seat, or can be an area biased towards the co-pilot seat. When the video picture is displayed on a split screen, the video picture captured in real time by the image acquisition device can be displayed in the specified split screen. Referring to Figure 3, a schematic diagram of the interface for displaying the video picture on a split screen is shown. As shown in Figure 3, the video picture captured by the image acquisition device 23 at the left rearview mirror is displayed in the area on the left, the video picture captured by the image acquisition device 21 at the right rearview mirror is displayed in the area on the right, and the video picture captured by the image acquisition device 22 at the front windshield is displayed in the area in the middle position.

[0111] When the display screen displays the video images captured by the image acquisition device, the user can operate the first video image currently displayed on the display screen to change at least one of the image size, field of view, shooting angle, image quality and display position of the video images subsequently displayed on the display screen.

[0112] The target operation may be a touch operation on the display screen. Different target operations may correspond to different differences, such as one target operation corresponding to a change in screen size, and another target operation corresponding to a change in the screen's field of view. The target operation may include a single-finger slide operation, a double-click operation, a single-click operation, a multi-finger slide operation, and the like.

[0113] In some other examples, the target operation may also be other operations different from the touch operation, such as voice operation, where the user inputs voice into the display screen to achieve any change in the angle, field of view, shooting angle, display position and picture quality of the video screen. In this case, the processor can recognize the input voice and identify keywords, thereby adjusting the image acquisition device and / or the video frame to be displayed according to the keywords. Of course, in some other examples, other methods of inputting target operations, such as gesture operations, are not excluded. For example, the user can make different gestures to achieve different adjustments to the video screen.

[0114] The processor may adjust the image acquisition device and / or the video frame to be displayed in response to the target operation. Specifically, as described above, different target operations correspond to adjustments to different angles of the video screen. Among the screen size, the field of view of the screen, the shooting angle corresponding to the screen, the screen quality, and the display position, the field of view of the screen and the shooting angle corresponding to the screen are related to the adjustment of the image acquisition device. In other words, the adjustment of the image acquisition device may cause the field of view of the video screen and the corresponding shooting angle to change. As shown in FIG2a , when the target operation indicates an adjustment to the field of view of the screen and the shooting angle corresponding to the screen, the image acquisition device may be adjusted, such as adjusting the focal length and shooting posture of the image acquisition device.

[0115] Among the image size, field of view, shooting angle, quality, and display position, the image size, quality, and display position are display attributes of the video frame, which can be achieved by adjusting the video frame. As shown in FIG2b , for example, the size of the video frame can be adjusted to change the size of the image displayed on the display screen, and the quality of the video frame can be adjusted to change the quality of the video image displayed on the display screen. Regarding the image size, for example, the size of the video frame can be reduced or enlarged to change the size; for example, only the width of the video frame or only the height of the video frame can be changed to change the size; for example, the width and height of the video frame can be changed unequally to change the size.

[0116] Among them, for the adjustment of the display position, in one way, it can be achieved by adjusting the picture size of the video frame, that is, the picture size is changed, and the area size of its display position is also changed accordingly; in another way, it can be achieved by changing the size of the display area of ​​the display screen. For example, in split-screen display, the video frame can be changed from the split screen where the first video picture is located to another split screen for display, so that the video picture is adjusted from one split screen to another split screen.

[0117] In this case, a user performing a target operation may correspond to any adjustment of the screen size, the screen's field of view, the corresponding shooting angle of the screen, the screen quality, and the display position. For example, as shown in FIG2a , a user performs a zoom operation on a first video screen on the display screen using two fingers. The processor may adjust the focal length of the image acquisition device in response to the zoom operation, thereby changing the screen's field of view. In this case, the first video screen is switched to a second video screen with a different field of view. Alternatively, a user performing a target operation may correspond to at least two adjustments of the screen size, the screen's field of view, the corresponding shooting angle of the screen, the screen quality, and the display position. For example, a user performs a zoom operation on a first video screen on the display screen using two fingers. The processor may adjust the focal length of the image acquisition device in response to the zoom operation, thereby changing the screen's field of view. The processor may also adjust the screen size of the video frame to be displayed, which is captured by the image acquisition device. In this case, the first video screen is switched to a second video screen with a different field of view and size, such as screen 1″ in FIG2a .

[0118] In some other examples, as shown in FIG2b , when the user performs a target operation indicating adjustment of picture quality, the picture quality of the video frame to be displayed, which is captured by the image capture device, can be adjusted. Then, based on the adjusted picture quality, the display position to be displayed can be determined. For example, the adjusted second video picture can be displayed at a display position different from that of the first video picture. In this way, the first video picture is switched to the second video picture with different picture quality, such as picture 1″ in FIG2b . From a visual perspective, the first video picture is switched to another display position and the picture quality of the second video picture is changed.

[0119] Among them, the difference in screen size between the second video screen and the first video screen may include the screen size of the second video screen being smaller or larger than the screen size of the first video screen; the difference in the field of view size of the screen may include the field of view of the second video screen being smaller or larger than the field of view of the first video screen, wherein, when the field of view of the second video screen is larger, it can help the user observe the long-range image of the vehicle, that is, the environment in the area farther away from the vehicle; when the field of view of the second video screen is smaller, it can help the user observe the close-up image of the vehicle, that is, the environment in the area closer to the vehicle.

[0120] The difference in shooting angles corresponding to the images may include the field of view of the second video image overlapping or not overlapping with the field of view of the first video image. In the case of overlap, the size of the overlapping area may not be limited. The difference in image quality may include the image quality of the second video image being higher or lower than the image quality of the first video image. When the image quality of the second video image is higher than that of the first video image, it can help the user observe a clearer image. Thus, when the vehicle is driving in harsh environments, such as rain, snow, or dust, image processing can help the user see the external environment more clearly, thereby avoiding the impact of weather factors on driving safety and improving driving safety. When the image quality of the second video image is lower than that of the first video image, the lower quality video frame can reduce the amount of video frame data and reduce image delay. Therefore, when the driving environment is good, the matching between the video image display and the driving speed can be improved, thereby reducing image delay. From a visualization perspective, the changes in the external environment observed by the human eye during vehicle driving are roughly the same as the changes in the video image displayed on the display screen.

[0121] It should be noted that the image acquisition device being adjusted is the acquisition device that acquires the first video frame, that is, the first video frame and the second video frame originate from the same image acquisition device; wherein, the video frame to be displayed is the next video frame of the first video frame, which can be a cached video frame that will not be displayed in time in the future, or it can be the next video frame acquired in real time by the image acquisition device. In the adjustment of the picture quality, display position and picture size, the processor can adjust each video frame acquired by the image acquisition device after the first video frame, so that each second video frame displayed after the first video frame has a difference in picture quality, display position and picture size from the first video frame, but the picture quality, display position and picture size of each second video frame can be consistent until a trigger event is detected. When the trigger event occurs, the adjustment of the picture quality, display position and picture size can be stopped.

[0122] It should also be noted that when the image acquisition device is adjusted, each video frame subsequently acquired by the image acquisition device will have differences in the field of view size and the corresponding shooting angle of the first video screen. However, when a trigger event occurs, the image acquisition device can be stopped from being restored to the settings corresponding to the first video screen, so that the video screen subsequently displayed on the display screen has the same configuration as the first video screen.

[0123] It is understood that, since the present disclosure is applied to in-vehicle displays, the image acquisition device can record a video stream of the vehicle's surrounding environment, and the image content of the first video frame and the second video frame can differ. For example, during driving, the first video frame is a picture of the vehicle's surrounding environment at time t1, and the second video frame is a picture of the vehicle's surrounding environment at time t2. For another example, when the vehicle is stationary, the video frame captured at time t3 and the video frame captured at time t4 can be the same.

[0124] With this display device, its processor can adjust the image capture device in response to a target operation on the first video frame being displayed in real time, thereby adjusting at least one of the field of view of the frame and the corresponding shooting angle of the frame. Furthermore, the processor can adjust the video frames captured by the image capture device, thereby adjusting at least one of the frame size, image quality, and display position. In this way, a user can switch the video frame displayed on the display screen by directly operating the display screen, so that the switched second video frame meets the user's observation needs, thereby improving the flexibility of in-vehicle display-assisted safe driving.

[0125] In some examples, the image acquisition device can be adjusted by performing a target operation on the display screen. Specifically, the posture of the image acquisition device can be adjusted to change the shooting angle corresponding to the image, or the focal length of the image acquisition device can be adjusted to change the field of view corresponding to the image. The target operation to indicate adjustment of the image acquisition device can be performed in a variety of ways, such as voice operation, touch operation, gesture operation, etc. Specifically, in this example, in order to reduce the impact on driving safety caused by operating the first video screen while driving and to improve the convenience of users adjusting the image acquisition device, an adjustment icon can be displayed on the first video screen. In this way, the driver can adjust the image acquisition device by performing corresponding operations on the adjustment icon.

[0126] In which, the display screen can also be configured to display an adjustment icon on the first video screen, and the processor can also be configured to adjust the posture and / or focal length of the image acquisition device in response to the target operation performed on the adjustment icon, so that there is a difference in at least one of the field of view size and shooting angle between the first video screen and the second video screen.

[0127] In this example, the adjustment icon can be an icon of any shape, such as a circular icon, a ring icon, a triangle icon, or an aperture icon. In practice, the target operation performed on the adjustment icon can include a touch operation performed by the user on the adjustment icon, such as sliding the adjustment icon with one finger, touching the adjustment icon with two fingers, single-clicking or double-clicking the adjustment icon with one finger, etc. Different types of target operations can indicate different types of adjustments to the image acquisition device. For example, a target operation performed by touching the adjustment icon with two fingers can indicate adjusting the focal length of the image acquisition device, while a target operation performed by touching the adjustment icon with one finger can indicate adjusting the posture of the image acquisition device to change the shooting angle.

[0128] In some exemplary examples, if a target operation includes multiple operations performed continuously, the processor can simultaneously adjust the posture and focal length of the image acquisition device. For example, if the target operation includes sliding an adjustment icon by touching it with a single finger, and then detecting an operation of touching the adjustment icon with two fingers after sliding to a position, the focal length of the image acquisition device can be adjusted, as well as the posture of the image acquisition device, thereby simultaneously changing the shooting angle and field of view of the next captured video frame.

[0129] In this example, the adjustment icon can be displayed on the first video screen. Specifically, the processor can add a layer on the first video screen, and the adjustment icon can be located in the layer. Specifically, the layer can be a transparent layer, and its adjustment icon can have a certain degree of transparency, such as 30% to 60% transparency, so as not to affect the complete display of the first video screen. In this case, the adjustment icon can be located entirely in the first video screen, or there can be an overlapping portion with the first video screen. Among them, when the adjustment icon is located above the first video screen, by touching the adjustment icon and performing the corresponding operation, the video screen below is switched to the second video screen, which can bring the user a more intuitive adjustment experience.

[0130] The adjustment icon may not be displayed on the first video screen, but may be displayed in an area different from the display position of the first video screen, such as the left or right area of ​​the first video screen, or displayed on both sides. When displayed in the left area, it is convenient for users on the left side of the display screen to operate the adjustment icon, and when displayed in the right area, it is convenient for users on the right side of the display screen to operate the adjustment icon. In this case, the display screen can be configured with a control area to display various controls for adjusting the video screen in the control area, and the adjustment icon can be located in the control area.

[0131] In a further example of this exemplary embodiment, the adjustment icon can be in the form of a frame to facilitate the user to adjust the size, position, and size of the frame, thereby adjusting the posture and / or focal length of the image acquisition device. Different operations on the frame can correspond to different operations on the posture and / or focal length. For example, an overall scaling operation on the size of the frame can correspond to a focal length adjustment operation, and an enlargement or reduction of a side of the frame can correspond to a posture adjustment operation. Of course, in practice, other operations can also be performed on the frame. For example, if the frame is a trapezoidal frame, the other operation can also be an adjustment operation on the tilt of the trapezoidal frame.

[0132] 4 , a schematic diagram of an interface of a display screen is shown. As shown in FIG4 , the adjustment icon includes a first frame, and the target operation may include a zoom operation on the first frame and a size change operation on at least one side length; wherein,

[0133] The processor is specifically configured to adjust the posture of the image acquisition device in response to a size change operation so that there is a change in picture content between the first video screen and the second video screen; and, in response to a zoom operation, adjust the focal length of the image acquisition device so that there is a change in field of view size between the first video screen and the second video screen.

[0134] In this example, the first frame can be a rectangular frame, a trapezoidal frame, and specifically a rounded rectangular frame or a right-angled rectangular frame, wherein different target operations can be performed on the first frame to adjust the posture or focal length of the image acquisition device. Specifically, the target operations performed on the first frame can include a zoom operation and an operation to change the size of at least one side, wherein the zoom operation can refer to the overall reduction or enlargement of the first frame, which can be used to adjust the focal length. For example, the zoom operation can include the operation of moving two fingers closer to or further away from each other by touching the first frame, such as moving closer to each other means shortening the focal length, thereby expanding the field of view, and moving further away from each other means increasing the focal length, thereby reducing the field of view.

[0135] Resizing at least one side can change the extension area of ​​the border on a particular side. The side whose extension area is changed can be used to indicate the offset direction of the image capture device's shooting angle. For example, as shown in FIG4 , if the border extends to the right, the image capture device can be offset outward, thereby shifting the shooting angle to the left, causing content on the left side that was not originally in the first video screen to appear in the second video screen. In a specific implementation, the offset direction and offset angle of the image capture device can be determined based on the difference between the adjusted first border and the unadjusted first border, thereby adjusting the position of the image capture device according to the offset direction and offset angle.

[0136] The difference between the adjusted first frame and the unadjusted first frame may include a difference in side lengths and a difference in inner angles of the frames.

[0137] For example, as shown in FIG4 , after adjustment, the first frame is extended to the right side compared to before adjustment. The extension size is assumed to be 1 cm. Then 1 cm can be mapped to the coordinate system of the image acquisition device to obtain the angle of the image acquisition device offset to the left. Assuming that the angle of offset to the left is 20 degrees, the processor can send a control instruction to the image acquisition device to make the image acquisition device offset 20 degrees to the right.

[0138] By adopting the technical solution of this example, when the adjustment icon is the first border, it can allow the first border to be scaled and the edge size to be changed. Thus, through different operations on the first border, the posture and field of view of the image acquisition device can be quickly adjusted.

[0139] Furthermore, in one example, the image acquisition device may be a panoramic type image acquisition device and a non-panoramic type image acquisition device. For the panoramic type image acquisition device, it may have a 360-degree viewing angle, which means that it may be offset within a 360-degree range, while the non-panoramic type image acquisition device generally has a 180-degree viewing angle, which means that it may be offset within a 180-degree range. Thus, when adjusting the posture of the image acquisition device, for different types of image acquisition devices, the process of calculating the offset angle and offset direction based on the adjusted first frame may be different.

[0140] In a specific implementation, when the position of the image capture device is adjusted in response to a resizing operation on the length of a side of the first bounding box, the adjusted size parameters of the first bounding box can be obtained based on the type of the image capture device. For example, different size parameters of the first bounding box can be obtained for different types of image capture devices. The size parameters can then be substituted into a preset function corresponding to the image capture device to obtain a rotation angle of the image capture device, and the image capture device can be rotated according to the rotation angle.

[0141] In this embodiment, the image acquisition device may include a panoramic image acquisition device and a non-panoramic image acquisition device. For a panoramic image acquisition device, the size parameters of the first frame obtained may be different from those for a non-panoramic image acquisition device. As described above, when adjusting the posture based on the difference between the adjusted first frame and the unadjusted first frame, the difference may include a difference in the size of the side lengths and a difference in the inner angles of the frame. Thus, for different types of image acquisition devices, the differences in the first frame obtained may be different. For example, for a panoramic image acquisition device, the difference in the inner angles of the frame may be obtained, while for a non-panoramic image acquisition device, the difference in the size of the side lengths may be obtained.

[0142] The size parameter can refer to the difference in the inner angles of the frame or the size difference in the side lengths. When adjusting the position of the image acquisition device according to the size parameter, the size parameter can be substituted into a preset function corresponding to the image acquisition device. The rotation angle of the image acquisition device is calculated by the preset function. Different types of image acquisition devices correspond to different preset functions. The preset function can be understood as a mapping function that converts the size parameter into the rotation angle of the image acquisition device in the world coordinate system, reflecting the conversion relationship between the pixel coordinates of the first frame on the display screen and the world coordinates of the image acquisition device.

[0143] By adopting the technical solution of this example, different size parameters can be obtained for different types of image acquisition devices. It should be noted that the size parameters can match the type of image acquisition device, thereby achieving precise adjustment of the posture of the image acquisition device.

[0144] Furthermore, in one example, different size parameters of the first bounding box can be obtained due to different types of image acquisition devices. In practice, the image acquisition devices may include panoramic image acquisition devices and non-panoramic image acquisition devices. A panoramic image acquisition device is a camera that can have a 360-degree horizontal viewing angle, or a viewing angle that can cover the entire sphere, which deviates from the real-world perspective; a non-panoramic image acquisition device has a horizontal viewing angle of no more than 180 degrees, and captures light rays from a hemisphere that fall on the camera's focal point.

[0145] Therefore, different types of image acquisition devices have different benchmarks when changing their postures. To facilitate users to more intuitively predict changes in perspective when viewing the posture of the first-frame image acquisition device, different first frames can be indicated for different types of image acquisition devices. In other words, the first frame displayed on the first video screen is related to the type of image acquisition device. Accordingly, the processor is further configured to display a corresponding type of first frame on the first video screen based on the type of image acquisition device. Specifically, the display screen can render the first frame on the first video screen.

[0146] In this example, when the type of the image acquisition device is the first type, the first frame is a rectangular frame, the size parameter includes the length of each side of the first frame, and the preset function includes a first preset function, which is used to represent the correspondence between the length of the first frame and the rotation angle;

[0147] Among them, when the type of the image acquisition device is the second type, the first border is a trapezoidal frame, the size parameters include the angles of each inner corner of the first border, the preset function includes a second preset function, and the second preset function is used to characterize the correspondence between the difference between the inner angles of the first border and the rotation angle.

[0148] The first type may be a non-panoramic type, and the second type may be a panoramic type. That is, when the image acquisition device is of a non-panoramic type, the first border displayed may be a rectangular frame, and the user may adjust the position of the image acquisition device by adjusting the side length of the rectangular frame; when the image acquisition device is of a panoramic type, the first border displayed may be a trapezoidal frame, and the user may also adjust the position of the image acquisition device by adjusting the side length of the rectangular frame. In this case, on the one hand, the trapezoidal frame can remind the user that it is a panoramic type image acquisition device, and on the other hand, the trapezoidal frame can intuitively display the deviation between the panoramic camera and the real world angle.

[0149] 5a and 5b , FIG5a shows a schematic diagram of an interface of a display screen under a first type of image acquisition device, and FIG5b shows a schematic diagram of an interface of a display screen under a second type of image acquisition device.

[0150] As shown in FIG5a , the rectangular frame corresponding to the non-panoramic image acquisition device is adjusted by extending the rectangular frame downward, that is, lengthening the lengths of the left and right sides, which means that the shooting angle of the image acquisition device needs to be offset toward the ground. The size parameters can be the lengths of the left and right sides. For example, the downward adjustment angle of the image acquisition device can be achieved by a first preset function, wherein the first preset function can be the following formula (1): y = (b / a + b) * π / 2 Formula (1)

[0151] In which, considering the rotation speed of the image acquisition device, the rotation coefficient y=ξ(b / a+b)*π / 2 can be added, where ξ∈(0,1).

[0152] In formula (1), y represents the offset angle of the image acquisition device, represents the size of the changed side length before the change, and b represents the size difference of the changed side length before and after the change, which can be an absolute value.

[0153] It should be noted that the downward adjustment described in Figure 5a is based on the manual instruction shown, with the midline of the current viewing angle of the image acquisition device as the reference line, and the adjustment angle is downward from the viewing angle midline. When the manual instruction is in other directions, the adjustment is also performed according to this principle.

[0154] As shown in Figure 5b, the panoramic image acquisition device corresponds to a trapezoidal frame. This is because the panoramic image deviates from the angle of the real world. The picture corresponding to the panoramic image acquisition device is an irregular trapezoid. It is difficult for the naked eye to distinguish the changes in these angles. Depending on the model of the image acquisition device, there is only a difference of about 1-3 degrees. The rotation angle of the image acquisition device can be determined based on the various internal angles of the trapezoid.

[0155] For example, as shown in FIG5b , when the image acquisition device is twisted downward, a increases and b decreases, so (ab) should be a positive number, and its absolute value increases as the twist angle increases; c increases and d decreases, so (cd) is a positive number, and its absolute value increases as the twist angle increases; when the image acquisition device is twisted upward, the angle changes in the opposite direction, a decreases and b increases, (ab) is a negative number, and its absolute value increases as the twist angle increases; c decreases and d increases, so (cd) is a negative number, and its absolute value increases as the twist angle increases;

[0156] Therefore, the sum of the differences (ab) + (cd) can be used as the final basis for the algorithm to calculate the rotation angle. In this way, the difference will be magnified 4 times, and whether the difference is a positive or negative number can be used to determine whether the image acquisition device should be twisted upward or downward.

[0157] The size parameter may be each inner angle of the trapezoidal frame after the change, and the second preset function may be as shown in the following formula (2): y=a'x2+b'x+c' Formula (2)

[0158] Where x represents the value of (ab) + (cd), a', b' and c' are coefficients, y represents the torsion angle of the image acquisition device, and the fitting method adopts the least squares method. The final fitting quadratic polynomial is shown in formula (3): y = 0.104x2 + 5.246x - 0.893 Formula (3)

[0159] The coefficients a', b' and c' may vary with different hardware tests and are not limited here.

[0160] For example, as shown in the following table, the rotation angles corresponding to the angle differences of different internal angles under adjustment of different scales are shown. The first row of Table 1 is the value of (ab) + (cd), which is recorded here as the angle difference; the second row is the rotation angle of the camera relative to the perpendicular bisector of the midline of the viewing angle, 0 degrees represents the vertical midline of the viewing angle, a positive angle indicates that the camera is twisted downward, and a negative angle indicates that the camera is twisted upward.

[0161] Table 1 - Rotation angles corresponding to different angle differences

[0162] When fitting from the above formula (2) to the above formula (3), the test data shown in Table 1 can be used for fitting, and finally formula (3) can be used as the second preset function. It should be noted that when the image acquisition device needs to be adjusted left and right, it is only necessary to replace X in formula (3) with: x = (ac) + (db). In this case, a positive angle indicates a left twist, and a negative angle indicates a right twist.

[0163] As described above, in some examples, the display device can be connected to multiple different image acquisition devices, and the different image acquisition devices can be located at different positions in the vehicle. The display screen can then display the video images captured by the multiple image acquisition devices in split screens. For example, if nine image acquisition devices are included, the display screen can be arranged in a nine-grid format, including nine display areas, each of which can display a video image captured by one image acquisition device. The first border can be entirely located in the split screen where the first video image is located, or partially located in the split screen where the first video image is located. In either case, when the size of the first border is changed, such as by scaling or adjusting the length of the side, the size of each split screen can be kept unchanged, while only the posture and / or focal length of the image acquisition device is adjusted.

[0164] Accordingly, after the size change operation, the first border can extend beyond the split screen where the first video screen is located and occupy a portion of at least one first split screen adjacent to the split screen; wherein, before and after the target operation, the processor is configured to keep the size of the split screen and the first split screen unchanged.

[0165] For example, reference can be made to Figure 6a, which shows a schematic diagram of changes in the display interface with the first border. As shown in Figure 6a, the first border is displayed above the first video screen. Before adjustment, as shown by the white box in Figure 6a, the entire first border is located above the first video screen. After adjustment, as shown by the black box in Figure 6a, the lower part of the first border extends to an adjacent first split screen. In this case, the first border exceeds the split screen where the first video screen is located, and occupies a portion of at least one first split screen adjacent to the split screen.

[0166] Of course, in some other examples, after the first border is adjusted, the first split screen where the excess part is located can be any split screen adjacent to the split screen above, below, left, and right. In practice, the part of the first border that exceeds the split screen where the first video screen is located can occupy two, three, or even four split screens at the same time, and the present disclosure does not impose specific restrictions on this.

[0167] Regardless of which first split screen the adjusted first border occupies, the adjustment of the first border may not affect the size of the split screen where the first video screen is located and the size of the first split screen. Thus, the size of the first video screen may remain unchanged before and after the target operation.

[0168] Of course, in some embodiments, as the first border occupies a portion of at least one first split screen adjacent to the split screen where the first video screen is located after being adjusted, the size of the split screen where the first video screen is located and the first split screen can be dynamically changed according to the adjusted first border. As shown in Figure 6a, when the posture and field of view of the image acquisition device are adjusted, the size of the split screen where the first video screen is located can be changed synchronously, that is, the screen size of the split screen where the first video screen is located can be changed synchronously, so that there are not only differences in field of view and shooting angle between the first video screen and the second video screen, but also differences in screen size.

[0169] In another example of this exemplary embodiment, when the display device displays video images captured by multiple image acquisition devices in a split screen, it can support the user to adjust the size of the split screen, such as increasing the size of the split screen or reducing the size of the split screen, so that the video image displayed in the split screen is adaptively adjusted. By adjusting the size of the split screen, the size of the video image is adjusted to meet the user's various viewing needs.

[0170] In this example, the adjustment icon may include a second border, and the display screen may also display the second border on the first video screen. By adjusting the second border, the size of the split screen may be adjusted. Specifically, the processor may be further configured to: upon detecting that the second border, after adjustment, exceeds the split screen where the first video screen is located and occupies a portion of at least one second split screen adjacent to the split screen where the first video screen is located, synchronously adjust the size of the split screen where the first video screen is located and the size of the second split screen so that the adjusted first border is entirely located in the split screen where the adjusted first video screen is located;

[0171] Furthermore, the screen size of the video frame to be displayed is adjusted so that the video screen of the image acquisition device covers the entire split screen where the adjusted first video screen is located.

[0172] For example, reference can be made to Figure 6b, which shows a schematic diagram of the change in the display interface displaying a second border. As shown in Figure 6b, the second border is displayed above the first video screen. Before adjustment, the entire second border is located above the first video screen. After adjustment, the lower part of the second border extends to an adjacent second split screen. In this case, the second border exceeds the split screen where the first video screen is located and occupies a portion of at least one adjacent second split screen.

[0173] In this example, after the second border is adjusted, the second split screen where the excess part is located can be any split screen adjacent to the split screen where the first video screen is located, in the top, bottom, left, and right directions. In practice, the part of the second border that exceeds the split screen where the first video screen is located can occupy two, three, or even four split screens at the same time, and this disclosure does not impose any specific restrictions on this.

[0174] The sizes of the split screen where the first video image is located and the second split screen may be adjusted, while the sizes of other split screens do not need to be adjusted.

[0175] In practice, after adjusting the second border, if the second border is reduced, the split screen where the first video screen is located will also be reduced synchronously. If the second border is enlarged, and the second border does not exceed the split screen where the first video screen is located after enlargement, the split screen where the first video screen is located will also be expanded synchronously. If the second border is enlarged, and the second border exceeds the split screen where the first video screen is located after enlargement and occupies part of the second split screen, the split screen where the first video screen is located will be enlarged and the second split screen will be reduced. Regardless of which adjustment is made, the adjusted second border can be located entirely in the split screen where the adjusted first video screen is located.

[0176] Among them, after the second frame is adjusted, the split screen where the adjusted first video screen is located can overlap with the adjusted second frame, or the split screen where the adjusted first video screen is located can roughly overlap with the adjusted second frame. In this case, the size of the second frame before being adjusted needs to overlap with the boundary of the split screen where the first video screen is located, so that when the processor adjusts the size of each split screen, it can directly adjust it according to the size of the adjusted second frame, thereby optimizing the algorithm and reducing the response time, that is, reducing the switching delay of the split screen, thereby improving the display effect and avoiding the long switching time that affects driving safety.

[0177] Among them, when adjusting the corresponding split screen, the picture size of the video frame to be displayed can also be adjusted synchronously so that the video picture of the image acquisition device fills the adjusted split screen. Among them, the adjustment of the picture size can be performed by changing the width and height of the video frame, which can be achieved by using relevant technologies and will not be elaborated here.

[0178] Of course, in some other examples, the split screen where the adjusted first video screen is located may not need to overlap with the adjusted second border. In this case, the size of the second border before being adjusted may be smaller than the size of the split screen where the first video screen is located. The user can scale the second border to adjust the size of the split screen where the first video screen is located and the second split screen. In this case, the processor can adjust the size of the corresponding split screen according to the size ratio between the second border and the split screen. In this way, there is no need to limit the display of the second border to strictly overlap with the first video screen. The second border only needs to be displayed on the first video screen, thereby improving the flexibility of the display.

[0179] It should be noted that the adjustment operation on the second frame can be a zooming operation on the second frame. The process of the zooming operation can refer to the process of the zooming operation of the first frame to change the field of view, which will not be repeated here.

[0180] Among them, the first frame and the second frame can be displayed on the first video screen at the same time, or they can be displayed on the first video screen at different times. When displayed at the same time, the display colors of the first frame and the second frame can be different, or the line styles of the first frame and the second frame can be different, so as to facilitate user distinction. In some examples, when the first frame and the second frame can be displayed on the first video screen at the same time, when an adjustment operation is performed on one of the frames, the display of the adjusted frame can be canceled and the display of the other frame can be retained. For example, if a zoom operation is performed on the first frame, the field of view of the image acquisition device can be adjusted. At this time, the display of the first frame is canceled, while the second frame remains displayed. For example, the second frame is displayed on the second video screen after switching, thereby allowing the user to change the size of the split screen where the second video screen is located by operating the second frame.

[0181] In the case where the first and second frames are not displayed simultaneously, if the currently displayed frame is adjusted, the other frame can be automatically displayed on the adjusted second video screen. For example, if a zoom operation is performed on the displayed first frame, the field of view of the image acquisition device can be adjusted. At this time, the second frame can be automatically displayed on the second video screen after switching, allowing the user to change the size of the split screen in which the second video screen is located by operating the second frame. In some cases, the display duration of the other frame (such as the second frame) can be set. If it is not operated within the specified display duration, the display of the other frame can be cancelled.

[0182] The display of the first and second frames may be triggered based on a preset instruction, such as a user's voice instruction containing specific words or a specific gesture instruction, or may be triggered by a control in the control area of ​​the display screen, which will not be described in detail here. It should be noted that when the preset instruction is triggered, the first and second frames may be displayed simultaneously, or a frame corresponding to the preset instruction may be displayed, and this disclosure does not impose any special restrictions.

[0183] As described in the above example, the processor can also be configured to process the image of the video frame to improve the image quality of the video frame, so that the image quality of the second video image is better than the image quality of the first video image. As mentioned above, in the in-vehicle display, the necessary video images are required to have low latency, such as the rearview mirror image requires low latency, and the important video images are required to have high quality. For the same video image, if it is a necessary video image and an important video image, it is required that the processing time of the video frame is less than the minimum delay time when the image quality is improved. However, it is difficult for the in-vehicle display in the related art to ensure that the delay time and image quality are taken into account at the same time when the same video image is both a necessary image and an important image. In this example, in order to ensure both low latency of the necessary image and high quality of the important image, the second video image can be displayed at the original position or other position of the first video image, so as to take into account both low latency and image quality of the video image captured by the image acquisition device.

[0184] Specifically, the processor can also be configured to display the second video screen at the first display position where the first video screen is located when the target operation indicates an adjustment to the picture quality, or to display the second video screen at a second display position different from the first display position; wherein the interval length between the display moment of the second video screen at the second display position and the target operation is greater than the interval length between the display moment of the second video screen at the first display position and the target operation.

[0185] In this example, the picture quality of the second video screen is higher than that of the first video screen. From the visual interface of the display screen, if the second video screen is displayed at a second display position different from the first display position, it means that the processing time from the video frame to the second video screen is longer, and the second video screen needs to be displayed separately at another display position to ensure the low latency of the video screen displayed at the first display position. In this case, the video screen displayed at the first display position can be an unprocessed video frame or an intermediate screen processed as the second video screen. Among them, if the second video screen is displayed at the first display position, the screen at the first display position switches from the first video screen to the second video screen, and the switching time meets the low latency. It should be noted that the interval between the second video screens displayed continuously in the first display position must meet the low latency time.

[0186] The low delay time is the low delay time corresponding to the image acquisition device, for example, it can be a value less than 100ms.

[0187] Therefore, if the second video image is displayed at the second display position, the time interval between the display time of the second video image at the second display position and the target operation can be longer than the time interval between the display time of the second video image at the first display position and the target operation. In other words, the display time of the second video image at the first display position is earlier than the display time of the second video image at the second display position. Therefore, by adjusting the display position of the second video image, low latency and high-quality video images can be displayed for users to watch.

[0188] In some examples, if the second video frame is displayed at the second display position, the first display can still maintain real-time display of the video frame captured by the image acquisition device, thereby ensuring low-latency display and high-definition display of the frame. Specifically, the processor can be further configured to display the third video frame captured by the image acquisition device at the first display position in real time while the second video frame is displayed at the second display position; wherein the interval between the video frames displayed at the first display position is less than the preset delay period.

[0189] In this example, when the second video screen is displayed at the second display position, the video frame captured in real time by the image acquisition device can be directly displayed at the first display position, that is, the unprocessed video frame is displayed at the first display position, so as to ensure the low-latency display of the video screen in the first display position. Among them, the video frame displayed at the first display position and not processed is called the third video screen. It should be noted that the third video screen is the video frame captured in real time by the image acquisition device. In this case, the video frames captured by the image acquisition device are divided into two paths, one path is processed into the second video screen and then displayed at the second display position, and the other path is directly displayed as the third video screen at the first display position.

[0190] Accordingly, the interval between the third video frames displayed at the first display position is less than the preset delay, which is the minimum delay corresponding to the image acquisition device, such as 80ms. Specifically, based on the parallax of the interval between the video frames displayed on the display screen, the interval between the third video frames displayed at the first display position is less than the interval between the second video frames displayed at the second display position.

[0191] For example, referring to Figures 7a and 7b, two schematic diagrams of changes in display interfaces are shown. As shown in Figure 7a, when switching from a first video screen to a second video screen, both the second video screen and the first video screen are displayed at the first display position, i.e., the original position of the first video screen. The second video screen is at the first display position at time t5, and the interval between t5 and the display time of the first video screen is T1. In this case, the video frame captured by the image acquisition device is processed and displayed at the first display position. As shown in Figure 7b, when switching from the first video screen to the second video screen, the second video screen is displayed at the second display position, and the first display position maintains the display of the original video frame captured by the image acquisition device. The second video screen is at the second display position at time t6. At this time t6, the original position, i.e., the first display position, has already displayed the N+j-th video frame, while the second display position displays the N-th video frame. The interval between t6 and the display time of the first video screen is T2, and T2 is greater than T1.

[0192] Of course, in some examples, it is not ruled out that when the second video screen is displayed at the second display position, the interval between each second video screen displayed in the second display position is also less than the preset delay time, and the disclosure does not impose special restrictions on this situation.

[0193] In another example of this exemplary embodiment, whether the second video image is displayed at the first display position or the second display position can be determined based on the time required to process the video frame into the second video image. If the time is long, such as exceeding a preset delay time, the second video image can be displayed at the second display position. If the time is short, such as not exceeding the preset delay time, the second video image can be displayed at the first display position. Accordingly, the processor can be specifically configured to determine the time required to adjust the image quality of the video frame when the target operation indicates adjusting the image quality, and schedule the second video image to the first display position or the second display position based on the adjustment time.

[0194] As can be seen from this example, the adjustment time consumed by processing the video frame into the second video screen determines the display position of the second video screen, wherein, when the adjustment time exceeds the preset delay time, the second video screen can be displayed at the second display position, and when the adjustment time does not exceed the preset delay time, the second video screen can be displayed at the first display position. Specifically, when the processor responds to the target operation indicating the improvement of the picture quality, it can obtain the video frame to be displayed and schedule the video frame to the corresponding processing thread to estimate the adjustment time consumed by processing the video frame into the second video screen. When estimating the adjustment, feature extraction can be performed on the video frame to determine the current picture quality of the video frame, and then the adjustment time consumed by processing the video frame into the second video screen can be calculated based on the extracted features. The extracted features can include pixel features, texture features, etc. of the video frame, wherein the pixel features can reflect the brightness and clarity of the video frame, etc., and the texture features can reflect the continuity of the texture, thereby reflecting whether there is any weather factor blocking. Therefore, the current picture quality of the video frame can be determined by the extracted features.

[0195] Among them, the process of estimating the adjustment time can be carried out simultaneously with the process of processing the video frame into the second video picture. This is because the estimated adjustment time is less than its processing time, and does not occupy the time for processing the second video picture, so that the low delay of the video picture can be guaranteed as much as possible.

[0196] In this example, the difference between the picture quality of the video frame and the picture quality indicated by the target operation can reflect the depth of processing of the video frame. For example, the greater the difference, the deeper the processing depth, indicating that more levels of processing are required to achieve the indicated picture quality; the smaller the difference, the shallower the processing depth, indicating that fewer levels of processing are required to achieve the indicated picture quality. Therefore, based on this difference, it is possible to determine which processing needs to be performed on the video frame, that is, which types of processing are to be performed. Then, after determining which types of processing are to be performed, the preset adjustment time corresponding to each processing can be estimated, and the sum of the preset adjustment times corresponding to multiple processing can be used as the adjustment time.

[0197] Specifically, when determining the display position of the second video screen based on the adjustment time, the preset delay time corresponding to the level can be determined based on the level of the image acquisition device. If the adjustment time is greater than the preset delay time, the second video screen can be displayed at the second display position. If the adjustment time is less than the preset delay time, the second video screen can be displayed at the first display position.

[0198] Specifically, image capture devices at different locations on the vehicle may correspond to different levels. For example, image capture devices at the left and right rearview mirrors may correspond to a higher level and a shorter preset delay time, while image capture devices at the front windshield may correspond to a lower level and a longer preset delay time. More specifically, image capture devices at the same location may also correspond to different levels under different driving conditions. For example, the preset delay time corresponding to the left rearview mirror in a lane-changing condition may be shorter than that in a non-lane-changing condition, i.e., the level corresponding to the lane-changing condition is higher than that in a non-lane-changing condition. Therefore, when determining the display position of the second video image based on the adjustment time, the processor may retrieve the current vehicle operating condition parameters and the position parameters of the image capture device, determine the current level based on the operating condition parameters and the position parameters, and then, based on the corresponding relationship between the level and the preset delay time, determine the final preset delay time. The processor then compares the preset delay time with the adjustment time to determine the final display position.

[0199] In a further example of this exemplary embodiment, when determining the adjustment time, the adjustment time may be related to the picture quality of the video frame and the current performance parameters of the processor. For example, the worse the picture quality, the longer the adjustment time may be, and the higher the current performance parameters representing the current performance of the processor are, the shorter the adjustment time may be.

[0200] In practice, the processor can be specifically configured to: determine multiple processing to be performed based on the first picture quality of the video frame and the second picture quality indicated by the target operation; and compensate for the preset adjustment time corresponding to each processing based on the data volume of the video frame and / or the current performance parameters of the processor; and determine the adjustment time based on the compensated preset adjustment time.

[0201] As described above, multiple processes to be performed can be determined based on the difference between the first picture quality and the second picture quality indicated by the target operation. In this example, in order to more accurately calculate the adjustment time, the factors that affect the adjustment time can be used as compensation to adjust the preset adjustment time corresponding to each preset process. Specifically, when the processor obtains a video frame, it can determine its data volume from the attribute information of the video frame. The larger the data volume, the longer the adjustment time. In addition, the processor can obtain its own current performance parameters. The current performance parameters may include the proportion of available computing resources. The higher the proportion, the more available computing resources and the shorter the adjustment time.

[0202] In practice, the preset adjustment time corresponding to each processing can be compensated based on the data volume of the video frame; or, the preset adjustment time corresponding to each processing can be compensated based on the current performance parameters of the processor; or, the preset adjustment time corresponding to each processing can be compensated based on the data volume of the video frame and the current performance parameters.

[0203] During compensation, multiple data volume ranges and multiple proportion ranges can be preset, and a first adjustment coefficient can be set for each data volume range, and a second adjustment coefficient can be set for each proportion range. Then, based on the first adjustment coefficient corresponding to the data volume range in which the data volume of the video frame is located, the preset adjustment time corresponding to each processing can be compensated, and the second adjustment coefficient corresponding to the proportion range in which the current proportion is located can be used to compensate for the preset adjustment time corresponding to each processing. If it is necessary to compensate for the preset adjustment time based on both the data volume and the current performance parameters, the weighted average of the corresponding second adjustment coefficient and the first adjustment coefficient can be used as the final adjustment parameter to compensate for the preset adjustment time.

[0204] Accordingly, when determining the multiple processes to be performed, the different picture qualities of the video frames may correspond to different types of processing. Specifically, the video frames may be analyzed to determine the environment type corresponding to the video frames. The environment type may characterize the picture quality of the video frames. For low-quality video frames, multiple levels of processing are required to achieve the second picture quality. For high-quality video frames, fewer levels of processing are required to achieve the second picture quality. The environment type may also characterize the weather type of the vehicle's environment, such as a dusty environment, a rainy and snowy environment, a clear environment, a foggy and hazy environment, etc. The different weather types of the vehicle's environment directly affect the picture quality of the video frames it collects.

[0205] Specifically, referring to FIG8 , a schematic flow chart of the steps of the processor in determining multiple processes to be executed is shown. As shown in FIG8 , the following steps may be performed:

[0206] Step S101: determining an environment type corresponding to a video frame, where the environment type is used to characterize the type of interference factors that affect image quality in the environment where the image acquisition device is located;

[0207] Step S102: determining multiple processing methods based on the difference between the environment type and the preset environment type, and the second picture quality; wherein the preset environment type indicates that there are no interference factors that affect the picture quality.

[0208] In this embodiment, the preset environment type may be a clear environment, meaning the vehicle is located in a relatively clear environment. In this case, the captured video frames have a relatively high image quality and require less processing to achieve the second image quality. If the environment type is not the preset environment type, it means the vehicle is located in a relatively unfavorable environment. In this case, the captured video frames have a relatively low image quality and require more processing to achieve the second image quality.

[0209] Among them, the interference factors can be understood as weather factors in the environment where the vehicle is located, which generally include dust factors caused by a dusty environment, rain and snow factors caused by a rainy and snowy environment, and dense fog factors caused by a haze environment.

[0210] Referring to FIG9 , a schematic diagram of the processing flow for processing video frames of different environmental types is shown. As shown in FIG9 , the processor can determine the environmental type of the video frame based on the pixel data of the video frame. For example, the environmental detection algorithm is used to determine the environmental type. The preset environmental type is a clear environment type. When the environmental type of the video frame is the preset environmental type, the processing it undergoes includes: image quality enhancement and color enhancement processing, as well as appropriate post-processing. When the environmental type of the video frame is not the preset environmental type, such as a low-light type, rainy type, snowy type, foggy type, or dusty type, the processing it undergoes includes: extreme scene restoration processing, image quality enhancement and color enhancement processing, as well as appropriate post-processing. The extreme scene restoration processing includes low-light enhancement, rain removal, fog removal, and dust removal. In other words, it removes interference factors in the video frame and reconstructs the image of the video frame, thereby obtaining a more complete and clear image.

[0211] Therefore, when the environment type is a preset environment type, the types of processing included are relatively few, and when the environment type is not a preset environment type, the types of processing included are relatively more. Specifically, when the processor determines the environment type of the video frame, it can call the environment detection algorithm. In one example, the environment detection algorithm can be as follows:

[0212] The video frame is converted to a target color space, and the brightness value and color temperature value of each pixel in the target color space are extracted; and the environment type is determined according to the brightness value and the color temperature value.

[0213] The target color space may be an HSV (Hue, Saturation, Value) color space, which refers to a subset of visible light in the H, S, V three-dimensional color space, and includes all colors in a certain color domain. Next, the brightness value of the brightness component converted to the target color space may be extracted, and then the brightness average of the brightness values ​​of each pixel may be calculated. Next, a brightness threshold may be called, and by comparing the brightness threshold with the brightness average, whether the video frame belongs to a dark light environment may be determined; and the color temperature value of the color temperature component converted to the target color space may be extracted, and then the color temperature average of the color temperature values ​​of each pixel may be calculated. Next, a color temperature threshold may be called, and by comparing the color temperature threshold with the color temperature average, whether the video frame belongs to an environment such as dust, rain, or snow may be determined.

[0214] For example, if the average brightness ∈ [0-25], it is considered to be night, and the average color temperature ∈ [3000-4000] is considered to be dust, etc.

[0215] Accordingly, the environment detection algorithm is as follows:

[0216] Figure 9 illustrates the calculation logic for adjusting the time consumption. Specifically, based on the inference time of each link measured in advance on the processor, let the pre-processing time be t1, the post-processing time be t2, the environment detection algorithm time be t3, the image quality assessment algorithm time be t4, the extreme scene restoration algorithm time be t5, the image quality enhancement algorithm time be t6, and the color enhancement algorithm time be t7. If extreme scene restoration, image quality enhancement, or color enhancement are not required, the fastest total link time is t = t1 + t2 + t3 + t4. In other cases, based on the results of the environment detection algorithm and image quality assessment algorithm, the corresponding extreme scene restoration, image quality enhancement, and color enhancement module times are added to t to obtain the adjustment time consumption.

[0217] Among them, since the picture quality of the video frames captured by the image acquisition device can also be improved, in order to facilitate the user to directly operate through the display screen to indicate to improve the picture quality, multiple menu keys can be displayed on the display screen, and the user can select the desired picture quality by triggering the multiple menu keys. For example, referring to Figure 10, a display interface schematic diagram of a display screen is shown. As shown in Figure 10, the display screen can also be configured to display multiple menu keys, wherein the processor can also be configured to adjust the picture quality of the video frame in response to the triggering operation of at least one menu key among the multiple menu keys, so that the picture quality of the second video picture is higher than the picture quality of the first video picture.

[0218] In this example, multiple menu keys can be displayed in the control area of ​​the display screen. Different menu keys can correspond to different dimensions of picture quality, such as a menu key for adjusting brightness, a menu key for adjusting color, a menu key for adjusting style, a menu key for adjusting resolution, etc. Each menu key can be used alone or in combination. When used alone, it adjusts the picture quality of a certain dimension. When used in combination, it can adjust the picture quality of multiple dimensions.

[0219] When the processor detects a triggering operation on the menu key, it can adjust the video frame according to the picture quality indicated by the triggered menu key, so that the picture quality of the second video picture is higher.

[0220] In this case, each menu key can be associated with a processing logic, so that the multiple processes shown in Figure 9 can be visualized as a menu key displayed on the display screen. Therefore, when calculating the adjustment time, the corresponding multiple processes can be determined based on the processing logic associated with the triggered menu key. For example, if three menu keys are selected, three processes will correspond. In this way, there is no need to calculate the adjustment time based on the difference between the picture quality of the video frame and the indicated picture quality, thereby reducing the time and computing resources consumed by the environmental detection link and improving the response speed of the processor.

[0221] By adopting the technical solution of this embodiment, the menu key is used to help improve the picture quality of the video frame, thereby improving the convenience of user operation.

[0222] In some other examples, due to the need to improve the picture quality of the video frame to be displayed, as described above, the processing process for improving the picture quality of the video frame may be different depending on the difference in the picture quality of the video frame itself. For example, if the picture quality of the video frame is poor, more levels of picture processing are required to achieve the required picture quality. Accordingly, the processor can be configured to determine the environment type corresponding to the video frame when adjusting the picture quality of the video frame, and based on the environment type, determine multiple processing to be performed on the video frame, and dispatch the video frame to the target image processing model corresponding to each processing in turn, so as to improve the picture quality of the video frame through the target image processing model.

[0223] Among them, the process of determining the environment type corresponding to the video frame can refer to the description of the above embodiment and will not be repeated here. In this example, when multiple processing is determined, the processor can dispatch the video frame to the target image processing model corresponding to each processing in sequence according to the execution order of the multiple processing, so as to improve the picture quality of the video frame through the target image processing model; wherein, multiple target image processing models can be configured in the processor, or multiple target image processing models can be configured in the memory of the display device, or multiple target image processing models can be configured in the cloud and called when needed; when processing needs to be performed, the processor can call the target image processing model and input the video frame into the target image processing model corresponding to each processing in sequence for corresponding processing.

[0224] It should be noted that, according to the execution order of multiple processing, the processor can input the video frame output by the previous target image processing model into the next target image processing model, and so on. After processing by multiple target image processing models, the video frame output by the last target image processing model can be displayed as the second video screen; as shown in Figure 9, image quality enhancement, color enhancement and extreme scene restoration algorithms can correspond to three target image processing models respectively.

[0225] In some examples, multiple candidate processing models can also be preset for each type of processing. For example, for image quality enhancement, multiple candidate processing models can be preset. There are differences in computational time and image processing quality between the multiple candidate processing models. The processor can also select the best candidate processing model as the target image processing model to perform image processing of the video frame based on the picture quality of the video frame and the size of the preset delay time corresponding to the image acquisition device. For example, if the tolerance for the preset delay time corresponding to the image acquisition device is high, such as the level is low, it can select the candidate processing model with better image processing quality as the target image processing model; if the tolerance for the preset delay time corresponding to the image acquisition device is low, such as the level is high, it can select the candidate processing model with shorter computational time as the target image processing model. Thus, by presetting multiple candidate processing models, a dynamic balance between the delay time and the picture quality of the picture can be achieved.

[0226] In some examples, picture quality may include picture clarity, picture noise, and color, among which, the higher the resolution of the video picture and the fewer interference factors it contains, the higher the picture clarity; when improving the picture quality of the video frame, the picture clarity can be improved, or the picture noise can be reduced, or the resolution of the video picture can be improved, or the color of the video picture can be enhanced to enhance the contrast of the picture, or both the picture clarity and the picture color can be enhanced; or, both the picture clarity can be improved, the picture noise can be reduced, and the picture color can be enhanced.

[0227] Specifically, to improve image clarity, the processor can eliminate data from certain interfering factors in the image, thereby ensuring that the second video image contains no or minimal interference such as dimming, dust, dense fog, rain, and snow. In a specific implementation, the processor can be configured to divide the video frame into multiple image blocks and perform mask reconstruction on the contents of all the image blocks, so that the second video image contains no or minimal interference factors that affect image quality: wherein the interference factors include at least one of dimming, dust, dense fog, and rain.

[0228] As described in the above embodiment, interference factors generally come from weather factors in the vehicle's environment, such as dim light, dust, dense fog, and rain and snow. In this example, removing interference factors from a video frame can be performed using image reconstruction. During this image reconstruction process, the video frame can be divided into multiple image blocks. The division can be performed equally or unequally. In the case of unequal division, the video frame can be randomly divided into multiple image blocks. The number of image blocks obtained by division can be related to the clarity of the video frame. For higher clarity, the number of image blocks obtained after division can be larger. Thus, the video frame can be divided into fine-grained segments. During reconstruction, the number or area of ​​reconstructed image blocks can be appropriately reduced, thereby improving reconstruction efficiency. For lower clarity, the number of image blocks obtained after division can be smaller. Thus, the video frame can be divided into coarse-grained segments. During reconstruction, the number or area of ​​reconstructed image blocks can be appropriately increased, thereby ensuring reconstruction quality.

[0229] After dividing to obtain multiple image blocks, the contents of some image blocks can be masked and reconstructed. Specifically, mask reconstruction may refer to first masking the image block to be reconstructed, and then reconstructing the contents of the masked image block based on other unmasked image blocks, so that the masked image block contains no or fewer interference factors.

[0230] In a further example, when masking and reconstructing the contents of some or all image blocks, the granularity of dividing each image area can be determined based on the clarity of each image area in the video frame. For example, an image area with higher clarity can be divided into more image blocks to reduce the amount of reconstructed data. For example, an image area with lower clarity can be divided into fewer image blocks to increase the amount of reconstructed data and improve the degree of restoration of this image area. Referring to FIG11, a schematic flow chart of the steps of masking and reconstructing the contents of an image block is shown. As shown in FIG11, the steps may include:

[0231] Step S201: Divide the video frame to obtain a plurality of first image blocks;

[0232] Step S202: dividing each first image block into at least one second image block according to the definition corresponding to each first image block; wherein a first image block with a lower definition is divided into a smaller number of second image blocks;

[0233] Step S203: masking a portion of the second image block;

[0234] Step S204: reconstructing the content of the masked second image block based on the unmasked second image block.

[0235] Referring to Figure 12, a schematic diagram of the process of mask reconstruction of a video frame is shown. As shown in Figure 12, when the video frame is divided to obtain multiple first image blocks, the video frame can be equally divided, such as evenly dividing the video frame into multiple first image blocks, wherein the number of first image blocks may not be specially limited. Figure 12 takes the division into 16 first image blocks as an example. In practice, it can also be divided into 9 first image blocks.

[0236] After obtaining multiple first image blocks, each first image block can be further divided according to its clarity, thereby dividing each first image block into at least one second image block. First image blocks of different clarity can correspond to different numbers of second image blocks. Specifically, the number of second image blocks obtained by dividing a first image block of higher clarity can be greater than the number of second image blocks obtained by dividing a first image block of lower clarity. For example, as shown in FIG12 , the first image block in the upper left corner has a lower clarity and can be left undivided, while the first image block in the lower right corner has a higher clarity and can be divided into nine second image blocks.

[0237] In one example, the definition can be divided into three levels, with each level corresponding to a different division method. For example, for the first image block at the first level of definition, the original size of the first image block can be maintained. For the second level of definition, the first image block can be split twice, into 2*2 second image blocks. For the third level of definition, the first image block can be split four times, into 4*4 second image blocks. This allows the entire video frame to be divided into multiple second image blocks of different sizes, with the different-sized image blocks corresponding to image regions of different definition in different video frames.

[0238] During mask reconstruction, a portion of the second image block can be reconstructed. Specifically, a portion of the second image block can be masked first, with the masked second image block including the second image block having a lower definition. Specifically, the proportion of the masked second image block in the first image block having a lower definition can be higher than the proportion of the masked second image block in the first image block having a higher definition. As shown in Figure 12, the top row of first image blocks has a lower definition, so the proportion of masked second image blocks is higher, accounting for 50%. The second row of first image blocks has a higher definition than the first row, so the proportion of masked second image blocks is lower, accounting for 37.5%. Similarly, the proportion of masked second image blocks in the third row is 25%, and the proportion of second image blocks in the fourth row is also 25%.

[0239] As a result, it is possible to focus on reconstructing image blocks with lower definition in the video, so that the processing of low-definition image areas in the video frame can be targeted and strengthened through masks without the need for full image processing, thereby improving processing efficiency and shortening adjustment time.

[0240] In some examples, a neural network model can also be used when performing mask reconstruction. Specifically, a target model can be configured in the processor. When the target model is called by the processor, it can divide the video frame to be displayed into multiple image blocks and perform mask reconstruction on the contents of all or part of the image blocks according to the process shown in Figure 9 above.

[0241] The target model may be generated using a first image sample of a first type and a second image sample of a second type of the same image as training samples, and the clarity of the first image sample is higher than that of the second image sample.

[0242] In this example, the target model can be configured into the processor, so that when performing mask reconstruction of the image, the target model can be called, such as inputting the data of the video frame into the target model, wherein the target model can perform mask reconstruction on the video frame according to the process of steps S201 to S204 in the above example.

[0243] Specifically, referring to Figure 13, a schematic diagram of the process of training the target model is shown. As shown in Figure 13, when the target model is trained, a training sample can be constructed first. The training sample can include multiple image pairs. Each image pair can include a first image sample of a first type and a second image sample of a second type of the same image. Different image pairs are for different images. For example, if 100 images are included, a high-definition first image sample and a low-definition second image sample of the image can be generated for each image. During the training process, the second image sample can be input into the target model. The target model can perform the above-mentioned steps S201 to S204 on the second image sample to obtain a reconstructed image. Then, the first image sample can be used as supervision to calculate the loss of the target model, and then the parameters of the target model can be updated based on the loss.

[0244] In which, during the training process of the target model, it can generate a mask image based on the first image sample and the second image sample, and reconstruct the mask image to obtain a reconstructed prediction image; wherein, the masked third image block in the mask image comes from the second image sample, and the unmasked image block comes from the first image sample; and, during the training process of the target model, the parameters of the target model are updated based on the difference between the fourth image block corresponding to the position of the third image block in the prediction image and the fifth image block corresponding to the position of the third image block in the first image sample.

[0245] As shown in Figure 13, during the training of a target model, a first image sample can be used to guide the reconstruction of a second image sample, thereby improving the reconstruction quality of the target model. Specifically, a partial area of ​​the second image sample can be masked. The masking process can refer to steps S201 to S203 described above. Then, the first image sample and the second image sample can be merged, retaining the masked image area, to generate a mask image. In other words, the unmasked image blocks in the mask image are all high-definition image areas from the first image sample, while the masked third image blocks are image areas that need to be reconstructed and belong to the second image sample. As shown in Figure 13, based on the obtained mask Fmask, the fog image Ffoggy (second image sample) and the corresponding clear image Fclean (first image sample) in the open source fog image dataset are combined to obtain a combined image Finput (masked image). In this way, all unmasked images can be high-definition images, thereby enhancing the guiding significance of unmasked images for image reconstruction, improving the quality of image reconstruction, and making the reconstructed images more faithful to the real scene. In this way, the target model can gradually learn the rules of image reconstruction, thereby improving the quality of image reconstruction in the inference stage. Therefore, in a driving environment, driving safety can be improved through on-board displays that more faithfully restore the real driving scene.

[0246] In a specific implementation, as shown in FIG12 , a mask may be pre-set, and the mask, the first image sample, and the second image sample may be combined to obtain a mask image. The mask also includes multiple image blocks, and the pixel values ​​corresponding to the masked image blocks are 0, and the pixel values ​​corresponding to the unmasked image blocks are 1.

[0247] As described above, during the training process, the target model generates a mask image based on the first and second image samples. After reconstructing the mask image, a predicted image can be output. To simplify the loss calculation, in this example, the loss corresponding to the reconstructed image block can be calculated. Specifically, the loss corresponding to the masked third image block in the reconstructed image can be calculated. For example, FIG14 illustrates a schematic diagram of loss calculation for a target model. As shown in FIG14 , the loss can be calculated based on the difference between a fourth image block corresponding to the position of the third image block in the predicted image and a fifth image block corresponding to the position of the third image block in the first image sample. The fourth image block is the masked reconstructed image block in the mask image, and the fifth image block is the high-definition image block in the first image sample at the same position as the fourth image block. In other words, the fifth image block can serve as the label for the fourth image block, and the cosine distance between the fourth and fifth image blocks can be used as the loss to update the parameters of the target model. It should be noted that, generally, if multiple fourth image blocks are included, there will be multiple corresponding losses. The parameters of the target model can be updated based on the mean or sum of the multiple losses.

[0248] For example, as shown in Figure 14, the masked portion of the clear image Fclean (the first image sample) and the defogging image Fout (the predicted image) are removed, and L loss is calculated only for the masked portion. This allows the target model to optimize defogging performance. This loss calculation method reduces the computational complexity of the loss function, thereby improving model training efficiency.

[0249] In some examples, image quality may also include removing noise from video frames, and may include enhancing the color of the video frames to enhance color saturation and improve the contrast of the video images to make image details clearer. Specifically, in this example, the processor may also be configured to, in response to the target operation, denoise the video frames and perform super-resolution processing on the denoised video frames so that the resolution of the second video images is higher than the resolution of the first video images; and / or, the processor may be configured to, in response to the target operation, denoise the video frames and perform color enhancement processing on the denoised video frames so that the color saturation of the second video images is higher than the color saturation of the first video images.

[0250] In this embodiment, the target operation may also instruct to denoise the video frame and perform super-resolution processing on the video frame to improve the resolution of the video frame; or the target operation may instruct to denoise the video frame and perform color enhancement processing on the video frame to improve the image contrast of the video frame. As described above, multiple menu keys may be displayed on the display screen, and the user can select which method to improve the image quality of the video frame by selecting the multiple menu keys.

[0251] The processor may be configured to denoise the video frame and perform color enhancement on the denoised video frame. This denoising and color enhancement process may be referred to as the color enhancement process described in the above example. When denoising the video frame, the processor may invoke a Ufomer model to remove flare. The Ufomer model may remove flare from the video frame.

[0252] When performing color enhancement, you can use related color enhancement techniques, such as using generative adversarial networks (GANs) for image enhancement. GANs consist of two neural networks: a generator and a discriminator. The generator is responsible for generating new images, while the discriminator is responsible for determining whether these images are realistic. By training these two networks, the generator can learn to produce sufficiently realistic images, making them suitable for color enhancement.

[0253] The processor may be configured to denoise the video frames and perform super-resolution on the denoised video frames. The denoising and super-resolution processes may be referred to as the image quality enhancement process described in the above example. When denoising the video frames, the processor may invoke a Ufomer model to remove flare. The Ufomer model may remove flare from the video frames.

[0254] Among them, when performing super-resolution processing, the super-resolution processing can be performed in two ways:

[0255] The first method: Use VSR++ as the basic model for video super-resolution.

[0256] The training process of the model can be as follows:

[0257] 1) The first-stage model is trained for 60k iterations using 10 LR frames as input, a batch size of 8, and a patch size of 256. The model is optimized using Charbonnier loss with the Adam optimizer and a cosine annealing scheme, with an initial learning rate of 1e-4.

[0258] 2) Fine-tune the model using the pre-trained weights from the model in stage (1). The batch size is 8, the patch size is 256 (HR), 10 LR frames are used as input, and the total number of iterations is 60k. The model is optimized using the MSE loss and cosine annealing scheme of the Adam optimizer, with an initial learning rate of 1e-5.

[0259] 3) Continue fine-tuning the model from stage (2) using MSE loss, using 10 LR frames as input, batch size 8 and patch size 512 (HR), the optimizer type remains unchanged, and the initial learning rate is 1e-6.

[0260] 4) Finally, the model from stage (3) is fine-tuned using 30 LR frames as input to include more information, with a patch size of 256 and a batch size of 8. This stage requires 60k iterations with a learning rate of 1e-6.

[0261] After the above model is trained, the processor can call the above model to perform super-resolution processing on the video frames.

[0262] Second way:

[0263] Referring to Figure 15, a schematic diagram of the process of super-resolution processing of video frames is shown. As shown in Figure 15, the processor can extract pixel data from the video frame, and then input the pixel data into the convolution layer for feature extraction, wherein the convolution layer is a Conv convolution for deep learning, k3f12 indicates that the convolution kernel is 3x3 and the number of channels is 12 channels, that is, the convolution kernel of the convolution layer in Figure 14 is 3x3, the input channel is 3 channels (R / G / B), and the output channel is 12 channels.

[0264] Next, the extracted features are split into channels. The purpose of channel splitting is to split the 12 channels into 4 parts (3 channels each) in order, namely 1-2-3 as the first part (f1-3), 4-5-6 as the second part (f4-6), 7-8-9 as the third part (f7-9), and 10-11-12 as the fourth part (f10-12).

[0265] Next, the data of each channel is mapped to the corresponding processing unit. When the 12 channels are split into 4 parts, they correspond to 4 processing units. As shown in Figure 14, 3D-LUT1 performs pixel mapping on f1-3, 3D-LUT2 performs pixel mapping on f4-6, 3D-LUT3 performs pixel mapping on f7-9, and 3D-LUT4 performs pixel mapping on f10-12. Pixel mapping can be understood as upsampling the input data, thereby increasing the pixel data to improve the resolution. Upsampling can be performed by linear interpolation, nonlinear interpolation, etc., which are not limited here.

[0266] In a further example, when performing super-resolution processing, the process of super-resolution processing is relatively cumbersome and time-consuming, while in the vehicle display, it is a real-time video stream display. As described in the above example, each video frame collected by the image acquisition device needs to be super-resolution processed before being displayed on the display screen. If the super-resolution processing time is too long, it is not conducive to the low-latency display of the video picture. Therefore, it is necessary to minimize the delay of the process of improving the picture quality. In the process of super-resolution processing, a pipeline processing method can be adopted to split the super-resolution processing process into multiple processing stages, each processing stage is configured as a unit in a thread queue, and multiple units are connected in series in sequence, so that the video frames to be super-resolution processed can be processed in multiple units in sequence. In this way, for the multiple video frames collected in real time by the image acquisition device, the pipeline processing method can reduce the delay between adjacent video frames, thereby ensuring low latency of the vehicle display.

[0267] During specific implementation, a thread queue can be configured in the processor, and the thread queue can include multiple units connected in series; the multiple units include a decoding unit, a data scheduling unit, a convolution unit, a super-resolution unit and a rendering unit; the processor is specifically configured to input the video frame into the thread queue so as to perform pipeline super-resolution processing on the video frame through multiple units.

[0268] Referring to Figure 16, a flow chart of executing the super-resolution processing shown in Figure 15 in a processor is shown. As shown in Figure 16, the multi-thread queue includes multiple units connected in series in sequence, and the connection order of the multiple units is the super-resolution processing order of the video frames.

[0269] Among them, the decoding unit (decoding module in Figure 15) can be used to decode the video frame to obtain pixel data; the data scheduling unit (data IO in Figure 15) can read the pixel data from the processor and write the pixel data to the GPU (graphics processing unit); the convolution unit (convolution layer & channel splitting in Figure 15) can convolve the pixel data, extract the features of the video frame, and perform channel splitting on the extracted features. The function of channel splitting is to split the 12 channels into 4 parts (3 channels each) in sequence, namely 1-2-3 as the first part (f1-3), 4-5-6 as the second part (f4-6), 7-8-9 as the third part (f7-9), and 10-11-12 as the fourth part (f10-12).

[0270] The super-resolution unit (3D-LUT in Figure 16) can be used to perform pixel mapping on the multiple data output by the convolution unit to improve the resolution of each data, such as by using linear interpolation or nonlinear interpolation. As shown in Figure 16, it includes four processing units: 3D-LUT1 performs pixel mapping on f1-3, 3D-LUT2 performs pixel mapping on f4-6, 3D-LUT3 performs pixel mapping on f7-9, and 3D-LUT4 performs pixel mapping on f10-12.

[0271] Among them, the rendering unit (GPU rendering in Figure 16) is used to render the video frames. Of course, in some examples, the video frames also need to be color enhanced. In this case, the rendering unit can also perform image fusion on the color-enhanced video frames and the super-resolution video frames. This image fusion can be understood as the post-processing in the above example, so that the fused image is displayed on the display screen.

[0272] In this example, the convolution unit, the super-resolution unit, and the rendering unit may be located in the GPU, and the decoding unit may be located in the processor, for calling the processor to complete decoding in resources.

[0273] It should be noted that the functions of the above-mentioned units are independent of each other, but the data is interdependent. For example, the input data of the scheduling unit is the data output by the previous decoding unit, and the data output by the scheduling unit is the input data of the next convolutional layer unit. In actual applications, when the GPU is rendering the 10th frame of image data, the super-resolution unit (3D-LUT) is processing the 11th frame of image data, the convolution unit is processing the 12th frame of image data, the scheduling unit is processing the 13th frame of image data, and the decoding unit is decoding the 14th frame of image data. Therefore, the total time is the longest part of these five units, rather than the sum of these five parts. This can reduce the interval between the second video screen and thus reduce the latency.

[0274] As described above, in an in-vehicle display, the position and / or field of view of the image acquisition device can be changed based on the target operation performed by the user on the first video screen in the display screen, and the picture quality of the video screen subsequently acquired by the image acquisition device can be improved. In some cases, the user can also restore the video screen displayed on the display screen to the initial setting. Specifically, the display screen can also be configured to display an undo control; wherein, the processor is further configured to display the video frame currently acquired in real time by the image acquisition device in accordance with the display parameters of the first video screen in response to the triggering operation of the undo control, so that the display screen switches from the second video screen back to the fourth video screen that is consistent with the display parameters of the first video screen; wherein the display parameters include at least one of the screen size, field of view size, shooting angle and picture quality.

[0275] In this example, the undo control can be displayed in the control area of ​​the display screen, or it can be displayed on the second video screen, wherein, as shown in Figure 17, a schematic diagram of the undo control displayed on the video screen is shown, wherein the shape of the undo control can be not restricted. Specifically, when the processor processes the video frame into the second video screen in response to the target operation, the undo control can be displayed on the second video screen to indicate that the user can directly operate the undo control on the second video screen to restore the display settings before the target operation. Among them, the undo control can be located at the end corners of the second video screen, such as the upper left corner, the lower left corner, the upper right corner and the lower right corner, so as not to affect the viewing of the second video screen. There is no restriction on the display position of the undo control.

[0276] Among them, after the target operation, the display of the undo control can be kept, or the undo control can be displayed on the display screen when the triggering condition of the undo control is met. When the processor detects that the undo control is triggered, if the image acquisition device is adjusted according to the target operation, the image acquisition device can be restored to the posture and focal length before the adjustment. If the acquired video frame is processed according to the target operation, the processing thread of the target operation on the video frame can be stopped, so that the video screen displayed after the undo control is triggered can be restored to the display properties consistent with the first video screen. If both the image acquisition device and the video frame are processed, the image acquisition device can be restored to the posture and focal length before the adjustment, and the video screen can be restored to the display properties consistent with the first video screen.

[0277] In which, whether the image acquisition device is restored or the captured video frame is restored, it can be seen on the display screen that the video screen switches from the second video screen to a fourth video screen having display parameters consistent with the first video screen, and at least one of the screen size, the field of view size, the shooting angle, and the picture quality of the fourth video screen can be consistent with the first video screen. For example, if the target operation causes the posture and focal length of the image acquisition device to be adjusted, the processor can adjust the posture and focal length of the image acquisition device back to the state before the target operation in response to the triggering of the undo control, thereby making the fourth video screen that is subsequently captured and displayed have the same shooting angle and field of view size as the first video screen.

[0278] For example, if the target operation causes the picture size and picture quality of the video frame captured by the image acquisition device to change, the processor can stop processing the subsequently captured video frames in response to the triggering of the undo control, thereby making the fourth video picture that is subsequently captured and displayed have the same picture size and picture quality as the first video picture.

[0279] Among them, when the target operation causes the picture quality of the video frame captured by the image acquisition device to change, if the second video screen is displayed at a second display position different from the first video screen, the processor can stop displaying the second video screen at the second display position in response to the triggering of the undo control, while the display screen at the first display position can remain unchanged. In this way, the user can see that after the undo control is triggered, the second display position no longer displays the high-definition second video screen.

[0280] By setting this key, users can quickly return to the original monitoring state after performing complex operations.

[0281] The display device of the present disclosure is described below with reference to examples.

[0282] Referring to Figures 18-21, Figure 18 shows a schematic diagram of the split-screen interface of the display screen, Figure 19 shows a schematic diagram of the interface of the display screen, Figure 20 shows a schematic diagram of the interface when a split-screen of the display screen displays a second video screen, and Figure 21 shows a schematic diagram of the interface when another split-screen displays a second video screen. As shown in Figures 18-21:

[0283] The display device includes a display screen and a processor, the processor being connected to the display screen and configured to control the content displayed on the display screen. The display device is connected to multiple image acquisition devices, configured to display video images captured by the multiple image acquisition devices in real time on a split-screen basis. Specifically, the multiple image acquisition devices may include a first image acquisition device located at the left and right rearview mirrors of the vehicle, a second image acquisition device located at the front windshield of the vehicle, and a third image acquisition device located at the rear windshield of the vehicle. As shown in FIG18 , the video images captured by the first image acquisition device are displayed in the areas to the left and right of the first row, while the video images captured by the second and third image acquisition devices are displayed in the areas to the left and right of the second row, respectively.

[0284] As shown in FIG18 , the display screen may further include a main screen. The main screen and the split screen may be separated. As shown in FIG18 , the main screen is located to the right of the split screen. The main screen may display vehicle multimedia content, such as music videos, or a subsequent high-definition secondary video. The display screen may also include a control area, which may display multiple menu keys. As shown in FIG18 , the control area is located to the left of the split screen, and the main screen area is larger than the split screen area.

[0285] As shown in Figure 19, when the user triggers the settings menu in the control area, a submenu pops up. The submenu includes multiple function menu keys on the vehicle, such as Bluetooth, Media, Display, etc. Bluetooth can set the Bluetooth connection between the vehicle and the mobile device, Media can be used to select multimedia content to be played on the main screen, and Display can set the video image displayed on the display. This example focuses on the display. As shown in Figure 19, when the user triggers the display menu key, a submenu pops up. The submenu includes the main screen, left and right rearview mirrors, etc., that is, the submenu can provide display-related components, such as the image acquisition device and the display screen.

[0286] As shown in Figure 19, when the user triggers the left and right rearview mirrors, a menu key related to the first image acquisition device of the left and right rearview mirrors pops up. The menu key can provide multiple first menu keys related to picture quality, such as brightness and style, and can also provide multiple second menu keys related to picture field of view, size and shooting angle, such as picture range.

[0287] When the user triggers the screen range, as shown in Figure 20, the first frame is displayed on the first video screen on both sides of the left rear of the first row of the display screen. In this case, the processor can adjust the focal length of the first image acquisition device in response to the zoom operation on the first frame. Specifically, the user performs a zoom operation on the first frame on the left rearview mirror, and the focal length of the first image acquisition device on the left side of the vehicle is lengthened. The lengthening ratio can be determined according to the zoom ratio before and after the first frame is enlarged.

[0288] Next, the user touches the bottom edge of the first frame on the left rearview mirror and slides downward, the first frame is stretched, and the processor twists the first image acquisition device toward the ground so that the shooting angle is downward. As a result, the field of view of the second video picture displayed on the split screen on the left side of the first row of the display screen is smaller than that of the first video picture, and the displayed second video picture is more biased towards the perspective below the vehicle than the first video picture.

[0289] When the user triggers the first border on the left side of the first row, the focal length and posture of the first image acquisition device on the right side may be adjusted synchronously, or may not be adjusted.

[0290] Among them, such as the interior rearview mirror and the main screen, the interior rearview mirror and other split screens generally require low latency and can only perform simple image processing. However, when the user has more needs, such as observing a unit in the distance in front of the vehicle, a unit in the distance behind the vehicle, the left front side of the vehicle, the right front side of the vehicle, etc., blurred units will appear. At this time, image detection and identification of blurred targets and high-definition processing are required to determine what they are, such as a black plastic bag, a steel bar raised on the highway, and other objects. Timely identification helps the vehicle take necessary measures as soon as possible. You can operate it by pressing the menu key. At this time, the split screen still displays with low latency, and the main screen performs image processing, target analysis and other functions to display the highly processed second video screen of the split screen.

[0291] As shown in FIG21 , if the user triggers the brightness menu key and the style menu key to increase the brightness and clarity of the video image of the left rearview mirror, the processor sends the video frame captured by the first image capture device at the left rearview mirror to the target image processing model for brightness processing and the target image processing model for clarity processing, and uses these two image processing models to increase the brightness and clarity of the video frame; at the same time, the processor calculates the adjustment time required for the processing process;

[0292] If the adjustment time is calculated to be greater than the preset delay time corresponding to the first image acquisition device, the processed second video image will be displayed on the main screen, while the video image captured by the first image acquisition device will continue to be displayed on the split screen on the left side of the first row. In this way, although the second video image on the main screen has a longer delay time, the user can still obtain real-time images outside the vehicle through the video image with a shorter delay time on the split screen on the left side of the first row. Although the second video image on the main screen has a longer delay time, the image is high-definition, which can help the user identify surrounding obstacles.

[0293] Similarly, the video frames captured by the first image capture device of the right rearview mirror can also be processed in the same manner. Of course, if the video frames captured by the first image capture devices of the left and right rearview mirrors need to have their image quality improved and need to be displayed on other screens at the same time, in this case, the processed second video frames of the left and right rearview mirrors can be displayed on the main screen in split screens.

[0294] Among them, the process of how to adjust the split screen size through the second border is not described in detail in this example, and the details can be referred to the description of the above embodiment.

[0295] Based on the same inventive concept, the present disclosure further provides a display method, which can be applied to an in-vehicle display. Specifically, it can be performed by a display device configured in a vehicle. As shown in FIG22 , a schematic flow chart of the steps of the display method is shown. As shown in FIG22 , the display method can specifically include the following steps:

[0296] Step S301: displaying the video images captured in real time by the image acquisition device;

[0297] Step S302: In response to a target operation performed on the first video frame currently displayed on the display screen, the image acquisition device and / or the video frame to be displayed is adjusted so that there is at least one of the following differences between the second video frame displayed after adjustment and the first video frame: picture size, picture field size, shooting angle corresponding to the picture, picture quality and display position.

[0298] The process of the embodiment of the method can refer to the description of the embodiment of the display device mentioned above, and will not be repeated here.

[0299] This display method allows adjustments to be made to the image capture device in response to a target operation on a first video frame being displayed in real time, thereby adjusting at least one of the field of view of the frame and the corresponding shooting angle. Furthermore, adjustments can be made to the video frames captured by the image capture device, thereby adjusting at least one of the frame size, quality, and display position. This allows a user to switch the video frame displayed on the display screen through direct operation of the display screen, so that the switched second video frame meets the user's viewing needs, thereby enhancing the flexibility of in-vehicle display-assisted safe driving.

[0300] In some examples, an adjustment icon may also be displayed on the display screen; when the image acquisition device and / or the video frame to be displayed is adjusted in response to a target operation performed on the first video screen currently displayed on the display screen, the posture and / or focal length of the image acquisition device may be adjusted in response to the target operation performed on the adjustment icon so that there is a difference in at least one of the field of view size and the shooting angle between the first video screen and the second video screen.

[0301] In some examples, the adjustment icon includes a first border, and the target operation includes a scaling operation on the first border and a size change operation on at least one side; wherein, in response to the target operation on the adjustment icon, the posture and / or focal length of the image acquisition device are adjusted, which can be: in response to the size change operation, the posture of the image acquisition device is adjusted so that there is a change in shooting angle between the first video screen and the second video screen; and, in response to the scaling operation, the focal length of the image acquisition device is adjusted so that there is a change in field of view size between the first video screen and the second video screen.

[0302] In some examples, the step of adjusting the posture of the image acquisition device in response to the size change operation may include: obtaining size parameters of the adjusted first border based on the type of the image acquisition device; substituting the size parameters into a preset function corresponding to the image acquisition device to obtain a rotation angle of the image acquisition device; and rotating the image acquisition device according to the rotation angle.

[0303] In some examples, when the type of the image acquisition device is the first type, the first border is a rectangular frame, the size parameter includes the length of each side of the first border, and the preset function includes a first preset function, and the first preset function is used to represent the correspondence between the length of the side of the first border and the rotation angle;

[0304] When the type of the image acquisition device is the second type, the first border is a trapezoidal frame, the size parameters include the angles of each inner corner of the first border, and the preset function includes a second preset function, which is used to characterize the correspondence between the difference between the inner angles of the first border and the rotation angle.

[0305] In some examples, video images captured by multiple different image acquisition devices can be displayed in split screens; if the first border exceeds the split screen where the first video image is located after the size change operation, and occupies a portion of at least one first split screen adjacent to the split screen; then the size of the split screen and the first split screen are kept unchanged before and after the target operation.

[0306] In some examples, video images captured by multiple different image capture devices may be displayed in split screens, and the adjustment icon includes a second border. The method may further include: upon detecting that the second border, after adjustment, exceeds the split screen where the first video image is located and occupies a portion of at least one second split screen adjacent to the first split screen, synchronously adjusting the size of the split screen and the size of the second split screen so that the adjusted second border is entirely located in the adjusted split screen;

[0307] And, the screen size of the video frame to be displayed is adjusted so that the video screen of the image acquisition device fills the adjusted split screen.

[0308] In some examples, when the target operation indicates adjusting the picture quality, the second video picture may be displayed at a first display position where the first video picture is located, or the second video picture may be displayed at a second display position different from the first display position;

[0309] The duration between the display time of the second video image at the second display position and the target operation is greater than the duration between the display time of the second video image at the first display position and the target operation.

[0310] In some examples, while the second video image is displayed at the second display position, a third video image captured by the image capture device can be displayed in real time at the first display position;

[0311] The interval between the video frames displayed in the first display position is shorter than the preset delay time.

[0312] In some examples, when the target operation indicates adjusting the picture quality, the adjustment time for adjusting the picture quality of the video frame can be determined, and based on the adjustment time, the second video picture can be scheduled to the first display position or the second display position.

[0313] In some examples, the process of determining the adjustment time may be as follows:

[0314] Determining multiple processes to be performed based on a first picture quality of the video frame and a second picture quality indicated by the target operation; and

[0315] Compensating for a preset adjustment time corresponding to each of the processing steps based on the data volume of the video frame and / or the current performance parameters of the processor;

[0316] The adjustment time is determined based on the compensated preset adjustment time.

[0317] In some examples, the process of determining the multiple processing to be performed based on the first picture quality of the video frame and the second picture quality indicated by the first preset operation may include the following steps:

[0318] Determining an environment type corresponding to the video frame, where the environment type is used to characterize the type of interference factors that affect picture quality in the environment where the image acquisition device is located;

[0319] determining a plurality of the processing based on a difference between the environment type and a preset environment type, and the second picture quality;

[0320] The preset environment type indicates that there are no interference factors that affect the picture quality.

[0321] In some examples, the display screen is further configured to display a plurality of menu keys; wherein, in response to a triggering operation on at least one of the plurality of menu keys, the picture quality of the video frame can be adjusted so that the picture quality of the second video picture is higher than the picture quality of the first video picture.

[0322] In some examples, when the picture quality of the video frame is adjusted, the environment type corresponding to the video frame can be determined, and based on the environment type, multiple processing operations to be performed on the video frame can be determined, and the video frame can be scheduled to the target image processing model corresponding to each processing in turn to improve the picture quality of the video frame through the target image processing model.

[0323] In some examples, the picture quality includes picture clarity; the video frame may be divided into a plurality of image blocks, and mask reconstruction may be performed on the contents of some of the image blocks, so that the second video picture contains no or minimal interference factors that affect the picture quality.

[0324] The interference factors include at least one of dim light, dust, thick fog, and rain and snow.

[0325] In some examples, the step of dividing the video frame into a plurality of image blocks and performing mask reconstruction on contents of some of the image blocks may include:

[0326] Dividing the video frame to obtain a plurality of first image blocks;

[0327] Dividing the first image block into at least one second image block according to the definition corresponding to each of the first image blocks; wherein the first image block with lower definition is divided into a smaller number of second image blocks;

[0328] masking a portion of the second image block;

[0329] Based on the unmasked second image block, the content of the masked second image block is reconstructed.

[0330] In some examples, during mask reconstruction, a target model may be called, and the target model is used to perform the steps of dividing the video frame into a plurality of image blocks and performing mask reconstruction on contents of some of the image blocks;

[0331] The target model is generated using a first image sample of a first type and a second image sample of a second type of the same image as training samples, and the clarity of the first image sample is higher than that of the second image sample.

[0332] In some examples, the target model is used during training to generate a mask image based on the first image sample and the second image sample, and to reconstruct the mask image to obtain a reconstructed predicted image; wherein the masked third image blocks in the mask image are from the second image sample, and the unmasked image blocks are from the first image sample;

[0333] Furthermore, during the training of the target model, the parameters of the target model are updated based on the difference between the fourth image block corresponding to the position of the third image block in the predicted image and the fifth image block corresponding to the position of the third image block in the first image sample.

[0334] In some examples, in response to the target operation, the video frame may be denoised, and the denoised video frame may be super-resolution processed so that the resolution of the second video picture is higher than the resolution of the first video picture; and / or,

[0335] In response to the target operation, the video frame may be denoised, and the denoised video frame may be color enhanced so that the color saturation of the second video picture is higher than the color saturation of the first video picture.

[0336] In some examples, the video frame can be input into a thread queue so as to perform pipeline super-resolution processing on the video frame through multiple units; the thread queue includes multiple units connected in series; the multiple units include a decoding unit, a data scheduling unit, a convolution unit, a super-resolution unit and a rendering unit.

[0337] In some examples, an undo control may also be displayed on the display;

[0338] In response to a triggering operation of the cancel control, the video frame currently captured in real time by the image capture device can be displayed according to the display parameters of the first video screen, so that the display screen switches from the second video screen to a fourth video screen having the same display parameters as the first video screen;

[0339] The display parameters include at least one of the screen size, the field of view, the shooting angle, and the picture quality.

[0340] Based on the same inventive concept, a vehicle-mounted display system is also provided, comprising a plurality of image acquisition devices and the display device, wherein the plurality of image acquisition devices are respectively connected to the display device, and the display device is configured to display the video images acquired in real time by the plurality of image acquisition devices in a split-screen manner.

[0341] Based on the same inventive concept, an electronic device is also provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to support display by the display device.

[0342] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0343] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity, or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, commodity, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, commodity, or device that includes the element.

[0344] The above is a detailed introduction to a display method, device, system and medium provided by the present disclosure. Specific examples are used herein to illustrate the principles and implementation methods of the present disclosure. The description of the above embodiments is only used to help understand the method and core ideas of the present disclosure. At the same time, for those skilled in the art, according to the ideas of the present disclosure, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present disclosure.

[0345] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0346] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0347] References herein to "one embodiment," "an embodiment," or "one or more embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. Furthermore, please note that instances of the phrase "in one embodiment" do not necessarily all refer to the same embodiment.

[0348] In the description provided herein, numerous specific details are described. However, it is understood that embodiments of the present disclosure may be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0349] In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in the claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present disclosure may be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0350] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present disclosure.

Claims

1. A display device, applied to in-vehicle display, comprising: A display screen configured to display a video picture captured in real time by an image capturing device; A processor configured to, in response to a target operation performed on a first video picture currently displayed on the display screen, adjust the image capturing device and / or a video frame to be displayed, so that there is at least one of the following differences between a second video picture displayed after adjustment and the first video picture: picture size, field of view size of the picture, shooting angle corresponding to the picture, picture quality, and display position.

2. The display device according to claim 1, wherein The display screen is further configured to display an adjustment icon; The processor is further configured to, in response to a target operation performed on the adjustment icon, adjust the pose and / or focal length of the image capturing device, so that there is a difference in at least one of the field of view size and the shooting angle between the first video picture and the second video picture.

3. The display device according to claim 2, wherein, The adjustment icon includes a first border, and the target operation includes a scaling operation on the first border and an operation of changing the size of at least one side; wherein, The processor is specifically configured to, in response to the operation of changing the size, adjust the pose of the image capturing device, so that there is a change in the shooting angle between the first video picture and the second video picture; and, In response to the scaling operation, adjust the focal length of the image capturing device, so that there is a change in the field of view size between the first video picture and the second video picture.

4. The display device according to claim 3, wherein, The adjusting the pose of the image capturing device in response to the operation of changing the size includes: Based on the type of the image capturing device, obtaining size parameters of the adjusted first border; Substituting the size parameters into a preset function corresponding to the image capturing device to obtain a rotation angle of the image capturing device; Rotating the image capturing device according to the rotation angle.

5. The display device according to claim 4, wherein, When the type of the image capturing device is the first type, the first border is a rectangular frame, the size parameters include the side lengths of each side of the first border, and the preset function includes a first preset function, and the first preset function is used to represent the corresponding relationship between the side length of the first border and the rotation angle; When the type of the image capturing device is the second type, the first border is a trapezoidal frame, the size parameters include the angles of each interior angle of the first border, and the preset function includes a second preset function, and the second preset function is used to represent the corresponding relationship between the difference between the interior angles of the first border and the rotation angle.

6. The display device according to claim 3, wherein, The display screen is configured to display video pictures captured by a plurality of different image capturing devices in a split screen manner; The first border exceeds the split screen where the first video picture is located after the operation of changing the size, and occupies a part of at least one first split screen adjacent to the split screen; Wherein, before and after the target operation, the processor is configured to keep the sizes of the split screen and the first split screen unchanged.

7. The display device according to claim 2, wherein, The display screen is configured to display video pictures captured by a plurality of different image capturing devices in a split screen manner, and the adjustment icon includes a second border; The processor is further configured to: when it is detected that the second border exceeds the split screen where the first video picture is located after being adjusted and occupies a part of at least one second split screen adjacent to the first split screen, synchronously adjust the size of the split screen and the size of the second split screen so that the adjusted second border is entirely located within the adjusted split screen; and, adjust the picture size of the video frame to be displayed so that the video picture of the image acquisition device fills the adjusted split screen.

8. The display device according to any one of claims 1-7, wherein, The processor is further configured to, when the target operation instructs to adjust the picture quality, display the second video picture at the first display position where the first video picture is located, or display the second video picture at a second display position different from the first display position; wherein, the time interval between the display time of the second video picture at the second display position and the target operation is greater than the time interval between the display time of the second video picture at the first display position and the target operation.

9. The display device according to claim 8, wherein, The processor is further configured to, when displaying the second video picture at the second display position, display the third video picture acquired by the image acquisition device in real time at the first display position; wherein, the time interval between the video pictures displayed at the first display position is less than a preset delay time.

10. The display device according to claim 8, wherein, The processor is specifically configured to, when the target operation instructs to adjust the picture quality, determine the adjustment time taken to adjust the picture quality of the video frame, and based on the adjustment time taken, schedule the second video picture to the first display position or the second display position.

11. The display device according to claim 10, wherein, The processor is specifically configured to: based on the first picture quality of the video frame and the second picture quality indicated by the target operation, determine multiple processes to be executed; and, compensate the preset adjustment time taken corresponding to each of the processes based on the data volume of the video frame and / or the current performance parameters of the processor; determine the adjustment time taken based on the compensated preset adjustment time taken.

12. The display device according to claim 11, wherein, The determining multiple processes to be executed based on the first picture quality of the video frame and the second picture quality indicated by the first preset operation instruction includes: determining the environmental type corresponding to the video frame, where the environmental type is used to characterize the type of interference factors that affect the picture quality in the environment where the image acquisition device is located; determining multiple processes based on the difference between the environmental type and the preset environmental type, and the second picture quality; wherein, the preset environmental type characterizes the absence of interference factors that affect the picture quality.

13. The display device according to any one of claims 1-7, wherein, The display screen is further configured to display a plurality of menu keys; wherein, the processor is further configured to, in response to a trigger operation on at least one of the plurality of menu keys, adjust the picture quality of the video frame so that the picture quality of the second video picture is higher than the picture quality of the first video picture.

14. The display device according to claim 1, wherein, The processor is further configured to determine the environmental type corresponding to the video frame when adjusting the picture quality of the video frame, and based on the environmental type, determine various processes to be performed on the video frame, and sequentially schedule the video frame to target image processing models corresponding to each process, so as to improve the picture quality of the video frame through the target image processing models.

15. The display device according to any one of claims 1-7 and 14, wherein, The picture quality includes picture clarity; The processor is configured to divide the video frame into multiple image blocks, and perform mask reconstruction on the content of some of the image blocks, so that the second video picture does not contain or contains fewer interference factors that affect the picture quality: Wherein, the interference factors include at least one of low light factors, dust factors, thick fog factors, and rain and snow factors.

16. The display device according to claim 15, wherein, The dividing the video frame into multiple image blocks and performing mask reconstruction on the content of some of the image blocks includes: Dividing the video frame to obtain multiple first image blocks; Dividing each first image block into at least one second image block according to the clarity corresponding to each first image block; wherein, the first image block with smaller clarity is divided into a smaller number of second image blocks; Masking some of the second image blocks; Based on the unmasked second image blocks, reconstruct the content of the masked second image blocks.

17. The display device according to claim 16, wherein, A target model is configured in the processor, and when the target model is called by the processor, it is used to perform the steps of dividing the video frame into multiple image blocks and performing mask reconstruction on the content of some of the image blocks; Wherein, the target model is generated using first image samples of a first type and second image samples of a second type of the same image, and the clarity of the first image samples is higher than that of the second image samples.

18. The display device according to claim 17, wherein, During the training process of the target model, it is used to generate a mask image based on the first image samples and the second image samples, and reconstruct the mask image to obtain a reconstructed predicted image; wherein, the masked third image blocks in the mask image come from the second image samples, and the unmasked image blocks come from the first image samples; And, during the training process of the target model, update the parameters of the target model based on the difference between the fourth image block corresponding to the position of the third image block in the predicted image and the fifth image block corresponding to the position of the third image block in the first image sample.

19. The display device according to any one of claims 1-7 and any one of 14, wherein, The processor is configured to denoise the video frame in response to the target operation, and perform super-resolution processing on the denoised video frame, so that the resolution of the second video picture is higher than that of the first video picture; and / or The processor is configured to denoise the video frame in response to the target operation, and perform color enhancement processing on the denoised video frame, so that the color saturation of the second video picture is higher than that of the first video picture.

20. The display device according to claim 19, wherein, The processor is configured with a thread queue, and the thread queue includes a plurality of units connected in series in sequence; the plurality of units include a decoding unit, a data scheduling unit, a convolution unit, a super-resolution unit, and a rendering unit; The processor is specifically configured to input the video frame into the thread queue to perform pipelined super-resolution processing on the video frame through the plurality of units.

21. The display device according to claim 1, wherein, The display screen is further configured to display a cancel control; The processor is further configured to, in response to a trigger operation of the cancel control, display the video frame currently and real-time collected by the image acquisition device according to the display parameters of the first video screen, so that the display screen switches from the second video screen back to a fourth video screen consistent with the display parameters of the first video screen; Wherein, the display parameters include at least one of the screen size, the field of view size, the shooting angle, and the picture quality.

22. A display method, applied to in-vehicle display, the method includes: Display a video screen real-time collected by an image acquisition device; In response to a target operation on a currently displayed first video screen in the display screen, adjust the image acquisition device and / or the video frame to be displayed, so that there is at least one of the following differences between the adjusted second video screen and the first video screen: screen size, field of view size of the screen, shooting angle corresponding to the screen, picture quality, and display position.

23. A vehicle-mounted display system, comprising a plurality of image acquisition devices, and the display device according to any one of claims 1-21, wherein, The plurality of image acquisition devices are respectively connected to the display device, and the display device is configured to display the video screens real-time collected by the plurality of image acquisition devices in a split screen manner.

24. An electronic device, comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor is used to support the display of the display device according to any one of claims 1-21.

Images