Optimization method and device for image delay, terminal equipment and computer medium

Abstract

The application discloses an image delay optimization method and device, terminal equipment and computer medium, relates to the technical field of image processing, and is applied to a vehicle provided with an image acquisition device. The application obtains initial image data through the image acquisition device, determines a delay duration corresponding to the initial image data, and determines a speed vector corresponding to an initial pixel point in the initial image data; determines a frame number compensation formula according to the delay duration and the initial pixel point, and generates target image data according to the frame number compensation formula and the speed vector; and performs an interpolation operation based on the target image data to complete optimization of the delay duration. The application can realize the technical effect of optimizing the delay duration generated when the vehicle-mounted AR technology transmits image data, thereby reducing the discomfort and delay feeling of a driver when the driver obtains image data through AR glasses.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, terminal device, and computer-readable storage medium for optimizing image latency. Background Technology

[0002] As the application scenarios of AR (Augmented Reality) technology become increasingly diverse, in-vehicle AR technology has emerged. In-vehicle AR technology uses radar and cameras deployed outside the vehicle to capture real-time scene image data. This data is then processed through planar detection and image fusion, and transmitted to AR glasses worn by the driver. The driver can then access navigation or road information contained within the image data through the AR glasses, significantly improving the driving experience.

[0003] However, in the process of processing scene image data, in-vehicle AR technology needs to transmit the acquired scene image data multiple times to generate image data with information. As a result, there will inevitably be a certain delay between the scene image data finally acquired by the driver and the scene image data captured by the camera during the transmission of image data. How to optimize this delay has become a technical problem that the industry urgently needs to solve. Summary of the Invention

[0004] The main objective of this application is to provide a method, apparatus, terminal device, and computer-readable storage medium for optimizing image latency, with the aim of optimizing the latency of in-vehicle AR technology when transmitting image data.

[0005] To achieve the above objectives, this application provides a method for optimizing image latency. The method is applied to a vehicle equipped with an image acquisition device, and includes the following steps:

[0006] The image acquisition device acquires initial image data, determines the delay time corresponding to the initial image data, and determines the velocity vector corresponding to the initial pixel in the initial image data.

[0007] The frame rate compensation formula is determined based on the delay duration and the initial pixel points, and the target image data is generated based on the frame rate compensation formula and the velocity vector.

[0008] Frame interpolation is performed based on the target image data to optimize the delay duration.

[0009] Further, the step of determining the velocity vector corresponding to the initial pixel in the initial image data includes:

[0010] Determine the first image block contained within the initial image data;

[0011] Determine the second image block corresponding to the first image block in the first reference image data and / or the second reference image data corresponding to the initial image data;

[0012] The velocity vector corresponding to the initial pixel in the initial image data is determined based on the first image block and the second image block.

[0013] Further, the step of determining the second image block corresponding to the first image block in the first reference image data and / or the second reference image data corresponding to the initial image data includes:

[0014] Detect the data type corresponding to the initial image data, wherein the data type includes simple image data and complex image data;

[0015] If the data type is detected to be the simple image data, then a second image block corresponding to the first image block is determined in the first reference image data using a preset first motion estimation method;

[0016] If the data type is detected to be complex image data, then the second image block contained in the first reference image data and the second reference image data is determined by a preset second motion estimation method.

[0017] Further, the step of determining the second image block corresponding to the first image block in the first reference image data using a preset first motion estimation method includes:

[0018] Determine the minimum mean square error function corresponding to the initial pixel points within the first image block;

[0019] Based on the minimum mean square error function, a candidate motion estimation vector corresponding to the initial pixel is determined, and based on the candidate motion estimation vector, a second image block corresponding to the first image block is determined in the first reference image data.

[0020] Further, the step of determining the second image patch contained in the first reference image data and the second reference image data respectively by using a preset second motion estimation method includes:

[0021] A first minimum absolute error function is determined between the initial pixel and the first reference image data, and a first candidate motion estimation vector corresponding to the initial pixel is determined based on the first minimum absolute error function;

[0022] A second minimum absolute error function is determined between the initial pixel and the second reference image data, and a second candidate motion estimation vector corresponding to the initial pixel is determined based on the second minimum absolute error function;

[0023] The second image block contained in the first reference image data and the second reference image data is determined based on the first candidate motion estimation vector and the second candidate motion estimation vector.

[0024] Further, the initial pixel is located within the first image block, and the step of determining the velocity vector corresponding to the initial pixel in the initial image data based on the first image block and the second image block includes:

[0025] The partial derivatives are calculated for the initial pixels.

[0026] The target matrix is ​​determined by the partial derivatives, and the target pixel corresponding to the initial pixel in the second image block is determined based on the target matrix.

[0027] The velocity vector corresponding to the initial pixel is determined based on the initial pixel and the target pixel.

[0028] Further, the step of determining the frame number compensation formula based on the delay duration and the initial pixel points includes:

[0029] Determine the pixel expression corresponding to the initial pixel point;

[0030] The pixel expression is processed based on the delay duration to generate a frame rate compensation formula.

[0031] Furthermore, to achieve the above objectives, this application also provides an image delay optimization apparatus, which is applied to a vehicle equipped with an image acquisition device, the apparatus comprising:

[0032] The parameter determination module is used to acquire initial image data through the image acquisition device, and determine the delay duration corresponding to the initial image data and the velocity vector corresponding to the initial pixel in the initial image data;

[0033] An image generation module is used to determine a frame number compensation formula based on the delay duration and the initial pixel points, and to generate target image data based on the frame number compensation formula and the velocity vector.

[0034] The delay optimization module is used to perform frame interpolation operations based on the target image data to optimize the delay duration.

[0035] In addition, to achieve the above objectives, this application also provides a terminal device, the terminal device comprising: a memory, a processor, and an image latency optimization program stored in the memory and executable on the processor, wherein the image latency optimization program, when executed by the processor, implements the steps of the image latency optimization method described above.

[0036] In addition, to achieve the above objectives, this application also provides a computer-readable storage medium storing an image delay optimization program, which, when executed by a processor, implements the steps of the image delay optimization method described above.

[0037] The image latency optimization method, apparatus, terminal device, and computer medium provided in this application are applied to a vehicle equipped with an image acquisition device. The method involves acquiring initial image data through the image acquisition device, determining the latency duration corresponding to the initial image data, and determining the velocity vector corresponding to the initial pixels in the initial image data. A frame rate compensation formula is determined based on the latency duration and the initial pixels, and target image data is generated based on the frame rate compensation formula and the velocity vector. Frame interpolation is then performed based on the target image data to optimize the latency duration.

[0038] In this embodiment, when the terminal device is running, it first captures images of the environment around the vehicle using an image acquisition device deployed outside the vehicle to obtain initial image data. The acquired initial image data is then input into the image processing module configured within the terminal device. The image processing module reads the storage device configured within the terminal device to obtain the delay duration preset by the technician corresponding to the initial image data. Based on the delay duration, it determines the velocity vector corresponding to the initial pixel in the initial image data. Then, based on the delay duration and the initial pixel in the initial image data, the image processing module determines the frame number compensation formula corresponding to the initial pixel. The image processing module then calculates the velocity vector corresponding to the initial pixel according to the obtained frame number compensation formula to generate target image data. Finally, the image processing module determines the frame interpolation position based on the delay duration and inserts the target image data into the frame interpolation position to complete the optimization of the delay duration.

[0039] Thus, this application determines the motion pattern of the initial pixels contained in the initial image data based on the delay between when the image data is acquired and when it is output to the AR glasses. Then, it compensates the initial image data based on the motion pattern to generate target image data. By interpolating the initial image data with the target image data, this application achieves the technical effect of optimizing the delay in the transmission of image data by in-vehicle AR technology. This reduces the discomfort and delay experienced by the driver when acquiring image data through AR glasses, and improves the driver's user experience. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the structure of the terminal device in the hardware operating environment involved in the embodiments of this application;

[0041] Figure 2 This is a flowchart illustrating the first embodiment of the image delay optimization method of this application;

[0042] Figure 3 This is a schematic diagram of a block matching unidirectional motion estimation method involved in an embodiment of the image delay optimization method of this application;

[0043] Figure 4 This is a schematic diagram of a block matching bidirectional motion estimation method involved in one embodiment of the image delay optimization method of this application;

[0044] Figure 5 This is a schematic diagram of the functional modules involved in an embodiment of the image delay optimization method of this application.

[0045] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0046] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0047] Reference Figure 1 , Figure 1 This is a schematic diagram of the terminal device structure of the hardware operating environment involved in the embodiments of this application.

[0048] It should be noted that, Figure 1 This can be a schematic diagram of the hardware operating environment of the terminal device. In this embodiment, the terminal device can be a device that executes the image latency optimization method of this application. Specifically, the terminal device can be a mobile terminal, a data storage control terminal, a PC, or a portable computer, etc.

[0049] like Figure 1As shown, the terminal device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0050] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the terminal device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0051] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and an image latency optimization program.

[0052] exist Figure 1 In the terminal device shown, the network interface 1004 is mainly used for data communication with other devices; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the terminal device of this application can be set in the terminal device. The terminal device calls the image delay optimization program stored in the memory 1005 through the processor 1001 and executes the image delay optimization method provided in the embodiment of this application.

[0053] Based on the aforementioned terminal device, various embodiments of the image delay optimization method of this application are provided.

[0054] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the image delay optimization method of this application.

[0055] It should be understood that although the logical order is shown in the flowchart, in some cases, the optimization method for image delay in this application may of course be performed in a different order than that shown or described here.

[0056] In this embodiment, the image delay optimization method of this application is applied to a vehicle equipped with an image acquisition device, and may include the following steps:

[0057] Step S10: Acquire initial image data through the image acquisition device, and determine the delay duration corresponding to the initial image data and the velocity vector corresponding to the initial pixel in the initial image data;

[0058] It should be noted that, in this embodiment, the delay duration is the time interval between the moment the image data to be processed is captured by the vehicle camera and the moment the processed image data is output to the AR glasses worn by the driver. The specific value of the delay duration is set by the technician based on the image processing capability and image transmission efficiency of the rendering engine preset in the terminal device, and this application does not impose any restrictions on it.

[0059] In this embodiment, when the terminal device is running, it first takes pictures of the environment around the vehicle through an image acquisition device deployed outside the vehicle to obtain initial image data. The acquired initial image data is then input into the image processing module configured inside the terminal device. The image processing module reads the storage device configured inside the terminal device to obtain the delay time preset by the technician corresponding to the initial image data, and determines the velocity vector corresponding to the initial pixel points contained in the initial image data based on the delay time.

[0060] For example, when the terminal device is running, it first controls the vehicle-mounted cameras deployed around the vehicle to capture images of the surrounding environment, thereby acquiring initial image data I0 that needs to include the environment around the vehicle. The terminal device then inputs the acquired initial image data I0 into the image processing module configured within the terminal device. The image processing module reads the storage device configured within the terminal device to obtain the delay duration dt preset by the technician corresponding to the initial image data I0. The image processing module determines a first image block of size s×s within the initial image data I0 and determines the initial pixels contained within the first image block. The image processing module then determines the motion state corresponding to the initial pixels and determines the velocity vector u and velocity vector v corresponding to the initial pixels based on the motion state corresponding to the initial pixels.

[0061] Furthermore, in a feasible embodiment, the step of "determining the velocity vector corresponding to the initial pixel in the initial image data" in step S10 above may specifically include:

[0062] Step S101: Determine the first image block contained within the initial image data;

[0063] In this embodiment, the image processing module first selects the image block that needs to be compensated in the initial image data and determines the image block as the first image block.

[0064] Step S102: Determine the second image block corresponding to the first image block in the first reference image data and / or the second reference image data corresponding to the initial image data;

[0065] In this embodiment, the image processing device reads the terminal device to obtain first reference image data that is located before the initial image data and spaced one frame apart, and second reference image data that is located after the initial image data and spaced one frame apart. The image processing module then determines a second image block that is similar to the first image block within the first reference image data and / or the second reference image data.

[0066] Step S103: Determine the velocity vector corresponding to the initial pixel in the initial image data based on the first image block and the second image block;

[0067] In this embodiment, the image processing module calculates each initial pixel contained in the first image block to determine the target pixel corresponding to the initial pixel in the second image block, and then determines the velocity vector corresponding to the initial pixel in the initial image data based on the initial pixel and the target pixel.

[0068] For example, the image processing module first determines an image block of size s×s to be compensated in the initial image data I0, and identifies this image block as the first image block. Simultaneously, the image processing module reads the terminal device to obtain first reference image data I1 located in front of the initial image data I0 and spaced one frame apart from the initial image data I0, and second reference image data I2 located behind the initial image data I0 and spaced one frame apart from the initial image data I0. The image processing module then determines a second image block similar to the first image block based on the first reference image data I1 and the second reference image data I2. Afterward, the image processing module determines the target pixel corresponding to each initial pixel in the first image block within the second pixel block based on the optical flow matching method preset by the technician. The image processing module then determines the velocity vector u and velocity vector v corresponding to each initial pixel according to the positional relationship between each initial pixel and the target pixel.

[0069] Furthermore, in a feasible embodiment, step S102 above may specifically include:

[0070] Step S1021: Detect the data type corresponding to the initial image data, wherein the data type includes simple image data and complex image data;

[0071] In this embodiment, after acquiring the initial image data, the image processing module first determines whether the data type corresponding to the initial image data is simple image data or complex image data.

[0072] Step S1022: If the data type is detected to be the simple image data, then a second image block corresponding to the first image block is determined in the first reference image data using a preset first motion estimation method;

[0073] In this embodiment, when the image processing module determines that the data type corresponding to the initial image data is simple image data, it determines a second image block similar to the first image block in the first reference image data using a block matching unidirectional motion estimation method preset by the technician.

[0074] Step S1023: If the data type is detected to be the complex image data, then the second image block contained in the first reference image data and the second reference image data is determined by a preset second motion estimation method;

[0075] In this embodiment, when the image processing module determines that the data type corresponding to the initial image data is complex graphic data, it uses a block matching bidirectional motion estimation method preset by the technician to determine the second image blocks that are similar to the first image blocks contained in the first reference image data and the second reference image data, respectively.

[0076] For example, the image processing module determines whether the data type corresponding to the initial image data I0 is simple image data or complex image data. Then, if the terminal device determines that the data type corresponding to the initial image data I0 is simple image data, it determines the search area in the acquired first reference image data I1 and determines the second image block corresponding to the first image block in the search area contained in the first reference image data I1 based on the block matching unidirectional motion estimation method preset by the technician.

[0077] If the image processing module determines that the data type corresponding to the initial image data I0 is complex image data, it determines the search area in the acquired first reference image data I1 and second reference image data I2 respectively, and determines the second image block similar to the first image block in the search area corresponding to the first reference image data I1 and the second reference image data I2 respectively based on the block matching bidirectional motion estimation method preset by the technician.

[0078] Finally, the image processing module determines the target pixel corresponding to the initial pixel contained in the first image block in the second image block, and determines the velocity vector u and velocity vector v corresponding to the initial pixel based on the positional relationship between the initial pixel and the target pixel.

[0079] Furthermore, in a feasible embodiment, the step S1022 above, "determining the second image block corresponding to the first image block in the first reference image data using a preset first motion estimation method," may specifically include:

[0080] Step A10: Determine the minimum mean square error function corresponding to the initial pixel points within the first image block;

[0081] In this embodiment, when the image processing module uses the block matching unidirectional motion estimation method preset by the technician, it first determines the minimum mean square error function corresponding to the initial pixel based on the pixel values ​​of each pixel in the search area corresponding to the first image block and the first reference image data.

[0082] Step A20: Determine the candidate motion estimation vector corresponding to the initial pixel based on the minimum mean square error function, and determine the second image block corresponding to the first image block in the first reference image data according to the candidate motion estimation vector;

[0083] In this embodiment, the image processing module determines the forward motion estimation vector corresponding to the initial pixel based on the minimum mean square error function, and then determines the second image block in the search area contained in the first reference image data based on the forward motion estimation vector corresponding to the initial pixel.

[0084] For example, please refer to Figure 3 , Figure 3 This is a schematic diagram of a block matching unidirectional motion estimation method involved in an embodiment of the image delay optimization method of this application, as shown below. Figure 3 As shown, when the image processing module uses the block matching unidirectional motion estimation method preset by the technician, it first determines the initial pixel points q0(x,y) contained in the first image block, and the pixel points q1(x+Δx,y+Δy) corresponding to each pixel point in the search area within the first reference image data I1. Then, based on the initial pixel points q0(x,y) and q1(x+Δx,y+Δy), the image processing module determines the minimum mean square error function corresponding to the initial pixel points in the first image block as follows:

[0085]

[0086] The image processing module then determines (Δx,Δy) = (udx,vdy), thereby obtaining the forward motion estimation vector corresponding to the initial pixel point within the first image block. The image processing module then uses the forward motion estimation vector to determine the position of the second image block within the search area of ​​the first reference image data I1.

[0087] It should be noted that, in this embodiment, M and N included in the minimum mean square error function represent the length and width values ​​of the image data, respectively.

[0088] Furthermore, in a feasible embodiment, the step S1023 above, "determining the second image block contained in the first reference image data and the second reference image data respectively by a preset second motion estimation method," may specifically include:

[0089] Step A30: Determine the first minimum absolute error function between the initial pixel and the first reference image data, and determine the first candidate motion estimation vector corresponding to the initial pixel based on the first minimum absolute error function;

[0090] In this embodiment, when the image processing module uses the block matching bidirectional motion estimation method preset by the technician, it determines the first minimum absolute error function corresponding to the initial pixel based on the pixel values ​​of each pixel contained in the search area corresponding to the first image block and the first reference image data, and determines the forward motion estimation vector corresponding to the initial pixel based on the first minimum absolute error function.

[0091] Step A40: Determine the second minimum absolute error function between the initial pixel and the second reference image data, and determine the second candidate motion estimation vector corresponding to the initial pixel based on the second minimum absolute error function;

[0092] In this embodiment, when the image processing module uses the block matching bidirectional motion estimation method preset by the technician, it determines the second minimum absolute error function corresponding to the initial pixel point based on the pixel value of each pixel point contained in the search area corresponding to the first image block and the second reference image data, and determines the backward motion estimation vector corresponding to the initial pixel point based on the second minimum absolute error function.

[0093] Step A50: Determine the second image block contained in the first reference image data and the second reference image data respectively based on the first candidate motion estimation vector and the second candidate motion estimation vector;

[0094] In this embodiment, the image processing module determines a second image block similar to the first image block in the search area corresponding to the first reference image data based on the acquired forward motion estimation vector. Similarly, the image processing module determines a second image block similar to the first image block in the search area corresponding to the second reference image data based on the acquired backward motion estimation vector.

[0095] For example, please refer to Figure 4 , Figure 4 This is a schematic diagram of a block-matching bidirectional motion estimation method involved in one embodiment of the image delay optimization method of this application, as shown below. Figure 4 As shown, when the image processing module uses the block matching bidirectional motion estimation method preset by the technician, it first determines the first minimum absolute error function corresponding to each initial pixel in the first image block based on the initial pixel q0(x,y) contained in the first image block and each pixel q1(x+Δx,y+Δy) in the search area within the first reference image data I1.

[0096]

[0097] The image processing module then determines the forward motion estimation vector corresponding to the initial pixel in the first image block based on the first minimum absolute error function, and the image processing module then determines the second image block in the search area within the first reference image data I1 based on the forward motion estimation vector.

[0098] Similarly, when using the block matching bidirectional motion estimation method preset by the technician, the image processing module determines the second minimum absolute error function corresponding to each initial pixel in the first image block based on the initial pixel q0(x,y) contained in the first image block and each pixel q2(x+Δx,y+Δy) within the search area in the second reference image data I2, and based on the initial pixel q0(x,y) and pixel q2(x+Δx,y+Δy):

[0099]

[0100] The image processing module then determines the backward motion estimation vector corresponding to the initial pixel point within the first image block based on the second minimum absolute error function, and the image processing module then determines the second image block within the search area of ​​the second reference image data I2 based on the backward motion estimation vector.

[0101] Furthermore, in a feasible embodiment, step S103 above may specifically include:

[0102] Step S1031: Calculate the partial derivatives for the initial pixels located within the first image block;

[0103] Step S1032: Determine the target matrix using the partial derivatives, and determine the target pixel corresponding to the initial pixel in the second image block based on the target matrix;

[0104] Step S1033: Determine the velocity vector corresponding to the initial pixel based on the initial pixel and the target pixel;

[0105] For example, after determining the second image block, the image processing module calculates the initial pixel q within the first image block based on the Lucas-Kanade optical flow method preset by the technician, thereby determining that the partial derivative of the initial pixel q with respect to the x-axis is I. x (q n ), and, the partial derivative of the initial pixel q with respect to the y-axis is I. y (q n ), and, the partial derivative of the initial pixel q with respect to time t is I t (q n The image processing module then determines the matrix based on the partial derivatives:

[0106]

[0107] Simultaneously, the image processing module obtains the preset vector calculation formula: Av = b, and calculates the vector calculation formula based on the least squares method to obtain:

[0108]

[0109] The image processing module thus determines V x and V y The value of V, and thus V x and V y The velocity vectors u and v are respectively determined as the initial pixel points.

[0110] It should be noted that, in this embodiment, matrix A is the matrix obtained by combining the partial derivatives of each initial pixel q in the initial image data I0 with the x-axis and y-axis respectively. Similarly, matrix v is the matrix obtained by combining the velocity vector u and velocity vector v corresponding to the initial pixel q. Similarly, matrix b is the matrix obtained by combining the partial derivatives of each initial pixel q with the t-axis respectively.

[0111] Step S20: Determine the frame number compensation formula based on the delay duration and the initial pixel points, and generate target image data based on the frame number compensation formula and the velocity vector;

[0112] In this embodiment, the image processing module determines the frame number compensation formula corresponding to the initial pixel based on the delay duration and the initial pixel contained in the initial image data. The image processing module then calculates the velocity vector corresponding to the initial pixel according to the obtained frame number compensation formula to generate the target image data.

[0113] For example, the image processing module determines the frame number compensation formula corresponding to the initial pixel point within the initial image data I0 based on the delay time dt as: Ixu + Ixv + It = 0. The image processing module then calculates the velocity vector u and velocity vector v based on the frame number compensation formula to generate the target image data I. t .

[0114] Furthermore, in a feasible embodiment, the step of "determining the frame number compensation formula based on the delay duration and the initial pixel points" in step S20 above may specifically include:

[0115] Step S201: Determine the pixel expression corresponding to the initial pixel point;

[0116] In this embodiment, the image processing module determines the initial pixel point within the initial image data and determines the pixel expression corresponding to the initial pixel point.

[0117] Step S202: Process the pixel expression based on the delay duration to generate a frame rate compensation formula;

[0118] In this embodiment, the image processing module performs Taylor expansion on the generated pixel expression to obtain a second pixel expression, and integrates the second pixel expression with the pixel expression before expansion to obtain the frame number compensation formula corresponding to the initial pixel.

[0119] For example, the image processing module first determines that the initial pixel point contained in the initial image data I0 is q(x,y,t). The image processing module then determines the pixel expression corresponding to the initial pixel point as: q(x,y,t)=q(x+dx,y+dy,t+dt). After that, the image processing module performs a Taylor expansion on the right half of the pixel expression to obtain the second pixel expression as:

[0120]

[0121] Since ε in the second pixel expression represents a second-order infinitesimal, it is ignored. The image processing module then substitutes the second pixel expression into the expanded pixel expression, removes the delay duration dt included in the formula, and obtains the third pixel expression as follows:

[0122]

[0123] Next, the image processing module presets u and v as the velocity vectors generated by the initial pixel along the x-axis and y-axis, respectively, thus determining the velocity vector u as:

[0124]

[0125] The velocity vector v is:

[0126]

[0127] Meanwhile, the image processing module determines the partial derivative of the initial pixel along the x-axis as follows: Similarly, the image processing module determines the partial derivative of the initial pixel along the y-axis as follows: Similarly, the image processing module determines the partial derivative of the initial pixel on the t-axis as: Finally, the image processing module then imports the partial derivatives into the third pixel expression to obtain the frame number compensation formula corresponding to the initial pixel:

[0128] qxu+qxv+qt=0.

[0129] Step S30: Perform frame interpolation based on the target image data to optimize the delay duration;

[0130] In this embodiment, the image processing module determines the interpolation position based on the delay duration, thereby inserting the target image data into the interpolation position to optimize the delay duration.

[0131] For example, the image processing module determines the interpolation position between the initial image data I0 and the subsequent second reference image data I2 based on the delay duration dt, and then interpolates the target image data I... t Insert this interpolation position to complete the optimization of the delay duration dt.

[0132] In this embodiment, when the terminal device is running, it first captures images of the environment around the vehicle using an image acquisition device deployed outside the vehicle to obtain initial image data. The acquired initial image data is then input into the image processing module configured within the terminal device. The image processing module reads the storage device configured within the terminal device to obtain the delay duration preset by the technician corresponding to the initial image data. Based on the delay duration, it determines the velocity vector corresponding to the initial pixel in the initial image data. Then, based on the delay duration and the initial pixel in the initial image data, the image processing module determines the frame number compensation formula corresponding to the initial pixel. The image processing module then calculates the velocity vector corresponding to the initial pixel according to the obtained frame number compensation formula to generate target image data. Finally, the image processing module determines the frame interpolation position based on the delay duration and inserts the target image data into the frame interpolation position to complete the optimization of the delay duration.

[0133] Thus, this application determines the motion pattern of the initial pixels contained in the initial image data based on the delay between when the image data is acquired and when it is output to the AR glasses. Then, it compensates the initial image data based on the motion pattern to generate target image data. By interpolating the initial image data with the target image data, this application achieves the technical effect of optimizing the delay in the transmission of image data by in-vehicle AR technology. This reduces the discomfort and delay experienced by the driver when acquiring image data through AR glasses, and improves the driver's user experience.

[0134] Furthermore, to achieve the above objectives, this application also provides an image delay optimization device, which is applied to a vehicle equipped with an image acquisition device. Please refer to [reference needed]. Figure 5 , Figure 5 This is a schematic diagram of the functional modules involved in an embodiment of the image delay optimization method of this application, as shown below. Figure 5 As shown, the device includes:

[0135] The parameter determination module 10 is used to acquire initial image data through the image acquisition device, and determine the delay time corresponding to the initial image data and the velocity vector corresponding to the initial pixel in the initial image data.

[0136] The image generation module 20 is used to determine the frame number compensation formula based on the delay duration and the initial pixel points, and to generate target image data based on the frame number compensation formula and the velocity vector.

[0137] The delay optimization module is used to perform frame interpolation operations based on the target image data to optimize the delay duration.

[0138] Furthermore, the parameter determination module 10 also includes:

[0139] A region segmentation unit is used to determine the first image block contained within the initial image data;

[0140] A region matching unit is used to determine the second image block corresponding to the first image block in the first reference image data and / or the second reference image data corresponding to the initial image data.

[0141] The vector calculation unit is used to determine the velocity vector corresponding to the initial pixel in the initial image data based on the first image block and the second image block.

[0142] Furthermore, the region matching unit includes:

[0143] A type detection subunit is used to detect the data type corresponding to the initial image data, wherein the data type includes simple image data and complex image data;

[0144] The first matching subunit is used to determine, if the data type is detected to be the simple image data, a second image block corresponding to the first image block in the first reference image data using a preset first motion estimation method.

[0145] The second matching subunit is used to determine the second image block contained in the first reference image data and the second reference image data respectively by means of a preset second motion estimation method if the data type is detected to be the complex image data.

[0146] Furthermore, the region matching unit also includes:

[0147] The first function calculation subunit is used to determine the minimum mean square error function corresponding to the initial pixel point in the first image block;

[0148] A unidirectional motion estimation subunit is used to determine the candidate motion estimation vector corresponding to the initial pixel based on the minimum mean square error function, and to determine the second image block corresponding to the first image block in the first reference image data based on the candidate motion estimation vector.

[0149] Furthermore, the region matching unit also includes:

[0150] The second function calculation subunit is used to determine the first minimum absolute error function between the initial pixel and the first reference image data, and to determine the first candidate motion estimation vector corresponding to the initial pixel based on the first minimum absolute error function.

[0151] The third function calculation subunit is used to determine the second minimum absolute error function between the initial pixel and the second reference image data, and to determine the second candidate motion estimation vector corresponding to the initial pixel based on the second minimum absolute error function;

[0152] A bidirectional motion estimation subunit is used to determine the second image block contained in the first reference image data and the second reference image data respectively, based on the first candidate motion estimation vector and the second candidate motion estimation vector.

[0153] Furthermore, the initial pixel is located within the first image block, and the vector calculation unit includes:

[0154] A partial derivative calculation subunit is used to calculate the partial derivatives of the initial pixel.

[0155] A pixel matching subunit is used to determine a target matrix by means of the partial derivatives, and to determine the target pixel corresponding to the initial pixel in the second image block based on the target matrix;

[0156] A velocity calculation subunit is used to determine the velocity vector corresponding to the initial pixel point based on the initial pixel point and the target pixel point.

[0157] Furthermore, the image generation module 20 includes:

[0158] The formula determination unit is used to determine the pixel expression corresponding to the initial pixel point;

[0159] The formula generation unit is used to process the pixel expression based on the delay duration to generate a frame rate compensation formula.

[0160] Furthermore, this application also provides a terminal device having an image latency optimization program that can run on a processor. When the terminal device executes the image latency optimization program, it implements the steps of the image latency optimization method as described in any of the above embodiments.

[0161] The specific embodiments of the terminal device in this application are basically the same as the embodiments of the above-described image delay optimization method, and will not be described in detail here.

[0162] Furthermore, this application also provides a computer-readable storage medium storing an image delay optimization program, which, when executed by a processor, implements the steps of the image delay optimization method as described in any of the above embodiments.

[0163] The specific embodiments of the computer-readable storage medium of this invention are basically the same as the embodiments of the image delay optimization method described above, and will not be repeated here.

[0164] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0165] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0166] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which can be a device that executes the image delay optimization method of this application, specifically a mobile terminal, data storage control terminal, PC, or portable computer, etc.) to execute the methods described in the various embodiments of this application.

[0167] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for optimizing image latency, characterized in that, The method is applied to vehicles equipped with image acquisition devices, and the image delay optimization method includes the following steps: The image acquisition device acquires initial image data, determines the delay duration corresponding to the initial image data, and determines the first image block contained within the initial image data. Detect the data type corresponding to the initial image data, wherein the data type includes simple image data and complex image data; If the data type is detected to be the simple image data, then a second image block corresponding to the first image block is determined in the first reference image data using a preset first motion estimation method; If the data type is detected to be complex image data, then the second image block contained in the first reference image data and the second reference image data is determined by a preset second motion estimation method. Based on the first image block and the second image block, determine the velocity vector corresponding to the initial pixel in the initial image data; The frame rate compensation formula is determined based on the delay duration and the initial pixel points, and the target image data is generated based on the frame rate compensation formula and the velocity vector. Frame interpolation is performed based on the target image data to optimize the delay duration.

2. The image delay optimization method as described in claim 1, characterized in that, The step of determining the second image block corresponding to the first image block in the first reference image data using a preset first motion estimation method includes: Determine the minimum mean square error function corresponding to the initial pixel points within the first image block; Based on the minimum mean square error function, a candidate motion estimation vector corresponding to the initial pixel is determined, and based on the candidate motion estimation vector, a second image block corresponding to the first image block is determined in the first reference image data.

3. The image delay optimization method as described in claim 2, characterized in that, The step of determining the second image blocks contained in the first reference image data and the second reference image data respectively by using a preset second motion estimation method includes: A first minimum absolute error function is determined between the initial pixel and the first reference image data, and a first candidate motion estimation vector corresponding to the initial pixel is determined based on the first minimum absolute error function; A second minimum absolute error function is determined between the initial pixel and the second reference image data, and a second candidate motion estimation vector corresponding to the initial pixel is determined based on the second minimum absolute error function; The second image block contained in the first reference image data and the second reference image data is determined based on the first candidate motion estimation vector and the second candidate motion estimation vector.

4. The image delay optimization method as described in claim 1, characterized in that, The initial pixel is located within the first image block, and the step of determining the velocity vector corresponding to the initial pixel in the initial image data based on the first image block and the second image block includes: The partial derivatives are calculated for the initial pixels. The target matrix is ​​determined by the partial derivatives, and the target pixel corresponding to the initial pixel in the second image block is determined based on the target matrix. The velocity vector corresponding to the initial pixel is determined based on the initial pixel and the target pixel.

5. The image delay optimization method as described in claim 1, characterized in that, The step of determining the frame number compensation formula based on the delay duration and the initial pixel points includes: Determine the pixel expression corresponding to the initial pixel point; The pixel expression is processed based on the delay duration to generate a frame rate compensation formula.

6. An image delay optimization device, characterized in that, The image delay optimization device is applied to a vehicle equipped with an image acquisition device, and the device includes: The parameter determination module is used to acquire initial image data through the image acquisition device, determine the delay duration corresponding to the initial image data, and determine the first image block contained in the initial image data; detect the data type corresponding to the initial image data, wherein the data type includes simple image data and complex image data; if the data type is detected to be simple image data, then a second image block corresponding to the first image block is determined in the first reference image data using a preset first motion estimation method; if the data type is detected to be complex image data, then the second image blocks contained in the first reference image data and the second reference image data are determined using a preset second motion estimation method; and determine the velocity vector corresponding to the initial pixel in the initial image data based on the first image block and the second image block. An image generation module is used to determine a frame number compensation formula based on the delay duration and the initial pixel points, and to generate target image data based on the frame number compensation formula and the velocity vector. The delay optimization module is used to perform frame interpolation operations based on the target image data to optimize the delay duration.

7. A terminal device, characterized in that, The terminal device includes: a memory, a processor, and an image latency optimization program stored in the memory and executable on the processor, wherein when the image latency optimization program is executed by the processor, it implements the steps of the image latency optimization method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an image delay optimization program, which, when executed by a processor, implements the steps of the image delay optimization method as described in any one of claims 1 to 5.

Images