Method, device and equipment for generating intermediate frame image and storage medium

Abstract

The application discloses a method and device for generating an intermediate frame image, an equipment and a storage medium, and relates to the technical field of image processing. The method comprises the following steps: acquiring an original intermediate frame image and a lens distortion coefficient value; determining a pixel group according to the distance between any pixel in the original intermediate frame image and the center pixel of the original intermediate frame image, wherein the distance between any pixel in any pixel group and the center pixel is the same; for any pixel group, determining a distortion value according to the distance between a pixel in the pixel group and the center pixel and the lens distortion coefficient value; and for each pixel in any pixel group, determining an anti-distortion intermediate frame image according to the corresponding distortion value and the image coordinate value of each pixel in the original intermediate frame image. The method can avoid the problems of low GPU work efficiency and increased display delay, and can avoid waste of computing resources.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a method for generating intermediate frame images, an apparatus for generating intermediate frame images, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Currently, VR devices typically require two rendering processes from scene rendering to display. These two rendering processes are VR scene rendering and Asynchronous Timewarp (ATW) rendering, both of which are performed by the GPU.

[0003] like Figure 1 The diagram illustrates the timing of GPU rendering of VR scenes and ATW (Automatic Time-Wave). Within a rendering cycle (between VsyncN and VsyncN+1), since both ATW and VR scene rendering are performed by the GPU, resource contention inevitably occurs between these two processes, leading to GPU hardware thread switching. Frequent GPU hardware thread switching results in low GPU efficiency and increased display latency. (See also...) Figure 1 As shown, the theoretical time consumption for VR scene rendering is Δt. render = t0 + t2 + t4 + t6, the theoretical time consumption of ATW is Δt ATW = t1 + t3 + t5, but due to preemption between hardware threads, the actual time consumed for VR scene rendering is Δt. render = t0 + t1 + t2 + t3 + t4 + t5 + t6, the actual time consumption of ATW is Δt. ATW = t1+t2+t3+t4+t5, which means the display delay increases. Summary of the Invention

[0004] One objective of this application is to provide a new technical solution for generating intermediate frame images.

[0005] According to a first aspect of this application, a method for generating intermediate frame images is provided, applied to a digital signal processor, comprising:

[0006] Obtain the original intermediate frame image and lens distortion coefficient values;

[0007] A pixel group is determined based on the distance between any pixel in the original intermediate frame image and the center pixel of the original intermediate frame image, wherein any pixel in any pixel group is at the same distance from the center pixel;

[0008] For any pixel group, the distortion value is determined based on the distance between a pixel in the pixel group and the center pixel, as well as the lens distortion coefficient value.

[0009] For each pixel in any pixel group, an anti-distortion intermediate frame image is determined based on the corresponding distortion value and the image coordinate value of each pixel in the original intermediate frame image.

[0010] Optionally, determining the anti-distortion intermediate frame image for each pixel in any pixel group, based on the corresponding distortion value and the image coordinates of each pixel in the original intermediate frame image, includes:

[0011] For each pixel in any pixel group, based on DSP HVX vector operations, an anti-distortion intermediate frame image is determined according to the corresponding distortion value and the image coordinate value of each pixel in the original intermediate frame image.

[0012] Optionally, before determining the pixel group based on the distance between any pixel in the original intermediate frame image and the center pixel of the original intermediate frame image, the method further includes:

[0013] Obtain the pixel coordinates of any pixel in the original intermediate frame image, the pixel coordinates of the center pixel of the original intermediate frame image, and the lens focal length;

[0014] The image coordinates of any pixel in the original intermediate frame image are determined based on the pixel coordinates of any pixel in the original intermediate frame image, the pixel coordinates of the center pixel of the original intermediate frame image, and the lens focal length.

[0015] Based on the image coordinates of any pixel in the original intermediate frame image, determine the distance between any pixel in the original intermediate frame image and the center pixel of the original intermediate frame image.

[0016] Optionally, before determining the anti-distortion intermediate frame image based on the corresponding distortion value and the image coordinate value of each pixel in the original intermediate frame image, the method further includes:

[0017] Obtain the pixel coordinates of the center pixel, the pixel coordinates of each pixel, and the lens focal length;

[0018] Based on the pixel coordinates of the center pixel, the pixel coordinates of each pixel, and the lens focal length, the image coordinates of each pixel in the original intermediate frame image are determined.

[0019] Optionally, before obtaining the original intermediate frame image and the lens distortion coefficient values, the method further includes:

[0020] Detect whether the mode for generating intermediate frame images by the digital signal processor is enabled;

[0021] When enabled, the step of acquiring the original intermediate frame image and lens distortion coefficient value is triggered.

[0022] Optionally, obtaining the original intermediate frame image and lens distortion coefficient values ​​includes:

[0023] Obtain lens parameter values;

[0024] The lens distortion coefficient value is determined based on the lens parameter values ​​and preset mapping data, wherein the preset mapping data is data reflecting the correspondence between the lens parameter values ​​and the corresponding lens distortion coefficient values.

[0025] According to a second aspect of this application, an apparatus for generating intermediate frame images is provided, applied to a digital signal processor, comprising:

[0026] The acquisition module is used to acquire the original intermediate frame image and lens distortion coefficient values;

[0027] The first determining module is used to determine a pixel group based on the distance between any pixel in the original intermediate frame image and the center pixel of the original intermediate frame image, wherein any pixel in any pixel group is at the same distance from the center pixel;

[0028] The second determining module is used to determine the distortion value for any pixel group based on the distance between a pixel in the pixel group and the center pixel and the lens distortion coefficient value.

[0029] The third determining module is used to determine the anti-distortion intermediate frame image for each pixel in any pixel group based on the corresponding distortion value and the image coordinate value of each pixel in the original intermediate frame image.

[0030] Optionally, the third determining module is specifically used to determine the anti-distortion intermediate frame image for each pixel in any pixel group based on DSPHVX vector operation, according to the corresponding distortion value and the image coordinate value of each pixel in the original intermediate frame image.

[0031] According to a third aspect of this application, an electronic device is provided, the electronic device comprising means for generating an intermediate frame image as described in any one of the second aspects; or,

[0032] The electronic device includes a memory and a digital signal processor, wherein the memory is used to store computer instructions, and the processor is used to retrieve the computer instructions from the memory to execute the method for generating an intermediate frame image as described in any one of the first aspects.

[0033] According to a fourth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements a method for generating an intermediate frame image according to any one of the first aspects.

[0034] This application provides a method for generating intermediate frame images, applied to a digital signal processor (DSP), comprising: acquiring an original intermediate frame image and lens distortion coefficient values; determining a pixel group based on the distance between any pixel in the original intermediate frame image and the center pixel of the original intermediate frame image, wherein any pixel in any pixel group is at the same distance from the center pixel; for any pixel group, determining a distortion value based on the distance between a pixel in the pixel group and the center pixel and the lens distortion coefficient value; and for each pixel in any pixel group, determining an anti-distortion intermediate frame image based on the corresponding distortion value and the image coordinates of each pixel in the original intermediate frame image. On one hand, this method is executed by a digital signal processor (DSP). This eliminates the need for a GPU to implement ATW (Automatic Time-of-Warping), meaning the GPU only needs to perform scene rendering (e.g., VR scene rendering). This avoids resource contention, thereby avoiding GPU hardware thread switching. Therefore, it avoids the problems of low GPU efficiency and increased display latency. On the other hand, by grouping the pixels in the original intermediate frame image, the distortion value only needs to be calculated once for a pixel group. Compared with the traditional technology where the GPU calculates the distortion value for each pixel in the original intermediate frame image when implementing ATW, this avoids the waste of computing resources.

[0035] Other features and advantages of this application will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0036] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the present application and, together with their description, serve to explain the principles of the present application.

[0037] Figure 1 This is a timing diagram of VR scene rendering and ATW using a GPU based on traditional technology;

[0038] Figure 2 This is a hardware configuration outline of an electronic device for implementing a method for generating intermediate frame images according to an embodiment of this application. Figure 1 ;

[0039] Figure 3 This is a flowchart illustrating a method for generating intermediate frame images according to an embodiment of this application.

[0040] Figure 4This is a schematic diagram of a pixel group according to an embodiment of this application;

[0041] Figure 5a This is a schematic diagram of a pincushion distortion according to an embodiment of this application;

[0042] Figure 5b This is a schematic diagram illustrating a user viewing an intermediate frame image according to an embodiment of this application;

[0043] Figure 6 This is a schematic diagram of the structure of an intermediate frame image generation device according to an embodiment of this application;

[0044] Figure 7 This is a schematic diagram of a DSP HVX vector operation process according to an embodiment of this application;

[0045] Figure 8 This is a hardware configuration outline of an electronic device for implementing an image brightness adjustment method according to an embodiment of this application. Figure 2 . Detailed Implementation

[0046] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0047] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the scope of this application and its application or use.

[0048] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0049] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0050] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0051] Figure 2 This is a hardware configuration outline of an electronic device for implementing a method for generating intermediate frame images according to an embodiment of this application. Figure 1 .

[0052] The electronic device 2000 is exemplarily a head-mounted device such as AR, MR, VR, or XR. The head-mounted device can be a separate head-mounted device or an integrated head-mounted device. The head-mounted device is a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application does not specifically limit the specific operating system used.

[0053] Electronic device 2000 may include processor 2100, memory 2200, interface device 2300, communication device 2400, display device 2500, input device 2600, speaker 2700, microphone 2800, etc. The processor 2100 is a digital signal processor (DSP). The memory 2200 includes, for example, ROM (read-only memory), RAM (random access memory), or non-volatile memory such as a hard disk. The interface device 2300 includes, for example, a USB interface or a headphone jack. The communication device 2400 is capable of wired or wireless communication. The display device 2500 is, for example, an LCD screen or a touch screen. The input device 2600 may include, for example, a touch screen or a keyboard. Users can input / output voice information through the speaker 2700 and microphone 2800.

[0054] Despite Figure 2 The electronic device 2000 shows multiple devices, but this application may only involve some of them. For example, the electronic device 2000 may only involve the memory 2200, the processor 2100, and the display device 2500.

[0055] In the embodiments applied in this application, the memory 2200 of the electronic device 2000 is used to store instructions for controlling the processor 2100 to execute the method for generating intermediate frame images provided in the embodiments of this application.

[0056] In the above description, those skilled in the art can design instructions based on the scheme disclosed in this application. How the instructions control the processor to operate is well known in the art, and therefore will not be described in detail here.

[0057] The intermediate frame image generation method provided in this application is applied to a Digital Signal Processor (DSP). This eliminates the need for a GPU to implement ATW (Automatic Time Written), meaning the GPU only needs to perform scene rendering (e.g., VR scene rendering). This avoids resource contention and thus avoids switching of GPU hardware threads. Therefore, it avoids the problems of low GPU efficiency and increased display latency.

[0058] like Figure 3As shown, the method for generating intermediate frame images provided in this application embodiment includes the following steps S3100-S3400:

[0059] S3100: Obtain the original intermediate frame image and lens distortion coefficient values.

[0060] In this embodiment, the original intermediate frame image is a normal intermediate frame image that has not undergone anti-distortion, i.e., an intermediate frame image without distortion. Furthermore, the DSP can use conventional techniques to generate the intermediate frame image to achieve the acquisition of the intermediate frame image. It should be noted that this embodiment does not limit the method of generating the original intermediate frame image.

[0061] The lens distortion factor is the value corresponding to the distortion factor of the lens of an electronic device. When the lens is fixed, the lens distortion factor value is fixed and known.

[0062] In this embodiment, the lens distortion coefficient values ​​are specifically k1, k2, and k3. Of course, the lens distortion coefficient values ​​can be more than one value, and this embodiment does not limit this.

[0063] In one embodiment of this application, the lens distortion coefficient can be obtained through the following steps S3110 and S3111:

[0064] S3110, Obtain lens parameter values.

[0065] S3111. Determine the lens distortion parameter value based on the lens parameter value and the preset mapping data.

[0066] Among them, the preset mapping data is data that reflects the correspondence between lens parameter values ​​and corresponding lens distortion coefficient values.

[0067] In this embodiment, the electronic device stores preset mapping data and lens parameter values. The electronic device determines the lens distortion coefficient value by looking up the preset mapping data according to the lens parameter values. This eliminates the need to rewrite the corresponding lens distortion coefficient value when the electronic device is matched with different lenses.

[0068] S3200: Determine the pixel group based on the distance between any pixel in the original intermediate frame image and the center pixel of the original intermediate frame image.

[0069] In this system, any pixel in any pixel group is equidistant from the center pixel.

[0070] In this embodiment, the center pixel of the original intermediate frame image refers to the pixel located at the center of the original intermediate frame image. Furthermore, pixels in the original intermediate frame image that are equidistant from the center pixel are grouped together as a set of pixels. Based on this, the resulting pixel groups can be denoted as HVX1, HVX2, HVX3, ...

[0071] It is understandable that, such as Figure 4 As shown, a group of pixels is located on a circular ring. Among them, Figure 4 The following example illustrates the use of pixel group one and pixel group two as examples.

[0072] Based on the lens distortion model, pixels equidistant from the center pixel of the original intermediate frame image have the same lens distortion model value. Therefore, pixels in the original intermediate frame image are grouped according to their distance from the center pixel, with each group consisting of pixels equidistant from the center pixel. For each pixel group, the lens distortion model value (referred to as the distortion value in this application) only needs to be calculated once. In contrast, in conventional technologies, when ATW is implemented using a GPU, the GPU's processing mechanism calculates the lens distortion model value for each pixel in the original intermediate frame image. Therefore, in this embodiment, grouping the pixels in the original intermediate frame image avoids wasting computational resources.

[0073] In one embodiment of this application, the method for generating intermediate frame images provided in this application further includes, before S3200 above, a step of calculating the distance between any pixel in the original intermediate frame image and the center pixel of the original intermediate frame image, which includes the following S3210-S3212:

[0074] S3210: Obtain the pixel coordinates of any pixel in the original intermediate frame image, the pixel coordinates of the center pixel of the original intermediate frame image, and the lens focal length.

[0075] In this embodiment of the application, the pixel coordinate value of any pixel in the original intermediate frame image specifically refers to the coordinate value of any pixel in the original intermediate frame image in the pixel coordinate system, with the unit being pixels. Furthermore, the pixel coordinate value of any pixel in the original intermediate frame image is denoted as (X, Y).

[0076] Correspondingly, the pixel coordinates of the center pixel of the original intermediate frame image specifically refer to the coordinates of the center pixel of the original intermediate frame image in the pixel coordinate system, in pixels. And, the pixel coordinates of the center pixel of the original intermediate frame image are denoted as (c... x c y ).

[0077] And, the focal length of the lens is denoted as f.

[0078] S3211. Determine the image coordinates of any pixel in the original intermediate frame image based on the pixel coordinates of any pixel in the original intermediate frame image, the pixel coordinates of the center pixel of the original intermediate frame image, and the lens focal length.

[0079] In this embodiment of the application, the image coordinates of any pixel in the original intermediate frame image are denoted as (x, y), where x and y are determined by the following formulas one and two, respectively:

[0080]

[0081]

[0082] S3212. Based on the image coordinates of any pixel in the original intermediate frame image, determine the distance between any pixel in the original intermediate frame image and the center pixel of the original intermediate frame image.

[0083] In this embodiment of the application, the image coordinates of any pixel in the original intermediate frame image are denoted as (x, y). The distance between any pixel in the original intermediate frame image and the center pixel of the original intermediate frame image can be determined by the following formula three:

[0084]

[0085] S3300: For any pixel group, determine the distortion value based on the distance between a pixel in the pixel group and the center pixel, as well as the lens distortion coefficient value.

[0086] It is understandable that electronic device lenses exhibit distortion, which is typically as follows: Figure 5a The radial distortion shown is pincushion distortion.

[0087] To ensure users see a normal, distortion-free intermediate frame image through the lens of an electronic device, after generating the original, undistorted intermediate frame image, barrel distortion correction needs to be applied. This way, users will see the image through a lens with pincushion distortion as... Figure 5b When you see the original intermediate frame image after barrel distortion, you can see an intermediate frame image without distortion.

[0088] In order to reverse barrel distortion on the original intermediate frame image, it is first necessary to determine the distortion value of each group of pixels in the original intermediate frame image.

[0089] In this embodiment of the application, a distortion value k corresponding to any pixel group is determined by the following formula four. d :

[0090] kd =(1+k1*r 2 +k2*r 4 +k3*r 6 (Formula 4)

[0091] It should be noted that, in this embodiment of the application, based on the distortion model shown in Formula 4 above, it is known that, on the basis that the distance between any pixel in a pixel group and the center pixel is the same, the distortion value corresponding to the distortion model shown in Formula 4 only needs to be calculated once to obtain the distortion value corresponding to the distortion model of all pixels in the pixel group.

[0092] S3400. For each pixel in any pixel group, determine the anti-distortion intermediate frame image based on the corresponding distortion value and the image coordinate value of each pixel in the original intermediate frame image.

[0093] In one embodiment of this application, the method for generating intermediate frame images provided in this application further includes the following steps S3410 and S3411 before the above-described S3400:

[0094] S3410: Obtain the pixel coordinates of the center pixel, the pixel coordinates of each pixel, and the lens focal length.

[0095] S3411. Determine the image coordinates of each pixel in the original intermediate frame image based on the pixel coordinates of the center pixel, the pixel coordinates of each pixel, and the lens focal length.

[0096] In the embodiments of this application, S3411 described above can be obtained by referring to Formula 1 and Formula 2. Further details will not be provided.

[0097] In this embodiment of the application, taking the image coordinates of a pixel in a pixel group as (x1, y1) as an example, the image coordinates of the pixel after anti-distortion are denoted as (x1, y1). d y d ), where x d y d The sum is determined by the following formulas five and six:

[0098] x d =k d x1 (Formula 5);

[0099] y d =k d y1 (Formula 6).

[0100] Based on the above steps S3100-S3400, the distortion correction of the original intermediate frame image can be completed. When the distorted intermediate frame image obtained from step S3400 is displayed, the user sees an intermediate frame image without distortion.

[0101] This application provides a method for generating intermediate frame images, applied to a digital signal processor (DSP), comprising: acquiring an original intermediate frame image and lens distortion coefficient values; determining a pixel group based on the distance between any pixel in the original intermediate frame image and the center pixel of the original intermediate frame image, wherein any pixel in any pixel group is at the same distance from the center pixel; for any pixel group, determining a distortion value based on the distance between a pixel in the pixel group and the center pixel and the lens distortion coefficient value; and for each pixel in any pixel group, determining an anti-distortion intermediate frame image based on the corresponding distortion value and the image coordinates of each pixel in the original intermediate frame image. On one hand, this method is executed by a digital signal processor (DSP). This eliminates the need for a GPU to implement ATW (Automatic Time-of-Warping), meaning the GPU only needs to perform scene rendering (e.g., VR scene rendering). This avoids resource contention, thereby avoiding GPU hardware thread switching. Therefore, it avoids the problems of low GPU efficiency and increased display latency. On the other hand, by grouping the pixels in the original intermediate frame image, the distortion value only needs to be calculated once for a pixel group. Compared with the traditional technology where the GPU calculates the distortion value for each pixel in the original intermediate frame image when implementing ATW, this avoids the waste of computing resources.

[0102] In one embodiment of this application, the above-mentioned S3400 can be specifically implemented by the following S3420:

[0103] S3420. For each pixel in any pixel group, based on DSP HVX vector operation, determine the anti-distortion intermediate frame image according to the corresponding distortion value and the image coordinate value of each pixel in the original intermediate frame image.

[0104] In this embodiment of the application, a schematic diagram of the DSP HVX vector operation process is shown below. Figure 6 As shown in the figure. Vu.ub represents the image coordinates of each pixel in a pixel group in the original intermediate frame image; Rt.b represents the distortion value corresponding to the corresponding pixel group; and Vd.w represents the pixel in the anti-distortion intermediate frame image corresponding to the corresponding pixel group.

[0105] Furthermore, the intermediate frame image generation method provided in this application embodiment is executed by a DSP, which is a parallel processor. Based on this, when the DSP executes the above S3420, it only needs to execute the HVX instruction of the pixel group number to complete the calculation of the anti-distortion corresponding to each pixel of the original intermediate frame image. This greatly improves the speed of determining the anti-distortion intermediate frame image.

[0106] In one embodiment of this application, the method for generating intermediate frame images provided in this application further includes the following steps S3120 and S3121 before the above-described S3100:

[0107] S3120: Detect whether the mode for generating intermediate frame images by the digital signal processor is enabled.

[0108] In this embodiment of the application, the electronic device has two modes: a mode in which the digital signal processor generates intermediate frame images and a mode in which the GPU generates intermediate frame images.

[0109] The activation of the two modes of the electronic device is triggered by the user. If the user wishes to control the mode where the digital signal processor (DSP) generates intermediate frame images, then the DSP-generated intermediate frame image mode is activated. Otherwise, the DSP-generated intermediate frame image mode is activated by default.

[0110] S3121. When enabled, trigger the step of acquiring the original intermediate frame image and lens distortion coefficient value.

[0111] In this embodiment of the application, when the mode of generating intermediate frame images by the digital signal processor is enabled, the steps S3100 to S3400 described above are triggered so that the digital signal processor can determine the anti-distortion intermediate frame image.

[0112] In this embodiment of the application, the above-described S3120 and S3121 enable the method for generating intermediate frame images provided in this embodiment of the application to meet the personalized needs of users.

[0113] This application provides an intermediate frame image generation apparatus 700, such as... Figure 7 As shown, it is applied to a digital signal processor and includes:

[0114] The acquisition module 710 is used to acquire the original intermediate frame image and lens distortion coefficient values;

[0115] The first determining module 720 is used to determine a pixel group based on the distance between any pixel in the original intermediate frame image and the center pixel of the original intermediate frame image, wherein any pixel in any pixel group is at the same distance from the center pixel.

[0116] The second determining module 730 is used to determine the distortion value for any pixel group based on the distance between a pixel in the pixel group and the center pixel and the lens distortion coefficient value.

[0117] The third determining module 740 is used to determine an anti-distortion intermediate frame image for each pixel in any pixel group based on the corresponding distortion value and the image coordinate value of each pixel in the original intermediate frame image.

[0118] In this embodiment, an apparatus for generating intermediate frame images is provided, executed by a digital signal processor (DSP), comprising: an acquisition module for acquiring an original intermediate frame image and lens distortion coefficient values; a first determination module for determining a pixel group based on the distance between any pixel in the original intermediate frame image and the center pixel of the original intermediate frame image, wherein any pixel in any pixel group is at the same distance from the center pixel; a second determination module for determining a distortion value for any pixel group based on the distance between a pixel in the pixel group and the center pixel and the lens distortion coefficient values; and a third determination module for determining an anti-distortion intermediate frame image for each pixel in any pixel group based on the corresponding distortion value and the image coordinates of each pixel in the original intermediate frame image. On one hand, this apparatus is applied to a DSP. This eliminates the need for a GPU to implement ATW (Automatic Time-of-Warping), meaning the GPU only needs to perform scene rendering (e.g., VR scene rendering). This avoids resource contention, thereby avoiding GPU hardware thread switching. Therefore, it avoids the problems of low GPU efficiency and increased display latency. On the other hand, by grouping the pixels in the original intermediate frame image, the distortion value only needs to be calculated once for a pixel group. Compared with the traditional technology where the GPU calculates the distortion value for each pixel in the original intermediate frame image when implementing ATW, this avoids the waste of computing resources.

[0119] In one embodiment of this application, the third determining module 740 is specifically used to determine an anti-distortion intermediate frame image for each pixel in any pixel group based on DSP HVX vector operation, according to the corresponding distortion value and the image coordinate value of each pixel in the original intermediate frame image.

[0120] In one embodiment of this application, the acquisition module 710 is further configured to acquire the pixel coordinate value of any pixel in the original intermediate frame image, the pixel coordinate value of the center pixel of the original intermediate frame image, and the lens focal length.

[0121] Furthermore, the first determining module 720 is also used for:

[0122] The image coordinates of any pixel in the original intermediate frame image are determined based on the pixel coordinates of any pixel in the original intermediate frame image, the pixel coordinates of the center pixel of the original intermediate frame image, and the lens focal length.

[0123] Furthermore, based on the image coordinates of any pixel in the original intermediate frame image, the distance between any pixel in the original intermediate frame image and the center pixel of the original intermediate frame image is determined.

[0124] In one embodiment of this application, the acquisition module 710 is further configured to acquire the pixel coordinate value of the center pixel, the pixel coordinate value of each pixel, and the lens focal length.

[0125] Furthermore, the third determining module 740 is also configured to determine the image coordinates of each pixel in the original intermediate frame image based on the pixel coordinates of the center pixel, the pixel coordinates of each pixel, and the lens focal length.

[0126] In one embodiment of this application, the device 700 further includes:

[0127] The detection module is used to detect whether the mode for generating intermediate frame images by the digital signal processor is enabled;

[0128] The trigger module is used to trigger the step of acquiring the original intermediate frame image and lens distortion coefficient value when it is enabled.

[0129] In one embodiment of this application, the acquisition module 710 is specifically used to acquire lens parameter values;

[0130] Furthermore, the lens distortion coefficient value is determined based on the lens parameter values ​​and preset mapping data, wherein the preset mapping data is data reflecting the correspondence between the lens parameter values ​​and the corresponding lens distortion coefficient values.

[0131] This application embodiment also provides an electronic device 800, which includes any of the intermediate frame image generation devices 700 provided in the above-described device embodiments; or,

[0132] like Figure 8 As shown, the electronic device 800 includes a memory 810 and a digital signal processor 820, wherein the memory 810 is used to store computer instructions, and the processor 820 is used to call the computer instructions from the memory 810 to execute the intermediate frame image generation method as provided in any of the above method embodiments.

[0133] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method for generating an intermediate frame image according to any one of the above method embodiments.

[0134] This application may be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this application.

[0135] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0136] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0137] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from the computer-readable program instructions. These electronic circuits can execute the computer-readable program instructions to implement various aspects of this application.

[0138] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0139] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0140] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0141] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be well known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0142] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this application is defined by the appended claims.

Claims

1. A method for generating intermediate frame images, characterized in that, Applied to digital signal processors, it performs asynchronous time warping in VR scene rendering, including: Obtain the original intermediate frame image and lens distortion coefficient values; A pixel group is determined based on the distance between any pixel in the original intermediate frame image and the center pixel of the original intermediate frame image, wherein any pixel in any pixel group is at the same distance from the center pixel; For any pixel group, the distortion value is determined based on the distance between a pixel in the pixel group and the center pixel, as well as the lens distortion coefficient value. For each pixel in any pixel group, based on the HVX vector operation of the digital signal processor, according to the corresponding distortion value and the image coordinate value of each pixel in the original intermediate frame image, the HVX instruction of the pixel group number is executed to complete the calculation of the anti-distortion corresponding to each pixel of the original intermediate frame image and determine the anti-distortion intermediate frame image. The method further includes, before obtaining the original intermediate frame image and the lens distortion coefficient value: Detect whether the mode for generating intermediate frame images by the digital signal processor is enabled; When enabled, the step of acquiring the original intermediate frame image and lens distortion coefficient value is triggered.

2. The method according to claim 1, characterized in that, Before determining the pixel group based on the distance between any pixel in the original intermediate frame image and the center pixel of the original intermediate frame image, the method further includes: Obtain the pixel coordinates of any pixel in the original intermediate frame image, the pixel coordinates of the center pixel of the original intermediate frame image, and the lens focal length; The image coordinates of any pixel in the original intermediate frame image are determined based on the pixel coordinates of any pixel in the original intermediate frame image, the pixel coordinates of the center pixel of the original intermediate frame image, and the lens focal length. Based on the image coordinates of any pixel in the original intermediate frame image, determine the distance between any pixel in the original intermediate frame image and the center pixel of the original intermediate frame image.

3. The method according to claim 1, characterized in that, Before determining the anti-distortion intermediate frame image based on the corresponding distortion value and the image coordinate value of each pixel in the original intermediate frame image, the method further includes: Obtain the pixel coordinates of the center pixel, the pixel coordinates of each pixel, and the lens focal length; Based on the pixel coordinates of the center pixel, the pixel coordinates of each pixel, and the lens focal length, the image coordinates of each pixel in the original intermediate frame image are determined.

4. The method according to claim 1, characterized in that, The process of obtaining the original intermediate frame image and lens distortion coefficient values ​​includes: Obtain lens parameter values; The lens distortion coefficient value is determined based on the lens parameter values ​​and preset mapping data, wherein the preset mapping data is data reflecting the correspondence between the lens parameter values ​​and the corresponding lens distortion coefficient values.

5. An apparatus for generating intermediate frame images, characterized in that, Applied to digital signal processors, it performs asynchronous time warping in VR scene rendering, including: The acquisition module is used to acquire the original intermediate frame image and lens distortion coefficient values; The first determining module is used to determine a pixel group based on the distance between any pixel in the original intermediate frame image and the center pixel of the original intermediate frame image, wherein any pixel in any pixel group is at the same distance from the center pixel; The second determining module is used to determine the distortion value for any pixel group based on the distance between a pixel in the pixel group and the center pixel and the lens distortion coefficient value. The third determining module is used to perform HVX vector operation based on the digital signal processor for each pixel in any pixel group, and according to the corresponding distortion value and the image coordinate value of each pixel in the original intermediate frame image, execute the HVX instruction of the pixel group number of times to complete the calculation of the anti-distortion corresponding to each pixel of the original intermediate frame image, and determine the anti-distortion intermediate frame image. The detection module is used to detect whether the mode for generating intermediate frame images by the digital signal processor is enabled; The trigger module is used to trigger the step of acquiring the original intermediate frame image and lens distortion coefficient value when it is enabled.

6. An electronic device, characterized in that, The electronic device includes the intermediate frame image generation apparatus as described in claim 5; or, The electronic device includes a memory and a digital signal processor, wherein the memory is used to store computer instructions, and the processor is used to retrieve the computer instructions from the memory to execute the method for generating an intermediate frame image as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method for generating intermediate frame images according to any one of claims 1-4.

Images