A method, apparatus, and medium for image processing

Abstract

Embodiments of the present disclosure relate to a method, apparatus and medium for image processing. The method proposed herein comprises: determining scene description information based on at least one reference frame, the scene description information indicating complexity corresponding to the at least one reference frame, the motion complexity indicating motion vector information associated with the at least one reference frame, and the picture complexity indicating object distribution in the at least one reference frame; determining at least one control parameter based on the scene description information and a usage state of image processing resources, the at least one control parameter comprising at least one of: a resolution of a motion vector, an accuracy of depth information, and the accuracy indicating a bit width of the depth information; determining at least one interpolation parameter based on the at least one control parameter; and performing interpolation processing based on a target reference frame in the at least one reference frame and the at least one interpolation parameter to generate an intermediate frame. In this way, embodiments of the present disclosure can effectively improve the efficiency of generating the intermediate frame.

Description

Technical Field

[0001] The embodiments of this disclosure generally relate to image processing techniques, and more specifically, to an image processing method, apparatus, and medium. Background Technology

[0002] Today, image processing technology is being applied to all aspects of people's lives. It can provide people with a variety of video or game scenes. Traditional real-time frame interpolation technology has problems such as large amount of motion vector calculation, high bandwidth consumption for depth accuracy, and fixed frame interpolation rate, which can easily lead to performance bottlenecks on mobile or low-end devices. Summary of the Invention

[0003] To address the aforementioned technical problems, in a first aspect, an image processing method is proposed. The method includes: determining scene description information based on at least one constructed reference frame, wherein the scene description information indicates the motion complexity or image complexity corresponding to the at least one reference frame, the motion complexity indicates motion vector information associated with the at least one reference frame, and the image complexity at least indicates the object distribution in the at least one reference frame; determining at least one control parameter based on the scene description information and the usage status of image processing resources, wherein the at least one control parameter includes at least one of the following: resolution of the motion vector, precision of the depth information, and precision indicating the bit width of the depth information; determining at least one interpolation parameter based on the at least one control parameter; and performing interpolation processing based on a target reference frame in the at least one reference frame and the at least one interpolation parameter to generate an intermediate frame.

[0004] In a second aspect, an apparatus for image processing is proposed. The apparatus includes a processor and a non-transitory memory having instructions thereon. When executed by the processor, the instructions cause the processor to perform the method according to the first aspect of this disclosure.

[0005] In a third aspect, a non-transitory computer-readable storage medium is proposed. This non-transitory computer-readable storage medium stores instructions that cause a processor to perform the method according to the first aspect of this disclosure.

[0006] The embodiments of this disclosure can determine at least one control parameter based on scene description information and the usage status of image processing resources, and dynamically perform interpolation processing based on at least one interpolation parameter determined by the at least one control parameter. In this way, the embodiments of this disclosure can effectively improve the efficiency of generating intermediate frames.

[0007] The present invention is provided to present, in a simplified form, the selection of concepts further described below in the detailed description. The present invention is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0008] The above and other objects, features, and advantages of exemplary embodiments of the present disclosure will become more apparent from the following detailed description with reference to the accompanying drawings. In the exemplary embodiments of the present disclosure, the same reference numerals generally refer to the same components.

[0009] Figure 1 A schematic diagram of an example environment according to some embodiments of the present disclosure is shown;

[0010] Figure 2 An example flowchart of an image processing method according to some embodiments of the present disclosure is shown;

[0011] Figure 3 A block diagram of a computing device in which various embodiments of the present disclosure may be implemented is shown.

[0012] Throughout all the accompanying figures, the same or similar reference numerals generally refer to the same or similar elements. Detailed Implementation

[0013] The principles of this disclosure will now be described with reference to some embodiments. It should be understood that these embodiments are described for illustrative purposes only and to help those skilled in the art understand and implement this disclosure, and do not imply any limitation on the scope of this disclosure. In addition to the methods described below, the disclosure described herein can be implemented in various other ways.

[0014] In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0015] The terms "an embodiment," "embodiment," "example embodiment," etc., used in this disclosure refer to embodiments that may include specific features, structures, or characteristics, but not every embodiment is required to include that specific feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same embodiment. Moreover, when a specific feature, structure, or characteristic is described in conjunction with an example embodiment, it is claimed that, whether explicitly described or not, such a feature, structure, or characteristic affecting its relation to other embodiments is within the knowledge of those skilled in the art.

[0016] It should be understood that although the terms “first” and “second”, etc., may be used herein to describe various elements, these elements should not be limited to these terms. These terms are used only to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without departing from the scope of the exemplary embodiments. As used herein, the term “and / or” includes any and all combinations of one or more of the listed terms.

[0017] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments. As used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising,” “including,” “having,” “containing,” and / or “comprising” as used herein indicate the presence of the said features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof.

[0018] As mentioned above, image processing technology is now being applied to all aspects of people's lives. It can provide a variety of video and game scenarios. However, traditional real-time frame interpolation technology suffers from problems such as high computational demands for motion vectors, high bandwidth consumption for depth precision, and fixed interpolation rates, which can easily lead to performance bottlenecks on mobile or low-to-mid-range devices.

[0019] Embodiments of this disclosure propose an image processing scheme. The scheme includes: determining scene description information based on at least one constructed reference frame, the scene description information indicating the complexity corresponding to the at least one reference frame; determining at least one control parameter based on the scene description information and the usage status of image processing resources, the at least one control parameter including at least one of the following: resolution of motion vectors, accuracy of depth information; determining at least one interpolation parameter based on the at least one control parameter; and performing interpolation processing based on a target reference frame in the at least one reference frame and the at least one interpolation parameter to generate an intermediate frame.

[0020] The embodiments of this disclosure can determine at least one control parameter based on scene description information and the usage status of image processing resources, and dynamically perform interpolation processing based on at least one interpolation parameter determined by the at least one control parameter. In this way, the embodiments of this disclosure can effectively improve the efficiency of generating intermediate frames.

[0021] The following section provides a detailed description of various example implementations of this scheme, with reference to the accompanying drawings.

[0022] Example Environment

[0023] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. For example... Figure 1As shown, example environment 100 may include, for example, a software development kit / plugin 110, a graphics processing unit (GPU) 120, a digital signal processor (DSP) 130, and system memory (DDR) 140.

[0024] The software development kit / plugin 110 in example environment 100 can be, for example, a developer-oriented software tool layer. Its function is to enable applications to actively adapt to and invoke the image processing methods disclosed herein. Such applications could be, for example, game applications. The software development kit / plugin 110 can provide standard interfaces and capability queries to support collaborative interpolation operations between the graphics processing unit 120 and the digital signal processor 130. The software development kit / plugin 110 can also provide rendering data and metadata to the graphics processing unit 120. This rendering data and metadata may include, for example, 3D model information, texture information, camera transformation matrix, object motion matrix, depth buffer, motion vector buffer, and rendering timing markers. After the software development kit / plugin 110 provides this data to the graphics processing unit 120, the graphics processing unit 120 can render the virtual scene in the game application based on this data.

[0025] The graphics processing unit 120 in example environment 100 can obtain rendering data and metadata from software development kit / plugin 110. After obtaining the rendering data and metadata, the graphics processing unit 120 can perform rendering processing based on the rendering data and metadata. Rendering processing may include, for example, vertex processing, rasterization, pixel shading, etc. The graphics processing unit 120 can also determine scene description information based on at least one constructed reference frame. The scene description information indicates the complexity corresponding to at least one reference frame. This complexity may include, for example, motion complexity and image complexity; motion complexity indicates motion vector information associated with at least one reference frame, and image complexity indicates at least the object distribution in at least one reference frame. After determining the scene description information, the graphics processing unit 120 can determine at least one control parameter for interpolation operations based on the scene description information and its own usage state. Further, the graphics processing unit 120 can determine at least one interpolation parameter based on such at least one control parameter. After determining at least one interpolation parameter, the graphics processing unit 120 can transmit the at least one interpolation parameter and the target reference frame to the digital signal processor 130, so that the digital signal processor 130 can perform interpolation processing. For example, the graphics processing unit 120 can transmit the target reference frame and at least one interpolation parameter to the digital signal processor 130 via direct memory access (DMA).

[0026] In example environment 100, digital signal processor 130, upon receiving a target reference frame and at least one interpolation parameter, can perform interpolation operations to construct an intermediate frame. For example, after acquiring at least one interpolation parameter, digital signal processor 130 can determine the precision of the intermediate frame to be constructed. The precision of the intermediate frame to be constructed may, for example, include motion vectors and depth information actually used to construct the intermediate frame. Digital signal processor 130 can perform interpolation operations based on such motion vectors and depth information to construct the intermediate frame.

[0027] In example environment 100, system memory 140 can be configured for data interaction between graphics processing unit 120 and digital signal processor 130. For instance, graphics processing unit 120 can transmit image data and motion vector information to digital signal processor 130 via system memory 140, enabling digital signal processor 130 to perform interpolation operations based on this data. System memory 140 can be a high-speed storage medium that provides buffering capabilities to smooth the operating rhythm between processors and prevent fluctuations in the processing speed of one from blocking the other.

[0028] The software development kit / plugin 110, graphics processing unit 120, digital signal processor 130, and system memory 140 in example environment 100 may be deployed in electronic device 150, for example. Electronic device 150 may be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, VR / AR devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof.

[0029] Some exemplary embodiments of this disclosure will be described in detail below. It should be noted that section headings are used in this document for ease of understanding and not to limit the embodiments disclosed in a section to that section. Furthermore, although some embodiments are described with reference to example scenario 100, the disclosed techniques are also applicable to other image processing techniques.

[0030] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0031] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0032] Example process

[0033] Figure 2 A flowchart of an example process 200 for image processing according to some embodiments of the present disclosure is shown. Process 200 can be implemented at an electronic device 150. Reference is made below. Figure 1 The example environment 100 shown is used to describe process 200.

[0034] refer to Figure 2 In step 210, the electronic device 150 determines scene description information based on at least one constructed reference frame. The scene description information indicates the motion complexity or image complexity corresponding to at least one reference frame. The motion complexity indicates the motion vector information associated with at least one reference frame, and the image complexity indicates at least the object distribution in at least one reference frame.

[0035] As an example, the electronic device 150 can acquire at least one constructed reference frame through the graphics processing unit 120 deployed therein. This at least one reference frame can also be referred to as at least one historical frame. The graphics processing unit 120 can reverse-engineer scene description information by analyzing the at least one reference frame. For example, the graphics processing unit can determine the scene description information using methods such as optical flow or global motion estimation. The scene description information can, for example, indicate the motion complexity or image complexity corresponding to the virtual scene. The motion complexity can, for example, indicate motion vector information associated with the at least one reference frame. For example, the more directions indicated by the motion vector information associated with the at least one reference frame, the higher the motion complexity of the at least one reference frame. The greater the degree of change in the motion vector information associated with the at least one reference frame, the higher the motion complexity of the at least one reference frame. The image complexity can, for example, indicate the object distribution in at least one reference frame. The object distribution can, for example, indicate the spatial hierarchy and position of at least one object in the image of the at least one reference frame. The image complexity can also, for example, indicate the number, type, precision, etc., of at least one object in the image of the at least one reference frame. The virtual scene can, for example, be provided by a game application (e.g., a game engine). In some scenarios, the graphics processing unit 120 can also directly obtain various parameters from the renderer to determine the scene description information.

[0036] In some embodiments, the electronic device 150 may determine a first motion parameter based on a reference motion vector corresponding to at least one reference frame. Further, the electronic device 150 may determine motion complexity based on the first motion parameter. Here, the first motion parameter includes the maximum value of the motion vector and / or the degree of change in the motion vector.

[0037] As an example, electronic device 150 can determine a corresponding reference motion vector to at least one reference frame using graphics processing unit 120. Graphics processing unit 120 can determine a first motion parameter based on the reference motion vector. This first motion parameter may, for example, include the maximum value of the motion vector and / or the degree of change of the motion vector. The degree of change of the motion vector may, for example, be referred to as the gradient of the motion vector. After determining the maximum value of the motion vector and / or the degree of change of the motion vector, graphics processing unit 120 can determine the motion complexity based on the maximum value of the motion vector and / or the degree of change.

[0038] For example, the graphics processing unit 120 can acquire the values ​​of a set of candidate motion vectors corresponding to at least one reference frame. The graphics processing unit 120 can determine the largest target motion vector from the set of candidate motion vector values ​​and use the target value of that target motion vector as the maximum value of the motion vector. The graphics processing unit 120 can also determine the degree of difference between the set of candidate motion vectors. This difference can, for example, indicate directional differences between the set of candidate motion vectors. Furthermore, the graphics processing unit 120 can also determine the degree of magnitude variation of the set of candidate motion vectors. After determining the degree of difference and magnitude variation of the set of candidate motion vectors, the graphics processing unit 120 can determine the degree of change of the motion vectors based on the aforementioned degree of difference and / or magnitude variation.

[0039] After determining the maximum value and / or degree of change of the motion vector, the graphics processing unit 120 can determine the velocity level based on the maximum value of the motion vector. The graphics processing unit 120 can also determine the disorder level based on the degree of change of the motion vector. Furthermore, the graphics processing unit 120 can determine the motion complexity based on the velocity level and / or disorder level.

[0040] In this way, embodiments of the present disclosure can determine motion complexity based on the maximum value and / or degree of change of motion vectors, thereby enabling a more accurate determination of the motion complexity of at least one object in a frame of at least one reference frame.

[0041] In some embodiments, the electronic device 150 can determine the second motion parameters based on the motion information of a virtual camera in a virtual scene. Further, the electronic device 150 can determine the motion complexity based on the first and second motion parameters.

[0042] As an example, electronic device 150 can acquire motion information of a virtual camera in a virtual scene through graphics processing unit 120. This motion information may include, for example, the rotation speed and translation speed of the virtual camera. The rotation speed indicates how fast the virtual camera rotates around its own pitch, yaw, and roll axes. This motion information can also be referred to as global motion parameters. After determining the first motion parameter (including the maximum value of the motion vector and / or the degree of change of the motion vector) and the second motion parameter (e.g., the motion information of the virtual camera), the graphics processing unit can determine the motion complexity based on the first and second motion parameters.

[0043] In this way, the embodiments of this disclosure can not only determine the motion complexity based on the first motion parameter, but also determine the motion complexity by combining the second motion parameter, thereby effectively improving the accuracy of the determined motion complexity.

[0044] In some embodiments, the electronic device 150 can determine reference depth information for at least one reference frame. Further, the electronic device 150 can perform object recognition on the at least one reference frame to obtain object detection results associated with the at least one reference frame. Finally, the electronic device 150 can determine the image complexity based on the reference depth information and the object detection results.

[0045] As an example, the electronic device 150 can acquire reference depth information of at least one reference frame through the graphics processing unit 120. This reference depth information may, for example, be depth gradient information associated with at least one reference frame. Furthermore, the graphics processing unit 120 can also perform object recognition on objects in the at least one reference frame to obtain object detection results. These object detection results may, for example, include objects corresponding to a preset type and the object attributes of those objects. These attributes may, for example, indicate the edge characteristics of the object. Based on the object detection results, the graphics processing unit 120 can determine the edge complexity associated with objects in the at least one reference frame. Further, after the edge complexity and reference depth information are determined, the graphics processing unit 120 can determine the image complexity based on the edge complexity and reference depth information using an appropriate algorithm.

[0046] In this way, embodiments of the present disclosure can more accurately determine the complexity of the image based on reference depth information and object detection results.

[0047] In some embodiments, the electronic device 150 can identify objects of a predetermined type in at least one reference frame. Objects of the predetermined type have preset edge characteristics. As an example, the graphics processing unit 120 can perform object recognition on objects in at least one reference frame to obtain corresponding object detection results. These object detection results may include, for example, objects corresponding to the predetermined type and the object attributes of those objects. The predetermined type may include, for example, periodically repeating types, non-rigid types, and transparent types. Objects corresponding to periodically repeating types may include, for example, fences, blinds, etc. Objects corresponding to non-rigid types may include, for example, grass, hair, branches, etc. Objects corresponding to transparent types may include, for example, water surfaces, glass, smoke, etc. Different types of objects correspond to different edge characteristics. These edge characteristics may include, for example, edge density and average edge strength. Edge density may indicate, for example, the percentage of edge pixels. Average edge strength may indicate, for example, the degree of brightness variation in the edge region of an object. After determining the edge density and average edge strength, the graphics processing unit 120 can determine the edge complexity based on the edge density and edge strength. Furthermore, the graphics processing unit 120 can determine the image complexity based on the edge complexity and reference depth information in order to determine at least one subsequent control parameter.

[0048] Continue to refer to Figure 2 In step 220, the electronic device 150 determines at least one control parameter based on scene description information and the usage status of image processing resources. The at least one control parameter includes at least one of the resolution of the motion vector and the accuracy of the depth information. Here, accuracy may, for example, indicate the bit width of the depth information. The usage status of the image processing resources at least indicates the utilization rate of the image processing unit.

[0049] As an example, after the scene description information is determined, the graphics processing unit 120 can determine at least one control parameter based on at least one of the scene description information and the usage status of image processing resources. For example, the graphics processing unit 120 can determine the resolution of the motion vector based on the motion complexity and the usage status of the image processing resources. The image processing resources here include the graphics processing unit 120 and / or the digital signal processor 130. The usage status of the image processing resources here can, for example, indicate the utilization rate of the graphics processing unit 120. In some scenarios, the usage status of the image processing resources here can indicate not only the utilization rate of the graphics processing unit 120 but also the utilization rate of the digital signal processor 130. The resolution of the motion vector here can, for example, indicate the sampling density or precision of the motion vector map. The precision of the depth information here can, for example, indicate the data bit width (e.g., 16 bits or 32 bits) used when storing the depth information.

[0050] In some embodiments, the electronic device 150 may determine the motion complexity and image complexity of at least one reference frame based on scene description information. Further, the electronic device 150 may determine the resolution of the motion vector from a plurality of preset resolutions based on the usage status of image processing resources, motion complexity, and image complexity.

[0051] As an example, the electronic device 150 may be configured with multiple preset resolutions. These preset resolutions may, for example, be associated with motion vectors. That is, the preset resolutions may be motion vector resolutions. The multiple preset resolutions may include, for example, full resolution, 3 / 4 resolution, and 1 / 2 resolution. After the motion complexity and image complexity are determined, the graphics processing unit 120 may select a target resolution from the multiple preset resolutions based on the motion complexity, image complexity, and the usage status of image processing resources (e.g., the utilization rate of the graphics processing unit 120), to use the target resolution as the resolution of the motion vectors.

[0052] In some scenarios, the graphics processing unit 120 can determine the motion state of at least one object in the virtual scene based on motion complexity. When the motion state indicates that the motion of at least one object is below a first preset value, and the frequency of changes in the motion direction of at least one object is below a second preset value, the graphics processing unit 120 can select a 1 / 2 resolution from multiple preset resolutions as the resolution of the motion vector. In other scenarios, when the utilization rate of the graphics processing unit 120 is high, the graphics processing unit 120 can also use a 1 / 2 resolution as the resolution of the motion vector.

[0053] Additionally or alternatively, the graphics processing unit 120 can also determine the resolution of motion vectors corresponding to different object types. For example, the graphics processing unit 120 can determine the type of at least one object in the virtual scene. When the type of at least one object matches a preset type, the graphics processing unit 120 can select full resolution as the motion vector resolution of that object. The preset type here may include, for example, a person type, a tree branch type, a grass type, a hair type, etc.

[0054] In some embodiments, the electronic device 150 may determine the image complexity of at least one reference frame based on scene description information. Further, the electronic device 150 may determine the precision of depth information from multiple preset precisions based on the usage status of image processing resources and the image complexity.

[0055] As an example, electronic device 150 can determine the image complexity of at least one reference frame based on reference depth information and edge complexity. The process of determining image complexity has been described above and will not be repeated here. After determining the image complexity, electronic device 150 can determine the precision of depth information from multiple preset precisions based on the utilization rate of graphics processing unit 120 and image complexity.

[0056] In some scenarios, when it is determined that multiple objects corresponding to preset types need to be presented in the intermediate frame, the graphics processing unit 120 can select a first precision (e.g., 32 bits) from multiple preset precisions. For example, if multiple clumps of grass, multiple tree branches, or multiple water surfaces need to be presented in the intermediate frame, the graphics processing unit 120 can determine that the image complexity of the intermediate frame is high, and a higher precision is needed to better present the image. Furthermore, the graphics processing unit 120 can select a first precision with a storage bit width of 32 bits from multiple preset precisions as the precision of the depth information.

[0057] In other scenarios, the graphics processing unit 120 can also determine low-complexity regions that need to be presented in intermediate frames. These low-complexity regions may, for example, indicate areas with a complexity below a first threshold, such as background areas, wall areas, etc. The graphics processing unit 120 can select a second precision with a storage bit width of 16 bits from multiple preset precisions as the precision of the depth information corresponding to the low-complexity regions.

[0058] In this way, embodiments of the present disclosure can dynamically select the precision of the depth information required during interpolation processing, thereby significantly reducing depth bandwidth and storage pressure.

[0059] Continue to refer to Figure 2 In step 230, the electronic device 150 determines at least one interpolation parameter based on at least one control parameter. As an example, the electronic device 150 can determine the at least one interpolation parameter based on at least one control parameter via a graphics processing unit 120 deployed therein. For example, the graphics processing unit 120 can determine the target motion vector for interpolation processing based on the resolution of the motion vector. As another example, the graphics processing unit 120 can also determine the target depth information for interpolation processing based on depth accuracy.

[0060] Continue to refer to Figure 2In step 240, the electronic device 150 performs interpolation processing based on a target reference frame in at least one reference frame and at least one interpolation parameter to generate an intermediate frame. As an example, after determining at least one interpolation parameter, the electronic device 150 can obtain the target reference frame from at least one reference frame. The target reference frame here can be, for example, a preceding frame before the intermediate frame or an image frame after the intermediate frame. Further, the electronic device 150 can perform interpolation processing based on the target reference frame and at least one interpolation parameter to generate the intermediate frame.

[0061] In some embodiments, the image processing resources include a graphics processing unit and a digital signal processor. The electronic device 150 may transmit a target reference frame and at least one interpolation parameter from the graphics processing unit to the digital signal processor. Furthermore, the electronic device 150 may utilize the digital signal processor to perform interpolation processing.

[0062] As an example, such as Figure 1 As shown, after obtaining the target reference frame and at least one interpolation parameter, the graphics processing unit 120 can transfer the target reference frame and at least one interpolation parameter to a shared memory via direct memory access (DMA). This shared memory can be, for example, system memory 140. After transferring the target reference frame and at least one interpolation parameter to the shared memory, the digital signal processor 130 can retrieve the target reference frame and at least one interpolation parameter from the shared memory (e.g., system memory 140) via DMA as the basis for subsequent interpolation operations. After obtaining the target reference frame and at least one interpolation parameter, the digital signal processor 130 can perform interpolation processing based on the target reference frame and at least one interpolation parameter to generate an intermediate frame.

[0063] In some embodiments, the electronic device 150 may determine whether to perform frame interpolation based on scene description information and the usage status of image processing resources. Further, in response to determining to perform frame interpolation, the electronic device 150 may perform interpolation processing based on a target reference frame in at least one reference frame and at least one interpolation parameter.

[0064] As an example, electronic device 150 can determine whether the current usage status of image processing resources (e.g., the utilization rate of graphics processing unit 120) has reached a second threshold. When it is determined that the utilization rate of graphics processing unit 120 has reached the second threshold, electronic device 150 can send a first instruction to digital signal processor 130. This first instruction may, for example, instruct to stop frame interpolation. After receiving the first instruction, digital signal processor 130 can stop the interpolation operation.

[0065] Furthermore, after stopping frame interpolation, the electronic device 150 can continuously monitor the utilization rate of the graphics processing unit 120. When the utilization rate of the graphics processing unit 120 is less than a third threshold for a preset period of time, the electronic device 150 can send a second instruction to the digital signal processor 130. This second instruction may, for example, instruct the restart of frame interpolation. After receiving the second instruction, the digital signal processor 130 can perform interpolation processing based on the target reference frame and at least one interpolation parameter.

[0066] In some scenarios, the electronic device 150 can also determine whether to perform frame interpolation based on scene description information. For example, when the utilization rate of the graphics processing unit 120 does not reach a second threshold, the electronic device 150 can determine whether to perform frame interpolation based on scene description information. For example, the electronic device 150 can determine whether the motion complexity and image complexity meet preset conditions. When it is determined that the motion complexity and image complexity meet the preset conditions, the electronic device 150 can determine to perform frame interpolation. The preset conditions here may, for example, indicate that the motion complexity and image complexity have not reached a fourth threshold.

[0067] In other scenarios, after determining to perform frame interpolation, the electronic device 150 can also determine the interpolation ratio based on the scene description information. The interpolation ratio, for example, can indicate how many intermediate frames to generate. After determining the interpolation ratio, the digital signal processor 130 can generate a number of intermediate frames corresponding to the interpolation ratio.

[0068] In this way, the embodiments of this disclosure can determine whether to perform frame interpolation based on scene description information and the usage status of image processing resources, thereby enabling more flexible execution of interpolation operations.

[0069] Alternatively or concurrently, the resolution of the motion vector, the precision of the depth information, and whether to perform frame interpolation can be determined collaboratively. For example, if it is determined that the resolution of the motion vector is low (e.g., half resolution), the electronic device 150 can simultaneously reduce the precision of the depth information (e.g., from 32 bits to 16 bits) and reduce the frame interpolation rate.

[0070] In some embodiments, the electronic device 150 may determine whether the performance of the image processing resource remains stable within a preset time period in response to determining that frame interpolation is to be performed. Further, the electronic device 150 may perform interpolation processing based on a target reference frame in at least one reference frame and at least one interpolation parameter in response to determining that the performance parameters of the image processing resource meet constraints within the preset time period.

[0071] As an example, after determining to perform frame interpolation, the electronic device 150 can acquire performance parameters of the image processing resources. These performance parameters may include, for example, the utilization rate of the graphics processing unit 120, the graphics queue depth, and the video memory usage. The graphics queue depth may, for example, indicate the number of rendering commands waiting to be executed in the command queue of the graphics processing unit 120. The electronic device 150 can determine whether the aforementioned performance parameters meet constraints within a preset time period. These constraints may, for example, indicate the stability of the graphics processing unit 120 within the preset time period. Taking the utilization rate of the graphics processing unit 120 as an example, the electronic device 150 can acquire the utilization rate of the graphics processing unit 120 within the preset time period. Further, the electronic device 150 can determine information such as the average value, variance, maximum value, and minimum value of this utilization rate. If all of the above information is less than a third preset value within the preset time period, the electronic device 150 can determine that the performance parameters meet the constraints within the preset time period. After determining that the performance parameters meet the constraints within the preset time period, the electronic device 150 can perform interpolation processing using the digital signal processor 130.

[0072] In this way, the embodiments of this disclosure can effectively balance image quality and image smoothness, thereby providing users with a better image effect.

[0073] The embodiments of this disclosure can determine at least one control parameter based on scene description information and the usage status of image processing resources, and dynamically perform interpolation processing based on at least one interpolation parameter determined by the at least one control parameter. In this way, the embodiments of this disclosure can effectively improve the efficiency of generating intermediate frames.

[0074] Example devices and equipment

[0075] Figure 3 A block diagram of a computing device 300 in which various embodiments of the present disclosure may be implemented is shown. The computing device 300 may be implemented as an electronic device 150, or may be included in an electronic device 150.

[0076] It should be understood that, Figure 3 The computing device 300 shown is for illustrative purposes only and is not intended to imply any limitation on the functionality and scope of the embodiments of this disclosure.

[0077] like Figure 3 As shown, computing device 300 includes general-purpose computing device 300. Computing device 300 may include at least one or more processors or processing units 310, memory 320, storage unit 330, one or more communication units 340, one or more input devices 350, and one or more output devices 360.

[0078] In some embodiments, the computing device 300 can be implemented as any user terminal or server terminal with computing capabilities. The server terminal can be a server, large computing device, etc., provided by a service provider. The user terminal can be, for example, any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, stations, units, devices, multimedia computers, multimedia tablet computers, internet nodes, communicators, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. It is conceivable that the computing device 300 can support any type of interface to the user (such as "wearable" circuitry devices, etc.).

[0079] Processing unit 310 can be a physical processor or a virtual processor, and can perform various processes based on programs stored in memory 320. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of computing device 300. Processing unit 310 may also be referred to as a central processing unit (CPU), microprocessor, controller, or microcontroller.

[0080] Computing device 300 typically includes various computer storage media. Such media can be any media accessible by computing device 300, including but not limited to volatile and non-volatile media, or removable and non-removable media. Memory 320 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (such as read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), or flash memory) or any combination thereof. Storage cell 330 can be any removable or non-removable media and can include machine-readable media, such as memory, flash drives, disks, or other media that can be used to store information and / or data and can be accessed within computing device 300.

[0081] The computing device 300 may also include additional removable / non-removable storage media, volatile / non-volatile storage media. Although in Figure 3 Not shown, but a disk drive for reading from and / or writing to a removable non-volatile disk, and an optical disc drive for reading from and / or writing to a removable non-volatile optical disc may be provided. In this case, each drive may be connected to a bus (not shown) via one or more data media interfaces.

[0082] Communication unit 340 communicates with another computing device via a communication medium. Furthermore, the functionality of the components in computing device 300 can be implemented by a single computing cluster or multiple computing machines that can communicate via communication connections. Therefore, computing device 300 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other general-purpose network nodes.

[0083] Input device 350 can be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. Output device 360 ​​can be one or more of various output devices, such as a monitor, speaker, printer, etc. With the aid of communication unit 340, computing device 300 can also communicate with one or more external devices (not shown), such as storage devices and display devices. Computing device 300 can also communicate with one or more devices that enable a user to interact with computing device 300, or, if necessary, with any device that enables computing device 300 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via an input / output (I / O) interface (not shown).

[0084] In some embodiments, some or all of the components of computing device 300 may be deployed in a cloud computing architecture, rather than being integrated into a single device. In a cloud computing architecture, components may be remotely provided and work together to achieve the functionality described herein. In some embodiments, cloud computing provides computing, software, data access, and storage services without requiring end users to know the physical location or configuration of the systems or hardware providing these services. In various embodiments, cloud computing provides services via a wide area network (WAN), such as the Internet, using suitable protocols. For example, a cloud computing provider provides applications via a WAN that can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture, along with the corresponding data, may be stored on servers at remote locations. Computing resources in a cloud computing environment may be consolidated or distributed across remote data center locations. Cloud computing infrastructure may provide services through shared data centers, although to users they appear as a single access point. Therefore, a cloud computing architecture can be used to provide the components and functionality described herein from service providers at remote locations. Alternatively, the components and functionality described herein may be provided by conventional servers or installed directly or otherwise on client devices.

[0085] In embodiments of this disclosure, computing device 300 may be used to implement image processing. Memory 320 may include one or more image processing modules 325 having one or more program instructions. These modules are accessible and executable by processing unit 310 to perform the functions of the various embodiments described herein.

[0086] In an example embodiment of image processing, the virtual scene may be processed, for example, by the image processing module 325 to generate a picture of the virtual scene. The picture of the virtual scene may be provided as output 370 via the output device 360.

[0087] While this disclosure has been specifically shown and described with reference to preferred embodiments, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of this application as defined by the appended claims. These variations are intended to be covered by the scope of this application. Therefore, the foregoing description of embodiments of this application is not intended to be limiting.

Claims

1. An image processing method, characterized in that, The method includes: Based on at least one constructed reference frame, scene description information is determined, wherein the scene description information indicates the motion complexity or image complexity corresponding to the at least one reference frame, the motion complexity indicates motion vector information associated with the at least one reference frame, and the image complexity at least indicates the object distribution in the at least one reference frame; Based on the scene description information and the usage status of image processing resources, at least one control parameter is determined, the at least one control parameter including at least one of the following: the resolution of the motion vector, the accuracy of the depth information, the accuracy indicating the bit width of the depth information, wherein the image processing resources include a graphics processing unit and a digital signal processor, and the resolution of the motion vector indicates the sampling accuracy of the motion vector image. Based on the at least one control parameter, determine at least one interpolation parameter; and Interpolation is performed based on the target reference frame in the at least one reference frame and the at least one interpolation parameter to generate an intermediate frame.

2. The method according to claim 1, characterized in that, Based on the scene description information and the usage status of image processing resources, at least one control parameter is determined, including: Based on the scene description information, determine the motion complexity and the image complexity of the at least one reference frame; and Based on the usage status of the image processing resources, the motion complexity, and the image complexity, the resolution of the motion vector is determined from a plurality of preset resolutions.

3. The method according to claim 1, characterized in that, Based on the scene description information and the usage status of image processing resources, at least one control parameter is determined, including: Based on the scene description information, the image complexity of the at least one reference frame is determined; and Based on the usage status of the image processing resources and the image complexity, the precision of the depth information is determined from multiple preset precisions.

4. The method according to claim 1, characterized in that, Based on at least one constructed reference frame, the scene description information is determined to include: Based on the reference motion vector corresponding to the at least one reference frame, a first motion parameter is determined; and The motion complexity is determined based on the first motion parameters.

5. The method according to claim 4, characterized in that, The first motion parameter includes at least one of the following: The maximum value of the motion vector, The degree of change of the motion vector.

6. The method according to claim 4, characterized in that, The at least one reference frame is associated with the virtual scene, and determining the motion complexity based on the first motion parameters includes: Based on the motion information of the virtual camera in the virtual scene, the second motion parameters are determined; and The motion complexity is determined based on the first motion parameters and the second motion parameters.

7. The method according to claim 1, characterized in that, Based on at least one constructed reference frame, the scene description information is determined to include: Determine the reference depth information of the at least one reference frame; Perform object recognition on the at least one reference frame to obtain object detection results associated with the at least one reference frame; and The image complexity is determined based on the reference depth information and the object detection results.

8. The method according to claim 7, characterized in that, Performing object identification on the at least one reference frame includes: Identify an object of a predetermined type in the at least one reference frame, the object of the predetermined type having a preset edge characteristic.

9. The method according to claim 1, characterized in that, Based on the target reference frame in the at least one reference frame and the at least one interpolation parameter, the interpolation process includes: Based on the scene description information and the usage status of the image processing resources, determine whether to perform frame interpolation; and In response to determining that interpolation should be performed, interpolation processing is performed based on the target reference frame in the at least one reference frame and the at least one interpolation parameter.

10. The method according to claim 9, characterized in that, In response to determining to perform frame interpolation, performing interpolation processing based on a target reference frame in the at least one reference frame and the at least one interpolation parameter includes: In response to determining to perform frame interpolation, it is determined whether the performance of the image processing resources remains stable within a preset time period; and In response to determining that the performance parameters of the image processing resource meet the constraints within a preset time period, interpolation processing is performed based on the target reference frame in the at least one reference frame and the at least one interpolation parameter.

11. The method according to claim 1, characterized in that, Based on the target reference frame in the at least one reference frame and the at least one interpolation parameter, performing interpolation processing to generate an intermediate frame includes: The graphics processing unit transmits the target reference frame and the at least one interpolation parameter to the digital signal processor; and The interpolation process is performed using the digital signal processor.

12. The method according to claim 1, characterized in that, The usage status of the image processing resources at least indicates the utilization rate of the graphics processing unit.

13. An apparatus for image processing, characterized in that, The apparatus includes a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 12.

14. A non-transitory computer-readable storage medium for storing instructions, characterized in that, The instructions cause the processor to execute the instructions of the method according to any one of claims 1 to 12.

Images