Image frame processing method and device, electronic equipment and storage medium

Abstract

The invention provides an image frame processing method and device, electronic equipment and a medium, and relates to the field of computers, in particular to the field of image processing. The method comprises the following steps: obtaining a to-be-processed image frame; determining a brightness value of a segmentation point based on the brightness of each pixel in the image frame; determining a brightness mapping curve based on the brightness value of the segmentation point, the brightness mapping curve being used for mapping the brightness value of each pixel in the image frame to an adjusted brightness value, the brightness mapping curve being represented as a piecewise function taking the brightness value of the segmentation point as a boundary; and processing the image frame based on the brightness mapping curve.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to image processing, and specifically to an image frame processing method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0003] The present disclosure provides an image frame processing method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0004] According to one aspect of the present disclosure, there is provided an image frame processing method, comprising: obtaining an image frame to be processed; determining a segmentation point brightness value based on the brightness of each pixel in the image frame; determining a brightness mapping curve based on the segmentation point brightness value, the brightness mapping curve being used to map the brightness value of each pixel in the image frame to an adjusted brightness value, the brightness mapping curve being characterized as a piecewise function with the segmentation point brightness value as a demarcation; and processing the image frame based on the brightness mapping curve.

[0005] According to another aspect of the present disclosure, there is provided an image frame processing device, comprising: an image frame obtaining unit, for obtaining an image frame to be processed; a segmentation point determining unit, for determining a segmentation point brightness value based on the brightness of each pixel in the image frame; a curve determining unit, for determining a brightness mapping curve based on the segmentation point brightness value, the brightness mapping curve being used to map the brightness value of each pixel in the image frame to an adjusted brightness value, the brightness mapping curve being characterized as a piecewise function with the segmentation point brightness value as a demarcation; and a processing unit, for processing the image frame based on the brightness mapping curve.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform an image frame processing method according to one or more embodiments of the present disclosure.

[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the image frame processing method according to one or more embodiments of the present disclosure.

[0008] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements the image frame processing method according to one or more embodiments of the present disclosure.

[0009] According to one or more embodiments of the present disclosure, a better brightness adjustment effect can be obtained.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0012] Figure 1 A schematic diagram showing an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0013] Figure 2 A flowchart of an image frame processing method according to an embodiment of the present disclosure is shown;

[0014] Figure 3 A structural block diagram of an image frame processing device according to an embodiment of the present disclosure is shown;

[0015] Figure 4 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0016] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0017] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0018] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0019] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0020] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.

[0021] In an embodiment of the present disclosure, the server 120 may run one or more services or software applications that enable execution of the image frame processing method according to the present disclosure.

[0022] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtualized environments and virtualized environments. In some embodiments, these services may be provided as web-based services or cloud services, such as provided to users of client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.

[0023] exist Figure 1In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0024] The user may use client devices 101, 102, 103, 104, 105 and / or 106 to process image frames, etc. The client device may provide an interface that enables a user of the client device to interact with the client device. The client device may also output information to the user via the interface. Figure 1 Only six client devices are depicted, but one skilled in the art will appreciate that the present disclosure may support any number of client devices.

[0025] Client devices 101, 102, 103, 104, 105 and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, game systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices may include cellular phones, smart phones, tablet computers, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Game systems may include various handheld game devices, Internet-enabled game devices, etc. Client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and may use various communication protocols.

[0026] The network 110 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, etc. By way of example only, the one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0027] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0028] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0029] In some implementations, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0030] In some embodiments, the server 120 may be a server of a distributed system, or a server combined with a blockchain. The server 120 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in a cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0031] The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. The databases 130 may reside in various locations. For example, the database used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The databases 130 may be of different types. In some embodiments, the databases used by the server 120 may be, for example, relational databases. One or more of these databases may store, update, and retrieve data to and from the databases in response to commands.

[0032] In some embodiments, one or more of the databases 130 may also be used by applications to store application data. The databases used by the applications may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system. Figure 1 The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0033] Reference below Figure 2 An image frame processing method 200 according to an exemplary embodiment of the present disclosure is described.

[0034] In step S201, an image frame to be processed is obtained.

[0035] In step S202, based on the brightness of each pixel in the image frame, the brightness value of the segmentation point is determined.

[0036] At step S203, a brightness mapping curve is determined based on the segmentation point brightness values, the brightness mapping curve is used to map the brightness values ​​of each pixel in the image frame to the adjusted brightness values, and the brightness mapping curve is characterized as a piecewise function with the segmentation point brightness values ​​as demarcations.

[0037] In step S204, the image frame is processed based on the brightness mapping curve.

[0038] According to the method described in the embodiment of the present disclosure, a better brightness adjustment effect can be obtained.

[0039] Exemplarily, brightness adjustment may be contrast enhancement, brightness tone change, etc., and the present disclosure is not limited thereto. According to an embodiment of the present disclosure, instead of applying a uniform mapping function to all brightness intervals, a piecewise function is used in brightness mapping to process the bright part and the dark part separately, thereby obtaining a better brightness adjustment effect.

[0040] According to some embodiments, determining the segmentation point brightness value based on the brightness of each pixel in the image frame includes: obtaining a statistical result of the brightness of each pixel in the image frame; and determining the segmentation point brightness value based on the statistical result.

[0041] In such an embodiment, the segmentation point brightness is obtained based on the statistical results of the pixel brightness in the image frame, and the segmentation point of the function is obtained based on the brightness distribution of the entire image, so that the brightness can be better enhanced in combination with the image features.

[0042] According to some embodiments, the segmentation point brightness values ​​correspond to brightness values ​​of a predetermined proportion of the brightness range related to the statistical result. In such an embodiment, points located at a certain proportion can be obtained, thereby effectively segmenting the image pixel brightness.

[0043] For example, illustratively, based on the statistical result, determining the segmentation point brightness value includes determining the minimum segmentation point brightness value that satisfies the following condition: the number of pixels in the image frame whose brightness is lower than the segmentation point brightness value is greater than or equal to a predetermined ratio of the number of all pixels in the image frame. In such an embodiment, points that are probabilistically located at a certain ratio can be obtained, thereby effectively segmenting the image pixel brightness.

[0044] According to some embodiments, the predetermined ratio is 50%. When the ratio is 50%, the brightness value of the segmentation point may be a probability median point, so that pixels below and above the probability median point can be processed differently.

[0045] According to some embodiments, the statistical result is a histogram statistical result. The histogram can effectively reflect the distribution of pixel brightness in the image frame.

[0046] According to some embodiments, the brightness mapping curve is characterized as a concave function in a portion where the brightness is lower than the segmentation point brightness value, and is characterized as a convex function in a portion where the brightness is higher than the segmentation point brightness value.

[0047] In such an embodiment, by using a concave function at a lower brightness and a convex function at a higher brightness, a function curve with relatively gentle slopes at both ends and relatively steep slopes in the middle can be obtained, thereby effectively enhancing the middle area and reducing overexposure at both ends.

[0048] According to some embodiments, the method may further include adjusting the brightness mapping curve to obtain an adjusted brightness mapping curve, and wherein processing the image frame based on the brightness mapping curve includes processing the image frame based on the adjusted brightness mapping curve.

[0049] In such an embodiment, after obtaining the brightness mapping curve based on the piecewise function, an adjustment step may be performed, such as performing local adjustment, center point adjustment or two end adjustments, to obtain a better brightness adjustment effect.

[0050] According to some embodiments, adjusting the brightness mapping curve to obtain an adjusted brightness mapping curve includes: obtaining an edge adjustment interval corresponding to a brightness edge value; and adjusting a portion of the brightness mapping curve that falls within the edge adjustment interval.

[0051] For example, by making adjustments near the edge, edge overflow can be avoided. The edge value can be an extreme value. For example, the brightness adjustment interval can have the brightness edge value as one end boundary. The brightness adjustment range includes a brightness interval with the brightness edge value as one end boundary.

[0052] According to some embodiments, obtaining an edge adjustment interval corresponding to a brightness edge value comprises: obtaining an edge mapping interval based on pixels of the image frame mapped to the brightness edge value according to the brightness mapping curve; and obtaining the edge adjustment interval based on the edge mapping interval.

[0053] Exemplarily, the pixel points of the image frame mapped to the brightness edge value according to the brightness mapping curve can be based on the number of pixel points mapped to the brightness edge value, or can be based on the number of brightnesses mapped to the brightness edge value. For example, the brightness of the image after being mapped can be obtained to determine how many pixels / brightnesses are mapped to the edge value, that is, overflow has occurred. For example, the edge mapping interval is the number of pixel points of the image frame mapped to the brightness edge value according to the brightness mapping curve. For another example, the edge mapping interval is the number of brightness values ​​of the image frame mapped to the brightness edge value according to the brightness mapping curve.

[0054] According to some embodiments, obtaining the edge adjustment interval based on the edge mapping interval includes: multiplying the edge mapping interval by an edge adjustment coefficient to obtain a size of the edge adjustment interval, wherein the edge adjustment interval has the brightness edge value as one end boundary.

[0055] In such an embodiment, the brightness adjustment interval is obtained based on the edge adjustment coefficient. For example, if a number of pixels are mapped to the maximum value 255, the number of pixels to be adjusted can be determined based on the number of pixels multiplied by the coefficient.

[0056] According to some embodiments, the edge adjustment coefficient is greater than 1. In such an example, the edge adjustment coefficient is greater than 1 so that the adjustment range is greater than the range of pixels actually affected by overflow. For example, the edge adjustment coefficient is equal to 2, 5 pixels are mapped to the maximum value of 255, and 10 pixels at the edge can be adjusted, thereby better avoiding overflow at both ends.

[0057] According to some embodiments, adjusting the portion of the brightness mapping curve that falls within the edge adjustment interval includes adjusting the slope of the portion of the brightness mapping curve that falls within the edge adjustment interval. In such an embodiment, overflow can be reduced by adjusting the slope.

[0058] Exemplarily, adjusting the slope of the portion of the brightness mapping curve falling within the edge adjustment interval includes adjusting the power of the portion of the brightness mapping curve falling within the edge adjustment interval. In such an embodiment, the slope is adjusted by adjusting the power near the edge, thereby reducing overflow.

[0059] According to some embodiments, the brightness edge value includes a brightness maximum value and a brightness minimum value, and obtaining an edge adjustment interval corresponding to the brightness edge value includes respectively obtaining an upper edge adjustment interval corresponding to the brightness maximum value and a lower edge adjustment interval corresponding to the brightness minimum value.

[0060] Such an embodiment can avoid overflow of the brightest and darkest areas by adjusting both ends.

[0061] According to some embodiments, adjusting the brightness mapping curve to obtain an adjusted brightness mapping curve includes: obtaining a segmentation point adjustment interval corresponding to the segmentation point brightness value; and adjusting a portion of the brightness mapping curve falling within the segmentation point adjustment interval.

[0062] In such an embodiment, the segmentation point of the piecewise function is adjusted so that the transition of the function is more coherent and less abrupt.

[0063] According to some embodiments, adjusting the portion of the brightness mapping curve that falls within the segmentation point adjustment interval includes adjusting the slope of the portion of the brightness mapping curve that falls within the segmentation point adjustment interval. The slope near the segmentation point of the piecewise function is adjusted to obtain a more consistent effect.

[0064] According to some embodiments, the image frame is an image frame from a video. In such embodiments, the method may further include, after processing the image frame based on the brightness mapping curve, performing video encoding on the processed image frame.

[0065] In such an embodiment, the images in the video may be processed and processed before encoding to achieve brightness / contrast adjustment of the video.

[0066] In actual video transcoding applications, most of the original network videos go through multiple encoding and decoding processes, resulting in poor video quality. In the field of video processing, subjective video quality is a dimension of video processing quality, and its main goal is to automatically evaluate the user's (i.e., the audience's) opinion on the quality of the video processed by the video processing system. In the field of image enhancement, there are many contrast enhancement algorithms, such as histogram equalization, adaptive histogram equalization with block-limited contrast, convex function mapping, and contrast enhancement algorithms based on histogram statistics. However, in the field of image enhancement, the problem of low subjective video quality after processing is also common. For example, image processing may affect the clarity, original color tone, and day / night conditions of the video, thereby affecting the subjective video quality and reducing the user's satisfaction with watching the video. For example, global histogram equalization has a greater impact on the subjective quality of the video, local histogram equalization normalization will cause color banding and striping problems at the boundaries of video blocks, convex function mapping will partially enhance the contrast in the central image area and partially weaken the contrast, and the contrast enhancement algorithm based on histogram statistics selects a convex function or a concave function according to the histogram, and cannot optimize the contrast enhancement of global pixels. Moreover, most of the above calculation processes need to be normalized to [0,1] floating-point operations, which requires a large amount of calculation and is not conducive to fast coding implementation.

[0067] According to some exemplary embodiments of the present disclosure, a method for image contrast enhancement applied to video is proposed, which can be applied to HEVC encoding, AV1 encoding, and VVC encoding, for example. Most of the existing image contrast enhancement methods are implemented based on images, and direct application to video images will result in poor subjective quality or non-compliance with the preset scene of the original video. For example, histogram equalization will greatly change the subjective contrast of an image, and a dark image (such as dusk) will be converted into a bright image (such as daytime). Local histogram equalization and cropping normalization will cause color bands and stripes to appear at the block boundaries. Selecting a convex function or a concave function based on histogram statistics cannot maximize the contrast enhancement of the central pixel area. In contrast, the image contrast enhancement method applicable to video proposed according to the exemplary embodiments of the present disclosure is applied to the original image before video encoding (HEVC, AV1, VVC) encoding, and maximizes the subjective quality of the area of ​​interest in the video without affecting the encoding complexity.

[0068] The method of the present disclosure is described below in conjunction with a specific non-limiting embodiment. Here, processing image frames in a video is used as an example, but it is understood that the present disclosure is not limited thereto.

[0069] First, a video source frame of the original video to be encoded can be obtained, which can be obtained by decoding videos in other formats.

[0070] Next, the following steps can be performed separately for the YUV components of the original video:

[0071] Perform histogram statistics on the entire frame image.

[0072] After that, an operation to determine the brightness value of the segmentation point can be carried out. For example, in an example with a ratio of 50% and taking the minimum probability, L(i) can be defined as the number of pixel points where the brightness of all pixels ≤ i, where i takes values within the possible brightness range, such as 0 - 255. Thus, the smallest MID value that satisfies the following condition can be found, such that L(MID) >= 50% × image SIZE. Where image SIZE represents the image size, and there is image SIZE = image WIDTH × image HEIGHT, where WIDTH is the image width and HEIGHT is the image height.

[0073] After that, based on the statistics of the MID value for the entire frame image, the mapping function corresponding to all pixels can be calculated. Thus, a brightness mapping curve can be obtained, such as an initial brightness mapping curve without adjustment.

[0074] The piecewise function can be characterized with the MID value as the segmentation point. For example, when the brightness value Y > MID, the brightness mapping curve is a convex function mapping, and the slope decreases as Y increases. And when the brightness value Y < MID, the brightness mapping curve is a concave function mapping, and the slope decreases as Y decreases.

[0075] As a specific non - restrictive example, it can be characterized by the following formula

[0076] Y = A × (X - MID)^2 + B × (X - MID) + C, Y > MID.

[0077] Y = -A × (X - MID)^2 + B × (X - MID) + C, Y < MID.

[0078] Where A < 0, B > 0, C > 0, X is the pixel before mapping, and Y is the result after mapping.

[0079] To achieve contrast enhancement, the slopes of the two curves can be cut off to be above 1.

[0080] The mapping curve TRANS_TABLE[i] = CLIP3(Y[i], 0, PIXEL MAX ) can be calculated according to the above formula, where i represents the original pixel, i = 0…PIXEL MAX , Y[i] is the mapping value of pixel i calculated as described above, and where PIXELMAX is the pixel maximum value. CLIP3 represents a function for mapping a range, for example, the output value = clip3(input value, minimum limit, maximum limit), and thus CLIP3(Y[i], 0, PIXEL MAX ) represents restricting the range of the mapped value to the pixel maximum value PIXEL MAX . For example, in an 8-bit image, PIXEL MAX is 255, and for example, in a 10-bit image, it is 1023.

[0081] The brightness mapping curve can be adjusted. For example, for the edge adjustment interval, such as the pixels in the end-side overflow area, convex function (Y > MID) and concave function (Y < MID) power mapping calculations can be performed respectively to obtain a new pixel mapping curve.

[0082] Specifically, as an example, the number of pixels in the upper and lower overflow areas in TRANS_TABLE[i] can be counted. Exemplarily, the pixels in the lower overflow area can be counted as the maximum value of the brightness s corresponding to TRANS_TABLE[s] = 0, where 0 represents the edge value, such as the minimum value. For example, the pixels in the lower overflow area are counted as NUM0. The pixels in the upper overflow area can be counted as the minimum value of the brightness t where TRANS_TABLE[t] = PIXEL MAX , counted as NUM MAX , where PIXEL MAX represents the edge value, such as the brightness upper limit value or the maximum value.

[0083] Thus, the adjustment ranges of the upper and lower overflow areas can be calculated respectively. For example, the adjustment range X MAX of the lower overflow area can be calculated as X MIN = NUM0 × m, and the adjustment range X MAX of the upper overflow area can be calculated as X MAX = PIXEL MAX – (PIXEL MIN - NUM MAX ) × m, where m is a value greater than 1 for appropriate expansion. For example, m can take values between 1.1 and 10, etc.

[0084] Exemplarily, the mapped brightness values Y MIN and Y MAX corresponding to the upper and lower overflow areas X MIN and X MIN can be found according to the mapping table TRANS_TABLE, in the form of Y MAX = TRANS_TABLE[X MAX , and Y

[0085] The slope of the edge adjustment interval can be adjusted. This can be achieved, for example, by adjusting the power. For example, the power coefficient n (n>1, n = 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5, etc.) can be selected. For example, within the underflow region, the corresponding pixels can be adjusted to an n-th power concave function mapping, and within the overflow region, the corresponding pixels can be adjusted to an n-th root convex function mapping, thereby achieving the adjustment of the edge adjustment interval.

[0086] Specifically, for pixels X belonging to [0, X MAX , there can be an adjusted brightness mapping curve segment Y[i] = [(i) / (X MAX )] n ×Y MAX .

[0087] For pixels X belonging to [X MIN , PIXEL MAX , there can be an adjusted brightness mapping curve segment Y[i] = Y MIN +((X - X MIN ) / (PIXEL MAX - X MIN ) 1 / n ×(PIXEL MAX - Y MIN ).

[0088] All pixel points can be calculated according to the above formula, and the mapping curve can be modified or adjusted based on the calculation results. For example, an adjusted segmented brightness mapping TRANS_TABLE[i] = CLIP3(Y[i], 0, PIXEL MAX ) can be obtained, where i = 0...PIXEL MAX , Y[i] is the mapping value of pixel i calculated as described above, and PIXEL MAX is the maximum pixel value. CLIP3 represents a function for mapping a range. For example, the output value = clip3(input value, minimum limit, maximum limit), and thus CLIP3(Y[i], 0, PIXEL MAX ) represents restricting the range of the mapping value to the maximum pixel value PIXEL MAX . For example, in an 8-bit image, PIXEL MAX is 255, and in a 10-bit image, it is 1023.

[0089] Exemplarily, the curve of pixels near the center point MID can also be adjusted. For example, the pixel mapping of the brightness value interval X where the middle |X - MID| < T can be adjusted, where T can be a constant, and exemplarily it can be 5, and the present disclosure is not limited thereto.

[0090] For example, the coefficient Z corresponding to |X - MID| = T can be calculated T , such as the slope at the segmentation point. Thus, the corresponding pixels can be recalculated.

[0091] Exemplarily, for pixels satisfying MID ≤ X < MID + T, the adjusted Y[X] = (Z T - 1) / T × (X - MID) 2 + (X - MID) + C can be obtained; and for pixels satisfying MID – T < X < MID, the adjusted Y = -(Z T - 1) / T × (X - MID) 2 + (X - MID) + C can be obtained, where C can be an adjustment offset, which can be, for example, a constant or an adjustment amount varying according to the pixel range.

[0092] All pixel points can be calculated according to the above formula to adjust or modify the mapping curve. For example, the adjusted brightness mapping curve TRANS_TABLE[i] = CLIP3(Y[i], 0, PIXEL MAX ), i = 0…PIXEL MAX can be obtained, where PIXEL MAX is the pixel maximum value, and CLIP3(Y[i], 0, PIXEL MAX ) represents restricting the range of the mapping value to the pixel maximum value PIXEL MAX , which is 255 in an 8-bit image and 1023 in a 10-bit image, for example.

[0093] The pixels of the entire image can be mapped to new pixel values through the mapping curve: PIXEL_NEW(i, j) = TRANS_TABLE[PIXEL_OLD(i, j)], where PIXEL_OLD(i, j) is the pixel value in the original image, PIXEL_NEW(i, j) is the pixel value after mapping, and i, j are coordinates.

[0094] After that, the enhanced image can be used as the input to the encoder for normal video encoding, including prediction, residual calculation, transformation, quantization, entropy encoding, inverse quantization, inverse transformation, loop filtering, to obtain the reconstructed image.

[0095] Now refer to Figure 3An image frame processing device 300 according to an embodiment of the present disclosure is described. The image frame processing device 300 may include an image frame obtaining unit 301, a segmentation point determination unit 302, a curve determination unit 303 and a processing unit 304. The image frame obtaining unit 301 is used to obtain an image frame to be processed. The segmentation point determination unit 302 may be used to determine a segmentation point brightness value based on the brightness of each pixel in the image frame. The curve determination unit 303 may be used to determine a brightness mapping curve based on the segmentation point brightness value, the brightness mapping curve being used to map the brightness value of each pixel in the image frame to an adjusted brightness value, the brightness mapping curve being characterized as a piecewise function with the segmentation point brightness value as a demarcation. The processing unit 304 may be used to process the image frame based on the brightness mapping curve.

[0096] According to the device described in the embodiment of the present disclosure, a better brightness adjustment effect can be achieved.

[0097] In the technical solution disclosed herein, the collection, acquisition, storage, use, processing, transmission, provision and public application of user personal information involved are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0098] According to an embodiment of the present disclosure, an electronic device, a readable storage medium and a computer program product are also provided.

[0099] refer to Figure 4 , a block diagram of an electronic device 400 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0100] like Figure 4As shown, the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0101] Multiple components in the electronic device 400 are connected to the I / O interface 405, including: an input unit 406, an output unit 407, a storage unit 408, and a communication unit 409. The input unit 406 can be any type of device that can input information to the electronic device 400. The input unit 406 can receive input digital or character information and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 407 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 408 can include but is not limited to a disk, an optical disk. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0102] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above, such as method 200 and its variants, etc. For example, in some embodiments, method 200 and its variants, etc. may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the method 200 and its variants, etc. described above may be executed. Alternatively, in other embodiments, the computing unit 401 may be configured to execute the method 200 and its variations, etc., in any other appropriate manner (eg, by means of firmware).

[0103] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0104] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0105] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0106] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0107] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0108] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0109] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0110] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. An image frame processing method, comprising: Obtaining an image frame to be processed; Determining a segmentation point brightness value based on the brightness of each pixel in the image frame; Determine a brightness mapping curve based on the brightness value of the segmentation point, wherein the brightness mapping curve is used to map the brightness value of each pixel in the image frame to the adjusted brightness value, and the brightness mapping curve is characterized as a piecewise function with the brightness value of the segmentation point as a demarcation; as well as The image frame is processed based on the brightness mapping curve.

2. The method according to claim 1, wherein: Based on the brightness of each pixel in the image frame, determining the brightness value of the segmentation point includes: Obtaining statistical results of the brightness of each pixel in the image frame; and Based on the statistical result, the brightness value of the segmentation point is determined.

3. The method according to claim 2, wherein: The segmentation point brightness values ​​correspond to brightness values ​​of a predetermined proportion of a brightness range related to the statistical result.

4. The method according to claim 3, wherein: The predetermined ratio is 50%.

5. The method according to any one of claims 2 to 4, wherein: The statistical result is a histogram statistical result.

6. The method according to any one of claims 1 to 5, wherein: The brightness mapping curve is characterized as a concave function in a portion where the brightness is lower than the brightness value of the division point, and is characterized as a convex function in a portion where the brightness is higher than the brightness value of the division point.

7. The method according to any one of claims 1 to 6, further comprising adjusting the brightness mapping curve to obtain an adjusted brightness mapping curve, and wherein: Processing the image frame based on the brightness mapping curve includes processing the image frame based on the adjusted brightness mapping curve.

8. The method according to claim 7, wherein: Adjusting the brightness mapping curve to obtain an adjusted brightness mapping curve includes: Obtaining an edge adjustment interval corresponding to the brightness edge value; and The portion of the brightness mapping curve that falls within the edge adjustment interval is adjusted.

9. The method according to claim 8, wherein: Obtaining the edge adjustment interval corresponding to the brightness edge value includes: Obtaining an edge mapping interval based on pixel points of the image frame that are mapped to the brightness edge value according to the brightness mapping curve; and The edge adjustment interval is obtained based on the edge mapping interval.

10. The method according to claim 8 or 9, wherein: Obtaining the edge adjustment interval based on the edge mapping interval includes: The edge mapping interval is multiplied by the edge adjustment coefficient to obtain the size of the edge adjustment interval, and the edge adjustment interval uses the brightness edge value as one end boundary.

11. The method according to claim 10, wherein: The edge adjustment coefficient is greater than 1.

12. The method according to any one of claims 8 to 11, wherein: Adjusting the portion of the brightness mapping curve that falls within the edge adjustment interval includes adjusting the slope of the portion of the brightness mapping curve that falls within the edge adjustment interval.

13. The method according to any one of claims 8 to 12, wherein: The brightness edge value includes a brightness maximum value and a brightness minimum value, and obtaining an edge adjustment interval corresponding to the brightness edge value includes respectively obtaining an upper edge adjustment interval corresponding to the brightness maximum value and a lower edge adjustment interval corresponding to the brightness minimum value.

14. The method according to any one of claims 7 to 13, wherein: Adjusting the brightness mapping curve to obtain an adjusted brightness mapping curve includes: Obtaining a segmentation point adjustment interval corresponding to the segmentation point brightness value; and The portion of the brightness mapping curve that falls within the segmentation point adjustment interval is adjusted. 15 . The method according to claim 14 , wherein adjusting the portion of the brightness mapping curve that falls within the segmentation point adjustment interval comprises adjusting the slope of the portion of the brightness mapping curve that falls within the segmentation point adjustment interval.

16. The method according to any one of claims 1 to 15, wherein: The image frame is an image frame from a video, and the method further comprises, after processing the image frame based on the brightness mapping curve, performing video encoding on the processed image frame.

17. An image frame processing device, comprising: An image frame obtaining unit, used for obtaining an image frame to be processed; A segmentation point determination unit, configured to determine a segmentation point brightness value based on the brightness of each pixel in the image frame; a curve determining unit, configured to determine a brightness mapping curve based on the brightness value of the segmentation point, wherein the brightness mapping curve is used to map the brightness value of each pixel in the image frame to an adjusted brightness value, and the brightness mapping curve is characterized as a piecewise function with the brightness value of the segmentation point as a demarcation; as well as A processing unit is used to process the image frame based on the brightness mapping curve.

18. An electronic device, comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 16.

19. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-16.

20. A computer program product comprising a computer program, wherein: The computer program implements the method according to any one of claims 1 to 16 when executed by a processor.

Images