Image processing method, device and computer storage medium

Abstract

The present application proposes an image processing method, device and computer storage medium, the image processing method comprising: obtaining a foreground area image and a background area image of an image to be processed; obtaining a foreground level based on the foreground area image; obtaining a brightness level based on the image to be processed; performing image enhancement on the foreground area image using the foreground level and / or the brightness level; performing image filtering on the background area image using the brightness level; fusing the foreground area image after image enhancement with the background area image after image filtering to obtain a fused image and perform image encoding. In the above manner, the foreground area is enhanced according to the foreground level and the brightness level, the flexibility of video encoding is improved while the video compression rate is improved, thereby improving the transmission efficiency.

Description

Technical Field

[0001] The present application relates to the field of coding, and in particular to an image processing method, device and computer storage medium. Background Art

[0002] Video codec standards include H.264 / AVC, H.265 / HEVC, H.266 / VVC, VP8, VP9, ​​AV1, AVS, etc. Their main purpose is to compress the collected video signals into data in different standard formats for easy transmission or storage. Bit rate control, ROI, motion detection and other technologies are widely used in actual encoding.

[0003] In the prior art, the foreground area and the background area are identified, and the frame is encoded using different numbers of pixels to obtain encoding streams of different resolutions, which is inflexible and time-consuming. Summary of the invention

[0004] The present application provides an image processing method, an image processing device, and a computer storage medium.

[0005] To solve the above technical problems, the present application proposes an image processing method, which includes: obtaining a foreground area image and a background area image of an image to be processed; obtaining a foreground level based on the foreground area image; obtaining a brightness level based on the image to be processed; performing image enhancement on the foreground area image using the foreground level and / or the brightness level; performing image filtering on the background area image using the brightness level; and fusing the foreground area image after image enhancement with the background area image after image filtering to obtain a fused image and perform image encoding.

[0006] Among them, the obtaining of the foreground area image and the background area image of the image to be processed includes: performing target detection on the image to be processed to obtain the target area in the image to be processed; and obtaining the foreground area image according to the position of the target area in the image to be processed.

[0007] The obtaining of the foreground level based on the foreground area image includes: obtaining the target type in the foreground area image; and setting the foreground level of the foreground area image according to the importance of the target type.

[0008] Wherein, obtaining the brightness level based on the image to be processed includes: obtaining the average brightness value of the image to be processed; comparing the average brightness value with the brightness range of different brightness levels; setting the brightness level of the background area image according to the brightness level of the brightness range in which the average brightness value is located; wherein the average brightness value is the average brightness value of the entire frame of the image to be processed, or the regional average brightness value of the foreground area image, or the regional average brightness value of the background area image.

[0009] The performing image filtering on the background area image by using the brightness level includes: determining a filtering mode and a filtering strength according to the brightness level; and performing image filtering on the background area image according to the filtering mode and the filtering strength.

[0010] Among them, determining the filtering method and filtering strength according to the brightness level includes: inputting the brightness level into a preset function to determine the filtering level; determining the filtering method according to the noise type corresponding to the brightness level; determining the filtering strength according to the filtering level and the filtering method.

[0011] Among them, using the foreground level to perform image enhancement on the foreground area image includes: determining an enhancement method according to the brightness level; determining an enhancement level according to the foreground level; and performing image enhancement on the foreground area image according to the enhancement method and the enhancement level.

[0012] In order to solve the above technical problems, the present application proposes an image processing device, which includes an acquisition module, a processing module and a fusion module; the acquisition module is used to acquire the foreground area image and the background area image of the image to be processed; based on the foreground area image, a foreground level is acquired; based on the image to be processed, a brightness level is acquired; the processing module is used to perform image enhancement on the foreground area image using the foreground level; and image filtering on the background area image using the brightness level; the fusion module is used to fuse the foreground area image after image enhancement with the background area image after image filtering to obtain a fused image and perform image encoding.

[0013] In order to solve the above technical problems, the present application proposes an image processing device, which includes a memory and a processor coupled to the memory; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above image processing method.

[0014] In order to solve the above technical problem, the present application proposes a computer storage medium, which is used to store program data. When the program data is executed by a computer, it is used to implement the above image processing method.

[0015] Different from the prior art, the beneficial effects of the present application are as follows: the image processing device obtains the foreground area image and the background area image of the image to be processed; based on the foreground area image, obtains the foreground level; based on the image to be processed, obtains the brightness level; uses the foreground level and / or the brightness level to perform image enhancement on the foreground area image; uses the brightness level to perform image filtering on the background area image; fuses the foreground area image after image enhancement with the background area image after image filtering to obtain a fused image and perform image encoding. In the above manner, the foreground area is enhanced according to the foreground level and the brightness level, the flexibility of video encoding is improved while the video compression rate is improved, thereby improving the transmission efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 is a flowchart of a first embodiment of the image processing method provided by the present application;

[0018] Figure 2 It is a schematic diagram of the overall process of the image processing method provided by this application;

[0019] Figure 3 The image processing method provided by this application Figure 1 A schematic flow chart of sub-steps of step S13 in FIG.

[0020] Figure 4 is a flowchart of a second embodiment of the image processing method provided by the present application;

[0021] Figure 5 is a structural schematic diagram of an embodiment of an image processing device provided by the present application;

[0022] Figure 6 is a structural schematic diagram of another embodiment of the image processing device provided by the present application;

[0023] Figure 7 It is a structural diagram of an embodiment of a computer storage medium provided by the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] The image processing method of the present application is applied to an image processing device, wherein the image processing device of the present application can be a server, or a system composed of a server and a local terminal cooperating with each other. Accordingly, the various parts of the image processing device, such as various units, sub-units, modules, and sub-modules, can all be set in the server, or can be set in the server and the local terminal respectively.

[0026] Furthermore, the above-mentioned server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules used to provide distributed servers, or it can be implemented as a single software or software module, which is not specifically limited here. In some possible implementations, the image processing method of the embodiment of the present application can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0027] This application proposes an image processing method, see Figure 1-Figure 2 , Figure 1 is a flowchart of a first embodiment of the image processing method provided by the present application, Figure 2 It is a schematic diagram of the overall flow of the image processing method provided by this application.

[0028] like Figure 1 As shown, the specific steps are as follows:

[0029] Step S11: Acquire a foreground area image and a background area image of an image to be processed.

[0030] The image to be processed may be a continuous multi-frame image or a discontinuous image, which is not specifically limited in this application.

[0031] Specifically, in the embodiment of the present application, the foreground area and the background area are a pair of relative concepts. The present application uses the region of interest as the foreground area. The region of interest can be any area in the image to be processed, and the background area is the area other than the foreground area.

[0032] In one embodiment of the present application, an image processing device performs target detection on an image to be processed to obtain a target area in the image to be processed; and obtains a foreground area image according to a position of the target area in the image to be processed.

[0033] Among them, the target area is the target region of interest. Any target detection algorithm or frame difference method can be used to obtain the foreground area of ​​the current frame image. The target region of interest is a region selected from the image. This region is the focus of image analysis and attention. Circle the area for further processing, which can reduce processing time and increase accuracy.

[0034] In the embodiment of the present application, the target region of interest may be a single region or multiple regions.

[0035] Step S12: Based on the foreground area image, obtain the foreground level.

[0036] In an embodiment of the present application, different types of foreground area images correspond to different foreground levels, and an image processing device obtains the target type in the foreground area image; and sets the foreground level of the foreground area image according to the importance of the target type.

[0037] Specifically, the image content of the foreground area image is analyzed, and different foreground levels can be mapped by analyzing the image content. The specific announcement is as follows:

[0038]

[0039] in, Represents different target types, such as trees, piers, machines, etc. This application does not specifically limit the target type, which can be any type.

[0040] in, It is a positive integer ≥ 2, representing the maximum number of foreground levels.

[0041] For example, the target detection algorithm can distinguish areas of interest such as trees, fences, and machines. These areas are foreground areas, and the rest are background areas. The targets in the foreground can be mapped to the foreground level according to the importance level of the targets. is 3, the foreground detected object is a tree, and the corresponding tree is ,but .

[0042] Step S13: Based on the image to be processed, obtain the brightness level.

[0043] Specifically, the present application proposes an embodiment for determining the brightness level of an image to be processed. Figure 3 , Figure 3The image processing method provided by this application Figure 1 Schematic diagram of the sub-step flow chart of step S13 in FIG.

[0044] like Figure 3 As shown, the specific steps are as follows:

[0045] Step S131: Obtain the average brightness value of the image to be processed.

[0046] The average brightness value is the average brightness value of the entire frame of the image to be processed, or the regional average brightness value of the foreground area image, or the regional average brightness value of the background area image.

[0047] Specifically, the brightness value of the entire frame is calculated to obtain the average brightness value of the entire frame. , or calculate the brightness value of the foreground and background areas to obtain the average brightness value of the foreground or background areas .

[0048] Step S132: Compare the average brightness value with the brightness ranges of different brightness levels.

[0049] Specifically, the image processing device compares the average brightness value with the brightness ranges of different brightness levels.

[0050] Step S133: setting the brightness level of the background area image according to the brightness level of the brightness range in which the average brightness value is located.

[0051] Specifically, the image processing device obtains the brightness level according to the average brightness value of the whole frame or the average brightness value of the region. The specific announcement is as follows:

[0052]

[0053] or

[0054]

[0055] in, is a value between [0,255], It is a positive integer ≥ 2, representing the maximum number of brightness levels.

[0056] For example, suppose , Formula 2 is 60, is 100, the average brightness of the whole frame is calculated is 50, then the formula is as follows:

[0057]

[0058] Then, according to the formula is 1.

[0059] The average brightness value of the entire frame image or the average brightness value of the foreground area is mapped to a brightness level. This method can grade the brightness and is used to guide different image processing of the foreground and background areas to achieve better effects and improve the compression rate.

[0060] Step S14: performing image enhancement on the foreground area image using the foreground level and / or brightness level.

[0061] In an embodiment of the present application, the image processing device performs image enhancement on the foreground area image according to different foreground levels. In other embodiments of the present application, the image processing device performs image enhancement on the foreground area image according to different brightness levels.

[0062] In another embodiment of the present application, the image processing device performs image enhancement on the foreground area image using the foreground level and the brightness level.

[0063] It should be noted that in the embodiment of the present application, the image processing device can use the foreground level or the brightness level to perform image enhancement on the foreground area image, and can flexibly switch according to the specific application. The embodiment of the present application takes the use of the foreground level and the brightness level to enhance the foreground area image as an example for explanation.

[0064] Specifically, the image processing device determines the enhancement method according to the brightness level; determines the enhancement level according to the foreground level; and performs image enhancement on the foreground area image according to the enhancement method and the enhancement level.

[0065] Image enhancement is a technology that improves the quality of an image and highlights the edges or certain properties of the image through image processing technology. Common image enhancement techniques include sharpening, brightness enhancement, texture enhancement, histogram transformation, grayscale transformation, frequency domain filtering, etc.

[0066] Specifically, the image processing device processes the image according to the brightness level. , get the enhancement method; by foreground level and calculation functions , including but not limited to linear and nonlinear function calculation to obtain enhanced level After that, the foreground area of ​​the frame is enhanced. The specific formula is as follows:

[0067]

[0068] The image processing device selects different foreground enhancement methods according to the brightness level, and the enhancement methods include but are not limited to brightness enhancement, texture enhancement and other algorithms. The foreground enhancement level is selected according to the foreground level, and then the foreground area is processed.

[0069] For example, if the brightness level 1 indicates low brightness. At this brightness level, the foreground is enhanced using the brightness enhancement method.

[0070] set up The foreground calculation function is a linear function, and the enhancement level calculation formula is:

[0071]

[0072] Then, according to Example 1 calculate, .

[0073] The brightness enhancement corresponding to enhancement level 5 is to increase the brightness value of the foreground area of ​​the frame by +5.

[0074] If the brightness level If it is 2, which is medium brightness, the enhancement method is USM sharpening. , the corresponding USM sharpening intensity parameter value is 0.5, then the foreground area of ​​the frame is subjected to USM sharpening with an intensity parameter value of 0.5.

[0075] In the embodiment of the present application, the level of the foreground target is obtained by combining the foreground area processing method with the analysis of the foreground area, and the foreground area is enhanced in different ways and to different degrees according to the foreground level and the brightness level.

[0076] Step S15: Perform image filtering on the background area image using the brightness level.

[0077] The image filtering in the embodiments of the present application includes but is not limited to: image filtering, Gaussian filtering, median filtering, etc.

[0078] Image filtering: Suppress the noise of the target image while retaining the image details as much as possible; common filtering methods include Gaussian filtering, median filtering, morphological filtering, bilateral filtering, etc.

[0079] Gaussian filtering: Gaussian filtering is the process of performing weighted averaging of the entire image. The value of each pixel is obtained by weighted averaging the value of itself and other pixels in the neighborhood. The specific operation of Gaussian filtering is: use a template, or convolution, mask, scan each pixel in the image, and replace the value of the central pixel of the template with the weighted average gray value of the pixels in the neighborhood determined by the template. The template is determined by the mean and variance of the Gaussian filter.

[0080] Median filtering: The basic principle of median filtering is to replace the value of a point in a digital image or digital sequence with the median of the values ​​of all points in a neighborhood of the point, so that the surrounding pixel values ​​are close to the true value, thereby eliminating isolated noise points. Its template is usually 3*3, 5*5, 7*7, etc.

[0081] Specifically, the present application proposes an embodiment for performing image filtering on a background area image using brightness levels. Figure 4 , Figure 4 It is a flowchart of the second embodiment of the image processing method provided by the present application.

[0082] like Figure 4 As shown, the specific steps are as follows:

[0083] Step S21: Determine the filtering mode and filtering intensity according to the brightness level.

[0084] According to brightness level , get the filter level ,According to the brightness level, different filtering methods are obtained, and different filter intensities are selected according to the filtering level to filter the background area image.

[0085] Step S22: performing image filtering on the background area image according to the filtering mode and filtering strength.

[0086] In an embodiment of the present application, the brightness level is input into a preset function to determine the filtering level; the filtering method is determined according to the noise type corresponding to the brightness level; and the filtering intensity is determined according to the filtering level and the filtering method.

[0087] Among them, the preset function can be any linear function or unrestricted function. For example, a linear function can be used. This application only takes a preset function as an example and does not limit the specific function type.

[0088] The selection of filters includes but is not limited to Gaussian filtering, mean filtering, median filtering, etc.; the calculation function of the filtering level Including but not limited to linear and nonlinear function calculations, as follows:

[0089]

[0090] For example: If the brightness level 1 indicates low brightness. At this brightness level, noise is likely to appear, so median filtering is used.

[0091] set up It is a linear function, and the filter level calculation formula is:

[0092]

[0093] Then, according to calculate, .

[0094] The median filter kernel corresponding to the filter level 2 is 5x5, and a 5x5 median filter is performed on the background area of ​​the frame.

[0095] If the brightness level If the value is 2, which is medium brightness, Gaussian filtering can be used. At the filtering level 2, a Gaussian filtering with a mean of 1 and a variance of 1 can be used.

[0096] Step S16: Fusing the foreground area image after image enhancement with the background area image after image filtering to obtain a fused image and perform image encoding.

[0097] Since different image processing methods are performed on the foreground and background in this application, after the processing, the processed image still needs to be a complete frame of image and then sent to the encoder for encoding.

[0098] Through the above method, after different image processing is performed on the foreground and background respectively, they are fused into one image and sent to the encoder for encoding, achieving the best solution for reducing the encoding bit rate while ensuring flexible selection of the foreground area.

[0099] In order to implement the image processing method of the above embodiment, the present application also provides an image processing device. Figure 5 , Figure 5 It is a structural schematic diagram of an embodiment of an image processing device provided by the present application.

[0100] like Figure 5 As shown, the image processing device 400 includes an acquisition module 41 , a processing module 42 and a fusion module 43 .

[0101] The acquisition module 41 is used to acquire a foreground area image and a background area image of the image to be processed; acquire a foreground level based on the foreground area image; and acquire a brightness level based on the image to be processed.

[0102] The processing module 42 is used to perform image enhancement on the foreground area image using the foreground level; and perform image filtering on the background area image using the brightness level.

[0103] The fusion module 43 is used to fuse the foreground area image after image enhancement and the background area image after image filtering to obtain a fused image and perform image encoding.

[0104] In order to implement the image processing method of the above embodiment, the present application also provides an image processing device. Figure 6 , Figure 6 It is a structural schematic diagram of another embodiment of the image processing device provided by the present application.

[0105] like Figure 6As shown, the image processing apparatus 600 of this embodiment includes a processor 61 , a memory 62 , an input / output device 63 , and a bus 64 .

[0106] The processor 61 , the memory 62 , and the input / output device 63 are respectively connected to the bus 64 . The memory 62 stores a computer program, and the processor 61 is used to execute the computer program to implement the image processing method of the above embodiment.

[0107] In this embodiment, the processor 61 may also be referred to as a CPU (Central Processing Unit). The processor 61 may be an integrated circuit chip with signal processing capabilities. The processor 61 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components. The processor 61 may also be a GPU (Graphics Processing Unit), also known as a display core, a visual processor, or a display chip, which is a microprocessor that is specifically used for image computing on computers, workstations, game consoles, and some mobile devices (such as tablet computers, smart phones, etc.). The purpose of the GPU is to convert and drive the display information required by the computer system, and to provide a line scan signal to the display to control the correct display of the display. It is an important component that connects the display and the computer motherboard. As an important component of the computer host, the graphics card is responsible for outputting display graphics. The general-purpose processor may be a microprocessor or the processor 61 may also be any conventional processor, etc.

[0108] The present application also provides a computer storage medium, such as Figure 7 As shown, the computer storage medium 700 is used to store a computer program 71. When the computer program 71 is executed by a processor, it is used to implement the method described in the embodiment of the image processing method of the present application.

[0109] The method involved in the embodiment of the image processing method of the present application, when implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0110] The above description is only an implementation mode of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An image processing method, characterized in that: The image processing method comprises: Obtaining a foreground area image and a background area image of an image to be processed; Based on the foreground area image, obtaining a foreground level; Based on the image to be processed, obtaining a brightness level; Performing image enhancement on the foreground area image using the foreground level and the brightness level; Performing image filtering on the background area image using the brightness level; The foreground area image after image enhancement is fused with the background area image after image filtering to obtain a fused image and perform image encoding; The step of obtaining a foreground area image and a background area image of an image to be processed includes: Performing target detection on the image to be processed to obtain a target area in the image to be processed; Acquire the foreground area image according to the position of the target area in the image to be processed; The acquiring of a foreground level based on the foreground area image includes: Obtaining the target type in the foreground area image; According to the importance of the target type, setting the foreground level of the foreground area image; The image processing method further comprises: When it is determined based on the brightness level that the foreground area image is a low-brightness image, using the brightness level as a brightness increase value of the foreground area image, and performing image enhancement on the foreground area image according to the brightness increase value; When it is determined based on the brightness level that the foreground area image is a medium brightness image, performing image enhancement on the foreground area image according to a preset image enhancement method; When the foreground area image is determined to be a high-brightness image based on the brightness level, an intensity parameter value is determined according to the brightness level, and image enhancement is performed on the foreground area image according to the intensity parameter value.

2. The image processing method according to claim 1, characterized in that: The step of obtaining a brightness level based on the image to be processed includes: Obtaining an average brightness value of the image to be processed; comparing the average brightness value with brightness ranges of different brightness levels; Setting the brightness level of the background area image according to the brightness level of the brightness range in which the average brightness value is located; The average brightness value is the average brightness value of the entire frame of the image to be processed, or the regional average brightness value of the foreground area image, or the regional average brightness value of the background area image.

3. The image processing method according to claim 1 or 2, characterized in that: The performing image filtering on the background area image by using the brightness level comprises: According to the brightness level, determine the filtering mode and filtering intensity; The background area image is subjected to image filtering according to the filtering mode and the filtering strength.

4. The image processing method according to claim 3, characterized in that: The determining of the filtering mode and filtering intensity according to the brightness level includes: Input the brightness level into a preset function to determine the filtering level; Determining the filtering method according to the noise type corresponding to the brightness level; The filtering strength is determined according to the filtering level and the filtering method.

5. The image processing method according to claim 1, characterized in that: The step of performing image enhancement on the foreground area image by using the foreground level comprises: According to the brightness level, determining an enhancement method; Determining an enhancement level according to the foreground level; The foreground area image is enhanced according to the enhancement method and the enhancement level.

6. An image processing device, characterized in that: The image processing device comprises an acquisition module, a processing module and a fusion module; The acquisition module is used to acquire a foreground area image and a background area image of the image to be processed; acquire a foreground level based on the foreground area image; and acquire a brightness level based on the image to be processed; The processing module is used to perform image enhancement on the foreground area image using the foreground level; Performing image filtering on the background area image using the brightness level; The fusion module is used to fuse the foreground area image after image enhancement and the background area image after image filtering to obtain a fused image and perform image encoding; The acquisition module is used to perform target detection on the image to be processed to acquire the target area in the image to be processed; acquire the foreground area image according to the position of the target area in the image to be processed; and acquire the target type in the foreground area image; According to the importance of the target type, setting the foreground level of the foreground area image; The processing module is further used for: when it is determined based on the brightness level that the foreground area image is a low brightness image, using the brightness level as a brightness increase value of the foreground area image, and performing image enhancement on the foreground area image according to the brightness increase value; When it is determined based on the brightness level that the foreground area image is a medium brightness image, performing image enhancement on the foreground area image according to a preset image enhancement method; When the foreground area image is determined to be a high-brightness image based on the brightness level, an intensity parameter value is determined according to the brightness level, and image enhancement is performed on the foreground area image according to the intensity parameter value.

7. An image processing device, characterized in that: The image processing device includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the image processing method according to any one of claims 1 to 5.

8. A computer storage medium, characterized in that The computer storage medium is used to store program data, and when the program data is executed by a computer, it is used to implement the image processing method according to any one of claims 1 to 5.

Images