Image Processing Method, Image Processing Apparatus, Computer Device, and Storage Medium

Abstract

An embodiment of the present application provides an image processing method, an image processing device, a computer device, and a storage medium. The method includes: determining M to-be-processed pixel objects from a target image, with N pixel objects spaced between any two adjacent to-be-processed pixel objects among the M to-be-processed pixel objects; obtaining a set of sampling points corresponding to the M to-be-processed pixel objects from the target image; where the set of sampling points includes subsets of sampling points corresponding to each to-be-processed pixel object, and there are identical sampling points between the subsets of sampling points corresponding to any two adjacent to-be-processed pixel objects; performing optimization processing on the M to-be-processed pixel objects according to the set of sampling points to obtain an optimization processing result of the M to-be-processed pixel objects. Through the embodiment of the present application, the efficiency of image optimization processing can be effectively improved, and the time consumption of image optimization processing can be reduced.

Description

Technical Field

[0001] This application relates to the field of image processing technologies, and in particular, to an image processing method, an image processing device, a computer device, and a computer-readable storage medium. Background Art

[0002] With the development of electronic technologies, shooting devices have been widely used in people's daily lives. People can take photos or videos through shooting devices, and can also conduct video calls, video conferences, video live broadcasts, etc. through shooting devices. After obtaining photos or videos through shooting devices, or when conducting video calls, video conferences, or video live broadcasts through shooting devices, how to optimize the images collected by shooting devices is a current research hotspot. Summary of the Invention

[0003] Embodiments of this application provide an image processing method, an image processing device, a computer device, and a storage medium, which can effectively improve the efficiency of image optimization processing and reduce the time consumption of image optimization processing.

[0004] On the one hand, embodiments of this application provide an image processing method, and the method includes:

[0005] Determine M to-be-processed pixel objects from a target image; wherein, there are N pixel objects between any two adjacent to-be-processed pixel objects among the M to-be-processed pixel objects;

[0006] Obtain a set of sampling points corresponding to the M to-be-processed pixel objects from the target image; wherein, the set of sampling points includes: sub-sets of sampling points corresponding to each to-be-processed pixel object, the sampling points in the sub-set of sampling points are selected from the target image according to a sampling point selection rule, and there are the same sampling points between the sub-sets of sampling points corresponding to any two adjacent to-be-processed pixel objects;

[0007] Perform optimization processing on the M to-be-processed pixel objects according to the set of sampling points to obtain the optimization processing results of the M to-be-processed pixel objects.

[0008] On the one hand, embodiments of this application provide an image processing device, and the device includes:

[0009] A determination unit, configured to determine M to-be-processed pixel objects from a target image; wherein, there are N pixel objects between any two adjacent to-be-processed pixel objects among the M to-be-processed pixel objects;

[0010] An acquisition unit, configured to acquire a set of sampling points corresponding to the M to-be-processed pixel objects from the target image; wherein, the set of sampling points includes: a subset of sampling points corresponding to each to-be-processed pixel object, the sampling points in the subset of sampling points are selected from the target image according to a sampling point selection rule, and there are identical sampling points between the subsets of sampling points corresponding to any two adjacent to-be-processed pixel objects;

[0011] A processing unit, configured to perform optimization processing on the M to-be-processed pixel objects according to the set of sampling points to obtain an optimization processing result of the M to-be-processed pixel objects.

[0012] In one embodiment, M is a positive integer greater than 1, and the processing unit is specifically configured to perform parallel optimization processing on the M to-be-processed pixel objects according to the set of sampling points through multiple threads to obtain an optimization processing result of the M to-be-processed pixel objects.

[0013] In one embodiment, the processing unit is further configured to store the set of sampling points into a target storage space; wherein, the target storage space is a storage space directly accessible by a thread for processing the target image.

[0014] In one embodiment, the processing unit is specifically configured to:

[0015] Call multiple threads to determine the to-be-processed pixel objects that each thread needs to process from the M to-be-processed pixel objects; perform parallel optimization processing on the to-be-processed pixel objects that each thread needs to process through each thread to obtain an optimization processing result of the M to-be-processed pixel objects;

[0016] Wherein, during the parallel optimization processing, any one of the multiple threads acquires a subset of sampling points corresponding to the target to-be-processed pixel object to be processed from the target storage space, and performs optimization processing on the target to-be-processed pixel object according to the acquired subset of sampling points to obtain an optimization processing result of the target to-be-processed pixel object.

[0017] In one embodiment, the sampling point selection rule includes: taking the to-be-processed pixel object as the center, equally spacing and selecting at least two pixel objects as sampling points in the horizontal direction, equally spacing and selecting at least two pixel objects as sampling points in the vertical direction, and equally spacing and selecting at least two pixel objects as sampling points in one or more diagonal directions.

[0018] In one embodiment, the processing unit is further configured to: after optimizing and processing all the to-be-processed pixel objects in the target image, generate an optimized target image according to the optimization processing results of all the to-be-processed pixel objects; perform image rendering processing on the optimized target image.

[0019] In one embodiment, the processing unit is further configured to:

[0020] During the process of optimizing the target image, obtain the next frame image relative to the target image; during the process of performing image rendering processing on the optimized target image, perform image optimization processing on the next frame image relative to the target image.

[0021] In one embodiment, the processing unit is specifically configured to:

[0022] Determine a sampling reference value through any one of the multiple threads according to the obtained subset of sampling points and the target pixel object to be processed; perform enhancement processing on the sampling reference value; according to the optimization coefficient and the enhanced sampling reference value, perform optimization processing on the target pixel object to be processed to obtain the optimized processing result of the target pixel object to be processed.

[0023] On the one hand, an embodiment of the present application provides a computer device, including: a processor and a memory, the processor and the memory are connected to each other, wherein the memory stores executable program code, and the processor is used to call the executable program code to execute the image processing method provided by the embodiment of the present application.

[0024] Correspondingly, an embodiment of the present application further provides a computer-readable storage medium, in which instructions are stored, and when it runs on a computer, it causes the computer to execute the image processing method provided by the embodiment of the present application.

[0025] Correspondingly, an embodiment of the present application further provides a computer program product or a computer program, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the image processing method provided by the embodiment of the present application.

[0026] In the embodiment of the present application, the number of sampling points selected for each pixel object to be processed is small, and the selection method of the pixel object to be processed can make a certain number of the same sampling points exist between adjacent pixel objects to be processed. In this way, not only can the total number of sampling points to be processed in one batch be effectively reduced, but also only a small number of operations (such as reading and writing) can be performed on the same sampling points, and there is no need to perform multiple identical operations on the same sampling points, so that the efficiency of image optimization processing can be effectively improved and the time-consuming of image optimization processing can be reduced. Description of the Drawings

[0027] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0028] Figure 1 It is a schematic flowchart of an image optimization processing method provided by an embodiment of the present application;

[0029] Figure 2 It shows a sampling point selection method provided by an embodiment of the present application;

[0030] Figure 3 It shows a positional relationship of multiple pixel objects processed in one batch;

[0031] Figure 4 It is a schematic flowchart of another image optimization processing method provided by an embodiment of the present application;

[0032] Figure 5 It shows another sampling point selection method provided by an embodiment of the present application;

[0033] Figure 6 It shows the overlapping situation of sampling points of two pixel objects;

[0034] Figure 7 It shows another positional relationship of multiple pixel objects processed in one batch;

[0035] Figure 8 It shows two data access methods;

[0036] Figure 9 It is a schematic architecture diagram of an image processing system provided by an embodiment of the present application;

[0037] Figure 10 It is a schematic flowchart of an image processing method provided by an embodiment of the present application;

[0038] Figure 11 It is a schematic structural diagram of an image processing device provided by an embodiment of the present application;

[0039] Figure 12 It is a schematic structural diagram of a computer device provided by an embodiment of the present application. Detailed implementation manners

[0040] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts belong to the scope of protection of the present application.

[0041] To better understand the embodiments of the present application, the following terms related to the embodiments of the present application will be introduced first:

[0042] Filtering: Filtering is an operation to filter out specific band frequencies in a signal and is an important measure to suppress and prevent interference.

[0043] GPU: Graphics Processing Unit, a graphics processor, is a microprocessor specifically designed to perform image and graphics-related operations on personal computers, workstations, game consoles, and some mobile devices (such as tablets, smartphones, etc.).

[0044] YUV: It is a color encoding method and is often used in various video processing components. When encoding photos or videos, YUV takes into account the human perception ability and allows reducing the bandwidth of chrominance.

[0045] RGB: Represents the colors of the red, green, and blue channels. This standard includes almost all the colors that the human vision can perceive and is one of the most widely used color systems.

[0046] Compute Kernel: A routine compiled for high-throughput accelerators (such as graphics processing units (GPUs), digital signal processors (DSPs), or field-programmable gate arrays (FPGAs)), separated from the main program (usually running on the central processing unit) but used by the main program.

[0047] Thread Block: A group of threads that can be executed serially or in parallel.

[0048] Shared Local Memory: An address space that can be accessed quickly by the threads within a Thread Block.

[0049] FPU: A computing unit specifically designed for performing floating-point operations.

[0050] With the development of image processing technology, after obtaining a photo or video through a shooting device, or during video calls, video conferences, or video live broadcasts using a shooting device, in order to improve the image quality, it is usually necessary to perform image optimization processing on the images collected by the shooting device. The image optimization process often optimizes pixel points. It is necessary to obtain multiple sampling points corresponding to the pixel points to be processed from the image, and optimize the pixel points to be processed based on the filtering operations of these multiple sampling points. Taking beauty processing as an example of image optimization processing, beauty is a commonly used technology for beautifying portraits. This technology often needs to be implemented based on the filtering operations of multiple sampling points to remove acne marks, fine lines, etc. to achieve the effect of beautifying the skin. Commonly used filters include Gaussian filter, bilateral filter, etc. These filtering operations need to access several pixels (i.e., pixel points) around the pixel point when processing a pixel point. The more pixels that need to be accessed, the more data bandwidth will be occupied, and the more time-consuming the image optimization process will be. And how to improve the efficiency of image optimization processing and reduce the time-consuming of image optimization processing is an urgent problem to be solved currently.

[0051] Based on this, the embodiments of the present application provide an image optimization processing method. Taking the beauty processing of images in a video as an example for illustration, the processing flow is as Figure 1 shown. Since beauty processing needs to sample in the G channel of the image, and the image format collected by video is generally in YUV format, three Compute Kernels are required, which are respectively used for the operations of converting YUV format to RGB format, beauty processing, and converting RGB format back to YUV format.

[0052] In the Compute Kernel for beauty processing, the following operations need to be performed: First, sample L (L is a positive integer, such as 20) pixels around the pixel point to be optimized and perform filtering (such as bilateral filtering) in the G channel to obtain the sampling average value, and calculate the sampling reference value according to the sampling average value. The calculation method is as shown in Formula 1:

[0053] sample g = input g - sample avg + 0.5 Formula 1

[0054] Among them, sample g is the sampling reference value, sample avg is the sampling average value, input g is the pixel value of the pixel point to be processed. The subscript rgb represents the color channel (similarly hereinafter). For example, if the subscript in Formula 1 is g, it represents the G channel.

[0055] Then, the sampled reference value is subjected to multiple (e.g., 5 times) strong light treatments. The strong light treatment involves substituting the sampled reference value into Formula 2 for calculation, and then substituting the newly calculated sampled reference value into Formula 2 for repeated calculation until the preset number of strong light treatment times is reached. Among them, Formula 2 is as follows:

[0056]

[0057] Finally, according to the beauty degree (θ), color fusion is performed with the pixel points to be processed, and the optimized pixel value of the pixel points to be processed can be obtained. The processing method can be as shown in Formulas 3 - 8:

[0058]

[0059] sample rgb1 =α*input rgb -(1-α)*sample ggg Formula 4

[0060]

[0061]

[0062]

[0063]

[0064] Among them, Formula 3 is used to calculate the reference value α based on the sampled average value sample avg and the reference value 0.3. Formula 4 is used to determine the pixel value sample g (i.e., sample ggg ) of the first optimized processing of the pixel points to be processed based on the reference value α, the sampled reference value sample rgb , and the initial pixel value input rgb1 of the pixel points to be processed. Formula 5 is used to determine the pixel value sample rgb of the second optimized processing of the pixel points to be processed based on the reference value 0.33, the initial pixel value input ggg of the pixel points to be processed, the initial G-channel pixel value input rgb1 , and sample rgb2 . Formulas 6 and 7 are similar to Formula 5. After being processed by Formulas 6 and 7, the pixel value sample rgb4 of the fourth optimized processing of the pixel points to be processed is obtained. Formula 8 is used to determine the final pixel value output rgb4 of the optimized processing of the pixel points to be processed based on the reference value 0.96 and sample rgbIt should be noted that the reference values 0.3, 0.33, 0.39, and 0.96 in Formula 3 - Formula 8 are only for illustrative purposes.

[0065] In one embodiment, the sampling point selection rule in the embodiments of the present application may include: respectively and equally spacingly selecting a plurality of pixel points as sampling points on a plurality of circumferences with different radii centered on the pixel point to be processed. In a feasible implementation manner, a plurality of pixel points are equally spacingly selected as sampling points on a circumference centered on the pixel point to be processed and with a first radius; a plurality of pixel points are equally spacingly selected as sampling points on a circumference centered on the pixel point to be processed and with a second radius; the second radius is greater than the first radius. It should be noted that the circumferences mentioned above may refer to approximate circumferences, and the equal spacing may refer to approximate equal spacing, that is, the error is within a certain range; the intervals for selecting sampling points on each circumference may be the same or different.

[0066] For example, the sampling method corresponding to this sampling point selection rule can be as Figure 2 shown, Figure 2 One square in it represents one pixel point, Figure 2 The black - filled pixel points in it are the pixel points to be processed, and the gray - filled pixel points are the sampling points corresponding to the pixel points to be processed, Figure 2 A total of 20 sampling points are included, including: 8 pixel points equally spacingly selected on the circumference with a shorter radius centered on the pixel point to be processed, and 12 pixel points equally spacingly selected on the circumference with a longer radius.

[0067] The above - mentioned processing flow is the beauty - enhancing processing method for one pixel point to be processed. For a multi - threaded ThreadBlock, parallel beauty - enhancing processing can be performed on multiple pixel points to be processed. Taking a 4 * 4 Thread Block (i.e., including 16 threads) as an example, this 4 * 4 Thread Block can perform parallel beauty - enhancing processing on a 4 * 4 pixel block, as shown by the gray pixel points in Figure 3 for example.

[0068] Although the above - mentioned image optimization processing method can, to a certain extent, improve the efficiency of image optimization processing and reduce the time consumption of image optimization processing to a certain extent, the above - mentioned image optimization processing method still has the following deficiencies:

[0069] 1. When performing filtering, a relatively large number of pixels need to be sampled around the pixel point to be processed (such as Figure 2As shown, 20 pixel points need to be sampled, which will introduce a large amount of data reading and affect the processing efficiency. Moreover, according to the distribution of the sampling points selected by the above sampling point selection rule, the sampling points used by the threads within a Thread Block to process a pixel point cannot be reused by other threads when they are processing, so it is impossible to use the address space Shared Local Memory that can be accessed at high speed by the Thread Block to accelerate operations such as sampling data reading.

[0070] 2. There are three Compute Kernels in the image optimization processing process, and each Compute Kernel needs to process all pixel points, which means that the image needs to be read and written at least three times, consuming a large amount of processing time and further reducing the processing efficiency.

[0071] 3. There are many floating-point operations in the Compute Kernel. All the calculations and the storage of intermediate results in the above image optimization (or image beautification) processing process are implemented based on the floating-point data type Float. The Float data type needs to occupy 32 bits. For an image, its data is usually of the Uint8 (8-bit unsigned integer) type and does not require such high precision. Therefore, performing floating-point operations based on the Float data type will cause waste of the FPU computing power, and saving intermediate results based on the Float data type will reduce the data reading speed.

[0072] 4. For video application scenarios such as video conferencing and video calls, the screen display needs to go through three stages: being captured by the shooting device, pre-processing the image (i.e., beautification processing), and rendering to the UI layer before it can be displayed on the interface layer. Both the shooting device capture and the rendering stage will consume a long time, and the above image optimization processing method does not consider using asynchronous processing to optimize the corresponding time.

[0073] Based on the above disadvantages, when using the above image optimization processing method, it is very difficult to control the processing time within a short duration (such as 30 ms) for images with a large size, such as 1080P, so it usually cannot meet the scenarios that require real-time processing (such as a single-frame time less than 30 ms).

[0074] To further improve the efficiency of image optimization processing and reduce the time consumed by image optimization processing, the embodiment of the present application provides another image optimization processing method. Taking the beautification processing of images in a video as an example, the processing flow is as Figure 4 shown. The following optimizations have been made to this other image optimization processing method compared to the image optimization processing method described above:

[0075] 1. A new sampling point selection rule is proposed. The new sampling point selection rule may include: taking the pixel point to be processed as the center, equally spaced selecting at least two pixel points as sampling points in the horizontal direction, equally spaced selecting at least two pixel points as sampling points in the vertical direction, and equally spaced selecting at least two pixel points as sampling points in one or more diagonal directions. In one embodiment, the one or more diagonal directions may be one or more of the diagonal directions formed by the 45-degree diagonal and the 225-degree diagonal, and the diagonal directions formed by the 135-degree diagonal and the 315-degree diagonal. It should be noted that equally spaced may mean spacing the same number of pixel points.

[0076] For example, the sampling method corresponding to the new sampling point selection rule may be as Figure 5 shown, Figure 5 where the black-filled pixel points are the pixel points to be processed, and the gray-filled pixel points are the sampling pixel points corresponding to the pixel points to be processed. Figure 5 There are a total of 12 sampling pixel points, including: 4 pixel points equally spaced in the horizontal direction (two on each side of the pixel point to be processed) with the pixel point to be processed as the center, 4 pixel points equally spaced in the vertical direction (two on each side of the pixel point to be processed) with the pixel point to be processed as the center, 2 pixel points equally spaced in the diagonal direction formed by the 45-degree diagonal and the 225-degree diagonal (one on each of the 45-degree diagonal and 225-degree diagonal of the pixel point to be processed), and 2 pixel points equally spaced in the diagonal direction formed by the 135-degree diagonal and the 315-degree diagonal (one on each of the 135-degree diagonal and 315-degree diagonal of the pixel point to be processed). Among them, equally spaced may mean spacing 4 pixel points.

[0077] The number of sampling points selected according to the new sampling point selection rule is reduced. For example, the number of sampling points to be selected in Figure 2 is reduced from 20 to only 12 in Figure 5 . Although the number of sampling points becomes smaller, since the sampling points in the horizontal, vertical, and diagonal directions with the pixel point to be processed as the center are included, the selected sampling points still have good reference value. And when selecting sampling points according to the new sampling point selection rule, there will be a certain number of reusable sampling points for pixel points to be processed that are spaced a certain number (related to the number of equally spaced pixel points in the new sampling point selection rule) of pixels apart. As Figure 6 shown, for the sampling points selected for the black pixel point to be processed on the left in Figure 6 , they are the gray pixel points marked as 1, the gray pixel points marked as 12 in Figure 6 , and the black pixel point marked in Figure 6 on the right; for the sampling points selected for the black pixel point to be processed on the right in Figure 6 , they are Figure 6The gray pixel points labeled 2, the gray pixel points labeled 12, and Figure 6 the black pixel points labeled on the left in it. Among them, there are 4 pixel points between two black pixel points to be processed, and the reused pixel points for both are Figure 6 6 gray pixel points labeled 12 in it; in addition, since the pixel points to be processed also need to be read, the reused pixel points for both also include 2 black pixel points labeled 12, and there are a total of 8 pixel points that can be reused for both.

[0078] In addition, the parallel scheme of the thread tasks within a Thread Block is modified at the same time. Among the multiple pixel points to be processed that are processed by the modified Thread Block at one time, there is a certain number (related to the number of equally spaced pixel points in the new sampling point selection rule) of pixels between any two adjacent pixel points to be processed. Similarly, taking a 4*4 ThreadBlock as an example, this 4*4 Thread Block can perform parallel beauty processing on 16 pixel points to be processed at one time. As Figure 7 shown, there are 4 pixel points between any two adjacent pixel points among these 16 pixel points to be processed (which is determined based on the Figure 6 characteristic that there are a certain number of pixel points that can be reused when selecting new pixel points to be processed at an interval of 4 pixel points shown). Through the above modification, in this way, the sampling points between at least some of the thread tasks within a Thread Block can be reused with each other.

[0079] Based on the above improvement, for a Thread Block of n*n (n is a positive integer), all the sampling points corresponding to the n*n pixel points to be processed (for the Figure 5 sampling method shown, and Figure 7 the method of selecting pixel points to be processed shown, there are a total of (n + 4)*(n + 4) - 4 sampling points) can be first allocated to each thread for pre-reading, and then all the sampling points are written into the Shared Local Memory. After that, when reading, it can be directly read from the Shared Local Memory (such as the part shown as 801 in Figure 8 ). In this way, compared with reading from the system memory (SystemDARM), it is less the two steps of reading from the system memory to the LLC and from the LLC to the L3 Cache (such as the part shown as 802 in Figure 8 ), thereby effectively improving the speed of data access.

[0080] 2. Merge Compute Kernel. In the image optimization processing method described in the previous embodiments, three Compute Kernels are required, which are respectively used for converting the YUV format to the RGB format, beauty processing, and converting the RGB format back to the YUV format. For a 1080P-sized picture, excluding data reading and writing in the algorithm, at least 37324800 (1920 * 1080 * 3 * 6) pixels need to be read and written. After the optimization described in the above point 1, the YUV data can be directly read in a Compute Kernel and then converted to RGB data and stored in the Shared Local Memory, which can avoid repeated color conversion calculations for the sampled data. Then, after beauty processing, it is converted back to the YUV format for writing. For a 1080P-sized picture, only 12441600 (1920 * 1080 * 3 * 2) pixels need to be read and written, thus reducing the data reading and writing volume by 2 / 3.

[0081] 3. Quantize floating-point calculations. Currently, most image processing chips support 16-bit floating-point calculations, or in other words, support calculations on data of the Half data type. Each FPU responsible for calculations can support 8 32-bit data or 16 16-bit data for simultaneous calculations. For images, their storage type is generally uint8. Changing to the 16-bit Half data type has almost no impact on accuracy. Therefore, all calculations and storage of intermediate results in the image optimization (or image beauty) processing process can be implemented by changing from the Float data type to the Half data type, which can improve the speed of floating-point calculations.

[0082] 4. For video application scenarios such as video conferencing and video calls, optimize the process of displaying the picture asynchronously. For the first frame of image data, after being captured by the camera, it is passed into the beauty processing module. The beauty processing module copies a copy of the image data and puts it into an asynchronous thread for processing, and then directly returns the first frame of image data. In the rendering stage, the first frame of the picture (without beauty processing) is rendered on the interface. After that, each frame of image is captured and passed into the beauty processing module. The beauty processing module will first copy the image data, and then wait until the asynchronous thread finishes processing the previous frame of image data, and then pass the new image data to the asynchronous thread and return the beauty-processed previous frame of image data to the rendering module for rendering. In this way, the time for capturing and rendering images can be used to perform beauty processing on the images, which can improve the response speed, save the waiting time between modules, and accelerate the processing efficiency. For users, there will only be an increase in the delay of one frame, and almost no perception.

[0083] Another image optimization processing method provided by the embodiments of the present application can reduce the number of sampling points to decrease the number of data reads by redesigning the sampling point selection method; by redesigning the scheme of parallel thread operation within a Thread Block, the sampling points used by a pixel to be processed can also be reused when processing other pixels, so that the sampling points can be pre-read into the Shared Local Memory and data can be read from the Shared Local Memory, which is much faster than directly reading from the memory, thus effectively reducing the time consumed in data reading; by fusing the logics of three Compute Kernels, after reading YUV data in one Compute Kernel, converting it to RGB for beauty processing, and finally converting it back to YUV format data for writing back, the data read and write volume can be greatly reduced; by implementing all calculations in the image optimization (or image beauty) processing process based on the Half type, the speed of floating-point calculation can be increased, thus reducing the time consumed by intermediate calculations; by performing beauty processing on the image data while capturing and rendering the image, part of the time consumed by beauty processing can be masked, improving the beauty processing speed. Based on the above effects, the optimized image optimization processing method can greatly improve the efficiency of image optimization processing and greatly reduce the time consumed by image optimization processing. For example, for a 1080P image, the image optimization processing can be completed in a very short time (such as less than 20 ms). After enabling the asynchronous processing flow, the single-frame image response time can even be less than 5 ms. Therefore, the optimized image optimization processing method can be applied to real-time processing scenarios.

[0084] Based on the idea of the above optimized image optimization processing method, the embodiments of the present application provide an image processing method to automatically optimize the images collected by a photographing device and effectively improve the efficiency of image optimization processing and reduce the time consumed by image optimization processing after a photo or video is obtained by shooting with the photographing device, or during video calls, video conferences, or video live broadcasts using the photographing device.

[0085] Please refer to Figure 9 , Figure 9The structure of an image processing system applicable to the image processing method provided by an embodiment of the present application is shown. The image processing system includes an image processing device 10 and a photographing device 11. The photographing device 11 is used to take photos or videos, and the image processing device 10 is used to perform image optimization processing on the images collected by the photographing device 11. Among them, the image processing device 10 can process the images collected by the photographing device 11 in real time. The photographing device 11 can be set on the image processing device 10 or exist independently of the image processing device 10. When the photographing device 11 exists independently of the image processing device 10, the image processing device can obtain the photos or videos taken by the photographing device 11 through wired or wireless means.

[0086] The image processing method provided by an embodiment of the present application can be implemented based on cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or a local area network to achieve data calculation, storage, processing, and sharing. The image processing device 10 can be a server or a terminal with image processing capabilities, and the photographing device 11 can be a terminal with photographing capabilities. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto.

[0087] Please refer to Figure 10 , which is a schematic flowchart of an image processing method provided by an embodiment of the present application. The image processing method described in the embodiment of the present application can be applied to Figure 9 the image processing device in the image processing system shown, and the image processing method includes but is not limited to the following steps:

[0088] S101. Determine M to-be-processed pixel objects from a target image; wherein, there are N pixel objects between any two adjacent to-be-processed pixel objects among the M to-be-processed pixel objects. M and N are positive integers.

[0089] In the embodiments of the present application, the target image may be a single photo captured by a photographing device, or a frame image in a video captured by the photographing device. In one embodiment, the image processing device may directly obtain the image collected during the photographing or video shooting process from the photographing device. In this case, the photographing device may be provided on the image processing device, and the image processing device obtains the image collected by the photographing device through an internal communication connection; or the photographing device exists independently of the image processing device, and the image processing device obtains the image collected by the photographing device through an external communication connection with the photographing device. In another embodiment, it may also be that the photographing device sends the captured photo or video to a certain storage device (such as a storage server), and the image processing device obtains the photo or video captured by the photographing device from the storage device. It may also be that the photographing device sends the captured photo or video to a certain intermediate device (such as a forwarding server), and the intermediate device forwards the photo or video sent by the photographing device to the image processing device.

[0090] In the embodiments of the present application, the pixel object may include one or more pixel points. In one embodiment, when the pixel object includes multiple pixel points, a pixel block of C*K (C and K are positive integers) may be used as a pixel object. C and K may be equal. For example, a 2*2 pixel block is used as a pixel object.

[0091] The number M of pixel objects to be processed is determined according to the number of threads that can serially or parallelly process the pixel objects to be processed. For example, if the image optimization processing is performed on an n*n Thread Block, the number M of pixel objects to be processed in parallel is less than or equal to n*n.

[0092] Adjacent pixel objects to be processed may refer to pixel objects to be processed that are adjacent in the horizontal, vertical, or diagonal directions, that is, there are no other pixel objects to be processed between two pixel objects to be processed in a certain direction. The number N of pixel objects between adjacent pixel objects to be processed is determined according to the number of pixel objects between the sampled pixel objects indicated by the sampling point selection rule and between the pixel object to be processed and the adjacent sampled pixel object. For example, if the pixel object is a single pixel point, and the sampling point selection rule indicates that there are 4 pixel points between the sampled pixel points and between the pixel point to be processed and the adjacent sampled pixel point, then the number N of pixel points between adjacent pixel points to be processed is 4. For example, if the sampling points are selected according to the Figure 5 sampling method shown, the distribution of the M pixel objects to be processed in parallel is as shown in Figure 7 shown. For specific reference, please refer to the previous description and will not be elaborated here.

[0093] S102. Obtain a set of sampling points corresponding to the M to-be-processed pixel objects from the target image; wherein, the set of sampling points includes: a subset of sampling points corresponding to each to-be-processed pixel object, and the sampling points in the subset of sampling points are selected from the target image according to a sampling point selection rule, and there are the same sampling points between the subsets of sampling points corresponding to any two adjacent to-be-processed pixel objects.

[0094] In the embodiment of the present application, the set of sampling points includes all the sampling points corresponding to each to-be-processed pixel object, that is, the subset of sampling points corresponding to each to-be-processed pixel object is a subset of the set of sampling points. All the sampling points corresponding to each to-be-processed pixel object are selected from the target image according to the sampling point selection rule.

[0095] In one embodiment, the sampling point selection rule may include: taking the to-be-processed pixel object as the center, equally spacing and selecting at least two pixel objects as sampling points in the horizontal direction, equally spacing and selecting at least two pixel objects as sampling points in the vertical direction, and equally spacing and selecting at least two pixel objects as sampling points in one or more diagonal directions. In one implementation manner, the one or more diagonal directions may be one or more of the diagonal directions formed by the 45-degree diagonal and the 225-degree diagonal, and the diagonal directions formed by the 135-degree diagonal and the 315-degree diagonal. It should be noted that equally spacing may mean spacing the same number of pixel objects. The set intervals of the sampling points set in each direction may be the same or different.

[0096] For example, the sampling method corresponding to the sampling point selection rule may be as Figure 5 shown, Figure 5 wherein a square represents a pixel object, which may be a pixel point or a pixel block. Figure 5 The black-filled squares therein are the to-be-processed pixel objects, and the gray-filled squares are the sampling points corresponding to the to-be-processed pixel objects. Figure 5 There are a total of 12 sampling points in it, including: taking the to-be-processed pixel object as the center, 4 pixel objects equally spaced in the horizontal direction (two on each side of the to-be-processed pixel object), 4 pixel objects equally spaced in the vertical direction (two on each of the top and bottom of the to-be-processed pixel object), 2 pixel objects equally spaced in the diagonal direction formed by the 45-degree diagonal and the 225-degree diagonal (one on each of the 45-degree diagonal and the 225-degree diagonal of the to-be-processed pixel object), and 2 pixel objects equally spaced in the diagonal direction formed by the 135-degree diagonal and the 315-degree diagonal (one on each of the 135-degree diagonal and the 315-degree diagonal of the to-be-processed pixel object). Among them, equally spacing means spacing 4 pixel objects.

[0097] The number of sampling points selected for the to-be-processed pixel object according to the sampling point selection rule is small. For example, asFigure 5 Only 12 sampling points are selected as shown. Although the number of sampling points is small, since the sampling points include those in the horizontal, vertical, and diagonal directions centered on the pixel object to be processed, the selected sampling points still have good reference value. And when selecting sampling points according to this sampling point selection rule, there will be a certain number of reusable sampling points between the pixel objects to be processed that are spaced apart by a certain number (this number is related to the number of pixel objects between the sampling pixel objects indicated by the sampling point selection rule and between the pixel object to be processed and the adjacent sampling pixel objects). As Figure 6 shown, for Figure 6 the sampling points selected for the black pixel object to be processed on the left in Figure 6 are the gray pixel objects marked as 1, the gray pixel objects marked as 12, and Figure 6 the black pixel object marked on the right in Figure 6 ; for Figure 6 the sampling points selected for the black pixel object to be processed on the right in Figure 6 are the gray pixel objects marked as 2, the gray pixel objects marked as 12, and Figure 5 the black pixel object marked on the left in

[0098] Among them, the two black pixel objects to be processed are separated by 4 pixel objects, and the reusable sampling points for the two are Figure 5 the 6 gray pixel objects marked as 12; in addition, since the pixel object to be processed also needs to be read during the optimization process, the reusable pixel objects for the two also include 2 black pixel objects marked as 12, and there are a total of 8 sampling points that can be reused between the two.

[0098] In addition, after selecting sampling points according to this sampling point selection rule, there are a certain number of identical sampling points between any two adjacent pixel objects to be processed, that is, the above-mentioned reusable sampling points; for two non-adjacent pixel objects to be processed, there may or may not be identical sampling points; and the closer the distance between two pixel objects to be processed, the more identical sampling points there are, and the farther the distance between two pixel objects to be processed, the fewer identical sampling points there are.

[0099] In one embodiment, based on the feature that there are identical (i.e., reusable) sampling points among at least some of the M to-be-processed pixel objects when selecting sampling points for the to-be-processed pixel objects according to the sampling point selection rule, all the sampling points corresponding to the M to-be-processed pixel objects (i.e., the sampling point set described above) can be stored in a target storage space directly accessible by the threads for processing the target image (or the to-be-processed pixel objects). Then, when each thread performs optimization processing on the corresponding to-be-processed pixel object, it can quickly obtain the corresponding sampling points from the target storage space for processing, which can effectively improve the speed of data access. For example, when using a ThreadBlock to perform optimization processing on the M to-be-processed pixel objects, the sampling point set corresponding to the M to-be-processed pixel objects can be stored in the address space Shared Local Memory that can be accessed at high speed by the Thread Block. Then, when reading, it can be directly read from the Shared Local Memory (as shown in the part of 801 in Figure 8 ). In this way, compared with reading from the system memory (SystemDARM), two steps of reading from the system memory to the LLC and from the LLC to the L3 Cache are omitted (as shown in the part of 802 in Figure 8 ), thereby effectively improving the speed of data access.

[0100] S103. Optimize the M to-be-processed pixel objects according to the sampling point set to obtain the optimization processing results of the M to-be-processed pixel objects.

[0101] In the embodiments of the present application, the M to-be-processed pixel objects may be serially or parallelly optimized according to the sampling point set to obtain the optimization processing results of each to-be-processed pixel object, and the optimization processing results include the pixel values of the to-be-processed pixel objects after optimization. The optimization processing may be beauty processing.

[0102] In one embodiment, when there are multiple to-be-processed pixel objects, multiple threads may perform parallel optimization processing on the M to-be-processed pixel objects according to the sampling point set to obtain the optimization processing results of each to-be-processed pixel object. Parallel processing can improve the optimization processing speed and reduce the time-consuming of the optimization processing.

[0103] In one embodiment, the manner of obtaining the optimized processing results of each pixel object to be processed by parallelly optimizing and processing M pixel objects to be processed according to the set of sampling points by multiple threads may be as follows: call multiple threads to determine the pixel objects to be processed that each thread needs to process from the M pixel objects to be processed; parallelly optimize and process the pixel objects to be processed that each thread needs to process by each thread to obtain the optimized processing results of each pixel object to be processed; wherein, during the parallel optimization and processing, any one of the multiple threads quickly obtains the subset of sampling points corresponding to the target pixel object to be processed that it needs to process from the above-mentioned target storage space, and optimizes and processes the target pixel object to be processed according to the obtained subset of sampling points to obtain the optimized processing result of the target pixel object to be processed.

[0104] For example, it may be that a Thread Block parallelly optimizes and processes M pixel objects to be processed according to the set of sampling points. Each thread in the Thread Block obtains the subset of sampling points corresponding to the target pixel object to be processed that it needs to process from the address space SharedLocal Memory that it can access at high speed, and optimizes and processes the target pixel object to be processed according to the obtained subset of sampling points to obtain the optimized processing result of the target pixel object to be processed. In this way, the thread can quickly obtain the data to be processed, which is beneficial to reducing the time consumption of the optimization and processing process.

[0105] In one embodiment, the manner of obtaining the optimized processing result of the target pixel object to be processed by any one of the multiple threads according to the obtained subset of sampling points may be as follows: any one of the multiple threads determines the sampling average value according to the subset of sampling points corresponding to the target pixel object to be processed that it obtains, determines the sampling reference value according to the sampling average value and the pixel value of the target pixel object (when the pixel object is a pixel block, its pixel value may be the average pixel value of each pixel point in the pixel block), and the calculation method may adopt Formula 1 described above; then perform enhancement processing on the sampling reference value, including performing multiple strong light processing, and the manner of strong light processing may adopt Formula 2 described above; finally, optimize and process the target pixel object to be processed according to the optimization coefficient (such as the beauty degree θ described above) and the enhanced sampling reference value to obtain the optimized processing result of the target pixel object to be processed, and this process may be implemented by using Formulas 3-8 described above. It should be noted that when the optimization and processing is beauty processing, the optimization and processing manner may refer to the relevant descriptions in the foregoing embodiments and will not be elaborated here.

[0106] In the embodiments of the present application, after the optimization processing of the M to-be-processed pixel objects is completed, M to-be-processed pixel objects are reselected from the target image according to the to-be-processed pixel object selection method described above, and steps S102 and S103 are repeatedly executed based on the reselected M to-be-processed pixel objects until all the to-be-processed pixel objects in the target image (such as all pixel points, or all pixel points in the area where the person image is located) are optimized. The optimized target image can be generated according to the optimization processing results of all the to-be-processed pixel objects. For application scenarios where the optimized target image does not need to be immediately displayed, the optimized target image can be saved, while for application scenarios such as video conferencing and video calls that require immediate display of the optimized target image, the optimized target image needs to be further subjected to image rendering processing for display on the display screen.

[0107] In one embodiment, for video application scenarios such as video conferencing and video calls, it is necessary to optimize each frame of the image in the video stream. The image frame can be optimized during the process of collecting the image frame and rendering the optimized image, which can save the waiting time for each step and speed up the optimization and rendering of the image frame. Based on this, the process of optimizing the target image in steps S101 to S103 can be executed during the process of rendering the previous frame image (which can be optimized) relative to the target image in the video stream, and the target image can be collected during the process of optimizing the previous frame image. Similarly, during the process of optimizing the target image, the next frame image relative to the target image in the video stream is obtained; during the process of rendering the optimized target image, the next frame image is subjected to image optimization processing.

[0108] In one embodiment, all calculations and storage of intermediate results during the image optimization process can be implemented based on the Half data type, which can improve the speed of floating-point calculations during the image optimization process and can also improve the speed of data storage and access, both of which are beneficial to reducing the time-consuming of the image optimization process.

[0109] It should be noted that the specific implementation methods of some steps in the above image processing method can refer to the relevant descriptions in the two image optimization methods provided in the previous embodiments, and will not be elaborated here.

[0110] In the embodiments of the present application, the number of sampling points selected for each pixel object to be processed is small, and the selection method of the pixel object to be processed can ensure that there are a certain number of identical sampling points between adjacent pixel objects to be processed. In this way, not only can the total number of sampling points to be processed in one batch be effectively reduced, but also only a small number of operations (such as reading and writing) need to be performed on the identical sampling points, without performing multiple identical operations on the same sampling points. In addition, sampling data can be quickly obtained from a specific storage space for processing, and the data calculation speed can be accelerated based on the Half data type. In addition, image optimization processing can be performed while collecting images and rendering images, thereby accelerating the speed of image optimization processing. Based on the above effects, the image processing method provided by the embodiments of the present application can effectively improve the efficiency of image optimization processing and reduce the time-consuming of image optimization processing.

[0111] It should be noted that the execution subject for executing each step in the above method embodiments may be composed of hardware, software, or a combination of hardware and software.

[0112] Please refer to Figure 11 , which is a schematic structural diagram of an image processing device provided by the embodiments of the present application. The image processing device described in the embodiments of the present application corresponds to the image processing device described above. The device includes:

[0113] A determination unit 111, configured to determine M pixel objects to be processed from a target image; wherein, there are N pixel objects between any two adjacent pixel objects to be processed among the M pixel objects to be processed;

[0114] An acquisition unit 112, configured to acquire a set of sampling points corresponding to the M pixel objects to be processed from the target image; wherein, the set of sampling points includes: subsets of sampling points corresponding to each pixel object to be processed, and the sampling points in the subset of sampling points are selected from the target image according to a sampling point selection rule, and there are identical sampling points between the subsets of sampling points corresponding to any two adjacent pixel objects to be processed;

[0115] A processing unit 113, configured to perform optimization processing on the M pixel objects to be processed according to the set of sampling points to obtain an optimization processing result of the M pixel objects to be processed.

[0116] In an embodiment, M is a positive integer greater than 1. The processing unit 113 is specifically configured to perform parallel optimization processing on the M pixel objects to be processed according to the set of sampling points through multiple threads to obtain an optimization processing result of the M pixel objects to be processed.

[0117] In one embodiment, the processing unit 113 is further configured to store the set of sampling points into a target storage space, where the target storage space is a storage space directly accessible by a thread for processing the target image.

[0118] In one embodiment, the processing unit 113 is specifically configured to:

[0119] Call multiple threads to determine, from the M to-be-processed pixel objects, the to-be-processed pixel objects that each thread needs to process; and parallelly optimize and process the to-be-processed pixel objects that each thread needs to process through each thread to obtain the optimized processing results of the M to-be-processed pixel objects.

[0120] Wherein, during the parallel optimization processing, any one of the multiple threads obtains a subset of sampling points corresponding to the target to-be-processed pixel object to be processed from the target storage space, and optimizes and processes the target to-be-processed pixel object according to the obtained subset of sampling points to obtain the optimized processing result of the target to-be-processed pixel object.

[0121] In one embodiment, the sampling point selection rule includes: taking the to-be-processed pixel object as the center, equally spacing and selecting at least two pixel objects as sampling points in the horizontal direction, equally spacing and selecting at least two pixel objects as sampling points in the vertical direction, and equally spacing and selecting at least two pixel objects as sampling points in one or more diagonal directions.

[0122] In one embodiment, the processing unit 113 is further configured to: after optimizing and processing all the to-be-processed pixel objects in the target image, generate an optimized target image according to the optimized processing results of all the to-be-processed pixel objects; and perform image rendering processing on the optimized target image.

[0123] In one embodiment, the processing unit 113 is further configured to:

[0124] During the process of optimizing the target image, obtain the next frame of image relative to the target image; and during the process of performing image rendering processing on the optimized target image, perform image optimization processing on the next frame of image relative to the target image.

[0125] In one embodiment, the processing unit 113 is specifically configured to:

[0126] Determine a sampling reference value according to a subset of acquired sampling points and the target pixel object to be processed by any one of the multiple threads; perform enhancement processing on the sampling reference value; and perform optimization processing on the target pixel object to be processed according to an optimization coefficient and the enhanced sampling reference value to obtain an optimized processing result of the target pixel object to be processed.

[0127] It can be understood that the functions of the functional units of the image processing apparatus according to the embodiments of the present application can be specifically implemented according to the methods in the above method embodiments, and the specific implementation process can refer to the relevant descriptions in the above method embodiments, which will not be elaborated here.

[0128] In a feasible embodiment, the image processing apparatus provided by the embodiments of the present application can be implemented in a software manner. The image processing apparatus can be stored in a memory, and it can be software in the form of a program and a plug-in, etc., and includes a series of units, including a determination unit, an acquisition unit, and a processing unit; wherein, the determination unit, the acquisition unit, and the processing unit are used to implement the image processing method provided by the embodiments of the present application.

[0129] In other feasible embodiments, the image processing apparatus provided by the embodiments of the present application can also be implemented in a combination of software and hardware. As an example, the image processing apparatus provided by the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the image processing method provided by the embodiments of the present application. For example, a processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic components.

[0130] In the embodiments of the present application, the number of sampling points selected for each pixel object to be processed is small, and the selection method of the pixel object to be processed can make a certain number of the same sampling points exist between adjacent pixel objects to be processed. In this way, not only can the total number of sampling points to be processed in one batch be effectively reduced, but also only a small number of operations (such as reading and writing) can be performed on the same sampling points, and there is no need to perform multiple identical operations on the same sampling points, thereby effectively improving the efficiency of image optimization processing and reducing the time-consuming of image optimization processing.

[0131] Please refer to Figure 12, which is a schematic structural diagram of a computer device provided by an embodiment of the present application. The computer device described in the embodiment of the present application includes: a processor 121, a communication interface 122, and a memory 123. Among them, the processor 121, the communication interface 122, and the memory 123 can be connected through a bus or other means. In the embodiment of the present application, the connection through the bus is taken as an example.

[0132] Among them, the processor 121 (or CPU (Central Processing Unit)) is the computing core and control core of the computer device. It can parse various instructions in the computer device and process various data of the computer device. For example, the CPU can be used to parse the power-on and power-off instructions sent by the user to the computer device and control the computer device to perform power-on and power-off operations; Another example: The CPU can transmit various interactive data between the internal structures of the computer device, and so on. The communication interface 122 can be an internal communication interface of the computer device or an external communication interface of the computer device. The external communication interface can optionally include standard wired interfaces, wireless interfaces (such as Wi-Fi, mobile communication interfaces, etc.), and is controlled by the processor 121 to send and receive data. The memory 123 (Memory) is a memory device in the computer device, used to store programs and data. It can be understood that the memory 123 here can include both the built-in memory of the computer device and, of course, the extended memory supported by the computer device. The memory 123 provides a storage space, and this storage space stores the operating system of the computer device, which can include but is not limited to: Android system, iOS system, Windows Phone system, etc. The present application does not make any limitations in this regard.

[0133] In the embodiment of the present application, the processor 121 performs the following operations by running the executable program code in the memory 123:

[0134] Determine M to-be-processed pixel objects from the target image; where any two adjacent to-be-processed pixel objects among the M to-be-processed pixel objects are separated by N pixel objects; Obtain a set of sampling points corresponding to the M to-be-processed pixel objects from the target image; where the set of sampling points includes: a subset of sampling points corresponding to each to-be-processed pixel object, and the sampling points in the subset of sampling points are selected from the target image according to the sampling point selection rule, and there are identical sampling points between the subsets of sampling points corresponding to any two adjacent to-be-processed pixel objects; Optimize the M to-be-processed pixel objects according to the set of sampling points to obtain the optimized processing result of the M to-be-processed pixel objects.

[0135] In one embodiment, M is a positive integer greater than 1. When the processor 121 performs optimization processing on the M pixel objects to be processed according to the set of sampling points and obtains the optimization processing results of the M pixel objects to be processed, it is specifically configured to: perform parallel optimization processing on the M pixel objects to be processed according to the set of sampling points through multiple threads, and obtain the optimization processing results of the M pixel objects to be processed.

[0136] In one embodiment, the processor 121 is further configured to: store the set of sampling points in a target storage space; wherein, the target storage space is a storage space directly accessible by the threads for processing the target image.

[0137] In one embodiment, when the processor 121 performs parallel optimization processing on the M pixel objects to be processed according to the set of sampling points and obtains the optimization processing results of the M pixel objects to be processed, it is specifically configured to:

[0138] Call multiple threads to determine the pixel objects to be processed that each thread needs to process from the M pixel objects to be processed; perform parallel optimization processing on the pixel objects to be processed that each thread needs to process through each thread, and obtain the optimization processing results of the M pixel objects to be processed;

[0139] Wherein, during the parallel optimization processing, any one of the multiple threads obtains a subset of sampling points corresponding to the target pixel object to be processed from the target storage space, and performs optimization processing on the target pixel object to be processed according to the obtained subset of sampling points, and obtains the optimization processing result of the target pixel object to be processed.

[0140] In one embodiment, the sampling point selection rule includes: taking the pixel object to be processed as the center, selecting at least two pixel objects at equal intervals in the horizontal direction as sampling points, selecting at least two pixel objects at equal intervals in the vertical direction as sampling points, and selecting at least two pixel objects at equal intervals in one or more diagonal directions as sampling points.

[0141] In one embodiment, the processor 121 is further configured to: after optimizing all the pixel objects to be processed in the target image, generate an optimized target image according to the optimization processing results of all the pixel objects to be processed; perform image rendering processing on the optimized target image.

[0142] In one embodiment, the processor 121 is further configured to: during the process of optimizing the target image, obtain the next frame of image relative to the target image; during the process of performing image rendering processing on the optimized target image, perform image optimization processing on the next frame of image relative to the target image.

[0143] In one embodiment, when the processor 121 optimizes the target pixel object to be processed according to a subset of the acquired sampling points through any one of the multiple threads and obtains the optimization result of the target pixel object to be processed, it is specifically configured to:

[0144] Determine a sampling reference value according to a subset of the acquired sampling points and the target pixel object to be processed through any one of the multiple threads; perform enhancement processing on the sampling reference value; and optimize the target pixel object to be processed according to the optimization coefficient and the enhanced sampling reference value to obtain the optimization result of the target pixel object to be processed.

[0145] In a specific implementation, the processor 121, the communication interface 122, and the memory 123 described in the embodiments of the present application may execute the implementation manners of the image processing device described in the image processing method or the image processing manner provided in the embodiments of the present application, or may also execute the implementation manners described in the image processing apparatus provided in the embodiments of the present application, which will not be elaborated herein.

[0146] In the embodiments of the present application, the number of sampling points selected for each pixel object to be processed is small, and the selection manner of the pixel object to be processed can make a certain number of the same sampling points exist between adjacent pixel objects to be processed. In this way, not only can the total number of sampling points to be processed in one batch be effectively reduced, but also only a small number of operations (such as reading and writing) need to be performed on the same sampling points, without performing multiple identical operations on the same sampling points, thereby effectively improving the efficiency of image optimization processing and reducing the time consumption of image optimization processing.

[0147] The embodiments of the present application further provide a computer-readable storage medium, in which a computer program is stored. When the computer program runs on a computer, the computer is enabled to execute the image processing method or the image optimization processing manner as described in the embodiments of the present application. Its specific implementation manner may refer to the foregoing description and will not be elaborated herein.

[0148] The embodiments of the present application further provide a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the image processing method or the image optimization processing method as described in the embodiments of the present application. Its specific implementation manner may refer to the foregoing description and will not be elaborated herein.

[0149] It should be noted that, for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, some steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0150] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program, and this program can be stored in a computer-readable storage medium. The storage medium can include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc, etc.

[0151] The foregoing disclosure is only a part of the embodiments of this application. Of course, it cannot be used to limit the scope of the rights of this application. Therefore, equivalent changes made according to the claims of this application still fall within the scope covered by this application.

Claims

1. An image processing method, characterized in that, the method includes: determining M to-be-processed pixel objects from a target image; wherein, there are N pixel objects between any two adjacent to-be-processed pixel objects among the M to-be-processed pixel objects; obtaining a set of sampling points corresponding to the M to-be-processed pixel objects from the target image; wherein, the set of sampling points includes: a subset of sampling points corresponding to each to-be-processed pixel object, the sampling points in the subset of sampling points are selected from the target image according to a sampling point selection rule, and there are the same sampling points between the subsets of sampling points corresponding to any two adjacent to-be-processed pixel objects; the sampling point selection rule includes: taking the to-be-processed pixel object as the center, equally spacing and selecting at least two pixel objects as sampling points in the horizontal direction, equally spacing and selecting at least two pixel objects as sampling points in the vertical direction, and equally spacing and selecting at least two pixel objects as sampling points in one or more diagonal directions; invoking multiple threads to determine the to-be-processed pixel objects that each thread needs to process from the M to-be-processed pixel objects; parallelly optimizing and processing the to-be-processed pixel objects that each thread needs to process through each thread to obtain an optimized processing result of the M to-be-processed pixel objects; wherein, during the parallel optimization and processing, through any one of the multiple threads, obtaining a subset of sampling points corresponding to the target to-be-processed pixel object to be processed, and optimizing and processing the target to-be-processed pixel object according to the obtained subset of sampling points to obtain an optimized processing result of the target to-be-processed pixel object.

2. The method according to claim 1, characterized in that, the method further includes: storing the set of sampling points into a target storage space; wherein, the target storage space is a storage space directly accessible by the threads for processing the target image.

3. The method according to claim 2, characterized in that, during the parallel optimization and processing, through any one of the multiple threads, obtaining a subset of sampling points corresponding to the target to-be-processed pixel object to be processed, includes: during the parallel optimization and processing, through any one of the multiple threads, obtaining a subset of sampling points corresponding to the target to-be-processed pixel object to be processed from the target storage space.

4. The method according to any one of claims 1-3, characterized in that, the method further includes: after optimizing and processing all the to-be-processed pixel objects in the target image, generating an optimized target image according to the optimized processing results of all the to-be-processed pixel objects; performing image rendering processing on the optimized target image.

5. The method according to claim 4, characterized in that, the method further includes: during the process of optimizing and processing the target image, obtaining a next frame image relative to the target image; during the process of performing image rendering processing on the optimized target image, performing image optimization processing on the next frame image relative to the target image.

6. The method according to claim 1, characterized in that, Optimizing the target pixel object to be processed according to the obtained subset of sampling points through any one of the multiple threads to obtain an optimized processing result of the target pixel object to be processed, including: Determining a sampling reference value according to the obtained subset of sampling points and the target pixel object to be processed through any one of the multiple threads; Performing enhancement processing on the sampling reference value; Optimizing the target pixel object to be processed according to the optimization coefficient and the enhanced sampling reference value to obtain an optimized processing result of the target pixel object to be processed.

7. An image processing apparatus Characterized in that The apparatus includes: A determination unit, configured to determine M pixel objects to be processed from a target image; wherein, there are N pixel objects between any two adjacent pixel objects to be processed among the M pixel objects to be processed; An acquisition unit, configured to acquire a set of sampling points corresponding to the M pixel objects to be processed from the target image; wherein, the set of sampling points includes: a subset of sampling points corresponding to each pixel object to be processed, and the sampling points in the subset of sampling points are selected from the target image according to a sampling point selection rule, and there are identical sampling points between the subsets of sampling points corresponding to any two adjacent pixel objects to be processed; the sampling point selection rule includes: taking the pixel object to be processed as the center, equally spaced selecting at least two pixel objects as sampling points in the horizontal direction, equally spaced selecting at least two pixel objects as sampling points in the vertical direction, and equally spaced selecting at least two pixel objects as sampling points in one or more diagonal directions; A processing unit, configured to call multiple threads, determine the pixel objects to be processed that each thread needs to process from the M pixel objects to be processed, and perform parallel optimization processing on the pixel objects to be processed that each thread needs to process through each thread to obtain an optimized processing result of the M pixel objects to be processed; wherein, during the parallel optimization processing, through any one of the multiple threads, acquire a subset of sampling points corresponding to the target pixel object to be processed, and optimize the target pixel object to be processed according to the acquired subset of sampling points to obtain an optimized processing result of the target pixel object to be processed.

8. A computer device Characterized in that It includes: A processor and a memory, the processor and the memory are connected to each other, wherein, the memory stores executable program code, and the processor is configured to call the executable program code to execute the image processing method according to any one of claims 1-6.

9. A computer-readable storage medium Characterized in that The computer-readable storage medium stores a computer program, and when it runs on a computer, it causes the computer to execute the image processing method according to any one of claims 1-6.

Images