Image processing method, image processing device, terminal and readable storage medium

Abstract

The present application discloses an image processing method, an image processing device, a terminal and a readable storage medium. The image processing method comprises: obtaining an image pyramid according to an original image, the image pyramid comprising a bottom layer image with successively decreasing resolutions, and at least one upper layer image, the bottom layer image being the original image; performing scene recognition on the upper layer image with the lowest resolution to obtain an overall recognition result; performing scene recognition on multiple regions in the upper layer image and the bottom layer image respectively to obtain a local recognition result; performing image enhancement processing on the original image according to the overall recognition result and the local recognition result image to obtain a target image. The present application can improve the accuracy of scene recognition, and can use different enhancement effects for each scene in the image, and can also be integrated with the overall enhancement as needed, which can highlight the local or overall effect of the image, thereby facilitating the improvement of the image quality of the target image finally obtained.

Description

Technical Field

[0001] The present application relates to the field of imaging technology, and in particular to an image processing method, an image processing device, a terminal and a computer-readable storage medium. Background Art

[0002] Some current scene inspection-based image enhancement solutions mostly detect the entire image, obtain a scene category of the current image, and enhance the entire image according to this scene category. However, there are many problems with the solution of performing some kind of enhancement on the entire image. For example, when there are multiple targets in the picture, and only the enhancement solution of the identified target is used during enhancement, the final effect is that the enhancement of the identified target is beneficial, while the enhancement of other targets may be disadvantageous. Another problem is that if multiple targets coexist in the image and there is no main target, the current scene may not be recognized, and then the entire image will not be enhanced or only default enhancement will be performed. Summary of the invention

[0003] Embodiments of the present application provide an image processing method, an image processing device, a terminal, and a computer-readable storage medium.

[0004] The embodiment of the present application provides an image processing method. The image processing method includes: obtaining an image pyramid according to an original image, the image pyramid including a bottom layer image with successively decreasing resolutions and at least one upper layer image, the bottom layer image being the original image; performing scene recognition on the upper layer image with the lowest resolution to obtain an overall recognition result; performing scene recognition on multiple regions in the upper layer image and the bottom layer image respectively to obtain a local recognition result; performing image enhancement processing on the original image according to the overall recognition result and the local recognition result image to obtain a target image.

[0005] An embodiment of the present application provides an image processing device. The image processing device includes a first acquisition module, a first recognition module, a second recognition module and a processing module. The first acquisition module is used to acquire an image pyramid based on an original image, wherein the image pyramid includes a bottom layer image with successively decreasing resolutions, and at least one upper layer image, wherein the bottom layer image is the original image. The first recognition module is used to perform scene recognition on the upper layer image with the lowest resolution to obtain an overall recognition result. The second recognition module is used to perform scene recognition on multiple regions in the upper layer image and the bottom layer image, respectively, to obtain the local recognition result. The processing module is used to perform image enhancement processing on the image according to the overall recognition result and the local recognition result to obtain a target image.

[0006] An embodiment of the present application provides a terminal. The terminal includes one or more processors, a memory, and one or more programs. One or more of the programs are stored in the memory and executed by one or more of the processors, and the program includes a program for executing an image processing method. The image processing method includes: obtaining an image pyramid according to an original image, the image pyramid including a bottom layer image with successively decreasing resolutions, and at least one upper layer image and a top layer image, the bottom layer image being the original image; performing scene recognition on the upper layer image with the lowest resolution to obtain an overall recognition result; performing scene recognition on multiple regions in the upper layer image and the bottom layer image respectively to obtain a local recognition result; performing image enhancement processing on the original image according to the overall recognition result and the local recognition result image to obtain a target image.

[0007] The present application embodiment provides a non-volatile computer-readable storage medium of the present application embodiment, which includes a computer program. When the computer program is executed by one or more processors, the processors execute the following image processing method: obtaining an image pyramid according to an original image, the image pyramid including a bottom layer image with successively decreasing resolutions, and at least one upper layer image, the bottom layer image being the original image; performing scene recognition on the top layer image to obtain an overall recognition result; performing scene recognition on multiple regions in the upper layer image and the bottom layer image respectively to obtain local recognition results; performing image enhancement processing on the original image according to the overall recognition result and the local recognition result image to obtain a target image.

[0008] The image processing method, image processing device and terminal of the present application first obtain an image pyramid based on the original image, wherein the resolution of each layer of the image in the image pyramid is different. Then, scene recognition is performed on each area in each layer of the image in the image pyramid to obtain local recognition results and overall recognition results. Subsequently, the image is subjected to corresponding image enhancement processing according to the overall recognition result and the local recognition result, to obtain a first enhanced image for overall enhancement and a second enhanced image for local enhancement, and finally the first enhanced image and the second enhanced image are fused to obtain the target image. On the one hand, compared with directly performing scene recognition on the entire original image, this is conducive to improving the accuracy of scene recognition and can identify the scene of the local area of ​​the original image; on the other hand, it can have greater freedom to use different enhancement effects for each scene in the image, and can also be integrated with the overall enhancement as needed, so that the local or overall effect of the image can be freely highlighted to avoid using only one enhancement processing method for the entire image, which is conducive to improving the image quality of the target image finally obtained.

[0009] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0011] Figure 1 is a flowchart of an image processing method of certain embodiments of the present application;

[0012] Figure 2 is a schematic diagram of the structure of an image processing device in some embodiments of the present application;

[0013] Figure 3 is a schematic diagram of the structure of a terminal in some implementation modes of the present application;

[0014] Figure 4 is a schematic diagram of obtaining an image pyramid in certain embodiments of the present application;

[0015] Figure 5 is a flowchart of an image processing method of certain embodiments of the present application;

[0016] Figure 6 is a schematic diagram of images of each layer of an image pyramid in certain embodiments of the present application;

[0017] Figures 7 to 10 is a schematic diagram of obtaining an image pyramid in certain embodiments of the present application;

[0018] Fig.11 is a schematic diagram of images of each layer of an image pyramid in certain embodiments of the present application;

[0019] Fig.12 is a schematic diagram of an original image and a local recognition result in certain embodiments of the present application;

[0020] Fig.13 is a flowchart of an image processing method of certain embodiments of the present application;

[0021] Fig.14 is a schematic diagram of an original image and a local recognition result in certain embodiments of the present application;

[0022] Figures 15 to 17 is a flowchart of an image processing method of certain embodiments of the present application;

[0023] Fig.18 is a schematic diagram of a key area of ​​an original image in certain embodiments of the present application;

[0024] Fig.19 is a schematic diagram of acquiring a target image in certain embodiments of the present application;

[0025] Fig. 20 It is a connection diagram of a non-volatile computer-readable storage medium and a processor in certain embodiments of the present application. DETAILED DESCRIPTION

[0026] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions from beginning to end. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and cannot be understood as limiting the embodiments of the present application.

[0027] See also Figure 1 , the embodiment of the present application provides an image processing method. The image processing method includes:

[0028] 01: Obtain an image pyramid according to the original image, the image pyramid including bottom-layer images with successively decreasing resolutions and at least one upper-layer image, the bottom-layer image being the original image;

[0029] 02: Perform scene recognition on the upper layer image with the lowest resolution to obtain the overall recognition result;

[0030] 03: Perform scene recognition on multiple areas in the upper image and the bottom image respectively to obtain local recognition results;

[0031] 04: Perform image enhancement processing on the original image based on the overall recognition results and the local recognition results to obtain the target image.

[0032] Please combine Figure 2The embodiment of the present application also provides an image processing device 100, which includes a first acquisition module 10, a first recognition module 20, a second recognition module 30 and a processing module 40. The above step 01 can be implemented by the first acquisition module 10, step 02 can be implemented by the first recognition module 20, step 03 can be implemented by the second recognition module 30, and step 04 can be implemented by the processing module 40. That is, the first acquisition module 10 is used to acquire an image pyramid according to the original image, and the image pyramid includes a bottom layer image with successively decreasing resolutions, and at least one upper layer image, and the bottom layer image is the original image; the first recognition module 20 is used to perform scene recognition on the upper layer image with the lowest resolution to obtain an overall recognition result; the second recognition module 30 is used to perform scene recognition on multiple regions in the upper layer image and the bottom layer image respectively to obtain a local recognition result; the processing module 40 is used to perform image enhancement processing on the original image according to the overall recognition result and the local recognition result to obtain a target image.

[0033] Please combine Figure 3 , the embodiment of the present application also provides a terminal 1000, the terminal 1000 includes one or more processors 200, a memory 300, and one or more programs. Among them, the one or more programs are stored in the memory 300 and executed by the one or more processors 200, and the program includes instructions for executing the image processing method of the embodiment of the present application. That is, when one or more processors 30 execute the program, the processor 30 can implement the methods in step 01, step 02, step 03 and step 04. That is, the one or more processors 200 are used to: obtain an image pyramid according to the original image, the image pyramid includes a bottom layer image with successively decreasing resolutions, and at least one upper layer image, the bottom layer image is the original image; perform scene recognition on the upper layer image with the lowest resolution to obtain an overall recognition result; perform scene recognition on multiple regions in the upper layer image and the bottom layer image respectively to obtain a local recognition result; and perform image enhancement processing on the original image according to the overall recognition result and the local recognition result to obtain a target image.

[0034] The image processing method, image processing device 100 and terminal 1000 of the present application first obtain an image pyramid based on the original image, wherein the resolution of each layer of the image pyramid is different. Then, scene recognition is performed on each area in each layer of the image pyramid to obtain a local recognition result and an overall recognition result. Subsequently, the image is enhanced according to the overall recognition result and the local recognition result to obtain a first enhanced image for overall enhancement and a second enhanced image for local enhancement, and finally the first enhanced image and the second enhanced image are fused to obtain the target image. On the one hand, since each layer of images (image pyramid) with different resolutions is first obtained based on the original image, and then scene recognition is performed on each area of ​​each layer of images with different resolutions, it is beneficial to improve the accuracy of scene recognition compared to directly performing scene recognition on the entire original image, and the scene of the local area of ​​the original image can be identified; on the other hand, since local recognition results and overall recognition results can be obtained, and corresponding image enhancement processing is performed on the image for the overall recognition results and the local recognition results respectively, to obtain a first enhanced image for overall enhancement and a second enhanced image for local enhancement, and then the first enhanced image and the second enhanced image are fused to obtain the target image, there is greater freedom to use different enhancement effects for each scene in the image, and it can also be integrated with the overall enhancement as needed, and the local or overall effect of the image can be freely highlighted to avoid using only one enhancement processing method for the entire image, which is beneficial to improving the image quality of the target image finally obtained.

[0035] Specifically, after acquiring the original image, the first acquisition module 10 (or the processor 200) acquires an image pyramid according to the original image. The image pyramid includes bottom-level images with successively decreasing resolutions, and at least one upper-level image, where the bottom-level image is the original image. In some embodiments, the upper-level image includes a top-level image and an intermediate-level image, where the top-level image is the upper-level image with the lowest resolution, and all images in the upper-level image except the top-level image are intermediate-level images. For example, Figure 4 As shown, the image pyramid includes multiple layers of images with different resolutions, and the resolution of any frame of the image in the image pyramid is greater than the resolution of the image arranged in the upper layer. The image arranged at the top of the image pyramid is the top image, that is, the image with the lowest resolution in the image pyramid is the top image; the image arranged at the bottom of the image pyramid is the bottom image, that is, the image with the highest resolution in the image pyramid is the bottom image; the image arranged between the top image and the bottom image in the image pyramid is the middle layer image, and the resolution of the middle layer image is greater than the resolution of the top image, but the resolution of the middle layer image is less than the resolution of the bottom image.

[0036] It should be noted that the number of intermediate layer images can be one or more. When the number of intermediate layer images is more than one, the resolution of the intermediate layer images arranged closer to the top layer image is smaller. Figure 4 As shown, the image pyramid includes a bottom layer image, a first intermediate layer image, a second intermediate layer image and a top layer image. The second intermediate layer image is closer to the top layer image than the first intermediate layer image, and the resolution of the second intermediate layer image is greater than the resolution of the first intermediate layer image. In addition, when the number of upper layer images is one, the upper layer image is both a top layer image and an intermediate layer image.

[0037] For example, in some embodiments, the first acquisition module 10 (or the processor 200) first uses the original image as the bottom image of the image pyramid, scales the bottom image (i.e., the original image, the bottom image mentioned below is the original image) to obtain an image with a resolution smaller than that of the bottom image, and places the obtained image on the upper layer of the bottom image in the image pyramid, and then repeats the above steps (i.e., scaling the image to obtain a new image, and placing the obtained new image on the upper layer of the image in the image pyramid), until the preset number of times is repeated, the image pyramid can be obtained. The preset number of times can be determined by the image processing device 100 (or the terminal 200) before leaving the factory based on a large amount of experimental data; or the preset number of times can also be set by the user according to his own needs, which is not limited here.

[0038] In some embodiments, the first acquisition module 10 (or the processor 200) may perform Gaussian blurring and downsampling processing on the image to achieve scaling of the image. Figure 4 As shown, the preset number of times is 3, that is, the image pyramid includes a total of 4 layers of images for explanation. The first acquisition module 10 (or processor 200) first uses the original image as the bottom image of the image pyramid, and then performs Gaussian blur and downsampling on the bottom image to obtain a first intermediate layer image with a resolution smaller than that of the bottom image, and the first intermediate layer image is placed on the upper layer of the bottom image in the image pyramid. The first intermediate layer image is then Gaussian blurred and downsampled to obtain a second intermediate layer image with a resolution smaller than that of the first intermediate layer image, and the second intermediate layer image is placed on the upper layer of the first intermediate layer image in the image pyramid. The second intermediate layer image is then Gaussian blurred and downsampled to obtain a top layer image with a resolution smaller than that of the second intermediate layer image, and the top layer image is placed on the upper layer of the second intermediate layer image in the image pyramid. It should be noted that, in some embodiments, the length and width of the upper layer image in the image pyramid are both half of the adjacent lower layer image. For example, as Figure 4As shown, the length of the first intermediate layer image is half of the length of the underlying image, and the width of the first intermediate layer image is half of the width of the underlying image.

[0039] It should be noted that the multiple images in the image pyramid do not have to be stacked. For example, in some embodiments, the multiple images in the image pyramid can also be arranged in a queue. For example, after the first acquisition module 10 (or the processor 200) acquires the bottom image and the upper image with a lower resolution than the bottom image, it can also arrange the lower resolution upper image after the bottom image. For the convenience of explanation, in the embodiments of the present application, the multiple images in the image pyramid are stacked according to the size of the resolution (i.e., Figure 4 The image pyramid shown).

[0040] After obtaining the image pyramid, the upper image with the lowest resolution in the image pyramid is obtained, that is, the top image is obtained, and the first recognition module 20 (or the processor 200) performs scene recognition on the upper image with the lowest resolution to obtain an overall recognition result. That is, scene recognition is performed on the top image to obtain an overall recognition result. Among them, the overall recognition result is used to characterize the scene category of the original image as a whole. On the one hand, since the top images are obtained by scaling the bottom images (that is, the original images) multiple times, the content in the top image is roughly the same as the content in the original image. Therefore, the scene information in the top image can characterize what scene the original image as a whole belongs to, that is, as long as the scene information in the top image is obtained, the overall scene of the original image can be obtained. On the other hand, since the resolution of the top image is the lowest, that is, the size of the top image is the smallest, the scene recognition of the top image is faster and consumes less power than the scene recognition of other images with larger resolutions. That is to say, this embodiment adopts scene recognition on the image with the lowest resolution to obtain the overall recognition result, so that the power consumption can be reduced and the recognition speed can be improved while obtaining the overall scene of the original image.

[0041] Specifically, in some embodiments, scene recognition of the top-level image can be achieved based on a deep learning model. For example, a deep learning model that can recognize multiple scenes is pre-selected and trained based on a large number of experiments. After obtaining the top-level image after the image pyramid is obtained, the top-level image is input into the trained deep learning model, and a series of operations such as convolution and pooling are performed to obtain feature information, and then the scene information in the top-level image is identified based on the feature information, and the overall recognition result is obtained. Of course, in some embodiments, other methods can also be used to recognize the scene of the top-level image, as long as the scene information in the top-level image can be obtained, and there is no limitation here.

[0042] The second recognition module 30 (or processor 200) performs scene recognition on multiple regions in the upper layer image and the lower layer image respectively to obtain local recognition results. The local recognition results are used to indicate which scene the original image belongs to. Figure 1 , Figure 5 and Fig.12 In some embodiments, the local recognition result includes multiple sub-results, and the multiple sub-results correspond to multiple first regions of the underlying image. Step 03: Perform scene recognition on multiple regions in the upper image and the underlying image respectively to obtain the local recognition result, including:

[0043] 031: Divide the bottom layer image into a plurality of first regions, divide the upper layer image into a plurality of second regions, and perform scene recognition on each of the first regions and the second regions respectively to obtain a recognition result of each first region and a recognition result of each second region, wherein the size of the first region is the same as the size of the second region;

[0044] 032: Obtain a sub-result corresponding to each first region in the bottom layer image according to the recognition result of each first region and the recognition result of the second region corresponding to the first region in the upper layer image;

[0045] 033: Obtain local recognition results based on all sub-results.

[0046] Please combine Figure 2 and Figure 3 In some embodiments, the second recognition module 30 and the processor 200 can also be used to execute the methods in step 031, step 032 and step 033. That is, the second recognition module 30 and the processor 200 are also used to divide the bottom layer image into a plurality of first regions, divide the top layer image into a plurality of second regions, and perform scene recognition on each first region and each second region respectively to obtain a recognition result of each first region and a recognition result of each second region, wherein the size of the first region is the same as the size of the second region; obtain a sub-result corresponding to each first region in the bottom layer image according to the recognition result of each first region and the recognition result of the second region corresponding to the first region in the top layer image; and obtain a local recognition result according to all the sub-results.

[0047] It should be noted that if Figure 6 and Fig.12 As shown, the underlying image includes multiple first regions, and the local recognition result includes multiple sub-results, and the multiple sub-results correspond to the multiple first regions one by one. The sub-results are used to characterize what scene the corresponding first region belongs to.

[0048] Furthermore, in some embodiments, the image processing method further includes obtaining the working mode of the shooting device that shoots the original image. If the shooting device is in the first working mode, only the middle layer image and the bottom layer image in the upper layer image are used to obtain the local recognition result. That is, scene recognition is performed on multiple areas in the middle layer image and the bottom layer image respectively to obtain the local recognition result; if the shooting device is in the second working mode, the middle layer image and the top layer image in the upper layer image and the bottom layer image are used to obtain the local recognition result. That is, scene recognition is performed on multiple areas in the top layer image, the middle layer image and the bottom layer image respectively to obtain the local recognition result.

[0049] It should be noted that, in some embodiments, the shooting device includes a front camera and a rear camera. If the original image is obtained by shooting with the rear camera, the shooting device is in the first working mode. In some embodiments, the working mode of the shooting device can also be adjusted by the user according to his own needs. For example, if the user wants to perform local scene recognition only based on the middle layer image and the bottom layer image, the user can adjust the shooting device to the first working mode. Because when the shooting device is in the first working mode, scene recognition is performed on multiple areas in the middle layer image and the bottom layer image respectively to obtain a local recognition result. Since the top layer image is obtained by scaling the bottom layer image (original image) multiple times, the resolution of the top layer image is low. Even if a part of the top layer image is extracted for local scene recognition, there may be multiple different scenes in the area, which is not conducive to obtaining a local recognition result. Therefore, only performing local scene recognition based on the middle layer image and the bottom layer image to obtain a local recognition result can accurately complete the local scene recognition while also improving the recognition speed. If the original image is obtained by shooting with the front camera, the shooting device is in the second working mode. Since the application scene of shooting with the front camera is usually self-portrait, the area occupied by the face is large. The multiple first areas in the bottom layer image and the multiple second areas in the middle layer image may only include parts of the human face, and the characteristic information is very small. It is very likely that the scene information of the first area and the second area cannot be recognized, that is, it is very likely that the recognition result of the first area and the recognition result of the second area, which are obviously parts of the human face, are both the first result. At this time, using only the middle layer image and the bottom layer image for local scene recognition may result in misjudgment. Therefore, in this embodiment, when the shooting device is in the second working mode, local scene recognition is performed based on the top layer image, the middle layer image and the bottom layer image, which can reduce misjudgment and help improve the image quality of the final acquired image. Of course, in some embodiments, the working mode of the shooting device can also be adjusted by the user according to his own needs. For example, if the user wants to perform local scene recognition based on the top layer image, the middle layer image and the bottom layer image, then the user can adjust the shooting device to the second working mode.

[0050] Take the first working mode of the camera as an example. At this time, only the middle layer image and the bottom layer image in the upper layer image are used to obtain the local recognition result. That is, scene recognition is performed on multiple areas in the middle layer image and the bottom layer image respectively to obtain the local recognition result. For details, please refer to Figure 6 In some embodiments, the second recognition module 30 (or the processor 200) divides the bottom layer image into a plurality of first regions, divides the middle layer image into a plurality of second regions, and performs scene recognition on each first region and the second region respectively to obtain the recognition result of each first region and the recognition result of the second region. The size of the first region is the same as the size of the second region. It should be noted that the number of the middle layer images can be one or more. When the number of the middle layer images is multiple, each frame of the middle layer image needs to be divided into a plurality of second regions respectively.

[0051] For example, an image pyramid including 4 layers of images, that is, including two frames of intermediate images, is used as an example for description. Figure 6 As shown, the image pyramid includes a bottom layer image, a first intermediate layer image, a second intermediate layer image and a top layer image. The second recognition module 30 (or the processor 200) divides the bottom layer image into a plurality of first regions, divides the first intermediate layer image into a plurality of second regions, and divides the second intermediate layer image into a plurality of second regions. Among them, the size of the first region in the bottom layer image, the size of the second region in the first intermediate layer image, and the size of the second region in the second intermediate layer image are all the same. It should be noted that when the length and width of the upper layer image in the image pyramid are both half of the adjacent lower layer image, since the size of the first region is the same as the size of the second region, the number of regions divided in the upper layer image in the image pyramid is one fourth of the number of regions divided in the adjacent lower layer image. As shown Figure 6 In the illustrated embodiment, assuming that the bottom layer image is divided into 8*8 first regions, the first intermediate layer image is divided into 4*4 second regions, and the second intermediate layer image is divided into 2*2 second regions.

[0052] After the bottom layer image is divided into a plurality of first regions and the middle layer image is divided into a plurality of second regions, the second recognition module 30 (or the processor 200) performs scene recognition on each first region and each second region respectively to obtain the recognition result of each first region and the recognition result of each second region. The recognition result of the first region is used to characterize the scene information in the first region, and the recognition result of the second region is used to characterize the scene information in the second region. Among them, the specific implementation method of scene recognition on the first region and the second region is the same as the specific implementation method of scene recognition on the top layer image in the above embodiment, and will not be repeated here. Since the bottom layer image and the middle layer image are divided into a plurality of regions, the possibility of including multiple scenes in one region can be reduced. In this way, by separately recognizing a plurality of regions, it is possible to avoid the situation where only one scene can be recognized, and the scene information of each local area of ​​the image can be obtained, which is conducive to the subsequent enhancement processing of the image according to the scene information, thereby improving the quality of the image.

[0053] After obtaining the recognition result of each first area and the recognition result of each second area, the second recognition module 30 (or the processor 200) obtains the sub-result corresponding to each first area in the bottom layer image according to the recognition result of each first area and the recognition result of the second area corresponding to the first area in the intermediate layer image. Since the bottom layer image and the intermediate image are divided into multiple areas, after the multiple areas are recognized respectively, the sub-result corresponding to the first area is obtained together according to the recognition results of the corresponding areas, both large scene objects and small scene objects can be recognized, further improving the accuracy of local scene recognition, which is conducive to the subsequent enhancement processing of the image according to the scene information, thereby improving the quality of the image.

[0054] It should be noted that if the content in a second area includes the content in a first area, then the first area corresponds to the second area. Figure 6 In the embodiment shown, the length and width of the upper layer image in the image pyramid are both half of the adjacent lower layer image. Assuming that the bottom layer image is divided into a first area of ​​8*8, the first intermediate layer image is divided into a second area of ​​4*4, and the second intermediate layer image is divided into a second area of ​​2*2. Figure 6As shown, b11 in the first intermediate layer image is scaled together by the first area a11, the first area a12, the first area a21, and the first area a22 in the underlying image, that is, the content of b11 in the first intermediate layer image includes the content of the first area a11, the first area a12, the first area a21, and the first area a22 in the underlying image. Therefore, the second areas corresponding to the first area a11, the first area a12, the first area a21, and the first area a22 in the underlying image in the first intermediate layer image are all b11 in the first intermediate layer image, that is, the first area a11, the first area a12, the first area a21, the first area a22, the first area a21, and the first area a22 arranged in the 1st row and 1st column of the underlying image, the first area b11 arranged in the 1st row and 1st column of the first intermediate layer image, and the first area a22 in the 2nd row and 2nd column of the underlying image, correspond to the second areas in the first intermediate layer image.

[0055] like Figure 6As shown, the second area c11 in the second intermediate image is scaled together by the second area b11, the second area b12, the second area b21, and the second area b22 in the first intermediate image, that is, the content of the second area c11 in the second intermediate image includes the content of the second area b11, the second area b12, the second area b21, and the second area b22 in the first intermediate image. It can be understood that b11 in the first intermediate layer image is scaled together by the first area a11, the first area a12, the first area a21, and the first area a22 in the underlying image, b12 in the first intermediate layer image is scaled together by the first area a13, the first area a14, the first area a23, and the first area a24 in the underlying image, b21 in the first intermediate layer image is scaled together by the first area a31, the first area a32, the first area a41, and the first area a42 in the underlying image, and b22 in the first intermediate layer image is scaled together by the first area a33, the first area a34, the first area a43, and the first area a44 in the underlying image. That is to say, the second area c11 in the second intermediate image is scaled together by the first area a11, the first area a12, the first area a21, the first area a22, the first area a13, the first area a14, the first area a23, the first area a24, the first area a31, the first area a32, the first area a41, the first area a42, the first area a33, the first area a34, the first area a43, and the first area a44 in the underlying image, that is, the content of the second area c11 in the second intermediate image includes the contents of the first area a11, the first area a12, the first area a21, the first area a22, the first area a13, the first area a14, the first area a23, the first area a24, the first area a31, the first area a32, the first area a41, the first area a42, the first area a33, the first area a34, the first area a43, and the first area a44 in the underlying image. Therefore, the first area a11, first area a12, first area a21, first area a22, first area a13, first area a14, first area a23, first area a24, first area a31, first area a32, first area a41, first area a42, first area a33, first area a34, first area a43, and first area a44 in the underlying image correspond to second areas in the second intermediate layer image all are second area c11 in the second intermediate image.That is, the first area a11 in the 1st row and 1st column, the first area a12 in the 1st row and 2nd column, the first area a21 in the 2nd row and 1st column, the first area a22 in the 2nd row and 2nd column, the first area a13 in the 1st row and 3rd column, the first area a14 in the 1st row and 4th column, the first area a23 in the 2nd row and 3rd column, the first area a24 in the 2nd row and 4th column, the first area a31 in the 3rd row and 1st column, the first area a32 in the 3rd row and 2nd column, the first area a41 in the 4th row and 1st column, the first area a42 in the 4th row and 2nd column, the first area a33 in the 3rd row and 3rd column, the first area a34 in the 3rd row and 4th column, the first area a43 in the 4th row and 4th column, and the first area a44 in the 4th row and 4th column, which are arranged in the 1st row and 1st column of the underlying image, all correspond to the second area c11 arranged in the 1st row and 1st column of the second intermediate layer image.

[0056] It should be noted that when the shooting device is in the second working mode, the intermediate layer image and the top layer image in the upper layer image and the bottom layer image are used to obtain local recognition results. That is, scene recognition is performed on multiple areas in the top layer image, the intermediate layer image and the bottom layer image respectively to obtain local recognition results. At this time, the second recognition module 30 (or the processor 200) divides the bottom layer image into multiple first areas, divides the upper layer image (including the intermediate layer image and the top layer image) into multiple second areas, and performs scene recognition on each first area and the second area respectively to obtain the recognition result of each first area and the recognition result of the second area.

[0057] In some embodiments, the second recognition module 30 (or the processor 200) randomly extracts a first area in the underlying image, and obtains the recognition result of the first area and the recognition result of the second area corresponding to the first area in the upper image. According to the recognition result of the first area and the recognition result of the second area, the sub-result corresponding to the first area is obtained. Subsequently, another first area is extracted from the underlying image and the above steps are repeated until all the first areas in the underlying image have been extracted, so that the sub-result corresponding to each first area in the underlying image is obtained.

[0058] For further information, see Figure 7 and Figure 8 In some embodiments, step 032: obtaining a sub-result corresponding to each first region in the bottom layer image according to the recognition result of each first region and the recognition result of the second region corresponding to the first region in the upper layer image, further includes:

[0059] 0321: if the recognition result of the first area is the second result, the recognition result of the first area is used as the sub-result corresponding to the first area; or

[0060] 0322: If the recognition result of the first area pair is a second result, a sub-result corresponding to the first area is obtained according to the recognition result of the second area corresponding to the first area in the upper image; wherein the second result includes recognition of a specific scene category.

[0061] Please combine Figure 2 and Figure 3 In some embodiments, the second recognition module 40 and the processor 200 may also be used to execute the methods in step 0321 and step 0322. That is, the second recognition module 40 and the processor 200 are also used to, if the recognition result of the first region is the second result, use the recognition result of the first region as the sub-result corresponding to the first region; or, if the recognition result of the first region pair is the second result, obtain the sub-result corresponding to the first region according to the recognition result of the second region corresponding to the first region in the upper layer image; wherein the second result includes the recognition of a specific scene category.

[0062] Due to the high resolution of the underlying image, in the first and second regions of the same size, although the first region contains less content than the second region, the first region contains more detailed information than the corresponding second region. When performing scene recognition on the corresponding first and second regions respectively, scene information may be recognized in both the first and second regions, but they belong to different scene categories. Figure 6 As shown, the first area a11 arranged in the 1st row and 1st column of the bottom layer image corresponds to the second area b11 arranged in the 1st row and 1st column of the first intermediate layer image in the first intermediate layer image. Assuming that the content of the second area b11 arranged in the 1st row and 1st column of the first intermediate layer image is a large group of flowers, and there is a small bee on a flower in the upper left corner, the content of the first area a11 arranged in the 1st row and 1st column of the bottom layer image corresponds to the content in the upper left corner of the second area b11 in the 1st row and 1st column of the first intermediate layer image, and the bee on the flower can be clearly seen. At this time, when the second area b11 in the 1st row and 1st column of the first intermediate layer image is subjected to scene recognition, since the area occupied by the bee is smaller than that of the flowers, that is, the main target in the second area b11 is the flowers, the scene of the second area b11 will be recognized as flowers. However, when performing scene recognition on the first area a11 of the 1st row and 1st column of the bottom layer image, the bee occupies a larger area and is easily recognized, that is, the main target in the first area a11 is the bee, so the scene in the first area is recognized as an animal.

[0063] In some embodiments, after the second recognition module 30 (or processor 200) obtains the recognition result of a first area and the recognition result of the second area corresponding to the first area in the upper layer image, if the recognition result of the first area is the second result, the recognition result of the first area is used as the sub-result corresponding to the first area. Among them, the second result includes the specific scene category identified. When the recognition result of the first area is the second result, it indicates that the scene category to which the first area is specifically identified belongs, and the recognition result of the first area is directly used as the sub-result corresponding to the first area. In this way, small target scenes in the underlying image (original image) can be identified, which is conducive to the subsequent enhancement processing of the image, thereby improving the quality of the image.

[0064] Take the example that the shooting device is in the first working mode and only uses the middle layer image and the bottom layer image in the upper layer image to obtain the local recognition result. Figure 6 As shown, the image pyramid includes a bottom layer image and three upper layer images, wherein the upper layer image includes a first intermediate layer image, a second intermediate layer image and a top layer image. The bottom layer image includes 8*8 first regions, the first intermediate layer image includes 4*4 second regions, and the second intermediate layer image includes 2*2 second regions. Among them, the first region a11 arranged in the first row and first column of the bottom layer image corresponds to the second region b11 arranged in the first row and first column of the first intermediate layer image in the first intermediate layer image; the first region a11 arranged in the first row and first column of the bottom layer image corresponds to the second region c11 arranged in the first row and first column of the second intermediate layer image in the second intermediate layer image. When obtaining the sub-result corresponding to the first region a11 in the bottom layer image, the second recognition module 30 (or the processor 200) respectively obtains the recognition result of the first region a11 arranged in the first row and first column of the bottom layer image, the recognition result of the second region b11 arranged in the first row and first column of the first intermediate layer image, and the recognition result of the second region c11 arranged in the first row and first column of the second intermediate layer image. If the recognition result of the first area a11 arranged in the first row and first column of the underlying image is the second result, that is, the scene category of the area is identified, the recognition result of the first area a11 is directly used as the sub-result corresponding to the first area a11. The second result includes multiple scene categories, for example, the second result includes scene categories such as flowers, blue sky, and human faces. Assuming that the first area is identified as flowers, the sub-result corresponding to the first sub-area is flowers.

[0065] It should be noted that, in some embodiments, the second recognition module 30 (or processor 200) may first obtain the recognition result of a first area in the bottom layer image, and if the recognition result of the first area is the second result, the recognition result of the first area is directly used as the sub-result corresponding to the first area, without obtaining the recognition result of the second area corresponding to the first area in the intermediate layer image. This can speed up scene recognition and thus speed up image processing.

[0066] Similarly, when the shooting device is in the second working mode and uses the middle layer image and the top layer image in the upper layer image and the bottom layer image to obtain local recognition results, if the recognition result of the first area is the second result, the recognition result of the first area is directly used as the sub-result corresponding to the first area, which is not elaborated here.

[0067] In particular, in some embodiments, after the second recognition module 30 (or processor 200) obtains the recognition result of a first area and the recognition result of the second area corresponding to the first area in the upper image, if the recognition result of the first area is the second result, the sub-result corresponding to the first area is obtained according to the recognition result of the second area corresponding to the first area in the upper image. Since the small target scene in the underlying image (original image) may only occupy a very small part of the entire image, even if the corresponding enhancement processing is performed on the area where the small target scene is located, the overall effect of the image is not significantly improved, but the image processing time and power consumption will be increased. Therefore, in this embodiment, even if the first area of ​​the underlying image also recognizes the scene information, the sub-result corresponding to the first area is still obtained according to the recognition result of the second area corresponding to it in the upper image, so that the image processing speed can be improved and the power consumption can be reduced without affecting the overall effect of the image.

[0068] It should be noted that, in some embodiments, if the recognition result of the first area is the second result, that is, the first area recognizes the scene, the user can choose whether to directly use the recognition result of the first area as the sub-result corresponding to the first area, or to obtain the sub-result corresponding to the first area based on the recognition result of the second area corresponding to the first area in the upper image. For example, in one example, the user can choose whether to enhance the small target. If the user chooses to enhance the small target, then when the recognition result of the first area is the second result, the recognition result of the first area is directly used as the sub-result corresponding to the first area; if the user chooses not to enhance the small target, then when the recognition result of the first area is the second result, the sub-result corresponding to the first area is obtained based on the recognition result of the second area corresponding to the first area in the upper image.

[0069] See also Fig. 9In some embodiments, step 032: obtaining a sub-result corresponding to each first region in the bottom layer image according to the recognition result of each first region and the recognition result of the second region corresponding to the first region in the upper layer image, includes:

[0070] 0323: If the recognition result of the first area is the first result, a sub-result corresponding to the first area is obtained according to the recognition result of the second area corresponding to the first area in the upper image, wherein the first result indicates that the scene is not recognized.

[0071] Please combine Figure 2 and Figure 3 In some embodiments, the second recognition module 40 and the processor 200 may also be used to execute the method in step 0323. That is, the second recognition module 40 and the processor 200 may also be used to obtain a sub-result corresponding to the first area according to the recognition result of the second area corresponding to the first area in the upper image if the recognition result of the first area is the first result, wherein the first result indicates that the scene is not recognized.

[0072] Since the resolution of the underlying image is large, in the first and second regions of the same size, the content in the first region is only a part of the corresponding second region. When there is a large target scene in the underlying image (original image), it is possible that the first region corresponding to the large target scene contains only a small part of the target scene, and the first region contains less feature information of the target scene. At this time, when the first region is subjected to scene recognition, the scene in the first region cannot be recognized. However, since the resolution of the upper image is small, the content contained in the second region corresponding to the first region is more than the content contained in the first region, so the feature information containing the target scene in the second region is more. At this time, when the second region is subjected to scene recognition, the scene in the second region can be recognized. In addition, since the content in the first region is a part of the content in the second region corresponding to it, when it is recognized that there is no scene in the first region, and the scene category of the second region corresponding to it is recognized, the scene category of the first region should be the same as the scene category of the second region corresponding to it.

[0073] Therefore, in some embodiments, when the recognition result of the first area is the first result, the sub-result corresponding to the first area can also be obtained based on the recognition result of the second area corresponding to the first area in the upper image. Specifically, after the second recognition module 30 (or the processor 200) obtains the recognition result of a first area and the recognition result of the second area corresponding to the first area in the upper image, if the recognition result of the first area is the first result, that is, the first area does not recognize the scene information, then the sub-result corresponding to the first area is obtained based on the recognition result of the second area corresponding to the first area in the upper image.

[0074] In some embodiments, the number of intermediate layer images may be multiple. Fig.10 , when there are multiple intermediate layer images, obtaining corresponding sub-results according to the recognition result of the second region corresponding to the first region in the intermediate layer includes:

[0075] 051: performing a processing step on the upper image with the highest resolution, the processing step comprising: if the recognition result of the second area corresponding to the first area in the upper image is the second result, using the recognition result of the second area corresponding to the first area in the upper image as the sub-result corresponding to the first area; if the recognition result of the second area corresponding to the first area in the upper image is the first result, selecting an upper image arranged after the upper image, and repeatedly performing the processing step until the processing step is performed on all the upper images; or

[0076] 052: Execute a processing step on the upper image with the highest resolution, the processing step comprising: if the recognition result of the second area corresponding to the first area in the upper image is the second result, then use the recognition result of the second area corresponding to the first area in the upper image as the sub-result corresponding to the first area; if the recognition result of the second area corresponding to the first area in the upper image is the first result, select an upper image arranged after the upper image, and repeat the processing steps until the processing steps are completed for all upper images except the upper image with the lowest resolution.

[0077] Please combine Figure 2 and Figure 3In some embodiments, the second identification module 30 and the processor 200 are also used to execute the methods in step 051 and step 052. That is to say, the second recognition module 30 and the processor 200 are also used to perform processing steps on the upper image with the highest resolution, and the processing steps include: if the recognition result of the second area corresponding to the first area in the upper image is the second result, then the recognition result of the second area corresponding to the first area in the upper image is used as the sub-result corresponding to the first area; if the recognition result of the second area corresponding to the first area in the upper image is the first result, then an upper image arranged after the upper image is selected, and the processing steps are repeatedly performed until the processing steps are completed for all upper images; or the processing steps are performed on the upper image with the highest resolution, and the processing steps include: if the recognition result of the second area corresponding to the first area in the upper image is the second result, then the recognition result of the second area corresponding to the first area in the upper image is used as the sub-result corresponding to the first area; if the recognition result of the second area corresponding to the first area in the upper image is the first result, then an upper image arranged after the upper image is selected, and the processing steps are repeatedly performed until the processing steps are completed for all upper images except the upper image with the lowest resolution.

[0078] It should be noted that the resolution of the next upper-layer image arranged after a certain upper-layer image is second only to the resolution of the upper-layer image. That is, the resolution of the next upper-layer image arranged after a certain upper-layer image is smaller than the resolution of the upper-layer image, and the resolution difference between the two is smaller than the resolution difference between other upper-layer images with smaller resolution than the upper-layer image and the upper-layer image. For example, if multiple images in an image pyramid are stacked, that is, the lower the image, the greater the resolution, and the higher the image, the smaller the resolution. At this time, the next upper-layer image arranged after the upper-layer image refers to the image arranged in the upper layer of the upper-layer image. If multiple images in the image pyramid are arranged in a queue from large to small according to resolution, and the resolution of the image arranged in front is larger, the resolution of the image arranged in the back is smaller. At this time, the next upper-layer image arranged after the upper-layer image refers to the first image arranged after the upper-layer image. In particular, in some embodiments, if the upper-layer images are arranged according to other rules, all the upper-layer images can be sorted from large to small according to resolution, that is, the upper-layer images with large resolution are arranged before the upper-layer images with small resolution, so that the resolution of the subsequent upper-layer image of any upper-layer image is second only to the upper-layer image.

[0079] In some embodiments, when the shooting device is in the first working mode, since only the intermediate layer image and the bottom layer image in the upper layer image are used to obtain the local recognition result, that is, the top layer image is not considered at this time, and the upper layer image with the lowest resolution is the top layer image, and all upper layer images except the upper layer image with the lowest resolution refer to all intermediate images. Therefore, when the shooting device is in the first working mode and needs to obtain the sub-result corresponding to the first area according to the recognition result of the second area corresponding to the first area in the upper layer image, the second recognition module 30 (or the processor 200) performs the above step 052: perform the processing step on the upper layer image with the highest resolution, and the processing step includes: if the recognition result of the second area corresponding to the first area in the upper layer image is the second result, then the recognition result of the second area corresponding to the first area in the upper layer image is used as the sub-result corresponding to the first area; if the recognition result of the second area corresponding to the first area in the upper layer image is the first result, then select the upper layer image of the upper layer of the upper layer image, and repeat the processing step until the processing step is completed for all upper layer images except the upper layer image with the lowest resolution. That is to say, when the shooting device is in the first working mode, there is no need to perform the processing steps on the top layer image. The step is stopped when the recognition result of the second area corresponding to the first area in a certain intermediate layer image is the second result, or the processing step can be stopped after all the intermediate layer images are completed. Specifically, the second recognition module 30 (or the processor 200) performs the processing steps on the intermediate layer image with the highest resolution. Among them, the processing steps include: if the recognition result of the second area corresponding to the first area in the intermediate layer image is the second result, the recognition result of the second area corresponding to the first area in the intermediate layer image is used as the sub-result corresponding to the first area; if the recognition result of the second area corresponding to the first area in the intermediate layer image is the first result, an intermediate image arranged after the intermediate layer image is selected to repeat the processing steps until the processing steps are completed for all the intermediate layer images.

[0080] For example, taking the acquisition of the sub-result corresponding to the first region arranged in the 1st row and 1st column of the bottom layer image as an example for explanation, it is assumed that the image pyramid includes the bottom layer image, the first intermediate layer image, the second intermediate layer image, the third intermediate layer image and the top layer image, wherein the first region arranged in the 1st row and 1st column of the bottom layer image corresponds to the second region arranged in the 1st row and 1st column of the first intermediate layer image in the first intermediate layer image; the first region arranged in the 1st row and 1st column of the bottom layer image corresponds to the second region arranged in the 1st row and 1st column of the second intermediate layer image in the second intermediate layer image; the first region arranged in the 1st row and 1st column of the bottom layer image corresponds to the second region arranged in the 1st row and 1st column of the second intermediate layer image in the third intermediate layer image. When the shooting device is in the first working mode and needs to acquire the sub-result corresponding to the first region according to the recognition result of the second region corresponding to the first region in the upper layer, the second recognition module 30 (or the processor 200) performs the processing step on the intermediate layer image with the largest resolution, that is, the first intermediate layer image. Specifically, if the recognition result of the second area arranged in the first row and first column of the first intermediate layer image is the second result, that is, the scene is recognized in the second area, the recognition result of the second area arranged in the first row and first column of the first intermediate layer image is directly used as the sub-result corresponding to the first area arranged in the first row and first column of the underlying image. If the recognition result of the second area arranged in the first row and first column of the first intermediate layer image is the first result, that is, the scene is not recognized in the second area, the second intermediate layer image arranged after the first intermediate layer image is selected to repeat the processing steps. That is, if the recognition result of the second area arranged in the first row and first column of the second intermediate layer image is the second result, the recognition result of the second area arranged in the first row and first column of the second intermediate layer image is used as the sub-result corresponding to the first area arranged in the first row and first column of the underlying image; if the recognition result of the second area arranged in the first row and first column of the second intermediate layer image is the first result, the above processing steps are repeatedly performed on the third intermediate layer image arranged after the second intermediate layer image until all intermediate layer images have completed the above processing steps.

[0081] It should be noted that, in some embodiments, when the shooting device is in the first working mode and has executed the processing steps on the last intermediate layer image (i.e., the intermediate layer image with the smallest resolution), then regardless of whether the recognition result of the second area corresponding to the first area in the last intermediate layer image is the first result (the scene is not recognized) or the second result (specific scene category), the recognition result of the second area is used as the corresponding sub-result of the first area.

[0082] In particular, in some embodiments, Fig.11As shown, the number of intermediate layer images is one, and it is necessary to obtain the sub-result corresponding to the first area based on the recognition result of the second area corresponding to the first area in the upper layer image. The second recognition module 30 (or processor 200) can also directly use the recognition result of the second area corresponding to the first area in the intermediate layer as the corresponding sub-result of the first area. It should be noted that at this time, regardless of whether the recognition result of the second area is the first result (scene not recognized) or the second result (specific scene category), the recognition result of the second area is used as the corresponding sub-result of the first area.

[0083] Specifically, take obtaining the sub-result corresponding to the first region arranged in the first row and first column of the underlying image as an example for explanation, such as Fig.11 As shown, the image pyramid includes a bottom image, an intermediate layer image and a top layer image, the bottom image includes 4*4 first regions, and the intermediate layer image includes 2*2 second regions. Among them, the second region corresponding to the first region arranged in the first row and first column of the bottom image in the first intermediate layer image is the second region arranged in the first row and first column of the intermediate layer image. When it is necessary to obtain the sub-result corresponding to the first region based on the recognition result of the second region corresponding to the first region in the intermediate layer, the second recognition module 30 (or the processor 200) obtains the recognition result of the second region arranged in the first row and first column of the intermediate layer image, and uses the obtained recognition result of the second region as the sub-result corresponding to the first region arranged in the first row and first column of the bottom image.

[0084] In some embodiments, when the shooting device is in the second working mode, since it is necessary to obtain local recognition results based on the intermediate image and the top image and the bottom image in the upper image, that is, it is necessary to consider all the upper images and bottom images at this time. Therefore, when the shooting device is in the second working mode and needs to obtain the sub-result corresponding to the first area based on the recognition result of the second area corresponding to the first area in the upper image, the second recognition module 30 (or the processor 200) executes the above step 051: performing a processing step on the upper image with the highest resolution, the processing step includes: if the recognition result of the second area corresponding to the first area in the intermediate image is the second result, then the recognition result of the second area corresponding to the first area in the upper image is used as the sub-result corresponding to the first area; if the recognition result of the second area corresponding to the first area in the upper image is the first result, then select an upper image arranged after the upper image, and repeat the processing step until the processing step is completed for all upper images. That is to say, when the shooting device is in the second working mode, the step is stopped when the recognition result of the second area corresponding to the first area in a certain upper image is the second result, or it can be stopped after the processing steps are performed on all upper images.

[0085] Specifically, the second recognition module 30 (or the processor 200) performs a processing step on the upper image with the highest resolution. The processing step includes: if the recognition result of the second area corresponding to the first area in the upper image is the second result, the recognition result of the second area corresponding to the first area in the upper image is used as the sub-result corresponding to the first area; if the recognition result of the second area corresponding to the first area in the upper image is the first result, an upper image arranged after the upper image is selected to repeat the processing step until all the upper images (including the middle layer image and the top layer image) are processed.

[0086] Similarly, in some embodiments, when the shooting device is in the second working mode and has performed processing steps on the upper image with the smallest resolution (i.e., the top image), then regardless of whether the recognition result of the second area corresponding to the first area in the upper image with the smallest resolution is the first result (the scene is not recognized) or the second result (specific scene category), the recognition result of the second area is used as the corresponding sub-result of the first area.

[0087] After obtaining all sub-results corresponding to the first region, a local recognition result is obtained based on all sub-results. Fig.12 After obtaining all sub-results corresponding to the first region, all sub-results are arranged according to the positions of the corresponding first regions in the underlying image to form a local recognition result. Fig.12 The figure on the right is a schematic diagram of a local recognition result in an embodiment, and the numbers in the figure represent sub-results. Among them, "0" represents the first result, and "A" and "B" represent two different scene categories in the second result. The sub-result arranged in the 1st row and 1st column of the local recognition result is "A", indicating that the sub-result corresponding to the first area arranged in the 1st row and 1st column of the underlying image (original image) is the scene category corresponding to "A" in the second result; the number arranged in the 5th row and 1st column of the local recognition result is 0, indicating that the sub-result corresponding to the first area arranged in the 5th row and 1st column of the underlying image (original image) is the first result, that is, no scene is recognized in the first area arranged in the 5th row and 1st column of the underlying image (original image).

[0088] After obtaining the overall recognition result and the local recognition result, the processing module 40 (or the processor 200) performs image enhancement processing on the original image according to the overall recognition result and the local recognition result to obtain the target image. Figure 1 and Fig.13 In some embodiments, step 04: performing image enhancement processing on the image according to the overall recognition result and the local recognition result to obtain the target image includes:

[0089] 041: Performing image enhancement processing on the original image according to the overall recognition result to obtain a first enhanced image;

[0090] 042: Performing image enhancement processing on the original image according to the local recognition result to obtain a second enhanced image;

[0091] 043: The first enhanced image and the second enhanced image are fused to obtain a target image.

[0092] Please combine Figure 2 and Figure 3 In some embodiments, the processing module 40 and the processor 200 are used to execute the methods in step 041, step 042 and step 043. That is, the processing module 40 and the processor 200 are also used to perform image enhancement processing on the original image according to the overall recognition result to obtain a first enhanced image; perform image enhancement processing on the original image according to the local recognition result to obtain a second enhanced image; and fuse the first enhanced image and the second enhanced image to obtain a target image.

[0093] After obtaining the overall recognition result, the processing module 40 (or the processor 200) performs image enhancement processing on the original image according to the overall recognition result to obtain a first enhanced image. By way of example, in some embodiments, the image processing device 100 (or the terminal 1000) pre-stores image enhancement processing algorithms corresponding to different overall recognition results. After obtaining the overall recognition result, the image enhancement processing algorithm corresponding to the overall recognition result can be selected to process the original image to obtain a first enhanced image. For example, when the overall recognition result is a sunset, the corresponding image enhancement processing algorithm requires dynamic enhancement of the image. Assuming that the overall recognition result obtained is a sunset, the processing module 40 (or the processor 200) dynamically enhances the original image to obtain a first enhanced image.

[0094] After obtaining the local recognition result, the processing module 40 (or the processor 200) performs image enhancement processing on the original image according to the local recognition result to obtain a second enhanced image. Specifically, in some embodiments, the image processing device 100 (or the terminal 1000) pre-stores image enhancement processing algorithms corresponding to different scene categories. When the local recognition result includes only one scene category, that is, when the sub-results of the second result in the local recognition result are all the same scene category, the processing module 40 (or the processor 200) uses all the first areas corresponding to the sub-results of the second result as the part to be processed according to the local recognition result. Then select the image enhancement processing algorithm corresponding to the scene category in the local recognition result, and process the part to be processed to obtain a second enhanced image.

[0095] For example, Fig.14The figure on the right is a schematic diagram of partial recognition results in some embodiments. The numbers and letters in the partial recognition results represent sub-results. Among them, "0" represents the first result, and "B" represents a scene category in the second result. It can be seen that Fig.14 In the local recognition results shown, the sub-results of the second result in the local recognition results are all of the same scene category. Assuming that the scene category represented by "B" is flowers, and when the scene category is flowers, the corresponding image enhancement processing algorithm needs to enhance the color vividness of the flower part. Fig.14 As shown, the sub-results arranged in the 1st row and 1st column, the 1st row and 2nd column, the 2nd row and 1st column, and the 2nd row and 2nd column of the local recognition result are all "B", that is, the sub-results corresponding to the first area arranged in the 1st row and 1st column, the 1st row and 2nd column, the 2nd row and 1st column, and the 2nd row and 2nd column of the original image (underlying image) are the second results. Therefore, the processing module 40 (or the processor 200) takes the first area arranged in the 1st row and 1st column, the 1st row and 2nd column, the 2nd row and 1st column, and the 2nd row and 2nd column of the original image (underlying image) as the part to be processed. That is, the part to be processed is the area where the flowers are located. Subsequently, the processing module 40 (or the processor 200) enhances the color vividness of the part to be processed (that is, the first area arranged in the 1st row and 1st column, the 1st row and 2nd column, the 2nd row and 1st column, and the 2nd row and 2nd column of the original image (underlying image)) to obtain a second enhanced image.

[0096] See also Fig.15 In some embodiments, if at least two sub-results in the local recognition result are of different scene categories, step 042: performing image enhancement processing on the original image according to the local recognition result to obtain a second enhanced image includes:

[0097] 0421: Perform corresponding image enhancement on the image according to different scene categories and local recognition results to obtain multiple frames of second enhanced images.

[0098] Please combine Figure 2 and Figure 3 In some embodiments, the processing module 40 and the processor 200 are also used to execute the method in step 0421. That is, the processing module 40 and the processor 200 are also used to perform corresponding image enhancement on the image according to different sub-results to obtain multiple frames of second enhanced images.

[0099] Specifically, in some embodiments, the image processing device 100 (or terminal 1000) pre-stores algorithms for image enhancement processing corresponding to different scene categories. When at least two sub-results in the local recognition result are of different scene categories, the processing module 40 (or processor 200) selects a scene category from a plurality of different scene categories in the local recognition result, and the processing module 40 (or processor 200) uses all first areas corresponding to the sub-results of the same scene category as the selected scene category as the part to be processed according to the local recognition result. Then, the algorithm for image enhancement processing corresponding to the scene category is selected, and the part to be processed is processed to obtain a frame of a second enhanced image corresponding to the scene category. Subsequently, the processing module 40 (or processor 200) selects another scene category from a plurality of different scene categories in the local recognition result, and repeats the above steps until all scene categories in the local recognition result are selected, so that multiple frames of second enhanced images corresponding to different scene categories are obtained.

[0100] For example, Fig.12 The figure on the right is a schematic diagram of partial recognition results in some embodiments. The numbers and letters in the partial recognition results represent sub-results. Among them, "0" represents the first result, and "A" and "B" represent two different scene categories in the second result. It can be seen that Fig.12 The local recognition result shown in the figure on the right includes two different scene categories. Assuming that the scene category represented by "A" is blue sky, and when the scene category is blue sky, the corresponding image enhancement processing algorithm needs to increase the saturation of the blue sky part; the scene category represented by "B" is flowers, and when the scene category is flowers, the corresponding image enhancement processing algorithm needs to increase the color vividness of the flower part. The processing module 40 (or processor 200) selects the scene category corresponding to "A" from multiple different scene categories in the local recognition result. Fig.12 As shown, the sub-results arranged in the 1st row and 1st column, the 1st row and 2nd column, the 1st row and 3rd column, the 1st row and 4th column, the 2nd row and 1st column, the 2nd row and 2nd column, and the 3rd row and 1st column of the local recognition result are all "A", that is, the scene category in the sub-results corresponding to the first area arranged in the 1st row and 1st column, the 1st row and 2nd column, the 1st row and 3rd column, the 1st row and 4th column, the 2nd row and 1st column, the 2nd row and 2nd column, and the 3rd row and 1st column of the original image (underlying image) is blue sky. Therefore, the processing module 40 (or the processor 200) uses the first area arranged in the 1st row and 1st column, the 1st row and 2nd column, the 1st row and 3rd column, the 1st row and 4th column, the 2nd row and 1st column, the 2nd row and 2nd column, and the 3rd row and 1st column of the original image (underlying image) as the part to be processed. That is, at this time, the part to be processed is the area where the blue sky is located. Then, the processing module 40 (or the processor 200) increases the saturation of the portion to be processed to obtain a second enhanced image corresponding to the blue sky scene. Then, the scene category corresponding to "B" is selected. Fig.12As shown, the sub-results arranged in the 2nd row, 3rd column, 3rd row, 2nd column, 3rd row, 3rd column, 3rd row, 4th column, 4th row, 1st column, and 4th row, 2nd column of the local recognition result are all "B", that is, the scene category in the sub-results corresponding to the first area arranged in the 2nd row, 3rd column, 3rd row, 2nd column, 3rd row, 3rd column, 3rd row, 4th column, 4th row, 1st column, and 4th row, 2nd column of the original image (underlying image) is flowers. Therefore, the processing module 40 (or the processor 200) uses the first area arranged in the 2nd row, 3rd column, 3rd row, 2nd column, 3rd row, 3rd column, 3rd row, 4th column, 4th row, 1st column, and 4th row, 2nd column of the original image (underlying image) as the part to be processed. That is, at this time, the part to be processed is the area where the flowers are located. Subsequently, the processing module 40 (or the processor 200) increases the color vividness of the part to be processed to obtain a second enhanced image corresponding to the flower scene.

[0101] After obtaining the first enhanced image and the second enhanced image, the first enhanced image and the second enhanced image are fused to obtain the target image. In this way, image enhancement can be performed for the entire scene of the original image, and corresponding image enhancement can be performed for different local scenes of the original image, which can avoid using only one enhancement method for the entire image, and is conducive to improving the quality of the target image finally obtained.

[0102] Specifically, see Fig.16 and Fig.17 In some embodiments, the image processing method further comprises:

[0103] 07: Obtain the key area of ​​the original image based on the overall recognition result and / or the local recognition result;

[0104] At this time, step 043: fusing the first enhanced image and the second enhanced image to obtain the target image includes:

[0105] 0431: Traverse all pixels in the original image and determine whether each pixel is in the key area. If the current pixel is in the key area, the pixel value of the pixel at the position corresponding to the current pixel in the target image is obtained according to the pixel value of the pixel at the position corresponding to the current pixel in the first enhanced image, the first weight corresponding to the first enhanced image, the pixel value of the pixel at the position corresponding to the current pixel in the second enhanced image, and the second weight corresponding to the second enhanced image, wherein the first weight is less than the second weight; if the current pixel is in the non-key area, the pixel value of the pixel at the position corresponding to the current pixel in the target image is obtained according to the pixel value of the pixel at the position corresponding to the current pixel in the first enhanced image, the third weight corresponding to the first enhanced image, the pixel value of the pixel at the position corresponding to the current pixel in the second enhanced image, and the fourth weight corresponding to the second enhanced image, wherein the third weight is greater than the fourth weight.

[0106] Please combine Figure 2 and Figure 3 In some embodiments, the image processing device 100 further includes a second acquisition module 50, and the second acquisition module 50 and the processor 200 are both used to execute the method in step 07. The processing module 40 and the processor 200 are also used to execute the method in step 0431. That is, the second acquisition module 50 and the processor 200 are both used to acquire the key area of ​​the original image according to the overall recognition result and / or the local recognition result. The processing module 40 and the processor 200 are also used to traverse all pixels in the original image and determine whether each pixel is in the key area. If the current pixel is in the key area, the pixel value of the pixel at the position corresponding to the current pixel in the target image is obtained according to the pixel value of the pixel at the position corresponding to the current pixel in the first enhanced image, the first weight corresponding to the first enhanced image, the pixel value of the pixel at the position corresponding to the current pixel in the second enhanced image, and the second weight corresponding to the second enhanced image, wherein the first weight is less than the second weight; if the current pixel is in the non-key area, the pixel value of the pixel at the position corresponding to the current pixel in the target image is obtained according to the pixel value of the pixel at the position corresponding to the current pixel in the first enhanced image, the third weight corresponding to the first enhanced image, the pixel value of the pixel at the position corresponding to the current pixel in the second enhanced image, and the fourth weight corresponding to the second enhanced image, wherein the third weight is greater than the fourth weight.

[0107] In some embodiments, it is also possible to use the overall recognition result and / or the local recognition result. For example, in some implementations, the key area of ​​the original image can be obtained based only on the local recognition result. For example, in one example, based on the local recognition result, all the first areas corresponding to the sub-results being the second result are used as the key areas of the original image. In another example, based on the local recognition result, all the first areas corresponding to the sub-results being the preset scene categories are used as the key areas of the original image. Among them, the preset scene category can be the scene category set by default before the image processing device 100 (or terminal 1000) leaves the factory; it can also be a scene category set by the user according to his or her own needs; it can also be based on the local recognition result to count the number of sub-results corresponding to different scene categories, and select the scene category corresponding to the largest number as the preset category.

[0108] In some implementations, the key areas of the original image can also be obtained based on the overall recognition result. For example, in one example, a relationship table between the overall recognition result and the corresponding key areas is preset. After the overall recognition result of the original image is obtained, the key areas are obtained based on the relationship table between the overall recognition result and the corresponding key areas. For example, suppose that the relationship table between the overall recognition result and the corresponding key areas records: when the overall recognition result is the first preset result, the middle area of ​​the image is the key area; when the overall recognition result is the second preset result, the upper left corner area of ​​the image is the key area. Then, when the overall recognition result obtained is the second preset result, the upper left corner area of ​​the original image is used as the key area.

[0109] In some implementations, the key areas of the original image can also be obtained based on the overall recognition result and the local recognition result. For example, in one example, there are preset categories of scenes of interest corresponding to different overall recognition results. After the overall recognition result of the original image is obtained, if the sub-results in the local recognition result are the categories of scenes of interest corresponding to the overall recognition result, all the first areas corresponding to the sub-results for the scenes of interest are used as the key areas of the original image. For example, assuming that when the overall recognition result is a portrait, the corresponding category of scenes of interest is a face. If the overall recognition result obtained is a portrait, and the sub-results in the local recognition result are faces, all the first areas corresponding to the sub-results for faces are used as the key areas of the original image.

[0110] After acquiring the key area of ​​the original image, the first enhanced image and the second enhanced image are fused in combination with the key area of ​​the original image. Specifically, the processing module 40 (or the processor 200) can traverse all pixels in the original image and determine whether each pixel is in the key area. If the current pixel is in the key area, the pixel value of the pixel corresponding to the current pixel in the target image is obtained according to the pixel value of the pixel at the position corresponding to the current pixel in the first enhanced image, the first weight corresponding to the first enhanced image, the pixel value of the pixel at the position corresponding to the current pixel in the second enhanced image, and the second weight corresponding to the second enhanced image, wherein the first weight is less than the second weight. In one example, the pixel value of the pixel at the position corresponding to the current pixel in the target image is equal to the average of the product of the pixel value of the pixel at the position corresponding to the current pixel in the first enhanced image and the first weight, and the product of the pixel value of the pixel at the position corresponding to the current pixel in the second enhanced image and the second weight. If the current pixel is located in a non-key area, the pixel value of the pixel corresponding to the current pixel in the target image is obtained according to the pixel value of the pixel corresponding to the current pixel in the first enhanced image, the third weight corresponding to the first enhanced image, the pixel value of the pixel corresponding to the current pixel in the second enhanced image, and the fourth weight corresponding to the second enhanced image, wherein the third weight is greater than the fourth weight. In one example, the pixel value of the pixel corresponding to the current pixel in the target image is equal to the average of the product of the pixel value of the pixel corresponding to the current pixel in the first enhanced image and the third weight, and the pixel value of the pixel corresponding to the current pixel in the second enhanced image and the fourth weight. Since the second enhanced image is given a larger weight in the key area and the first enhanced image is given a larger weight in the non-key area, the local or overall effect of the image can be freely highlighted, which is conducive to making the effect of the target image finally obtained more in line with the user's expectations.

[0111] For example, Fig.18 and Fig.19 As shown, the pixels arranged in the 1st row and 1st column of the original image are located in the key area, and the pixels arranged in the 5th row and 5th column of the original image are located in the non-key area. Among them, the pixel value of the pixel arranged in the 1st row and 1st column of the first enhanced image is Z1, the pixel value of the pixel arranged in the 1st row and 1st column of the second enhanced image is Z2, the pixel value of the pixel arranged in the 5th row and 5th column of the first enhanced image is Z3, the pixel value of the pixel arranged in the 5th row and 5th column of the second enhanced image is Z4, the first weight is k1, the second weight is k2, the third weight is k3, and the fourth weight is k4, then the pixel value of the pixel arranged in the 1st row and 1st column of the target image is R=(Z1*k1+Z2*k2) / 2, and the pixel value of the pixel arranged in the 5th row and 5th column of the target image is R=(Z3*k3+Z4*k4) / 2.

[0112] See also Fig. 20 The embodiment of the present application further provides a non-volatile computer-readable storage medium 400 including a computer program 401. When the computer program 401 is executed by one or more processors 200, the processor 200 executes the image processing methods in 01, 02, 03, 04, 031, 032, 033, 0321, 0322, 0323, 051, 052, 041, 042, 043, 0421, 07, and 0431.

[0113] For example, please combine Figure 1 , when the computer program 401 is executed by one or more processors 200, the processor 200 performs the following method:

[0114] 01: Obtain an image pyramid according to the original image, where the image pyramid includes bottom-layer images with successively decreasing resolutions and at least one upper-layer image, where the bottom-layer image is the original image;

[0115] 02: Perform scene recognition on the upper layer image with the lowest resolution to obtain the overall recognition result;

[0116] 03: Perform scene recognition on multiple areas in the upper image and the bottom image respectively to obtain local recognition results;

[0117] 04: Perform image enhancement processing on the original image based on the overall recognition results and the local recognition results to obtain the target image.

[0118] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0119] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0120] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. An image processing method, characterized in that: include: Acquire an image pyramid according to the original image, wherein the image pyramid includes bottom-layer images with successively decreasing resolutions and at least one upper-layer image, wherein the bottom-layer image is the original image; Performing scene recognition on the upper layer image with the lowest resolution to obtain an overall recognition result; Dividing the bottom layer image into a plurality of first regions, dividing the upper layer image into a plurality of second regions, and performing scene recognition on each of the first regions and each of the second regions respectively to obtain a recognition result of each of the first regions and a recognition result of each of the second regions, wherein the size of the first region is the same as the size of the second region; If the recognition result of the first area is the second result, taking the recognition result of the first area as the corresponding sub-result of the first area; Or, if the recognition result of the first area is the second result, then according to the recognition result of the second area corresponding to the first area in the upper image, a sub-result corresponding to the first area is obtained; and a local recognition result is obtained according to all the sub-results; wherein a plurality of the sub-results correspond one-to-one to a plurality of the first areas of the underlying image, and the local recognition result includes a plurality of sub-results; and the second result includes the recognition of a specific scene category; and The original image is subjected to image enhancement processing according to the overall recognition result and the local recognition result image to obtain a target image.

2. The image processing method according to claim 1, characterized in that: The method further comprises: If the recognition result of the first area is a first result, the sub-result corresponding to the first area is obtained according to the recognition result of the second area corresponding to the first area in the upper image, wherein the first result indicates that the scene is not recognized.

3. The image processing method according to claim 1 or 2, characterized in that: There are multiple upper-layer images, and obtaining the sub-result corresponding to the first region according to the recognition result of the second region corresponding to the first region in the upper-layer image includes: Performing a processing step on the upper-layer image with the highest resolution, the processing step comprising: if the recognition result of the second area corresponding to the first area in the upper-layer image is the second result, using the recognition result of the second area corresponding to the first area in the upper-layer image as the sub-result corresponding to the first area; if the recognition result of the second area corresponding to the first area in the upper-layer image is the first result, selecting an upper-layer image arranged after the upper-layer image, and repeatedly performing the processing step until the processing step is completed for all the upper-layer images; or Execute processing steps on the upper image with the highest resolution, the processing steps comprising: if the recognition result of the second area corresponding to the first area in the upper image is the second result, then use the recognition result of the second area corresponding to the first area in the upper image as the sub-result corresponding to the first area; if the recognition result of the second area corresponding to the first area in the upper image is the first result, select an upper image arranged after the upper image, and repeat the processing steps until the processing steps are completed for all the upper images except the upper image with the lowest resolution.

4. The image processing method according to claim 1, characterized in that: The performing image enhancement processing on the image according to the overall recognition result and the local recognition result to obtain a target image includes: Performing image enhancement processing on the original image according to the overall recognition result to obtain a first enhanced image; Performing image enhancement processing on the original image according to the local recognition result to obtain a second enhanced image; The target image is obtained by fusing the first enhanced image and the second enhanced image.

5. The image processing method according to claim 4, characterized in that: If at least two sub-results in the local recognition result are of different scene categories, performing image enhancement processing on the image according to the local recognition result to obtain a second enhanced image includes: The image is enhanced accordingly according to the different scene categories and the local recognition results to obtain multiple frames of the second enhanced image.

6. The image processing method according to claim 4, characterized in that: Also includes: Acquire a key area of ​​the original image according to the overall recognition result and / or the local recognition result; The fusing the first enhanced image and the second enhanced image to obtain the target image comprises: Traversing all pixels in the original image and determining whether each pixel is in the key area, if the current pixel is in the key area, obtaining the pixel value of the pixel at the position corresponding to the current pixel in the target image according to the pixel value of the pixel at the position corresponding to the current pixel in the first enhanced image, the first weight corresponding to the first enhanced image, the pixel value of the pixel at the position corresponding to the current pixel in the second enhanced image, and the second weight corresponding to the second enhanced image, wherein the first weight is less than the second weight; If the current pixel is located in a non-key area, the pixel value of the pixel at the position corresponding to the current pixel in the target image is obtained based on the pixel value of the pixel at the position corresponding to the current pixel in the first enhanced image, the third weight corresponding to the first enhanced image, the pixel value of the pixel at the position corresponding to the current pixel in the second enhanced image, and the fourth weight corresponding to the second enhanced image, wherein the third weight is greater than the fourth weight.

7. The image processing method according to claim 1, characterized in that: The upper layer image includes a top layer image, and the top layer image is an image with the lowest resolution among the upper layer images; The performing scene recognition on the upper layer image with the lowest resolution to obtain an overall recognition result includes: Scene recognition is performed on the top-level image to obtain an overall recognition result.

8. The image processing method according to claim 1, characterized in that: The upper-layer images include a top-layer image and a middle-layer image, wherein the top-layer image is the image with the lowest resolution among the upper-layer images, and all images in the upper-layer images except the top-layer image are middle-layer images; if the shooting device is in the first working mode, the local recognition result is obtained by using the middle-layer image and the top-layer image; if the shooting device is in the second working mode, the local recognition result is obtained by using the middle-layer image, the top-layer image, and the bottom-layer image.

9. An image processing device, characterized in that: include: A first acquisition module, configured to acquire an image pyramid according to an original image, wherein the image pyramid includes a bottom layer image with successively decreasing resolutions and at least one upper layer image, wherein the bottom layer image is the original image; A first recognition module, used for performing scene recognition on the upper layer image with the lowest resolution to obtain an overall recognition result; a second recognition module, configured to divide the bottom layer image into a plurality of first regions, divide the upper layer image into a plurality of second regions, and perform scene recognition on each of the first regions and each of the second regions respectively, so as to obtain a recognition result of each of the first regions and a recognition result of each of the second regions, wherein the size of the first region is the same as the size of the second region; if the recognition result of the first region is the second result, the recognition result of the first region is used as a corresponding sub-result of the first region; Or, if the recognition result of the first area is the second result, then according to the recognition result of the second area corresponding to the first area in the upper image, a sub-result corresponding to the first area is obtained; and a local recognition result is obtained according to all the sub-results; wherein a plurality of the sub-results correspond one-to-one to a plurality of the first areas of the underlying image, and the local recognition result includes a plurality of sub-results; and the second result includes the recognition of a specific scene category; A processing module is used to perform image enhancement processing on the image according to the overall recognition result and the local recognition result to obtain a target image.

10. A terminal, characterized in that: The terminal comprises: One or more processors, memory; and One or more programs, wherein one or more of the programs are stored in the memory and executed by one or more of the processors, and the programs include instructions for executing the image processing method according to any one of claims 1 to 8.

11. A non-volatile computer-readable storage medium storing a computer program, which, when executed by one or more processors, implements the image processing method according to any one of claims 1 to 8.

Images