Image processing method, apparatus, device, and medium

Abstract

The image processing method, apparatus, device, and medium provided in this application include: acquiring original image data containing sparse phase detection pixels through a preset interactive interface, and extracting first pixel data corresponding to the sparse phase detection pixels, performing pixel compensation processing on it to obtain second pixel data; performing multi-scale downsampling processing on the original image data and the second pixel data respectively to obtain first and second multi-level image features; fusing the first multi-level image features and the second multi-level image features of the same level to obtain fused image data, and performing multi-scale upsampling processing on it to obtain upsampled image data; obtaining multi-level image detail residuals based on the first multi-level image features, the second multi-level image features, and the upsampled image data, and combining them with the upsampled image data to obtain target image data, and then outputting it to the preset interactive interface, thereby improving image clarity.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus, device and medium. Background Technology

[0002] With the rapid development of mobile internet and the widespread application of smart terminals, users' requirements for image quality are increasing, and image sharpness has become a core indicator for measuring the performance of imaging devices. In scenarios with extremely high image quality requirements, such as mobile phones, digital cameras, and drones, image sensors based on sparse phase detection (Sparse PD) are widely used. However, due to the sparse distribution of PD pixels, compensation from neighboring pixels is required when generating a complete image, resulting in the loss of original texture information. This is especially true in low light, dynamic, or out-of-focus conditions, where image sharpness decreases significantly. Therefore, how to improve image sharpness while preserving the original information of PD pixels has become a challenging and significant issue.

[0003] In existing technologies, the processing flow for image sensors containing sparse phase detection (Sparse PD) pixels includes: first, acquiring the original image output by the sensor containing sparse PD pixels; since PD pixels only cover a part of the image area, compensation is required through adjacent pixels in the same channel; a common method is neighbor pixel mean interpolation, that is, taking the average value of surrounding pixels to fill the missing PD pixel area; the compensated image is used as the final output original image for subsequent image signal processor (ISP) and high dynamic range (HDR) synthesis and other backend algorithms.

[0004] However, existing technologies suffer from insufficient image clarity due to factors such as loss of original information and unutilization of out-of-focus information. Summary of the Invention

[0005] The image processing methods, apparatus, devices, and media provided in this application are used to achieve the technical effect of improving image clarity.

[0006] In a first aspect, this application provides an image processing method, comprising:

[0007] The original image data containing sparse phase detection pixels is obtained through a preset interactive interface;

[0008] Based on the original image data, the first pixel data corresponding to the sparse phase detection pixels is extracted, and pixel compensation processing is performed on the first pixel data to obtain the compensated second pixel data.

[0009] Multi-scale downsampling processing is performed on the original image data and the second pixel data respectively to obtain the corresponding first multi-level image features and second multi-level image features, wherein the first multi-level image features are the multi-level features corresponding to the original image data, and the second multi-level image features are the multi-level features corresponding to the second pixel data.

[0010] The first and second multi-level image features at the same level are fused to obtain fused image data;

[0011] Multi-scale upsampling processing is performed on the fused image data to obtain upsampled image data;

[0012] Based on the first multi-level image features, the second multi-level image features, and the upsampled image data, the multi-level image detail residuals are obtained;

[0013] Based on the multi-level image detail residuals and upsampled image data, the target image data is obtained and output to the preset interactive interface.

[0014] In one possible implementation, multi-scale downsampling processing is performed on the original image data and the second pixel data respectively to obtain corresponding first multi-level image features and second multi-level image features, including:

[0015] Obtain the preset image pyramid algorithm;

[0016] Based on the image pyramid algorithm, filtering, convolution and downsampling are performed sequentially on the original image data to iteratively generate the first multi-level image features.

[0017] According to the image pyramid algorithm, filtering, convolution and downsampling are performed on the second pixel data in sequence to iteratively generate a second multi-level image feature that matches the size of the first multi-level image feature.

[0018] In one possible implementation, the first multi-level image features and the second multi-level image features of the same level are fused to obtain fused image data, including:

[0019] Based on the second multi-level image features, determine the target level that is consistent with the second pixel data size;

[0020] The second multi-level image features are decomposed into left phase detection pixel features and right phase detection pixel features;

[0021] The right phase detection pixel features corresponding to the target level are used as the reference features, and the left phase detection pixel features corresponding to the target level are subjected to pixel translation processing to obtain the translated left phase detection pixel features.

[0022] Based on the left phase detection pixel features and the reference features, determine the corresponding sum of absolute differences;

[0023] The target alignment position is determined based on the minimum value of the sum of absolute differences;

[0024] Based on the target alignment position, the left phase detection pixel features and the right phase detection pixel features are synthesized to obtain the center phase detection pixel features.

[0025] The central phase detection pixel features and the first multi-level image features corresponding to the target level are weighted and superimposed to obtain fused image data.

[0026] In one possible implementation, based on the scene acquisition parameters, the center phase detection pixel features and the first multi-level image features corresponding to the target level are weighted and superimposed to obtain fused image data, including:

[0027] Based on the original image data, obtain the corresponding acquisition scene parameters;

[0028] Based on the scene parameters, adjust the fusion weights of the central phase detection pixel features and the first multi-level image features;

[0029] Based on the adjusted fusion weights, the center phase detection pixel features and the first multi-level image features are weighted and calculated to obtain the fused image data.

[0030] In one possible implementation, the fusion weights of the center phase detection pixel features and the first multi-level image features are adjusted according to the scene acquisition parameters, including:

[0031] Obtain the preset data collection scenario parameter matching table;

[0032] Based on the parameter matching table for the collected scene, determine the parameter type of the collected scene parameters;

[0033] If the parameter type is a low-light scene parameter, increase the fusion weight of the central phase detection pixel features;

[0034] If the scene parameters collected are high dynamic scene parameters, reduce the fusion weight of the central phase detection pixel features;

[0035] If the scene parameters collected are standard scene parameters, the fusion weights of the center phase detection pixel features and the first multi-level image features are set to be equally distributed.

[0036] In one possible implementation, multi-level image detail residuals are obtained based on first multi-level image features, second multi-level image features, and upsampled image data, including:

[0037] The upsampled image data is split into sub-upsampled image data that match the size of each level of the first multi-level image features and the second multi-level image features;

[0038] Based on the first multi-level image features and the sub-upsampled image data, determine the first pixel difference between the first multi-level image features of each level and the sub-upsampled image data of the corresponding size;

[0039] Based on the second multi-level image features and the sub-upsampled image data, determine the second pixel difference between the second multi-level image features of each level and the sub-upsampled image data of the corresponding size;

[0040] Based on the first pixel difference and the second pixel difference, the image detail residuals corresponding to each level are integrated to obtain the image detail residuals. Each level corresponds to one image detail residual.

[0041] The image detail residuals corresponding to each level are summarized to obtain multi-level image detail residuals.

[0042] In one possible implementation, target image data is obtained based on multi-level image detail residuals and upsampled image data, including:

[0043] Based on the multi-level image detail residuals, obtain the image detail residuals corresponding to each level;

[0044] The corresponding generation order of the layers is determined based on the first and second multi-layer image features;

[0045] Based on the generation order of the layers, determine the superposition order of the image detail residuals corresponding to each layer;

[0046] According to the stacking order, the image detail residuals of the bottom level are stacked onto the sub-upsampled image data of the corresponding size to obtain the image data after the bottom stack, and the image data after the bottom stack is used as the input data for the next level of upsampling processing;

[0047] Upsampling is performed on the input data. After the upsampling is completed, the image detail residual overlay processing of the previous level is iteratively performed until the processing of each level is completed, and the image data after the highest level overlay is obtained.

[0048] The image data after the highest layer is overlaid is used as the target image data.

[0049] Secondly, this application provides an image processing apparatus, comprising:

[0050] The first acquisition module is used to acquire raw image data containing sparse phase detection pixels through a preset interactive interface;

[0051] The first processing module is used to extract the first pixel data corresponding to the sparse phase detection pixels based on the original image data, and perform pixel compensation processing on the first pixel data to obtain the compensated second pixel data.

[0052] The second processing module is used to perform multi-scale downsampling processing on the original image data and the second pixel data respectively to obtain the corresponding first multi-level image features and second multi-level image features, wherein the first multi-level image features are the multi-level features corresponding to the original image data, and the second multi-level image features are the multi-level features corresponding to the second pixel data.

[0053] The fusion module is used to fuse the first and second multi-level image features at the same level to obtain fused image data.

[0054] The third processing module is used to perform multi-scale upsampling processing on the fused image data to obtain upsampled image data.

[0055] The fourth processing module is used to obtain multi-level image detail residuals based on the first multi-level image features, the second multi-level image features, and the upsampled image data.

[0056] The fifth processing module is used to obtain target image data based on multi-level image detail residuals and upsampled image data, and output the target image data to a preset interactive interface.

[0057] In one possible implementation, the second processing module is further configured to:

[0058] Obtain the preset image pyramid algorithm;

[0059] Based on the image pyramid algorithm, filtering, convolution and downsampling are performed sequentially on the original image data to iteratively generate the first multi-level image features.

[0060] According to the image pyramid algorithm, filtering, convolution and downsampling are performed on the second pixel data in sequence to iteratively generate a second multi-level image feature that matches the size of the first multi-level image feature.

[0061] In one possible implementation, the fusion module is also used for:

[0062] Based on the second multi-level image features, determine the target level that is consistent with the second pixel data size;

[0063] The second multi-level image features are decomposed into left phase detection pixel features and right phase detection pixel features;

[0064] The right phase detection pixel features corresponding to the target level are used as the reference features, and the left phase detection pixel features corresponding to the target level are subjected to pixel translation processing to obtain the translated left phase detection pixel features.

[0065] Based on the left phase detection pixel features and the reference features, determine the corresponding sum of absolute differences;

[0066] The target alignment position is determined based on the minimum value of the sum of absolute differences;

[0067] Based on the target alignment position, the left phase detection pixel features and the right phase detection pixel features are synthesized to obtain the center phase detection pixel features.

[0068] The central phase detection pixel features and the first multi-level image features corresponding to the target level are weighted and superimposed to obtain fused image data.

[0069] In one possible implementation, the fusion module is also used for:

[0070] Based on the original image data, obtain the corresponding acquisition scene parameters;

[0071] Based on the scene parameters, adjust the fusion weights of the central phase detection pixel features and the first multi-level image features;

[0072] Based on the adjusted fusion weights, the center phase detection pixel features and the first multi-level image features are weighted and calculated to obtain the fused image data.

[0073] In one possible implementation, the fusion module is also used for:

[0074] Obtain the preset data collection scenario parameter matching table;

[0075] Based on the parameter matching table for the collected scene, determine the parameter type of the collected scene parameters;

[0076] If the parameter type is a low-light scene parameter, increase the fusion weight of the central phase detection pixel features;

[0077] If the scene parameters collected are high dynamic scene parameters, reduce the fusion weight of the central phase detection pixel features;

[0078] If the scene parameters collected are standard scene parameters, the fusion weights of the center phase detection pixel features and the first multi-level image features are set to be equally distributed.

[0079] In one possible implementation, the fourth processing module is further configured to:

[0080] The upsampled image data is split into sub-upsampled image data that match the size of each level of the first multi-level image features and the second multi-level image features;

[0081] Based on the first multi-level image features and the sub-upsampled image data, determine the first pixel difference between the first multi-level image features of each level and the sub-upsampled image data of the corresponding size;

[0082] Based on the second multi-level image features and the sub-upsampled image data, determine the second pixel difference between the second multi-level image features of each level and the sub-upsampled image data of the corresponding size;

[0083] Based on the first pixel difference and the second pixel difference, the image detail residuals corresponding to each level are integrated to obtain the image detail residuals. Each level corresponds to one image detail residual.

[0084] The image detail residuals corresponding to each level are summarized to obtain multi-level image detail residuals.

[0085] In one possible implementation, the fifth processing module is further configured to:

[0086] Based on the multi-level image detail residuals, obtain the image detail residuals corresponding to each level;

[0087] The corresponding generation order of the layers is determined based on the first and second multi-layer image features;

[0088] Based on the generation order of the layers, determine the superposition order of the image detail residuals corresponding to each layer;

[0089] According to the stacking order, the image detail residuals of the bottom level are stacked onto the sub-upsampled image data of the corresponding size to obtain the image data after the bottom stack, and the image data after the bottom stack is used as the input data for the next level of upsampling processing;

[0090] Upsampling is performed on the input data. After the upsampling is completed, the image detail residual overlay processing of the previous level is iteratively performed until the processing of each level is completed, and the image data after the highest level overlay is obtained.

[0091] The image data after the highest layer is overlaid is used as the target image data.

[0092] Thirdly, this application provides an image processing device, including: a memory and a processor;

[0093] The memory stores the instructions that the computer executes;

[0094] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0095] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible embodiments of the first aspect.

[0096] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0097] This application provides an image processing method, apparatus, device, and medium. It acquires original image data containing sparse phase detection pixels through a preset interactive interface, providing a basic data source for subsequent image processing; extracts and compensates for the pixel data corresponding to the sparse phase detection pixels, filling in the information gaps of the sparse phase detection pixels and restoring complete phase detection pixel information; performs multi-scale downsampling processing on the original image data and the compensated pixel data to extract image features at different levels, realizing the mining and extraction of multi-scale image information; fuses two types of multi-level image features at the same level, integrating the effective information of phase detection pixel features and original image features to compensate for the feature deficiencies of the original image; performs multi-scale upsampling processing on the fused image data to restore the image's size hierarchy, preparing for detail enhancement; combines the two types of multi-level features with the upsampled image data to obtain multi-level detail residuals, extracting detail information at each level of the image; obtains and outputs the target image based on the residuals and upsampled data, superimposing the detail information back into the image, ultimately achieving the technical effect of improving image clarity. Attached Figure Description

[0098] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0099] Figure 1 This application provides a schematic diagram of an application data processing system architecture.

[0100] Figure 2 A flowchart illustrating the image processing method provided in the embodiments of this application. Figure 1 ;

[0101] Figure 3 This application provides an image pyramid multi-scale downsampling hierarchy generation map for embodiments of the present application;

[0102] Figure 4 A flowchart illustrating the image processing method provided in the embodiments of this application. Figure 2 ;

[0103] Figure 5This is a correlation diagram of defocus and offset of phase detection pixel features provided in an embodiment of this application;

[0104] Figure 6 A graph showing the relationship between the sum of absolute differences and pixel translation amount provided in the embodiments of this application;

[0105] Figure 7 A flowchart illustrating the image processing method provided in the embodiments of this application. Figure 3 ;

[0106] Figure 8 A flowchart illustrating the image processing method provided in the embodiments of this application. Figure 4 ;

[0107] Figure 9 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application;

[0108] Figure 10 This is a schematic diagram of the structure of the image processing device provided in the embodiments of this application.

[0109] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0110] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0111] Due to factors such as loss of original information and unutilization of out-of-focus information, existing technologies suffer from insufficient image clarity.

[0112] To address the aforementioned issues, this application provides an image processing method, apparatus, device, and medium. The method acquires raw image data containing sparse phase detection pixels through a pre-defined interactive interface, providing a basic data source for subsequent image processing. It extracts and compensates for the pixel data corresponding to the sparse phase detection pixels, filling in the information gaps and restoring complete phase detection pixel information. Multi-scale downsampling is performed on the raw image data and the compensated pixel data to extract image features at different levels, achieving multi-scale information mining and extraction. Two types of multi-level image features at the same level are fused, integrating the effective information of phase detection pixel features and raw image features to compensate for the feature deficiencies of the raw image. Multi-scale upsampling is performed on the fused image data to restore the image's size hierarchy, preparing for detail enhancement. Multi-level detail residuals are obtained by combining the two types of multi-level features with the upsampled image data, extracting detail information at each level of the image. The target image is obtained and output based on the residuals and upsampled data, and the detail information is superimposed back into the image, ultimately achieving the technical effect of improving image clarity.

[0113] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0114] Figure 1 This is a schematic diagram of an application data processing system architecture provided in an embodiment of this application. The application data processing system is a computer device. Figure 1 As shown, the above architecture includes at least one of a data acquisition device 101, a processing device 102, and a display device 103.

[0115] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the architecture of the application data processing system. In other feasible embodiments of this application, the above architecture may include more or fewer components than illustrated, or combine some components, or split some components, or arrange different components, which can be determined according to the actual application scenario and is not limited here. Figure 1 The components shown can be implemented in hardware, software, or a combination of both.

[0116] In the specific implementation process, the data acquisition device 101 may include an input / output interface or a communication interface, and the data acquisition device 101 can be connected to the processing device through the input / output interface or the communication interface.

[0117] The processing device 102 can first obtain the original image data containing sparse phase detection pixels from the preset interactive interface, extract and compensate the sparse phase detection pixel data, and then perform multi-scale downsampling processing on the original image data and the compensated pixel data to obtain corresponding multi-level features. By fusing the same-level features, the effective image information is integrated. After multi-scale upsampling of the fused data, the residual details of each level of the image are extracted. Finally, the residuals and upsampled data are combined to generate a clear target image and output it to the preset interactive interface. This fully utilizes the effective information of sparse phase detection pixels to make up for the information loss of the original image and improves the image clarity.

[0118] The display device 103 can also be a touch screen or the screen of a terminal device, used to receive user commands while displaying the above-mentioned content, so as to realize interaction with the user.

[0119] It should be understood that the aforementioned processing device can be implemented by a processor reading instructions from memory and executing those instructions, or it can be implemented by a chip circuit.

[0120] Furthermore, the network architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0121] Figure 2 A flowchart illustrating the image processing method provided in the embodiments of this application. Figure 1 ,like Figure 2 As shown, this embodiment provides an image processing method, including:

[0122] S201. Obtain the original image data containing sparse phase detection pixels through a preset interactive interface.

[0123] The system obtains raw image data containing sparse phase detection pixels through a pre-set interactive interface. This raw image data, which is the basic input data for all subsequent image processing steps, is the raw image data with sparse phase detection.

[0124] S202. Based on the original image data, extract the first pixel data corresponding to the sparse phase detection pixel, and perform pixel compensation processing on the first pixel data to obtain the compensated second pixel data.

[0125] From the acquired raw image data, the first pixel data corresponding to the sparse phase detection pixels is extracted, and then pixel compensation processing is performed on these first pixel data. The compensation method can be to calculate the average of neighboring pixels in the same channel, etc. The result obtained after processing is the compensated second pixel data.

[0126] S203. Perform multi-scale downsampling processing on the original image data and the second pixel data respectively to obtain the corresponding first multi-level image features and second multi-level image features.

[0127] In this embodiment, the first multi-level image feature is the multi-level feature corresponding to the original image data, and the second multi-level image feature is the multi-level feature corresponding to the second pixel data.

[0128] In one possible implementation, multi-scale downsampling processing is performed on the original image data and the second pixel data respectively to obtain corresponding first multi-level image features and second multi-level image features, including:

[0129] Obtain the preset image pyramid algorithm;

[0130] Based on the image pyramid algorithm, filtering, convolution and downsampling are performed sequentially on the original image data to iteratively generate the first multi-level image features.

[0131] According to the image pyramid algorithm, filtering, convolution and downsampling are performed on the second pixel data in sequence to iteratively generate a second multi-level image feature that matches the size of the first multi-level image feature.

[0132] For example, a pre-defined image pyramid algorithm is first obtained. Based on this algorithm, the original image data is sequentially filtered, convolved, and downsampled. After multiple iterations, a first multi-level image feature is generated. Then, the same filtering, convolution, and downsampling process is performed on the second pixel data using the same algorithm. A second multi-level image feature is generated iteratively. The size of this feature matches that of the first multi-level image feature. The first and second multi-level image features correspond to the multi-level features of the original image data and the second pixel data, respectively.

[0133] Specifically, Figure 3 The image pyramid multi-scale downsampling hierarchy generation map provided in the embodiments of this application, such as Figure 3As shown, using a Gaussian pyramid as an example, this demonstrates the process of generating the first multi-level image features by performing filtering, convolution, and downsampling iterative processing on the original image data. It presents the hierarchical changes and decreasing size states of the generated level 1, level 2, level 3, and level 4 images, starting from the level 0 original image, intuitively reflecting the hierarchical generation results of multi-scale downsampling. The size of each level image gradually decreases with downsampling iterations, and precisely matches the size of the original image from sparse phase detection. Each level completely preserves the parametric features of the original image.

[0134] It is important to note that Figure 3 This is for illustrative purposes only and does not affect the scope of protection of this application.

[0135] S204. The first and second multi-level image features of the same level are fused to obtain fused image data.

[0136] The first and second level image features of the same level are fused together. During the fusion process, the weight distribution ratio of each feature can be set according to the requirements. The image data obtained after the fusion calculation is the fused image data.

[0137] It should be noted that this embodiment does not limit the specific method of fusion processing.

[0138] S205. Perform multi-scale upsampling processing on the fused image data to obtain upsampled image data.

[0139] Specifically, multi-scale upsampling is performed on the fused image data obtained after fusion processing. Upsampling can magnify the image to twice its original size in all directions, and the newly added rows and columns are filled with 0. The image data obtained after this processing is the upsampled image data.

[0140] S206. Based on the first multi-level image features, the second multi-level image features, and the upsampled image data, obtain the multi-level image detail residuals.

[0141] The first and second multi-level image features of the same size are compared with the corresponding upsampled image data. The calculation method is to subtract the upsampled image data from the original features. The difference obtained in this way is the multi-level image detail residual, which corresponds to the image detail at different scales.

[0142] S207. Based on the multi-level image detail residuals and upsampled image data, obtain the target image data and output the target image data to the preset interactive interface.

[0143] For example, the obtained multi-level image detail residuals are superimposed on the corresponding upsampled image data, and the superimposed image is used as the input for the next upsampling. After processing at each scale in sequence, the target image data is obtained, and finally the target image data is output to the pre-set interactive interface.

[0144] This application provides an image processing method that acquires original image data containing sparse phase detection pixels through a preset interactive interface, providing a basic data source for subsequent image processing; extracts and compensates for the pixel data corresponding to the sparse phase detection pixels, filling in the information gaps of the sparse phase detection pixels and restoring complete phase detection pixel information; performs multi-scale downsampling processing on the original image data and the compensated pixel data to extract image features at different levels, realizing the mining and extraction of multi-scale image information; fuses two types of multi-level image features at the same level, integrating the effective information of phase detection pixel features and original image features to compensate for the feature deficiencies of the original image; performs multi-scale upsampling processing on the fused image data to restore the size hierarchy of the image, preparing for detail enhancement; combines the two types of multi-level features with the upsampled image data to obtain multi-level detail residuals, extracting detail information at each level of the image; obtains and outputs the target image based on the residuals and upsampled data, and superimposes the detail information back into the image, ultimately achieving the technical effect of improving image clarity.

[0145] Figure 4 A flowchart illustrating the image processing method provided in the embodiments of this application. Figure 2 ,like Figure 4 As shown, this embodiment, based on the above embodiments, provides a detailed description of the specific process for obtaining fused image data, including:

[0146] S401. Based on the second multi-level image features, determine the target level that is consistent with the second pixel data size.

[0147] For example, in the second multi-level image features, the level whose pixel size is completely consistent with the second pixel data is selected and the level is determined as the target level. The target level is the core level of the subsequent phase detection pixel feature processing. It can be accurately matched from the multi-level features according to the downsampling multiple relationship. For example, the level downsampled to 4 times can be matched with the second pixel data of the corresponding size as the target level.

[0148] S402. Decompose the second multi-level image features into left phase detection pixel features and right phase detection pixel features.

[0149] The second-level image features are split according to the phase detection attributes to obtain left phase detection pixel features and right phase detection pixel features. Both types of features retain the original pixel information of phase detection and are the basic data for subsequent alignment and synthesis processing. The splitting is strictly based on the left and right attributes of phase detection for accurate division without information deviation.

[0150] S403. Using the right phase detection pixel features corresponding to the target level as the reference features, perform pixel translation processing on the left phase detection pixel features corresponding to the target level to obtain the translated left phase detection pixel features.

[0151] Specifically, the right phase detection pixel feature corresponding to the target level is selected as the reference feature. The left phase detection pixel feature corresponding to the target level is shifted with a step size of 1 pixel. The shifted left phase detection pixel feature is obtained. The shift range can be adjusted to the pixel range of -r to r according to actual needs. The maximum value of r is the width of the effective feature area.

[0152] S404. Based on the left phase detection pixel features and the reference features, determine the corresponding sum of absolute differences, and based on the minimum value of the sum of absolute differences, determine the corresponding target alignment position.

[0153] Calculate the sum of absolute differences between the left phase detection pixel features after translation and the reference features. Iterate through the sum of absolute differences corresponding to all translation positions and determine the position with the smallest value as the target alignment position. During the calculation, you can select integer features or specified regions as the effective calculation area for the sum of absolute differences as needed.

[0154] Figure 5 This is a correlation diagram of defocus and offset of phase detection pixel features provided in an embodiment of this application. Figure 5This diagram illustrates the correlation between left and right phase-detection pixel features and image defocus. It visually presents the correspondence between the degree of defocus (i.e., the sum of absolute differences) and the positional offset of the left and right phase-detection pixel features. The greater the defocus, the greater the offset; the smaller the defocus, the closer the two are to overlapping. This provides a visual basis for determining pixel translation and target alignment. The red dashed line with arrows at the top of the diagram represents the direction and distance of the positional offset of the left and right phase-detection pixel features when the image is significantly defocused, with the line length reflecting the magnitude of the offset. The green solid line with arrows represents the direction and distance of the offset of the left and right phase-detection pixel features when the image is slightly defocused or nearly in focus, with the line length reflecting the magnitude of the slight offset. The red curve below represents the distribution of pixel phase offset with image region / depth of field under significant defocus, with an overall high offset level. The green curve represents the distribution of pixel phase offset under slight defocus, with an overall low offset level. The brown curve is the phase offset baseline curve for a fully in-focus state, which is nearly horizontal and is the core reference for determining the degree of defocus and pixel feature alignment. Figure 5 It clearly demonstrates the correlation between the degree of defocus and the pixel parameter position offset.

[0155] Figure 6 The graph showing the relationship between the sum of absolute differences and pixel translation provided in the embodiments of this application is as follows: Figure 6 As shown in the figure, this figure illustrates the trend of the sum of absolute differences (SAD) calculated from the baseline features after the left phase detection pixel features are shifted by different numbers of pixels. It intuitively presents the correspondence between the number of shifted pixels and the sum of absolute differences, and clearly identifies the pixel shift position corresponding to the minimum value of the sum of absolute differences, which is the target alignment position of the left and right phase detection pixel features. In the figure, the horizontal axis represents the number of translation pixels, and the vertical axis represents the total absolute difference parameter. The lowest point of the curve is the parameter node corresponding to the target alignment position. At this position, the matching degree between the left phase detection pixel parameter and the pixel parameter of the reference feature reaches the highest, which can provide a precise positional basis for subsequent synthesis of clear center phase detection pixel features. Here, r represents the boundary value of the pixel translation range of the left phase detection pixel feature relative to the reference feature, and the value range of the translation pixels is -r to r. The black dots are the points corresponding to the number of translation pixels and the corresponding total absolute difference, which intuitively present the distribution of SAD values ​​under different numbers of translation pixels. The red dots are the points with the smallest corresponding SAD values ​​among all black dots. The number of pixel translations on the horizontal axis corresponding to this point is the target alignment position of the left and right phase detection pixel features, which is the core identifier for determining the overlapping position of pixel features.

[0156] It is important to note that Figure 5 , Figure 6 This is for illustrative purposes only and does not affect the scope of protection of this application.

[0157] S405. Based on the target alignment position, the left phase detection pixel features and the right phase detection pixel features are synthesized to obtain the center phase detection pixel features.

[0158] According to the determined target alignment position, the left phase detection pixel features and the right phase detection pixel features are synthesized to obtain the center phase detection pixel features.

[0159] It should be noted that the synthesis method can be to directly add pixel values, take the average value, or add them with weights. This embodiment does not make specific limitations on the synthesis method. The selected synthesis method must ensure that the pixel information of the synthesized feature is complete and clear.

[0160] S406. Obtain the corresponding acquisition scene parameters based on the original image data.

[0161] The corresponding acquisition scene parameters are extracted from the acquisition association information of the original image data. The acquisition scene parameters include scene-related information such as ambient lighting effect and dynamic range during image acquisition. These parameters are the core basis for subsequent adjustment of fusion weights and can be accurately obtained directly from the acquisition metadata of the original image data without additional processing.

[0162] S407. Adjust the fusion weights of the central phase detection pixel features and the first multi-level image features according to the scene acquisition parameters.

[0163] In one possible implementation, the fusion weights of the center phase detection pixel features and the first multi-level image features are adjusted according to the scene acquisition parameters, including:

[0164] Obtain the preset data collection scenario parameter matching table;

[0165] Based on the parameter matching table for the collected scene, determine the parameter type of the collected scene parameters;

[0166] If the parameter type is a low-light scene parameter, increase the fusion weight of the central phase detection pixel features;

[0167] If the scene parameters collected are high dynamic scene parameters, reduce the fusion weight of the central phase detection pixel features;

[0168] If the scene parameters collected are standard scene parameters, the fusion weights of the center phase detection pixel features and the first multi-level image features are set to be equally distributed.

[0169] For example, a pre-set acquisition scene parameter matching table is first retrieved, and the type of the extracted acquisition scene parameters is determined according to the table. If it is a low-light scene parameter, the fusion weight of the central phase detection pixel feature is increased. If it is a high dynamic scene parameter, the fusion weight of the feature is decreased. Under normal scene parameters, the fusion weights of the two types of features are equally distributed. The weight adjustment is only performed on the central phase detection pixel feature and the first multi-level image features.

[0170] S408. Based on the adjusted fusion weights, the center phase detection pixel features and the first multi-level image features are weighted and calculated to obtain the fused image data.

[0171] Based on the adjusted fusion weights, weighted calculations are performed on the central phase detection pixel features and the first multi-level image features. During the calculation, the pixel values ​​of the two types of features are multiplied by their corresponding weights and summed, then divided by the total weights. The final fused image data is obtained through this weighted calculation method. The calculation process ensures that the feature pixel size matching is without deviation.

[0172] The image processing method provided in this application defines the core range for subsequent phase feature processing by accurately matching the feature level adapted to the second pixel data, splitting the left and right phase detection pixel features to provide independent basic data for subsequent alignment synthesis, and providing multiple sets of comparison data for finding the target alignment position by pixel translation. The target alignment position can be accurately determined by summing the absolute differences, thus accurately judging the ideal overlap state of the left and right phase features. The synthesized center phase detection pixel features can integrate the effective information of the left and right phases to form high-quality features. The extraction of acquisition scene parameters provides a realistic basis for adjusting the fusion weights. The fusion weights are adjusted according to the scene to adapt the feature fusion to different acquisition environments. The weighted calculation can fuse the two types of features according to the adapted weights, so that the fused image data takes into account the advantages of different features.

[0173] Figure 7 A flowchart illustrating the image processing method provided in the embodiments of this application. Figure 3 ,like Figure 7 As shown, this embodiment, based on the above embodiments, elaborates on the specific process of obtaining multi-level image detail residuals and subsequent processes, including:

[0174] S701. The upsampled image data is split into sub-upsampled image data that match the size of each level of the first multi-level image features and the second multi-level image features.

[0175] The obtained upsampled image data is split into layers. Based on the size specifications of each layer of the first and second multi-layer image features, sub-upsampled image data that match each layer is split out. During implementation, the pixel size of each layer is accurately divided to ensure that each layer has sub-upsampled image data of the corresponding size. After splitting, the size of the sub-upsampled image data of each layer is completely consistent with the corresponding layer features.

[0176] S702. Based on the first multi-level image features and the sub-upsampled image data, determine the first pixel difference between the first multi-level image features of each level and the sub-upsampled image data of the corresponding size.

[0177] Based on the data of each level of the first multi-level image features and the corresponding sub-upsampled image data, the pixel values ​​of the two types of data at the same level are subtracted to obtain the first pixel difference between the first multi-level image features of each level and the corresponding size sub-upsampled image data. In practice, the level and size are strictly matched, and the difference calculation is completed pixel by pixel to ensure the integrity of the first pixel difference data of each level.

[0178] S703. Based on the second multi-level image features and the sub-upsampled image data, determine the second pixel difference between the second multi-level image features of each level and the sub-upsampled image data of the corresponding size.

[0179] The data of each level of the second multi-level image features are matched with the sub-upsampled image data of the corresponding size. The pixel value of the two types of data at the same level is subtracted to obtain the second pixel difference between the second multi-level image features of each level and the corresponding sub-upsampled image data. In practice, the calculation is performed sequentially by level to ensure that the second pixel difference result of each level accurately corresponds to the level to which it belongs.

[0180] S704. Based on the first pixel difference and the second pixel difference, integrate to obtain the image detail residuals corresponding to each level.

[0181] In this embodiment, each level corresponds to an image detail residual.

[0182] The first pixel difference and the second pixel difference corresponding to each level are integrated. During integration, the two types of difference data in the same level are merged into a unique image detail residual for that level. Only one corresponding image detail residual is generated for each level. In practice, integration is carried out one by one according to the level to ensure that the image detail residual of each level is independent and accurately corresponds to the level.

[0183] S705. Summarize the image detail residuals corresponding to each level to obtain multi-level image detail residuals.

[0184] The image detail residuals of all levels obtained after integration are summarized and aggregated. The individual image detail residuals of each level are integrated into a complete set of multi-level image detail residuals. During implementation, the level attributes of each level residual are preserved. After aggregation, the image detail residual data corresponding to each level can still be distinguished, and the residual information of different levels is not confused.

[0185] The image processing method provided in this application provides a precise size matching foundation for subsequent difference calculation by splitting upsampled image data by level. Calculating the first pixel difference can extract the detail differences of the first multi-level image features relative to the upsampled data. Calculating the second pixel difference can obtain the detail differences of the second multi-level image features relative to the upsampled data. Integrating the differences of the same level can obtain the complete image detail residual of a single level. Summarizing the residuals of each level can form a multi-level image detail residual covering the entire level, providing full-scale detail data support for subsequent image processing.

[0186] Figure 8 A flowchart illustrating the image processing method provided in the embodiments of this application. Figure 4 ,like Figure 8 As shown, this embodiment, based on the above embodiments, provides a detailed description of the specific process for obtaining target image data, including:

[0187] S801. Based on the multi-level image detail residuals, obtain the image detail residuals corresponding to each level.

[0188] From the aggregated multi-level image detail residuals, the independent image detail residuals corresponding to each level are extracted. During implementation, the residuals are precisely split according to the attribute characteristics of each level, and the original data information of the image detail residuals of each level is preserved. This ensures that the extracted single-level residuals correspond one-to-one with their respective levels, without any level confusion or missing data.

[0189] S802. Determine the corresponding generation order of the layers based on the first multi-level image features and the second multi-level image features.

[0190] Referring to the generation logic of the first and second multi-level image features, the generation order of the two from the bottom to the top is clarified. During implementation, the generation order of the layers from low to high is determined according to the iterative generation rules of filtering, convolution, and downsampling, ensuring that the order is completely consistent with the actual generation order of the two types of multi-level image features.

[0191] S803. Determine the stacking order of image detail residuals corresponding to each level according to the generation order of the levels.

[0192] The stacking order of image detail residuals corresponding to each level is kept consistent with the determined level generation order. During implementation, the stacking logic of residuals is set according to the order of level generation. The image detail residuals of the lowest level are stacked first, and the residuals of the higher levels are stacked in turn. The stacking order is completely matched with the level generation order.

[0193] S804. According to the stacking order, the image detail residuals of the bottom layer are stacked onto the sub-upsampled image data of the corresponding size to obtain the image data after the bottom layer is stacked, and the image data after the bottom layer is stacked is used as the input data for the upsampling processing of the next layer.

[0194] According to the set stacking order, the image detail residuals at the bottom level are first stacked with the sub-upsampled image data of the corresponding size. After stacking, the image data of the bottom layer is obtained. This data is directly used as the input data for the next level of upsampling. During implementation, it is ensured that the size of the residuals and the sub-upsampled image data are accurately matched, and there is no data deviation in the stacking operation.

[0195] S805. Perform upsampling processing on the input data. After the upsampling processing is completed, iteratively perform the image detail residual overlay processing of the previous level until the processing of each level is completed, and obtain the image data after the highest level overlay. Then, use the image data after the highest level overlay as the target image data.

[0196] Upsampling is performed on the input data of the next level obtained after the bottom layer is overlaid. Upsampling can magnify the image by 2 times in all directions and fill the new rows and columns with 0. After confirming that the upsampling is completed, the image detail residuals of the previous level are iteratively overlaid in sequence. All levels of overlay and upsampling are completed according to this rule, and finally the image data after the highest level overlay is obtained. This data is directly identified as the target image data. During implementation, the iteration process strictly follows the hierarchical order to ensure that no level of processing is missed.

[0197] The image processing method provided in this application provides independent detailed data support for subsequent layer overlay by extracting residuals at each level. Determining the generation order of the layers defines a clear logical sequence for residual overlay. Matching the residual overlay order ensures the logical consistency between overlay operation and feature generation. The bottom layer residual overlay is used as input for the upper layer to lay the foundation data for subsequent iterative processing. Sequential iterative upsampling and residual overlay can completely restore the details of all levels to the image, and finally obtain target image data with rich details and improved clarity.

[0198] Figure 9 This is a schematic diagram of the image processing apparatus provided in an embodiment of this application. The apparatus in this embodiment can be in the form of software and / or hardware. Figure 9As shown, the image processing apparatus 900 provided in this application embodiment includes: a first acquisition module 901, a first processing module 902, a second processing module 903, a fusion module 904, a third processing module 905, a fourth processing module 906, and a fifth processing module 907.

[0199] The first acquisition module 901 is used to acquire raw image data containing sparse phase detection pixels through a preset interactive interface.

[0200] The first processing module 902 is used to extract the first pixel data corresponding to the sparse phase detection pixel points according to the original image data, and perform pixel compensation processing on the first pixel data to obtain the compensated second pixel data.

[0201] The second processing module 903 is used to perform multi-scale downsampling processing on the original image data and the second pixel data respectively to obtain the corresponding first multi-level image features and the second multi-level image features, wherein the first multi-level image features are the multi-level features corresponding to the original image data, and the second multi-level image features are the multi-level features corresponding to the second pixel data.

[0202] The fusion module 904 is used to fuse the first multi-level image features and the second multi-level image features at the same level to obtain fused image data;

[0203] The third processing module 905 is used to perform multi-scale upsampling processing on the fused image data to obtain upsampled image data.

[0204] The fourth processing module 906 is used to obtain multi-level image detail residuals based on the first multi-level image features, the second multi-level image features, and the upsampled image data.

[0205] The fifth processing module 907 is used to obtain target image data based on multi-level image detail residuals and upsampled image data, and output the target image data to a preset interactive interface.

[0206] In one possible implementation, the second processing module 903 is further configured to:

[0207] Obtain the preset image pyramid algorithm;

[0208] Based on the image pyramid algorithm, filtering, convolution and downsampling are performed sequentially on the original image data to iteratively generate the first multi-level image features.

[0209] According to the image pyramid algorithm, filtering, convolution and downsampling are performed on the second pixel data in sequence to iteratively generate a second multi-level image feature that matches the size of the first multi-level image feature.

[0210] In one possible implementation, the fusion module 904 is further configured to:

[0211] Based on the second multi-level image features, determine the target level that is consistent with the second pixel data size;

[0212] The second multi-level image features are decomposed into left phase detection pixel features and right phase detection pixel features;

[0213] The right phase detection pixel features corresponding to the target level are used as the reference features, and the left phase detection pixel features corresponding to the target level are subjected to pixel translation processing to obtain the translated left phase detection pixel features.

[0214] Based on the left phase detection pixel features and the reference features, determine the corresponding sum of absolute differences;

[0215] The target alignment position is determined based on the minimum value of the sum of absolute differences;

[0216] Based on the target alignment position, the left phase detection pixel features and the right phase detection pixel features are synthesized to obtain the center phase detection pixel features.

[0217] The central phase detection pixel features and the first multi-level image features corresponding to the target level are weighted and superimposed to obtain fused image data.

[0218] In one possible implementation, the fusion module 904 is further configured to:

[0219] Based on the original image data, obtain the corresponding acquisition scene parameters;

[0220] Based on the scene parameters, adjust the fusion weights of the central phase detection pixel features and the first multi-level image features;

[0221] Based on the adjusted fusion weights, the center phase detection pixel features and the first multi-level image features are weighted and calculated to obtain the fused image data.

[0222] In one possible implementation, the fusion module 904 is further configured to:

[0223] Obtain the preset data collection scenario parameter matching table;

[0224] Based on the parameter matching table for the collected scene, determine the parameter type of the collected scene parameters;

[0225] If the parameter type is a low-light scene parameter, increase the fusion weight of the central phase detection pixel features;

[0226] If the scene parameters collected are high dynamic scene parameters, reduce the fusion weight of the central phase detection pixel features;

[0227] If the scene parameters collected are standard scene parameters, the fusion weights of the center phase detection pixel features and the first multi-level image features are set to be equally distributed.

[0228] In one possible implementation, the fourth processing module 906 is further configured to:

[0229] The upsampled image data is split into sub-upsampled image data that match the size of each level of the first multi-level image features and the second multi-level image features;

[0230] Based on the first multi-level image features and the sub-upsampled image data, determine the first pixel difference between the first multi-level image features of each level and the sub-upsampled image data of the corresponding size;

[0231] Based on the second multi-level image features and the sub-upsampled image data, determine the second pixel difference between the second multi-level image features of each level and the sub-upsampled image data of the corresponding size;

[0232] Based on the first pixel difference and the second pixel difference, the image detail residuals corresponding to each level are integrated to obtain the image detail residuals. Each level corresponds to one image detail residual.

[0233] The image detail residuals corresponding to each level are summarized to obtain multi-level image detail residuals.

[0234] In one possible implementation, the fifth processing module 907 is further configured to:

[0235] Based on the multi-level image detail residuals, obtain the image detail residuals corresponding to each level;

[0236] The corresponding generation order of the layers is determined based on the first and second multi-layer image features;

[0237] Based on the generation order of the layers, determine the superposition order of the image detail residuals corresponding to each layer;

[0238] According to the stacking order, the image detail residuals of the bottom level are stacked onto the sub-upsampled image data of the corresponding size to obtain the image data after the bottom stack, and the image data after the bottom stack is used as the input data for the next level of upsampling processing;

[0239] Upsampling is performed on the input data. After the upsampling is completed, the image detail residual overlay processing of the previous level is iteratively performed until the processing of each level is completed, and the image data after the highest level overlay is obtained.

[0240] The image data after the highest layer is overlaid is used as the target image data.

[0241] The image processing apparatus provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0242] Figure 10 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application. Figure 10 As shown, the image processing device 1000 provided in this embodiment includes at least one processor 1001 and a memory 1002. Optionally, the device 1000 further includes a communication component 1003. The processor 1001, memory 1002, and communication component 1003 are connected via a bus.

[0243] In a specific implementation, at least one processor 1001 executes computer execution instructions stored in memory 1002, causing at least one processor 1001 to perform the above-described method.

[0244] The specific implementation process of processor 1001 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0245] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0246] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0247] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0248] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0249] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0250] This application also provides a chip, including at least one chip processor, which is used to execute program instructions to perform the above-described method.

[0251] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0252] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0253] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0254] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0255] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0256] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0257] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0258] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, include: The original image data containing sparse phase detection pixels is obtained through a preset interactive interface; Based on the original image data, the first pixel data corresponding to the sparse phase detection pixel is extracted, and pixel compensation processing is performed on the first pixel data to obtain the compensated second pixel data. Multi-scale downsampling processing is performed on the original image data and the second pixel data respectively to obtain the corresponding first multi-level image features and second multi-level image features, wherein the first multi-level image features are the multi-level features corresponding to the original image data, and the second multi-level image features are the multi-level features corresponding to the second pixel data; The first multi-level image features and the second multi-level image features at the same level are fused to obtain fused image data; Perform multi-scale upsampling processing on the fused image data to obtain upsampled image data; Based on the first multi-level image features, the second multi-level image features, and the upsampled image data, multi-level image detail residuals are obtained; Based on the multi-level image detail residuals and the upsampled image data, target image data is obtained and output to the preset interactive interface.

2. The method according to claim 1, characterized in that, The step of performing multi-scale downsampling processing on the original image data and the second pixel data respectively to obtain the corresponding first multi-level image features and second multi-level image features includes: Obtain the preset image pyramid algorithm; According to the image pyramid algorithm, the original image data is sequentially processed by filtering, convolution and downsampling to iteratively generate the first multi-level image features. According to the image pyramid algorithm, the second pixel data is sequentially processed by filtering, convolution and downsampling to iteratively generate the second multi-level image features that match the size of the first multi-level image features.

3. The method according to claim 2, characterized in that, The process of fusing the first multi-level image features and the second multi-level image features at the same level to obtain fused image data includes: Based on the second multi-level image features, a target level consistent with the second pixel data size is determined; The second multi-level image features are decomposed into left phase detection pixel features and right phase detection pixel features; Using the right phase detection pixel features corresponding to the target level as the reference features, the left phase detection pixel features corresponding to the target level are subjected to pixel translation processing to obtain the translated left phase detection pixel features; Based on the left phase detection pixel features and the reference features, determine the corresponding sum of absolute differences; The corresponding target alignment position is determined based on the minimum value of the sum of the absolute differences; Based on the target alignment position, the left phase detection pixel features and the right phase detection pixel features are synthesized to obtain the center phase detection pixel features; The fused image data is obtained by weighted superposition of the central phase detection pixel features and the first multi-level image features corresponding to the target level.

4. The method according to claim 3, characterized in that, The step of weighted superposition of the central phase detection pixel features and the first multi-level image features corresponding to the target level to obtain the fused image data includes: Based on the original image data, obtain the corresponding acquisition scene parameters; Based on the acquired scene parameters, adjust the fusion weights of the central phase detection pixel features and the first multi-level image features; Based on the adjusted fusion weights, the center phase detection pixel features and the first multi-level image features are weighted and calculated to obtain the fused image data.

5. The method according to claim 4, characterized in that, The step of adjusting the fusion weights of the center phase detection pixel features and the first multi-level image features according to the acquired scene parameters includes: Obtain the preset data collection scenario parameter matching table; Based on the scene parameter matching table, determine the parameter type of the scene parameters. If the parameter type is a low-light scene parameter, increase the fusion weight of the center phase detection pixel features; If the scene parameters being collected are high dynamic scene parameters, reduce the fusion weight of the central phase detection pixel features; If the scene parameters are conventional scene parameters, the fusion weights of the center phase detection pixel features and the first multi-level image features are set to be equally distributed.

6. The method according to any one of claims 1-5, characterized in that, The step of obtaining multi-level image detail residuals based on the first multi-level image features, the second multi-level image features, and the upsampled image data includes: The upsampled image data is split into sub-upsampled image data that match the size of each level of the first multi-level image feature and the second multi-level image feature; Based on the first multi-level image features and the sub-upsampled image data, determine the first pixel difference between the first multi-level image features of each level and the sub-upsampled image data of the corresponding size; Based on the second multi-level image features and the sub-upsampled image data, determine the second pixel difference between the second multi-level image features of each level and the sub-upsampled image data of the corresponding size; Based on the first pixel difference and the second pixel difference, the image detail residuals corresponding to each level are integrated to obtain the image detail residuals corresponding to each level. The image detail residuals corresponding to each level are summarized to obtain the multi-level image detail residuals.

7. The method according to claim 6, characterized in that, The step of obtaining target image data based on the multi-level image detail residuals and the upsampled image data includes: Based on the multi-level image detail residuals, obtain the image detail residuals corresponding to each level; Based on the first multi-level image features and the second multi-level image features, the corresponding generation order of the levels is determined; Based on the generation order of the layers, determine the superposition order of the image detail residuals corresponding to each layer; According to the stacking order, the image detail residuals of the bottom level are stacked onto the sub-upsampled image data of the corresponding size to obtain the image data after the bottom stack, and the image data after the bottom stack is used as the input data for the next level upsampling process; Upsampling is performed on the input data. After the upsampling is completed, the image detail residual overlay processing of the previous level is iteratively performed until the processing of each level is completed, and the image data after the highest level overlay is obtained. The image data after the highest level is superimposed is used as the target image data.

8. An image processing apparatus, characterized in that, include: The first acquisition module is used to acquire raw image data containing sparse phase detection pixels through a preset interactive interface; The first processing module is used to extract the first pixel data corresponding to the sparse phase detection pixel based on the original image data, and perform pixel compensation processing on the first pixel data to obtain the compensated second pixel data. The second processing module is used to perform multi-scale downsampling processing on the original image data and the second pixel data respectively to obtain the corresponding first multi-level image features and second multi-level image features, wherein the first multi-level image features are the multi-level features corresponding to the original image data, and the second multi-level image features are the multi-level features corresponding to the second pixel data. The fusion module is used to fuse the first multi-level image features and the second multi-level image features at the same level to obtain fused image data; The third processing module is used to perform multi-scale upsampling processing on the fused image data to obtain upsampled image data. The fourth processing module is used to obtain multi-level image detail residuals based on the first multi-level image features, the second multi-level image features, and the upsampled image data. The fifth processing module is used to obtain target image data based on the multi-level image detail residuals and the upsampled image data, and output the target image data to the preset interactive interface.

9. An image processing device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

Images