Image processing method and device, communication equipment and chip

Abstract

The invention provides an image processing method and device, communication equipment and a chip, and relates to the field of image processing. The method comprises the following steps: acquiring a first confidence coefficient of a to-be-processed image, wherein the first confidence coefficient is the confidence coefficient of pixel points of the to-be-processed image; based on the first confidence coefficient, determining a second confidence coefficient of the to-be-processed image, the second confidence coefficient being a confidence coefficient of an image frame of the to-be-processed image; and processing the to-be-processed image based on the second confidence coefficient. According to the scheme, the confidence coefficient of the image frame of the to-be-processed image is determined based on the confidence coefficient of the pixel points of the to-be-processed image, and image processing is performed by using the confidence coefficient of the image frame of the to-be-processed image, so that global alignment and local alignment selection based on the confidence coefficient of the image frame and switching among different alignment modes can be realized; the accuracy of an image quality judgment result is improved, and meanwhile the image processing quality is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to an image processing method, device, communication equipment, chip and chip. Background Art

[0002] With the popularization of smart terminal devices and the development of image processing, users have put forward higher requirements for the image quality obtained from the terminal. Image alignment is an important direction in the field of image processing. When facing the alignment information of multi-source images and the alignment information of different local images, it is necessary to make image quality judgment and output decisions based on the confidence of the image frame. Therefore, an image processing method that effectively solves the above problems is needed. Summary of the invention

[0003] The present disclosure provides an image processing method, apparatus, communication equipment and chip, which can upgrade the confidence of pixel points to confidence information of image frames to solve problems in related technologies, expand the scope of application of image processing solutions, and improve image processing efficiency and quality.

[0004] The first aspect embodiment of the present disclosure proposes an image processing method, which includes: obtaining a first confidence of an image to be processed, where the first confidence is the confidence of a pixel point of the image to be processed; based on the first confidence, determining a second confidence of the image to be processed, where the second confidence is the confidence of an image frame of the image to be processed; and based on the second confidence, processing the image to be processed.

[0005] In some embodiments, determining the second confidence based on the first confidence includes: determining a third confidence of the image to be processed based on the first confidence, the third confidence being used to determine the second confidence; determining the spatial weight and / or content weight of the third confidence; performing weighted summation processing on the third confidence based on the spatial weight and / or content weight, respectively, to determine the spatial confidence and / or content confidence of the image to be processed; and normalizing the spatial confidence and / or content confidence, respectively, to determine the second confidence.

[0006] In some embodiments, based on the first confidence level, determining a third confidence level of the image to be processed includes: performing a first filtering process on the first confidence level to determine the third confidence level.

[0007] In some embodiments, determining the content weight of the third confidence level includes: acquiring a content saliency map of the image to be processed; and determining the content weight of the third confidence level based on the content saliency map and according to a content saliency lookup table.

[0008] In some embodiments, the method further includes: filtering the second confidence level to determine a filtering confidence level; and removing abnormal data of the filtering confidence level.

[0009] In some embodiments, filtering the second confidence to determine the filtering confidence includes: determining a buffer area and a second confidence weight; and determining the filtering confidence based on the buffer area and the second confidence weight.

[0010] The second aspect embodiment of the present disclosure proposes an image processing device, which includes: an acquisition unit, used to acquire a first confidence of an image to be processed, the first confidence being the confidence of a pixel point of the image to be processed; a determination unit, used to determine a second confidence of the image to be processed based on the first confidence, the second confidence being the confidence of an image frame of the image to be processed; and a processing unit, used to process the image to be processed based on the second confidence.

[0011] The third aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the first aspect embodiment of the present disclosure.

[0012] The fourth aspect embodiment of the present disclosure proposes a communication device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect embodiment of the present disclosure.

[0013] The fifth aspect embodiment of the present disclosure proposes a chip, which includes one or more interfaces and one or more processors; the interface is used to receive signals from a memory of a communication device and send signals to the processor, the signals include computer instructions stored in the memory, and when the processor executes the computer instructions, the communication device executes the method described in the first aspect embodiment of the present disclosure.

[0014] In summary, according to the image processing method proposed in the present disclosure, the method includes: obtaining a first confidence of the image to be processed, the first confidence being the confidence of the pixel points of the image to be processed; determining a second confidence based on the first confidence, the second confidence being the confidence of the image frame of the image to be processed; and processing the image to be processed based on the second confidence. The scheme of the present disclosure determines the confidence of the image frame of the image to be processed based on the confidence of the pixel points of the image to be processed, and uses the confidence of the image frame of the image to be processed for image processing, thereby realizing global alignment based on the confidence of the image frame, local alignment selection, and switching between different alignment modes, thereby improving the accuracy of the image quality determination result and the image processing quality.

[0015] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0017] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0018] Figure 2 A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0019] Figure 3 A flowchart of another image processing method provided by an embodiment of the present disclosure;

[0020] Figure 4 A flowchart of another image processing method provided by an embodiment of the present disclosure;

[0021] Figure 5 A flowchart of another image processing method provided by an embodiment of the present disclosure;

[0022] Figure 6 A flowchart of another image processing method provided by an embodiment of the present disclosure;

[0023] Figure 7 A schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure;

[0024] Figure 8 A schematic diagram of the structure of a communication device provided in an embodiment of the present disclosure;

[0025] Fig. 9 A schematic diagram of the structure of a chip provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0027] With the popularization of smart terminal devices and the development of image processing, people's technical requirements for various aspects of terminals are also rising. The quality of output images has become one of the most concerned indicators for users. At the same time, image alignment technology is an important direction in the field of image processing. When the terminal outputs images to users, whether it can ensure the alignment of the foreground and background of the image, and whether it can control the local jitter caused by local alignment of the image, directly determines the user experience. Therefore, image alignment technology has become an important focus of various manufacturers.

[0028] Image alignment is an important direction in the field of image processing. When faced with the alignment information of multi-source images and the alignment information of locally different images, it is necessary to make image quality judgments and output decisions based on the confidence of the image frames. When the confidence of the image frames is lacking or the confidence quality of the image frames is insufficient, it will affect the image quality judgment and output decisions.

[0029] Therefore, in order to solve the problems existing in the related art, the present disclosure proposes an image processing solution, which determines the confidence of the image frame of the image to be processed based on the confidence of the pixel points of the image to be processed, and uses the confidence of the image frame of the image to be processed to perform image processing, thereby realizing global alignment based on the confidence of the image frame, local alignment selection and switching between different alignment modes, thereby improving the accuracy of the image quality judgment results, and at the same time improving the image processing quality.

[0030] Before introducing the detailed solution of the present disclosure, the scenario to which the solution of the present disclosure is applied is described first. Figure 1 FIG. 1 is an application scenario diagram of an image processing method in an embodiment. Figure 1 As shown, the application scenario includes an electronic device 104, in which a camera module and several applications can be installed. The application can initiate an image acquisition instruction to acquire an image, and the camera module acquires the image 102. Among them, the camera module can include a front camera module and / or a rear camera module. Finally, the target image is sent to the target application. Among them, the electronic device 104 can be a smart phone, a tablet computer, a personal digital assistant, a wearable device, etc.

[0031] In some optional embodiments, the electronic device may also be a vehicle-mounted device or an Internet of Vehicles device, such as a smart car, etc. In the present disclosure, only a smart phone is taken as an example, but it does not limit the scope of the present disclosure.

[0032] A camera can be installed on an electronic device, and images can be acquired through the installed camera. Cameras can be divided into laser cameras, visible light cameras and other types according to the different images acquired. Laser cameras can acquire images formed by irradiating lasers onto objects, and visible light cameras can acquire images formed by irradiating visible light onto objects. Several cameras can be installed on an electronic device, and the installation locations are not limited. For example, a camera can be installed on the front panel of an electronic device, and two cameras can be installed on the back panel. The camera can also be installed inside the electronic device in an embedded manner, and then the camera can be opened by rotating or sliding. Specifically, a front camera and a rear camera can be installed on an electronic device. The front camera and the rear camera can acquire images from different perspectives. Generally, the front camera can acquire images from the front perspective of the electronic device, and the rear camera can acquire images from the back perspective of the electronic device.

[0033] It should be understood that in the present disclosure, the front camera or the rear camera is only used as an example to distinguish the shooting angles of different cameras, rather than to limit the functions of multiple cameras. The multiple cameras in the present disclosure can be rear cameras or front cameras at the same time, which is not limited in this disclosure.

[0034] An electronic device can have several applications installed. An application is software written for a certain purpose in an electronic device. An electronic device can use an application to provide services to users. When an application needs to capture an image, it will initiate an image capture instruction. The electronic device will call the camera module to capture the image according to the image capture instruction. An image capture instruction is an instruction used to trigger an image capture operation.

[0035] The electronic device is also provided with a processor, and the image processing module in the processor can perform image processing on the image captured by the camera module, for example, execute the image processing method provided in the present disclosure.

[0036] Figure 2 A flowchart of an image processing method provided by an embodiment of the present disclosure. The method can be applied to Figure 1 The application scenarios shown are, for example, executed by a terminal with an integrated image processing function or an image processor in the terminal, or by other devices suitable for image processing, which are not limited by the present disclosure. Figure 2 As shown, the image processing method includes steps 201-203.

[0037] Step 201: Obtain a first confidence level of an image to be processed.

[0038] In some embodiments, the image to be processed may be an image with a reference image in a data set, or may be an actual application scenario without a reference image.

[0039] In some embodiments, the image to be processed may be an image with only one frame or an image with multiple frames.

[0040] In some embodiments, the first confidence level is the confidence level of a pixel of the image to be processed. Alternatively, the first confidence level may be a local confidence level of the image to be processed.

[0041] Optionally, in some embodiments, the first confidence level of the image to be processed may be directly acquired by reading a processing result of the processor on the image to be processed.

[0042] Optionally, in some embodiments, for an image to be processed having a reference image (Grand Truth), such as a KITTI dataset, an alignment index of the image to be processed and the reference image may be determined to obtain a first confidence level of the image to be processed.

[0043] For example, if the image to be processed is an image in the KITTI dataset, the following alignment indicators can be used: D1 (Percentage of stereo disparity outliers in first frame), the percentage of stereo disparity (or optical flow motion vector) outliers in the first frame, D2: (Percentage of stereo disparity outliers in second frame), the percentage of stereo disparity outliers in the second frame, Fl (Percentage of optical flow outliers), the percentage of optical flow outliers, SF (Percentage of scene flow outliers = outliers in either D0, D1 or Fl), which is equivalent to the outliers in D0, D1 or Fl, bg (Percentage of outliers averaged only over background regions), the percentage of outliers averaged only over background regions, fg (Percentage of outliers averaged only over foreground regions), and all (Percentage of outliers averaged over all ground truth pixels), the average percentage of outliers over all real pixels. The first confidence level is calculated by selecting one or more of the above alignment indicators and setting weights for the one or more alignment indicators.

[0044] Optionally, in some embodiments, for an image to be processed without a reference image, it is impossible to construct a difference between the image to be processed and the reference image to determine the first confidence level, and feature extraction of the image to be processed and iterative processing of the image to be processed can be performed to obtain the first confidence level of the image to be processed when there is no reference image, wherein the feature extraction can include extraction of feature strength (Feature Strength) and extraction of feature distribution uniformity (Feature Coverage).

[0045] It should be understood that the above description of obtaining the first confidence of the image to be processed is only an example and should not limit the solution of the present disclosure. The present disclosure does not limit the method for obtaining the first confidence of the image to be processed.

[0046] Step 202: Determine a second confidence level of the image to be processed based on the first confidence level.

[0047] In some embodiments, the second confidence level of the image to be processed may be determined based on the first confidence level, thereby upgrading the confidence level of the pixel points of the image to be processed to the confidence level of the image frame.

[0048] In some embodiments, the second confidence level is a confidence level of an image frame of the image to be processed.

[0049] In some embodiments, the second confidence level may be determined by setting a spatial weight and / or a content weight of the first confidence level and performing a weighted summation.

[0050] Step 203: Process the image to be processed based on the second confidence level.

[0051] In some embodiments, the image to be processed can be processed based on the second confidence level, thereby achieving global alignment based on the confidence level of the image frame, local alignment selection, and switching between different alignment modes, while improving the image processing quality of the image to be processed.

[0052] In some embodiments, there is no limitation on the processing method of the image to be processed, which may include image noise reduction processing, smoothing processing between multiple frames of images, global alignment of images, etc.

[0053] Optionally, in some embodiments, the second confidence level may be combined with the global confidence level of the image to be processed to process the image to be processed. For example, weights may be set for the second confidence level and the global confidence level of the image to be processed, and a fused confidence level may be generated by weighted summation, and the image to be processed may be processed using the fused confidence level.

[0054] In some embodiments, the global confidence is similar to the local confidence. It can be the full image confidence of the image to be processed determined based on the alignment index, or it can be the full image confidence determined based on the feature distribution of the full image of the image to be processed. The present disclosure does not limit this.

[0055] In summary, according to the image processing method disclosed in the present invention, the method includes: obtaining a first confidence of an image to be processed, the first confidence being the confidence of a pixel of the image to be processed; determining a second confidence based on the first confidence, the second confidence being the confidence of an image frame of the image to be processed; and processing the image to be processed based on the second confidence. The scheme disclosed in the present invention determines the confidence of the image frame of the image to be processed based on the confidence of the pixel of the image to be processed, and uses the confidence of the image frame of the image to be processed for image processing, thereby realizing global alignment based on the confidence of the image frame, local alignment selection, and switching between different alignment modes, thereby improving the accuracy of the image quality determination result and the image processing quality.

[0056] Figure 3 The following further shows a flowchart of an image processing method proposed in the present disclosure. Figure 2 The embodiment shown, Figure 3 Further description of step 202 includes the following steps.

[0057] Step 301: Determine a third confidence level of the image to be processed based on the first confidence level.

[0058] In some embodiments, the first confidence level may be subjected to a first filtering process to determine a third confidence level of the image to be processed, wherein the third confidence level is used to determine the second confidence level. In other words, the third confidence level is the first confidence level after noise is filtered out.

[0059] In some embodiments, the first filtering process is used to smooth the first confidence level and to filter out noise.

[0060] Optionally, in some embodiments, the first filtering process may be filtering process performed on the first confidence level using one or more of an infinite impulse response (IIR) filter, a finite impulse response (FIR) filter, a box filter, etc., which is not limited in the present disclosure.

[0061] For example, the first confidence level is subjected to Box Filter filtering to filter out regional noise of the first confidence level through a preset window of the Box Filter, thereby achieving smooth filtering of the first confidence level. The preset window of the Box Filter may be a window of 3*3 units, or a window of other sizes, such as a window of 5*5 units, etc., which is not limited in the present disclosure.

[0062] Step 302: Determine the spatial weight and / or content weight of the third confidence level.

[0063] In some embodiments, the spatial confidence and / or content confidence of the image to be processed is determined by determining the spatial weight and / or content weight of the third confidence.

[0064] In some embodiments, there is no limitation on the method for determining the spatial weight of the third confidence. For example, the spatial weight of the third confidence can be constructed by a center-focused method, that is, a higher weight is set for the third confidence in the central area of ​​the image to be processed, and a relatively lower weight is set for the third confidence in the surrounding area. For another example, a higher weight can be set for the third confidence in the area with a complex spatial structure of the image to be processed, and a relatively lower weight can be set for the third confidence in the area with a simple spatial structure.

[0065] For example, the spatial weight of the third confidence is determined based on the Euclidean distance between all pixels of the image to be processed and the central pixel, wherein the closer the Euclidean distance to the central pixel is, the higher the spatial weight of the third confidence is.

[0066] In some embodiments, there is no limitation on the method of determining the content weight of the third confidence level. For example, a lower weight may be set for static areas in the image, and a higher weight may be set for moving object areas. For another example, a higher weight may be set for significant areas, and a lower weight may be set for insignificant areas.

[0067] Step 303: Perform weighted sum processing on the third confidence based on the spatial weight and / or content weight to determine the spatial confidence and / or content confidence of the image to be processed.

[0068] In some embodiments, based on the spatial weight, the third confidence is weighted and summed to determine the spatial confidence of the image to be processed.

[0069] Specifically, the third confidence level may be multiplied by its corresponding spatial weight, and the calculation results may be accumulated and summed to obtain the spatial confidence level of the image to be processed.

[0070] In some embodiments, when there are multiple third confidences, weights may be assigned again to multiple calculation results obtained by weighted summation of the multiple third confidences, and based on the weights assigned again, weighted summation is performed on the multiple calculation results, and the weighted summation result is determined as the spatial confidence of the image to be processed. However, it is not limited thereto, and multiple calculation results obtained by weighted summation of the multiple third confidences may also be directly determined as the spatial confidence of the image to be processed.

[0071] In other words, when there are multiple third confidences, the determined spatial confidence of the image to be processed may be one or more spatial confidences.

[0072] In some embodiments, based on the content weight, the third confidences are weighted and summed to determine the content confidence of the image to be processed.

[0073] Specifically, the third confidence level may be multiplied by its corresponding content weight, and the calculation results may be accumulated and summed to obtain the content confidence level of the image to be processed.

[0074] In some embodiments, when there are multiple third confidences, weights may be assigned again to multiple calculation results obtained by weighted summation of the multiple third confidences, and based on the weights assigned again, the multiple calculation results are weighted summed, and the weighted sum result is determined as the content confidence of the image to be processed. However, it is not limited thereto, and multiple calculation results obtained by weighted summation of the multiple third confidences may also be directly determined as the content confidence of the image to be processed.

[0075] In other words, when there are multiple third confidences, the determined content confidence of the image to be processed may be one or more content confidences.

[0076] Step 304: Normalize the spatial confidence and / or the content confidence respectively to determine a second confidence.

[0077] In some embodiments, the normalization of the spatial confidence can be achieved by the following formula:

[0078]

[0079] Among them, frame_level_space_conf represents the normalized spatial confidence, sum_space_weighted_conf represents the spatial confidence, and sum_space_weight represents the sum of the spatial weights corresponding to the third confidence.

[0080] In some embodiments, the normalization of content confidence can be achieved by the following formula:

[0081]

[0082] Among them, frame_level_content_conf represents the normalized spatial confidence, sum_content_weig hted_conf represents the spatial confidence, and sum_content_weight represents the sum of the spatial weights corresponding to the third confidence.

[0083] In some embodiments, the normalized spatial confidence and / or content confidence may be directly determined as the second confidence.

[0084] Optionally, in some embodiments, the normalized spatial confidence and content confidence may be subjected to weighted sum processing, and the weighted sum processing result may be determined as the second confidence.

[0085] In summary, according to the image processing method disclosed in the present invention, the method includes: determining a third confidence of the image to be processed based on the first confidence, the third confidence is used to determine the second confidence; determining the spatial weight and / or content weight of the third confidence; performing weighted processing on the third confidence based on the spatial weight and / or content weight respectively to determine the spatial confidence and / or content confidence of the image to be processed; normalizing the spatial confidence and / or content confidence respectively to determine the second confidence. This scheme realizes noise filtering of the first confidence by performing a first filtering process on the first confidence, and at the same time determines the spatial confidence and / or content confidence of the image to be processed, thereby determining the second confidence of the image to be processed, so as to realize upgrading the pixel confidence of the image to be processed to the confidence of the image frame, so as to realize global alignment based on the confidence of the image frame, local alignment selection and switching between different alignment modes, thereby improving the accuracy of the image quality judgment result and improving the image processing quality.

[0086] Figure 4 The following further shows a flowchart of an image processing method proposed in the present disclosure. Figure 3 The illustrated embodiment further describes step 301, which includes the following steps.

[0087] Step 401: Obtain a content saliency map of an image to be processed.

[0088] In some embodiments, a content saliency map of the image to be processed is obtained, so as to determine the content weight of the third confidence level based on the content saliency map.

[0089] In some embodiments, there is no limitation on the method of obtaining the content saliency map of the image to be processed. For example, AI saliency detection is used to determine the salient area of ​​the image to be processed, thereby obtaining the content saliency map of the image to be processed. Another example is: the image to be processed is divided into different areas through image segmentation technology, and the content saliency map of the image to be processed is obtained based on the differences in texture, color, etc. between each area and the adjacent areas.

[0090] Optionally, in some embodiments, the content saliency map of the image to be processed can also be determined based on the motion vector strength of the image to be processed, that is, the area with a larger motion vector strength is determined as a salient area, and the area with a smaller motion vector strength is determined as a non-salient area, so as to obtain the content saliency map of the image to be processed.

[0091] Step 402: Based on the content saliency map and according to the content saliency lookup table, determine the content weight of the third confidence level.

[0092] In some embodiments, the content weights corresponding to different saliencies in the content saliency map may be determined according to a content saliency lookup table, wherein the content saliency lookup table is a preset lookup table. In other words, the content weight of the third confidence level may be determined based on the content saliency map and according to the content saliency lookup table.

[0093] The content weight of the third confidence level can be determined by the following formula:

[0094] weight_content=content_weight_LUT(significance_map) (Formula 3)

[0095] Among them, weight_content is the content weight of the third confidence level, content_weight_LUT is the content significance lookup table, and significance_map represents the content significance map.

[0096] In some embodiments, step 402 is optional, that is, the content weight of the third confidence level may be preset directly based on the content saliency map of the image to be processed in step 401 .

[0097] In summary, according to the image processing method proposed in the present disclosure, the method includes: obtaining a content saliency map of the image to be processed; based on the content saliency map, determining the content weight of the third confidence level according to the content saliency lookup table. The solution of the present disclosure obtains the content saliency map of the image to be processed, and then determines the content weight of the third confidence level, which lays a foundation for determining the second confidence level of the image to be processed.

[0098] Figure 5 The following further shows a flowchart of an image processing method proposed in the present disclosure. Figure 1 The embodiment shown further explains the method. Figure 1 The illustrated embodiment may further include the following steps.

[0099] Step 501: Perform a second filtering process on the second confidence level to determine the filtering confidence level of the image to be processed.

[0100] In some embodiments, the second confidence level may be further subjected to a second filtering process to ensure smoothness of the second confidence level in the time domain. In other words, the filtered confidence level of the image to be processed is the second confidence level after the second filtering process.

[0101] In some embodiments, the second filtering process is used to perform a smoothing process on the second confidence level in the time domain.

[0102] In some embodiments, the second filtering process may be a sliding window filtering process in the time domain, but is not limited thereto, and may also be a filtering method such as a Gaussian filter.

[0103] Step 502: filter out abnormal data with low confidence.

[0104] In some embodiments, abnormal data of the filter confidence is eliminated to ensure the data stability and correctness of the filter confidence, thereby improving the image processing quality.

[0105] In some embodiments, a filter confidence value threshold may be set to determine filter confidence data exceeding the threshold as abnormal data, and the abnormal data may be removed.

[0106] In some embodiments, image processing may also be performed on the image to be processed based on the filtering confidence of removing abnormal data.

[0107] In summary, according to the image processing method proposed in the present disclosure, the method includes: performing a second filtering process on the second confidence level to determine the filtering confidence level of the image to be processed; and removing abnormal data of the filtering confidence level. The method disclosed in the present disclosure performs a second filtering process on the second confidence level to improve the smoothness of the second confidence level in the time domain, and removes abnormal data in the second confidence level after the second filtering process to ensure the data stability and correctness of the confidence level, thereby further improving the image processing quality.

[0108] Figure 6 The following further shows a flowchart of an image processing method proposed in the present disclosure. Figure 5 The illustrated embodiment further describes step 501 and includes the following steps.

[0109] Step 601: Determine a buffer area and a second confidence weight.

[0110] In some embodiments, by determining the buffer region and the second confidence weight, a basis is laid for determining the filtering confidence.

[0111] In some embodiments, the second confidence weight may be determined in a preset manner, and the present disclosure does not limit the manner for determining the second confidence weight.

[0112] In some embodiments, the buffer area is used to buffer the second confidences of the plurality of frames.

[0113] In some embodiments, there is no limitation on determining the size of the buffer area, which may be, for example, the size of the confidence data of image frames of 5 frames, 7 frames, etc., of the image to be processed.

[0114] It should be understood that the larger the buffer area, the better the smoothness of the second confidence in the time domain, but the less obvious the confidence change between frames; the smaller the buffer area, the smoother the second confidence in the time domain, but the better the confidence change between frames. Therefore, the above buffer area can be determined according to the demand for the second confidence and the performance of the device executing this solution.

[0115] Step 602: Determine filtering confidence based on the buffer area and the second confidence weight.

[0116] In some embodiments, the filtering confidence may be determined based on the buffer area and the second confidence weight by taking a weighted sum and performing a normalization process.

[0117] For example, the second confidence level that the buffer area can buffer N frames of images to be processed is used as an example for explanation, and the weighted sum can be performed using the following formula:

[0118] current_frame_conf_filtered=weight0*conf_buffer[0]+weight1*conf_buffer[1]

[0119] +…+weightN*conf_buffer[N](Formula 4)

[0120] Among them, [0] to [N] represent the frame numbers of the images to be processed, weight0 to weightN represent the second confidence weights corresponding to the second confidences of frames 0 to N of the images to be processed, conf_buffer represents the second confidence, and current_frame_con f_filtered represents the calculation result of the weighted sum of the second confidences of N frames of images to be processed.

[0121] For example, the second confidence level that the buffer area can buffer N frames of images to be processed is used as an example for explanation, and the weighted sum result can be normalized by the following formula:

[0122] current_frame_conf_filtered=current_frame_conf_filtered / sum(weight1+

[0123] weight2+…weightN)(Formula 5)

[0124] Among them, sum(weight1+weight2+…weightN) represents the sum of the second confidence weights of N frames of images to be processed, current_frame_conf_filtered represents the normalized processing result, that is, the filtering confidence, and the remaining parameters can be explained with reference to Formula 4, which will not be repeated here.

[0125] It should be understood that, to determine the filtering confidence based on the buffer area and the second confidence weight, the weighted sum of the second confidences of multiple frames in the buffer area can be calculated first, and then normalized; or the second confidence weights of multiple frames can be normalized first, and then the weighted sum is calculated. The present disclosure does not limit this.

[0126] In summary, according to the image processing method proposed in the present disclosure, the method includes: determining a buffer area and a second confidence weight; and determining a filter confidence based on the buffer area and the second confidence weight. The scheme of the present disclosure determines the buffer area and the second confidence weight, and determines the filter confidence based on the buffer area and the second confidence weight, thereby achieving smoothing of the second confidence, and further improving the quality of image processing.

[0127] The scheme disclosed in the present invention upgrades the confidence of the pixel points of the image to be processed to the confidence of the image frame, thereby realizing global alignment, local alignment selection and switching between different alignment modes based on the confidence of the image frame; filtering is performed through the confidence of the image frame, and abnormal data is eliminated, thereby improving the image processing quality.

[0128] Therefore, this solution has the following beneficial effects:

[0129] 1. By upgrading the confidence of the pixels of the image to be processed to the confidence of the image frame, global alignment, local alignment selection and switching between different alignment modes based on the confidence of the image frame can be realized, thereby improving the image processing quality.

[0130] 2. The image processing quality is further improved by filtering the confidence of the image frame and eliminating abnormal data.

[0131] Figure 7 The present invention provides a schematic diagram of the structure of an image processing device 700, wherein the image processing device 700 includes:

[0132] An acquisition unit 710 is used to acquire a first confidence level of the image to be processed, where the first confidence level is a confidence level of a pixel point of the image to be processed;

[0133] A determination unit 720, configured to determine a second confidence level based on the first confidence level, where the second confidence level is a confidence level of an image frame of the image to be processed;

[0134] The processing unit 730 is configured to process the image to be processed based on the second confidence level.

[0135] According to the image processing device proposed in the present disclosure, by obtaining the first confidence of the image to be processed, the first confidence is the confidence of the pixel points of the image to be processed; based on the first confidence, the second confidence of the image to be processed is determined, the second confidence is the confidence of the image frame of the image to be processed; based on the second confidence, the image to be processed is processed. The scheme of the present disclosure determines the confidence of the image frame of the image to be processed based on the confidence of the pixel points of the image to be processed, and uses the confidence of the image frame of the image to be processed to perform image processing, so that global alignment based on the confidence of the image frame, local alignment selection and switching between different alignment modes can be realized, thereby improving the accuracy of the image quality judgment result and the image processing quality.

[0136] In some embodiments, the determination unit 720 is also used to determine a third confidence of the image to be processed based on the first confidence, and the third confidence is used to determine the second confidence; determine the spatial weight and / or content weight of the third confidence; perform weighted processing on the third confidence based on the spatial weight and / or content weight, respectively, to determine the spatial confidence and / or content confidence of the image to be processed; and perform normalization processing on the spatial confidence and / or content confidence, respectively, to determine the second confidence.

[0137] In some embodiments, the determination unit 720 is further configured to perform a first filtering process on the first confidence level to determine a third confidence level.

[0138] In some embodiments, the acquisition unit 710 is further used to acquire a content saliency map of the image to be processed;

[0139] In some embodiments, the determination unit 720 is further configured to determine a content weight of a third confidence level based on the content saliency map and according to a content saliency lookup table.

[0140] In some embodiments, the processing unit 730 is further configured to perform a second filtering process on the second confidence level to determine the filtering confidence level of the image to be processed; and to eliminate abnormal data of the filtering confidence level.

[0141] In some embodiments, the determination unit 720 is further configured to determine a buffer area and a second confidence weight; and determine a filtering confidence based on the buffer area and the second confidence weight.

[0142] Since the device provided in the embodiment of the present disclosure corresponds to the methods provided in the above-mentioned embodiments, the implementation of the method is also applicable to the device provided in the embodiment and will not be described in detail in this embodiment.

[0143] Figure 8 800 is a schematic diagram of the structure of a communication device 800 provided in an embodiment of the present application. The communication device 800 may be a network device, or a terminal device, or a chip, a chip system, or a processor that supports the network device to implement the above method, or a chip, a chip system, or a processor that supports the terminal device to implement the above method. The device may be used to implement the method described in the above method embodiment, and the details may refer to the description in the above method embodiment.

[0144] The communication device 800 may include one or more processors 801. The processor 801 may be a general-purpose processor or a dedicated processor, etc. For example, it may be a baseband processor or a central processing unit. The baseband processor may be used to process the communication protocol and communication data, and the central processing unit may be used to control the communication device (such as a base station, a baseband chip, a terminal device, a terminal device chip, a DU or a CU, etc.), execute a computer program, and process the data of the computer program.

[0145] Optionally, the communication device 800 may further include one or more memories 802, on which a computer program 804 may be stored, and the processor 801 executes the computer program 804 so that the communication device 800 performs the method described in the above method embodiment. Optionally, data may also be stored in the memory 802. The communication device 800 and the memory 802 may be provided separately or integrated together.

[0146] Optionally, the communication device 800 may further include a transceiver 805 and an antenna 806. The transceiver 805 may be referred to as a transceiver unit, a transceiver, or a transceiver circuit, etc., and is used to implement a transceiver function. The transceiver 805 may include a receiver and a transmitter, the receiver may be referred to as a receiver or a receiving circuit, etc., and is used to implement a receiving function; the transmitter may be referred to as a transmitter or a transmitting circuit, etc., and is used to implement a transmitting function.

[0147] Optionally, the communication device 800 may further include one or more interface circuits 807. The interface circuit 807 is used to receive code instructions and transmit them to the processor 801. The processor 801 executes the code instructions to enable the communication device 800 to execute the method described in the above method embodiment.

[0148] In one implementation, the processor 801 may include a transceiver for implementing receiving and sending functions. For example, the transceiver may be a transceiver circuit, an interface, or an interface circuit. The transceiver circuit, interface, or interface circuit for implementing the receiving and sending functions may be separate or integrated. The above-mentioned transceiver circuit, interface, or interface circuit may be used for reading and writing code / data, or the above-mentioned transceiver circuit, interface, or interface circuit may be used for transmitting or delivering signals.

[0149] In one implementation, the processor 801 may store a computer program 803, which runs on the processor 801 and enables the communication device 800 to perform the method described in the above method embodiment. The computer program 803 may be fixed in the processor 801, in which case the processor 801 may be implemented by hardware.

[0150] In one implementation, the communication device 800 may include a circuit that can implement the functions of sending or receiving or communicating in the aforementioned method embodiments. The processor and transceiver described in the present application can be implemented in an integrated circuit (IC), an analog IC, a radio frequency integrated circuit RFIC, a mixed signal IC, an application specific integrated circuit (ASIC), a printed circuit board (PCB), an electronic device, etc. The processor and transceiver can also be manufactured using various IC process technologies, such as complementary metal oxide semiconductor (CMOS), N-type metal oxide semiconductor (NMOS), P-type metal oxide semiconductor (positive channel metal oxide semiconductor, PMOS), bipolar junction transistor (BJT), bipolar CMOS (BiCMOS), silicon germanium (SiGe), gallium arsenide (GaAs), etc.

[0151] The communication device described in the above embodiments may be a network device or a terminal device, but the scope of the communication device described in this application is not limited thereto, and the structure of the communication device may not be limited thereto. Figure 8 The communication device may be a stand-alone device or may be part of a larger device. For example, the communication device may be:

[0152] (1) Independent integrated circuit IC, or chip, or chip system or subsystem;

[0153] (2) having a set of one or more ICs, and optionally, the IC set may also include a storage component for storing data and computer programs;

[0154] (3) ASIC, such as modem;

[0155] (4) Modules that can be embedded in other devices;

[0156] (5) Receivers, terminal devices, intelligent terminal devices, cellular phones, wireless devices, handheld devices, mobile units, vehicle-mounted devices, network devices, cloud devices, artificial intelligence devices, etc.;

[0157] (6)Others

[0158] For the case where the communication device may be a chip or a chip system, see Fig. 9 Schematic diagram of the chip structure shown.

[0159] The embodiment of the present disclosure also provides a chip, such as Fig. 9 The chip shown includes a processor 901 and an interface 902. The number of the processor 901 may be one or more, and the number of the interface 902 may be multiple.

[0160] Optionally, the chip further includes a memory 903, and the memory 903 is used to store necessary computer programs and data.

[0161] The embodiments of the present disclosure further provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the above embodiments of the present disclosure.

[0162] Those skilled in the art may also appreciate that the various illustrative logic blocks and steps listed in the embodiments of the present application may be implemented by electronic hardware, computer software, or a combination of the two. Whether such functions are implemented by hardware or software depends on the design requirements of the specific application and the entire system. Those skilled in the art may use various methods to implement the functions for each specific application, but such implementation should not be understood as exceeding the scope of protection of the embodiments of the present application.

[0163] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0164] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0165] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0166] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (control method), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0167] It should be understood that the various parts of the embodiments of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (P GA), a field programmable gate array (FPGA), etc.

[0168] A person skilled in the art may understand that all or part of the steps in the above-mentioned embodiment method may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0169] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0170] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a first confidence level of the image to be processed, where the first confidence level is the confidence level of a pixel point of the image to be processed; Based on the first confidence level, determining a second confidence level of the image to be processed, where the second confidence level is a confidence level of an image frame of the image to be processed; Based on the second confidence level, the image to be processed is processed.

2. The method according to claim 1, characterized in that Determining a second confidence level of the image to be processed based on the first confidence level includes: Based on the first confidence level, determining a third confidence level of the image to be processed, wherein the third confidence level is used to determine the second confidence level; Determining a spatial weight and / or a content weight of the third confidence level; Based on the spatial weight and the content weight, respectively, a third confidence is weighted and summed to determine the spatial confidence and / or content confidence of the image to be processed; The spatial confidence and / or the content confidence are respectively normalized to determine a second confidence.

3. The method according to claim 2, characterized in that Determining a third confidence level based on the first confidence level includes: A first filtering process is performed on the first confidence level to determine the third confidence level.

4. The method according to claim 2, characterized in that: Determining the content weight of the third confidence level includes: Obtaining a content saliency map of the image to be processed; Based on the content saliency map, a content weight of the third confidence level is determined according to a content saliency lookup table.

5. The method according to claim 1, characterized in that The method further comprises: Performing filtering processing on the second confidence level to determine a filtering confidence level; The abnormal data of the filtering confidence is eliminated.

6. The method according to claim 5, characterized in that The filtering the second confidence level to determine the filtering confidence level includes: determining a buffer area and the second confidence weight; A filtering confidence is determined based on the buffer area and the second confidence weight.

7. An image processing device, characterized in that: include: An acquisition unit, used to acquire a first confidence level of the image to be processed, where the first confidence level is a confidence level of a pixel point of the image to be processed; A determining unit, configured to determine a second confidence level based on the first confidence level, wherein the second confidence level is a confidence level of the image frame of the image to be processed; A processing unit is used to process the image to be processed based on the second confidence level.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. A communication device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

10. A chip, characterized in that: The invention comprises one or more interfaces and one or more processors; the interface is used to receive a signal from a memory of a communication device and send the signal to the processor, wherein the signal includes a computer instruction stored in the memory; when the processor executes the computer instruction, the communication device executes the method described in any one of claims 1 to 6.

Images