Image processing method, device and electronic system

Abstract

The present invention provides an image processing method, device and electronic system, which determine the pixel points to be interpolated from the image to be interpolated; obtain the color reference information of the pixel points to be interpolated from the color reference map corresponding to the image to be interpolated; determine the pixel point group to which the pixel points to be interpolated belong according to the color reference information of the pixel points to be interpolated; and determine the pixel value of the pixel points to be interpolated based on the pixel values ​​of each pixel point in the pixel point group to which the pixel points to be interpolated belong. In this method, the pixel point group is determined using the color reference information of the image to be interpolated, and then the pixel value of the pixel point to be interpolated is determined based on the pixel values ​​of the pixels in the group. Since the color reference information of the pixels in the same group is similar, this method can avoid using the pixels with large color differences on both sides of the edge for interpolation calculation, and for special images such as disparity maps and optical flow maps, the accuracy of the interpolated image is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method, device and electronic system. Background Art

[0002] Image interpolation is the process of calculating the pixel values ​​of unknown pixel points using pixel points with known pixel values ​​on an image. Pixel points with unknown pixel values ​​are also called interpolation points. In related technologies, the pixel values ​​of interpolation points are usually calculated based on the pixel values ​​of the pixels around the interpolation points, such as the nearest neighbor interpolation algorithm, bilinear interpolation algorithm, cubic polynomial interpolation algorithm, etc. Usually, there are obvious differences between the pixel points on both sides of the edge of an area in an image. When the interpolation point is located at the edge, the pixel value of the interpolation point is usually calculated based on the pixel points on both sides of the edge, resulting in blurred edges of the interpolated image. This problem can easily lead to inaccurate images after special images such as disparity maps and optical flow maps are interpolated in the above manner. Summary of the invention

[0003] In view of this, an object of the present invention is to provide an image processing method, device and electronic system to improve the accuracy of images after interpolation of special images such as disparity maps and optical flow maps.

[0004] In a first aspect, an embodiment of the present invention provides an image processing method, the method comprising: determining a pixel to be interpolated from an image to be interpolated; obtaining color reference information of the pixel to be interpolated from a color reference map corresponding to the image to be interpolated; determining a pixel group to which the pixel to be interpolated belongs based on the color reference information of the pixel to be interpolated; wherein, based on the color reference information of the pixel whose pixel values ​​have been determined in the image to be interpolated, the pixel whose pixel values ​​have been determined are pre-divided into a plurality of pixel groups; and determining the pixel value of the pixel to be interpolated based on the pixel values ​​of each pixel in the pixel group to which the pixel to be interpolated belongs.

[0005] Furthermore, the image to be interpolated includes: a disparity map or an optical flow map generated based on a color reference map.

[0006] Furthermore, among the multiple pixel point groups, each pixel point group corresponds to a pixel feature; different pixel point groups correspond to different pixel features; the step of determining the pixel point group to which the pixel point to be interpolated belongs based on the color reference information of the pixel point to be interpolated includes: determining the target pixel feature of the pixel point to be interpolated based on the color reference information of the pixel point to be interpolated; and grouping the pixel points corresponding to the target pixel feature as the pixel point group to which the pixel point to be interpolated belongs.

[0007] Furthermore, the step of determining the target pixel features of the pixel to be interpolated based on the color reference information of the pixel to be interpolated includes: determining the target pixel features of the pixel to be interpolated based on the color reference information of the pixel to be interpolated and the position information of the pixel to be interpolated in the image to be interpolated.

[0008] Furthermore, based on the color reference information of the pixel to be interpolated and the position information of the pixel to be interpolated in the image to be interpolated, the step of determining the target pixel feature of the pixel to be interpolated includes: based on a preset conversion rule, converting the color reference information of the pixel to be interpolated and the position information of the pixel to be interpolated in the image to be interpolated into a one-dimensional feature value; wherein the color reference information of the pixel to be interpolated includes: a plurality of color channel values ​​of the pixel to be interpolated in the color reference image; the position information of the pixel to be interpolated in the image to be interpolated includes: an X coordinate value and a Y coordinate value; and determining the target pixel feature of the pixel to be interpolated according to the one-dimensional feature value.

[0009] Further, based on a preset conversion rule, the step of converting the color reference information of the pixel to be interpolated and the position information of the pixel to be interpolated in the image to be interpolated into a one-dimensional eigenvalue includes: one-dimensional eigenvalue key=x / sigma_s+y / sigma_s*255+r / sigma_c*255^2+g / sigma_c*255^3+b / sigma_c*255^4; wherein key is the one-dimensional eigenvalue; x is the X coordinate value; y is the Y coordinate value; r is the R channel value; g is the G channel value; b is the B channel value; sigma_s is the spatial smoothing parameter corresponding to the image to be interpolated, and sigma_c is the color smoothing parameter corresponding to the image to be interpolated.

[0010] Furthermore, the step of determining the target pixel features of the pixel points to be interpolated based on the one-dimensional feature values ​​includes: determining the pixel features corresponding to the pixel features of each pixel group, whose distance from the one-dimensional feature values ​​is less than a preset distance threshold, as the target pixel features of the pixel points to be interpolated.

[0011] Furthermore, before the step of determining the pixel points to be interpolated from the image to be interpolated, the method also includes: obtaining pixel points with determined pixel values ​​from the image to be interpolated; obtaining color reference information of the pixel points with determined pixel values ​​from a color reference map corresponding to the image to be interpolated; for each pixel point with determined pixel value, determining the pixel feature of the pixel point based on the color reference information of the pixel point; and dividing the pixel points with the same pixel features into the same pixel point group to obtain multiple pixel point groups.

[0012] Furthermore, based on the pixel values ​​of each pixel in the pixel group to which the pixel to be interpolated belongs, the step of determining the pixel value of the pixel to be interpolated includes: calculating the weighted average of the pixel values ​​of each pixel in the pixel group to which the pixel to be interpolated belongs, and determining the weighted average as the pixel value of the pixel to be interpolated.

[0013] Furthermore, the step of determining the pixel points to be interpolated from the image to be interpolated includes: obtaining a mask image of the image to be interpolated based on the image to be interpolated or the color reference image; and determining the pixel points to be interpolated from the image to be interpolated based on the mask image.

[0014] Furthermore, the method also includes: if there are multiple pixels to be interpolated, traverse each pixel to be interpolated according to a preset traversal rule; for each pixel to be interpolated, execute the step of obtaining color reference information of the pixel to be interpolated from a color reference map corresponding to the image to be interpolated until the pixel value of the pixel to be interpolated is obtained; when each pixel to be interpolated in the pixel group to which the pixel to be interpolated belongs obtains a pixel value, update each pixel to be interpolated in the pixel group to which the pixel to be interpolated belongs to a pixel with a determined pixel value.

[0015] In a second aspect, an embodiment of the present invention provides an image processing device, comprising: an acquisition module, used to determine a pixel to be interpolated from an image to be interpolated; acquiring color reference information of the pixel to be interpolated from a color reference map corresponding to the image to be interpolated; a grouping module, used to determine the pixel group to which the pixel to be interpolated belongs according to the color reference information of the pixel to be interpolated; wherein, based on the color reference information of the pixel whose pixel values ​​have been determined in the image to be interpolated, the pixel whose pixel values ​​have been determined are pre-divided into a plurality of pixel groups; a determination module, used to determine the pixel value of the pixel to be interpolated based on the pixel values ​​of each pixel in the pixel group to which the pixel to be interpolated belongs.

[0016] In a third aspect, an embodiment of the present invention provides an electronic system, the electronic system comprising: a processing device and a storage device; a computer program is stored on the storage device, and the computer program executes the image processing method as any one of the first aspects when the processed device is running.

[0017] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processing device, the steps of the image processing method as described in any one of the first aspects are executed.

[0018] The embodiments of the present invention bring the following beneficial effects:

[0019] The embodiment of the present invention provides an image processing method, device and electronic system, which determines the pixel points to be interpolated from the image to be interpolated; obtains the color reference information of the pixel points to be interpolated from the color reference map corresponding to the image to be interpolated; determines the pixel point group to which the pixel points to be interpolated belong according to the color reference information of the pixel points to be interpolated; wherein, based on the color reference information of the pixel points with determined pixel values ​​in the image to be interpolated, the pixel points with determined pixel values ​​are pre-divided into a plurality of pixel point groups; and the pixel values ​​of the pixel points to be interpolated are determined based on the pixel values ​​of each pixel point in the pixel point group to which the pixel points to be interpolated belong. In this method, the pixel point group is determined using the color reference information of the image to be interpolated, and then the pixel values ​​of the pixel points to be interpolated are determined based on the pixel values ​​of the pixels in the group. Since the color reference information of the pixels in the same group is similar, this method can avoid using the pixels with large color differences on both sides of the edge for interpolation calculation, and improves the accuracy of the interpolated image for special images such as disparity maps and optical flow maps.

[0020] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0021] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 A schematic diagram of the positions of pixels of an interpolation calculation method provided by an embodiment of the present invention;

[0024] Figure 2 A schematic diagram of the structure of an electronic system provided by an embodiment of the present invention;

[0025] Figure 3 A flow chart of an image processing method provided by an embodiment of the present invention;

[0026] Figure 4 A flow chart of another image processing method provided by an embodiment of the present invention;

[0027] Figure 5A flow chart of another image processing method provided by an embodiment of the present invention;

[0028] Figure 6 A schematic diagram of pixel grouping provided by an embodiment of the present invention;

[0029] Figure 7 A schematic diagram of another pixel grouping method provided by an embodiment of the present invention;

[0030] Figure 8 A flowchart of a specific image processing method provided by an embodiment of the present invention;

[0031] Fig. 9 A schematic structural diagram of an image processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0033] The current image interpolation algorithm uses the area information corresponding to the known pixels in the image to interpolate the pixels in the unknown area. Common interpolation algorithms include nearest neighbor interpolation, bilinear interpolation, cubic polynomial interpolation, etc. The above algorithms are single-input and single-output. For example, in the bilinear interpolation algorithm, the pixel value of the pixel to be interpolated is calculated based on the linear expression of the pixels around the pixel to be interpolated, such as Figure 1 As shown, P0 is an unknown pixel point, P1 to P4 are known pixels points, and assuming that P0 is at the center of the rectangle, the value of P0 obtained by bilinear interpolation is:

[0034]

[0035] I p0 is the pixel value of P0 pixel; I p1 is the pixel value of pixel P1; I p2 is the pixel value of P2 pixel; I p3 is the pixel value of pixel P3; I p4 is the pixel value of pixel P4;

[0036] In addition, when the P0 pixel is not in the center, the bilinear method: the pixel value of the P0 pixel is calculated by the linear expression of the P1~P4 pixel values; the nearest neighbor method: the pixel value of the pixel closest to the P0 pixel is directly assigned; the cubic polynomial: it is calculated by the nonlinear expression of the pixel values ​​of the P1~P4 pixel points.

[0037] The above traditional interpolation methods have good image interpolation effects in flat non-edge areas of the image. However, when the area to be interpolated is relatively large and there are pixels located in the edge area, the pixel values ​​of the pixels to be interpolated in the edge part are calculated from the pixel values ​​on both sides of the edge, resulting in blurred edges of the image after interpolation. In image processing, disparity maps and optical flow maps often need to be interpolated in occluded areas. If the above traditional interpolation algorithms are used, the disparity values ​​of some occluded areas will be linearly smoothed and between the foreground disparity and the background disparity, or, after interpolation, the occluded areas are partially interpolated to the values ​​of the foreground area, while the correct values ​​should be the values ​​of the background. This results in inaccurate images after interpolation of special images such as disparity maps and optical flow maps in the above manner.

[0038] Based on this, an image processing method, device and electronic system provided by an embodiment of the present invention can be applied to computers, smart phones, cameras, tablet computers and other devices. The technology can be implemented using relevant software and hardware, and is described below through embodiments. To facilitate understanding of this embodiment, an image processing method disclosed in an embodiment of the present invention is first introduced in detail.

[0039] Embodiment 1:

[0040] First, refer to Figure 2 An exemplary electronic system 100 for implementing the image processing method, apparatus, and electronic system according to the embodiments of the present invention is described.

[0041] like Figure 2 The electronic system 100 includes one or more processing devices 102, one or more storage devices 104, an input device 106, an output device 108, and may also include one or more image acquisition devices 110. These components are interconnected through a bus system 112 and / or other forms of connection mechanisms (not shown). It should be noted that Figure 2 The components and structure of the electronic system 100 shown are merely exemplary and non-limiting. The electronic system may also have other components and structures as required.

[0042] The processing device 102 can be a gateway, or a smart terminal, or a device including a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities. It can process data of other components in the electronic system 100 and control other components in the electronic system 100 to perform desired functions.

[0043] The storage device 104 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processing device 102 may run the program instructions to implement the client functions and / or other desired functions in the embodiments of the present invention (implemented by the processing device) described below. Various applications and various data, such as various data used and / or generated by the application, may also be stored in the computer-readable storage medium.

[0044] The input device 106 may be a device used by a user to input instructions, and may include one or more of a keyboard, a mouse, a microphone, a touch screen, and the like.

[0045] The output device 108 may output various information (eg, images or sounds) to the outside (eg, a user), and may include one or more of a display, a speaker, and the like.

[0046] The image acquisition device 110 can acquire preview video frames or picture data (such as images to be processed or target video frames), and store the acquired preview video frames or image data in the storage device 104 for use by other components.

[0047] Exemplarily, the components in the example electronic system for implementing the image processing method, device and electronic system according to the embodiment of the present invention can be integrated or dispersed, such as integrating the processing device 102, the storage device 104, the input device 106 and the output device 108 into one body, and setting the image acquisition device 110 at a specified position where the image can be acquired. When the components in the above electronic system are integrated, the electronic system can be implemented as a smart terminal such as a camera, a smart phone, a tablet computer, a computer, a vehicle-mounted terminal, a video camera, etc.

[0048] Embodiment 2:

[0049] This embodiment provides an image processing method, such as Figure 3 As shown, the method comprises the following steps:

[0050] Step S302, determining a pixel to be interpolated from the image to be interpolated; obtaining color reference information of the pixel to be interpolated from a color reference map corresponding to the image to be interpolated;

[0051] The above-mentioned image to be interpolated can be various types of images, such as a multi-channel color image, a single-channel or multi-channel depth image, a disparity map, an optical flow map, etc.; the above-mentioned pixel points to be interpolated usually refer to unknown areas in the image to be interpolated, such as occluded areas in special images such as disparity maps, optical flow maps or depth maps, which often need to be interpolated, and the occluded areas are usually determined as pixels to be interpolated. The above-mentioned color reference map can be a multi-channel color map, such as a three-channel color map, etc., which corresponds to the image to be interpolated. When the image to be interpolated is a depth map, a disparity map, or an optical flow map, the image to be interpolated corresponding to the above can be generated based on the color map; the color reference information of the above-mentioned pixel points to be interpolated usually includes information of multiple color channels, such as the most basic RGB color channel information, which is obtained by changing the three color channels of red (R), green (G), and blue (B) and their superposition information; since the image to be interpolated may not be in color, the above-mentioned color reference information is obtained from the color reference map corresponding to the image to be interpolated.

[0052] In actual implementation, firstly, based on the color reference map, the corresponding image to be interpolated can be generated through image transformation and other methods, and the pixel to be interpolated can be determined according to the occluded area in the image to be interpolated; each pixel in the image to be interpolated corresponds one-to-one to each pixel in the corresponding color reference map, so according to the corresponding relationship, the color reference information of the pixel to be interpolated can be obtained from the color reference map, that is, the color information of the pixel to be interpolated in the color reference map, which usually includes the superposition information between the three primary colors of the pixel to be interpolated in the color reference map and the corresponding color channel value.

[0053] Step S304, determining the pixel group to which the pixel to be interpolated belongs according to the color reference information of the pixel to be interpolated; wherein the pixel points with determined pixel values ​​are pre-divided into a plurality of pixel groups based on the color reference information of the pixel points with determined pixel values ​​in the image to be interpolated;

[0054] The above-mentioned pixel point grouping includes pixel points with determined pixel values ​​in the image to be interpolated, wherein the positions of the pixel points with determined pixel values ​​in the same pixel point group in the image to be interpolated may be different, and the corresponding color reference color information may be different, but the characteristics of the individual pixel points in each pixel point group are the same.

[0055] Specifically, according to the color reference information of the pixel points with determined pixel values ​​in the image to be interpolated, the pixel points with similar color reference information are divided into the same pixel point group, and the pixel points included in the same pixel point group are determined as a pixel point group; usually a color reference map corresponding to an image to be interpolated includes multiple colors, so according to the color reference information of the pixel points with determined pixel values, it can be divided into multiple pixel point groups, each pixel point group has a similar color reference information; the color reference information of the pixel points to be interpolated can be matched with the color reference information corresponding to the multiple pixel point groups, and the pixel point group matching the color reference information of the pixel points to be interpolated is determined.

[0056] Step S306 , determining the pixel value of the pixel to be interpolated based on the pixel values ​​of each pixel in the pixel group to which the pixel to be interpolated belongs.

[0057] When the image to be interpolated is a disparity map, the above pixel values ​​are disparity values; when the image to be interpolated is an optical flow map, the above pixel values ​​are optical flow values.

[0058] Specifically, the average value of the pixel values ​​of each pixel in the pixel group to which the pixel to be interpolated belongs can be calculated, and the average value can be determined as the pixel value of the pixel to be interpolated; the pixel values ​​of each pixel in the pixel group to which the pixel to be interpolated belongs can also be mathematically counted, and the median or mode can be determined as the pixel value of the pixel to be interpolated; and a pixel value can also be randomly selected from the pixel values ​​of each pixel in the pixel group to which the pixel to be interpolated belongs, and determined as the pixel value of the pixel to be interpolated.

[0059] The embodiment of the present invention provides an image processing method, which determines the pixel points to be interpolated from the image to be interpolated; obtains the color reference information of the pixel points to be interpolated from the color reference map corresponding to the image to be interpolated; determines the pixel point group to which the pixel points to be interpolated belong according to the color reference information of the pixel points to be interpolated; wherein, based on the color reference information of the pixel points whose pixel values ​​have been determined in the image to be interpolated, the pixel points whose pixel values ​​have been determined are pre-divided into a plurality of pixel point groups; and determines the pixel value of the pixel points to be interpolated based on the pixel values ​​of each pixel point in the pixel point group to which the pixel points to be interpolated belong. In this method, the pixel point group is determined using the color reference information of the image to be interpolated, and then the pixel value of the pixel points to be interpolated is determined based on the pixel values ​​of the pixels in the group. Since the color reference information of the pixels in the same group is similar, this method can avoid using the pixels with large color differences on both sides of the edge for interpolation calculation, and improves the accuracy of the interpolated image for special images such as disparity maps and optical flow maps.

[0060] Embodiment three:

[0061] This embodiment provides another image processing method, which is implemented on the basis of the above embodiment. In the above multiple pixel point groups, each pixel point group corresponds to a pixel feature; different pixel point groups correspond to different pixel features.

[0062] The above pixel features can be determined based on the color reference information of each pixel point; the color reference information of each pixel point has certain differences, so different pixel point groups correspond to different pixel features.

[0063] This embodiment focuses on describing the specific implementation process of the step of determining the pixel group to which the pixel to be interpolated belongs according to the color reference information of the pixel to be interpolated (implemented through steps S404-S406). Figure 4 As shown, the method comprises the following steps:

[0064] Step S402, determining the pixel to be interpolated from the image to be interpolated; obtaining the color reference information of the pixel to be interpolated from the color reference map corresponding to the image to be interpolated;

[0065] Step S404, determining the target pixel feature of the pixel to be interpolated based on the color reference information of the pixel to be interpolated;

[0066] Specifically, the color reference information of the pixel to be interpolated can be converted into a value of fixed length according to an information conversion method, such as hash function, list, etc., and the fixed-length value is determined as the target pixel feature of the pixel to be interpolated; or, the characteristic information value is extracted from the color reference information of the pixel to be interpolated, and the characteristic information value is determined as the target pixel feature of the pixel to be interpolated.

[0067] In a preferred embodiment, the step of determining the target pixel features of the pixel to be interpolated based on the color reference information of the pixel to be interpolated includes: determining the target pixel features of the pixel to be interpolated based on the color reference information of the pixel to be interpolated and the position information of the pixel to be interpolated in the image to be interpolated.

[0068] The above-mentioned position information can be coordinate information of a two-dimensional space; for example, a two-dimensional coordinate is established in the image to be interpolated, and the position information of the pixel to be interpolated is the horizontal coordinate value and the vertical coordinate value of the two-dimensional coordinate. Specifically, the color reference information of the pixel to be interpolated and the position information of the pixel to be interpolated in the image to be interpolated can be converted into a fixed-length value according to an information conversion method, such as a hash function or a list method, and the fixed-length value is determined as the target pixel feature of the pixel to be interpolated; or, the color reference information of the pixel to be interpolated and the position information of the pixel to be interpolated in the image to be interpolated are used to extract a feature information value, and the feature information value is determined as the target pixel feature of the pixel to be interpolated.

[0069] A preferred implementation of the step of determining the target pixel feature of the pixel to be interpolated based on the color reference information of the pixel to be interpolated and the position information of the pixel to be interpolated in the image to be interpolated includes:

[0070] Step A1, based on a preset conversion rule, converting the color reference information of the pixel to be interpolated and the position information of the pixel to be interpolated in the image to be interpolated into a one-dimensional feature value; wherein the color reference information of the pixel to be interpolated includes: a plurality of color channel values ​​of the pixel to be interpolated in the color reference image; the position information of the pixel to be interpolated in the image to be interpolated includes: an X coordinate value and a Y coordinate value;

[0071] The above preset conversion rules can be understood as a hash method, which can be understood as transforming an input of any length (also called a pre-image) into an output of a fixed length through a hash algorithm, and the output is the hash value; this conversion is a compression mapping, that is, the spatial dimension of the hash value is usually much smaller than the spatial dimension of the input, and different inputs may hash to the same output.

[0072] In this embodiment, the multiple color channel values ​​of the above-mentioned pixel points to be interpolated in the color reference image can be the color channel values ​​of the three colors of RGB, and the position information of the pixel points to be interpolated in the image to be interpolated can be the horizontal and vertical coordinate values ​​of the XY coordinate axes. Therefore, the color reference information of the pixel points to be interpolated and the position information of the pixel points to be interpolated in the image to be interpolated can be understood as information in a five-dimensional space. Through a preset hash algorithm, for example, the color reference information and position information of the five-dimensional space are converted into a one-dimensional eigenvalue; the above-mentioned one-dimensional eigenvalue can be understood as a hash value.

[0073] With respect to the above-mentioned step of converting the color reference information of the pixel to be interpolated and the position information of the pixel to be interpolated in the image to be interpolated into a one-dimensional feature value based on the preset conversion rule, a preferred implementation manner includes:

[0074] One-dimensional eigenvalue key=x / sigma_s+y / sigma_s*255+r / sigma_c*255^2+g / sigma_c*255^3+b / sigma_c*255^4; wherein key is a one-dimensional eigenvalue; x is an X coordinate value; y is a Y coordinate value; r is an R channel value; g is a G channel value; b is a B channel value; sigma_s is a spatial smoothing parameter corresponding to the image to be interpolated, and sigma_c is a color smoothing parameter corresponding to the image to be interpolated.

[0075] The above r is the R channel value, g is the G channel value, and b is the B channel value, which are the color reference information of the pixel to be interpolated; the above x is the X coordinate value, and y is the Y coordinate value, which are the position information of the pixel to be interpolated in the image to be interpolated; where sigma_s is the spatial smoothing parameter corresponding to the image to be interpolated, and sigma_c is the color smoothing parameter corresponding to the image to be interpolated. Usually, the spatial smoothing parameter and the color smoothing parameter are fixed for the current image to be interpolated. When the image to be interpolated changes, the corresponding spatial smoothing parameter and the color smoothing parameter will also change according to specific needs. For example, if the color requirements are relatively strict, the value of the color smoothing parameter can be set relatively small; for another example, if the occluded area in the image to be interpolated is large, the value of the spatial smoothing parameter can be set relatively large. The above formula can be used to convert a five-dimensional spatial information into a one-dimensional feature value.

[0076] It should be noted that the calculation method of the pixel features corresponding to each pixel group is also calculated in the above method, that is, based on the color reference information of the pixels in the pixel group and the position information of the pixels in the pixel group in the image to be interpolated, the target pixel features of the pixels in the pixel group are determined. For the pixels in the same pixel group, even if the position information and the color reference information are different, the pixel features of each pixel determined are the same; therefore, the pixel features of each pixel included in each pixel group are the same.

[0077] Step A2: determining the target pixel feature of the pixel to be interpolated according to the one-dimensional feature value.

[0078] After calculating the one-dimensional feature value of the pixel to be interpolated, the one-dimensional feature value can be directly determined as the target pixel feature of the pixel to be interpolated; the one-dimensional feature value of the pixel to be interpolated can also be matched with the pixel features corresponding to each pixel group, and the pixel features that meet the threshold can be determined as the target pixel features of the pixel to be interpolated.

[0079] In a preferred implementation, pixel features corresponding to each pixel point group, whose distance to the one-dimensional feature value is less than a preset distance threshold, are determined as target pixel features of the pixel point to be interpolated.

[0080] The above-mentioned preset distance threshold can be set specifically according to the area size and color of the pixel to be interpolated in the image to be interpolated; specifically, since pixels with similar distances usually have similar color reference information and the same pixel features, the position distance of the one-dimensional feature value, that is, the pixel feature whose horizontal and vertical coordinate values ​​are less than the preset distance threshold, can be determined as the target pixel feature of the pixel to be interpolated.

[0081] Step S406, grouping the pixel points corresponding to the target pixel feature and determining them as the pixel point group to which the pixel point to be interpolated belongs;

[0082] It can be understood that the pixel values ​​of the pixels included in the pixel point group corresponding to the target pixel feature and the pixel features of each pixel are very close to the pixel values ​​and pixel features of the pixel point to be interpolated, that is, the pixel point group corresponding to the target pixel feature can be determined as the pixel point group to which the pixel point to be interpolated belongs.

[0083] Step S408 : determining the pixel value of the pixel to be interpolated based on the pixel values ​​of each pixel in the pixel group to which the pixel to be interpolated belongs.

[0084] In this method, the one-dimensional feature value of the pixel to be interpolated and the pixel features corresponding to the pixel group are calculated. Based on the hash method, the high-dimensional color reference information and spatial position information of the pixel are converted into a one-dimensional feature value; the pixel features corresponding to each pixel group whose distance from the one-dimensional feature value is less than the preset distance threshold are determined as the target pixel features of the pixel to be interpolated; the pixel points corresponding to the target pixel features are grouped and determined as the pixel group to which the pixel to be interpolated belongs; based on the pixel values ​​of each pixel in the pixel group to which the pixel to be interpolated belongs, the pixel value of the pixel to be interpolated is determined. The inherent defects of the traditional interpolation algorithm that completely relies on the pixel information of the image to be interpolated and the interpolation result is inconsistent with the original image mapping are solved, and the edge-preserving interpolation is achieved. After interpolation, the effect of real parallax can be obtained. At the same time, this method is based on high-dimensional hash implementation, and the performance is comparable to that of traditional interpolation. Since the color reference information of the pixels in the same group is similar, this method can avoid using pixels with large color differences on both sides of the edge for interpolation calculation, and improves the accuracy of the interpolated image for special images such as disparity maps and optical flow maps.

[0085] Embodiment 4:

[0086] This embodiment provides another image processing method, which is implemented on the basis of the above embodiment. This embodiment focuses on the specific implementation process of the steps before the step of determining the pixel point to be interpolated from the image to be interpolated (implemented by steps S502-S508), and the specific implementation process of the step of determining the pixel value of the pixel point to be interpolated based on the pixel value of each pixel point in the pixel point group to which the pixel point to be interpolated belongs (implemented by step S510), and the specific implementation process of the step of determining the pixel point to be interpolated from the image to be interpolated (implemented by step S516), such as Figure 5 As shown, the method comprises the following steps:

[0087] Step S502, obtaining pixel points with determined pixel values ​​from the image to be interpolated;

[0088] The pixel points with determined pixel values ​​mentioned above usually refer to the non-occluded areas in the image to be interpolated; for example, the pixel points in the non-black area in the depth map.

[0089] Step S504, obtaining color reference information of pixel points whose pixel values ​​have been determined from a color reference map corresponding to the image to be interpolated;

[0090] Since the pixels in the image to be interpolated correspond one-to-one to the pixels in the corresponding color reference image, for each pixel with a determined pixel value in the image to be interpolated, the color reference information of the pixel at that position can be obtained from the corresponding color reference image; the color reference information can be the color channel values ​​of the three colors RGB.

[0091] Step S506, for each pixel point whose pixel value has been determined, determining the pixel feature of the pixel point based on the color reference information of the pixel point;

[0092] Preferably, according to the aforementioned preset conversion rule, the color reference information of the pixel point with the determined pixel value and the position information of the pixel point with the determined pixel value in the image to be interpolated can be converted into a one-dimensional feature value of the pixel point; the one-dimensional feature value can be determined as the pixel feature of the pixel point. The color reference information of the pixel point with the determined pixel value includes: multiple color channel values ​​of the pixel point with the determined pixel value in the color reference map; the position information of the pixel point with the determined pixel value in the image to be interpolated includes: X coordinate value and Y coordinate value.

[0093] Step S508, dividing the pixels with the same pixel features into the same pixel group to obtain a plurality of pixel groups;

[0094] For each pixel point with a determined pixel value, the pixel features of multiple pixel points can be determined; the pixel features of each pixel point may be the same or different, and usually the pixel features of pixels with similar distances and similar colors are the same; therefore, the pixels with the same pixel features can be divided into the same pixel point group, and multiple pixel point groups can be obtained, and the pixel features corresponding to each pixel point group are different.

[0095] See also Figure 6 The schematic diagram of pixel grouping shown in the figure is a five-dimensional space pixel represented by a two-dimensional space, wherein the horizontal axis is the RGB color channel value of the pixel with a determined pixel value, and the vertical axis is the X-coordinate value and Y-coordinate value of the pixel with a determined pixel value in the image to be interpolated; each circle in the figure, namely table1, table2, and table3, represents each pixel grouping, the number of symbols in each circle represents the number of pixels included in the pixel grouping, and the symbol in the circle represents a pixel feature; it can be seen from the figure that there are three symbols in total, and each symbol represents a pixel feature.

[0096] Step S510, based on the image to be interpolated or the color reference image, a mask image of the image to be interpolated is obtained; based on the mask image, pixels to be interpolated are determined from the image to be interpolated;

[0097] The above-mentioned mask graphic can be a mask image of the corresponding image to be interpolated generated by image conversion or the like from the image to be interpolated or the color reference image. The mask image can be a single-channel mask image. Since the pixels in the image to be interpolated and the corresponding mask image correspond to each other, the pixels in the black area of ​​the mask image can be determined as the pixels to be interpolated. Specifically, the pixels at the corresponding positions in the image to be interpolated can be obtained according to the positions of the pixels in the black area of ​​the mask image, and the pixels are determined as the pixels to be interpolated.

[0098] Step S512, obtaining color reference information of the pixel to be interpolated from the color reference map corresponding to the image to be interpolated; and determining the target pixel feature of the pixel to be interpolated based on the color reference information of the pixel to be interpolated;

[0099] Step S514, grouping the pixel points corresponding to the target pixel feature and determining them as the pixel point group to which the pixel point to be interpolated belongs;

[0100] Step S516, calculating the weighted average of the pixel values ​​of each pixel in the pixel group to which the pixel to be interpolated belongs, and determining the weighted average as the pixel value of the pixel to be interpolated.

[0101] After obtaining the pixel group to which the pixel to be interpolated belongs, for example Figure 7The schematic diagram of pixel grouping shown in the figure, table1 corresponds to the first pixel grouping, table2 corresponds to the second pixel grouping, and table3 corresponds to the third pixel grouping; the pixel points to be interpolated p1 and p2 in the figure belong to the first pixel grouping; the pixel point to be interpolated p3 belongs to the second pixel grouping; the weighted average of the pixel values ​​of each pixel in the pixel group to which the pixel point to be interpolated belongs can be calculated, as shown in the following formula:

[0102]

[0103]

[0104] Among them, I p1 represents the pixel value of the pixel point p1 to be interpolated, wherein when the image to be interpolated is a disparity map, the pixel value can also be called a disparity value; when the image to be interpolated is an optical flow map, the pixel value can also be called an optical flow value; I p2 I represents the pixel value of the pixel point p2 to be interpolated; p3 represents the pixel value of the pixel point p3 to be interpolated; n and m represent the number of pixels in table1 and table2; table1[i] and table2[i] represent the pixel values ​​of each pixel in the pixel point group to which the pixel point to be interpolated belongs; therefore, according to the above formula, when p1 and p2 are in the same pixel point group, the pixel values ​​of p1 and p2 are equal.

[0105] In this method, pixel information in a five-dimensional space is established based on a color reference image and a mask image, and pixel points with determined pixel values ​​are grouped; the weighted average of the pixel values ​​of each pixel in the pixel point group to which the pixel point to be interpolated belongs is calculated, and the weighted average is determined as the pixel value of the pixel point to be interpolated. This solves the inherent defect of the traditional interpolation algorithm that it completely relies on the pixel point information of the image to be interpolated, and the interpolation result is inconsistent with the original image mapping. Edge-preserving interpolation is achieved, and the effect of real parallax can be obtained after interpolation. At the same time, this method is based on high-dimensional hash implementation, and its performance is comparable to that of traditional interpolation. Since the color reference information of pixels in the same group is similar, this method can avoid using pixels with large color differences on both sides of the edge for interpolation calculations. For special images such as disparity maps and optical flow maps, the accuracy of the interpolated image is improved.

[0106] Furthermore, the above method also includes: if there are multiple pixel points to be interpolated, traversing each pixel point to be interpolated according to a preset traversal rule; for each pixel point to be interpolated, executing the step of obtaining color reference information of the pixel point to be interpolated from a color reference map corresponding to the image to be interpolated until the pixel value of the pixel point to be interpolated is obtained; when each pixel point to be interpolated in the pixel point group to which the pixel point to be interpolated belongs obtains a pixel value, updating each pixel point to be interpolated in the pixel point group to which the pixel point to be interpolated belongs to a pixel point with a determined pixel value.

[0107] The preset traversal rule can be to traverse the pixels row by row according to the image to be interpolated, or when the area with determined pixel values ​​is large, it can be based on the image to be interpolated, starting from the pixel to be interpolated near the area with determined pixel values, and traversing from the outside to the inside using the interpolation order in the form of "onion skin"; for each pixel to be interpolated, complete the above steps until the pixel value of the pixel to be interpolated is obtained; if each pixel to be interpolated in the pixel group to which the pixel to be interpolated belongs has obtained a pixel value; in the mask image, update the pixel value of each pixel to be interpolated in the pixel group, and update each pixel to be interpolated in the pixel group to a pixel with a determined pixel value; repeat the above operation of updating the pixel value of each pixel to be interpolated in each pixel group until the pixel value of each pixel to be interpolated in all pixel groups is updated. The number of pixels to be interpolated in the pixel group to which the above pixel to be interpolated belongs may be one or more.

[0108] See also Figure 8 As shown in the flowchart of a specific image processing method, the color guide image src_image corresponds to the aforementioned color reference image; the depth map to be interpolated target_image corresponds to the aforementioned image to be interpolated; the area to be interpolated mask corresponds to the aforementioned mask image; and the table set corresponds to the aforementioned pixel grouping.

[0109] Specifically, when executing the image processing method, the color guide map src_image, the depth map to be interpolated target_image, and the mask of the area to be interpolated are input, wherein the depth map to be interpolated target_image is generated by the color guide map src_image; the mask of the area to be interpolated can be generated by the color guide map src_image or by the depth map to be interpolated target_image; according to the color guide map src_image and the mask of the area to be interpolated, the position information and color reference information of the pixel points to be interpolated are obtained, and according to the position information Information and color reference information are used to establish pixel point grouping in the pixel point area with determined pixel values ​​under the (x, y, r, g, b) coordinates. The hash method is used to calculate key = x / sigma_s + y / sigma_s*255 + r / sigma_c*255^2 + g / sigma_c*255^3 + b / sigma_c*255^4; where key corresponds to the aforementioned one-dimensional eigenvalue; x is the X coordinate value; y is the Y coordinate value; r is the R channel value; g is the G channel value; b is the B channel value; sigma_s is the spatial smoothing parameter corresponding to the image to be interpolated, sigm a_c is the color smoothing parameter corresponding to the image to be interpolated; then set the number of iterations, and traverse the pixels to be interpolated in the area to be interpolated according to the preset traversal rules; for the traversed pixels to be interpolated, calculate the key value of each pixel to be interpolated, and the calculation method is the same as that of the pixel area with known pixel values; according to the calculated key value of the pixel to be interpolated, find the target key value in the table set that matches the key value of the pixel to be interpolated, calculate the weighted average of the pixel values ​​of all pixels in the table corresponding to the target key value, and determine the average value as the pixel to be interpolated. ; add the pixel points to be interpolated that have been interpolated to the table, and update the pixel values ​​of the pixel points to be interpolated in the mask of the interpolation area; count whether all the interpolation calculations are completed, and then add 1 to the number of iterations. If the number of iterations does not meet the set maximum number of iterations or the interpolation calculation is not completed, return to the traversal step; if the number of iterations meets the set maximum number of iterations or the interpolation calculation is completed, check the mask of the interpolation area. If there is an uninterpolated area, the area can be determined as a noise point and filled with a traditional interpolation method; finally, output the completed interpolation image and check the difference result.

[0110] This method is based on the color space interpolation technology guided by the color image. During interpolation, the original color image, the image to be interpolated, and the interpolation area mask are input simultaneously. Through the fast guided interpolation in the high-dimensional space, a result closer to the real parallax can be obtained.

[0111] Embodiment five:

[0112] Corresponding to the above method embodiment, see Fig. 9 A structural schematic diagram of an image processing device is shown, the device comprising:

[0113] The acquisition module 91 is used to determine the pixel to be interpolated from the image to be interpolated; and to acquire the color reference information of the pixel to be interpolated from the color reference map corresponding to the image to be interpolated;

[0114] A grouping module 92 is used to determine the pixel group to which the pixel to be interpolated belongs according to the color reference information of the pixel to be interpolated; wherein the pixel points with determined pixel values ​​are pre-divided into a plurality of pixel groups based on the color reference information of the pixel points with determined pixel values ​​in the image to be interpolated;

[0115] The determination module 93 is used to determine the pixel value of the pixel to be interpolated based on the pixel values ​​of each pixel in the pixel group to which the pixel to be interpolated belongs.

[0116] The embodiment of the present invention provides an image processing device, which determines the pixel points to be interpolated from the image to be interpolated; obtains the color reference information of the pixel points to be interpolated from the color reference map corresponding to the image to be interpolated; determines the pixel point group to which the pixel points to be interpolated belong according to the color reference information of the pixel points to be interpolated; wherein, based on the color reference information of the pixel points with determined pixel values ​​in the image to be interpolated, the pixel points with determined pixel values ​​are pre-divided into a plurality of pixel point groups; and the pixel values ​​of the pixel points to be interpolated are determined based on the pixel values ​​of each pixel point in the pixel point group to which the pixel points to be interpolated belong. In this method, the pixel point group is determined using the color reference information of the image to be interpolated, and then the pixel values ​​of the pixel points to be interpolated are determined based on the pixel values ​​of the pixels in the group. Since the color reference information of the pixels in the same group is similar, this method can avoid using the pixels with large color differences on both sides of the edge for interpolation calculation, and improves the accuracy of the interpolated image for special images such as disparity maps and optical flow maps.

[0117] Furthermore, the above-mentioned image to be interpolated includes: a disparity map or an optical flow map generated based on a color reference map.

[0118] Furthermore, in the above-mentioned multiple pixel point groups, each pixel point group corresponds to a pixel feature; different pixel point groups correspond to different pixel features; the above-mentioned grouping module is also used to: determine the target pixel features of the pixel points to be interpolated based on the color reference information of the pixel points to be interpolated; group the pixel points corresponding to the target pixel features, and determine them as the pixel point group to which the pixel points to be interpolated belong.

[0119] Furthermore, the grouping module is further used to determine target pixel features of the pixels to be interpolated based on color reference information of the pixels to be interpolated and position information of the pixels to be interpolated in the image to be interpolated.

[0120] Furthermore, the grouping module is also used to: based on a preset conversion rule, convert the color reference information of the pixel to be interpolated and the position information of the pixel to be interpolated in the image to be interpolated into a one-dimensional feature value; wherein the color reference information of the pixel to be interpolated includes: multiple color channel values ​​of the pixel to be interpolated in the color reference image; the position information of the pixel to be interpolated in the image to be interpolated includes: X coordinate value and Y coordinate value; determine the target pixel feature of the pixel to be interpolated according to the one-dimensional feature value.

[0121] Furthermore, the above grouping module is also used for: one-dimensional eigenvalue key=x / sigma_s+y / sigma_s*255+r / sigma_c*255^2+g / sigma_c*255^3+b / sigma_c*255^4; wherein key is a one-dimensional eigenvalue; x is an X coordinate value; y is a Y coordinate value; r is an R channel value; g is a G channel value; b is a B channel value; sigma_s is a spatial smoothing parameter corresponding to the image to be interpolated, and sigma_c is a color smoothing parameter corresponding to the image to be interpolated.

[0122] Furthermore, the grouping module is also used to: determine the pixel features corresponding to each pixel point group, whose distance to the one-dimensional feature value is less than a preset distance threshold, as the target pixel features of the pixel point to be interpolated.

[0123] Furthermore, the above-mentioned device is also used to: obtain pixel points with determined pixel values ​​from the image to be interpolated; obtain color reference information of pixel points with determined pixel values ​​from the color reference map corresponding to the image to be interpolated; for each pixel point with determined pixel value, determine the pixel feature of the pixel point based on the color reference information of the pixel point; divide the pixel points with the same pixel features into the same pixel point group to obtain multiple pixel point groups.

[0124] Furthermore, the determination module is further used to calculate a weighted average value of pixel values ​​of each pixel in the pixel group to which the pixel to be interpolated belongs, and determine the weighted average value as the pixel value of the pixel to be interpolated.

[0125] Furthermore, the acquisition module is further used to: acquire a mask image of the image to be interpolated based on the image to be interpolated or the color reference image; and determine pixels to be interpolated from the image to be interpolated based on the mask image.

[0126] Furthermore, the above-mentioned device is also used for: if there are multiple pixel points to be interpolated, traversing each pixel point to be interpolated according to a preset traversal rule; for each pixel point to be interpolated, executing the step of obtaining the color reference information of the pixel point to be interpolated from the color reference map corresponding to the image to be interpolated until the pixel value of the pixel point to be interpolated is obtained; when each pixel point to be interpolated in the pixel point group to which the pixel point to be interpolated belongs obtains a pixel value, updating each pixel point to be interpolated in the pixel point group to which the pixel point to be interpolated belongs to a pixel point with a determined pixel value.

[0127] The image processing device provided by the embodiment of the present invention has the same technical features as the image processing method provided by the above embodiment, so it can also solve the same technical problems and achieve the same technical effects.

[0128] Embodiment six:

[0129] An embodiment of the present invention provides an electronic system, which includes: an image acquisition device, a processing device and a storage device; the image acquisition device is used to obtain preview video frames or image data; a computer program is stored on the storage device, and the computer program executes the steps of the above-mentioned image processing method when the processed device is running.

[0130] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the electronic system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0131] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processing device, the steps of the above-mentioned image processing method are executed.

[0132] The computer program products of the image processing methods, devices, and electronic systems provided in the embodiments of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.

[0133] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0134] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0135] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0136] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0137] Finally, it should be noted that the above embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions recorded in the above embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: The method comprises: Determine a pixel to be interpolated from the image to be interpolated; obtain color reference information of the pixel to be interpolated from a color reference map corresponding to the image to be interpolated; Determine the pixel group to which the pixel to be interpolated belongs according to the color reference information of the pixel to be interpolated; wherein, based on the color reference information of the pixel whose pixel value has been determined in the image to be interpolated, the pixel whose pixel value has been determined is pre-divided into a plurality of pixel groups; determine the pixel group to which the pixel to be interpolated belongs according to the color reference information of the pixel to be interpolated, including: dividing the pixel points with similar color reference information into the same pixel group according to the color reference information of the pixel whose pixel value has been determined in the image to be interpolated, and determining the pixel points included in the same pixel group as one pixel group; The pixel value of the pixel to be interpolated is determined based on the pixel value of each pixel in the pixel group to which the pixel to be interpolated belongs.

2. The method according to claim 1, characterized in that: The image to be interpolated includes: a disparity map or an optical flow map generated based on the color reference map.

3. The method according to claim 1, characterized in that In the plurality of pixel point groups, each pixel point group corresponds to a pixel feature; different pixel point groups correspond to different pixel features; The step of determining the pixel group to which the pixel to be interpolated belongs according to the color reference information of the pixel to be interpolated comprises: Determining a target pixel feature of the pixel to be interpolated based on the color reference information of the pixel to be interpolated; The pixel points corresponding to the target pixel feature are grouped and determined as the pixel point group to which the pixel point to be interpolated belongs.

4. The method according to claim 3, characterized in that: The step of determining the target pixel feature of the pixel to be interpolated based on the color reference information of the pixel to be interpolated comprises: Based on the color reference information of the pixel to be interpolated and the position information of the pixel to be interpolated in the image to be interpolated, the target pixel feature of the pixel to be interpolated is determined.

5. The method according to claim 4, characterized in that The step of determining the target pixel feature of the pixel to be interpolated based on the color reference information of the pixel to be interpolated and the position information of the pixel to be interpolated in the image to be interpolated includes: Based on a preset conversion rule, the color reference information of the pixel to be interpolated and the position information of the pixel to be interpolated in the image to be interpolated are converted into a one-dimensional feature value; wherein the color reference information of the pixel to be interpolated includes: a plurality of color channel values ​​of the pixel to be interpolated in the color reference image; the position information of the pixel to be interpolated in the image to be interpolated includes: an X coordinate value and a Y coordinate value; The target pixel feature of the pixel to be interpolated is determined according to the one-dimensional feature value.

6. The method according to claim 5, characterized in that The step of converting the color reference information of the pixel to be interpolated and the position information of the pixel to be interpolated in the image to be interpolated into a one-dimensional feature value based on a preset conversion rule includes: The one-dimensional eigenvalue key = x / sigma_s + y / sigma_s*255 + r / sigma_c *255^2 + g / sigma_c*255^3+ b / sigma_c*255^4; Among them, key is the one-dimensional eigenvalue; x is the X coordinate value; y is the Y coordinate value; r is the R channel value; g is the G channel value; b is the B channel value; sigma_s is the spatial smoothing parameter corresponding to the image to be interpolated, and sigma_c is the color smoothing parameter corresponding to the image to be interpolated.

7. The method according to claim 5, characterized in that The step of determining the target pixel feature of the pixel to be interpolated according to the one-dimensional feature value comprises: Among the pixel features corresponding to each pixel point group, the pixel features whose distance from the one-dimensional feature value is less than a preset distance threshold are determined as the target pixel features of the pixel point to be interpolated.

8. The method according to any one of claims 1 to 7, characterized in that: Before the step of determining the pixel points to be interpolated from the image to be interpolated, the method further includes: Acquire pixel points with determined pixel values ​​from the image to be interpolated; Acquire color reference information of the pixel point whose pixel value has been determined from a color reference map corresponding to the image to be interpolated; For each pixel point whose pixel value has been determined, determining a pixel feature of the pixel point based on the color reference information of the pixel point; Pixels with the same pixel features are divided into the same pixel group to obtain multiple pixel groups.

9. The method according to any one of claims 1 to 7, characterized in that: The step of determining the pixel value of the pixel to be interpolated based on the pixel value of each pixel in the pixel group to which the pixel to be interpolated belongs comprises: A weighted average of the pixel values ​​of each pixel in the pixel group to which the pixel to be interpolated belongs is calculated, and the weighted average is determined as the pixel value of the pixel to be interpolated.

10. The method according to claim 1, characterized in that The step of determining the pixel points to be interpolated from the image to be interpolated comprises: Based on the image to be interpolated or the color reference image, a mask image of the image to be interpolated is acquired; based on the mask image, pixels to be interpolated are determined from the image to be interpolated.

11. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: If the pixel points to be interpolated include multiple ones, traverse each pixel point to be interpolated according to a preset traversal rule; For each pixel to be interpolated, performing the step of acquiring the color reference information of the pixel to be interpolated from the color reference map corresponding to the image to be interpolated, until a pixel value of the pixel to be interpolated is obtained; When each pixel to be interpolated in the pixel point group to which the pixel to be interpolated belongs obtains a pixel value, each pixel to be interpolated in the pixel point group is updated to a pixel point with a determined pixel value.

12. An image processing device, characterized in that: The device comprises: An acquisition module is used to determine the pixel to be interpolated from the image to be interpolated; and to acquire the color reference information of the pixel to be interpolated from the color reference map corresponding to the image to be interpolated; A grouping module, configured to determine the pixel group to which the pixel to be interpolated belongs according to the color reference information of the pixel to be interpolated; wherein, based on the color reference information of the pixel whose pixel value has been determined in the image to be interpolated, the pixel whose pixel value has been determined is pre-divided into a plurality of pixel groups; determining the pixel group to which the pixel to be interpolated belongs according to the color reference information of the pixel to be interpolated, comprising: dividing the pixel points with similar color reference information into the same pixel group according to the color reference information of the pixel whose pixel value has been determined in the image to be interpolated, and determining the pixel points included in the same pixel group as one pixel group; The determination module is used to determine the pixel value of the pixel point to be interpolated based on the pixel value of each pixel point in the pixel point group to which the pixel point to be interpolated belongs.

13. An electronic system, characterized in that: The electronic system includes: a processing device and a storage device; The storage device stores a computer program, which, when executed by the processing device, executes the image processing method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processing device, the steps of the image processing method according to any one of claims 1 to 11 are executed.

Images