Image processing method and device, electronic equipment and storage medium

Abstract

This invention provides an image processing method, apparatus, electronic device, and storage medium. The method includes: an image processing operation, comprising: acquiring a sequence of images to be processed, the sequence including a reference image and non-reference images; performing a search operation for each target image block in the non-reference image, including: searching for a reference image block in the reference image that matches the target image block; and for each target image block in the non-reference image, when a reference image block is found, determining the algorithm processing result corresponding to the reference image block as the algorithm processing result corresponding to the target image block, wherein the algorithm processing result corresponding to any image block includes the result obtained after processing the image block using the target image processing algorithm. This approach can effectively save time spent repeatedly processing the same image content in multiple images using image processing algorithms.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to an image processing method and apparatus, an electronic device, a storage medium, and computer program instructions. Background Technology

[0002] When an image acquisition device continuously acquires images of a target object, the resulting image sequence can constitute a video stream. Currently, processing the video stream using relevant algorithms requires processing each frame of the video stream. For example, when using an endoscope system to continuously acquire images of a target object, a preset endoscopic algorithm is needed to process each acquired image in the YUV or RGB space. This algorithm processing is time-consuming, and the display latency of the video stream is relatively large. Summary of the Invention

[0003] The present invention addresses the aforementioned problems. It provides an image processing method, an image processing apparatus, an electronic device, a storage medium, and a computer program product. This solution effectively saves processing time associated with repeatedly processing the same image content in multiple images using image processing algorithms, and effectively avoids redundant data generated by repeated processing, thereby saving memory resources.

[0004] According to one aspect of the present invention, an image processing method is provided. The method includes: image processing operations, which include: acquiring a sequence of images to be processed, the sequence of images to be processed including a reference image and non-reference images; determining the algorithm processing result corresponding to the non-reference image, wherein the algorithm processing result corresponding to any image includes the result obtained after processing the image using a target image processing algorithm; wherein determining the algorithm processing result corresponding to the non-reference image includes: performing a search operation for each target image block in the non-reference image, the search operation including: searching for a reference image block that matches the target image block in the reference image; for each target image block in the non-reference image, when a reference image block is found, determining the algorithm processing result corresponding to the reference image block as the algorithm processing result corresponding to the target image block, wherein the algorithm processing result corresponding to any image block includes the result obtained after processing the image block using a target image processing algorithm; when no reference image block is found, processing the target image block using a target image processing algorithm to obtain the algorithm processing result corresponding to the target image block; wherein the algorithm processing result corresponding to the non-reference image includes: the algorithm processing results corresponding to the target image block in the non-reference image and the remaining image blocks other than the target image block.

[0005] Optionally, the images in the image sequence to be processed are images in the Y channel or images in the RGB channel.

[0006] Optionally, the image processing operation includes a first image processing operation, in which the images in the image sequence to be processed are images in the Y channel. In the first image processing operation, obtaining the image sequence to be processed includes: obtaining an initial image sequence, which includes a reference initial image and a non-reference initial image; converting the images in the initial image sequence to the YUV color space respectively to obtain a Y channel image sequence, wherein the Y channel image in the Y channel image sequence corresponding to the reference initial image is the reference image, and the Y channel image in the Y channel image sequence corresponding to the non-reference initial image is the non-reference image.

[0007] Optionally, the reference image containing the algorithm processing result in the first image processing operation is the reference result image, and the non-reference image containing the algorithm processing result in the first image processing operation is the non-reference result image; wherein, the image processing operation further includes a second image processing operation, in which, when performing the step of converting the images in the initial image sequence to the YUV color space respectively, a U-channel image sequence and a V-channel image sequence are also obtained, wherein, in the second image processing operation, obtaining the image sequence to be processed includes: converting the images in the YUV image sequence to the RGB color space respectively to obtain an RGB image sequence, wherein the RGB image in the RGB image sequence corresponding to the reference initial image is the non-reference result image in the second image processing operation. The reference image is the RGB image in the RGB image sequence that corresponds to the non-reference initial image. The non-reference image in the second image processing operation is the RGB image in the RGB image sequence. The YUV image sequence includes the reference YUV image corresponding to the reference initial image and the non-reference YUV image corresponding to the non-reference initial image. The reference YUV image includes the reference result image, the U channel image corresponding to the reference initial image in the U channel image sequence, and the V channel image corresponding to the reference initial image in the V channel image sequence. The non-reference YUV image includes the non-reference result image, the U channel image corresponding to the non-reference initial image in the U channel image sequence, and the V channel image corresponding to the non-reference initial image in the V channel image sequence.

[0008] Optionally, the image processing operation includes a first image processing operation and a second image processing operation. In the first image processing operation, the images in the image sequence to be processed are Y-channel images. In the second image processing operation, the images in the image sequence to be processed are RGB images. The images in the image sequence to be processed in the second image processing operation are obtained one-to-one based on the images in the image sequence to be processed in the first image processing operation. The target image blocks included in the non-reference images in the second image processing operation correspond one-to-one with the target image blocks included in the non-reference images in the first image processing operation. For any target image block in the non-reference images in the first image processing operation, if a reference image block matching the target image block is found, the image block corresponding to the target image block in the second image processing operation is taken as the target image block in the second image processing operation.

[0009] Optionally, a search operation is performed for each target image block in the non-reference image, including: obtaining at least two initial image blocks of equal scale based on the non-reference image, wherein each initial image block is used as an initial target image block; for each target image block for which no reference image block is found, a search operation is performed based on the target image block; for any target image block, if no reference image block matching the target image block is found in the search operation, and a preset condition is met, the target image block is divided into at least two new image blocks of equal scale, and each of the divided new image blocks is used as a new target image block, and the process returns to the step of performing a search operation based on each target image block for which no reference image block is found; wherein the preset condition includes: the scale of the new image blocks obtained from the division based on the target image block is greater than or equal to a preset scale.

[0010] Optionally, obtaining at least two initial image patches of equal scale based on the non-reference image includes: if the non-reference image cannot be evenly divided according to the scale of the initial image patches, then expanding outward along at least a portion of the edges of the non-reference image to obtain a new non-reference image, wherein the pixel values ​​of the expanded region in the new non-reference image are determined by mirror mapping the pixel values ​​of the regions adjacent to the expanded region in the non-reference image; and dividing the new non-reference image evenly according to the scale of the initial image patches to obtain at least two initial image patches of equal scale.

[0011] Optionally, finding a reference image block that matches the target image block from the reference image includes: determining at least one image block to be matched in the reference image based on the position of the target image block in the corresponding non-reference image; calculating the similarity between the image block to be matched and the target image block for each of the at least one image block to be matched; comparing the minimum similarity among the similarities corresponding to each of the at least one image block to be matched with a preset similarity threshold; when the minimum similarity is less than or equal to the preset similarity threshold, determining the image block to be matched corresponding to the minimum similarity as the reference image block corresponding to the target image block; wherein, at least one image block to be matched is located within a preset range around a specific image block, the position of the specific image block in the reference image is consistent with the position of the target image block in the corresponding non-reference image, and the scale of each image block to be matched is the same as the scale of the corresponding target image block.

[0012] Optionally, calculating the similarity between the image patch to be matched and the target image patch includes: subtracting the pixel value of each pixel in the image patch to be matched from the pixel value of the corresponding pixel in the target image patch to determine the pixel difference corresponding to each pixel in the image patch to be matched; and averaging the pixel differences corresponding to each pixel in the image patch to be matched to determine the image deviation corresponding to the image patch to be matched, wherein the similarity is represented by the image deviation.

[0013] Optionally, the center point of the at least one image patch to be matched includes one or more of the four vertices of the specific image patch, the midpoints of the four edges of the specific image patch, and the center point of the specific image patch.

[0014] Optionally, based on the position of the target image patch in the corresponding non-reference image, at least one image patch to be matched is determined in the reference image, including: for any target point determined based on the position of the target image patch in the corresponding non-reference image, if all image patches centered at the target point and with a scale equal to the scale of the corresponding target image patch are located within the reference image, then the image patch centered at the target point and with a scale equal to the scale of the corresponding target image patch is determined as the image patch to be matched; wherein, the target point includes the four vertices of the specific image patch, the midpoints of the four edges of the specific image patch, and the center point of the specific image patch.

[0015] Optionally, when a reference image block is found, the algorithm processing result corresponding to the reference image block is determined as the algorithm processing result corresponding to the target image block, including: when a reference image block is found, determining and caching the motion vector between the reference image block and the target image block; when processing image blocks in non-reference images using the target image processing algorithm, assigning the algorithm processing result corresponding to the reference image block as the algorithm processing result corresponding to the target image block according to the motion vector.

[0016] Optionally, determining the algorithm processing result corresponding to the non-reference image further includes: for each remaining image block in the non-reference image other than the target image block, processing the remaining image block using the target image processing algorithm to obtain the algorithm processing result corresponding to the remaining image block.

[0017] According to another aspect of the present invention, an image processing apparatus is also provided, the apparatus comprising: an image processing module, the image processing module comprising: an acquisition submodule, configured to acquire a sequence of images to be processed, the sequence of images to be processed including a reference image and non-reference images; and a determination submodule, configured to determine an algorithm processing result corresponding to a non-reference image, wherein the algorithm processing result corresponding to any image includes the result obtained after processing the image using a target image processing algorithm; wherein the determination submodule comprises: an execution unit, configured to perform a search operation for each target image block in the non-reference image, the search operation including: searching for a reference image block matching the target image block from the reference image; and a first determination unit, configured to determine the image block in the non-reference image... For each target image block, when a reference image block is found, the algorithm processing result corresponding to the reference image block is determined as the algorithm processing result corresponding to the target image block. The algorithm processing result corresponding to any image block includes the result obtained after processing the image block using the target image processing algorithm. The second determining unit is used to process each target image block in the non-reference image using the target image processing algorithm when no reference image block is found, to obtain the algorithm processing result corresponding to the target image block. The algorithm processing result corresponding to the non-reference image includes the algorithm processing results corresponding to the target image block in the non-reference image and the remaining image blocks other than the target image block.

[0018] According to another aspect of the present invention, an electronic device is also provided, including a processor and a memory, wherein a computer program is stored in the memory, and the computer program instructions are executed by the processor to perform the above-described image processing method.

[0019] According to another aspect of the present invention, a storage medium is also provided, which stores a computer program / instructions, which, when executed, are used to perform the above-described image processing method.

[0020] The above technical solution finds a reference image block in the reference image that matches each target image block of each non-reference image. When a matching reference image block is found, the algorithm processing result corresponding to the matching reference image block can be directly determined as the algorithm processing result corresponding to the target image block. This can effectively save the processing time of repeatedly processing target image blocks containing the same image content as the reference image block using the target image processing algorithm, and can effectively avoid redundant data caused by repeated processing, thereby saving memory resources.

[0021] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0022] The above and other objects, features, and advantages of the present invention will become more apparent from the more detailed description of the embodiments of the invention in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same parts or steps.

[0023] Figure 1 A schematic flowchart of an image processing method according to an embodiment of the present invention is shown;

[0024] Figure 2 A schematic diagram is shown of at least one image block to be matched within a preset range around a specific image block according to an embodiment of the present invention;

[0025] Figure 3 A schematic diagram is shown of at least one image block to be matched within a preset range around a specific image block according to another embodiment of the present invention;

[0026] Figure 4 A schematic block diagram of an image processing module of an image processing apparatus according to an embodiment of the present invention is shown; and

[0027] Figure 5 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the present invention more apparent, exemplary embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely a part of the embodiments of the present invention, and not all of the embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments described in the present invention, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of the present invention.

[0029] To at least partially solve the above-mentioned technical problems, the present invention provides an image processing method. Figure 1 A schematic flowchart of an image processing method 100 according to an embodiment of the present invention is shown, such as Figure 1 As shown, the method 100 may include image processing operations, which include steps S110 and S120.

[0030] In step S110, a sequence of images to be processed is obtained, which includes a reference image and non-reference images.

[0031] For example, the image sequence to be processed can be an image sequence acquired using any type of image acquisition device (e.g., camera, industrial camera, ultrasound system, endoscope system, etc.). Taking an endoscope system as an example, the image sequence to be processed can be an image sequence acquired using an endoscope system targeting a biological organism (e.g., human body, animal body) and its target tissue (e.g., stomach, gallbladder, pancreas, etc.). The image sequence to be processed can include multiple images, which can be continuously acquired initial images, or images obtained by image preprocessing operations (e.g., filtering, color space conversion, etc.) on continuously acquired initial images. In this case, the image sequence to be processed can be a real-time online video stream or a stored offline video stream. The multiple images in the image sequence to be processed can also be non-continuously acquired initial images, or images obtained by image preprocessing operations on non-continuously acquired initial images. In this case, the image sequence to be processed can be an image sequence obtained by frame extraction processing of a stored offline video stream acquired for the target object, or an image sequence obtained by arranging multiple individually acquired images of the target object according to a preset rule. When the image sequence to be processed is an online video stream, the first image acquired in the sequence (i.e., the first frame) can be used as the reference image, and images acquired after the first frame can be used as non-reference images. When the image sequence to be processed is an offline video stream, any image among multiple images in the sequence can be used as the reference image, and other images besides the reference image can be used as non-reference images.

[0032] In step S120, the algorithm processing result corresponding to the non-reference image is determined, wherein the algorithm processing result corresponding to any image includes the result obtained after processing the image using the target image processing algorithm.

[0033] For example, a reference image may include reference image blocks. A reference image block may be all image blocks in the reference image divided according to a preset rule, or it may be a portion of the all image blocks in the reference image divided according to a preset rule. For each reference image block of the reference image, a target image processing algorithm can be applied to process the reference image block to obtain the algorithm processing result corresponding to that reference image block. The target image processing algorithm can be any type of image processing algorithm, such as: color enhancement algorithm, image contrast enhancement algorithm, cropping algorithm, inverse gamma correction (Degamma) algorithm, scaling (Zoom) algorithm, etc. For example, a non-reference image may include target image blocks. A target image block may be all image blocks in the non-reference image divided according to a preset rule, or it may be a portion of the all image blocks in the non-reference image divided according to a preset rule. A non-reference image may also include remaining image blocks other than the target image blocks. For each remaining image block of each non-reference image other than the target image block, a target image processing algorithm can be applied to process the image block to obtain the algorithm processing result corresponding to that image block. It can be understood that when the target image block is all the image blocks in the non-reference image divided according to the preset rules, the number of remaining image blocks in the non-reference image other than the target image block is 0. When the target image block is part of the image blocks in the non-reference image divided according to the preset rules, the number of remaining image blocks in the non-reference image other than the target image block is greater than or equal to 1.

[0034] In step S120, determining the algorithm processing result corresponding to the non-reference image includes: performing a search operation for each target image block in the non-reference image, the search operation including: searching for a reference image block that matches the target image block in the reference image; for each target image block in the non-reference image, when a reference image block is found, determining the algorithm processing result corresponding to the reference image block as the algorithm processing result corresponding to the target image block, wherein the algorithm processing result corresponding to any image block includes the result obtained after processing the image block using the target image processing algorithm; when no reference image block is found, processing the target image block using the target image processing algorithm to obtain the algorithm processing result corresponding to the target image block; wherein the algorithm processing result corresponding to the non-reference image includes: the algorithm processing results corresponding to the target image block in the non-reference image and the remaining image blocks other than the target image block.

[0035] For example, for each target image patch in each non-reference image, a search operation can be performed on the target image patch. Specifically, the search operation can include: finding a reference image patch in the reference image that matches the target image patch. "Matching" can mean at least satisfying the following conditions: consistent scale and similarity meeting requirements. Alternatively, "matching" can mean consistent scale and positional deviation and similarity meeting requirements. When a reference image patch matching the target image patch is found, the algorithm processing result corresponding to the found reference image patch can be determined as the algorithm processing result corresponding to the target image patch. When no reference image patch matching the target image patch is found, the target image patch can be processed using a target image processing algorithm to obtain the algorithm processing result corresponding to the target image patch. It can be understood that for each non-reference image, the algorithm processing result corresponding to the non-reference image can include the algorithm processing result corresponding to the target image patch in the non-reference image and the algorithm processing results corresponding to the remaining image patches in the non-reference image excluding the target image patch.

[0036] In many cases, the content of adjacent frames in the image sequence to be processed does not change much. For example, when the image acquisition scene is relatively simple or the target object changes slowly or little during the image acquisition period, the similarity between adjacent frames of the acquired video stream is high. Specifically, when using medical imaging devices (such as ultrasound systems, endoscope systems, etc.) to image target tissues (such as the stomach, pancreas, gallbladder, intestines, etc.), the similarity between frames of the obtained image sequence is high because the scene in which the target tissue is located is relatively simple. In this case, when the target image processing algorithm processes each image in the image sequence to be processed, it will perform many repetitive calculations and generate a large amount of redundant data.

[0037] The above technical solution finds a reference image block in the reference image that matches each target image block of each non-reference image. When a matching reference image block is found, the algorithm processing result corresponding to the matching reference image block can be directly determined as the algorithm processing result corresponding to the target image block. This can effectively save the processing time of repeatedly processing target image blocks containing image content that is the same as or similar to the reference image block using the target image processing algorithm, and can effectively avoid redundant data caused by repeated processing, thereby saving memory resources.

[0038] Optionally, the images in the image sequence to be processed are images in the Y channel or images in the RGB channel.

[0039] For example, when acquiring images of a target object, the initial images in the acquired initial image sequence can be RGB images / YUV images / YCbCr images. When the initial images in the initial image sequence are RGB images, each initial image can be converted from the RGB color space to the YUV color space to obtain the image in the Y channel. The images in the image sequence to be processed can be the Y channel images obtained from the initial images after color space conversion; alternatively, the images in the image sequence to be processed can also be the initial images contained in the initial image sequence, i.e., the images in the RGB channels. When the initial images in the initial image sequence are YUV images / YCbCr images, the images in the image sequence to be processed can be the images in the Y channel; alternatively, each initial image can also be converted from the YUV / YCbCr color space to the RGB color space to obtain the image in the RGB channel. The images in the image sequence to be processed can also be the RGB channel images obtained from the initial images after color space conversion. By processing the images in the Y channel or the images in the RGB channel, it is easier to determine the algorithm processing result corresponding to each image, because different processing algorithms often need to be implemented in different color spaces.

[0040] Optionally, the image processing operation includes a first image processing operation, in which the images in the image sequence to be processed are images in the Y channel. In the first image processing operation, obtaining the image sequence to be processed includes: obtaining an initial image sequence, which includes a reference initial image and a non-reference initial image; converting the images in the initial image sequence to the YUV color space respectively to obtain a Y channel image sequence, wherein the Y channel image in the Y channel image sequence corresponding to the reference initial image is the reference image, and the Y channel image in the Y channel image sequence corresponding to the non-reference initial image is the non-reference image.

[0041] For example, the image processing operation may include a first image processing operation. In the first image processing operation, the images in the acquired image sequence to be processed may be images in the Y channel. Specifically, the images in the initial image sequence acquired by the image acquisition device (denoted as initial images) may be images in a color space other than the YUV color space. For example, each initial image may be an image in the RGB color space (i.e., an RGB image). Each initial image in the initial image sequence may include a reference initial image and non-reference initial images. Any image in each initial image may be used as a reference initial image; for example, the first frame initial image may be used as the reference initial image. Alternatively, the reference initial image in the initial image sequence may be determined according to a preset interval. Specifically, for example, at least one reference initial image in the initial image sequence may be determined at an interval of 30 frames. In other words, the 30i+1th frame initial image in the initial image sequence may be used as the reference initial image, where i≥0, and N-30≤30i+1≤N, where N is the total number of initial images in the initial sequence. For example, when the initial image sequence is a video stream, a reference initial image can be determined according to a preset time period. For instance, at least one reference initial image can be determined at 0.5-second intervals. That is, starting from the first initial frame, a reference initial image is determined every 0.5 seconds, and the first initial frame is also used as the reference initial image. It can be understood that other initial images in the initial image sequence besides the reference initial image are non-reference initial images. For each non-reference initial image, the corresponding reference initial image can be the one that precedes the non-reference initial image in the initial image sequence and has the fewest intervals with it. Converting each initial image to the YUV color space yields the corresponding Y-channel image (denoted as the Y-channel image). The Y-channel images of each initial image can constitute a Y-channel image sequence, which can be determined as the image sequence to be processed. In a Y-channel image sequence, the Y-channel image obtained by converting the reference initial image to the YUV color space can be used as the reference image, and the Y-channel image obtained by converting the non-reference initial image to the YUV color space can be used as the non-reference image. In other words, all other Y-channel images in the Y-channel image sequence except for the Y-channel image used as the reference image can be non-reference images.

[0042] In the above technical solution, by converting the color space of each image in the initial image sequence to the YUV color space, a Y-channel image sequence that can be used as the image sequence to be processed is obtained, thereby enabling image algorithm optimization in the Y space. On the other hand, based on the reference initial image and non-reference initial image in the initial image sequence, the reference and non-reference images corresponding to the Y-channel image sequence can be determined. This allows for the rapid acquisition of the Y-channel image corresponding one-to-one with each image in the initial image sequence, and the rapid identification of the reference and non-reference images in the image sequence to be processed. Furthermore, for images acquired by certain image acquisition devices, the algorithm processing results obtained by processing the images in the YUV color space using target image processing algorithms are better. The target image processing algorithms configured for these image acquisition devices can be image processing algorithms in the YUV space. For example, for images acquired using an endoscope system, the endoscope system is usually configured with a target image processing algorithm for the Y channel. Therefore, by converting the color space of each initial image acquired by the endoscope system to the YUV color space, image processing in the YUV color space can be achieved.

[0043] Optionally, the reference image containing the algorithm processing result in the first image processing operation is the reference result image, and the non-reference image containing the algorithm processing result in the first image processing operation is the non-reference result image; wherein, the image processing operation further includes a second image processing operation, in which, when performing the step of converting the images in the initial image sequence to the YUV color space respectively, a U-channel image sequence and a V-channel image sequence are also obtained, wherein, in the second image processing operation, obtaining the image sequence to be processed includes: converting the images in the YUV image sequence to the RGB color space respectively to obtain an RGB image sequence, wherein the RGB image in the RGB image sequence corresponding to the reference initial image is the non-reference result image in the second image processing operation. The reference image is the RGB image in the RGB image sequence that corresponds to the non-reference initial image. The non-reference image in the second image processing operation is the RGB image in the RGB image sequence. The YUV image sequence includes the reference YUV image corresponding to the reference initial image and the non-reference YUV image corresponding to the non-reference initial image. The reference YUV image includes the reference result image, the U channel image corresponding to the reference initial image in the U channel image sequence, and the V channel image corresponding to the reference initial image in the V channel image sequence. The non-reference YUV image includes the non-reference result image, the U channel image corresponding to the non-reference initial image in the U channel image sequence, and the V channel image corresponding to the non-reference initial image in the V channel image sequence.

[0044] For example, for the Y-channel image sequence obtained in the first image processing operation, the algorithm processing result corresponding to the reference image can be denoted as the reference result image, and the algorithm processing result corresponding to the non-reference image can be denoted as the non-reference result image. When converting each initial image in the initial image sequence to the YUV color space, the image in the U channel (denoted as the U-channel image) and the image in the V channel (denoted as the V-channel image) corresponding to each initial image can also be obtained. The U-channel images corresponding to each initial image can constitute a U-channel image sequence, and similarly, the V-channel images corresponding to each initial image can constitute a V-channel image sequence. It can be understood that each image in the Y-channel image sequence corresponds one-to-one with each image in the U-channel image sequence, and each image in the Y-channel image sequence corresponds one-to-one with each image in the V-channel image sequence. For example, for a reference result image or any non-reference result image in a Y-channel image sequence, the image, the corresponding V-channel image in a V-channel image sequence, and the corresponding U-channel image in a U-channel image sequence can be combined into a single YUV image. The resulting YUV images can constitute a YUV image sequence. In the YUV image sequence, the YUV image corresponding to the reference initial image can be denoted as the reference YUV image, and the YUV image corresponding to the non-reference initial image can be denoted as the non-reference YUV image. In other words, the reference YUV image can include the reference result image, the U-channel image corresponding to the reference initial image in the U-channel image sequence, and the V-channel image corresponding to the reference initial image in the V-channel image sequence; the non-reference YUV image can include the non-reference result image, the U-channel image corresponding to the non-reference initial image in the U-channel image sequence, and the V-channel image corresponding to the non-reference initial image in the V-channel image sequence. For example, the image processing operation can also include a second image processing operation. In the second image processing operation, the images in the acquired image sequence to be processed can be RGB images. Specifically, after obtaining the YUV image sequence, each YUV image in the sequence can be converted to the RGB color space to obtain an RGB image corresponding to each YUV image. The resulting RGB images can constitute an RGB image sequence. In the RGB image sequence, the RGB image corresponding to the reference initial image (i.e., the reference YUV image) can be used as the reference image in the second image processing operation, and the non-reference image corresponding to the non-reference initial image (i.e., the non-reference YUV image) can be used as the non-reference image.

[0045] The above technical solution can convert an image in YUV space to RGB color space after processing it with an algorithm targeting the Y channel, and then continue to use it as an image sequence to be processed. This can preserve the algorithm processing results of the RGB image sequence before color space conversion when processing the RGB image sequence, which is beneficial to the image optimization effect.

[0046] Optionally, the image processing operation includes a first image processing operation and a second image processing operation. In the first image processing operation, the images in the image sequence to be processed are Y-channel images. In the second image processing operation, the images in the image sequence to be processed are RGB images. The images in the image sequence to be processed in the second image processing operation are obtained one-to-one based on the images in the image sequence to be processed in the first image processing operation. The target image blocks included in the non-reference images in the second image processing operation correspond one-to-one with the target image blocks included in the non-reference images in the first image processing operation. For any target image block in the non-reference images in the first image processing operation, if a reference image block matching the target image block is found, the image block corresponding to the target image block in the second image processing operation is taken as the target image block in the second image processing operation.

[0047] Exemplary examples show that the specific process of obtaining the image sequence to be processed in the first image processing operation and the specific process of obtaining the image sequence to be processed in the second image processing operation can both refer to the foregoing embodiments, and will not be repeated here. Each image in the image sequence to be processed in the first image processing operation corresponds one-to-one with each image in the image sequence to be processed in the second image processing operation. For each non-reference image in the image sequence to be processed in the second image processing operation, the target image block contained in the non-reference image corresponds one-to-one with the target image block contained in the corresponding non-reference image in the image sequence to be processed in the first image processing operation. For any target image block in each non-reference image in the image sequence to be processed in the first image processing operation, the non-reference image is denoted as image A, and the target image block is denoted as image block a. If a reference image block matching image block a is found in the reference image of the image sequence to be processed in the first image processing operation, then the image block corresponding to image block a contained in the non-reference image corresponding to image A in the image sequence to be processed in the second image processing operation can be used as the target image block in the second image processing operation. For example, for any target image block in the non-reference image during the first image processing operation, if no reference image block matching the target image block is found, the target image processing algorithm can be directly applied to process the image block corresponding to the target image block in the second image processing operation to obtain the algorithm processing result of the image block corresponding to the target image block in the second image processing operation, without having to perform a search operation on the image block corresponding to the target image block in the second image processing operation. It is understood that the target image processing algorithm may include image processing algorithms for the YUV color space, and may also include image processing algorithms for the RGB color space. That is, the first image processing operation may use an image processing algorithm for the YUV color space, and the second image processing operation may use an image processing algorithm for the RGB color space. In the aforementioned embodiment, "the image block corresponding to the target image block in the second image processing operation" refers to the image block obtained after converting the algorithm processing result corresponding to the target image block in the first image processing operation to the RGB color space.

[0048] In the above technical solution, by using the target image block that can be matched by the reference image block in the first image processing operation as the target image block in the second image processing operation, the time for repeatedly determining the target image block in the second image processing operation can be effectively saved. In particular, the target image block that fails to match the reference image block in the first image processing operation is likely to also fail to find a matching reference image block in the RGB space. By reducing the search operation time, the processing efficiency of the algorithm can be effectively improved.

[0049] Optionally, a search operation is performed for each target image block in the non-reference image, including: obtaining at least two initial image blocks of equal scale based on the non-reference image, wherein each initial image block is used as an initial target image block; for each target image block for which no reference image block is found, a search operation is performed based on the target image block; for any target image block, if no reference image block matching the target image block is found in the search operation, and a preset condition is met, the target image block is divided into at least two new image blocks of equal scale, and each of the divided new image blocks is used as a new target image block, and the process returns to the step of performing a search operation based on each target image block for which no reference image block is found; wherein the preset condition includes: the scale of the new image blocks obtained from the division based on the target image block is greater than or equal to a preset scale.

[0050] For example, for each non-reference image, at least two initial image patches can be obtained from the non-reference image, and the scale of each initial image patch is equal. The entire area covered by each initial image patch can be part or all of the non-reference image. Each initial image patch can serve as an initial target image patch. After determining the initial target image patch, a search operation can be performed on each initial target image patch. If no reference image patch matching the initial target image patch is found, a partitioning operation can be performed. The specific process of the partitioning operation can include: partitioning the initial target image patch to obtain at least two new image patches, each new image patch having the same scale, and the scale of the new image patch being greater than or equal to a preset scale. Similarly, the entire area covered by each new image patch can be the entire area or part of the initial target image patch. Each new image patch can serve as a new target image patch. After determining the new target image patch, a search operation can be performed based on the new target image patch. The aforementioned partitioning and search operations can be repeated. When a matching reference image block is found, or when the scale of the current target image block is smaller than a preset scale, the partitioning and search operations for the current target image block are stopped. For example, taking a non-reference image with a scale of 64*32 as an example, the non-reference image can be divided into two initial image blocks of 32*32, which can serve as initial target image blocks. Then, a search operation can be performed on each initial target image block. If no matching reference image block is found, the initial target image block can be partitioned at a scale of 16*16 to obtain four new image blocks, which can serve as new target image blocks. In this embodiment, the preset scale can be 8*8. A search operation is performed on each of the four new target image blocks. If no matching reference image block is found, the initial target image block can be further partitioned at a scale of 8*8 to obtain four new image blocks, which can serve as new target image blocks. For each of the four new target image blocks, a search operation is performed. If no matching reference image block is found, the partitioning operation is no longer performed, and the target image block can be processed using a target image processing algorithm.

[0051] In the above technical solution, when performing a search operation on the current target image block, if no matching reference image block is found, the current target image block can be divided equally to obtain a new image block. When the scale of the new image block meets the conditions, the new image block can be used as the current target image block to continue the search operation. This method can perform multiple search operations and reduce the scale of the target image block after each search operation if no reference image block is found, instead of dividing the non-reference image into multiple target image blocks according to the preset scale and performing the search operation one by one. The number of target image blocks for the entire search operation is small.

[0052] Optionally, obtaining at least two initial image patches of equal scale based on the non-reference image includes: if the non-reference image cannot be evenly divided according to the scale of the initial image patches, then expanding outward along at least a portion of the edges of the non-reference image to obtain a new non-reference image, wherein the pixel values ​​of the expanded region in the new non-reference image are determined by mirror mapping the pixel values ​​of the regions adjacent to the expanded region in the non-reference image; and dividing the new non-reference image evenly according to the scale of the initial image patches to obtain at least two initial image patches of equal scale.

[0053] For example, for any non-reference image, if the non-reference image cannot be evenly divided according to the scale of the initial image patch—for example, the ratio of the scale of the non-reference image in the first direction (e.g., horizontal direction) to the scale of the initial image patch in the first direction is not an integer, and / or the ratio of the scale of the non-reference image in the second direction (e.g., vertical direction) to the scale of the initial image patch in the second direction is not an integer, and the first and second directions are perpendicular to each other—in this case, the extended region of the non-reference image in the first and / or second directions can be determined. Specifically, taking the first direction as an example, if the scale of the non-reference image in the first direction is M, the scale of the initial image patch in the first direction is N, and M / N is not an integer, then the reserved scale of the non-reference image in the first direction is MP*N, where M / N-1 < P < M / N, and P is an integer. The first reserved scale of the non-reference image at the first end along the first direction can be denoted as a, and the second reserved scale at the second end can be denoted as b, then a+b=MP*N, where a>0 and b≥0, or a≥0 and b>0. It can be understood that the sum of the scale of the non-reference image extending outward from the first end along the first direction and the scale of the non-reference image extending outward from the second end along the first direction can be N-(a+b). The scale of the non-reference image extending outward from the second direction is similar to that of the first direction, and will not be elaborated here. After determining the scales of the non-reference image extending outward from the first and second directions, the extended region obtained by extending outward from each edge of the non-reference image can be determined accordingly. The pixel values ​​of the extended region can be determined by mirroring the pixel values ​​of the regions adjacent to the extended region in the non-reference image. Specifically, taking the example that the first and second reserved scales of the non-reference image are both not equal to 0, and the third and fourth reserved scales of the non-reference image at the first end and the second end along the second direction are also not equal to 0, in this embodiment, the extended region can include a first extended region and a second extended region. Each first extended region corresponds one-to-one with the edge line of the non-reference image, and each first extended region and the non-reference image share the corresponding edge line. For each first extended region, with the corresponding edge line as the axis of symmetry, the pixel value of each pixel within the first extended region is equal to the pixel value of the symmetrical pixel in the non-reference image. Each second extended region corresponds one-to-one with a vertex of the non-reference image, and each second extended region shares a corresponding vertex with the non-reference image. For each second extended region, with the corresponding vertex as the center of symmetry (i.e., the mirror center), corresponding pixels in the non-reference image that are centrally symmetrical (i.e., mirror symmetrical) with each pixel of the second extended region can be obtained. Furthermore, the pixel value of any pixel in the second extended region can be equal to the pixel value of its corresponding pixel in the non-reference image. For example, the non-reference image and the extended regions can constitute a new non-reference image. The new non-reference image can be evenly divided according to the scale of the initial image blocks, thereby obtaining at least two initial image blocks of equal scale.

[0054] When the non-reference image cannot be evenly divided according to the scale of the initial image block, the above technical solution can obtain a new non-reference image that can be squared according to the scale of the initial image block by expanding the non-reference image outward along the edge. This can make the coverage area of ​​the obtained initial target image block include the entire area of ​​the non-reference image, so that the search operation can be performed on the entire area of ​​the non-reference image.

[0055] Optionally, finding a reference image block that matches the target image block from the reference image includes: determining at least one image block to be matched in the reference image based on the position of the target image block in the corresponding non-reference image; calculating the similarity between the image block to be matched and the target image block for each of the at least one image block to be matched; comparing the minimum similarity among the similarities corresponding to each of the at least one image block to be matched with a preset similarity threshold; when the minimum similarity is less than or equal to the preset similarity threshold, determining the image block to be matched corresponding to the minimum similarity as the reference image block corresponding to the target image block; wherein, at least one image block to be matched is located within a preset range around a specific image block, the position of the specific image block in the reference image is consistent with the position of the target image block in the corresponding non-reference image, and the scale of each image block to be matched is the same as the scale of the corresponding target image block.

[0056] For example, when searching for a matching reference image block in a reference image for each target image block, a specific image block corresponding to the target image block can first be determined in the reference image. The position of the specific image block in the reference image is consistent with the position of the target image block in the corresponding non-reference image. At least one image block to be matched can be determined within a preset range surrounding the specific image block, and each image block to be matched has the same scale as the target image block. The preset range surrounding the specific image block can include the area covered by the specific image block, and can also include a preset surrounding area outside the specific image block. For example, the preset range surrounding the specific image block can be the area covered by magnifying the edges of the specific image block by a factor of two, with the center point of the specific image block as the scaling center. It should be noted that when the specific image block shares an edge with the reference image (including the case of shared vertices), the preset surrounding area of ​​the specific image block is always within the reference image; in other words, the determined image blocks to be matched are all located within the reference image. For example, after determining the image blocks to be matched, the similarity between each image block to be matched and the target image block can be calculated. This similarity can be determined by the difference between the pixel values ​​of the image block to be matched and the pixel values ​​of the target image block. The smaller the difference, the smaller the similarity, indicating that the image block to be matched is more similar to the target image block. The minimum similarity among the similarities corresponding to each image block to be matched is compared with a preset similarity threshold. When the minimum similarity is less than or equal to the preset similarity threshold, the image block to be matched corresponding to the minimum similarity can be determined as the reference image block that matches the target image block.

[0057] The above technical solution determines a specific image block in the reference image that is consistent with the position of the target image block, and determines the image block to be matched within a preset range around the specific image block. This can fully take into account the situation that the objects in the acquisition area undergo small movements during the acquisition period of the image sequence to be processed, so that one or more image blocks to be matched can be determined in a more reasonable location area. On the other hand, by calculating the similarity between the image block to be matched and the target image block, when the minimum similarity is less than or equal to the preset similarity threshold, the image block to be matched corresponding to the minimum similarity is determined as the reference image block. This is beneficial to make the image content contained in the determined reference image block as consistent as possible with that of the target image block, so as to ensure that the algorithm processing result corresponding to the target image block can be well integrated into the non-reference image.

[0058] Optionally, calculating the similarity between the image patch to be matched and the target image patch includes: subtracting the pixel value of each pixel in the image patch to be matched from the pixel value of the corresponding pixel in the target image patch to determine the pixel difference corresponding to each pixel in the image patch to be matched; and averaging the pixel differences corresponding to each pixel in the image patch to be matched to determine the image deviation corresponding to the image patch to be matched, wherein the similarity is represented by the image deviation.

[0059] For example, when calculating the similarity between a target image patch and each corresponding image patch to be matched, the pixel values ​​of each pixel in the image patch to be matched can be subtracted from the pixel values ​​of the corresponding pixels in the target image patch to obtain the pixel differences for each pixel in the image patch to be matched. The average value of these pixel differences can be calculated, and the absolute value of this average value can be used as the image deviation for the image to be matched. The similarity between the target image patch and the image patch to be matched can be represented by the image deviation; the smaller the image deviation, the greater the similarity.

[0060] The above technical solution can quickly and accurately determine the difference between the target image block and the image block to be matched by taking the average of the difference between the pixel values ​​of each pixel in the image block to be matched and the pixel values ​​of the corresponding pixels in the target image block, thereby quickly and accurately determining the reference image.

[0061] Optionally, the center point of the at least one image patch to be matched includes one or more of the four vertices of the specific image patch, the midpoints of the four edges of the specific image patch, and the center point of the specific image patch.

[0062] For example, when one vertex of a specific image patch is located at any of the four vertices of a reference image, at least one image patch to be matched within a preset range around the specific image patch includes: an image patch whose center point is located at the center point of the specific image patch (i.e., the specific image patch itself), two image patches whose center points are each located at the midpoint of two edges in the specific image patch that do not overlap with the reference image, and an image patch whose center point is located at a vertex in the specific image patch that is not on an edge of the reference image. Please refer to [link to relevant documentation]. Figure 2 The diagram illustrates at least one image block to be matched within a preset range surrounding a specific image block according to an embodiment of the present invention. Figure 2 In the illustrated embodiment, the area formed by the nine square frames marked A1-A9 and the white area constitutes the entire area of ​​the reference image, and the specific image block is the image block composed of A1, A2, A4, and A5. At least one image block to be matched within a preset range around the specific image block includes: (1) an image block whose center point is located at the center point of the specific image block (within...). Figure 2(a1) The image block shown is the specific image block itself; (2) The center point is located at the midpoint of the right edge of the specific image block (in Figure 2 The image patch shown is a2), which is in Figure 2 The image may include four rectangular boxes: A2, A3, A5, and A6; (3) the center point is located at the midpoint of the lower edge of a specific image block (in Figure 2 The image block shown is a3), which is in Figure 2 The image may include four rectangular boxes: A4, A5, A7, and A8; (4) the center point is located at the lower right vertex of a specific image block (in Figure 2 The image block shown is a4), which is in Figure 2 The image block can include four rectangular boxes: A5, A6, A8, and A9. It can be understood that the two edges in a specific image block that do not overlap with the reference image are the right edge and the bottom edge, and the midpoints of these two edges can be used as the center points of the image blocks to be matched. The vertex in a specific image block that is not on an edge of the reference image is the bottom right vertex, and this vertex can also be used as the center point of the image block to be matched.

[0063] For example, when the vertices of a specific image patch do not coincide with any of the four vertices of the reference image, and share an edge with the reference image, at least one image patch to be matched within a preset range around the specific image patch includes: an image patch whose center point is located at the center point of the specific image patch; three image patches whose center points are each located at the midpoint of three edges in the specific image patch that do not overlap with the reference image; and two image patches whose center points are each located at two vertices in the specific image patch that are not on the edge of the reference image. Please refer to [link to relevant documentation]. Figure 3 The diagram shown illustrates at least one image block to be matched within a preset range surrounding a specific image block according to another embodiment of the present invention. Figure 3 In the illustrated embodiment, the area formed by the 12 square frames marked B1-B12 and the white area constitutes the entire area of ​​the reference image, and the specific image block is the image block composed of B1-4. At least one image block to be matched within a preset range around the specific image block includes: (1) the center point is located at the center point of the specific image block (within Figure 3 (b2) is the image patch itself; (2) the center point is located at the midpoint of the right edge of the image patch. Figure 3 The image block shown is b3), which is in Figure 3 The image may include four square boxes: B2, B4, B5, and B6; (3) the center point is located at the midpoint of the lower edge of a specific image block (in Figure 3 The image block shown is b5), which is in Figure 3 It may include four square boxes B3, B4, B7 and B8; (4) the center point is located at the midpoint of the left edge of a specific image block (in Figure 3The image block shown is b1), which is in Figure 3 The image can include four rectangular boxes: B1, B3, B10, and B11; (5) the center point is located at the lower right vertex of a specific image block (in Figure 3 The image block shown is b6), which is in Figure 3 The image may include four rectangular boxes: B4, B6, B8, and B9; (5) the center point is located at the lower left vertex of a specific image block (in Figure 3 The image block shown is b4), which is in Figure 3 The image block can include four rectangular boxes: B3, B7, B11, and B12. It can be understood that the three edges in a specific image block that do not overlap with the reference image are the right edge, the bottom edge, and the left edge, and the midpoints of these three edges can be used as the center points of the image block to be matched. The vertices in a specific image block that are not on the edges of the reference image are the bottom right vertex and the bottom left vertex, and these two vertices can also be used as the center points of the image block to be matched.

[0064] For example, when none of the edges of a specific image block overlap with any of the edges of a reference image, at least one image block to be matched within a preset range around the specific image block may include: four image blocks whose center points are each located at the four vertices of the specific image block, four image blocks whose center points are each located at the midpoints of the four edges of the specific image block, and an image block whose center point is located at the center point of the specific image block.

[0065] The above technical solution can determine up to nine image blocks to be matched within a preset range around a specific image block. When multiple image blocks to be matched are determined, each image block to be matched can be arranged relatively evenly within the preset range around the image block. This can help avoid missing image blocks that meet the requirements of a reference image among the determined image blocks to be matched. It can also limit the number of image blocks to be matched within the preset range around the image block, which helps save the time of the search operation.

[0066] Optionally, based on the position of the target image patch in the corresponding non-reference image, at least one image patch to be matched is determined in the reference image, including: for any target point determined based on the position of the target image patch in the corresponding non-reference image, if all image patches centered at the target point and with a scale equal to the scale of the corresponding target image patch are located within the reference image, then the image patch centered at the target point and with a scale equal to the scale of the corresponding target image patch is determined as the image patch to be matched; wherein, the target point includes the four vertices of the specific image patch, the midpoints of the four edges of the specific image patch, and the center point of the specific image patch.

[0067] For example, for each target image patch, after determining the specific image patch corresponding to the target image patch within the reference image, the four vertices, the midpoints of the four edges, and the center point of the specific image patch can be determined as nine target points in the reference image corresponding to the target image patch. For any one of the determined nine target points, an image patch centered at that target point, with a scale equal to that target image patch, and whose covered area is entirely within the reference image, can be determined as the image patch to be matched corresponding to that target image patch.

[0068] The above technical solution can ensure that all the determined image blocks to be matched are within the reference image. This helps to avoid the situation where a certain image block may be located at the edge of the reference image, causing some areas within a preset range around the specific image block to be outside the reference image. Since the areas outside the reference image do not have corresponding algorithm processing results, this can effectively avoid the situation where the algorithm processing results corresponding to the determined reference image block are invalid, thus making it impossible to determine the algorithm processing results corresponding to the target image block.

[0069] Optionally, when a reference image block is found, the algorithm processing result corresponding to the reference image block is determined as the algorithm processing result corresponding to the target image block, including: when a reference image block is found, determining and caching the motion vector between the reference image block and the target image block; when processing image blocks in non-reference images using the target image processing algorithm, assigning the algorithm processing result corresponding to the reference image block as the algorithm processing result corresponding to the target image block according to the motion vector.

[0070] For example, for any target image patch, the motion vector can be understood as the distance and direction in which the target image patch in the non-reference image moves relative to the reference image patch in the reference image. For ease of description and understanding, the position of the reference image patch in the reference image can be represented by image coordinates (x0, y0), and the position of the target image patch in the non-reference image can be represented by image coordinates (x1, y1). The motion vector can then be represented as (x1-x0, y1-y0). In some embodiments, the reference image patch and the target image patch do not actually have their own image coordinates; the motion vector can be directly represented by the offset of the target image patch relative to the reference image patch. For example, after determining the motion vector between the reference image patch and the target image patch, the determined motion vector can be cached. When processing the image patch in the reference image patch using a target image processing algorithm, the algorithm processing result corresponding to the reference image patch can be assigned to the algorithm processing result corresponding to the target image patch according to the cached motion vector.

[0071] In the above technical solution, motion vectors can represent the motion trajectory of a reference image block to its corresponding non-reference image block. This allows for the storage of only the motion vectors, eliminating the need to store the position data of each image block in all images. Furthermore, motion vectors enable a more accurate matching of the target image block's location with the reference image block's location, allowing the algorithm processing result corresponding to the reference image block to be accurately assigned to the corresponding target image block.

[0072] Optionally, determining the algorithm processing result corresponding to the non-reference image further includes: for each remaining image block in the non-reference image other than the target image block, processing the remaining image block using the target image processing algorithm to obtain the algorithm processing result corresponding to the remaining image block.

[0073] For example, for each remaining image block in each non-reference image other than the target image block, the target image processing algorithm can be used to process the remaining image block, and the processing result can be used as the algorithm processing result corresponding to the remaining image block.

[0074] The above technical solution can use the target image processing algorithm to process the remaining image blocks when the non-reference image also includes the remaining image blocks other than the target image block. This helps to ensure that all regions of the non-reference image can correspond to the algorithm processing results.

[0075] According to another aspect of the present invention, an image processing apparatus is also provided. Figure 4 A schematic diagram of an image processing module of an image processing apparatus according to an embodiment of the present invention is shown, such as... Figure 4 As shown, the device may include an image processing module 400, which includes:

[0076] The acquisition submodule 410 is used to acquire the image sequence to be processed, which includes a reference image and non-reference images.

[0077] The determination submodule 420 is used to determine the algorithm processing result corresponding to the non-reference image, wherein the algorithm processing result corresponding to any image includes the result obtained after processing the image using the target image processing algorithm;

[0078] The determination submodule 420 includes:

[0079] The execution unit 4201 is used to perform a search operation for each target image block in the non-reference image, the search operation including: searching for a reference image block in the reference image that matches the target image block;

[0080] The first determining unit 4202 is used to determine the algorithm processing result corresponding to the reference image block as the algorithm processing result corresponding to the target image block when a reference image block is found for each target image block in the non-reference image. The algorithm processing result corresponding to any image block includes the result obtained after processing the image block using the target image processing algorithm.

[0081] The second determining unit 4203 is used to process each target image block in the non-reference image using a target image processing algorithm when no reference image block is found, so as to obtain the algorithm processing result corresponding to the target image block.

[0082] The algorithm processing results corresponding to the non-reference image include: the target image block in the non-reference image and the algorithm processing results corresponding to the remaining image blocks other than the target image block.

[0083] Those skilled in the art can understand the specific implementation scheme and beneficial effects of the above-described image processing device by reading the relevant description of the image processing method 100. For the sake of brevity, it will not be described in detail here.

[0084] According to another aspect of the present invention, an electronic device is also provided. Figure 5 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. Figure 5 As shown, the electronic device 500 includes a processor 510 and a memory 520. The memory 520 stores a computer program, and the computer program instructions are executed by the processor 510 to perform the above-described image processing method.

[0085] According to another aspect of the present invention, a storage medium storing a computer program / instructions is also provided. The storage medium may include, for example, a storage component of a tablet computer, a hard disk of a personal computer, an erasable programmable read-only memory (EPROM), a portable read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The storage medium may be any combination of one or more computer-readable storage media. The computer program / instructions are used by a processor during runtime to execute the above-described image processing method.

[0086] According to another aspect of the present invention, a computer program product is also provided, including computer program instructions, which, when executed, are used to perform the image processing method described above.

[0087] Those skilled in the art can understand the specific implementation schemes of the above-mentioned electronic devices, storage media, and computer program products by reading the relevant descriptions of the image processing methods above, and for the sake of brevity, they will not be described in detail here.

[0088] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of the invention. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of the invention. All such changes and modifications are intended to be included within the scope of the invention as claimed in the appended claims.

[0089] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0090] In the several embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0091] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0092] Similarly, it should be understood that, in order to streamline the invention and aid in understanding one or more of the various aspects of the invention, features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of the invention. However, this approach should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, its inventive point lies in solving the corresponding technical problem with fewer features than all of those in a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0093] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0094] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0095] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules in the image processing apparatus according to embodiments of the present invention. The present invention can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing some or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0096] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0097] The above description is merely a specific embodiment of the present invention or an explanation of that embodiment. The scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, The method includes image processing operations, which include: Obtain a sequence of images to be processed, which includes a reference image and non-reference images; Determine the algorithm processing result corresponding to the non-reference image, wherein the algorithm processing result corresponding to any image includes the result obtained after processing the image using the target image processing algorithm; The step of determining the algorithm processing result corresponding to the non-reference image includes: For each target image patch in the non-reference image, a search operation is performed, the search operation including: finding a reference image patch in the reference image that matches the target image patch; For each target image patch in the non-reference image When the reference image block is found, the algorithm processing result corresponding to the reference image block is determined as the algorithm processing result corresponding to the target image block. The algorithm processing result corresponding to any image block includes the result obtained after processing the image block using the target image processing algorithm. If the reference image block is not found, the target image block is processed by the target image processing algorithm to obtain the algorithm processing result corresponding to the target image block; The algorithm processing results corresponding to the non-reference image include: the algorithm processing results corresponding to the target image block in the non-reference image and the remaining image blocks other than the target image block.

2. The method as described in claim 1, characterized in that, The images in the image sequence to be processed are either images in the Y channel or images in the RGB channel.

3. The method as described in claim 2, characterized in that, The image processing operation includes a first image processing operation, in which the images in the image sequence to be processed are images in the Y channel. In the first image processing operation, obtaining the image sequence to be processed includes: Obtain an initial image sequence, which includes a reference initial image and non-reference initial images; The images in the initial image sequence are converted to the YUV color space to obtain a Y channel image sequence. The Y channel image in the Y channel image sequence that corresponds to the reference initial image is the reference image, and the Y channel image in the Y channel image sequence that corresponds to the non-reference initial image is the non-reference image.

4. The method as described in claim 3, characterized in that, The reference image containing the algorithm processing result in the first image processing operation is the reference result image, and the non-reference image containing the algorithm processing result in the first image processing operation is the non-reference result image; wherein, the image processing operation further includes a second image processing operation, in which, when performing the step of converting the images in the initial image sequence to the YUV color space respectively, a U-channel image sequence and a V-channel image sequence are also obtained, wherein, In the second image processing operation, obtaining the image sequence to be processed includes: The images in the YUV image sequence are converted to the RGB color space to obtain an RGB image sequence. The RGB image in the RGB image sequence that corresponds to the reference initial image is the reference image in the second image processing operation, and the RGB image in the RGB image sequence that corresponds to the non-reference initial image is the non-reference image in the second image processing operation. The YUV image sequence includes a reference YUV image corresponding to the reference initial image and a non-reference YUV image corresponding to the non-reference initial image. The reference YUV image includes the reference result image, the U-channel image in the U-channel image sequence corresponding to the reference initial image, and the V-channel image in the V-channel image sequence corresponding to the reference initial image. The non-reference YUV image includes the non-reference result image, the U-channel image in the U-channel image sequence corresponding to the non-reference initial image, and the V-channel image in the V-channel image sequence corresponding to the non-reference initial image.

5. The method as described in claim 2, characterized in that, The image processing operation includes a first image processing operation and a second image processing operation. In the first image processing operation, the images in the image sequence to be processed are Y-channel images. In the second image processing operation, the images in the image sequence to be processed are RGB images. In the second image processing operation, the images in the image sequence to be processed are obtained one-to-one based on the images in the image sequence to be processed in the first image processing operation. The target image blocks included in the non-reference image in the second image processing operation correspond one-to-one with the target image blocks included in the non-reference image in the first image processing operation. Specifically, for any target image block in the non-reference image in the first image processing operation, if a reference image block matching the target image block is found, the image block corresponding to the target image block in the second image processing operation is taken as the target image block in the second image processing operation.

6. The method according to any one of claims 1-5, characterized in that, The step of performing a lookup operation for each target image patch in the non-reference image includes: Based on the non-reference image, at least two initial image blocks of equal scale are obtained, wherein each initial image block is used as an initial target image block; For each target image block for which no reference image block is found, the search operation is performed based on that target image block; For any target image block, if no reference image block matching the target image block is found in the search operation, and the preset condition is met, then the target image block is divided into at least two new image blocks of equal size, and each of the divided new image blocks is used as a new target image block. Then, the step of performing the search operation based on the target image block for each target image block for which no reference image block is found is returned. The preset conditions include: the scale of the new image block obtained based on the target image block is greater than or equal to the preset scale.

7. The method as described in claim 6, characterized in that, The step of obtaining at least two initial image patches of equal scale based on the non-reference image includes: If the non-reference image cannot be evenly divided according to the scale of the initial image block, a new non-reference image is obtained by extending outward along at least part of the edges of the non-reference image. The pixel values ​​of the extended region in the new non-reference image are determined by mirror mapping the pixel values ​​of the region adjacent to the extended region in the non-reference image. The new non-reference image is divided equally according to the scale of the initial image patch to obtain at least two initial image patches of equal scale.

8. The method according to any one of claims 1-5, characterized in that, The step of finding a reference image patch that matches the target image patch from the reference image includes: Based on the position of the target image patch in the corresponding non-reference image, at least one image patch to be matched is determined in the reference image; For each of the at least one image blocks to be matched, calculate the similarity between the image block to be matched and the target image block; The minimum similarity among the similarities corresponding to each of the at least one image block to be matched is compared with a preset similarity threshold; When the minimum similarity is less than or equal to the preset similarity threshold, the image block to be matched corresponding to the minimum similarity is determined as the reference image block corresponding to the target image block; Wherein, the at least one image block to be matched is located within a preset range around a specific image block, the position of the specific image block in the reference image is consistent with the position of the target image block in the corresponding non-reference image, and the scale of each image block to be matched is the same as the scale of the corresponding target image block.

9. The method as described in claim 8, characterized in that, The calculation of the similarity between the image patch to be matched and the target image patch includes: Subtract the pixel value of each pixel in the image block to be matched from the pixel value of the corresponding pixel in the target image block to determine the pixel difference value corresponding to each pixel in the image block to be matched. The average of the pixel differences corresponding to each pixel in the image block to be matched is calculated to determine the image deviation corresponding to the image block to be matched, and the similarity is represented by the image deviation.

10. The method as described in claim 8, characterized in that, The center point of the at least one image patch to be matched includes one or more of the four vertices of the specific image patch, the midpoints of the four edges of the specific image patch, and the center point of the specific image patch.

11. The method as described in claim 10, characterized in that, The step of determining at least one image patch to be matched in the reference image based on the position of the target image patch in the corresponding non-reference image includes: For any target point determined based on the position of the target image patch in the corresponding non-reference image, if all image patches centered at the target point and with a scale equal to the scale of the corresponding target image patch are located within the reference image, then the image patch centered at the target point and with a scale equal to the scale of the corresponding target image patch is determined as the image patch to be matched. The target point includes the four vertices of the specific image patch, the midpoints of the four edges of the specific image patch, and the center point of the specific image patch.

12. The method according to any one of claims 1-5, characterized in that, When the reference image block is found, determining the algorithm processing result corresponding to the reference image block as the algorithm processing result corresponding to the target image block includes: When the reference image block is located, the motion vector between the reference image block and the target image block is determined; When processing image blocks in the non-reference image using the target image processing algorithm, the algorithm processing result corresponding to the reference image block is assigned to the algorithm processing result corresponding to the target image block according to the motion vector.

13. The method according to any one of claims 1-5, characterized in that, The algorithm processing result for determining the non-reference image also includes: For each remaining image block in the non-reference image other than the target image block, the target image processing algorithm is applied to the remaining image block to obtain the algorithm processing result corresponding to the remaining image block.

14. An image processing apparatus, characterized in that, The device includes an image processing module, which comprises: The acquisition submodule is used to acquire the image sequence to be processed, which includes a reference image and non-reference images. The determination submodule is used to determine the algorithm processing result corresponding to the non-reference image, wherein the algorithm processing result corresponding to any image includes the result obtained after processing the image using the target image processing algorithm; The determining submodule includes: An execution unit is configured to perform a search operation for each target image block in the non-reference image, the search operation including: searching for a reference image block in the reference image that matches the target image block; The first determining unit is configured to, for each target image block in the non-reference image, when the reference image block is found, determine the algorithm processing result corresponding to the reference image block as the algorithm processing result corresponding to the target image block, wherein the algorithm processing result corresponding to any image block includes the result obtained after processing the image block using the target image processing algorithm; The second determining unit is used to process each target image block in the non-reference image using the target image processing algorithm when the reference image block is not found, so as to obtain the algorithm processing result corresponding to the target image block. The algorithm processing results corresponding to the non-reference image include: the algorithm processing results corresponding to the target image block in the non-reference image and the remaining image blocks other than the target image block.

15. An electronic device comprising a processor and a memory, characterized in that, The memory stores a computer program, the computer program instructions of which, when executed by the processor, are used to perform the image processing method as described in any one of claims 1-13.

16. A storage medium storing a computer program / instructions, characterized in that, The computer program / instructions are used to perform the image processing method as described in any one of claims 1-13 when the program is executed.

17. A computer program product comprising computer program instructions, characterized in that, The computer program instructions, when executed, are used to perform the image processing method as described in any one of claims 1-13.

Images