Image processing method and device, electronic equipment, chip and medium

Abstract

The invention provides an image processing method and device, electronic equipment, a chip and a medium, and relates to the field of digital image processing.The method comprises the steps that a first image pyramid of a current frame image and a second image pyramid of a reference frame are determined, and the reference frame image comprises one or more frames of images adjacent to the current frame image in an image sequence; determining a plurality of static image blocks of the first image pyramid based on the first image pyramid and the second image pyramid; and outputting the probability of scene switching of each pixel point in the current frame image based on the plurality of static image blocks, thereby solving the problem of relatively rough granularity in the scene switching detection process, improving the scene switching detection fineness under the condition that the scene switching in the current frame image is not in the whole frame level, and improving the scene switching detection accuracy. And determining the probability of scene switching of each pixel point in the current frame image.

Description

Technical Field

[0001] The present disclosure relates to the field of digital image processing, and in particular to an image processing method, device, electronic device, chip and medium. Background Art

[0002] Scene switch detection is currently widely used in various fields of the multimedia industry. As a highly versatile technical solution, this technology can enhance the image quality, effects, and performance of other related solutions. In video encoding and decoding, scene switch detection is used to update the adaptive GOP (Group of Pictures) to improve the performance of video encoding; scene switch detection is used to switch the bit rate control method to improve the accuracy of bit rate control. In the ISP full channel or in the display video playback, scene switch detection is used to adjust the color and brightness of different styles for scene pictures of different styles. Therefore, scene switch detection is crucial in practical applications.

[0003] However, in the related technical solutions, the main focus of scene switching detection is on the accuracy of detection, and the detection result outputted is only whether the scene switching occurs in the frame, and the granularity of the detection process is relatively coarse. Summary of the invention

[0004] The present disclosure provides an image processing method, device, electronic device, chip and medium to solve the problem of coarse granularity of the scene switching detection process in the related art, so as to improve the precision of scene switching detection when the scene switching in the current frame image is not at the whole frame level, and clarify the probability of scene switching for each pixel in the current frame image.

[0005] A first aspect of the present disclosure provides an image processing method, the method comprising: determining a first image pyramid of a current frame image and a second image pyramid of a reference frame, the reference frame image comprising one or more frames of images adjacent to the current frame image in an image sequence; determining multiple static image blocks of the first image pyramid based on the first image pyramid and the second image pyramid; and outputting a probability of scene switching for each pixel in the current frame image based on the multiple static image blocks.

[0006] In some embodiments of the present disclosure, determining a first image pyramid of a current frame image and a second image pyramid of a reference frame includes: acquiring a current frame image and a reference frame image corresponding to the current frame image in an image sequence; and performing sampling processing on the reference frame image and the current frame image at different scales to determine a first image pyramid corresponding to the current frame image and a second image pyramid corresponding to the reference frame image.

[0007] In some embodiments of the present disclosure, determining a plurality of static image blocks of the first image pyramid based on the first image pyramid and the second image pyramid includes: dividing the first image pyramid and the second image pyramid by number of layers, determining a first image block of a first layer in the first image pyramid and a second image block of a first layer in the second image pyramid corresponding to a position and a size of the first image block; and determining a plurality of static image blocks based on the second image block and the first image block.

[0008] In some embodiments of the present disclosure, the plurality of static image blocks include a first static image block and a second static image block, and determining the plurality of static image blocks based on the second image block and the first image block includes: determining an error value between a pixel value of each pixel point in the first image block and a pixel value of each pixel point in the second image block; when the error value is less than a preset threshold, determining the first image block to be the first static image block; and determining a second static image block of a second layer in a first image pyramid based on the first static image block, the second static image block being a third image block having the same mapping position of the first static image block in the second layer of the first image pyramid.

[0009] In some embodiments of the present disclosure, the method further includes: determining a first coefficient based on the size of the first image block; and determining a preset threshold based on the first coefficient.

[0010] In some embodiments of the present disclosure, based on multiple static image blocks, outputting the probability of scene switching for each pixel in the current frame image includes: connecting the first static image block and the second static image block according to a preset rule to obtain a static area; based on the static area, the first static image block and the second static image block, determining the probability of scene switching for each pixel in the current frame.

[0011] In some embodiments of the present disclosure, the preset rule includes at least one of the following: distance correlation, area density correlation, color brightness correlation, and texture direction correlation.

[0012] A second aspect of the present disclosure provides an image processing device, which includes: a first determination unit, used to determine a first image pyramid of a current frame image and a second image pyramid of a reference frame, wherein the reference frame image includes one or more frames of images adjacent to the current frame image in an image sequence; a second determination unit, used to determine multiple static image blocks of the first image pyramid based on the first image pyramid and the second image pyramid; and an output unit, used to output the probability of scene switching for each pixel in the current frame image based on the multiple static image blocks.

[0013] The third aspect embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect embodiment of the present disclosure.

[0014] The fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the first aspect embodiment of the present disclosure.

[0015] The fifth aspect embodiment of the present disclosure proposes a chip, which includes one or more interface circuits and one or more processors; the interface circuit is used to receive signals from a memory of an electronic device and send signals to the processor, the signals including computer instructions stored in the memory, and when the processor executes the computer instructions, the electronic device executes the method described in the first aspect embodiment of the present disclosure.

[0016] In summary, according to the image processing method proposed in the present disclosure, by determining a first image pyramid of a current frame image and a second image pyramid of a reference frame, the reference frame image includes one or more frames of images adjacent to the current frame image in a video image sequence; based on the first image pyramid and the second image pyramid, multiple static image blocks of the first image pyramid are determined; based on the multiple static image blocks, the probability of scene switching for each pixel in the current frame image is output, which solves the problem of coarse granularity in the scene switching detection process, and achieves the improvement of the precision of scene switching detection when the scene switching in the current frame image is not at the whole frame level, and clarifies the probability of scene switching for each pixel in the current frame image.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0019] Figure 1 A schematic diagram of a display scene detection method provided by an embodiment of the present disclosure;

[0020] Figure 2 A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0021] Figure 3 A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0022] Figure 4 A schematic diagram of image pyramid decomposition and mapping provided by an embodiment of the present disclosure;

[0023] Figure 5 A schematic diagram of connecting a regional density correlation and a distance correlation provided by an embodiment of the present disclosure;

[0024] Figure 6 A schematic diagram of a specific scene switching method provided by an embodiment of the present disclosure;

[0025] Figure 7 A schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure;

[0026] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure;

[0027] Fig. 9 A schematic diagram of the structure of a chip provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] The embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions.

[0029] Scene switching detection is currently widely used in various fields of the multimedia industry. As a highly versatile technical solution, this technology can enhance the image quality, effects, and performance of other related solutions. In video encoding and decoding, scene switching detection is used to update the adaptive GOP to improve the performance of video encoding; scene switching detection is used to switch the bit rate control method to improve the accuracy of bit rate control. In the ISP full channel or in the display video playback, scene switching detection is used to adjust the color and brightness of different styles for different styles of scene pictures. Therefore, scene switching detection is crucial in practical applications.

[0030] In the relevant technical solution, a deep learning algorithm is adopted, and the discrimination threshold of the model is automatically adjusted to the optimal value during the training process, so there is no need to set the threshold; since the model input adds the frame difference between two frames, the model converges faster; since the model adopts batch normalization technology to prevent training overfitting, the generalization ability of the model is improved.

[0031] It can be seen that the main focus of scene switching detection in related technologies is the accuracy of detection. The detection result output is only whether the scene switching in the frame has occurred. It does not take into account that the scene switching in a frame is not at the level of the entire frame, and the granularity of the detection process is relatively coarse. For the display field, this coarse granularity will reduce the ability of the effect enhancement module in the display system.

[0032] In order to solve the problems existing in the related technology, the present disclosure proposes an image processing method, which can better realize multi-window targeted continuous enhancement by determining the probability of scene switching for each pixel in the current frame image. Taking into account that the scene switching in a frame is not at the whole frame level, the fineness of the judgment granularity is improved, and at the same time, the possibility of sudden change of the entire picture style in the non-scene switching area is reduced, thereby further improving the image quality.

[0033] In the display architecture, such as Figure 1 As shown, the scene switching detection of image processing can be applied to the adaptive image inspection process, such as the logo area, the advertising pop-up area, and multiple applications on the same screen (such as computers, mobile phones, and the same display interface with video playback, chat windows, basic interfaces, etc.). For different areas, a mapping map of non-scene switching areas is output, and different areas are enhanced in different colors and brightness according to the mapping map.

[0034] The image processing method proposed in the present disclosure can be applied to image processing fields such as ISP (Image Signal Processing), image display and image encoding and decoding.

[0035] The image processing method provided in this application is described in detail below with reference to the accompanying drawings.

[0036] Figure 2 Flow chart of an image processing method provided by an embodiment of the present disclosure. Figure 2 As shown, the image processing method can be applied to the scene of scene switching detection, and the image processing method includes steps 101-103.

[0037] Step 101 : determining a first image pyramid of a current frame image and a second image pyramid of a reference frame, wherein the reference frame image includes one or more frame images adjacent to the current frame image in an image sequence.

[0038] In the embodiment of the present disclosure, the current frame image refers to a frame image currently played in the video. The reference frame image refers to the previous frame image or the next frame image in the video playback order of the current frame image. The number of reference frame images can be one frame or multiple frames. The specific number can be manually set according to actual conditions and is not limited in the embodiment of the present disclosure. The first image pyramid image and the second image pyramid image can be Gaussian pyramids, which are obtained by scaling the current frame image and the reference frame image.

[0039] The image pyramid is an image set consisting of multiple sub-images of different resolutions of an image, which is generated by continuously reducing the sampling rate of an image. In the present disclosure, the first image pyramid includes an image set of different resolutions of the current frame image, and the second image pyramid includes an image set of different resolutions of the reference frame image.

[0040] Step 102: Determine a plurality of static image blocks of the first image pyramid based on the first image pyramid and the second image pyramid.

[0041] In an embodiment of the present disclosure, the number of layers of the first image pyramid and the second image pyramid and the first image block and the second image block are determined according to the first image pyramid and the second image pyramid, and a plurality of static image blocks of the first image pyramid are determined according to the number of layers and the first image block and the second image block.

[0042] The first image pyramid and the second image pyramid are divided to obtain the number of layers of the first image pyramid and the second image pyramid. The number of layers can be manually set according to actual conditions, for example, it can be 2 layers, 3 layers, 4 layers or 5 layers, which is not limited in the embodiment of the present disclosure, and the number of layers of a common pyramid image is 3 layers.

[0043] The size of the first image block of each layer in the first image pyramid may be different, and the size may be manually set according to actual conditions, and is not limited in the embodiments of the present disclosure. A common first image block may be a matrix of size 8*8.

[0044] The second image block is determined according to the first image pyramid and the first image block. The second image block has the same number of layers, position and size as the first image block in the second image pyramid.

[0045] The static image block (ROIBLK) refers to a first image block whose error value between the first image block and the second image block meets a preset threshold.

[0046] Step 103: Based on the multiple static image blocks, output the probability of scene switching for each pixel in the current frame image.

[0047] In the embodiments of the present disclosure, multiple static image blocks can be connected to each other through preset rules to determine the static area, and the scene switching detection is performed on the current frame image through the static area to determine the probability of scene switching for each pixel in the current frame image. At the same time, since the probability of switching for each pixel of the current image is obtained, the probability of scene switching for each pixel can be formed into a confidence image and output in the form of a confidence image. The scene switching is performed on the current frame based on the confidence image, and subsequent image processing operations such as image enhancement are performed.

[0048] Scene switching refers to the change in color, content and other factors between the current frame image and the reference frame image. Static region (ROI, region of interest) can also be called region of interest. During the image processing process, a specific area of ​​the image is of interest, and this area is called the region of interest. After setting the region of interest, you can perform overall operations on the region.

[0049] In summary, according to the image processing method proposed in the present disclosure, by determining a first image pyramid of a current frame image and a second image pyramid of a reference frame, the reference frame image includes one or more frames of images adjacent to the current frame image in an image sequence; based on the first image pyramid and the second image pyramid, multiple static image blocks of the first image pyramid are determined; based on the multiple static image blocks, the probability of scene switching for each pixel in the current frame image is output, which solves the problem of coarse granularity in the scene switching detection process, and achieves the improvement of the precision of scene switching detection when the scene switching in the current frame image is not at the whole frame level, and clarifies the probability of scene switching for each pixel in the current frame image.

[0050] based on Figure 2 The embodiment shown, Figure 3 The following further illustrates a flow chart of an image processing method proposed in the present disclosure. Figure 3 based on Figure 2 In the embodiment shown, step 101, step 102 and step 103 are further defined. Figure 3 In the embodiment shown, step 101 includes step 201 and step 202, step 102 includes step 203 and step 204, and step 103 includes step 205 and step 206. The method is applied to the training stage of the result reparameterization method, such as Figure 3 As shown, the method comprises the following steps:

[0051] Step 201: Acquire a current frame image in an image sequence and a reference frame image corresponding to the current frame image.

[0052] In an embodiment of the present disclosure, the number of reference frame images may be manually set, and the reference frame image may be a previous frame image or a subsequent frame image of a current frame image in a video playback sequence.

[0053] Step 202 : Sampling processes of different scales are performed on the reference frame image and the current frame image to determine a first image pyramid corresponding to the current frame image and a second image pyramid corresponding to the reference frame image.

[0054] In the embodiments of the present disclosure, sampling processing at different scales can be understood as the process of upsampling and / or downsampling the current frame image and the reference frame image. The downsampling algorithm can be bicubic, lanczos, or other algorithms, which are not limited in the embodiments of the present disclosure.

[0055] In the process of sampling at different scales, the reduction ratio mapping of each layer of pyramid image relative to the previous layer of pyramid image can be integer level pixels (e.g., 2:1) or non-integer level pixels (e.g., 2.5:1). The horizontal and vertical scaling ratios in the process of sampling at different scales can be the same or different, for example, a horizontal scaling ratio of 2.5:1, a vertical scaling ratio of 3:1, or a normal horizontal scaling ratio and a vertical scaling ratio of 2:1.

[0056] Step 203 , dividing the first image pyramid and the second image pyramid into layers, and determining a first image block in the first layer of the first image pyramid and a second image block in the first layer of the second image pyramid corresponding to the position and size of the first image block.

[0057] In an optional embodiment of the present disclosure, a first image block of a first image pyramid is first determined, and then a second image block of the same layer in a second image pyramid is determined according to the position and size of the first image block.

[0058] It can be understood that the first image block and the second image block have the same position, size, and number of layers. Since there is more than one layer in the image pyramid, according to the order of the layers of the image pyramid from top to bottom, for each layer of the pyramid image, the first image block in the first image pyramid and the second image block with the same position, size, and number of layers in the second pyramid image are determined.

[0059] Step 204: determine a plurality of static image blocks based on the second image block and the first image block.

[0060] In the present disclosure, the plurality of static image blocks include a first static image block and a second static image block.

[0061] In an embodiment of the present disclosure, an error value between a pixel value of each pixel point in a first image block and a pixel value of each pixel point in a second image block is determined; whether the error value is less than a preset threshold is judged, and if the error value is less than the preset threshold, the first image block is determined to be a first static image block; based on the first static image block, a second static image block of a second layer in a first image pyramid is determined, the second static image block being a third image block having the same position as the first static image block mapped in the second layer of the first image pyramid.

[0062] The error value refers to the difference between the first image block and the second image block, and this difference may also be referred to as a motion vector (MV) in the present disclosure. The error value may be calculated by SD (sum of difference), SAD (sum of absolute difference) or other algorithms based on SD / SAD.

[0063] The preset threshold may be manually set according to actual needs, or the first coefficient may be determined based on the size of the first image block; and the preset threshold may be adaptively determined based on the first coefficient.

[0064] In an optional embodiment of the present disclosure, the preset threshold is adaptively calculated according to the size of the first image block, the component weight, and the level of the image pyramid, as shown in Formula 1:

[0065] Thr[cpt]=a[cpt]*BLKsize+b[cpt]*level

[0066] MVthr = c[0]* Thr[0] + c[1]* Thr[1] + c[2]* Thr[2] (Formula 1)

[0067] Wherein, Thr[cpt] is the first coefficient. BLKsize refers to the size of the first image block. level refers to the layer number where the current first image block is located. cpt refers to the component (captain) for the size and layer number of the first image block in the current image channel. a[cpt] and b[cpt] refer to the component weights of the size and layer number of the first image block. c[0], c[1], c[2] refer to the component weights of each image channel. [0], [1], [2] refer to image channels, which can be RGB channels or YUV channels, which are not limited in the embodiments of the present disclosure.

[0068] In an embodiment of the present disclosure, when it is determined that the first image block is a first static image, Figure 4In the image pyramid decomposition diagram and mapping diagram shown, the third image block at the same position after the next layer mapping corresponding to the first static image block can be directly determined as the second static image block, and the error value is no longer calculated for the preset threshold judgment. Due to the loss of pixel accuracy during the downsampling process, there may be some image block areas in the third image block that are not covered by the mapping. The uncovered area can be manually set as the second static image block or the error value can be calculated again for judgment, which is not limited in the disclosed embodiment.

[0069] It can be understood that the third image block is an image block in a next layer of the first image block in the first image pyramid.

[0070] Step 205 : Connect the first static image block and the second static image block according to a preset rule to obtain a static area.

[0071] In the embodiments of the present disclosure, the preset rule refers to a rule for connecting image blocks based on the correlation between the image blocks, such as Figure 5 The connection example of regional density correlation and distance correlation shown. The preset rules may specifically include: distance correlation, regional density correlation, color brightness correlation, texture direction correlation, etc. Among them, distance correlation means that if the distance between two adjacent static image blocks is less than a threshold, they can be connected to form a static area; regional density correlation means that if the area ratio of all static image blocks in a fixed area is greater than a threshold, the area can be set as a static area, wherein the shape and size of the fixed area can be manually set, for example, it can be a static area composed of the upper left corner point of the upper left static image block and the lower right corner point of the lower right static image block; color brightness correlation means that if the average color or brightness difference between two adjacent static image blocks is less than a threshold, they can be connected to form a static area; texture direction correlation means that if the main texture direction angle difference between two adjacent static image blocks is less than a threshold, they can be connected to form a static area.

[0072] The shape of the static area can be manually set according to actual needs, for example, it can be a rectangle, a circle, etc.; the number of static areas can also be manually specified. In the present disclosure, the final static area can be determined by sorting according to certain rules determined manually, and the rules can be the size, position, etc. of the static area.

[0073] Step 206, based on the static area, the first static image block and the second static image block, determine the probability of scene switching for each pixel in the current frame, and output the probability of scene switching for each pixel in the current frame image.

[0074] In the embodiment of the present disclosure, by determining whether the error value of each pixel in the non-static image block in the static area and the error value of each pixel in the non-static area meets the preset threshold, the probability of scene switching for each pixel in the current frame is determined according to the judgment result, that is, the possibility of scene switching for each pixel. The probability of scene switching corresponding to each pixel is output, which can be output in the form of a confidence image in the present disclosure.

[0075] In an optional embodiment of the present disclosure, the scene switching possibility is set to 0 to K. Excluding the static area, the scene switching detection is performed on other non-static areas, wherein the scene switching detection method is not limited in the embodiment of the present disclosure, and is determined by judging the error value in the present disclosure. If a pixel point in the non-static area is detected to have a scene switching, the corresponding confidence of the pixel point is determined to be K, otherwise it is 0.

[0076] For pixels in static image blocks, the corresponding confidence of the pixels is set to 0. For the confidence of pixels in non-static image blocks in the static area connected according to preset rules, the confidence is determined to be an intermediate value in the range of 0 to K.

[0077] In summary, the method provided by the present disclosure obtains a current frame image and a reference frame image corresponding to the current frame image in an image sequence, performs sampling processing on the reference frame image and the current frame image at different scales, determines a first image pyramid corresponding to the current frame image and a second image pyramid corresponding to the reference frame image, divides the first image pyramid and the second image pyramid into layers, determines a first image block in the first layer of the first image pyramid and a second image block in the first layer of the second image pyramid corresponding to the position and size of the first image block, determines a plurality of static image blocks based on the second image block and the first image block, connects the first static image block and the second static image block according to a preset rule to obtain a static area, determines a probability of scene switching for each pixel in the current frame based on the static area, the first static image block and the second static image block, and outputs the probability of scene switching for each pixel in the current frame image, so as to better realize multi-window targeted continuous enhancement, and considering that the scene switching in a frame is not at the whole frame level, improves the fineness of the granularity of the judgment, and reduces the possibility of sudden change of the style of the entire picture in the non-scene switching area, so as to further improve the image quality.

[0078] based on Figure 2 , Figure 3 The embodiment shown, as Figure 6 A schematic diagram of a specific scene switching method is shown.

[0079] In an embodiment of the present disclosure, a reference frame image of a current frame image is confirmed. The first image pyramid and the second image pyramid of the reference frame image and the current frame image are hierarchically divided, and the image block size of each layer of the first pyramid image is determined. According to the top-down order of the image pyramid layers, for each layer of the first image pyramid, a running vector (MV) is determined for each first image block of the current frame image and the second image block at the same position of the same layer of the image pyramid in the reference frame image. If the running vector is less than a preset threshold, the first image block in the current frame image is a static image block (ROI BLK), wherein it should be noted that the third image block at the same position after mapping of the next layer of the image pyramid of the current frame image corresponding to the static image block is also determined as a static image block, and no running vector determination is performed. All the static image blocks determined are connected in a prescribed shape according to certain rules, and finally a fixed number of static regions (ROIs) are formed; according to the static regions and static image blocks, an arbitrary scene switching detection method is used to obtain the probability of scene switching for each pixel point of the current frame image, and a scene switching confidence image is formed, and the confidence image is output.

[0080] In summary, the method disclosed in the present invention takes into account that the scene switching in a frame is not at the whole frame level, and the judgment granularity is relatively fine, which can better perform multi-window targeted continuous enhancement based on the confidence image; at the same time, it can reduce the possibility of sudden changes in the style of the entire picture in the non-scene switching area, thereby further improving the image quality.

[0081] Figure 7 FIG. 7 is a schematic diagram of the structure of an image processing device 700 provided in an embodiment of the present disclosure. Figure 7 As shown, the image processing device comprises:

[0082] A first determining unit 710 is configured to determine a first image pyramid of a current frame image and a second image pyramid of a reference frame, wherein the reference frame image includes one or more frames of images adjacent to the current frame image in an image sequence;

[0083] A second determining unit 720, configured to determine a plurality of static image blocks of the first image pyramid based on the first image pyramid and the second image pyramid;

[0084] The output unit 730 is used to output the probability of scene switching for each pixel in the current frame image based on the multiple static image blocks.

[0085] In some embodiments, the first determination unit 710 is used to: obtain a current frame image in an image sequence and a reference frame image corresponding to the current frame image; perform sampling processing on the reference frame image and the current frame image at different scales to determine a first image pyramid corresponding to the current frame image and a second image pyramid corresponding to the reference frame image.

[0086] In some embodiments, the second determining unit 720 is used to: divide the first image pyramid and the second image pyramid into layers, determine a first image block of the first layer in the first image pyramid and a second image block of the first layer in the second image pyramid corresponding to the position and size of the first image block; and determine a plurality of static image blocks based on the second image block and the first image block.

[0087] In some embodiments, the multiple static image blocks include a first static image block and a second static image block, and the second determination unit 720 is used to: determine an error value between a pixel value of each pixel point in the first image block and a pixel value of each pixel point in the second image block; when the error value is less than a preset threshold, determine that the first image block is the first static image block; based on the first static image block, determine a second static image block of the second layer in the first image pyramid, the second static image block being a third image block having the same mapping position of the first static image block in the second layer of the first image pyramid.

[0088] In some embodiments, the second determination unit 720 is further used to: determine the first coefficient based on the size of the first image block; and determine a preset threshold based on the first coefficient.

[0089] In some embodiments, the output unit 730 is used to: connect the first static image block and the second static image block according to a preset rule to obtain a static area; and determine the probability of scene switching for each pixel in the current frame based on the static area, the first static image block and the second static image block.

[0090] In some embodiments, the preset rule includes at least one of the following: distance correlation, area density correlation, color brightness correlation, and texture direction correlation.

[0091] In summary, an image processing device determines a first image pyramid of a current frame image and a second image pyramid of a reference frame, wherein the reference frame image includes one or more frames of images adjacent to the current frame image in an image sequence; based on the first image pyramid and the second image pyramid, multiple static image blocks of the first image pyramid are determined; based on the multiple static image blocks, the probability of scene switching for each pixel in the current frame image is output, thereby solving the problem of coarse granularity in the scene switching detection process, and achieving, when the scene switching in the current frame image is not at the whole frame level, improving the precision of scene switching detection, and clarifying the probability of scene switching for each pixel in the current frame image.

[0092] Corresponding to the methods provided in the above-mentioned embodiments, the present disclosure also provides an image processing device. Since the device provided in the embodiment of the present disclosure corresponds to the methods provided in the above-mentioned embodiments, the implementation method of the method is also applicable to the device provided in this embodiment and will not be described in detail in this embodiment.

[0093] In the embodiments provided in the present application, the methods and devices provided in the embodiments of the present application are introduced. In order to implement the functions in the methods provided in the embodiments of the present application, the electronic device may include a hardware structure and a software module, and implement the functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. A function of the functions may be executed in the form of a hardware structure, a software module, or a hardware structure plus a software module.

[0094] Figure 8 8 is a block diagram of an electronic device 800 for implementing the above-mentioned image processing method according to an exemplary embodiment. For example, the electronic device 800 may be a mobile phone, a computer, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0095] Reference Figure 8 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0096] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0097] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0098] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0099] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0100] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0101] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0102] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0103] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 1000 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G LTE, 5G NR (NewRadio) or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0104] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0105] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the above instructions can be executed by the processor 820 of the electronic device 800 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0106] An embodiment of the present disclosure further proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the image processing method described in the above embodiment of the present disclosure.

[0107] An embodiment of the present disclosure further provides a computer program product, including a computer program, which executes the image processing method described in the above embodiment of the present disclosure when a processor is used.

[0108] The embodiment of the present disclosure also provides a chip, such as Fig. 9 As shown, the chip includes one or more interface circuits 901 and one or more processors 902; the interface circuit is used to receive signals and send signals to the processor, the signals include computer instructions stored in the memory, and when the processor executes the computer instructions, the chip executes the image processing method described in the above embodiments of the present disclosure.

[0109] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0110] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0111] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0112] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (control method), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0113] It should be understood that the various parts of the embodiments of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0114] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0115] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0116] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. An image processing method, characterized in that: The method comprises: Determine a first image pyramid of a current frame image and a second image pyramid of a reference frame, wherein the reference frame image includes one or more frame images adjacent to the current frame image in an image sequence; Determine a plurality of static image blocks of the first image pyramid based on the first image pyramid and the second image pyramid; Based on the multiple static image blocks, the probability of scene switching for each pixel in the current frame image is output.

2. The method according to claim 1, characterized in that The determining of a first image pyramid of a current frame image and a second image pyramid of a reference frame comprises: Acquire a current frame image in the image sequence and a reference frame image corresponding to the current frame image; Sampling processes of different scales are performed on the reference frame image and the current frame image to determine a first image pyramid corresponding to the current frame image and a second image pyramid corresponding to the reference frame image.

3. The method according to claim 1, characterized in that: The determining, based on the first image pyramid and the second image pyramid, a plurality of static image blocks of the first image pyramid comprises: Dividing the first image pyramid and the second image pyramid into layers, and determining a first image block of a first layer in the first image pyramid and a second image block of a first layer in the second image pyramid corresponding to a position and a size of the first image block; Based on the second image block and the first image block, a plurality of static image blocks are determined.

4. The method according to claim 3, characterized in that The plurality of static image blocks include a first static image block and a second static image block, and the determining the plurality of static image blocks based on the second image block and the first image block includes: Determine an error value between a pixel value of each pixel point in the first image block and a pixel value of each pixel point in the second image block; When the error value is less than a preset threshold, determining that the first image block is a first static image block; Based on the first static image block, a second static image block of a second layer in the first image pyramid is determined, where the second static image block is a third image block at the same position as the first static image block mapped in the second layer of the first image pyramid.

5. The method according to claim 4, characterized in that The method further comprises: Determining a first coefficient based on the size of the first image block; Based on the first coefficient, a preset threshold is determined.

6. The method according to claim 5, characterized in that The outputting the probability of scene switching for each pixel in the current frame image based on the multiple static image blocks includes: According to a preset rule, connecting the first static image block and the second static image block to obtain a static area; Based on the static area, the first static image block, and the second static image block, a probability of scene switching for each pixel in the current frame is determined.

7. The method according to claim 6, characterized in that The preset rules include at least one of the following: Distance correlation, area density correlation, color brightness correlation, and texture direction correlation.

8. An image processing device, characterized in that: The device comprises: A first determining unit, configured to determine a first image pyramid of a current frame image and a second image pyramid of a reference frame, wherein the reference frame image includes one or more frame images adjacent to the current frame image in an image sequence; a second determining unit, configured to determine a plurality of static image blocks of the first image pyramid based on the first image pyramid and the second image pyramid; An output unit is used to output the probability of scene switching for each pixel in the current frame image based on the multiple static image blocks.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

11. A chip, characterized in that: The chip comprises one or more interface circuits and one or more processors; the interface circuit is used to receive a signal and send the signal to the processor, the signal includes a computer instruction stored in a memory, and when the processor executes the computer instruction, the chip executes the method described in any one of claims 1 to 7.

Images