Image processing method and device, electronic equipment and readable storage medium

Abstract

The invention discloses an image processing method and device, electronic equipment and a readable storage medium, and belongs to the technical field of videos. The method comprises the steps that a first frame image and a second frame image are acquired, the first frame image comprises hard subtitles, and the second frame image is an image after the dynamic range of the first frame image is increased; determining at least two first connected regions in the first frame image, wherein the first connected regions comprise at least two adjacent pixel points belonging to the hard subtitles in the first frame image; screening a second connected region from the at least two first connected regions, wherein the second connected region is a first connected region of which the brightness of a corresponding region in the second frame image is not less than a brightness threshold; and determining a hard caption region based on the second connected region. According to the method and the device, the connected region which does not belong to the hard caption can be removed to obtain the second connected region, so that the hard caption region with relatively high accuracy can be determined based on the second connected region.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of video technology, and in particular to an image processing method, device, electronic device, and readable storage medium. Background Art

[0002] Many videos on the Internet include subtitles, including hard subtitles. Hard subtitles are also called embedded subtitles or embedded subtitles, which are subtitles that are embedded in a frame image and integrated with the frame image. In the field of video technology, it is necessary to determine the hard subtitle area from the frame image to optimize the hard subtitle area. Summary of the invention

[0003] The present application provides an image processing method, device, electronic device and readable storage medium, which can determine a hard subtitle area with high accuracy. The technical solution includes the following contents.

[0004] On the one hand, an image processing method is provided, the method comprising: acquiring a first frame image and a second frame image, the first frame image comprising a hard subtitle, and the second frame image being an image after the dynamic range of the first frame image is improved; determining at least two first connected areas in the first frame image, the first connected area comprising at least two adjacent pixel points in the first frame image that belong to the hard subtitle; screening a second connected area from the at least two first connected areas, the second connected area being a first connected area in which the brightness of the corresponding area in the second frame image is not less than a brightness threshold; and determining a hard subtitle area based on the second connected area.

[0005] On the other hand, an image processing device is provided, the device comprising: an acquisition module, used to acquire a first frame image and a second frame image, the first frame image comprising a hard subtitle, and the second frame image is an image after the dynamic range of the first frame image is improved; a determination module, used to determine at least two first connected areas in the first frame image, the first connected area comprising at least two adjacent pixels in the first frame image that belong to the hard subtitle; a screening module, used to screen a second connected area from the at least two first connected areas, the second connected area being a first connected area whose brightness of the corresponding area in the second frame image is not less than a brightness threshold; the determination module is further used to determine a hard subtitle area based on the second connected area.

[0006] In a possible implementation, the screening module is used to determine brightness data for the i-th first connected area based on the brightness values ​​of each pixel in the second frame image, and when the brightness data is not less than the brightness threshold, the i-th first connected area is used as the second connected area, where i is a positive integer, and the brightness data represents the maximum brightness value of the area corresponding to the i-th first connected area in the second frame image.

[0007] In a possible implementation, the determination module is further used to determine at least one of a first area, a second area or a third area, the first area is an image after adjusting the brightness of the hard subtitle area, the second area is an image after adjusting the color of the hard subtitle area, and the third area is an image after improving the dynamic range of the hard subtitle area; based on at least one of the first area, the second area or the third area, a fourth area is determined.

[0008] In a possible implementation, the determination module is used to determine a weight image, which includes the weights of each pixel in the hard subtitle area; and determine the fourth area based on at least one of the first area, the second area or the third area and the weight image.

[0009] In a possible implementation, the determination module is used to perform binarization on the hard subtitle area to obtain a binarized image, wherein the value of a pixel in the binarized image represents whether the brightness of the pixel is higher than a reference brightness; perform dilation on the pixel in the binarized image whose brightness is higher than the reference brightness to obtain a dilated image; and determine a weight image based on the dilated image.

[0010] In a possible implementation manner, the determination module is further configured to determine a reference area located around the hard subtitle area in the first frame image or the second frame image; and determine a reference brightness based on the reference area.

[0011] In a possible implementation, the determination module is used to crop the first frame image to obtain an area to be detected, where the area to be detected includes the hard subtitles; determine a detection result based on the area to be detected, where the detection result represents the possibility that each pixel point in the area to be detected belongs to the hard subtitles; and determine at least two first connected areas based on the detection result.

[0012] In a possible implementation, the pixel points in the area to be detected correspond to pixel values ​​of at least three channels; the determination module is used to preprocess the area to be detected to obtain a preprocessed area; and the detection result is determined based on the preprocessed area; wherein the preprocessed area includes the pixel values ​​of each channel corresponding to the first pixel point and the grayscale value of the second pixel point, the first pixel point is a pixel point in the area to be detected whose minimum pixel value is not less than a first threshold and whose maximum extreme value is not greater than a second threshold, the minimum pixel value is the minimum value among the pixel values ​​of the at least three channels, the maximum extreme value is the maximum value of the difference between the pixel values ​​of every two channels, the second pixel point includes the pixel points in the area to be detected except the first pixel point, and the grayscale value of the second pixel point is determined based on the pixel values ​​of each channel corresponding to the second pixel point.

[0013] In a possible implementation, there are at least three second connected areas; the determination module is used to determine the envelope frames of each second connected area; group the envelope frames of each second connected area to obtain at least one group, the group includes at least one envelope frame, and the positional relationship between two envelope frames belonging to the same group satisfies the position condition; and merge the envelope frames of each group to obtain a hard subtitle area.

[0014] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the electronic device implements any of the above-mentioned image processing methods.

[0015] On the other hand, a computer-readable storage medium is provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor so that an electronic device can implement any of the above-mentioned image processing methods.

[0016] On the other hand, a computer program is also provided. The computer program is at least one, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned image processing methods.

[0017] On the other hand, a computer program product is also provided, wherein at least one computer program is stored in the computer program product, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any one of the above-mentioned image processing methods.

[0018] The technical solution provided by this application brings at least the following beneficial effects: In the technical solution provided by the present application, at least two first connected areas are determined from the first frame image including hard subtitles, and the first connected area includes at least two adjacent pixels belonging to the hard subtitles. Since in the high dynamic range image, the brightness of the pixels belonging to the hard subtitles is significantly higher than the brightness of other pixels, and the dynamic range of the second frame image is higher than the dynamic range of the first frame image. Therefore, by screening the first connected area whose brightness of the corresponding area in the second frame image is not less than the brightness threshold from the at least two first connected areas, it is achieved to remove the connected area that does not belong to the hard subtitles and obtain the second connected area, thereby improving the accuracy of the hard subtitle area determined based on the second connected area. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 It is a structural diagram of a computer system provided in an embodiment of the present application.

[0021] Figure 2 It is a flow chart of detection and optimization of hard subtitle area provided in an embodiment of the present application.

[0022] Figure 3 It is a flowchart of an image processing method provided in an embodiment of the present application.

[0023] Figure 4 It is a detection schematic diagram of a hard subtitle area provided in an embodiment of the present application.

[0024] Figure 5 This is a schematic diagram of an optimization of a hard subtitle area provided in an embodiment of the present application.

[0025] Figure 6 It is a schematic diagram of a display interface provided in an embodiment of the present application.

[0026] Figure 7 It is a schematic diagram of detection and optimization of a hard subtitle area provided in an embodiment of the present application.

[0027] Figure 8 It is a structural schematic diagram of an image processing device provided in an embodiment of the present application.

[0028] Fig. 9 It is a structural diagram of a terminal device provided in an embodiment of the present application.

[0029] Fig.10It is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0031] It should be noted that the terms "first", "second", etc. in this application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0032] First, the abbreviations and key terms involved in the embodiments of the present application are defined.

[0033] Dynamic Range (DR): Also known as exposure range, it refers to the range of light intensity distribution from the darkest shadow to the brightest highlight in the picture. Simply put, dynamic range refers to the difference between the highest brightness and the lowest brightness in the image.

[0034] SDR (Standard Dynamic Range): A technology that compresses video to a smaller dynamic range. Typically, SDR video is 8-bit video with a dynamic range of 0 to 255.

[0035] HDR (High Dynamic Range): A processing technology that improves video brightness and contrast. Compared with SDR, HDR can provide a larger dynamic range. Typically, HDR video is 10-bit video with a dynamic range of 0 to 1023.

[0036] Hard subtitles: Also known as embedded subtitles or embedded subtitles, they are subtitles that are embedded in the video screen and integrated with the video screen, as opposed to soft subtitles. Among them, soft subtitles are also called plug-in subtitles. Generally, soft subtitles are saved separately. When the video is played, soft subtitles will be automatically rendered to the specified position of the screen.

[0037] like Figure 1 As shown, Figure 11 is a schematic diagram of the structure of a computer system provided in an embodiment of the present application, and the computer system includes: a terminal device 101 and a server 102. Among them, a client is installed and running in the terminal device 101, and the server 102 is used to provide background services for the client. The image processing method provided in the embodiment of the present application can be executed by the terminal device 101, or by the server 102, or by the terminal device 101 and the server 102 together, and the embodiment of the present application does not limit this.

[0038] In a possible implementation, the server 102 undertakes the main computing work, and the terminal device 101 undertakes the secondary computing work. Alternatively, the server 102 undertakes the secondary computing work, and the terminal device 101 undertakes the main computing work. Alternatively, the terminal device 101 and the server 102 use a distributed computing architecture to perform collaborative computing.

[0039] Optionally, the terminal device 101 may be any electronic device product that can interact with the user through one or more methods such as a keyboard, a touchpad, a remote control, voice interaction, or a handwriting device. For example, the terminal device 101 may be a mobile phone or a computer, wherein the mobile phone includes but is not limited to a smart phone, a folding phone, a slide phone, or a flip phone, and the computer includes but is not limited to a tablet computer, a laptop computer, a desktop computer, a PPC (Pocket PC) or a PC (Personal Computer). In actual application, the terminal device 101 may also include a smart speaker, a smart watch, a PDA (Personal Digital Assistant), a wearable device, a smart car machine, a smart TV, etc.

[0040] The terminal device 101 may generally refer to one of a plurality of terminal devices, and this embodiment is only illustrated by taking the terminal device 101 as an example. Those skilled in the art may know that the number of the terminal devices 101 may be more or less. For example, the terminal device 101 may be only one, or the terminal devices 101 may be dozens or hundreds, or more. The embodiment of the present application does not limit the number and device type of the terminal devices 101.

[0041] The server 102 is a single server, or a server cluster consisting of multiple servers, or any one of a cloud computing platform and a virtualization center, which is not limited in the embodiments of the present application. The server 102 is directly or indirectly connected to the terminal device 101 by means of a wired network or a wireless network. The server 102 has a data receiving function, a data processing function, and a data sending function. Of course, the server 102 may also have other functions, which are not limited in the embodiments of the present application.

[0042] Those skilled in the art should understand that the above-mentioned terminal device 101 and server 102 are only for illustration, and other existing or future terminal devices or servers, if applicable to the present application, should also be included in the scope of protection of the present application and are included here by reference.

[0043] In actual application, the brightness and color of the video can be optimized through the terminal device 101 or the server 102, and the optimized video can be displayed through the terminal device 101. When the video includes hard subtitles, usually, the brightness of the hard subtitles is higher than the screen content, resulting in a glaring visual effect of the hard subtitles in the optimized video. Based on this, it is necessary to detect the hard subtitle area from the frame image and optimize the hard subtitle area separately. Figure 2 As shown, the terminal device 101 or the server 102 can perform the following steps.

[0044] Step 1, detect the hard subtitle area. In the embodiment of the present application, the terminal device 101 or the server 102 can obtain the SDR frame image 201, and determine the detection result based on the SDR frame image 201, and the detection result represents the possibility that each pixel in the SDR frame image 201 belongs to a hard subtitle. If the possibility that the pixel belongs to a hard subtitle is higher than the possibility threshold, it is determined that the pixel belongs to a hard subtitle; if the possibility that the pixel belongs to a hard subtitle is lower than the possibility threshold, it is determined that the pixel does not belong to a hard subtitle. A first connected area can be determined based on adjacent pixels belonging to hard subtitles, and there are at least two first connected areas. Since the brightness of the hard subtitles in the HDR frame image 202 is significantly higher than the picture content, a second connected area whose brightness in the corresponding area in the HDR frame image 202 is not less than the brightness threshold can be screened out from each first connected area based on the HDR frame image 202, and a hard subtitle area 203 is determined based on the second connected area.

[0045] Step 2, optimize the hard subtitle area. In the embodiment of the present application, on the one hand, the terminal device 101 or the server 102 can determine the reference brightness based on the reference area around the hard subtitle area 203 in the SDR frame image 201, and adjust the brightness of the hard subtitle area 203 according to the reference brightness to obtain the first area 204. On the other hand, the terminal device 101 or the server 102 can adjust the color of the hard subtitle area 203 to obtain the second area 205. On the other hand, the terminal device 101 or the server 102 can crop the third area 206 corresponding to the hard subtitle area 203 from the HDR frame image 202. Afterwards, the first area 204, the second area 205 and the third area 206 are fused to obtain the fourth area 207. The fourth area 207 is the image area after the brightness and color of the hard subtitle area 203 are improved.

[0046] The present application embodiment provides an image processing method, which can be applied to the above-mentioned computer system and is executed by at least one of the terminal device 101 or the server 102. For ease of description, the terminal device 101 or the server 102 is collectively referred to as an electronic device, that is, the method of the present application embodiment is executed by the electronic device. Figure 3 As shown, Figure 3 It is a flowchart of an image processing method provided in an embodiment of the present application, and the method includes the following steps.

[0047] Step 301 , obtaining a first frame image and a second frame image, wherein the first frame image includes hard subtitles, and the second frame image is an image obtained by improving the dynamic range of the first frame image.

[0048] That is, the content of the second frame image is the same as that of the first frame image, but the dynamic range of the second frame image is higher than that of the first frame image. For example, the first frame image is an SDR image, and the second frame image is an HDR image with the same content as the first frame image.

[0049] The embodiments of the present application do not limit the method for acquiring the first frame image and the second frame image. For example, the electronic device can obtain the first frame image input by the user, and obtain the second frame image by performing an up-conversion on the first frame image. Among them, up-conversion is a technology that converts the format of an image from SDR to HDR, which usually involves remapping of brightness and contrast to make the color expression richer. Through up-conversion, the dynamic range and color depth can be increased, and the visual quality and realism of the image can be improved. Optionally, the first frame image is any frame image in the video.

[0050] The first frame image includes hard subtitles. Generally, the brightness of the hard subtitles in the first frame image is higher than the brightness of other screen contents. In the process of performing up-conversion on the first frame image including the hard subtitles, the brightness of the hard subtitles is likely to be significantly higher than the screen contents due to brightness stretching, causing the hard subtitles to have a glaring visual impact. In addition, during the up-conversion process, the color of the hard subtitles is easily affected by the color of the surrounding pixels, resulting in the presence of color noise in the hard subtitles. Based on this, the hard subtitle area can be determined from the first frame image so that during the up-conversion process, the hard subtitle area can be optimized separately to improve the visual effect of the hard subtitle area.

[0051] Step 302: determine at least two first connected regions in the first frame image, where the first connected region includes at least two adjacent pixels belonging to hard subtitles in the first frame image.

[0052] The electronic device can obtain a trained image detection model. The first frame image is input into the image detection model, and the image detection model outputs whether each pixel in the first frame image belongs to a hard subtitle. Among them, the image detection model can be implemented based on any one of the CRAFT (Character Region Awareness for Text Detection) algorithm, the YOLO (You Only Look Once) algorithm, and the DBNet++ (Differentiable Binarization Network++) algorithm. In addition, the embodiments of the present application do not limit the structure, calculation process, etc. of the image detection model.

[0053] Based on the above, the electronic device can obtain the detection result of the first frame image, and the detection result includes information on whether each pixel in the first frame image belongs to a hard subtitle. For example, the detection result includes the value of each pixel. If the value of the pixel is a first value, it indicates that the pixel belongs to a hard subtitle; if the value of the pixel is a second value, it indicates that the pixel does not belong to a hard subtitle. The first value and the second value are different values, for example, the first value is 1 and the second value is 0.

[0054] The electronic device can determine at least two pixels belonging to hard subtitles and adjacent to each other based on the detection result of the first frame image, and these pixels constitute a first connected area. That is, each pixel in the first connected area belongs to a hard subtitle, and these pixels are adjacent to each other. There are at least two first connected areas.

[0055] In a possible implementation, step 302 includes: cropping the first frame image to obtain an area to be detected, where the area to be detected includes hard subtitles; determining a detection result based on the area to be detected, where the detection result represents the possibility that each pixel point in the area to be detected belongs to a hard subtitle; and determining at least two first connected areas based on the detection result.

[0056] The embodiment of the present application does not limit the cropping method of the first frame image. For example, in general, the hard subtitles are located in the bottom area of ​​the frame image. Based on this, the electronic device can crop an area with a height of 30% of the height of the first frame image from the bottom of the first frame image to obtain the area to be detected. For another example, the electronic device can crop an area of ​​a predetermined area from the bottom of the first frame image to obtain the area to be detected. Optionally, use I roi Characterize the area to be inspected.

[0057] Next, the electronic device inputs the area to be detected into the image detection model, and outputs the detection result through the image detection model. Generally, the input of the image detection model needs to meet the size condition. Based on this, the electronic device can fill the pixels of the area to be detected to obtain the target area that meets the size condition, input the target area into the image detection model, and output the detection result through the image detection model.

[0058] For example, the width and height of the input of the image detection model are both integer multiples of 32. In response, the electronic device fills the right and bottom of the area to be detected with 0 pixels (i.e., pixels with a pixel value of 0) to obtain the target area, so that the width and height of the target area are both integer multiples of 32. I roi The width is w roi , Gao Wei h roi , then the width of the target area and high They satisfy the following formula (1) respectively.

[0059] Formula (1).

[0060] in, is a round-down operator. The above formula (1) indicates that the right side of the area to be detected is filled Column 0 pixels, padding at the bottom Line 0 pixels.

[0061] The detection result of the area to be detected is a confidence score map, including the score of each pixel in the area to be detected, and the score of the pixel represents the possibility that the pixel belongs to a hard subtitle. Optionally, the score of the pixel is a probability value ranging from 0 to 1, representing the probability that the pixel belongs to a hard subtitle. In this example, the higher the score of the pixel, the greater the possibility that the pixel belongs to a hard subtitle. It can be understood that the detection result of the target area is also a confidence score map. By removing the padded areas on the right and bottom of this confidence score map, the detection result of the area to be detected can be obtained.

[0062] The electronic device may obtain a confidence threshold, which may be a value preset based on manual experience or a randomly generated value. Based on the confidence threshold and the detection result of the area to be detected, it is determined whether each pixel in the area to be detected belongs to a hard subtitle. Optionally, the score of the pixel is I score , the confidence threshold is score ths .if I score > score ths, the electronic device determines that the pixel belongs to a hard subtitle; if I score < score ths , the electronic device determines that the pixel does not belong to a hard subtitle; if I score = score ths , the electronic device can determine that the pixel point belongs to a hard subtitle, or can determine that the pixel point does not belong to a hard subtitle.

[0063] For example, score ths If the pixel point belongs to a hard subtitle, the value of the pixel point is a first value, for example, the value of the pixel point is 1. If the pixel point does not belong to a hard subtitle, the value of the pixel point is a second value, for example, the value of the pixel point is 0. That is, the electronic device performs binarization on the detection result of the detection area according to the confidence threshold to obtain a binarized area, and the value of the pixel point in the binarized area is the first value or the second value. Afterwards, the first connected area is determined based on the binarized area, that is, the information on whether each pixel point belongs to a hard subtitle.

[0064] By cropping the first frame image, the area to be detected including the hard subtitles is obtained, thereby removing image content irrelevant to the hard subtitles, reducing the size of the area to be detected, and improving the efficiency of determining the detection result.

[0065] In an exemplary embodiment, pixel points in the area to be detected correspond to pixel values ​​of at least three channels; determining the detection result according to the area to be detected includes: preprocessing the area to be detected to obtain a preprocessed area; and determining the detection result based on the preprocessed area.

[0066] Among them, the preprocessed area includes the pixel values ​​of each channel corresponding to the first pixel point and the grayscale value of the second pixel point. The first pixel point is a pixel point whose minimum pixel value in the area to be detected is not less than the first threshold and whose maximum extreme value is not greater than the second threshold. The minimum pixel value is the minimum value among the pixel values ​​of at least three channels, and the maximum extreme value is the maximum value among the differences between the pixel values ​​of every two channels. The second pixel point includes the pixel points in the area to be detected except the first pixel point, and the grayscale value of the second pixel point is determined based on the pixel values ​​of each channel corresponding to the second pixel point.

[0067] The embodiments of the present application do not limit the channels corresponding to the pixel points. Exemplarily, the channels corresponding to the pixel points include at least three of the R (Red) channel, the G (Green) channel, the B (Blue) channel, the Y (Luminance or Luma) channel, the U (Chrominance or Chroma) channel, and the V (Chroma) channel.

[0068] Each pixel in the area to be detected corresponds to at least three channels. For each pixel, on the one hand, the minimum pixel value can be determined from the pixel values ​​of at least three channels corresponding to the pixel, and on the other hand, the difference between the pixel values ​​of every two channels can be determined based on the pixel values ​​of at least three channels corresponding to the pixel, and the maximum pixel value difference can be determined from the difference between the pixel values ​​of every two channels to obtain the maximum extreme value. If the minimum pixel value is not less than the first threshold and the maximum extreme value is not greater than the second threshold, then this pixel is used as the first pixel. If at least one of the following is satisfied: the minimum pixel value is less than the first threshold and the maximum extreme value is greater than the second threshold, then this pixel is used as the second pixel. Among them, the determination method of the first threshold and the second threshold is not limited here. For example, the first threshold and the second threshold can be set according to artificial experience, or the first threshold and the second threshold can be randomly generated.

[0069] Optionally, each pixel point in the area to be detected corresponds to an R channel, a G channel and a B channel. Then: the first pixel point is determined according to the following formula (2).

[0070] Formula (2).

[0071] in, Represents the minimum pixel value, Characterizes the maximum extreme value. represents the pixel value of the R channel corresponding to the pixel point, G represents the pixel value of the G channel corresponding to the pixel point, and B represents the pixel value of the B channel corresponding to the pixel point.

[0072] Since the color of hard subtitles is generally close to white, and the value of white in the R channel, G channel, and B channel is 255. Therefore, the first pixel point determined according to the above formula (2) is a potential pixel point belonging to the hard subtitle. For the first pixel point, the pixel values ​​of each channel corresponding to the first pixel point can be retained, and for the second pixel point, the grayscale value of the second pixel point can be determined based on the pixel values ​​of each channel corresponding to the second pixel point. For example, according to , determine the gray value of the second pixel .

[0073] Based on the pixel values ​​of each channel corresponding to each first pixel point and the grayscale value of each second pixel point, a preprocessed region is obtained. Afterwards, the preprocessed region is input into an image detection model, and a detection result is output by the image detection model to determine a first connected region based on the detection result.

[0074] From the above content, it can be seen that in the preprocessed area, the potential pixels belonging to hard subtitles correspond to the pixel values ​​of each channel, while the pixels not belonging to hard subtitles correspond to grayscale values. In this way, the non-hard subtitle pixels are grayed to highlight the pixels belonging to hard subtitles, reduce the complexity of the picture, and thus improve the accuracy of the detection results.

[0075] Step 303: Filter a second connected area from at least two first connected areas, where the second connected area is a first connected area whose brightness of a corresponding area in the second frame image is not less than a brightness threshold.

[0076] In this example, the content and size of the first frame image are the same as the content and size of the second frame image, and each pixel in the first frame image has a corresponding pixel in the second frame image. That is, assuming that the first frame image includes M×N pixels, the second frame image also includes M×N pixels, and the pixel in the i-th row and j-th column of the first frame image corresponds to the pixel in the i-th row and j-th column of the second frame image. Wherein, M and N are both positive integers, i is any positive integer from 1 to M, and j is any positive integer from 1 to N.

[0077] Based on this, for the first connected area in the first frame image, the pixel points corresponding to each pixel point in the first connected area can be determined in the second frame image, and the determined pixel points constitute the area corresponding to the first connected area in the second frame image. If the brightness of the area corresponding to the first connected area is not less than the brightness threshold, the electronic device uses the first connected area as the second connected area; if the brightness of the area corresponding to the first connected area is less than the brightness threshold, the electronic device filters out the first connected area. In this way, the electronic device can filter out part of the first connected area and select the second connected area.

[0078] In an exemplary embodiment, step 303 includes: for the i-th first connected area, based on the brightness value of each pixel in the second frame image, determining the brightness data, and when the brightness data is not less than the brightness threshold, taking the i-th first connected area as the second connected area, i is a positive integer, and the brightness data represents the maximum brightness of the area corresponding to the i-th first connected area in the second frame image.

[0079] The electronic device can obtain the brightness value of each pixel in the second frame image. Optionally, when the first frame image is cropped to obtain the area to be detected, the area corresponding to the area to be detected can be determined from the second frame image, and the brightness value of each pixel in the corresponding area can be obtained. The method for determining the corresponding area is not limited here. For example, the corresponding area can be cropped from the second frame image, or the corresponding area can be determined in the second frame image based on the position information of the area to be detected in the first frame image.

[0080] Optionally, the information of any pixel in the second frame image includes a brightness value. Based on this, the electronic device can directly obtain the brightness value of each pixel. Alternatively, the electronic device can calculate the brightness value based on the pixel value of any pixel in the second frame image.

[0081] For example, any pixel point in the second frame image corresponds to pixel values ​​of at least three channels. In this example, for any pixel point, the pixel values ​​of each channel can be normalized to obtain the normalized pixel values ​​of each channel, and the normalized pixel value of each channel is within the value range of 0 to 1.

[0082] Electronic devices can perform electro-optical conversion on the normalized pixel values ​​of each channel to obtain the converted pixel values ​​of each channel. Electro-optical conversion is the process of converting electrical signals into optical signals. Electro-optical conversion occurs on display devices and can convert digital signals into optical signals to show the brightness range and rich details of the video. Photoelectric conversion is the inverse transformation of electro-optical conversion, which is used to convert optical signals into electrical signals. Generally, cameras or sensors capture optical signals and convert them into electrical signals for subsequent processing and storage.

[0083] Optionally, follow , perform electro-optical conversion on the normalized pixel value of any channel to obtain the converted pixel value of the channel. Represents the pixel value after the channel conversion, which belongs to the linear domain. Represents the normalized pixel value of this channel, which belongs to the nonlinear domain. are all constant parameters, and their specific values ​​are not limited here. For example, .

[0084] Next, the electronic device calculates the brightness value of each pixel based on the converted pixel values ​​of each channel corresponding to each pixel. , calculate the brightness value of the pixel. Among them, The brightness value of a pixel is the value of the Y channel of the pixel in the YUV color space. Represents the pixel value after the pixel is converted in the R channel. Represents the pixel value after the pixel is converted in the G channel. Represents the pixel value after the pixel is converted in the B channel.

[0085] According to the above method, the electronic device can determine the brightness value of each pixel in the second frame image (or the area corresponding to the area to be detected in the second frame image). The electronic device can determine the area corresponding to the i-th first connected area from the second frame image, determine the maximum brightness value from the brightness values ​​of each pixel in the area, and obtain brightness data. If the brightness data is not less than the brightness threshold, the i-th first connected area is used as the second connected area; if the brightness data is less than the brightness threshold, the i-th first connected area is filtered out.

[0086] The embodiments of the present application do not limit the method for determining the brightness threshold. Exemplarily, the electronic device can randomly generate a brightness threshold, or the electronic device can obtain a brightness threshold input by a user. Taking the brightness threshold of 300 nits as an example, in this example, if the brightness data of the area corresponding to the i-th first connected area in the second frame image is not less than 300 nits, the i-th first connected area is used as the second connected area; if the brightness data of the area corresponding to the i-th first connected area in the second frame image is less than 300 nits, the i-th first connected area is filtered out.

[0087] By filtering the first connected area by the maximum brightness of the area corresponding to the first connected area in the second frame image, rather than filtering the first connected area by the brightness of the corresponding area, the amount of calculation can be reduced and the filtering efficiency can be improved. In addition, since the brightness of the pixel points belonging to the hard subtitle is higher than the brightness of other pixel points, by filtering out the first connected area whose maximum brightness is lower than the brightness threshold, the first connected area that does not belong to the hard subtitle is filtered, which is conducive to improving the accuracy of the hard subtitle area.

[0088] It should be noted that the above filtering is only exemplary, and in practical applications, there may be other filtering methods. For example, the first connected regions whose area is less than an area threshold (such as 10) may be filtered out, and the first connected regions whose area is greater than or equal to the area threshold may be used as the second connected regions.

[0089] Step 304: determine a hard subtitle area based on the second connected area.

[0090] If there is only one second connected area, the second connected area or the envelope surrounding it is used as the hard subtitle area. If there are at least two second connected areas, the envelope surrounding at least two second connected areas is used as the hard subtitle area, or the hard subtitle area is determined as follows.

[0091] In a possible implementation, there are at least three second connected areas. Step 304 includes: determining the envelopes of the second connected areas; grouping the envelopes of the second connected areas to obtain at least one group, wherein the group includes at least one envelope, and the positional relationship between two envelopes belonging to the same group satisfies the position condition; and merging the envelopes of the groups to obtain a hard subtitle area.

[0092] In this example, the envelope is the smallest polygon that surrounds the second connected area. Generally, the shape of the envelope is a rectangle. The electronic device includes a tool for determining the envelope. For example, the tool includes a findContours function and a minAreaRect function in the OpenCV library. Among them, the findContours function is a function for finding contours in a binary image, and the minAreaRect function is a function for calculating the minimum bounding rectangle of a point set. The minimum bounding rectangle of each second connected area is obtained by the tool, and the minimum bounding rectangle is the envelope.

[0093] The electronic device can obtain the position information of each envelope frame, and the position information includes but is not limited to at least one of the following: the coordinates of the center of the envelope frame, the coordinates of any vertex of the envelope frame, and the coordinates of the center of gravity of the envelope frame. The electronic device can group the envelope frames of each second connected area based on the position information of each envelope frame to obtain at least one group, and each group includes at least one envelope frame. If a group includes at least two envelope frames, the positional relationship between the two envelope frames satisfies the position condition. The content of the position condition is not limited here. Exemplarily, the position condition includes: the difference between the center coordinates of the two envelope frames is less than a threshold value.

[0094] After the groups are obtained, for any group, if the group includes only one envelope, the envelope is used as the hard subtitle area, and if the group includes at least two envelopes, the envelopes of the group are merged to obtain the hard subtitle area. The merging method is not limited here. For example, the vertex with the smallest coordinate and the vertex with the largest coordinate can be determined from the position information of the envelopes included in the group, and the merged envelope is determined based on these two vertices, and the envelope is the hard subtitle area.

[0095] The embodiment of the present application does not limit the specific implementation method of merging the envelope frame to obtain the hard subtitle area. For example, the envelope frame is a rectangular frame, and the electronic device can perform steps S1 to S9 as shown below to obtain the hard subtitle area. The details are as follows.

[0096] Step S1, initialize a horizontal rectangular box list and an inclined rectangular box list, wherein the horizontal rectangular box list is used to store horizontal rectangular boxes, and the inclined rectangular box list is used to store inclined rectangular boxes. Based on this, traverse all envelope boxes of the second connected area to determine whether each envelope box is a horizontal rectangular box or an inclined rectangular box. Optionally, for each envelope box, calculate the slope of at least one edge in the envelope box, based on the slopes of each edge and the slope threshold slope ths , determine whether the envelope is a horizontal rectangular frame or an inclined rectangular frame. Put all horizontal rectangular frames into the horizontal rectangular frame list, and put all inclined rectangular frames into the inclined rectangular frame list.

[0097] For example, when the slopes of the upper and lower sides of the envelope are both less than the slope threshold slope ths , the envelope is judged to be a horizontal rectangular frame. In this case, the envelope is placed in the horizontal rectangular frame list; when the slope of one of the upper and lower edges of the envelope is greater than the slope threshold slope ths When the envelope is determined to be an inclined rectangular frame, the envelope is placed in the inclined rectangular frame list. It should be noted that the value of the slope threshold can be flexibly set according to the actual application. For example, the slope threshold slope ths is 0.1.

[0098] Step S2: traverse all envelope frames in the horizontal rectangular frame list, and sort these envelope frames in the order of the vertical coordinates of the central pixels from small to large.

[0099] Step S3, initialize the current merge list 1 and the total merge list 1. Traverse the horizontal rectangular box list and obtain each envelope box in turn. Step S3 includes steps S3-1 to S3-3 as shown below. For the current envelope box, determine the list to which it belongs according to steps S3-1 to S3-3.

[0100] Step S3 - 1 : if the current merge list 1 is empty, that is, there is no envelope frame in the current merge list 1 , then put the current envelope frame into the current merge list 1 .

[0101] Step S3-2: If the current merge list 1 is not empty, that is, the current merge list 1 has at least one envelope frame and satisfies the first position condition: , put the current envelope into the current merge list 1. Represents the mean ordinate value of the center points of each envelope in the current merge list 1. Represents the average height of each envelope box in the current merge list 1, y center The vertical coordinate representing the center point of the current envelope. ths center It should be noted that the threshold can be flexibly set according to the actual application situation. ths center For example, ths center .

[0102] Step S3-3: If the current merge list 1 is not empty and does not meet the first position condition, each envelope frame in the current merge list 1 is taken as a group and put into the total merge list 1, the current merge list 1 is cleared, and the current envelope frame is put into the current merge list 1.

[0103] Through the above step S3, all horizontal rectangular frames are grouped according to the first position condition to obtain groups, each group includes at least one horizontal rectangular frame, and the position relationship between the envelope frames belonging to the same group meets the first position condition.

[0104] Step S4, initialize the current merge list 2 and the total merge list 2. Traverse the total merge list 1 and obtain each group in turn. For the current group, if the current group only includes one envelope frame, put the current group into the total merge list 2, skip steps S5 to S7, and directly execute step S8. If the current group includes at least two envelope frames, execute from step S5.

[0105] Step S5: in the case where the current group includes at least two envelope frames, traverse each envelope frame in the current group and sort each envelope frame in ascending order of the horizontal coordinate of the upper left corner of the envelope frame.

[0106] Step S6, traverse the current group and obtain each envelope frame in sequence. Step S6 includes steps S6-1 to S6-3 as shown below. For the current envelope frame, determine the list to which it belongs according to steps S6-1 to S6-3.

[0107] Step S6 - 1 : if the current merge list 2 is empty, that is, there is no envelope frame in the current merge list 2 , then put the current envelope frame into the current merge list 2 .

[0108] Step S6-2: If the current merge list 2 is not empty, that is, the current merge list 2 has at least one envelope frame and satisfies the second position condition: , put the current envelope into the current merge list 2. Among them, Represents the average height of each envelope box in the current merge list 2, height is the height of the current envelope, The minimum value of the horizontal coordinates of each vertex of the current envelope. The maximum value of the vertical coordinates of each vertex representing the current envelope. The minimum value of the ordinates of the vertices of the current envelope. is the maximum value of the horizontal coordinates of the vertices of the previous envelope frame of the current envelope frame in each sorted envelope frame, is the height threshold is the width threshold. It should be noted that the height threshold can be flexibly set according to the actual application situation. , width threshold For example, .

[0109] Step S6-3: If the current merge list 2 is not empty and does not meet the second position condition, each envelope frame in the current merge list 2 is taken as a group and put into the total merge list 2, the current merge list 2 is cleared, and the current envelope frame is put into the current merge list 2.

[0110] Through the above-mentioned step S6, for the group including at least two envelope frames obtained by the division through step S3, the envelope frames in the group are grouped again according to the second position condition to obtain individual groups, each group includes at least one horizontal rectangular frame, and the positional relationship between the envelope frames belonging to the same group satisfies the first position condition and the second position condition.

[0111] Step S7: if the current merge list 2 is not empty, each envelope frame in the current merge list 2 is taken as a group and put into the total merge list 2.

[0112] Step S8: Take each inclined envelope frame as a group and put it into the total merge list 2.

[0113] From then on, the envelope frames of all second connected areas are grouped to obtain various groups. Each group includes at least one envelope frame. Optionally, a group includes an inclined envelope frame, or a group includes a horizontal envelope frame, or a group includes at least two horizontal envelope frames that meet the first position condition and the second position condition.

[0114] Step S9, traverse the total merge list 2, and obtain each group in turn. For the current group, determine the minimum value from the horizontal coordinates of the vertices of each envelope box in the current group. and maximum value , and determine the minimum value from the ordinates of the vertices of each envelope in the current group and maximum value .according to , determine the coordinates of the upper left corner, according to , determine the coordinates of the lower right corner. According to the coordinates of the upper left corner, the coordinates of the lower right corner, and the heighty max -y min and width x max -x min , determine a hard subtitle area. Among them, The expansion factor can be flexibly set according to the actual application situation. For example, It is understandable that each group in the total merged list 2 can determine a hard subtitle area. That is, there is at least one hard subtitle area.

[0115] By grouping and merging the envelope frames of each second connected area, the envelope frames that meet the position conditions are merged into a hard subtitle area, which facilitates subsequent optimization of the hard subtitle area and improves the visual effect of the hard subtitle area.

[0116] The detection process of the hard subtitle area is as follows: Figure 4 As shown. In this example, the electronic device cuts out the area to be detected 401 from the bottom of the first frame image, and the area to be detected 401 includes hard subtitles. The area to be detected 401 is input into the image detection model, and the detection result 402 is output by the image detection model. The detection result 402 includes the probability that each pixel in the area to be detected 401 belongs to a hard subtitle. The detection result is binarized according to the confidence threshold to obtain a binarized area 403. The binarized area 403 represents whether each pixel in the area to be detected 401 belongs to a hard subtitle. The connected area can be determined based on the binarized area 403, the brightness threshold is determined based on the second frame image, and the connected area is filtered by the brightness threshold. The envelope box 404 of the filtered connected domain is determined, and these envelope boxes are merged to obtain the hard subtitle area 405.

[0117] In a possible implementation, step 304 further includes steps 305 to 306 (not shown in the figure).

[0118] Step 305, determining at least one of the first area, the second area or the third area, the first area is the image after adjusting the brightness of the hard subtitle area, the second area is the image after adjusting the color of the hard subtitle area, and the third area is the image after improving the dynamic range of the hard subtitle area.

[0119] The electronic device can adjust the brightness of the hard subtitle area to obtain the first area. In different application scenarios, the brightness adjustment method may be different. Exemplarily, the brightness adjustment method is as follows: First, binarization is performed on the hard subtitle area to obtain a binarized image. The method for determining the binarized image is described below and will not be repeated here. The binarized image includes multiple pixels, and each pixel in the binarized image corresponds one-to-one to each pixel in the hard subtitle area (or the third area). The value of any pixel in the binarized image represents whether the brightness of the corresponding pixel in the hard subtitle area (or the third area) is higher than the reference brightness.

[0120] Assuming that the pixels of interest are pixels with brightness higher than or equal to the reference brightness, and the pixels of non-interest are pixels with brightness lower than the reference brightness, the binary image includes the pixels of interest and the pixels of non-interest, and the brightness range of the corresponding pixels of the pixels of interest in the hard subtitle area (or the third area) is . Characterizes the reference brightness, Represents the maximum brightness of pixels in the subtitle area (or the third area).

[0121] Electronic devices can obtain the extreme brightness difference value The brightness extreme difference value represents the difference between the maximum brightness and the minimum brightness of the pixel of interest, and its unit is nit. The specific value is not limited here. For example, Based on the brightness extreme difference value, the brightness of the corresponding pixel of each pixel of interest in the hard subtitle area (or the third area) is adjusted to obtain the first area. It can be understood that after the brightness adjustment, the brightness range of the pixel of interest is .

[0122] Assume that the brightness value of the corresponding pixel of interest in the hard subtitle area (or the third area) is , then after brightness adjustment, the brightness value of the pixel of interest for: .in, Characterizing the reference brightness, the method for determining the reference brightness is described below and will not be repeated here. Represents the maximum value of the brightness values ​​of each pixel in the subtitle area (or the third area).

[0123] Since the brightness of the hard subtitle is higher than the screen content, and the brightness of the pixel of interest is higher than or equal to the reference brightness, adjusting the brightness value of the pixel of interest in the hard subtitle area is to adjust the brightness value of the pixel that potentially belongs to the hard subtitle. By adjusting the brightness of the hard subtitle to obtain the first area, and determining the fourth area based on the first area, the hard subtitle can be prevented from being too bright or too dark, thereby ensuring the visual effect of the hard subtitle.

[0124] Optionally, the reference brightness is determined based on a reference area around the hard subtitle area. By adjusting the brightness of the hard subtitles based on the reference brightness, the probability of the following two situations can be reduced. Situation 1: When the pixels around the hard subtitles are darker, the brightness of the hard subtitles is higher. Situation 2: When the pixels around the hard subtitles are brighter, the brightness of the hard subtitles is darker. In other words, after the brightness of the hard subtitle area is adjusted based on the reference brightness, the brightness of the hard subtitles is adapted to the brightness of the surrounding pixels, which can avoid a glaring visual effect.

[0125] During the image processing process, the hard subtitles may be affected by the colors of the surrounding pixels due to local operations, encoding, etc., resulting in the appearance of noise in the hard subtitles, affecting the visual experience of the hard subtitles. In this regard, the electronic device can adjust the color of the hard subtitle area to obtain the second area, so that after determining the fourth area based on the second area, the color effect of the hard subtitles can be guaranteed. In different application scenarios, the color adjustment method may be different.

[0126] Exemplarily, the hard subtitle area is converted to obtain a converted area to obtain the original subtitle color information. The electronic device obtains a color gamut conversion matrix, and converts the hard subtitle area to obtain a converted area using the color gamut conversion matrix.

[0127] Optionally, the color gamut conversion matrix is: After the color gamut conversion of the hard subtitle area is performed according to the color gamut conversion matrix, the converted area satisfies: .in, Characterize the transformed region, Represents the hard subtitle area, R, G, B are the pixel values ​​of the pixel in the R channel, G channel, and B channel respectively. The converted area is the second area, which retains the color and brightness information of the hard subtitle area.

[0128] In addition, the electronic device may crop the area corresponding to the hard subtitle area from the second frame image to obtain the third area. Alternatively, the electronic device may perform up-conversion on the hard subtitle area to obtain the third area. Since the dynamic range of the third area is greater than the dynamic range of the hard subtitle area, after determining the fourth area based on the third area, the dynamic range and color depth of the hard subtitles can be increased, thereby improving the visual effect of the hard subtitles.

[0129] Step 306: Determine a fourth area based on at least one of the first area, the second area, or the third area.

[0130] If the electronic device acquires any one of the first area, the second area or the third area, the electronic device uses the acquired area as the fourth area. If the electronic device acquires at least two of the first area, the second area or the third area, the electronic device merges the acquired areas to obtain the fourth area.

[0131] In an exemplary embodiment, step 306 includes: determining a weight image, the weight image including weights of each pixel in the hard subtitle area; and determining a fourth area based on at least one of the first area, the second area, or the third area and the weight image.

[0132] The size of the weighted image is consistent with the size of the hard subtitle area. For each pixel in the hard subtitle area, there is a corresponding pixel in the weighted image. The value of any pixel on the weighted image represents the weight of the corresponding pixel in the hard subtitle area. The manner in which the electronic device obtains the weighted image is not limited herein. For example, the electronic device can obtain the weighted image input by the user. Alternatively, determining the weighted image includes: performing binarization on the hard subtitle area to obtain a binarized image, the value of the pixel in the binarized image representing whether the brightness of the pixel is higher than a reference brightness; performing dilation on the pixel in the binarized image whose brightness is higher than the reference brightness to obtain an expanded image; and determining the weighted image based on the expanded image.

[0133] In practical applications, the reference brightness can be determined based on manual experience or based on the area around the hard subtitle area. For example, before binarizing the hard subtitle area to obtain a binarized image, the method further includes: determining a reference area around the hard subtitle area in the first frame image or the second frame image; and determining the reference brightness based on the reference area.

[0134] In this example, the first frame image includes a hard subtitle area, and the electronic device can determine a reference area located around the hard subtitle area in the first frame image. The second frame image includes a corresponding area of ​​the hard subtitle area, and the electronic device can determine a reference area located around the corresponding area in the second frame image, and the reference area can also be regarded as an area around the hard subtitle area.

[0135] Assume the hard subtitle area is R sub , the upper left corner coordinates of the hard subtitle area are ( x l , y l ), the coordinates of the lower right corner are ( x r , y r The electronic device may expand the boundary box of the hard subtitle area outward, in the height direction, by H pixels upward and downward, and in the width direction, by W pixels left and right, to obtain a rectangular areaR expand The coordinates of the upper left corner of the rectangular area are , the coordinates of the lower right corner are The values ​​of H and W are not limited here. For example, if the resolution of the first frame image is 1080P (i.e., the size is 1920×1080), then H=W=50. R expand With hard subtitle area R sub The difference is R expand Internal and R sub The outer overlapping area is used as a reference area, and the reference area is a rectangular ring. It can be understood that if there are at least two hard subtitle areas, then for each hard subtitle area, the corresponding reference area can be determined in the above manner.

[0136] Electronic device obtains subtitle brightness lower limit The lower limit of brightness represents the minimum value of the reference brightness, and its unit is nit. The value of is not limited here, for example, For each reference area, obtain the brightness value of each pixel in the reference area, and take the 98th percentile of the brightness value (that is, 98% of the pixel brightness is less than this value) as the candidate brightness to eliminate the influence of noise. If the candidate brightness is less than the lower limit of the subtitle brightness , the alternative brightness is set to If the alternative brightness is greater than or equal to the lower limit of the subtitle brightness , then keep the candidate brightness unchanged. Traverse the candidate brightness of each reference area and take the maximum value as the reference brightness .

[0137] Next, based on the reference brightness Binarization is performed on the hard subtitle area to obtain a binary image. In this example, the hard subtitle area is obtained. R sub The brightness value of each pixel in the For hard subtitle area R sub Perform binarization to obtain a binary image. For any pixel, if the brightness of the pixel is higher than the reference brightness, the value of the pixel is 1, indicating that the pixel is a pixel of interest; if the brightness of the pixel is equal to or lower than the reference brightness, the value of the pixel is 0, indicating that the pixel is a pixel of no interest. The binary image includes each pixel of interest and each pixel of no interest. The binary image can be recorded as M subIt is understandable that the brightness of hard subtitles is significantly higher than that of the picture content, and the vast majority of the pixels of interest belong to hard subtitles, while a small number of pixels belong to non-hard subtitles.

[0138] Afterwards, the dilate(·) function in the OpenCV library is used to dilate the pixels in the binary image whose brightness is higher than the reference brightness (i.e., the pixels of interest) to obtain a dilated image. Optionally, for each pixel of interest in the binary image, the pixel of interest is dilated outward by a number of pixels (e.g., 31) to obtain a dilated pixel. These dilated pixels include the pixels of interest and can form a dilated region. The dilated image includes the dilated region and the region outside the dilated region, the dilated region includes the dilated pixels, and the region outside the dilated region includes the pixels of non-interest.

[0139] For each expanded pixel in the expanded area, calculate the shortest distance between the expanded pixel and the pixel of interest d , and d Perform normalization to get the normalized distance . The value range is 0~1, which can represent the weight of the expanded pixel. In other words, the weight of the pixel in the expanded area w fusion Set to , the weight of the pixel outside the expansion area w fusion Set to 1. From this we can get the weight image.

[0140] The reference brightness is determined by the reference area around the hard subtitle area, and the hard subtitle area is binarized based on the reference brightness, so that the area around the hard subtitle is referenced to determine the potential pixel points belonging to the hard subtitle, thereby improving the accuracy of the binarized image. The pixel points that potentially belong to the hard subtitle in the binarized image are expanded, and a weight image is determined based on the expanded image, so as to fuse at least one of the first area, the second area and the third area based on the weight image to obtain a fourth area. The fourth area is determined by the weight image, so that the brightness change, color change and dynamic range change of the hard subtitle and the surrounding area are smoothed, and the visual effect of the fourth area is improved.

[0141] It is understandable that in different scenarios, the fusion methods of each region are also different. For example, the fourth region can be obtained by fusion of at least one of the first region, the second region, and the third region and the hard subtitle region based on the weight image. Alternatively, the first region and the third region can be fused first, and then the fused region is fused with the second region. Alternatively, the first region and the second region can be fused first, and then the fused region is fused with the third region. A possible fusion method is shown below.

[0142] In this example, the first region and the third region are first fused based on the weight image to obtain a fused region. HDR impbr = w fusion × HDR + (1- w fusion )× HDR br , determine the fusion area. Among them, HDR impbr Characterize the fusion region, w fusion Representation weight image, HDR Characterize the third region, HDR br Characterize the first region.

[0143] Next, the second region and the fused region are further fused based on the weight image to obtain a fourth region. Optionally, since the second region is obtained by color gamut conversion of the hard subtitle region, its dynamic range is consistent with the dynamic range of the first frame image, which is much lower than the dynamic range of the second frame image. HDR impbr The dynamic range of the second frame image is consistent with the dynamic range of the second frame image. HDR fmt Brightness stretching is performed so that the dynamic range of the stretched area matches the dynamic range of the second frame image.

[0144] The brightness stretching process is as follows: Get the second area HDR fmt The brightness value of each pixel in Y fmt , and get the fusion area HDR impbr The brightness value of each pixel in Y impbr . Based on the brightness value of any pixel Y fmt and brightness value Y impbr , calculate the brightness improvement ratio of the pixel. Optionally, the brightness improvement ratio of the pixel ratio satisfy: .in, is a hyperparameter that prevents the divisor from being zero, for example, Next, a Gaussian kernel (whose size is not limited, for example, the size is 5) is used to average the pixels in the second area to smooth the brightness of the pixels belonging to the hard subtitles. Then, based on the brightness enhancement ratio of each pixel ratio , the second area after the average processing The brightness of each pixel in the second area after the improvement satisfy: .

[0145] As mentioned above, when calculating the brightness of a pixel, it is necessary to first convert the pixel value of the pixel from the nonlinear domain to the linear domain, and determine the brightness value of the pixel based on the converted pixel value. In other words, the brightness adjustment is performed in the linear domain, so the electronic device can adjust the brightness of the second area after the improvement. Perform photoelectric conversion to obtain a second region after photoelectric conversion. , the linear domain Transformed into the nonlinear domain .

[0146] The electronic device may further fuse the second region after photoelectric conversion and the fused region based on the weight image to obtain a fourth region. , determine the fourth area. Among them, HDR impcol Characterizes the fourth region, w fusion Representation weight image, HDR impbr Characterize the fusion region, The region after brightness stretching is performed on the second region is represented.

[0147] In an exemplary embodiment, the optimization process of the hard subtitle area is as follows Figure 5 As shown. In this example, the hard subtitle area is binarized to obtain a binarized image 501, and the value of any pixel in the binarized image 501 represents whether the brightness of the pixel is higher than the reference brightness. The pixel points in the binarized image 501 whose brightness is higher than the reference brightness are dilated to obtain a dilated image 502, and the weight image 503 is determined based on the dilated image 502. For the pixel points in the binarized image 501 whose brightness is higher than the reference brightness, the brightness of the corresponding pixel points of the pixel point in the hard subtitle area 504 is adjusted to obtain a first area 505. The hard subtitle area 504 and the first area 505 are fused based on the weight image 503 to obtain a hard subtitle area 506 with optimized brightness, which can be the fourth area mentioned above.

[0148] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant regions. For example, the first frame image, the second frame image, etc. involved in this application are all obtained with full authorization.

[0149] In the above method, at least two first connected areas are determined from the first frame image including hard subtitles, and the first connected area includes at least two adjacent pixels belonging to the hard subtitles. Since in the high dynamic range image, the brightness of the pixels belonging to the hard subtitles is significantly higher than the brightness of other pixels, and the dynamic range of the second frame image is higher than the dynamic range of the first frame image. Therefore, by selecting the first connected area whose brightness of the corresponding area in the second frame image is not less than the brightness threshold from the at least two first connected areas, it is achieved to remove the connected area that does not belong to the hard subtitles and obtain the second connected area, thereby improving the accuracy of the hard subtitle area determined based on the second connected area.

[0150] The above describes the image processing method of the embodiment of the present application from the perspective of method steps. The following is a systematic and comprehensive description. The method of the embodiment of the present application can be applied in the video playback scenario. Generally, the electronic device is equipped with video playback software, and the video is played through the video playback software. During the video playback process, the user can switch the video quality at any time. Figure 6 As shown, Figure 6 The display interface of a video playback software is shown, and the display interface includes a frame image 601 of the video being played and a plurality of image quality switching controls 602. It can be understood that different video playback software may correspond to different types or the same type of image quality switching controls. Figure 6 The image quality switching controls shown are for exemplary purposes only and can be flexibly set in actual applications.

[0151] Assume that when the user selects the image quality switching control 604, the video playback software plays the SDR video, and when the user selects the image quality switching control 603, the video playback software plays the HDR video. Based on this, if the user selects the image quality switching control 603 while the video playback software is playing the SDR video, in this case, on the one hand, the electronic device identifies the hard subtitle area from the SDR video based on the image processing method of the embodiment of the present application, and optimizes the hard subtitle area, and on the other hand, the electronic device converts the picture content of the SDR video except the hard subtitle area into the picture content of the HDR video. The electronic device displays the converted picture content and the optimized hard subtitle area. Processing the video in this way can make the processed video close to the real light and shadow effects, light and dark contrast, color tone and retina (i.e., real human eye) level of particle clarity, improve the video quality, and thus improve the user's visual experience.

[0152] Optionally, the electronic device is configured as follows: Figure 7 The process shown in the figure identifies the hard subtitle area from the SDR video and optimizes the hard subtitle area. Specifically, the process includes the following contents.

[0153] Step 701: Identify the hard subtitle area.

[0154] In this example, the electronic device obtains the SDR video being played. When the user selects the image quality switching control 603, the electronic device can obtain the HDR video obtained by manually processing the SDR video, or process the SDR video through a conversion algorithm to generate an HDR video. Among them, the content of the SDR video and the HDR video is the same, but the dynamic range of the two is different. If the SDR video includes hard subtitles, the electronic device decodes the SDR video into each SDR frame image, and decodes each HDR video into each HDR frame image, so as to identify and optimize the hard subtitle area in the frame image frame by frame based on each SDR frame image and each HDR frame image. For ease of description, the implementation process of step 701 and step 702 is described below, taking an SDR frame image and a corresponding HDR frame image as an example. Among them, the SDR frame image is the first frame image mentioned above, and the HDR frame image is the second frame image mentioned above.

[0155] In this example, on the one hand, the SDR frame image is cropped to obtain an area to be detected including hard subtitles, the area to be detected is preprocessed to obtain a preprocessed area, and a connected domain is detected based on the preprocessed area, wherein the connected domain is the first connected domain mentioned above. On the other hand, the connected domain is filtered according to the HDR frame image corresponding to the SDR frame image, so as to remove the connected domains whose brightness in the corresponding area in the HDR frame image is less than the brightness threshold, and retain the connected domains whose brightness in the corresponding area in the HDR frame image is not less than the brightness threshold. Among them, the retained connected domain is the second connected domain mentioned above. The electronic device determines the envelope boxes of each retained connected domain, and obtains the hard subtitle area by merging these envelope boxes. As can be seen from the foregoing, the hard subtitle area is the image area where the hard subtitles are located in the SDR frame image.

[0156] Step 702: Optimize the hard subtitle area.

[0157] In this example, the hard subtitle area is a rectangular area in the SDR frame image, and the electronic device can obtain the size and position of the rectangular area. Based on the size and position of the rectangular area, a reference area is determined in the HDR frame image, and the reference area is a rectangular ring surrounding the rectangular area. The electronic device can determine the reference brightness based on the reference area. Binarization is performed on the hard subtitle area based on the reference brightness to obtain a binary image, and the value of any pixel in the binary image represents whether the brightness of the pixel is higher than the reference brightness. Dilation is performed on the pixels in the binary image whose brightness is higher than the reference brightness to obtain a dilated image, and a weight image is determined based on the dilated image. In addition, each pixel in the binary image corresponds to each pixel in the hard subtitle area one by one. For the pixel in the binary image whose brightness is higher than the reference brightness, the brightness of the corresponding pixel in the hard subtitle area is adjusted. In this way, the brightness of the hard subtitle area is adjusted to obtain the first area. The corresponding area of ​​the hard subtitle area is cropped from the HDR frame image to obtain the third area. The first area and the third area are fused based on the weight image to obtain the hard subtitle area with optimized brightness. Afterwards, the SDR frame image is color-adjusted, and the area corresponding to the hard subtitle area is cropped from the color-adjusted SDR frame image to obtain the second area. Alternatively, the hard subtitle area is directly color-adjusted to obtain the second area. The second area and the hard subtitle area after brightness optimization are fused based on the weight image to obtain the hard subtitle area after color optimization. Among them, the hard subtitle area after color optimization is the fourth area mentioned above. From then on, the optimization of the hard subtitle area is completed. Afterwards, the electronic device obtains and outputs the optimized HDR frame image based on the hard subtitle area after color optimization and the picture content in the HDR frame image except the hard subtitle area. The electronic device displays the optimized HDR frame image.

[0158] It can be understood that after the user selects the image quality switching control 603, the electronic device can process each SDR frame image in the SDR video according to the implementation principles of steps 701 to 702, obtain and display the corresponding optimized HDR frame image, and these HDR frame images can form an HDR video.

[0159] Figure 8 FIG. 1 is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. Figure 8 As shown, the device includes the following contents.

[0160] The acquisition module 801 is used to acquire a first frame image and a second frame image, wherein the first frame image includes hard subtitles, and the second frame image is an image obtained by improving the dynamic range of the first frame image.

[0161] The determination module 802 is configured to determine at least two first connected regions in the first frame image, where the first connected regions include at least two adjacent pixels belonging to hard subtitles in the first frame image.

[0162] The screening module 803 is used to screen a second connected area from at least two first connected areas, where the second connected area is a first connected area whose brightness of the corresponding area in the second frame image is not less than a brightness threshold.

[0163] The determination module 802 is further configured to determine a hard subtitle area based on the second connected area.

[0164] In one possible implementation, the screening module 803 is used to determine brightness data for the i-th first connected area based on the brightness values ​​of each pixel in the second frame image, and when the brightness data is not less than a brightness threshold, the i-th first connected area is used as the second connected area, where i is a positive integer and the brightness data represents the maximum brightness value of the area corresponding to the i-th first connected area in the second frame image.

[0165] In one possible implementation, the determination module 802 is further used to determine at least one of the first area, the second area or the third area, the first area is the image after adjusting the brightness of the hard subtitle area, the second area is the image after adjusting the color of the hard subtitle area, and the third area is the image after improving the dynamic range of the hard subtitle area; based on at least one of the first area, the second area or the third area, the fourth area is determined.

[0166] In a possible implementation, the determination module 802 is used to determine a weight image, where the weight image includes the weights of each pixel in the hard subtitle area; and determine the fourth area based on at least one of the first area, the second area, or the third area and the weight image.

[0167] In a possible implementation, the determination module 802 is used to perform binarization on the hard subtitle area to obtain a binarized image, where the value of a pixel in the binarized image represents whether the brightness of the pixel is higher than a reference brightness; perform dilation on the pixel whose brightness in the binarized image is higher than the reference brightness to obtain a dilated image; and determine a weight image based on the dilated image.

[0168] In a possible implementation, the determination module 802 is further configured to determine a reference area around the hard subtitle area in the first frame image or the second frame image; and determine a reference brightness based on the reference area.

[0169] In a possible implementation, the determination module 802 is used to crop the first frame image to obtain an area to be detected, where the area to be detected includes hard subtitles; determine a detection result based on the area to be detected, where the detection result represents the possibility that each pixel point in the area to be detected belongs to a hard subtitle; and determine at least two first connected areas based on the detection result.

[0170] In one possible implementation, the pixel points in the area to be detected correspond to pixel values ​​of at least three channels; a determination module 802 is used to preprocess the area to be detected to obtain a preprocessed area; based on the preprocessed area, a detection result is determined; wherein the preprocessed area includes the pixel values ​​of each channel corresponding to the first pixel point and the grayscale value of the second pixel point, the first pixel point is a pixel point in the area to be detected whose minimum pixel value is not less than a first threshold and whose maximum extreme value is not greater than a second threshold, the minimum pixel value is the minimum value among the pixel values ​​of at least three channels, the maximum extreme value is the maximum value among the differences between the pixel values ​​of every two channels, the second pixel point includes the pixel points in the area to be detected except the first pixel point, and the grayscale value of the second pixel point is determined based on the pixel values ​​of each channel corresponding to the second pixel point.

[0171] In a possible implementation, there are at least three second connected areas; a determination module 802 is used to determine the envelope frames of each second connected area; the envelope frames of each second connected area are grouped to obtain at least one group, each group includes at least one envelope frame, and the positional relationship between two envelope frames belonging to the same group satisfies the position condition; and the envelope frames of each group are merged to obtain a hard subtitle area.

[0172] In the above device, at least two first connected areas are determined from the first frame image including hard subtitles, and the first connected area includes at least two adjacent pixels belonging to the hard subtitles. Since in the high dynamic range image, the brightness of the pixels belonging to the hard subtitles is significantly higher than the brightness of other pixels, and the dynamic range of the second frame image is higher than the dynamic range of the first frame image. Therefore, by selecting the first connected area whose brightness of the corresponding area in the second frame image is not less than the brightness threshold from the at least two first connected areas, it is achieved to remove the connected area that does not belong to the hard subtitles and obtain the second connected area, thereby improving the accuracy of the hard subtitle area determined based on the second connected area.

[0173] It should be understood that the above Figure 8 When the device provided realizes its functions, only the division of the above-mentioned functional modules is used as an example for illustration. In practical applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0174] Fig. 9 The structure block diagram of a terminal device 900 provided by an exemplary embodiment of the present application is shown. The terminal device 900 includes: a processor 901 and a memory 902 .

[0175] The processor 901 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 901 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 901 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 901 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 901 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0176] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 902 is used to store at least one computer program, which is used to be executed by the processor 901 to implement the image processing method provided in the method embodiment of the present application.

[0177] In some embodiments, the terminal device 900 may further optionally include: a peripheral device interface 903 and at least one peripheral device. The processor 901, the memory 902 and the peripheral device interface 903 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 903 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 904, a display screen 905, a camera assembly 906, an audio circuit 907 and a power supply 908.

[0178] The peripheral device interface 903 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 901 and the memory 902. In some embodiments, the processor 901, the memory 902, and the peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 901, the memory 902, and the peripheral device interface 903 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0179] The radio frequency circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 904 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 904 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 904 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 904 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 904 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0180] The display screen 905 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 905 is a touch display screen, the display screen 905 also has the ability to collect touch signals on the surface or above the surface of the display screen 905. The touch signal can be input to the processor 901 as a control signal for processing. At this time, the display screen 905 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 905 can be one, which is set on the front panel of the terminal device 900; in other embodiments, the display screen 905 can be at least two, which are respectively set on different surfaces of the terminal device 900 or are folded; in other embodiments, the display screen 905 can be a flexible display screen, which is set on the curved surface or folded surface of the terminal device 900. Even, the display screen 905 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 905 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0181] The camera assembly 906 is used to capture images or videos. Optionally, the camera assembly 906 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal, and the rear camera is disposed on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 906 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0182] The audio circuit 907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 901 for processing, or input them into the radio frequency circuit 904 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal device 900. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 907 may also include a headphone jack.

[0183] The power supply 908 is used to power various components in the terminal device 900. The power supply 908 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 908 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery charged through a wired line, and a wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0184] In some embodiments, the terminal device 900 further includes one or more sensors 909 . The one or more sensors 909 include but are not limited to: an acceleration sensor 911 , a gyroscope sensor 912 , a pressure sensor 913 , an optical sensor 914 , and a proximity sensor 915 .

[0185] The acceleration sensor 911 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal device 900. For example, the acceleration sensor 911 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 901 can control the display screen 905 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 911. The acceleration sensor 911 can also be used for collecting game or user motion data.

[0186] The gyro sensor 912 can detect the body direction and rotation angle of the terminal device 900, and the gyro sensor 912 can cooperate with the acceleration sensor 911 to collect the user's 3D actions on the terminal device 900. The processor 901 can implement the following functions based on the data collected by the gyro sensor 912: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0187] The pressure sensor 913 can be set on the side frame of the terminal device 900 and / or the lower layer of the display screen 905. When the pressure sensor 913 is set on the side frame of the terminal device 900, it can detect the user's holding signal of the terminal device 900, and the processor 901 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 913. When the pressure sensor 913 is set on the lower layer of the display screen 905, the processor 901 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 905. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0188] The optical sensor 914 is used to collect the ambient light intensity. In one embodiment, the processor 901 can control the display brightness of the display screen 905 according to the ambient light intensity collected by the optical sensor 914. Specifically, when the ambient light intensity is high, the display brightness of the display screen 905 is increased; when the ambient light intensity is low, the display brightness of the display screen 905 is reduced. In another embodiment, the processor 901 can also dynamically adjust the shooting parameters of the camera assembly 906 according to the ambient light intensity collected by the optical sensor 914.

[0189] The proximity sensor 915, also called a distance sensor, is usually arranged on the front panel of the terminal device 900. The proximity sensor 915 is used to collect the distance between the user and the front of the terminal device 900. In one embodiment, when the proximity sensor 915 detects that the distance between the user and the front of the terminal device 900 is gradually decreasing, the processor 901 controls the display screen 905 to switch from the screen-on state to the screen-off state; when the proximity sensor 915 detects that the distance between the user and the front of the terminal device 900 is gradually increasing, the processor 901 controls the display screen 905 to switch from the screen-off state to the screen-on state.

[0190] Those skilled in the art will understand that Fig. 9 The structure shown in the figure does not constitute a limitation on the terminal device 900, and may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0191] Fig.10This is a schematic diagram of the structure of the server provided in the embodiment of the present application. The server 1000 may have relatively large differences due to different configurations or performances, and may include one or more processors 1001 and one or more memories 1002, wherein the one or more memories 1002 store at least one computer program, and the at least one computer program is loaded and executed by the one or more processors 1001 to implement the image processing methods provided in the above-mentioned various method embodiments. Exemplarily, the processor 1001 is a CPU. Of course, the server 1000 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server 1000 may also include other components for implementing device functions, which will not be described in detail here.

[0192] In an exemplary embodiment, a computer-readable storage medium is further provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned image processing methods.

[0193] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0194] In an exemplary embodiment, a computer program is further provided. The computer program is at least one, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any one of the above-mentioned image processing methods.

[0195] In an exemplary embodiment, a computer program product is further provided. The computer program product stores at least one computer program, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any one of the above-mentioned image processing methods.

[0196] It should be understood that the "plurality" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0197] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0198] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a first frame image and a second frame image, wherein the first frame image includes hard subtitles, and the second frame image is an image obtained by improving the dynamic range of the first frame image; Determine at least two first connected regions in the first frame of image, where the first connected regions include at least two adjacent pixels in the first frame of image that belong to the hard subtitles; Selecting a second connected area from the at least two first connected areas, where the second connected area is a first connected area where the brightness of the corresponding area in the second frame image is not less than a brightness threshold; Based on the second connected area, a hard subtitle area is determined.

2. The method according to claim 1, characterized in that The step of selecting a second connected area from the at least two first connected areas comprises: For the i-th first connected area, brightness data is determined based on the brightness values ​​of each pixel in the second frame image. When the brightness data is not less than the brightness threshold, the i-th first connected area is used as the second connected area, where i is a positive integer and the brightness data represents the maximum brightness value of the area corresponding to the i-th first connected area in the second frame image.

3. The method according to claim 1, characterized in that The method further comprises: Determine at least one of a first area, a second area, or a third area, wherein the first area is an image after adjusting the brightness of the hard subtitle area, the second area is an image after adjusting the color of the hard subtitle area, and the third area is an image after improving the dynamic range of the hard subtitle area; A fourth area is determined based on at least one of the first area, the second area, or the third area.

4. The method according to claim 3, characterized in that The determining of the fourth area based on at least one of the first area, the second area or the third area comprises: Determining a weight image, wherein the weight image includes the weight of each pixel in the hard subtitle area; A fourth region is determined based on at least one of the first region, the second region, or the third region and the weight image.

5. The method according to claim 4, characterized in that The step of determining a weight image comprises: Binarization is performed on the hard subtitle area to obtain a binary image, wherein the value of a pixel in the binary image represents whether the brightness of the pixel is higher than a reference brightness; Performing dilation on pixel points in the binary image whose brightness is higher than the reference brightness to obtain a dilated image; Based on the dilated image, a weight image is determined.

6. The method according to claim 5, characterized in that Before performing binarization on the hard subtitle area to obtain a binarized image, the method further includes: Determining a reference area around the hard subtitle area in the first frame image or the second frame image; A reference brightness is determined based on the reference area.

7. The method according to any one of claims 1 to 6, characterized in that: The determining of at least two first connected areas in the first frame image includes: Cropping the first frame image to obtain a region to be detected, where the region to be detected includes the hard subtitles; Determine a detection result according to the area to be detected, wherein the detection result represents the possibility that each pixel point in the area to be detected belongs to the hard subtitle; Based on the detection result, at least two first connected areas are determined.

8. The method according to claim 7, characterized in that The pixel points in the area to be detected correspond to pixel values ​​of at least three channels; The determining of the detection result according to the area to be detected includes: Preprocessing the area to be detected to obtain a preprocessed area; Determining a detection result based on the preprocessed area; Among them, the preprocessed area includes pixel values ​​of each channel corresponding to a first pixel point and a grayscale value of a second pixel point, the first pixel point is a pixel point whose minimum pixel value in the area to be detected is not less than a first threshold and whose maximum extreme value is not greater than a second threshold, the minimum pixel value is the minimum value among the pixel values ​​of the at least three channels, the maximum extreme value is the maximum value of the difference between the pixel values ​​of every two channels, the second pixel point includes pixel points in the area to be detected except the first pixel point, and the grayscale value of the second pixel point is determined based on the pixel values ​​of each channel corresponding to the second pixel point.

9. The method according to any one of claims 1 to 6, characterized in that: The number of the second connected areas is at least three; and determining the hard subtitle area based on the second connected areas includes: Determine the envelope of each second connected area; Grouping the envelope frames of the second connected regions to obtain at least one group, wherein the group includes at least one envelope frame, and the positional relationship between two envelope frames belonging to the same group satisfies a position condition; Merge the envelopes of each group to get the hard subtitle area.

10. An image processing device, characterized in that: The device comprises: An acquisition module, used for acquiring a first frame image and a second frame image, wherein the first frame image includes hard subtitles, and the second frame image is an image obtained by improving the dynamic range of the first frame image; A determination module, configured to determine at least two first connected regions in the first frame of image, wherein the first connected regions include at least two adjacent pixels in the first frame of image that belong to the hard subtitles; A screening module, configured to screen a second connected area from the at least two first connected areas, wherein the second connected area is a first connected area whose brightness of a corresponding area in the second frame image is not less than a brightness threshold; The determination module is further used to determine a hard subtitle area based on the second connected area.

11. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the image processing method as claimed in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor to enable the electronic device to implement the image processing method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that the electronic device implements the image processing method according to any one of claims 1 to 9.

Images