Image processing method, device, electronic device and readable storage medium

Abstract

The present application discloses an image processing method, apparatus, electronic device, and readable storage medium, belonging to the field of video technology. The method includes: obtaining a first frame image and a second frame image, wherein the first frame image includes hard subtitles, and the second frame image is an image after the dynamic range of the first frame image is increased; determining at least two first connected regions in the first frame image, wherein the first connected region includes at least two adjacent pixels in the first frame image that belong to hard subtitles; selecting a second connected region from the at least two first connected regions, wherein the second connected region is a first connected region in which the brightness of the corresponding region in the second frame image is not less than a brightness threshold; and determining the hard subtitle region based on the second connected region. The present application can remove connected regions that do not belong to hard subtitles to obtain a second connected region, so that the hard subtitle region can be determined with higher accuracy based on the second connected region.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of video technology, and in particular to an image processing method, device, electronic device, and readable storage medium. Background Art

[0002] Many videos on the internet include subtitles, including hard subtitles. Hard subtitles, also known as embedded subtitles or inline subtitles, are subtitles that are embedded within the frame image and blend seamlessly into it. In the field of video technology, it's necessary to identify and optimize the hard subtitle area within the frame image. Summary of the Invention

[0003] The present application provides an image processing method, apparatus, electronic device, and readable storage medium, which can determine hard subtitle areas with high accuracy. The technical solution includes the following contents.

[0004] On the one hand, an image processing method is provided, the method comprising: acquiring a first frame image and a second frame image, the first frame image including hard subtitles, and the second frame image being an image after the dynamic range of the first frame image is improved; determining at least two first connected areas in the first frame image, the first connected area including at least two adjacent pixels in the first frame image belonging to the hard subtitles; screening a second connected area from the at least two first connected areas, the second connected area being a first connected area in which the brightness of the corresponding area in the second frame image is not less than a brightness threshold; and determining a hard subtitle area based on the second connected area.

[0005] On the other hand, an image processing device is provided, comprising: an acquisition module for acquiring a first frame image and a second frame image, wherein the first frame image includes a hard subtitle, and the second frame image is an image after the dynamic range of the first frame image is improved; a determination module for determining at least two first connected areas in the first frame image, wherein the first connected area includes at least two adjacent pixels in the first frame image that belong to the hard subtitle; a screening module for screening a second connected area from the at least two first connected areas, wherein the second connected area is a first connected area in which the brightness of the corresponding area in the second frame image is not less than a brightness threshold; and the determination module is further configured to determine a hard subtitle area based on the second connected area.

[0006] In one possible implementation, the screening module is used to determine brightness data for the i-th first connected area based on the brightness values ​​of each pixel in the second frame image, and when the brightness data is not less than the brightness threshold, the i-th first connected area is used as the second connected area, where i is a positive integer, and the brightness data represents the maximum brightness of the area corresponding to the i-th first connected area in the second frame image.

[0007] In one possible implementation, the determination module is further used to determine at least one of a first area, a second area, or a third area, where the first area is an image after adjusting the brightness of the hard subtitle area, the second area is an image after adjusting the color of the hard subtitle area, and the third area is an image after improving the dynamic range of the hard subtitle area; and a fourth area is determined based on at least one of the first area, the second area, or the third area.

[0008] In one possible implementation, the determination module is used to determine a weight image, where the weight image includes the weights of each pixel in the hard subtitle area; and determine the fourth area based on at least one of the first area, the second area, or the third area and the weight image.

[0009] In one possible implementation, the determination module is configured to perform binarization on the hard subtitle area to obtain a binarized image, wherein the value of a pixel in the binarized image represents whether the brightness of the pixel is higher than a reference brightness; perform dilation on the pixels in the binarized image whose brightness is higher than the reference brightness to obtain a dilated image; and determine a weighted image based on the dilated image.

[0010] In a possible implementation, the determination module is further configured to determine a reference area located around the hard subtitle area in the first frame image or the second frame image; and determine a reference brightness based on the reference area.

[0011] In one possible implementation, the determination module is configured to crop the first frame image to obtain an area to be detected, where the area to be detected includes the hard subtitles; determine a detection result based on the area to be detected, where the detection result represents the likelihood that each pixel in the area to be detected belongs to the hard subtitles; and determine at least two first connected areas based on the detection result.

[0012] In one possible implementation, the pixel points in the area to be detected correspond to pixel values ​​of at least three channels; the determination module is used to preprocess the area to be detected to obtain a preprocessed area; and the detection result is determined based on the preprocessed area; wherein the preprocessed area includes the pixel values ​​of each channel corresponding to the first pixel point and the grayscale value of the second pixel point, the first pixel point is the pixel point in the area to be detected whose minimum pixel value is not less than the first threshold and whose maximum extreme value is not greater than the second threshold, the minimum pixel value is the minimum value among the pixel values ​​of the at least three channels, the maximum extreme value is the maximum value of the difference between the pixel values ​​of each two channels, the second pixel point includes the pixel points in the area to be detected other than the first pixel point, and the grayscale value of the second pixel point is determined based on the pixel values ​​of each channel corresponding to the second pixel point.

[0013] In one possible implementation, there are at least three second connected areas; the determination module is used to determine the envelope frames of each second connected area; group the envelope frames of each second connected area to obtain at least one group, each group including at least one envelope frame, and the positional relationship between two envelope frames belonging to the same group meets a position condition; and merge the envelope frames of each group to obtain a hard subtitle area.

[0014] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the electronic device implements any of the above-mentioned image processing methods.

[0015] On the other hand, a computer-readable storage medium is provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned image processing methods.

[0016] On the other hand, a computer program is further provided. The computer program is at least one, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned image processing methods.

[0017] On the other hand, a computer program product is further provided. The computer program product stores at least one computer program, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned image processing methods.

[0018] The technical solution provided by this application brings at least the following beneficial effects:

[0019] In the technical solution provided by this application, at least two first connected regions are determined from a first frame image containing hard subtitles. The first connected regions include at least two adjacent pixels belonging to the hard subtitles. Because the brightness of pixels belonging to hard subtitles in a high dynamic range image is significantly higher than that of other pixels, and the dynamic range of the second frame image is higher than that of the first frame image, by selecting from the at least two first connected regions those first connected regions whose corresponding regions in the second frame image have a brightness not less than a brightness threshold, connected regions that do not belong to hard subtitles are removed, resulting in second connected regions, thereby improving the accuracy of hard subtitle regions determined based on the second connected regions. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] Figure 1 It is a structural diagram of a computer system provided in an embodiment of the present application.

[0022] Figure 2 This is a flow chart for detecting and optimizing a hard subtitle area provided in an embodiment of the present application.

[0023] Figure 3 This is a flowchart of an image processing method provided in an embodiment of the present application.

[0024] Figure 4 This is a schematic diagram of detecting a hard subtitle area provided in an embodiment of the present application.

[0025] Figure 5 This is a schematic diagram of an optimization of a hard subtitle area provided in an embodiment of the present application.

[0026] Figure 6 This is a schematic diagram of a display interface provided in an embodiment of the present application.

[0027] Figure 7 This is a schematic diagram of detection and optimization of a hard subtitle area provided in an embodiment of the present application.

[0028] Figure 8 It is a structural diagram of an image processing device provided in an embodiment of the present application.

[0029] Figure 9 This is a structural diagram of a terminal device provided in an embodiment of the present application.

[0030] Figure 10This is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0032] It should be noted that the terms "first," "second," and the like in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0033] First, the abbreviations and key terms involved in the embodiments of the present application are defined.

[0034] Dynamic Range (DR): Also known as exposure range, it refers to the range of light intensity distribution from the darkest shadows to the brightest highlights in the image. Simply put, dynamic range refers to the difference between the highest and lowest brightness levels in an image.

[0035] SDR (Standard Dynamic Range): A technology that compresses video to a smaller dynamic range. Typically, SDR video is 8-bit video with a dynamic range of 0 to 255.

[0036] HDR (High Dynamic Range): A processing technology that improves video brightness and contrast. Compared to SDR, HDR provides a wider dynamic range. Typically, HDR video is 10-bit video with a dynamic range of 0 to 1023.

[0037] Hard subtitles, also known as embedded subtitles or built-in subtitles, are subtitles that are embedded in the video and blend seamlessly into the video. They contrast with soft subtitles. Soft subtitles, also known as external subtitles, are typically stored separately and automatically rendered to a designated location on the screen during video playback.

[0038] like Figure 1 As shown, Figure 11 is a schematic diagram of the structure of a computer system provided in an embodiment of the present application. The computer system includes: a terminal device 101 and a server 102. The terminal device 101 has a client installed and running, and the server 102 is used to provide background services for the client. The image processing method provided in the embodiment of the present application can be executed by the terminal device 101, the server 102, or both the terminal device 101 and the server 102, and this embodiment of the present application is not limited thereto.

[0039] In one possible implementation, server 102 performs primary computing tasks, and terminal device 101 performs secondary computing tasks. Alternatively, server 102 performs secondary computing tasks, and terminal device 101 performs primary computing tasks. Alternatively, terminal device 101 and server 102 utilize a distributed computing architecture for collaborative computing.

[0040] Optionally, the terminal device 101 may be any electronic device capable of human-computer interaction with a user via one or more methods, such as a keyboard, touchpad, remote control, voice interaction, or handwriting device. For example, the terminal device 101 may be a mobile phone or a computer. Mobile phones include, but are not limited to, smartphones, foldable phones, slider phones, or flip phones; computers include, but are not limited to, tablet computers, laptops, desktop computers, PPCs (Pocket PCs), or PCs (Personal Computers). In practical applications, the terminal device 101 may also include smart speakers, smart watches, PDAs (Personal Digital Assistants), wearable devices, smart car systems, smart televisions, and the like.

[0041] Terminal device 101 may generally refer to one of multiple terminal devices. This embodiment uses terminal device 101 as an example. Those skilled in the art will appreciate that the number of terminal devices 101 may be greater or lesser. For example, there may be only one terminal device 101, or there may be dozens, hundreds, or even more terminal devices 101. This embodiment of the application does not limit the number or type of terminal devices 101.

[0042] The server 102 is a single server, or a server cluster consisting of multiple servers, or any one of a cloud computing platform and a virtualization center, which is not limited in the embodiments of the present application. The server 102 is directly or indirectly connected to the terminal device 101 via a wired network or a wireless network. The server 102 has a data receiving function, a data processing function, and a data sending function. Of course, the server 102 may also have other functions, which are not limited in the embodiments of the present application.

[0043] Those skilled in the art should understand that the above-mentioned terminal device 101 and server 102 are merely examples, and other existing or future terminal devices or servers, if applicable to the present application, should also be included in the scope of protection of the present application and are incorporated herein by reference.

[0044] In actual application, the brightness and color of the video can be optimized through the terminal device 101 or the server 102, and the optimized video can be displayed through the terminal device 101. When the video includes hard subtitles, the brightness of the hard subtitles is usually higher than the screen content, resulting in a glaring visual effect of the hard subtitles in the optimized video. Based on this, it is necessary to detect the hard subtitle area from the frame image and optimize the hard subtitle area separately. Figure 2 As shown, the terminal device 101 or the server 102 can perform the following steps.

[0045] Step 1: Detect the hard subtitle area. In an embodiment of the present application, the terminal device 101 or the server 102 can obtain an SDR frame image 201 and determine a detection result based on the SDR frame image 201. The detection result represents the likelihood that each pixel in the SDR frame image 201 belongs to a hard subtitle. If the likelihood that the pixel belongs to a hard subtitle is higher than a likelihood threshold, the pixel is determined to belong to a hard subtitle; if the likelihood that the pixel belongs to a hard subtitle is lower than the likelihood threshold, the pixel is determined not to belong to a hard subtitle. A first connected region can be determined based on adjacent pixels belonging to hard subtitles, and there are at least two first connected regions. Since the brightness of hard subtitles in the HDR frame image 202 is significantly higher than that of the image content, a second connected region whose brightness in the corresponding region in the HDR frame image 202 is not less than a brightness threshold can be screened from each first connected region based on the HDR frame image 202, and a hard subtitle region 203 is determined based on the second connected region.

[0046] Step 2: Optimize the hard subtitle area. In this embodiment of the present application, on one hand, the terminal device 101 or server 102 can determine a reference brightness based on a reference area surrounding the hard subtitle area 203 in the SDR frame image 201, and adjust the brightness of the hard subtitle area 203 according to the reference brightness to obtain a first area 204. On the other hand, the terminal device 101 or server 102 can adjust the color of the hard subtitle area 203 to obtain a second area 205. On another hand, the terminal device 101 or server 102 can crop a third area 206 corresponding to the hard subtitle area 203 from the HDR frame image 202. The first area 204, the second area 205, and the third area 206 are then fused to obtain a fourth area 207. The fourth area 207 is the image area after improving the brightness and color of the hard subtitle area 203.

[0047] The embodiment of the present application provides an image processing method, which can be applied to the above-mentioned computer system and executed by at least one of the terminal device 101 or the server 102. For the convenience of description, the terminal device 101 or the server 102 is collectively referred to as an electronic device, that is, the method of the embodiment of the present application is executed by the electronic device. Figure 3 As shown, Figure 3 This is a flowchart of an image processing method provided in an embodiment of the present application, which includes the following steps.

[0048] Step 301: Acquire a first frame image and a second frame image, wherein the first frame image includes hard subtitles, and the second frame image is an image obtained by increasing the dynamic range of the first frame image.

[0049] That is, the content of the second frame image is the same as the content of the first frame image, but the dynamic range of the second frame image is higher than the dynamic range of the first frame image. For example, the first frame image is an SDR image, and the second frame image is an HDR image with the same content as the first frame image.

[0050] The embodiments of the present application do not limit the method for obtaining the first frame image and the second frame image. For example, the electronic device can obtain the first frame image input by the user, and obtain the second frame image by performing up-conversion on the first frame image. Among them, up-conversion is a technology that converts the format of an image from SDR to HDR, which usually involves remapping of brightness and contrast to make the color expression richer. Through up-conversion, the dynamic range and color depth can be increased, and the visual quality and realism of the image can be improved. Optionally, the first frame image is any frame image in the video.

[0051] The first frame includes hard subtitles. Generally, the brightness of the hard subtitles in the first frame is higher than that of other screen content. During up-conversion of the first frame including the hard subtitles, brightness stretching can easily cause the brightness of the hard subtitles to be significantly higher than the screen content, resulting in a glaring visual effect. Furthermore, during the up-conversion process, the color of the hard subtitles is easily affected by the color of surrounding pixels, resulting in color noise in the hard subtitles. Based on this, the hard subtitle area can be determined from the first frame so that the hard subtitle area can be optimized separately during the up-conversion process to improve the visual effect of the hard subtitle area.

[0052] Step 302: Determine at least two first connected regions in the first frame image, where the first connected regions include at least two adjacent pixels belonging to hard subtitles in the first frame image.

[0053] The electronic device can obtain a trained image detection model. The first frame image is input into the image detection model, and the image detection model outputs whether each pixel in the first frame image belongs to a hard subtitle. The image detection model can be implemented based on any one of the CRAFT (Character Region Awareness for Text Detection) algorithm, the YOLO (You Only Look Once) algorithm, and the DBNet++ (Differentiable Binarization Network++) algorithm. In addition, the embodiments of the present application do not limit the structure, calculation process, etc. of the image detection model.

[0054] Based on the above, the electronic device can obtain a detection result for the first frame of image. The detection result includes information about whether each pixel in the first frame of image belongs to a hard subtitle. For example, the detection result includes the value of each pixel. If the pixel value is a first value, it indicates that the pixel belongs to a hard subtitle; if the pixel value is a second value, it indicates that the pixel does not belong to a hard subtitle. The first value and the second value are different values, for example, the first value is 1 and the second value is 0.

[0055] Based on the detection result of the first frame image, the electronic device may determine at least two adjacent pixels belonging to hard subtitles, where these pixels constitute a first connected region. In other words, each pixel in the first connected region belongs to a hard subtitle, and these pixels are adjacent to each other. There are at least two first connected regions.

[0056] In one possible implementation, step 302 includes: cropping the first frame image to obtain an area to be detected, where the area to be detected includes hard subtitles; determining a detection result based on the area to be detected, where the detection result represents the possibility that each pixel point in the area to be detected belongs to a hard subtitle; and determining at least two first connected areas based on the detection result.

[0057] The embodiment of the present application does not limit the cropping method of the first frame image. For example, in general, the hard subtitles are located in the bottom area of ​​the frame image. Based on this, the electronic device can crop an area with a height of 30% of the height of the first frame image from the bottom of the first frame image to obtain the area to be detected. For another example, the electronic device can crop an area of ​​a predetermined area from the bottom of the first frame image to obtain the area to be detected. Optionally, use I roi Characterize the area to be inspected.

[0058] Next, the electronic device inputs the area to be inspected into the image detection model, which then outputs the detection results. Generally, the input to the image detection model must meet size requirements. Based on this, the electronic device can pad the area to be inspected with pixels to obtain a target area that meets the size requirements. This target area is then input into the image detection model, which then outputs the detection results.

[0059] For example, the width and height of the input of the image detection model are both integer multiples of 32. In response, the electronic device fills the right and bottom of the area to be detected with 0 pixels (i.e., pixels with a pixel value of 0) to obtain the target area, so that the width and height of the target area are both integer multiples of 32. I roi The width is w roi , Gao Wei h roi , the width of the target area and high They satisfy the following formula (1).

[0060] Formula (1).

[0061] in, The above formula (1) indicates that the right side of the area to be detected is filled with Column 0 pixels, padding at the bottom Row 0 pixels.

[0062] The detection result of the area to be detected is a confidence score map, including the score of each pixel in the area to be detected. The score of the pixel represents the possibility that the pixel belongs to hard subtitles. Optionally, the score of the pixel is a probability value ranging from 0 to 1, representing the probability that the pixel belongs to hard subtitles. In this example, the higher the score of the pixel, the greater the possibility that the pixel belongs to hard subtitles. It can be understood that the detection result of the target area is also a confidence score map. By removing the padded areas on the right and bottom of this confidence score map, the detection result of the area to be detected can be obtained.

[0063] The electronic device can obtain a confidence threshold, which can be a value preset based on manual experience or a randomly generated value. Based on the confidence threshold and the detection result of the area to be detected, it is determined whether each pixel in the area to be detected belongs to a hard subtitle. Optionally, the score of the pixel is I score , the confidence threshold is score ths .if I score > score ths, the electronic device determines that the pixel belongs to a hard subtitle; if I score < score ths , the electronic device determines that the pixel does not belong to a hard subtitle; if I score = score ths , the electronic device can determine that the pixel point belongs to a hard subtitle, or can determine that the pixel point does not belong to a hard subtitle.

[0064] For example, score ths If the pixel belongs to a hard subtitle, the value of the pixel is a first value, for example, the value of the pixel is 1. If the pixel does not belong to a hard subtitle, the value of the pixel is a second value, for example, the value of the pixel is 0. That is, the electronic device performs binarization on the detection result of the to-be-detected area according to the confidence threshold to obtain a binarized area, and the values ​​of the pixels in the binarized area are the first value or the second value. Then, based on the binarized area, that is, the information on whether each pixel belongs to a hard subtitle, a first connected area is determined.

[0065] By cropping the first frame of image, the area to be detected including the hard subtitles is obtained, thereby removing image content irrelevant to the hard subtitles, reducing the size of the area to be detected, and improving the efficiency of determining the detection result.

[0066] In an exemplary embodiment, pixel points in the area to be detected correspond to pixel values ​​of at least three channels; determining the detection result according to the area to be detected includes: preprocessing the area to be detected to obtain a preprocessed area; and determining the detection result based on the preprocessed area.

[0067] Among them, the preprocessed area includes the pixel values ​​of each channel corresponding to the first pixel point and the grayscale value of the second pixel point. The first pixel point is the pixel point whose minimum pixel value in the area to be detected is not less than the first threshold and the maximum extreme value is not greater than the second threshold. The minimum pixel value is the minimum value among the pixel values ​​of at least three channels, and the maximum extreme value is the maximum value of the difference between the pixel values ​​of each two channels. The second pixel point includes the pixel points in the area to be detected except the first pixel point, and the grayscale value of the second pixel point is determined based on the pixel values ​​of each channel corresponding to the second pixel point.

[0068] This embodiment of the application does not limit the channels corresponding to the pixels. For example, the channels corresponding to the pixels include at least three of the following: the R (Red) channel, the G (Green) channel, the B (Blue) channel, the Y (Luminance or Luma) channel, the U (Chrominance or Chroma) channel, and the V (Chroma) channel.

[0069] Each pixel point in the area to be detected corresponds to at least three channels. For each pixel point, on the one hand, the minimum pixel value can be determined from the pixel values ​​of at least three channels corresponding to the pixel point. On the other hand, the difference between the pixel values ​​of each two channels can be determined based on the pixel values ​​of at least three channels corresponding to the pixel point, and the maximum pixel value difference can be determined from the difference between the pixel values ​​of each two channels to obtain the maximum extreme value. If the minimum pixel value is not less than the first threshold and the maximum extreme value is not greater than the second threshold, then this pixel point is used as the first pixel point. If at least one of the following items is met: the minimum pixel value is less than the first threshold and the maximum extreme value is greater than the second threshold, then this pixel point is used as the second pixel point. The method of determining the first threshold and the second threshold is not limited here. For example, the first threshold and the second threshold can be set according to manual experience, or the first threshold and the second threshold can be randomly generated.

[0070] Optionally, each pixel point in the area to be detected corresponds to the R channel, the G channel, and the B channel. Then: the first pixel point is determined according to the following formula (2).

[0071] Formula (2).

[0072] in, Represents the minimum pixel value, Represents the maximum extreme value. represents the pixel value of the R channel corresponding to the pixel point, G represents the pixel value of the G channel corresponding to the pixel point, and B represents the pixel value of the B channel corresponding to the pixel point.

[0073] Since the color of hard subtitles is generally close to white, and the value of white in the R channel, G channel, and B channel is 255. Therefore, the first pixel point determined according to the above formula (2) is a potential pixel point belonging to the hard subtitle. For the first pixel point, the pixel values ​​of each channel corresponding to the first pixel point can be retained, and for the second pixel point, the grayscale value of the second pixel point can be determined based on the pixel values ​​of each channel corresponding to the second pixel point. For example, according to , determine the gray value of the second pixel .

[0074] A preprocessed region is obtained based on the pixel values ​​of each channel corresponding to each first pixel and the grayscale value of each second pixel. The preprocessed region is then input into an image detection model, and a detection result is output by the image detection model to determine a first connected region based on the detection result.

[0075] As can be seen above, in the preprocessed area, pixels potentially belonging to hard subtitles are mapped to pixel values ​​across all channels, while pixels not belonging to hard subtitles are mapped to grayscale values. This method grayscales non-hard subtitle pixels, highlighting those belonging to hard subtitles, reducing image complexity and improving detection accuracy.

[0076] Step 303: Filter a second connected area from the at least two first connected areas, where the second connected area is a first connected area whose brightness of the corresponding area in the second frame image is not less than a brightness threshold.

[0077] In this example, the content and size of the first image frame are identical to those of the second image frame, and each pixel in the first image frame has a corresponding pixel in the second image frame. That is, assuming the first image frame includes M×N pixels, the second image frame also includes M×N pixels, and the pixel in the i-th row and j-th column of the first image frame corresponds to the pixel in the i-th row and j-th column of the second image frame. Where M and N are both positive integers, i is any positive integer from 1 to M, and j is any positive integer from 1 to N.

[0078] Based on this, for the first connected area in the first frame image, the pixel points corresponding to each pixel point in the first connected area can be determined in the second frame image. The determined pixel points form the area corresponding to the first connected area in the second frame image. If the brightness of the area corresponding to the first connected area is not less than a brightness threshold, the electronic device will treat the first connected area as the second connected area. If the brightness of the area corresponding to the first connected area is less than the brightness threshold, the electronic device will filter out the first connected area. In this way, the electronic device can filter out part of the first connected area and select the second connected area.

[0079] In an exemplary embodiment, step 303 includes: for the i-th first connected area, determining brightness data based on the brightness values ​​of each pixel point in the second frame image, and when the brightness data is not less than the brightness threshold, taking the i-th first connected area as the second connected area, i is a positive integer, and the brightness data represents the maximum brightness of the area corresponding to the i-th first connected area in the second frame image.

[0080] The electronic device can obtain the brightness value of each pixel in the second frame image. Optionally, when the area to be detected is obtained by cropping the first frame image, the area corresponding to the area to be detected can be determined from the second frame image, and the brightness value of each pixel in the corresponding area can be obtained. The method for determining the corresponding area is not limited here. For example, the corresponding area can be obtained by cropping from the second frame image, or the corresponding area can be determined in the second frame image based on the position information of the area to be detected in the first frame image.

[0081] Optionally, the information of any pixel in the second frame image includes a brightness value. Based on this, the electronic device can directly obtain the brightness value of each pixel. Alternatively, the electronic device can calculate the brightness value based on the pixel value of any pixel in the second frame image.

[0082] For example, any pixel in the second frame image corresponds to pixel values ​​of at least three channels. In this example, for any pixel, the pixel values ​​of each channel can be normalized to obtain the normalized pixel values ​​of each channel, and the normalized pixel values ​​of each channel are within the numerical range of 0 to 1.

[0083] Electronic devices can perform electro-optical conversion on the normalized pixel values ​​of each channel to obtain the converted pixel values ​​for each channel. Electro-optical conversion is the process of converting electrical signals into optical signals. Electro-optical conversion occurs on the display device and can convert digital signals into optical signals to display the brightness range and rich details of the video. Photoelectric conversion is the inverse transformation of electro-optical conversion, which converts optical signals into electrical signals. Generally, cameras or sensors capture optical signals and convert them into electrical signals for subsequent processing and storage.

[0084] Optionally, follow , perform electro-optical conversion on the normalized pixel value of any channel to obtain the converted pixel value of the channel. Represents the pixel value after the channel conversion, which belongs to the linear domain. Represents the normalized pixel value of this channel, which belongs to the nonlinear domain. are all constant parameters, and their specific values ​​are not limited here. For example, .

[0085] Next, the electronic device calculates the brightness value of each pixel based on the pixel values ​​converted from each channel. , calculate the brightness value of the pixel. Among them, The brightness value of a pixel is the value of the Y channel in the YUV color space. Represents the pixel value after the pixel is converted in the R channel. Represents the pixel value after the pixel is converted in the G channel, Represents the pixel value after the pixel is converted in the B channel.

[0086] In this manner, the electronic device can determine the brightness value of each pixel in the second image frame (or the area corresponding to the area to be detected in the second image frame). The electronic device can determine the area corresponding to the i-th first connected region from the second image frame, and determine the maximum brightness value from the brightness values ​​of the pixels in this area to obtain brightness data. If the brightness data is not less than a brightness threshold, the i-th first connected region is treated as the second connected region; if the brightness data is less than the brightness threshold, the i-th first connected region is filtered out.

[0087] The embodiments of the present application do not limit the method for determining the brightness threshold. For example, the electronic device can randomly generate a brightness threshold, or the electronic device can obtain a brightness threshold input by the user. Taking the brightness threshold of 300 nits as an example, in this example, if the brightness data of the area corresponding to the i-th first connected area in the second frame image is not less than 300 nits, the i-th first connected area is used as the second connected area; if the brightness data of the area corresponding to the i-th first connected area in the second frame image is less than 300 nits, the i-th first connected area is filtered out.

[0088] Filtering the first connected region based on the maximum brightness of the region corresponding to the first connected region in the second frame image, rather than filtering based on the brightness of the corresponding region, reduces computational complexity and improves filtering efficiency. Furthermore, because pixels belonging to hard subtitles have higher brightness than other pixels, filtering out first connected regions whose maximum brightness falls below a brightness threshold eliminates first connected regions that do not belong to hard subtitles, improving accuracy in hard subtitle regions.

[0089] It should be noted that the above filtering is merely exemplary. In actual applications, other filtering methods are possible. For example, first connected regions with an area smaller than an area threshold (e.g., 10) can be filtered out, and first connected regions with an area greater than or equal to the area threshold can be used as second connected regions.

[0090] Step 304: Determine a hard subtitle area based on the second connected area.

[0091] If there is only one second connected area, the second connected area or the envelope surrounding it is used as the hard subtitle area. If there are at least two second connected areas, the envelope surrounding the at least two second connected areas is used as the hard subtitle area. Alternatively, the hard subtitle area is determined as follows.

[0092] In one possible implementation, there are at least three second connected regions. Step 304 includes: determining the envelopes of each second connected region; grouping the envelopes of each second connected region to obtain at least one group, each group including at least one envelope, wherein the positional relationship between two envelopes belonging to the same group satisfies a position condition; and merging the envelopes of each group to obtain a hard subtitle region.

[0093] In this example, the envelope is the smallest polygon that encloses the second connected area. Generally, the shape of the envelope is a rectangle. The electronic device includes a tool for determining the envelope. For example, the tool includes the findContours function and the minAreaRect function in the OpenCV library. The findContours function is a function for finding contours in a binary image, and the minAreaRect function is a function for calculating the minimum bounding rectangle of a point set. The minimum bounding rectangle of each second connected area is obtained through the tool. The minimum bounding rectangle is the envelope.

[0094] The electronic device can obtain the location information of each envelope frame, which includes but is not limited to at least one of the following: the coordinates of the center of the envelope frame, the coordinates of any vertex of the envelope frame, and the coordinates of the center of gravity of the envelope frame. Based on the location information of each envelope frame, the electronic device can group the envelope frames of each second connected area to obtain at least one group, each group including at least one envelope frame. If a group includes at least two envelope frames, the positional relationship between the two envelope frames satisfies the location condition. The content of the location condition is not limited herein. Exemplarily, the location condition includes: the difference between the center coordinates of the two envelope frames is less than a threshold.

[0095] After grouping to obtain the individual groups, for any group containing only one envelope, this envelope is used as the hard subtitle area. If the group contains at least two envelopes, the envelopes of the group are merged to obtain the hard subtitle area. The merging method is not limited here. For example, the vertex with the smallest coordinate and the vertex with the largest coordinate can be determined from the position information of the envelopes of the group. Based on these two vertices, a merged envelope is determined, and this envelope is the hard subtitle area.

[0096] The embodiment of the present application does not limit the specific implementation method of merging the envelope to obtain the hard subtitle area. For example, if the envelope is a rectangular frame, the electronic device can perform steps S1 to S9 as shown below to obtain the hard subtitle area. The details are as follows.

[0097] Step S1: Initialize a horizontal rectangular box list and a tilted rectangular box list, wherein the horizontal rectangular box list is used to store horizontal rectangular boxes, and the tilted rectangular box list is used to store tilted rectangular boxes. Based on this, traverse all envelope boxes of the second connected area to determine whether each envelope box is a horizontal rectangular box or a tilted rectangular box. Optionally, for each envelope box, calculate the slope of at least one edge in the envelope box, and based on the slope of each edge and the slope threshold, calculate the slope of at least one edge in the envelope box. slope ths , determine whether the envelope is a horizontal rectangle or an inclined rectangle. Put all horizontal rectangles into the horizontal rectangle list, and put all inclined rectangles into the inclined rectangle list.

[0098] For example, when the slopes of the upper and lower edges of the envelope are both less than the slope threshold slope ths When the envelope is judged to be a horizontal rectangular box, in this case, the envelope is placed in the horizontal rectangular box list; when the slope of the upper and lower sides of the envelope is greater than the slope threshold slope ths When the envelope is judged to be an inclined rectangular frame, the envelope is placed in the inclined rectangular frame list. It should be noted that the slope threshold value can be flexibly set according to the actual application. For example, the slope threshold slope ths is 0.1.

[0099] Step S2: traverse all envelope frames in the horizontal rectangular frame list and sort these envelope frames in ascending order of the vertical coordinates of the central pixels.

[0100] Step S3: Initialize the current merge list 1 and the total merge list 1. Traverse the horizontal rectangular box list and obtain each envelope box in sequence. Step S3 includes steps S3-1 to S3-3 as shown below. For the current envelope box, determine the list it belongs to according to steps S3-1 to S3-3.

[0101] Step S3 - 1 : If the current merge list 1 is empty, that is, the current merge list 1 does not have an envelope frame, then the current envelope frame is placed in the current merge list 1 .

[0102] Step S3-2: If the current merge list 1 is not empty, that is, the current merge list 1 contains at least one envelope frame and satisfies the first position condition: , put the current envelope into the current merge list 1. Represents the mean of the vertical coordinates of the center points of each envelope in the current merge list 1, Represents the average height of each envelope box in the current merge list 1, y center The vertical coordinate representing the center point of the current envelope, ths center It should be noted that the threshold can be flexibly set according to the actual application situation. ths center For example, ths center .

[0103] Step S3-3: If the current merge list 1 is not empty and does not meet the first position condition, each envelope frame in the current merge list 1 is grouped into the total merge list 1, the current merge list 1 is cleared, and the current envelope frame is placed in the current merge list 1.

[0104] Through the above step S3, all horizontal rectangular frames are grouped according to the first position condition to obtain groups, each group includes at least one horizontal rectangular frame, and the position relationship between the envelope frames belonging to the same group meets the first position condition.

[0105] Step S4: Initialize current merge list 2 and total merge list 2. Traverse total merge list 1 and obtain each group in sequence. If the current group contains only one envelope, place the current group in total merge list 2, skip steps S5 through S7, and proceed directly to step S8. If the current group contains at least two envelopes, proceed again from step S5.

[0106] Step S5: When the current group includes at least two envelope frames, traverse the envelope frames in the current group and sort the envelope frames in ascending order of the horizontal coordinates of the upper left corners of the envelope frames.

[0107] Step S6: traverse the current group and obtain each envelope frame in sequence. Step S6 includes steps S6-1 to S6-3 as shown below. For the current envelope frame, determine the list it belongs to according to steps S6-1 to S6-3.

[0108] Step S6 - 1 : If the current merge list 2 is empty, that is, the current merge list 2 does not have an envelope frame, then the current envelope frame is placed in the current merge list 2 .

[0109] Step S6-2: If the current merge list 2 is not empty, that is, the current merge list 2 contains at least one envelope frame and satisfies the second position condition: , put the current envelope into the current merge list 2. Represents the average height of each envelope box in the current merge list 2, height is the height of the current envelope, Represents the minimum value of the horizontal coordinates of each vertex of the current envelope, Represents the maximum value of the vertical coordinates of each vertex of the current envelope, Represents the minimum value of the vertical coordinates of each vertex of the current envelope, is the maximum value of the horizontal coordinates of the vertices of the previous envelope frame of the current envelope frame in the sorted envelope frames. is the height threshold It should be noted that the height threshold can be flexibly set according to the actual application situation. , width threshold For example, .

[0110] Step S6-3: If the current merge list 2 is not empty and does not meet the second position condition, each envelope frame in the current merge list 2 is grouped as a group and placed in the total merge list 2, the current merge list 2 is cleared, and the current envelope frame is placed in the current merge list 2.

[0111] Through the above-mentioned step S6, for the group including at least two envelope frames obtained by the division in step S3, the envelope frames in the group are grouped again according to the second position condition to obtain individual groups, each group includes at least one horizontal rectangular frame, and the positional relationship between the envelope frames belonging to the same group satisfies the first position condition and the second position condition.

[0112] Step S7: If the current merge list 2 is not empty, each envelope frame in the current merge list 2 is grouped as a group and put into the total merge list 2.

[0113] Step S8: Treat each tilted envelope as a group and put it into the total merge list 2.

[0114] From this point on, the envelopes of all second connected regions are grouped to obtain groups. Each group includes at least one envelope. Optionally, a group includes an inclined envelope, or a group includes a horizontal envelope, or a group includes at least two horizontal envelopes that meet the first position condition and the second position condition.

[0115] Step S9: traverse the total merge list 2 and obtain each group in turn. For the current group, determine the minimum value from the horizontal coordinates of the vertices of each envelope in the current group. and maximum value , and determine the minimum value from the vertical coordinates of the vertices of each envelope in the current group and maximum value .according to , determine the coordinates of the upper left corner, according to , determine the coordinates of the lower right corner. According to the coordinates of the upper left corner, the coordinates of the lower right corner, the heighty max -y min and width x max -x min , determine a hard subtitle area. Among them, The expansion coefficient can be flexibly set according to the actual application situation. For example, It is understandable that each group in the total merged list 2 can determine a hard subtitle area. In other words, there is at least one hard subtitle area.

[0116] By grouping and merging the envelope frames of each second connected area, the envelope frames that meet the position conditions are merged into a hard subtitle area, which facilitates subsequent optimization of the hard subtitle area and improves the visual effect of the hard subtitle area.

[0117] The detection process of the hard subtitle area is as follows: Figure 4 As shown. In this example, the electronic device crops an area to be detected 401 from the bottom of the first frame image, and the area to be detected 401 includes hard subtitles. The area to be detected 401 is input into the image detection model, and the image detection model outputs a detection result 402. The detection result 402 includes the probability that each pixel point in the area to be detected 401 belongs to a hard subtitle. The detection result is binarized according to the confidence threshold to obtain a binarized area 403. The binarized area 403 represents whether each pixel point in the area to be detected 401 belongs to a hard subtitle. The connected area can be determined based on the binarized area 403, and the brightness threshold is determined based on the second frame image, and the connected area is filtered by the brightness threshold. The envelope box 404 of the filtered connected domain is determined, and these envelope boxes are merged to obtain the hard subtitle area 405.

[0118] In a possible implementation, step 304 further includes steps 305 to 306 (not shown in the figure).

[0119] Step 305, determining at least one of the first area, the second area, or the third area, wherein the first area is the image after adjusting the brightness of the hard subtitle area, the second area is the image after adjusting the color of the hard subtitle area, and the third area is the image after improving the dynamic range of the hard subtitle area.

[0120] The electronic device can adjust the brightness of the hard subtitle area to obtain the first area. In different application scenarios, the brightness adjustment method may be different. Exemplarily, the brightness adjustment method is as follows: first, binarization is performed on the hard subtitle area to obtain a binarized image. The method for determining the binarized image is described below and will not be repeated here. The binarized image includes multiple pixels, and each pixel in the binarized image corresponds one-to-one to each pixel in the hard subtitle area (or the third area). The value of any pixel in the binarized image represents whether the brightness of the corresponding pixel in the hard subtitle area (or the third area) is higher than the reference brightness.

[0121] Assuming that the pixels of interest are pixels with brightness higher than or equal to the reference brightness, and the pixels of non-interest are pixels with brightness lower than the reference brightness, the binary image includes the pixels of interest and the pixels of non-interest, and the brightness range of the corresponding pixels of the pixels of interest in the hard subtitle area (or the third area) is . Characterizes the reference brightness, Represents the maximum brightness of pixels in the subtitle area (or third area).

[0122] Electronic devices can obtain extreme brightness differences The brightness extreme difference value represents the difference between the maximum brightness and the minimum brightness of the pixel of interest. Its unit is nit, and the specific value is not limited here. For example, Based on the brightness extreme difference value, the brightness of the corresponding pixel of each pixel of interest in the hard subtitle area (or the third area) is adjusted to obtain the first area. It can be understood that after the brightness adjustment, the brightness range of the pixel of interest is .

[0123] Assume that the brightness value of the corresponding pixel of interest in the hard subtitle area (or the third area) is , then after brightness adjustment, the brightness value of the pixel of interest for: .in, Characterizing the reference brightness, the method for determining the reference brightness is described below and will not be repeated here. Represents the maximum value of the brightness values ​​of each pixel in the subtitle area (or the third area).

[0124] Because hard subtitles have a higher brightness than the image content, and the brightness of pixels of interest is higher than or equal to the reference brightness, adjusting the brightness of pixels of interest in the hard subtitle region effectively adjusts the brightness of pixels potentially belonging to the hard subtitle. By adjusting the brightness of the hard subtitles to obtain the first region and determining the fourth region based on the first region, we can prevent the hard subtitles from being too bright or too dark, ensuring the visual quality of the hard subtitles.

[0125] Optionally, the reference brightness is determined based on a reference area surrounding the hard subtitle area. By adjusting the brightness of the hard subtitles based on the reference brightness, the probability of the following two situations can be reduced: Situation 1: When the pixels surrounding the hard subtitles are darker, the brightness of the hard subtitles is higher. Situation 2: When the pixels surrounding the hard subtitles are brighter, the brightness of the hard subtitles is darker. In other words, after adjusting the brightness of the hard subtitle area based on the reference brightness, the brightness of the hard subtitles matches the brightness of the surrounding pixels, avoiding a glaring visual effect.

[0126] During image processing, the colors of surrounding pixels may be affected by local operations and encoding, resulting in color noise within the hard subtitles and affecting the visual experience. To address this, electronic devices can adjust the color of the hard subtitle area to obtain a second area. By determining a fourth area based on the second area, the color quality of the hard subtitles can be maintained. The color adjustment method may vary in different application scenarios.

[0127] Exemplarily, the hard subtitle area is subjected to color gamut conversion to obtain a converted area to obtain original subtitle color information. The electronic device obtains a color gamut conversion matrix and performs color gamut conversion on the hard subtitle area using the color gamut conversion matrix to obtain the converted area.

[0128] Optionally, the color gamut conversion matrix is: After the color gamut conversion of the hard subtitle area is performed according to the color gamut conversion matrix, the converted area satisfies: .in, Characterize the transformed region, Represents the hard subtitle area, where R, G, and B are the pixel values ​​of the pixel in the R channel, G channel, and B channel respectively. The converted area is the second area, which retains the color and brightness information of the hard subtitle area.

[0129] Furthermore, the electronic device may crop the area corresponding to the hard subtitle area from the second frame image to obtain a third area. Alternatively, the electronic device may perform up-conversion on the hard subtitle area to obtain the third area. Because the dynamic range of the third area is greater than that of the hard subtitle area, determining the fourth area based on the third area can increase the dynamic range and color depth of the hard subtitles, thereby improving the visual quality of the hard subtitles.

[0130] Step 306: Determine a fourth area based on at least one of the first area, the second area, or the third area.

[0131] If the electronic device acquires any one of the first area, the second area, or the third area, the electronic device uses the acquired area as the fourth area. If the electronic device acquires at least two of the first area, the second area, or the third area, the electronic device merges the acquired areas to obtain the fourth area.

[0132] In an exemplary embodiment, step 306 includes: determining a weight image, the weight image including weights of each pixel in the hard subtitle area; and determining a fourth area based on at least one of the first area, the second area, or the third area and the weight image.

[0133] The size of the weighted image is consistent with the size of the hard subtitle area. For each pixel in the hard subtitle area, there is a corresponding pixel in the weighted image. The value of any pixel on the weighted image represents the weight of the corresponding pixel in the hard subtitle area. The manner in which the electronic device obtains the weighted image is not limited here. For example, the electronic device can obtain the weighted image input by the user. Alternatively, determining the weighted image includes: performing binarization on the hard subtitle area to obtain a binarized image, the value of the pixel in the binarized image represents whether the brightness of the pixel is higher than the reference brightness; performing dilation on the pixel whose brightness is higher than the reference brightness in the binarized image to obtain an dilated image; and determining the weighted image based on the dilated image.

[0134] In practical applications, the reference brightness can be determined based on manual experience or based on the area surrounding the hard subtitle area. For example, before binarizing the hard subtitle area to obtain a binarized image, the method further includes: determining a reference area surrounding the hard subtitle area in the first or second image frame; and determining the reference brightness based on the reference area.

[0135] In this example, the first frame image includes a hard subtitle area, and the electronic device can determine a reference area surrounding the hard subtitle area in the first frame image. The second frame image includes a corresponding area of ​​the hard subtitle area, and the electronic device can determine a reference area surrounding the corresponding area in the second frame image. This reference area can also be considered as the area surrounding the hard subtitle area.

[0136] Assume that the hard subtitle area is R sub , the upper left corner coordinates of the hard subtitle area are ( x l , y l ), the coordinates of the lower right corner are ( x r , y r The electronic device can expand the boundary box of the hard subtitle area outward, in the height direction, by H pixels upward and downward, and in the width direction, by W pixels left and right, to obtain a rectangular areaR expand The coordinates of the upper left corner of the rectangular area are , the coordinates of the lower right corner are The values ​​of H and W are not limited here. For example, if the resolution of the first frame image is 1080P (i.e. the size is 1920×1080), then H=W=50. R expand With hard subtitle area R sub The difference, that is R expand Internal and R sub The outer overlapping area is used as a reference area, and the reference area is a rectangular ring. It can be understood that if there are at least two hard subtitle areas, then for each hard subtitle area, a corresponding reference area can be determined in the above manner.

[0137] Electronic device obtains the lower limit of subtitle brightness The lower limit of brightness represents the minimum value of the reference brightness, and its unit is nit. The value of is not limited here, for example, For each reference area, obtain the brightness value of each pixel in the reference area, and take the 98th percentile of the brightness value (that is, 98% of the pixel brightness is less than this value) as the alternative brightness to eliminate the noise effect. If the alternative brightness is less than the lower limit of the subtitle brightness , the alternative brightness is set to If the alternative brightness is greater than or equal to the subtitle brightness lower limit , then keep the alternative brightness unchanged. Traverse the alternative brightness of each reference area and take the maximum value as the reference brightness .

[0138] Next, based on the reference brightness Binarization is performed on the hard subtitle area to obtain a binary image. In this example, the hard subtitle area is obtained. R sub The brightness value of each pixel in the reference brightness For hard subtitle area R sub Perform binarization to obtain a binary image. For any pixel, if the brightness value of the pixel is higher than the reference brightness, the value of the pixel is 1, indicating that the pixel is a pixel of interest; if the brightness value of the pixel is equal to or lower than the reference brightness, the value of the pixel is 0, indicating that the pixel is a pixel of no interest. The binary image includes all pixels of interest and all pixels of no interest, and the binary image can be recorded as M subIt is understandable that the brightness of hard subtitles is significantly higher than that of the screen content, and the vast majority of pixels of interest belong to hard subtitles, while a small number of pixels belong to non-hard subtitles.

[0139] Next, use the dilate(·) function in the OpenCV library to dilate pixels in the binary image whose brightness is higher than the reference brightness (i.e., pixels of interest) to produce a dilated image. Optionally, for each pixel of interest in the binary image, dilate it outward by a number of pixels (e.g., 31) to produce a dilated pixel. These dilated pixels, including the pixel of interest, form a dilated region. The dilated image consists of the dilated region, which includes the dilated pixels, and the region outside the dilated region, which includes pixels of non-interest.

[0140] For each expanded pixel in the expanded area, calculate the shortest distance between the expanded pixel and the pixel of interest d , and d Perform normalization to obtain the normalized distance . The value range is 0~1, which can represent the weight of the expanded pixel. In other words, the weight of the pixel in the expanded area is w fusion Set to , the weight of the pixel outside the expansion area w fusion Set to 1. From this we can get the weight image.

[0141] By determining a reference brightness from a reference area surrounding the hard subtitle area and binarizing the hard subtitle area based on the reference brightness, the system identifies potential hard subtitle pixels by referencing the area surrounding the hard subtitles, improving the accuracy of the binarized image. By dilating the potential hard subtitle pixels in the binarized image and determining a weight image based on the dilated image, a fourth area is generated by fusing at least one of the first, second, and third areas based on the weighted image. Determining the fourth area using the weighted image smoothes variations in brightness, color, and dynamic range between the hard subtitles and the surrounding area, improving the visual quality of the fourth area.

[0142] It's understandable that the fusion methods for each region may vary in different scenarios. For example, the fourth region may be obtained by fusing at least one of the first, second, and third regions, along with the hard subtitle region, based on a weighted image. Alternatively, the first and third regions may be fused first, and then the fused region may be fused with the second region. Alternatively, the first and second regions may be fused first, and then the fused region may be fused with the third region. A possible fusion method is shown below.

[0143] In this example, the first region and the third region are first fused based on the weight image to obtain a fused region. HDR impbr = w fusion × HDR + (1- w fusion )× HDR br , determine the fusion area. Among them, HDR impbr Characterize the fusion region, w fusion Representation weight image, HDR Characterizing the third region, HDR br Characterize the first region.

[0144] Next, the second region and the fused region are further fused based on the weighted image to obtain a fourth region. Optionally, since the second region is obtained by color gamut conversion of the hard subtitle region, its dynamic range is consistent with the dynamic range of the first frame image, which is much lower than the dynamic range of the second frame image. HDR impbr The dynamic range of the second frame image is consistent with the dynamic range of the second frame image. HDR fmt Brightness stretching is performed so that the dynamic range of the stretched area matches the dynamic range of the second frame image.

[0145] The brightness stretching process is as follows: Get the second area HDR fmt The brightness value of each pixel in Y fmt , and obtain the fusion area HDR impbr The brightness value of each pixel in Y impbr Based on the brightness value of any pixel Y fmt and brightness values Y impbr , calculate the brightness improvement ratio of the pixel point. Optionally, the brightness improvement ratio of the pixel point ratio satisfy: .in, is a hyperparameter that prevents the divisor from being 0, for example, Next, a Gaussian kernel (its size is not limited, for example, the size is 5) is used to average the pixels in the second area to smooth the brightness of the pixels belonging to the hard subtitles. ratio , the second area after the average processing The brightness of each pixel in the second area after improvement satisfy: .

[0146] As mentioned above, when calculating the brightness of a pixel, it is necessary to first convert the pixel value of the pixel from the nonlinear domain to the linear domain, and then determine the brightness value of the pixel based on the converted pixel value. In other words, the brightness adjustment is performed in the linear domain, so the electronic device can adjust the brightness of the second area after the improvement. Perform photoelectric conversion to obtain a second region after photoelectric conversion. , the linear domain Converted to nonlinear domain .

[0147] The electronic device can further fuse the second region after photoelectric conversion and the fused region based on the weighted image to obtain a fourth region. , determine the fourth area. Among them, HDR impcol Characterizing the fourth region, w fusion Representation weight image, HDR impbr Characterize the fusion region, The region represents the region after brightness stretching is performed on the second region.

[0148] In an exemplary embodiment, the optimization process for the hard subtitle area is as follows Figure 5 As shown. In this example, the hard subtitle area is binarized to obtain a binarized image 501. The value of any pixel in the binarized image 501 indicates whether the brightness of the pixel is higher than the reference brightness. Pixels in the binarized image 501 with brightness higher than the reference brightness are dilated to obtain a dilated image 502. A weighted image 503 is determined based on the dilated image 502. For pixels in the binarized image 501 with brightness higher than the reference brightness, the brightness of the corresponding pixels in the hard subtitle area 504 is adjusted to obtain a first area 505. The hard subtitle area 504 and the first area 505 are fused based on the weighted image 503 to obtain a hard subtitle area 506 with optimized brightness. The hard subtitle area 506 with optimized brightness can be the fourth area mentioned above.

[0149] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, storage, and display, etc.), and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant region. For example, the first and second frames of images involved in this application were obtained with full authorization.

[0150] In the above method, at least two first connected regions are determined from a first frame image containing hard subtitles. These first connected regions include at least two adjacent pixels belonging to the hard subtitles. Because the brightness of pixels belonging to hard subtitles in a high dynamic range image is significantly higher than that of other pixels, and the dynamic range of the second frame image is higher than that of the first frame image, by selecting from the at least two first connected regions those first connected regions whose corresponding regions in the second frame image have a brightness not less than a brightness threshold, connected regions that do not belong to hard subtitles are removed, resulting in second connected regions. This improves the accuracy of hard subtitle region determination based on these second connected regions.

[0151] The above describes the image processing method of the embodiment of the present application from the perspective of method steps. The following is a systematic and comprehensive description. The method of the embodiment of the present application can be applied in the video playback scenario. Generally, electronic devices are equipped with video playback software, and videos are played through the video playback software. During the video playback process, the user can switch the video quality at any time. Figure 6 As shown, Figure 6 The display interface of a video player software is shown, which includes a frame image 601 of the video being played and multiple image quality switching controls 602. It is understandable that different video player software may correspond to different types or the same type of image quality switching controls. Figure 6 The image quality switching controls shown are for illustrative purposes only and can be flexibly set in actual applications.

[0152] Assume that when the user selects the image quality switching control 604, the video playback software plays the SDR video, and when the user selects the image quality switching control 603, the video playback software plays the HDR video. Based on this, if the user selects the image quality switching control 603 while the video playback software is playing the SDR video, in this case, on the one hand, the electronic device identifies the hard subtitle area from the SDR video based on the image processing method of the embodiment of the present application and optimizes the hard subtitle area. On the other hand, the electronic device converts the picture content of the SDR video except the hard subtitle area into the picture content of the HDR video. The electronic device displays the converted picture content and the optimized hard subtitle area. Processing the video in this way can make the processed video close to the realistic lighting effects, light and dark contrast, color tone, and retinal (i.e., real human eye) level of particle clarity, improve the video quality, and thus enhance the user's visual experience.

[0153] Optionally, the electronic device is configured as follows: Figure 7 The process shown in FIG. 1 identifies a hard subtitle area from an SDR video and optimizes the hard subtitle area. Specifically, the process includes the following contents.

[0154] Step 701: Identify the hard subtitle area.

[0155] In this example, the electronic device obtains the SDR video being played. When the user selects the image quality switching control 603, the electronic device can obtain the HDR video obtained by manually processing the SDR video, or it can process the SDR video through a conversion algorithm to generate an HDR video. The content of the SDR video and the HDR video are the same, but the dynamic ranges of the two are different. If the SDR video includes hard subtitles, the electronic device decodes the SDR video into individual SDR frame images and decodes the individual HDR videos into individual HDR frame images, so as to identify and optimize the hard subtitle areas in the frame images frame by frame based on the individual SDR frame images and the individual HDR frame images. For ease of description, the implementation process of steps 701 and 702 is explained below using an SDR frame image and a corresponding HDR frame image as an example. The SDR frame image is the first frame image mentioned above, and the HDR frame image is the second frame image mentioned above.

[0156] In this example, on the one hand, the SDR frame image is cropped to obtain an area to be detected including hard subtitles, the area to be detected is preprocessed to obtain a preprocessed area, and a connected domain is detected based on the preprocessed area, wherein the connected domain is the first connected domain mentioned above. On the other hand, the connected domain is filtered according to the HDR frame image corresponding to the SDR frame image, so as to remove the connected domains whose brightness in the corresponding area in the HDR frame image is less than the brightness threshold, and retain the connected domains whose brightness in the corresponding area in the HDR frame image is not less than the brightness threshold. Among them, the retained connected domain is the second connected domain mentioned above. The electronic device determines the envelope boxes of each retained connected domain, and obtains the hard subtitle area by merging these envelope boxes. As can be seen from the foregoing, the hard subtitle area is the image area where the hard subtitles are located in the SDR frame image.

[0157] Step 702: Optimize the hard subtitle area.

[0158] In this example, the hard subtitle area is a rectangular area in the SDR frame image. The electronic device can obtain the size and position of the rectangular area. Based on the size and position of the rectangular area, a reference area is determined in the HDR frame image. The reference area is a rectangular ring surrounding the rectangular area. The electronic device can determine a reference brightness based on the reference area. Binarization is performed on the hard subtitle area based on the reference brightness to obtain a binary image. The value of any pixel in the binary image indicates whether the brightness of the pixel is greater than the reference brightness. Pixels in the binary image with brightness greater than the reference brightness are dilated to obtain a dilated image, and a weighted image is determined based on the dilated image. Furthermore, each pixel in the binary image corresponds one-to-one to each pixel in the hard subtitle area. For pixels in the binary image with brightness greater than the reference brightness, the brightness of the corresponding pixel in the hard subtitle area is adjusted. This adjustment of the brightness of the hard subtitle area results in a first area. The corresponding area of ​​the hard subtitle area is cropped from the HDR frame image to obtain a third area. The first and third areas are fused based on the weighted image to obtain a hard subtitle area with optimized brightness. Afterwards, the SDR frame image is color-adjusted, and the area corresponding to the hard subtitle area is cropped from the color-adjusted SDR frame image to obtain a second area. Alternatively, the hard subtitle area is directly color-adjusted to obtain the second area. The second area and the hard subtitle area after brightness optimization are fused based on the weight image to obtain a hard subtitle area after color optimization. The hard subtitle area after color optimization is the fourth area mentioned above. From then on, the optimization of the hard subtitle area is completed. Afterwards, the electronic device obtains and outputs an optimized HDR frame image based on the hard subtitle area after color optimization and the image content in the HDR frame image other than the hard subtitle area. The electronic device displays the optimized HDR frame image.

[0159] It can be understood that after the user selects the image quality switching control 603, the electronic device can process each SDR frame image in the SDR video according to the implementation principles of steps 701 to 702, obtain and display the corresponding optimized HDR frame image, and these HDR frame images can form an HDR video.

[0160] Figure 8 FIG. 1 is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. Figure 8 As shown, the device includes the following contents.

[0161] The acquisition module 801 is configured to acquire a first frame image and a second frame image, wherein the first frame image includes hard subtitles, and the second frame image is an image obtained by increasing the dynamic range of the first frame image.

[0162] The determining module 802 is configured to determine at least two first connected regions in the first frame image, where the first connected regions include at least two adjacent pixels belonging to hard subtitles in the first frame image.

[0163] The screening module 803 is configured to screen a second connected area from the at least two first connected areas, where the second connected area is a first connected area whose brightness of a corresponding area in the second frame image is not less than a brightness threshold.

[0164] The determination module 802 is further configured to determine a hard subtitle area based on the second connected area.

[0165] In one possible implementation, the screening module 803 is used to determine brightness data for the i-th first connected area based on the brightness values ​​of each pixel in the second frame image, and when the brightness data is not less than a brightness threshold, the i-th first connected area is used as the second connected area, where i is a positive integer and the brightness data represents the maximum brightness of the area corresponding to the i-th first connected area in the second frame image.

[0166] In one possible implementation, the determination module 802 is further used to determine at least one of a first area, a second area, or a third area, where the first area is an image after adjusting the brightness of the hard subtitle area, the second area is an image after adjusting the color of the hard subtitle area, and the third area is an image after improving the dynamic range of the hard subtitle area; and a fourth area is determined based on at least one of the first area, the second area, or the third area.

[0167] In one possible implementation, the determination module 802 is configured to determine a weight image, where the weight image includes weights of each pixel in the hard subtitle area; and determine a fourth area based on at least one of the first area, the second area, or the third area and the weight image.

[0168] In one possible implementation, the determination module 802 is configured to perform binarization on the hard subtitle area to obtain a binarized image, where the value of a pixel in the binarized image represents whether the brightness of the pixel is higher than a reference brightness; perform dilation on the pixels in the binarized image whose brightness is higher than the reference brightness to obtain a dilated image; and determine a weighted image based on the dilated image.

[0169] In a possible implementation, the determination module 802 is further configured to determine a reference area located around the hard subtitle area in the first frame image or the second frame image; and determine a reference brightness based on the reference area.

[0170] In one possible implementation, the determination module 802 is configured to crop the first frame image to obtain an area to be detected, where the area to be detected includes hard subtitles; determine a detection result based on the area to be detected, where the detection result represents the likelihood that each pixel point in the area to be detected belongs to a hard subtitle; and determine at least two first connected areas based on the detection result.

[0171] In one possible implementation, the pixel points in the area to be detected correspond to pixel values ​​of at least three channels; a determination module 802 is used to preprocess the area to be detected to obtain a preprocessed area; and a detection result is determined based on the preprocessed area; wherein the preprocessed area includes the pixel values ​​of each channel corresponding to the first pixel point and the grayscale value of the second pixel point, the first pixel point is a pixel point in the area to be detected whose minimum pixel value is not less than the first threshold and whose maximum extreme value is not greater than the second threshold, the minimum pixel value is the minimum value among the pixel values ​​of at least three channels, the maximum extreme value is the maximum value among the differences between the pixel values ​​of each two channels, the second pixel point includes the pixel points in the area to be detected other than the first pixel point, and the grayscale value of the second pixel point is determined based on the pixel values ​​of each channel corresponding to the second pixel point.

[0172] In one possible implementation, there are at least three second connected areas; a determination module 802 is configured to determine envelope frames of each second connected area; group the envelope frames of each second connected area to obtain at least one group, each group including at least one envelope frame, and the positional relationship between two envelope frames belonging to the same group satisfies a position condition; and merge the envelope frames of each group to obtain a hard subtitle area.

[0173] In the above-described device, at least two first connected regions are determined from a first frame image containing hard subtitles. The first connected regions include at least two adjacent pixels belonging to the hard subtitles. Because the brightness of pixels belonging to hard subtitles in a high dynamic range image is significantly higher than that of other pixels, and the dynamic range of the second frame image is higher than that of the first frame image, by selecting from the at least two first connected regions those first connected regions whose corresponding regions in the second frame image have a brightness not less than a brightness threshold, connected regions not belonging to hard subtitles are removed, resulting in second connected regions, thereby improving the accuracy of hard subtitle region determination based on the second connected regions.

[0174] It should be understood that the above Figure 8 The provided device is illustrated only by the division of the above-mentioned functional modules when implementing its functions. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0175] Figure 9 FIG2 shows a block diagram of a terminal device 900 provided by an exemplary embodiment of the present application. The terminal device 900 includes a processor 901 and a memory 902 .

[0176] Processor 901 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 901 may be implemented in hardware using at least one of the following: a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), or a PLA (Programmable Logic Array). Processor 901 may also include a main processor and a coprocessor. The main processor is used to process data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 901 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing content displayed on the display screen. In some embodiments, processor 901 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0177] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 902 is used to store at least one computer program, which is executed by the processor 901 to implement the image processing method provided in the method embodiment of the present application.

[0178] In some embodiments, the terminal device 900 may optionally include a peripheral device interface 903 and at least one peripheral device. The processor 901, memory 902, and peripheral device interface 903 may be connected via a bus or signal lines. Each peripheral device may be connected to the peripheral device interface 903 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 904, a display screen 905, a camera assembly 906, an audio circuit 907, and a power supply 908.

[0179] The peripheral device interface 903 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 901 and the memory 902. In some embodiments, the processor 901, the memory 902, and the peripheral device interface 903 are integrated on the same chip or circuit board. In other embodiments, any one or two of the processor 901, the memory 902, and the peripheral device interface 903 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0180] The RF circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 904 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the RF circuit 904 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The RF circuit 904 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 904 may also include circuitry related to Near Field Communication (NFC), although this application does not limit this.

[0181] The display screen 905 is used to display a user interface (UI). This UI may include graphics, text, icons, videos, or any combination thereof. If the display screen 905 is a touchscreen display, it is also capable of detecting touch signals on or above the surface of the display screen 905. These touch signals can be input as control signals to the processor 901 for processing. In this case, the display screen 905 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there can be a single display screen 905, located on the front panel of the terminal device 900. In other embodiments, there can be at least two display screens 905, located on different surfaces of the terminal device 900 or in a foldable design. In still other embodiments, the display screen 905 can be a flexible display, located on a curved or foldable surface of the terminal device 900. Furthermore, the display screen 905 can be configured as a non-rectangular, irregular shape, also known as a special-shaped screen. The display screen 905 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0182] The camera component 906 is used to capture images or videos. Optionally, the camera component 906 includes a front camera and a rear camera. Typically, the front camera is set on the front panel of the terminal, and the rear camera is set on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera component 906 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0183] The audio circuit 907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals to be input into the processor 901 for processing, or input into the radio frequency circuit 904 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there can be multiple microphones, which are respectively arranged in different parts of the terminal device 900. The microphone can also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signals from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker can be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signals into sound waves audible to humans, but also convert the electrical signals into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 907 may also include a headphone jack.

[0184] The power supply 908 is used to power the various components in the terminal device 900. The power supply 908 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When the power supply 908 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0185] In some embodiments, the terminal device 900 further includes one or more sensors 909 , including but not limited to: an acceleration sensor 911 , a gyroscope sensor 912 , a pressure sensor 913 , an optical sensor 914 , and a proximity sensor 915 .

[0186] The accelerometer 911 can detect the magnitude of acceleration along the three coordinate axes of the coordinate system established by the terminal device 900. For example, the accelerometer 911 can be used to detect the components of gravity acceleration along the three coordinate axes. The processor 901 can control the display screen 905 to display the user interface in a landscape or portrait view based on the gravity acceleration signal collected by the accelerometer 911. The accelerometer 911 can also be used to collect game or user motion data.

[0187] The gyroscope sensor 912 can detect the orientation and rotation angle of the terminal device 900. It can work in conjunction with the accelerometer 911 to collect the user's 3D movements of the terminal device 900. Based on the data collected by the gyroscope sensor 912, the processor 901 can implement the following functions: motion sensing (for example, changing the UI based on the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0188] The pressure sensor 913 can be set on the side frame of the terminal device 900 and / or the lower layer of the display screen 905. When the pressure sensor 913 is set on the side frame of the terminal device 900, it can detect the user's grip signal of the terminal device 900, and the processor 901 performs left and right hand recognition or shortcut operations based on the grip signal collected by the pressure sensor 913. When the pressure sensor 913 is set on the lower layer of the display screen 905, the processor 901 controls the operable controls on the UI interface based on the user's pressure operation on the display screen 905. The operable controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0189] Optical sensor 914 is used to detect ambient light intensity. In one embodiment, processor 901 can control the display brightness of display screen 905 based on the ambient light intensity detected by optical sensor 914. Specifically, when the ambient light intensity is high, the display brightness of display screen 905 is increased; when the ambient light intensity is low, the display brightness of display screen 905 is decreased. In another embodiment, processor 901 can also dynamically adjust the shooting parameters of camera assembly 906 based on the ambient light intensity detected by optical sensor 914.

[0190] The proximity sensor 915, also known as a distance sensor, is typically located on the front panel of the terminal device 900. The proximity sensor 915 is used to detect the distance between the user and the front of the terminal device 900. In one embodiment, when the proximity sensor 915 detects that the distance between the user and the front of the terminal device 900 is gradually decreasing, the processor 901 controls the display screen 905 to switch from the screen-on state to the screen-off state. When the proximity sensor 915 detects that the distance between the user and the front of the terminal device 900 is gradually increasing, the processor 901 controls the display screen 905 to switch from the screen-off state to the screen-on state.

[0191] Those skilled in the art will understand that Figure 9 The structure shown in the figure does not constitute a limitation on the terminal device 900, and the terminal device 900 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0192] Figure 10This is a schematic diagram of the structure of the server provided in an embodiment of the present application. The server 1000 may vary significantly due to different configurations or performance, and may include one or more processors 1001 and one or more memories 1002. The one or more memories 1002 store at least one computer program, which is loaded and executed by the one or more processors 1001 to implement the image processing methods provided in the above-mentioned various method embodiments. Exemplarily, the processor 1001 is a CPU. Of course, the server 1000 may also have components such as a wired or wireless network interface, a keyboard, and an input / output interface for input and output. The server 1000 may also include other components for implementing device functions, which will not be described in detail here.

[0193] In an exemplary embodiment, a computer-readable storage medium is further provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned image processing methods.

[0194] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, or the like.

[0195] In an exemplary embodiment, a computer program is further provided. The computer program is at least one, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any one of the above-mentioned image processing methods.

[0196] In an exemplary embodiment, a computer program product is further provided. The computer program product stores at least one computer program. The at least one computer program is loaded and executed by a processor to enable an electronic device to implement any one of the above-mentioned image processing methods.

[0197] It should be understood that the term "plurality" used herein refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0198] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0199] The above description is merely an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a first frame of image and a second frame of image, wherein the first frame of image includes hard subtitles, and the second frame of image is an image obtained by increasing the dynamic range of the first frame of image; determining at least two first connected regions in the first frame of image, where the first connected regions include at least two adjacent pixels in the first frame of image that belong to the hard subtitles; Selecting a second connected area from the at least two first connected areas, where the second connected area is a first connected area whose brightness of a corresponding area in the second frame image is not less than a brightness threshold; determining a hard subtitle area based on the second connected area; performing binarization on the hard subtitle area to obtain a binarized image, wherein the value of a pixel in the binarized image represents whether the brightness of the pixel is higher than a reference brightness; Taking the difference between the maximum brightness and the minimum brightness of a pixel of interest as the extreme brightness difference value, wherein the pixel of interest is a pixel in the binarized image having a brightness higher than or equal to the reference brightness; Adjusting the brightness of the pixel point corresponding to the pixel point of interest in the hard subtitle area based on the brightness extreme difference value to obtain a first area; Acquire at least one of a second region or a third region, wherein the second region is an image obtained by performing color gamut conversion on the hard subtitle region, and the third region is an image obtained by increasing the dynamic range of the hard subtitle region; Expanding the pixel of interest in the binary image outward to obtain an expanded pixel, calculating the shortest distance between the expanded pixel and the pixel of interest, and normalizing the shortest distance to obtain a weight of the expanded pixel; Based on a weight image, at least one of the second region or the third region and the first region are fused to obtain a fourth region, wherein the weight image includes weights of the dilated pixels.

2. The method according to claim 1, characterized in that The step of screening the second connected area from the at least two first connected areas comprises: For the i-th first connected area, brightness data is determined based on the brightness values ​​of each pixel in the second frame image. If the brightness data is not less than the brightness threshold, the i-th first connected area is used as the second connected area, where i is a positive integer and the brightness data represents the maximum brightness value of the area corresponding to the i-th first connected area in the second frame image.

3. The method according to claim 1, characterized in that Before performing binarization on the hard subtitle area to obtain a binarized image, the method further includes: determining a reference area around the hard subtitle area in the first frame image or the second frame image; A reference brightness is determined based on the reference area.

4. The method according to any one of claims 1 to 3, characterized in that The determining of at least two first connected areas in the first frame image includes: Cropping the first frame image to obtain a region to be detected, where the region to be detected includes the hard subtitles; determining a detection result based on the area to be detected, wherein the detection result represents the possibility that each pixel point in the area to be detected belongs to the hard subtitle; Based on the detection result, at least two first connected areas are determined.

5. The method according to claim 4, characterized in that The pixel points in the area to be detected correspond to pixel values ​​of at least three channels; The determining of the detection result according to the area to be detected includes: Preprocessing the area to be detected to obtain a preprocessed area; Determining a detection result based on the preprocessed area; In which, the preprocessed area includes the pixel values ​​of each channel corresponding to the first pixel point and the grayscale value of the second pixel point, the first pixel point is the pixel point whose minimum pixel value in the area to be detected is not less than the first threshold and the maximum extreme value is not greater than the second threshold, the minimum pixel value is the minimum value among the pixel values ​​of the at least three channels, the maximum extreme value is the maximum value of the difference between the pixel values ​​of each two channels, the second pixel point includes the pixel points in the area to be detected other than the first pixel point, and the grayscale value of the second pixel point is determined based on the pixel values ​​of each channel corresponding to the second pixel point.

6. The method according to any one of claims 1 to 3, characterized in that The number of the second connected areas is at least three; and determining the hard subtitle area based on the second connected areas includes: Determining the envelope of each second connected area; Grouping the envelope frames of the second connected regions to obtain at least one group, wherein the group includes at least one envelope frame, and a positional relationship between two envelope frames belonging to the same group satisfies a position condition; Merge the envelopes of each group to get the hard subtitle area.

7. An image processing device, characterized in that The device comprises: an acquisition module, configured to acquire a first frame image and a second frame image, wherein the first frame image includes hard subtitles, and the second frame image is an image obtained by increasing the dynamic range of the first frame image; a determining module, configured to determine at least two first connected regions in the first frame of image, where the first connected regions include at least two adjacent pixels in the first frame of image that belong to the hard subtitles; A screening module, configured to screen a second connected area from the at least two first connected areas, wherein the second connected area is a first connected area whose brightness of a corresponding area in the second frame image is not less than a brightness threshold; The determination module is further configured to determine a hard subtitle region based on the second connected region; perform binarization on the hard subtitle region to obtain a binarized image, wherein the value of a pixel in the binarized image represents whether the brightness of the pixel is higher than a reference brightness; use the difference between the maximum brightness and the minimum brightness of a pixel of interest as a brightness extreme difference value, wherein the pixel of interest is a pixel in the binarized image having a brightness higher than or equal to the reference brightness; adjust the brightness of a pixel corresponding to the pixel of interest in the hard subtitle region based on the brightness extreme difference value to obtain a first region; obtain at least one of a second region or a third region, wherein the second region is an image obtained by performing color gamut conversion on the hard subtitle region, and the third region is an image obtained by increasing the dynamic range of the hard subtitle region; dilate the pixel of interest in the binarized image outward to obtain a dilated pixel, calculate the shortest distance between the dilated pixel and the pixel of interest, and normalize the shortest distance to obtain a weight of the dilated pixel; and fuse at least one of the second region or the third region and the first region based on a weight image to obtain a fourth region, wherein the weight image includes the weight of the dilated pixel.

8. The device according to claim 7, characterized in that The screening module is configured to determine brightness data for the i-th first connected area based on the brightness values ​​of each pixel in the second frame image, and to use the i-th first connected area as the second connected area if the brightness data is not less than the brightness threshold, where i is a positive integer and the brightness data represents the maximum brightness of the area corresponding to the i-th first connected area in the second frame image.

9. The device according to claim 7, characterized in that The determination module is further configured to determine a reference area located around the hard subtitle area in the first frame image or the second frame image; and determine a reference brightness based on the reference area.

10. The device according to any one of claims 7 to 9, characterized in that The determination module is configured to crop the first frame image to obtain a region to be detected, where the region to be detected includes the hard subtitles; determine a detection result based on the region to be detected, where the detection result indicates a likelihood that each pixel in the region to be detected belongs to the hard subtitles; Based on the detection result, at least two first connected areas are determined.

11. The device according to claim 10, characterized in that The pixel points in the area to be detected correspond to pixel values ​​of at least three channels; The determining module is used to pre-process the area to be detected to obtain a pre-processed area; Determining a detection result based on the preprocessed area; In which, the preprocessed area includes the pixel values ​​of each channel corresponding to the first pixel point and the grayscale value of the second pixel point, the first pixel point is the pixel point whose minimum pixel value in the area to be detected is not less than the first threshold and the maximum extreme value is not greater than the second threshold, the minimum pixel value is the minimum value among the pixel values ​​of the at least three channels, the maximum extreme value is the maximum value of the difference between the pixel values ​​of each two channels, the second pixel point includes the pixel points in the area to be detected other than the first pixel point, and the grayscale value of the second pixel point is determined based on the pixel values ​​of each channel corresponding to the second pixel point.

12. The device according to any one of claims 7 to 9, characterized in that There are at least three second connected areas; The determination module is configured to determine the envelope frames of each second connected area; group the envelope frames of each second connected area to obtain at least one group, each group including at least one envelope frame, wherein the positional relationship between two envelope frames belonging to the same group satisfies a position condition; and merge the envelope frames of each group to obtain a hard subtitle area.

13. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor, so that the electronic device implements the image processing method according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor to enable the electronic device to implement the image processing method according to any one of claims 1 to 6.

15. A computer program product, characterized in that The computer program product stores at least one computer program, and the at least one computer program is loaded and executed by a processor to enable the electronic device to implement the image processing method according to any one of claims 1 to 6.

Images