Border detection method, apparatus, device, and medium

Abstract

The present disclosure provides a method, device and equipment for edge detection and storage medium. The method comprises: performing feature extraction on a target image to obtain a feature map corresponding to the target image, wherein the target image comprises a video content region and a background region, the junction position of the video content region and the background region is a target edge region to be identified in the target image, and the feature value of each position in the feature map reflects the probability that the position belongs to the edge region; determining the feature value of a candidate edge region in the target image according to the feature value of each position in the feature map; and determining the target edge region in the target image based on the feature value of the candidate edge region. In this way, the edge region in the target image is determined based on the probability that each position in the feature map belongs to the edge region, so that the display mode of the image is adjusted based on the edge region, and thus the display mode of the image is adapted to the screen display mode, thereby ultimately improving the interactive experience of the user.

Description

Technical Field

[0001] This disclosure relates to the field of border detection technology, and in particular to a border detection method, apparatus, device and medium. Background Technology

[0002] When displaying images on electronic devices, the devices usually need to adjust the vertical image to a horizontal image and then embed it in a landscape screen for display, or adjust the horizontal image to a vertical image and then embed it in a portrait screen for display, so that the displayed image is adapted to the screen display mode and can improve the user's interactive experience.

[0003] In the process of displaying images on electronic devices, it is usually necessary to detect the image's border region in order to adapt the displayed image to the screen display method. However, based on current border detection methods, it is difficult to accurately determine the image's border region. Therefore, proposing an accurate border detection method is a technical problem that urgently needs to be solved. Summary of the Invention

[0004] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides a border detection method, apparatus, device, and medium.

[0005] In a first aspect, this disclosure provides a border detection method, which includes:

[0006] Feature extraction is performed on the target image to obtain a feature map corresponding to the target image. The target image includes a video content region and a background region. The boundary between the video content region and the background region is the target bounding box region to be identified in the target image. The feature value of each position in the feature map reflects the probability that the position belongs to the bounding box region.

[0007] Based on the feature values ​​at each location in the feature map, determine the feature values ​​of the candidate bounding box region in the target image;

[0008] Based on the feature values ​​of the candidate border regions, the target border region in the target image is determined.

[0009] Secondly, this disclosure provides a border detection device, the device comprising:

[0010] The feature map acquisition module is used to extract features from the target image and acquire a feature map corresponding to the target image. The target image includes a video content region and a background region. The boundary between the video content region and the background region is the target bounding box region to be identified in the target image. The feature value of each position in the feature map reflects the probability that the position belongs to the bounding box region.

[0011] The feature value determination module is used to determine the feature value of the candidate bounding box region in the target image based on the feature values ​​at each position in the feature map;

[0012] The border region determination module is used to determine the target border region in the target image based on the feature values ​​of the candidate border regions.

[0013] Thirdly, this disclosure provides a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to implement the above-described method.

[0014] Fourthly, this disclosure provides an apparatus comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method described above.

[0015] Fifthly, this disclosure provides a computer program product comprising a computer program / instruction that, when executed by a processor, implements the method described above.

[0016] The technical solution provided in this disclosure has at least the following advantages compared with the prior art:

[0017] This disclosure provides a border detection method, apparatus, device, and medium. First, feature extraction is performed on a target image to obtain a feature map corresponding to the target image. The target image includes a video content region and a background region. The boundary between the video content region and the background region is the target border region to be identified in the target image. The feature value at each position in the feature map reflects the probability that the position belongs to the border region. Then, based on the feature values ​​at each position in the feature map, the feature values ​​of candidate border regions in the target image are determined. Next, based on the feature values ​​of the candidate border regions, the target border region in the target image is determined. Through this method, the border region in the target image can be accurately determined based on the probability that each position in the feature map belongs to the border region. This allows the image display method to be adjusted based on the border region, thus adapting the image display method to the screen display method and ultimately improving the user's interactive experience. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0019] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A schematic flowchart of a border detection method provided in an embodiment of this disclosure;

[0021] Figure 2 A schematic flowchart of another border detection method provided in this disclosure embodiment;

[0022] Figure 3a A target image provided in an embodiment of this disclosure;

[0023] Figure 3b A type of activation heatmap provided in this disclosure embodiment;

[0024] Figure 4 A line graph provided in an embodiment of this disclosure;

[0025] Figure 5 A flowchart illustrating another border detection method provided in this disclosure embodiment;

[0026] Figure 6 This is a schematic diagram of the structure of a border detection device provided in an embodiment of the present disclosure;

[0027] Figure 7 This is a schematic diagram of the structure of a border detection device provided in an embodiment of this disclosure. Detailed Implementation

[0028] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0029] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0030] Taking video images as an example, during the user's video playback, the electronic device will adjust the horizontal video to a vertical video by filling the top and bottom borders and then embed it in the vertical screen for display, or adjust the vertical video to a horizontal video by filling the left and right borders and then embed it in the horizontal screen for display, so that the displayed video is adapted to the screen display mode and improves the user's video viewing experience.

[0031] Optionally, the fill border can be a solid color border, a blurred border, or a colored border.

[0032] Understandably, while videos with filled borders may look aesthetically pleasing, they can negatively impact subsequent video processing. Generally, subsequent video processing requires cropping out content outside the border area and only processing the content within the border region. Therefore, the accuracy of image border detection significantly affects subsequent image processing.

[0033] In existing technologies, traditional border detection methods typically involve scanning an image line by line and calculating scan values ​​to detect and locate borders. However, this approach requires a border threshold to accurately detect and locate border regions, and this threshold is difficult to determine. Therefore, traditional border detection methods cannot accurately detect border regions; for example, the identified border regions may cover important image information. Consequently, existing border detection methods are difficult to widely apply.

[0034] To address the aforementioned problems, this disclosure provides a border detection method. Figure 1 A schematic flowchart of a border detection method provided in an embodiment of this disclosure is shown.

[0035] like Figure 1 As shown, the border detection method may include the following steps.

[0036] S110. Perform feature extraction on the target image to obtain a feature map corresponding to the target image. The target image includes a video content region and a background region. The boundary between the video content region and the background region is the target bounding box region to be identified in the target image. The feature value of each position in the feature map reflects the probability that the position belongs to the bounding box region.

[0037] In this embodiment of the disclosure, when it is necessary to detect the border of the target image, the electronic device can use a preset feature extraction algorithm to detect the video content region and the background region of the target image, and generate a feature map corresponding to the target image based on the feature values ​​of the video content region and the feature values ​​of each position in the background region.

[0038] In this embodiment of the disclosure, the target image can be any image that requires border detection. Specifically, the target image includes a video content area for displaying video content, a background area for displaying video content (which does not need to display content or content that the user will not pay attention to), and the boundary position is used as the target border area to be identified in the target image.

[0039] Optionally, the target image can be a still image or a video image.

[0040] In some embodiments, the preset feature extraction algorithm can be a pre-trained feature extraction model. This model extracts features layer by layer from each pixel in the target image to obtain a feature map corresponding to the target image. Specifically, the feature extraction model can be trained based on sample images and their corresponding labeled bounding box regions.

[0041] In other embodiments, an image feature extraction algorithm from a preset feature extraction algorithm can be used to extract image features from the target image and obtain a feature map corresponding to the target image. Optionally, the feature extraction algorithm can be an algorithm such as a feature descriptor operator.

[0042] Optionally, image features may include at least one of the following: color features, brightness features, contrast, grayscale features, texture features, etc.

[0043] It is understandable that since the feature values ​​at each position in the feature map reflect the probability that the position belongs to the bounding box region, the larger the feature value, the greater the probability that the position belongs to the bounding box region. Therefore, multiple positions with large and similar feature values ​​are selected from the feature map to determine the bounding box region.

[0044] In this feature map, the feature values ​​at each location can be values ​​that have a positive influence on bounding box detection.

[0045] Optionally, the feature value at each location in the feature map can be at least one of the following: color value, brightness value, contrast value, texture feature value, and grayscale value.

[0046] To further improve the accuracy of bounding box detection, prior to S110, the method also includes:

[0047] S101. Select the image with the border region from the target image;

[0048] Accordingly, S110 specifically includes the following steps:

[0049] S1101. Extract features from the image with the border region to obtain the feature map corresponding to the target image.

[0050] Images with bordered regions can be positive examples in the target image, while images without bordered regions can be negative examples in the target image.

[0051] Among them, an image with a border region can be an image in which a border is detected but the border region is not accurately determined.

[0052] In some embodiments, image features are extracted from the target image, and images with border regions are selected from the target image based on the image features. Then, feature extraction is performed on the images with border regions based on the image features to obtain a feature map corresponding to the target image.

[0053] In other embodiments, image features are extracted from the target image, and based on the image features, images with border regions are selected from the target image. Furthermore, a pre-trained feature extraction model is used to extract features from the border images to obtain a feature map corresponding to the target image.

[0054] In some other embodiments, a pre-trained classification model is used to select images with border regions from the target image, and image features are extracted from the images with border regions. Based on the image features, further feature extraction is performed on the images with border regions to obtain a feature map corresponding to the target image.

[0055] In some other embodiments, a pre-trained classification model is used to select images with border regions from the target image, and a pre-trained feature extraction model is used to extract features from the images with border regions to obtain feature maps corresponding to the target image.

[0056] The pre-trained classification model is trained using sample images and sample images with bounding box region annotations.

[0057] Therefore, in this embodiment of the disclosure, an image with a border region can be selected from the target image first, and then feature extraction can be performed on the image with the border region to obtain a feature map corresponding to the target image, which can improve the accuracy of feature extraction.

[0058] S120. Based on the feature values ​​at each location in the feature map, determine the feature values ​​of the candidate bounding box region in the target image.

[0059] It is understandable that the larger the feature value, the greater the probability that the position corresponding to that feature value appears in the bounding box region of the target image; conversely, the smaller the feature value, the smaller the probability that the position corresponding to that feature value appears in the bounding box region of the target image.

[0060] In some embodiments, when embedding a horizontal video within a portrait screen, top and bottom borders need to be filled in the horizontal video to convert it into a portrait video before embedding it in the portrait screen, thus adapting the video display method to the screen display method. Therefore, to fill the top and bottom borders, the feature values ​​of each row position can be determined based on the feature values ​​at each position in the feature map, and the probability that each row position belongs to the top and bottom border region can be determined based on the feature values ​​of each row position.

[0061] In other embodiments, for cases where a vertical video is embedded in a landscape screen, left and right borders need to be filled on the vertical video so that it can be converted into a landscape video and embedded in the landscape screen, thus adapting the video display method to the screen display method. Therefore, to fill the left and right borders, the feature values ​​of each column position can be determined based on the feature values ​​at each position in the feature map, and the probability that each column position belongs to the left or right border region can be determined based on the feature values ​​of each column position.

[0062] In other embodiments, left and right borders can be filled on the video. Then, based on the feature values ​​of each position in the feature map, the feature values ​​of each row position and each column position can be determined. Based on the feature values ​​of each row position and each column position, the probability that each row position belongs to the top and bottom border regions and the probability that each column position belongs to the left and right border regions can be determined.

[0063] S130. Based on the feature values ​​of the candidate bounding box regions, determine the target bounding box regions in the target image.

[0064] It is understandable that if the feature value of the candidate bounding box region is larger, the probability that the row or column corresponding to that feature value belongs to the target bounding box region in the target image is greater. Conversely, if the feature value of the candidate bounding box region is smaller, the probability that the row or column corresponding to that feature value belongs to the target bounding box region in the target image is smaller.

[0065] In some embodiments, if you want to fill the top and bottom borders, you can use the two rows with the largest feature values ​​of the candidate border regions as the target border regions in the target image.

[0066] In other embodiments, if it is desired to fill the left and right borders, the two columns with the largest feature values ​​of the candidate border regions can be used as the target border regions in the target image.

[0067] In other embodiments, if you want to fill the top and bottom borders and the left and right borders, you can use the two rows and two columns with the largest feature values ​​of the candidate border regions as the target border regions in the target image.

[0068] This disclosure provides a border detection method. First, feature extraction is performed on the target image to obtain a feature map corresponding to the target image. The target image includes a video content region and a background region. The boundary between the video content region and the background region is the target border region to be identified in the target image. The feature value of each position in the feature map reflects the probability that the position belongs to the border region. Then, based on the feature values ​​of each position in the feature map, the feature values ​​of candidate border regions in the target image are determined. Next, based on the feature values ​​of the candidate border regions, the target border region in the target image is determined. Through the above method, the border region in the target image can be accurately determined based on the probability that each position in the feature map belongs to the border region. This allows the image display method to be adjusted based on the border region, thus adapting the image display method to the screen display method and ultimately improving the user's interactive experience.

[0069] In another embodiment of this disclosure, a pre-trained feature extraction model can be used to extract features from the target image to obtain a category activation heatmap corresponding to the target image, which serves as a feature map corresponding to the target image.

[0070] Figure 2 A flowchart illustrating another border detection method provided in an embodiment of this disclosure is shown.

[0071] like Figure 2 As shown, the border detection method may include the following steps.

[0072] S210. Use a pre-trained feature extraction model to extract features from the target image and obtain the category activation heatmap corresponding to the target image, which serves as the feature map corresponding to the target image.

[0073] In this embodiment, a category activation heatmap can be used to detect which part of the target image contributes more to the final result. Specifically, it can be generated from the activation feature values ​​at various locations near the bounding box region in the target image. The larger the activation feature value in the category activation heatmap, the greater its contribution to the final result.

[0074] Optionally, in this embodiment of the disclosure, S210 may specifically include the following steps:

[0075] S2101. Extract gradient features of the target image through the target feature layer of the pre-trained feature extraction model, and generate a category activation heatmap based on the gradient features and the input features corresponding to the target feature layer, wherein the target feature layer is the last hidden layer in the pre-trained feature extraction model.

[0076] The pre-trained feature extraction model can be a model obtained based on the grad_cam algorithm.

[0077] The gradient feature can be the gradient of the target feature layer with respect to the current loss function.

[0078] In this context, the last hidden layer in the pre-trained feature extraction model can be understood as the feature extraction layer or the feature layer to be visualized. In other words, the target feature layer is either the feature extraction layer of the pre-trained feature extraction model or the feature layer to be visualized.

[0079] Specifically, S2101, "extracting gradient features of the target image through the target feature layer of a pre-trained feature extraction model," may include the following steps:

[0080] S1. Obtain the input and output features of the classification layer of the pre-trained feature extraction model;

[0081] S2. Using the target feature layer, determine the gradient features of the target image based on the input and output features of the classification layer.

[0082] Optionally, the gradient features of the target image can be determined in the following way:

[0083]

[0084] Where k is the channel dimension index of the feature map, c is the category index, and i and j are the width and height dimension indices, respectively. It is the gradient feature of the target image. It is the input feature of the classification layer, y c These are the output features of the classification layer. and y c It can be obtained through backpropagation of a pre-trained feature extraction model. It is a global average pooling process.

[0085] Specifically, S2101, "generating a category activation heatmap based on gradient features and the input features corresponding to the target feature layer," may include the following steps:

[0086] S3. Perform a weighted summation of the gradient features of all channels with the input features corresponding to the target feature layer to obtain the weighted summation result;

[0087] S4. Filter the weighted summation result to obtain the activation feature value corresponding to each position;

[0088] S5. Generate a category activation heatmap of the target image based on the activation feature values ​​corresponding to each location.

[0089] Optionally, the ReLU function can be used to filter the weighted summation result to obtain the activation feature value corresponding to each position.

[0090] Optionally, the activation feature value can be determined in the following way:

[0091]

[0092] Among them, ReLU(*) is the ReLU function.

[0093] Therefore, in this embodiment of the present disclosure, the gradient features of the target image can be extracted using the target feature layer of the pre-trained feature extraction model, so that the category activation heatmap corresponding to the target image can be accurately generated based on the extracted gradient features of the target image, thus obtaining the feature map corresponding to the target image. Furthermore, the bounding region is determined based on the category activation heatmap, thereby improving the accuracy of the bounding region determination method.

[0094] It should be noted that, in order to make the detected bounding box region fit the target image, after obtaining the category activation heatmap, the image size of the category activation heatmap can be adjusted to be the same as the image size of the target image.

[0095] For ease of understanding, Figure 3a The target image is shown. Figure 3b A category activation heatmap is shown.

[0096] Figure 3b The category activation heatmap includes Figure 3a Feature activation values ​​at various locations within the middle border region.

[0097] S220. Based on the feature values ​​at each location in the feature map, determine the feature values ​​of the candidate bounding box region in the target image.

[0098] S230. Based on the feature values ​​of the candidate bounding box regions, determine the target bounding box regions in the target image.

[0099] S220 to S230 are similar to S120 to S130, and will not be described in detail here.

[0100] In another embodiment of this disclosure, the feature values ​​at each position can be summed row by row in the horizontal direction, and the target border region of the target image can be determined based on the sum of the feature values ​​in the horizontal direction; and / or, the feature values ​​at each position can be summed row by row in the vertical direction, and the target border region of the target image can be determined based on the sum of the feature values ​​in the vertical direction.

[0101] In some embodiments of this disclosure, S120 may specifically include the following steps:

[0102] S1201. If the video content area is a vertical screen area and a horizontal video is displayed in the video content area, then the feature values ​​at each position are summed row by row in the horizontal direction to obtain the sum of the feature values ​​in the horizontal direction, and the sum of the feature values ​​in the horizontal direction is used as the feature values ​​of the candidate border area.

[0103] S130 may specifically include the following steps:

[0104] S1301. The candidate bounding box region corresponding to the sum of the largest eigenvalues ​​in the horizontal direction is taken as the target bounding box region of the target image.

[0105] Understandably, for the scenario of embedding a horizontal video within a vertical screen—that is, the video content area is vertical, and the horizontal video is displayed within that area—top and bottom borders need to be filled on the horizontal video. This requires determining the top and bottom borders based on the feature values ​​at each location. Since the feature values ​​at each location reflect the probability that that location belongs to a border region in the target image, the sum of the feature values ​​in the horizontal direction reflects the probability that each row in the target image will form a border. Therefore, the larger the sum of the feature values ​​in the horizontal direction, the greater the probability that the row corresponding to that feature value will form a border; conversely, the smaller the sum of the feature values ​​in the horizontal direction, the smaller the probability that the row corresponding to that feature value will form a border.

[0106] It should be noted that the feature values ​​at each location in this embodiment can be the feature activation values ​​at each location of the category activation heatmap mentioned in the above embodiments, and the feature activation values ​​at each location of the category activation heatmap are used for illustrative purposes.

[0107] To facilitate understanding, the target bounding box region of the target image is determined in the horizontal direction. Figure 4 A line graph is shown. 41 is the line graph corresponding to the category activation heatmap, and 42 is the line graph corresponding to the target image.

[0108] like Figure 4 As shown, the horizontal coordinates of both polyline 41 and polyline 42 are the row numbers, the vertical coordinate of polyline 41 is the sum of the activation feature values ​​at each row position, and the vertical coordinate of polyline 42 is the sum of the image features at each row position.

[0109] like Figure 4 As shown, there are two maximum activation feature values ​​summed on line 41. The sums of the two maximum activation feature values ​​on line 41 correspond to the 80th and 230th rows, respectively. Furthermore, the sum of the image features of the target image changes abruptly in the 80th and 230th rows, indicating that the target image has upper and lower target bounding box regions in the 80th and 230th rows.

[0110] Therefore, in this embodiment of the disclosure, the target border region in the horizontal direction of the target image is accurately determined based on the sum of the feature values ​​in the horizontal direction.

[0111] In some embodiments of this disclosure, S120 may specifically include the following steps:

[0112] S1202. If the video content area is a landscape area and a vertical video is displayed in the video content area, then the feature values ​​at each position are summed column by column in the vertical direction to obtain the sum of the feature values ​​in the vertical direction, and the sum of the feature values ​​in the vertical direction is used as the feature values ​​of the candidate border area.

[0113] S130 may specifically include the following steps:

[0114] S1302. The candidate bounding box region corresponding to the sum of the largest eigenvalues ​​in the vertical direction is taken as the target bounding box region of the target image.

[0115] Understandably, for the scenario of embedding a vertical video within a landscape screen—that is, the video content area is the landscape area, and the vertical video is displayed within that area—left and right borders need to be filled on the vertical video. This requires determining the left and right borders based on the feature values ​​at each location. Since the feature values ​​at each location reflect the probability that that location belongs to a border region in the target image, the sum of the feature values ​​in the vertical direction reflects the probability that each column position in the target image will form a border. Therefore, the larger the sum of the feature values ​​in the vertical direction, the greater the probability that the row position corresponding to that feature value will form a border; conversely, the smaller the sum of the feature values ​​in the vertical direction, the smaller the probability that the column position corresponding to that feature value will form a border.

[0116] It should be noted that the feature values ​​at each location in the embodiments of this disclosure can also be the feature activation values ​​at each location of the category activation heatmap mentioned in the above embodiments.

[0117] Therefore, in this embodiment of the disclosure, the target border region in the vertical direction of the target image is accurately determined based on the sum of the feature values ​​in the vertical direction.

[0118] In some embodiments of this disclosure, S120 may specifically include the following steps:

[0119] S1203. Sum the feature values ​​at each position row by row in the horizontal direction to obtain the sum of the feature values ​​in the horizontal direction, and use the sum of the feature values ​​in the horizontal direction as the feature values ​​of the candidate bounding box region.

[0120] S1204. Summing the feature values ​​at each position column by column in the vertical direction to obtain the sum of the feature values ​​in the vertical direction, and using the sum of the feature values ​​in the vertical direction as the feature values ​​of the candidate bounding box region.

[0121] S130 may specifically include the following steps:

[0122] S1303. The candidate bounding box region corresponding to the sum of the largest eigenvalues ​​in the horizontal direction is taken as the target bounding box region of the target image.

[0123] S1304. The candidate bounding box region corresponding to the sum of the largest eigenvalues ​​in the vertical direction is taken as the target bounding box region of the target image.

[0124] Therefore, in this embodiment of the disclosure, the target border regions in the horizontal and vertical directions of the target image are determined based on the sum of feature values ​​in the horizontal direction and the sum of feature values ​​in the vertical direction, respectively.

[0125] In another embodiment of this disclosure, an initial border region of the target image can be obtained, and a target border region in the target image can be determined based on the feature values ​​of the initial border region and the candidate border regions.

[0126] Figure 5 A flowchart illustrating another border detection method provided in an embodiment of this disclosure is shown.

[0127] like Figure 5 As shown, the border detection method may include the following steps.

[0128] S510, Obtain the initial border region of the target image.

[0129] In this embodiment of the disclosure, after acquiring the target image, the electronic device can detect and locate the border of the target image to obtain the image features of the target image, and obtain the initial border region of the target image based on the image features of the target image.

[0130] In this embodiment of the disclosure, the initial border region may be a rough border region of the target image.

[0131] S520. Perform feature extraction on the target image to obtain a feature map corresponding to the target image. The target image includes a video content region and a background region. The boundary between the video content region and the background region is the target bounding box region to be identified in the target image. The feature value of each position in the feature map reflects the probability that the position belongs to the bounding box region.

[0132] S530. Based on the feature values ​​at each position in the feature map, determine the feature values ​​of the candidate bounding box region in the target image.

[0133] S520 to S530 are similar to S110 to S120, and will not be described in detail here.

[0134] S540. Determine the target border region in the target image based on the feature values ​​of the initial border region and the candidate border region.

[0135] Optionally, in this embodiment, S530 may specifically include the following steps:

[0136] S5301. Determine the border region to be determined in the target image based on the feature values ​​of the candidate border region.

[0137] S5302. The bounding region to be determined is used as the constraint condition of the initial bounding region to obtain the target bounding region in the target image.

[0138] Specifically, for S5301, the feature values ​​of the candidate bounding box region can be the sum of feature values ​​in the horizontal direction and / or the sum of feature values ​​in the vertical direction. Then, the candidate bounding box region corresponding to the maximum sum of feature values ​​in the horizontal direction and / or the candidate bounding box region corresponding to the sum of feature values ​​in the vertical direction will be used as the bounding box region to be determined in the target image.

[0139] For S5302, the initial border region of the border region to be determined within a preset range can be obtained as the target border region in the target image.

[0140] The preset range can be a pixel range.

[0141] Optionally, the preset range can be 50 pixels. That is, the initial border region within 50 pixels of the border region to be determined is obtained as the target border region in the target image, and the initial border region exceeding 50 pixels of the border region to be determined is discarded.

[0142] Therefore, in this embodiment of the disclosure, the initial border region of the target image can be obtained, and the target border region in the target image can be determined by combining the feature values ​​of the initial border region and the candidate border region, thereby further improving the accuracy of border detection.

[0143] Based on the same inventive concept as the above-described method embodiments, this disclosure also provides a border detection device, with reference to... Figure 6 This is a schematic diagram of the structure of a border detection device 600 provided in an embodiment of this disclosure. The device includes:

[0144] The feature map acquisition module 610 is used to extract features from the target image and acquire a feature map corresponding to the target image. The target image includes a video content region and a background region. The boundary between the video content region and the background region is the target bounding box region to be identified in the target image. The feature value of each position in the feature map reflects the probability that the position belongs to the bounding box region.

[0145] The feature value determination module 620 is used to determine the feature value of the candidate bounding box region in the target image based on the feature values ​​at each position in the feature map;

[0146] The border region determination module 630 is used to determine the target border region in the target image based on the feature values ​​of the candidate border regions.

[0147] This disclosure provides a border detection device. First, feature extraction is performed on a target image to obtain a feature map corresponding to the target image. The target image includes a video content region and a background region. The boundary between the video content region and the background region is the target border region to be identified in the target image. The feature value of each position in the feature map reflects the probability that the position belongs to the border region. Then, based on the feature values ​​of each position in the feature map, the feature values ​​of candidate border regions in the target image are determined. Next, based on the feature values ​​of the candidate border regions, the target border region in the target image is determined. Through the above method, the border region in the target image can be accurately determined based on the probability that each position in the feature map belongs to the border region. This allows the image display method to be adjusted based on the border region, thus adapting the image display method to the screen display method and ultimately improving the user's interactive experience.

[0148] In one optional implementation, the feature map acquisition module 610 is specifically used to extract features from the target image using a pre-trained feature extraction model to obtain a category activation heatmap corresponding to the target image, which serves as the feature map corresponding to the target image.

[0149] In one optional implementation, the feature map acquisition module 610 is specifically used for,

[0150] The gradient features of the target image are extracted through the target feature layer of the pre-trained feature extraction model, and...

[0151] The category activation heatmap is generated based on the gradient features and the input features corresponding to the target feature layer, wherein the target feature layer is the last hidden layer in the pre-trained feature extraction model.

[0152] In one optional embodiment, the device further includes:

[0153] An image selection module is used to select an image with a bordered region from the target image;

[0154] Accordingly, the feature map acquisition module 610 is specifically used to extract features from the image with the border region and obtain a feature map corresponding to the target image.

[0155] In one optional implementation, the feature value determination module 620 includes:

[0156] The row-by-row summation unit is used to perform row-by-row summation of the feature values ​​at each position in the horizontal direction if the video content area is a vertical screen area and a horizontal video is displayed in the video content area, so as to obtain the sum of the feature values ​​in the horizontal direction, and use the sum of the feature values ​​in the horizontal direction as the feature value of the candidate border area.

[0157] The border area determination module 630 includes:

[0158] The first border determination unit is used to take the candidate border region corresponding to the sum of the maximum feature values ​​in the horizontal direction as the target border region of the target image.

[0159] In one optional implementation, the feature value determination module 620 includes:

[0160] The column-by-column summation unit is used to sum the feature values ​​of each position in the vertical direction if the video content area is a horizontal screen area and a vertical video is displayed in the video content area, so as to obtain the sum of the feature values ​​in the vertical direction, and use the sum of the feature values ​​in the vertical direction as the feature value of the candidate border area.

[0161] The border area determination module 630 includes:

[0162] The first border determination unit is used to take the candidate border region corresponding to the sum of the maximum feature values ​​in the vertical direction as the target border region of the target image.

[0163] In one optional embodiment, the device further includes:

[0164] An initial border region acquisition module is used to acquire the initial border region of the target image;

[0165] Accordingly, the border region determination module 630 is specifically used to determine the target border region in the target image based on the feature values ​​of the initial border region and the candidate border region.

[0166] In addition to the methods and apparatus described above, this disclosure also provides a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to implement the border detection method of this disclosure.

[0167] This disclosure also provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, they implement the border detection method of this disclosure.

[0168] Figure 7 A schematic diagram of a border detection device provided in an embodiment of this disclosure is shown. This border detection device may be the electronic device described in the foregoing embodiments.

[0169] like Figure 7 As shown, the border detection device may include a controller 701 and a memory 702 storing computer program instructions.

[0170] Specifically, the controller 701 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0171] Memory 702 may include a large-capacity storage for information or instructions. For example, and not limitingly, memory 702 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 702 may include removable or non-removable (or fixed) media. Where appropriate, memory 702 may be internal or external to the integrated gateway device. In a particular embodiment, memory 702 is a non-volatile solid-state memory. In a particular embodiment, memory 702 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (Electrically Programmable ROM, EPROM), an electrically erasable programmable PROM (EEPROM), an electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0172] The controller 701 performs the steps of the border detection method provided in the embodiments of this disclosure by reading and executing computer program instructions stored in the memory 702.

[0173] In one example, the border detection device may further include a transceiver 703 and a bus 704. Wherein, as... Figure 7 As shown, the controller 701, memory 702 and transceiver 703 are connected via bus 704 and communicate with each other.

[0174] Bus 704 includes hardware, software, or both. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 704 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0175] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0176] The above are merely specific embodiments of this disclosure, enabling those skilled in the art to understand or implement this disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to these embodiments, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A border detection method, characterized in that, The method includes: Feature extraction is performed on the target image to obtain a feature map corresponding to the target image. The target image includes a video content region and a background region. The boundary between the video content region and the background region is the target bounding box region to be identified in the target image. The feature value of each position in the feature map reflects the probability that the position belongs to the bounding box region. Based on the feature values ​​at each location in the feature map, determine the feature values ​​of the candidate bounding box region in the target image; Based on the feature values ​​of the candidate bounding box regions, the target bounding box region in the target image is determined; The step of extracting features from the target image to obtain a feature map corresponding to the target image includes: The gradient features of the target image are extracted through the target feature layer of a pre-trained feature extraction model, and... A category activation heatmap is generated based on the gradient features and the input features corresponding to the target feature layer, which serves as the feature map corresponding to the target image. The target feature layer is the last hidden layer in the pre-trained feature extraction model.

2. The method according to claim 1, characterized in that, Before performing feature extraction on the target image to obtain the feature map corresponding to the target image, the method further includes: Select an image with a bordered region from the target image; Accordingly, the step of extracting features from the target image to obtain a feature map corresponding to the target image includes: Feature extraction is performed on the image with the border region to obtain a feature map corresponding to the target image.

3. The method according to claim 1, characterized in that, The step of determining the feature values ​​of the candidate bounding box region in the target image based on the feature values ​​at each position in the feature map includes: If the video content area is a vertical screen area and a horizontal video is displayed in the video content area, then the feature values ​​at each position are summed row by row in the horizontal direction to obtain the sum of the feature values ​​in the horizontal direction, and the sum of the feature values ​​in the horizontal direction is used as the feature value of the candidate border area. Determining the target border region in the target image based on the feature values ​​of the candidate border regions includes: The candidate bounding box region corresponding to the sum of the maximum feature values ​​in the horizontal direction is taken as the target bounding box region of the target image.

4. The method according to claim 1, characterized in that, The step of determining the feature values ​​of the candidate bounding box region in the target image based on the feature values ​​at each position in the feature map includes: If the video content area is a landscape area and a vertical video is displayed in the video content area, then the feature values ​​at each position are summed column by column in the vertical direction to obtain the sum of the feature values ​​in the vertical direction, and the sum of the feature values ​​in the vertical direction is used as the feature value of the candidate border area. Determining the target border region in the target image based on the feature values ​​of the candidate border regions includes: The candidate bounding box region corresponding to the sum of the maximum eigenvalues ​​in the vertical direction is taken as the target bounding box region of the target image.

5. The method according to any one of claims 1 to 4, characterized in that, Before determining the target bounding box region in the target image based on the feature values ​​of the candidate bounding box regions, the method further includes: Obtain the initial border region of the target image; Accordingly, determining the target bounding box region in the target image based on the feature values ​​of the candidate bounding box regions includes: The target border region in the target image is determined based on the feature values ​​of the initial border region and the candidate border region.

6. A border detection device, characterized in that, The device includes: The feature map acquisition module is used to extract features from the target image and acquire a feature map corresponding to the target image. The target image includes a video content region and a background region. The boundary between the video content region and the background region is the target bounding box region to be identified in the target image. The feature value of each position in the feature map reflects the probability that the position belongs to the bounding box region. The feature value determination module is used to determine the feature value of the candidate bounding box region in the target image based on the feature values ​​at each position in the feature map; The border region determination module is used to determine the target border region in the target image based on the feature values ​​of the candidate border regions; The feature map acquisition module is specifically used for: The gradient features of the target image are extracted through the target feature layer of a pre-trained feature extraction model, and... A category activation heatmap is generated based on the gradient features and the input features corresponding to the target feature layer, which serves as the feature map corresponding to the target image. The target feature layer is the last hidden layer in the pre-trained feature extraction model.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a terminal device, cause the terminal device to perform the method as described in any one of claims 1-5.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as described in any one of claims 1-5.

9. A computer program product, characterized in that, The computer program product includes a computer program / instruction that, when executed by a processor, implements the method as described in any one of claims 1-5.

Images