Target area extraction for digital content addition

Abstract

The present disclosure provides a method, apparatus and computer program product for extracting a target region for adding digital content. An image may be obtained. At least one initial quadrilateral region may be detected in the image. A line segment set may be detected in the image. The initial quadrilateral region may be calibrated using the line segment set to obtain a target region for adding digital content.

Description

Background Art

[0001] With the development of digital devices, communication technology, video coding technology, etc., people can use terminal devices such as smart phones, desktop computers, tablet computers, etc. to easily access videos on the Internet. In this article, video can widely refer to various visual contents that can be presented on terminal devices, such as movies, video news, variety shows, video speeches, game interfaces, online education videos, etc. People can watch videos online smoothly through video service platforms, or download videos from them to watch later. The traffic generated by accessing videos has occupied a large proportion of the total Internet traffic, and this proportion is still growing. It is desirable to insert specific digital content in a video and make the digital content visible to the public as the video is played. In this article, digital content can widely refer to content intended to be presented together with a video for various purposes, such as commercial information, public service announcements, etc. The format of digital content can include pictures, videos, animations, etc. Summary of the invention

[0002] This Summary is provided to introduce a group of concepts that will be further described in the following Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0003] Embodiments of the present disclosure provide a method, apparatus, and computer program product for extracting a target region for adding digital content. An image may be obtained. At least one initial quadrilateral region may be detected in the image. A line segment set may be detected in the image. The initial quadrilateral region may be calibrated using the line segment set to obtain a target region for adding digital content.

[0004] It should be noted that one or more of the above aspects include the features specifically pointed out in the following detailed description and claims. The following description and drawings set forth in detail certain illustrative features of the one or more aspects. These features are merely indicative of the various ways in which the principles of various aspects may be implemented, and the present disclosure is intended to include all such aspects and their equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The disclosed aspects will be described below in conjunction with the accompanying drawings, which are provided to illustrate rather than limit the disclosed aspects.

[0006] Figure 1 An exemplary process of extracting a target area for adding digital content according to an embodiment of the present disclosure is shown.

[0007] Figure 2 is a schematic diagram showing a target area and its circumscribed rectangle according to an embodiment of the present disclosure.

[0008] Figure 3 An exemplary process for detecting an initial quadrilateral region in an image according to an embodiment of the present disclosure is shown.

[0009] Figure 4 An exemplary process for directly calibrating an initial quadrilateral region by using a set of line segments according to an embodiment of the present disclosure is shown.

[0010] Figure 5 is a schematic diagram showing the calculation of the distance between two line segments according to an embodiment of the present disclosure.

[0011] Figure 6 FIG. 4 is a schematic diagram showing an example of directly calibrating an initial quadrilateral region by using a set of line segments according to an embodiment of the present disclosure.

[0012] Figure 7 An exemplary process for calibrating an initial quadrilateral region using a reference quadrilateral region constructed by a set of line segments according to an embodiment of the present disclosure is shown.

[0013] Figure 8 A set of exemplary adjacency types according to an embodiment of the present disclosure is shown.

[0014] Fig. 9 An exemplary invalid reference quadrilateral region according to an embodiment of the present disclosure is shown.

[0015] Fig.10 is a schematic diagram showing an example of calibrating an initial quadrilateral region using a reference quadrilateral region constructed by a set of line segments according to an embodiment of the present disclosure.

[0016] Fig.11 is a flowchart of an exemplary method for extracting a target area for adding digital content according to an embodiment of the present disclosure.

[0017] Fig.12 An exemplary device for extracting a target area for adding digital content according to an embodiment of the present disclosure is shown.

[0018] Fig.13 An exemplary device for extracting a target area for adding digital content according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0019] The present disclosure will now be discussed with reference to several exemplary embodiments. It should be understood that the discussion of these embodiments is only used to enable those skilled in the art to better understand and thereby implement the embodiments of the present disclosure, and does not teach any limitation on the scope of the present disclosure.

[0020] There are some existing ways of inserting digital content in video. One way can provide digital content independently of video. For example, digital content can be inserted before, during or after video playing, and these ways can be called pre-roll, mid-roll or post-roll accordingly. Another way can provide digital content in a suspended way on the screen of video, for example, by making digital content suspended on the partial screen in video to play digital content. Another way can extract the target area for adding digital content in one or more frames of video, and add digital content to the extracted target area.

[0021] The embodiments of the present disclosure provide an improved method for extracting a target region for adding digital content, which can be performed on an image. For example, at least one initial quadrilateral region and a line segment set can be detected in the image, and the initial quadrilateral region can be calibrated using the line segment set to obtain a target region for adding digital content.

[0022] In one aspect, an embodiment of the present disclosure proposes detecting an initial quadrilateral region in an image by means of a trained machine learning model. In this article, the machine learning model used to detect an initial quadrilateral region in an image may be referred to as a quadrilateral detection model. The quadrilateral detection model may predict the positions of the center points and corner points in the image, and combine the center points and corner points belonging to the same quadrilateral region to determine at least one initial quadrilateral region. The training data of the quadrilateral detection model may include a plurality of training images, and each training image may be marked with one or more quadrilateral regions suitable for adding digital content. The quadrilateral region suitable for adding digital content may be, for example, a photo frame, a poster, a bulletin board, and the like. The quadrilateral detection model trained using such training data may detect a quadrilateral region suitable for adding digital content in an input image when it is actually deployed.

[0023] In another aspect, the embodiments of the present disclosure propose to calibrate the initial quadrilateral region using a line segment set detected in an image. For example, the line segment set can be used to calibrate at least one side of the initial quadrilateral region. Additionally or alternatively, the initial quadrilateral region can be calibrated using a reference quadrilateral region constructed by the line segment set. The calibrated quadrilateral region obtained by calibrating the detected initial quadrilateral region using the line segment set can be more closely aligned with the quadrilateral region in the image. The digital content added to the quadrilateral region calibrated in this way can be more harmonious with other parts in the image.

[0024] In yet another aspect, the embodiments of the present disclosure propose that after the initial quadrilateral area is calibrated to obtain the target area for adding digital content, it is possible to further verify whether the target area is suitable for adding digital content based on the content of the target area. For example, the target area can be classified into a target area suitable for adding digital content and a target area not suitable for adding digital content by a trained classification model. If the target area is classified as a target area suitable for adding digital content, the target area can be retained. If the target area is classified as a target area not suitable for adding digital content, the target area can be removed.

[0025] In yet another aspect, an embodiment of the present disclosure proposes that when an image is a group of images from a video, a group of target regions corresponding to the group of images can be obtained. Each target region in the group of target regions can be scored, and one or more target regions with the highest scores can be selected from the group of target regions for adding digital content. In particular, when scoring the target regions, the target regions can be scored based on quality metrics such as the confidence, aspect ratio, area, rectangularity, clarity, etc. of the target regions. These quality metrics can comprehensively reflect the quality of the target regions, so that the quality of the target regions can be accurately evaluated to facilitate the selection of the target regions.

[0026] Figure 1 Exemplary processes 100a and 100b of target region extraction for adding digital content according to an embodiment of the present disclosure are shown. Through process 100a, a target region for adding digital content can be extracted from an image. In the case of having a group of images, a group of target regions corresponding to the group of images can be obtained through process 100b, and one or more target regions can be selected from the group of target regions for adding digital content.

[0027] At 102, an image may be obtained. The image may be, for example, a single image existing independently by itself, or one of a group of images extracted from various types of files such as videos, web pages, emails, productivity tool documents, and the like.

[0028] At 104, at least one initial quadrilateral region may be detected in the image. In one embodiment, at least one initial quadrilateral region may be detected in the image by using a trained quadrilateral detection model. Figure 3hereinafter is used to illustrate an exemplary structure of a quadrilateral detection model and an exemplary process of detecting an initial quadrilateral region. The training data of the quadrilateral detection model may include multiple training images, and each training image may be annotated with one or more quadrilateral regions suitable for adding digital content. Quadrilateral regions suitable for adding digital content may include, for example, photo frames, picture frames, easels, menus, posters, placards, New Year paintings, hanging paintings, banners, mirrors, bulletin boards, box facades, unopened screens, etc. When the quadrilateral detection model trained using such training data is actually deployed, it can detect quadrilateral regions suitable for adding digital content in the input image.

[0029] At 106, a set of line segments may be detected in the image. The set of line segments may be detected in the image by various known line segment detection techniques such as Line Segment Detector (LSD), Attraction Field Map (AFM), Edge Drawing Lines (EDLines), etc.

[0030] At 108, the initial quadrilateral region may be calibrated using the line segment set to obtain a target region for adding digital content. In the first embodiment, the initial quadrilateral region may be calibrated directly using the line segment set. Figure 4 In the second embodiment, the initial quadrilateral region can be calibrated using a reference quadrilateral region constructed by a set of line segments. Figure 7 To illustrate an exemplary process of calibrating an initial quadrilateral region using a reference quadrilateral region constructed by a set of line segments. The two embodiments can be performed independently of each other. For example, for a specific initial quadrilateral region, any one of the embodiments can be used to calibrate it. Alternatively, the two embodiments can be performed in combination with each other. For example, for a specific initial quadrilateral region, the first embodiment can be used to calibrate it, and then the second embodiment can be used to further calibrate it.

[0031] At 110, it can be further verified whether the target area is suitable for adding digital content. Through the operation at 104, an initial quadrilateral area suitable for adding digital content can be detected in the image. At 110, the target area obtained by calibrating the initial quadrilateral area can be further verified to further determine whether the target area is suitable for adding digital content. For example, it can be determined based on the content of the target area whether the target area is suitable for adding digital content. In one embodiment, the target area can be classified into a target area suitable for adding digital content and a target area not suitable for adding digital content by a trained classification model. If the target area is classified as a target area suitable for adding digital content, the target area can be retained. If the target area is classified as a target area not suitable for adding digital content, the target area can be removed. The classification model can be any known binary classification model such as a mobile network (MobileNet) model, an efficient network (EffcientNet) model, etc. The target area can be first expanded to a rectangle, and then the size of the rectangle is adjusted to the size required by the classification model by operations such as cutting and standardization, thereby obtaining an input image of the classification model. The classification model may output a prediction value indicating that the input image is suitable for adding digital content, or a prediction value indicating that the input image is not suitable for adding digital content. The training data used to train the classification model may include a plurality of training images, each training image being labeled accordingly as to whether it is suitable for adding digital content. As an example, training images of the following contents may be labeled as suitable for adding digital content: photo frames, picture frames, easels, menus, posters, placards, New Year paintings, hanging paintings, banners, mirrors, bulletin boards, box facades, unopened screens, etc.

[0032] Through steps 102 to 110 in process 100a, a target area for adding digital content can be extracted from an image.

[0033] The image may be one of a group of images extracted from various types of files such as videos, web pages, emails, productivity tool documents, etc. For example, the image may be a group of images from a video. Preferably, the group of images may be shot using different camera parameters. During the shooting of the video, the camera may move, and accordingly, the camera parameters may change. The video may be processed by a multimedia processing tool such as FFmpeg to segment the video into multiple video segments according to the camera movement or the camera parameters used to shoot the video. One or more frames of images may be extracted from each video segment to form a group of images corresponding to the video.

[0034] A set of target areas corresponding to a set of images may be obtained through process 100 b , and one or more target areas may be selected from the set of target areas for adding digital content.

[0035] At 112, a set of target regions corresponding to a set of images may be obtained. For example, for each image in a set of images, a target region for adding digital content may be extracted from the image through steps 102 to 110 in process 100a. Accordingly, a set of target regions may be obtained.

[0036] At 114, each target region in the set of target regions may be scored. For example, the target regions may be scored based on the quality of the target regions. In one embodiment, the target regions may be scored based on one or more of confidence, aspect ratio, area, rectangularity, and clarity of the target regions.

[0037] The confidence of the target area may include a detection confidence associated with the initial quadrilateral area detection performed by the quadrilateral detection model at 104. The detection confidence may be scored to obtain a detection confidence score P(det). The target area may have four corner points. The quadrilateral detection model may provide the position of each corner point of the predicted target area and the heat value of the corner point. The heat value of the corner point may indicate the probability that the point at the position is the corner point. The detection confidence score P(det) may be calculated by summing and averaging the heat values ​​of the four corner points of the target area provided by the quadrilateral detection model, as shown in the following formula:

[0038]

[0039] in, Represents the thermal value of corner point i in the target area.

[0040] The confidence of the target region may also include classification confidence related to the classification at 110. A classification confidence score related to the classification confidence may be represented as P(cls), which may be provided by the classification model while outputting a prediction value indicating whether the input image is suitable for adding digital content.

[0041] The aspect ratio of the target region may reflect the ratio between the width and the height of the target region. The aspect ratio of the target region may be scored to obtain an aspect ratio score P(aspect) of the target region. The aspect ratio score P(aspect) may be calculated, for example, by the following formula:

[0042]

[0043] Where w and h represent the width and height of the target area, respectively, and TH aspect Represents the aspect ratio related threshold.

[0044] The area of ​​the target region may be scored to obtain an area score P(area) of the target region. The area score P(area) may be calculated, for example, by the following formula:

[0045]

[0046] Where w and h represent the width and height of the target area, W and H represent the width and height of the image where the target area is located, and [TH area ,TH′ area ] represents the threshold range related to the area.

[0047] The rectangular degree of the target area can reflect the degree of proximity between the target area and the rectangle. The rectangular degree of the target area can be scored to obtain a rectangular degree score P(geo) of the target area. In one embodiment, the bounding rectangle of the target area can be first determined, and then the distance between each vertex of the target area and the corresponding vertex of the bounding rectangle is obtained by connecting the vertex of the target area with the corresponding vertex of the bounding rectangle. Figure 2 200 is a schematic diagram showing a target area and its circumscribed rectangle according to an embodiment of the present disclosure. Target area 202 and its corresponding circumscribed rectangle 204 are shown in FIG. A vector 210 from the upper left corner point 206 to the upper left corner point 208 can be obtained by connecting the upper left corner point 206 of the circumscribed rectangle 204 with the upper left corner point 208 of the target area 202. The vector 210 can be decomposed into components 216 along the side 212 in the θ0 direction of the circumscribed rectangle 204. and the component 218 of the side 214 along the θ1 direction of the circumscribed rectangle 204 Then, the difference ratio of the θ0 direction between the upper left corner point 206 and the upper left corner point 208 can be calculated by, for example, the following formula: The difference ratio in the θ1 direction

[0048]

[0049]

[0050] in, and Respectively represent the lengths of side 212 and side 214.

[0051] The above operation can be performed for each corner point of the target area 202 to obtain a set of difference ratios: Then, the maximum value may be selected from the set of difference ratios as the comprehensive difference ratio x corresponding to the target area 202. The rectangular degree score P(geo) may be calculated, for example, based on the comprehensive difference ratio x by the following formula:

[0052]

[0053] It should be understood that the above process is only an example for calculating the rectangular degree score P(geo). The rectangular degree score P(geo) can also be calculated in other ways. For example, when decomposing the vector 210, the vector 210 can also be decomposed into a horizontal component and a vertical component, and the difference ratio is calculated based on the horizontal component and the vertical component accordingly.

[0054] The clarity of the target area may be scored to obtain a clarity score P(blur) of the target area. The clarity score P(blur) may be calculated, for example, by the following formula:

[0055]

[0056] Wherein, nrss represents the sharpness index corresponding to the target area obtained according to the No Reference Structural Sharpness (NRSS) processing algorithm, and [TH blur ,TH′ blur ] indicates the threshold range associated with this indicator.

[0057] After obtaining the detection confidence score P(det), classification confidence score P(cls), aspect ratio score P(aspect), area score P(area), rectangular degree score P(geo), and clarity score P(blur) of the target area, the comprehensive score P of the target area can be calculated. In one embodiment, the comprehensive score P can be calculated as the product of the detection confidence score P(det), the classification confidence score P(cls), the aspect ratio score P(aspect), the area score P(area), the rectangular degree score P(geo), and the clarity score P(blur), as shown in the following formula:

[0058] P=P(det)·P(cls)·P(aspect)·P(area)·P(geo)·P(blur) (8)

[0059] It should be understood that the above formula (8) is merely exemplary, and the comprehensive score P can also be generated based on any other calculation method, or based on any one or more of the detection confidence score P(det), classification confidence score P(cls), aspect ratio score P(aspect), area score P(area), rectangular degree score P(geo) and clarity score P(blur).

[0060] After obtaining the comprehensive score of each target region in the set of target regions at 114, the set of target regions may be updated at 116. In one embodiment, the set of target regions may be updated by removing at least one of the following from the set of target regions: a repeated target region, an occluded target region, a target region that fails to be tracked, a target region with a duration below a threshold, and a target region with a score below a threshold.

[0061] For example, for each target area in the group of target areas, it can be determined whether there is a target area that repeats the target area. In one embodiment, a group of neighbor images adjacent to the image where the target area is located in a group of images, and a group of neighbor target areas in the group of neighbor images can be first obtained, wherein a group of neighbor target areas is a target area extracted in the group of neighbor images. Then, the features of the target area and the features of each neighbor target area in the group of neighbor target areas can be generated by a known method, and the features can be, for example, Scale-Invariant Feature Transform (SIFT) features. Subsequently, the SIFT features of each neighbor target area can be matched with the SIFT features of the target area. If the SIFT features of the neighbor target area match the SIFT features of the target area, it indicates that the neighbor target area and the target area may correspond to the same area, and thus the neighbor target area and the target area are repeated target areas. All neighbor target areas that repeat with the target area can be obtained. Only one target area in the target area and all neighbor target areas that repeat with the target area can be retained. For example, only the target area with the highest comprehensive score P in the target area and all neighbor target areas that repeat with the target area can be retained.

[0062] After the target area is detected in the image, it can be determined whether it is occluded by a person, a vehicle, or other foreground object. If the target area is occluded by the foreground object, the target area can be removed. In one embodiment, the bounding box of the foreground object can be detected. Subsequently, it can be confirmed whether there is overlap between the bounding box of the target area and the foreground object. If there is overlap between the bounding box of the target area and the foreground object, it can be determined that the target area is occluded by the foreground object.

[0063] In addition, after the target area is detected in the image, it can be tracked to know its position in subsequent images, which can be, for example, subsequent multiple frames of images. In the case where the target area is blocked by a person, a vehicle or other foreground object, or the lighting conditions during video shooting vary greatly, or the camera shooting the video moves suddenly, etc., the tracking of the target area may fail, which will affect the subsequent addition of digital content. Therefore, the target area that fails to be tracked can be removed from the set of target areas.

[0064] The duration of the target region can also be determined. For example, the duration of the target region in the video can be determined. If the duration of the target region is below a threshold, the target region may not be able to adequately display the digital content, and the target region can be removed.

[0065] In addition, in the case where the score of each target area, such as the comprehensive score P of each target area, has been obtained at 114, the target area with a score lower than the threshold may be removed from the set of target areas. For example, when the comprehensive score P is calculated by step 114, the comprehensive score P of some target areas may be calculated as zero. The target area with a comprehensive score P of zero may be removed from the set of target areas.

[0066] After the set of target areas is updated at 116 , one or more highest-scoring target areas may be selected from the updated set of target areas for adding digital content at 118 .

[0067] It should be understood that Figure 1 The processes 100a and 100b in the figure are merely examples of the process for extracting a target area for adding digital content. According to the actual application requirements, the steps in the process for extracting the target area can be replaced or modified in any manner, and the process can include more or fewer steps. For example, the step 110 for further verifying the target area can be omitted from the process 100a. In addition, the specific order or hierarchy of the steps in the process 100 is only exemplary, and the process for extracting the target area can be performed in an order different from the described order. For example, the target area update operation at 116 can be performed first, and then the target area scoring operation at 114 can be performed.

[0068] Figure 3 FIG. 3 shows an exemplary process 300 for detecting an initial quadrilateral region in an image according to an embodiment of the present disclosure. The process 300 may correspond to Figure 1304 in step 104. For example, the image 302 may be provided to the quadrilateral detection model 310. The quadrilateral detection model 310 may detect at least one initial quadrilateral region 312 in the image 302. Through the quadrilateral detection model 310, a heatmap set corresponding to the center point and the corner point in the image 302 may be obtained, an offset set corresponding to each center point in the image 302 may be obtained based on the offset between the center point and the corresponding corner point, and an initial quadrilateral region 312 may be detected based on the heatmap set and a plurality of offset sets corresponding to a plurality of center points in the image 302.

[0069] The quadrilateral detection model 310 may include an image representation generation unit 320. The image representation generation unit 320 may be a known image representation generation model, such as a High-Resolution Network (HR-Net) model, a Residual Network (ResNet) model, etc. The image representation generation unit 320 may generate an image representation 322 of the image 302.

[0070] The image representation 322 may be provided to the heat map prediction unit 330. The heat map prediction unit 330 may predict a heat map corresponding to each type of key point in the image 302, thereby obtaining a heat map set 332 corresponding to multiple types of key points. Key points may include, for example, center points and corner points, and corner points may further include, for example, top-left corner points, top-right corner points, bottom-right corner points, bottom-left corner points, etc. Therefore, key points may include, for example, center points, top-left corner points, top-right corner points, bottom-right corner points, bottom-left corner points, etc. The heat map prediction unit 330 may predict heat maps corresponding to each type of key point based on the image representation 322 of the image 302. The heat map corresponding to a specific type of key point may reflect the heat value of the point at each position in the image, which may indicate the probability that the point is a key point of that type. For example, the heat map prediction unit 330 may predict the center point heat map corresponding to the center point, the upper left corner point heat map corresponding to the upper left corner point, the upper right corner point heat map corresponding to the upper right corner point, the lower right corner point heat map corresponding to the lower right corner point, and the lower left corner point heat map corresponding to the lower left corner point, respectively, based on the image representation 322 of the image 302, thereby obtaining the heat map set 332. The center point heat map may indicate the probability of a point at each position being a center point, the upper left corner point heat map may indicate the probability of a point at each position being an upper left corner point, the upper right corner point heat map may indicate the probability of a point at each position being an upper right corner point, the lower right corner point heat map may indicate the probability of a point at each position being a lower right corner point, and the lower left corner point heat map may indicate the probability of a point at each position being a lower left corner point.

[0071] The image representation 322 may also be provided to the offset prediction unit 340. The offset prediction unit 340 may predict multiple key points in the image 302 based on the image representation 322, and regard each key point as a center point and predict the offset between each center point and each of its four corner points to obtain an offset set corresponding to the center point. In this article, the distance between two key points may be referred to as the offset between the two key points. Multiple offset sets corresponding to multiple center points may be combined into offset information 342. Each heat map included in the heat map set 332 may indicate the probability that a point at each position is a corresponding key point. Preferably, the heat map set 332 may be provided to the offset prediction unit 340 as guidance information. The offset prediction unit 340 may optimize the prediction of the key point by referring to the probability that a point at each position indicated by the heat map set 332 is a corresponding key point, so that more accurate offset information 342 can be output.

[0072] At least one initial quadrilateral region 312 can be detected based on the heat map set 332 and the offset information 342 by the combination unit 350. In one embodiment, the center point can be selected based on the center point heat map in the heat map set 332. For example, each key point can have a thermal value indicating the probability that the key point is the center point. The key points can be sorted according to the thermal value. Then, the key point with a thermal value greater than a predetermined thermal value threshold can be selected as the center point from the predetermined number of key points with the highest sorting. After the center point is selected, the offset set corresponding to the selected center point can be obtained based on the offset information 342, so that the positions of the four corner points corresponding to the selected center point can be determined. Subsequently, the positions of each corner point can be optimized based on the heat map set 332, so that the initial quadrilateral region 312 can be determined.

[0073] In process 300, at least one initial quadrilateral region may be determined by predicting the positions of the center points and corner points in the image and combining the center points and corner points belonging to the same quadrilateral region. This approach may also be referred to as bottom-up quadrilateral region detection.

[0074] The training data of the quadrilateral detection model may include multiple training images, and each training image may be marked with one or more quadrilateral regions suitable for adding digital content. The quadrilateral regions suitable for adding digital content may include, for example, photo frames, picture frames, easels, menus, posters, placards, New Year pictures, hanging paintings, banners, mirrors, bulletin boards, box facades, unopened screens, etc. When the quadrilateral detection model trained using such training data is actually deployed, it can detect quadrilateral regions suitable for adding digital content in the input image.

[0075] When training the quadrilateral detection model, the training loss can be calculated by the following formula:

[0076]

[0077] in, represents the loss associated with the heat map prediction, M is the weight used to balance the number of positive and negative samples, represents the predicted thermal value, Indicates the true mark of thermal value, represents the loss associated with the offset prediction, Area i represents the area of ​​the ith quadrilateral, o ij represents the offset from the jth corner point of the predicted i-th quadrilateral region to the center point of the i-th quadrilateral region, represents the true annotation of the offset from the jth corner point of the predicted i-th quadrilateral region to the center point of the i-th quadrilateral region, and λ is the coefficient for the loss associated with the offset prediction.

[0078] It should be understood that Figure 3 The process 300 in the figure is merely an example of a process for detecting an initial quadrilateral region in an image. The steps in the process for detecting an initial quadrilateral region may be replaced or modified in any manner according to actual application requirements, and the process may include more or fewer steps. For example, the offset prediction unit 340 may also obtain the offset information 342 based only on the image representation 322. In addition, the specific order or hierarchy of the steps in the process 300 is merely exemplary, and the process for detecting an initial quadrilateral region may be performed in an order different from the order described.

[0079] Figure 4 An exemplary process 400 for calibrating an initial quadrilateral region directly using a set of line segments according to an embodiment of the present disclosure is shown. The process 400 may correspond to Figure 1 In step 108 of the process 400, at least one side of the initial quadrilateral region may be calibrated using the line segment set to obtain a calibrated quadrilateral region.

[0080] At 402, a line segment set may be obtained. The obtained line segment set may correspond to, for example, Figure 1 The set of line segments detected in step 106.

[0081] At 404, it can be determined whether there is a line segment subset in the line segment set whose angle and / or distance with one side of the initial quadrilateral region meet the predetermined requirements. For example, it can be determined whether there is a line segment subset in the line segment set whose angle with one side of the initial quadrilateral region is within a predetermined range. The angle between two line segments / sides can be determined by any known method. Additionally or alternatively, it can be determined whether there is a line segment subset in the line segment set whose distance with one side of the initial quadrilateral region is lower than a predetermined distance threshold.

[0082] Figure 5 500a and 500b are schematic diagrams showing how to calculate the distance between two line segments according to an embodiment of the present disclosure. Assume that there is a line segment 502s i and line segment 504s j Each line segment can have a start endpoint and an end endpoint. Figure 5 In , the starting endpoint and the ending endpoint are indicated by a solid dot and a hollow dot, respectively. For example, the upper left corner of the image can be defined as the origin, and the endpoint of the line segment closer to the origin is defined as the starting endpoint, and the endpoint of the line segment farther from the origin is defined as the ending endpoint. Assume that the line segment si can be expressed as in and can represent line segments s i The coordinate values ​​of the two endpoints of . The starting endpoint p can be determined by, for example, the following formula start and the end endpoint p end :

[0083]

[0084] The line segment 502s may be calculated, for example, by the following formula: i and line segment 504s j The distance d between:

[0085] d=λ0*d0+λ1*d1 (12)

[0086]

[0087]

[0088] In formula (12), λ0 and λ1 indicate coefficients associated with distance d0 and distance d1, respectively. In formula (13), d0(s i (start),s j ) may correspond to distance 506 in FIG. 500a, which indicates that line segment s i The starting endpoint and line segment s j The vertical distance between i (center),s j ) may correspond to distance 508 in FIG. 500a, which indicates that line segment s i The center point and line segment s j The vertical distance between them, and d0(s i (end),s j ) may correspond to distance 510 in FIG. 500a, which indicates that line segment s i The end point of the line segment s j In formula (14), d1(s i (start),s j (start)) may correspond to distance 512 in diagram 500b, which indicates that line segment s i The starting endpoint and line segment s j The distance between the starting and ending points, and d1(s i (end),s j (end)) may correspond to distance 514 in FIG. 500b, which indicates that line segment s iThe end point of the line segment s j The distance between the end points of the

[0089] If it is determined at 404 that there is no line segment subset in the line segment set whose angle and / or distance with one edge of the initial quadrilateral region meets the predetermined requirement, the process may proceed to 406. At 406, the edge may be retained.

[0090] If it is determined at 404 that there is a line segment subset in the line segment set whose angle and / or distance with one side of the initial quadrilateral region meets the predetermined requirement, the process may proceed to 408. At 408, a line segment closest to the side may be selected from the line segment subset to replace the side.

[0091] Steps 404 to 408 can be performed for each side of the initial quadrilateral area to obtain a set of current sides. At 410, a calibrated quadrilateral area can be formed based on a set of current sides. In the case where each pair of adjacent sides in a set of current sides intersects, a calibrated quadrilateral area can be formed by connecting the intersection points of each pair of adjacent sides. In the case where there is one or more pairs of adjacent sides that do not intersect, for each pair of adjacent sides that do not intersect, one or two sides in the pair of adjacent sides can be extended so that the pair of adjacent sides intersect, and an intersection point is obtained. Subsequently, a calibrated quadrilateral area can be formed by connecting the intersection points of each pair of adjacent sides.

[0092] It should be understood that Figure 4 The process 400 in the figure is merely an example of a process for calibrating an initial quadrilateral region directly using a set of line segments. According to actual application requirements, the steps in the process for calibrating an initial quadrilateral region directly using a set of line segments may be replaced or modified in any manner, and the process may include more or fewer steps. In addition, the specific order or hierarchy of the steps in the process 400 is merely exemplary, and the process for calibrating an initial quadrilateral region directly using a set of line segments may be performed in an order different from the order described.

[0093] Figure 6 600a to 600c are schematic diagrams showing an example of calibrating an initial quadrilateral region directly using a set of line segments according to an embodiment of the present disclosure. In Figures 600a to 600c, an image 602, a picture 604, and an initial quadrilateral region 606 are shown. The image 602 may be, for example, a frame of an image from a video. The picture 604 may be a picture in the image 602. The initial quadrilateral region 606 may be, for example, a Figure 3602 is detected by the process 300 in the image 602. It should be understood that for the sake of clarity, the image 602, the picture 604 and the initial quadrilateral area 606 may not be drawn in actual proportion and actual position. For example, in actual situations, the picture 604 and the initial quadrilateral area 606 may only occupy a small part of the image 602, and the position of the initial quadrilateral area 606 may be closer to the position of the picture 604.

[0094] In diagram 600a, an outer frame 608 and an inner frame 610 of the painting 604 and four sides 612, 614, 616, and 618 of the initial quadrilateral region 606 are shown.

[0095] In Figure 600b, a set of line segments detected in the image 602 is shown. The detected set of line segments may include, for example, line segments 620, 622, and 624 corresponding to three of the four sides of the outer frame 608, while the line segment corresponding to the side 626 of the outer frame 608 is not detected. The detected set of line segments may also include line segments 628, 630, 632, and 634 corresponding to the four sides of the inner frame 610.

[0096] After detecting the line segment set, you can use, for example, Figure 4 The process 400 in FIG. 4 is used to directly use the line segment set to calibrate the initial quadrilateral region 606. For example, the edge 612 of the initial quadrilateral region 606 can be replaced by the line segment 620, the edge 614 can be replaced by the line segment 630, and the edge 616 can be replaced by the line segment 632. As for the edge 618, since there is no line segment whose angle and / or distance with the edge 618 meet the predetermined requirements in the detected line segment set, the edge 618 can be retained. The calibrated quadrilateral region 636 can be formed based on the line segment 620, the line segment 630, the line segment 632 and the edge 618, as shown in the thick solid frame in FIG. 600c. Specifically, when forming the calibrated quadrilateral region 636, the adjacent line segments 620 and the line segment 630 are not intersecting, and the line segment 630 can be extended to intersect with the line segment 620 and obtain the intersection point. Similarly, the intersection points of other adjacent edges can be obtained. A calibrated quadrilateral area 636 may be formed by connecting the four intersection points.

[0097] As can be seen from Figure 600c, the calibrated quadrilateral region 636 includes an uncalibrated edge, such as edge 618. In addition, the calibrated quadrilateral region 636 includes edges in the outer frame 608 of the picture 604, such as edge 620, and edges in the inner frame 610, such as edges 630 and 632. In order to make the calibrated quadrilateral region more closely fit the quadrilateral region in the image, or in the case where the quadrilateral region in the image includes an outer frame and an inner frame, so that the calibrated quadrilateral region only includes the edges corresponding to the outer frame or only includes the edges corresponding to the inner frame, the embodiments of the present disclosure also propose to calibrate the initial quadrilateral region using a reference quadrilateral region constructed by a set of line segments.

[0098] Figure 7 An exemplary process 700 for calibrating an initial quadrilateral region using a reference quadrilateral region constructed by a set of line segments according to an embodiment of the present disclosure is shown. The process 700 may correspond to, for example, Figure 1 In step 108 of the process 700, a reference quadrilateral region set may be constructed using a line segment set. The line segment set may be, for example, Figure 1 Then, a reference quadrilateral region that is closest to the initial quadrilateral region and whose distance from the initial quadrilateral region is lower than a threshold value may be selected from the reference quadrilateral region set, and the initial quadrilateral region may be replaced by the selected reference quadrilateral region.

[0099] The reference quadrilateral region set can be constructed by steps 702 to 712. For example, each line segment in the line segment set can be used as the starting line segment, and the next line segment adjacent to the current line segment can be iteratively searched from the line segment set to form at least one reference quadrilateral region corresponding to the line segment. Then, a plurality of reference quadrilateral regions corresponding to the line segment set can be combined into a reference quadrilateral region set. In one embodiment, a group of adjacency matrix can be first established. This group of adjacency matrix can indicate the adjacency relationship between every two line segments in the line segment set. Subsequently, each line segment in the line segment set can be used as the starting line segment based on this group of adjacency matrix, and the next line segment adjacent to the current line segment can be iteratively searched from the line segment set to form at least one reference quadrilateral region corresponding to the line segment.

[0100] At 702, a set of adjacency types can be defined based on a combination of endpoint types of adjacent endpoints of two line segments. Figure 8 FIG. 8 shows a set of exemplary adjacency types 800 according to an embodiment of the present disclosure. A quadrilateral region may be formed by connecting four line segments end to end. The endpoint types of adjacent endpoints of adjacent line segments may be cycled through a set of adjacency types. Figure 8As shown, starting from the start endpoint above the vertical line segment on the left side of the quadrilateral area, the adjacency types along the clockwise direction can be {start endpoint-start endpoint, end endpoint-start endpoint, end endpoint-end endpoint, start endpoint-end endpoint}. The adjacency type ADJACENT_TYPE can be defined as, for example, ADJACENT_TYPE = {0: ['start', 'start'], 1: ['end', 'start'], 2: ['end', 'end'], 3: ['start', 'end']}, where "['start', 'start']", "['end', 'start']", "['end', 'end']" and "['start', 'end']" can represent the adjacency types "start endpoint-start endpoint", "end endpoint-start endpoint", "end endpoint-end endpoint" and "start endpoint-end endpoint" respectively.

[0101] At 704, for each adjacency type in a set of adjacency types and for each two line segments in the line segment set, an adjacency value between the two line segments may be determined based on an angle between the two line segments and / or a distance between two endpoints of the two line segments corresponding to the adjacency type to obtain a set of adjacency values. i and line segment s j The adjacency value between can be expressed as M k , which can be determined, for example, by the following formula:

[0102]

[0103] Among them, arccos(s i ,s j )∈[a0,a1] represents the line segment s i and line segment s j The angle between is within the threshold range [a0, a1], and Dist(s i [ADJACENT_TYPE[k]],s j [ADJACENT_TYPE[k]])<d thres Represents line segment s i and line segment s j The distance between the two endpoints of the adjacency type ADJACENT_TYPE[k] is less than the distance threshold d thres .

[0104] At 706, a set of adjacency values ​​may be combined into an adjacency matrix for the adjacency type.

[0105] Steps 704 and 706 may be performed for each adjacency type, and thus at 708, a set of adjacency matrices for a set of adjacency types may be obtained.

[0106] At 710, each line segment in the line segment set can be used as a starting line segment, and based on a set of adjacency matrices, the next line segment adjacent to the current line segment can be iteratively searched from the line segment set to form at least one reference quadrilateral area corresponding to the line segment. For example, for the current line segment, a line segment with an adjacency value of "1" with the current line segment can be searched from the line segment set as the next line segment adjacent to the current line segment. For each line segment, it can be regarded as a starting line segment, and a certain number of line segments can be searched out from the line segment set in sequence. The starting line segment and the line segments searched out from the line segment set can be combined into a reference quadrilateral area corresponding to the starting line segment. The above search can be performed by, for example, a depth-first search (DFS) algorithm. The maximum value of the number of obtained line segments including the starting line segment can be set to "4", that is, at most three line segments can be searched out from the line segment set. In the case where the number of obtained line segments is less than or equal to "2", the combination operation can not be performed on it. When the number of obtained line segments is "3", that is, two line segments are searched out, a reference quadrilateral region can be formed by connecting the endpoints of the starting line segment and the last searched out ending line segment. When the number of obtained line segments is "4", that is, three line segments are searched out, the four obtained line segments can be combined into a reference quadrilateral region. When the starting line segment and the last searched out ending line segment do not intersect, the starting line segment and / or the last searched out ending line segment can be extended to make them intersect, thereby forming a reference quadrilateral region.

[0107] Step 710 may be performed for each line segment in the line segment set to obtain a plurality of reference quadrilateral regions corresponding to the line segment set. At 712, the plurality of reference quadrilateral regions corresponding to the line segment set may be combined into a reference quadrilateral region set.

[0108] Optionally, the reference quadrilateral region set may be updated by removing invalid reference quadrilateral regions from the reference quadrilateral region set at 714. For example, the reference quadrilateral region set may be updated by removing at least one of the following from the reference quadrilateral region set: a reference quadrilateral region of a non-convex quadrilateral, a reference quadrilateral region where the intersection of the start line segment and the end line segment falls on the start line segment, and a reference quadrilateral region where the angle between the start line segment and the end line segment is outside a predetermined range, etc. Fig. 9 Exemplary invalid reference quadrilateral regions 900a to 900c are shown according to an embodiment of the present disclosure.

[0109] Invalid reference quadrilateral region 900a can be a reference quadrilateral region of a non-convex quadrilateral. For each reference quadrilateral region in the reference quadrilateral region set, the vector product of every two adjacent line segments can be calculated, for example, in a clockwise direction to obtain a set of vector products corresponding to the reference quadrilateral region. If the signs of the set of vector products are the same, the reference quadrilateral region is a convex quadrilateral and can be retained. If the signs of the set of vector products are not the same, the reference quadrilateral region is not a convex quadrilateral and can be considered as an invalid reference quadrilateral region, and can thus be removed. Taking invalid reference quadrilateral region 900a as an example, it can include line segments 902, 904, 906 and 908, and a quadrilateral region can be formed by extending line segment 902 to intersect with line segment 908. The vector product of line segment 902 and line segment 904, the vector product of line segment 904 and line segment 906, the vector product of line segment 906 and line segment 908, and the vector product of line segment 908 and line segment 902 can be calculated respectively to obtain a set of vector products. In the set of vector products, the sign of the vector product of line segment 904 and line segment 906, the sign of the vector product of line segment 906 and line segment 908, and the sign of the vector product of line segment 908 and line segment 902 are the same, and the sign of the vector product of line segment 902 and line segment 904 is different from the signs of the first three, then the quadrilateral region is not a convex quadrilateral and can be removed.

[0110] Invalid reference quadrilateral region 900b may be a reference quadrilateral region in which the intersection of the starting line segment and the ending line segment falls on the starting line segment. The ending line segment may be the last line segment searched out. In the case where the starting line segment and the ending line segment do not directly intersect, the intersection of the starting line segment and the ending line segment may be obtained by extending the starting line segment or the ending line segment. In invalid reference quadrilateral region 900b, the intersection 914 of the starting line segment 910 and the ending line segment 912 falls on the starting line segment 902, and thus the invalid reference quadrilateral region 900b may be considered as an invalid reference quadrilateral region and may be removed.

[0111] The invalid reference quadrilateral region 900c may be a reference quadrilateral region in which the angle between the starting line segment and the ending line segment is outside a predetermined range. The predetermined range may be, for example, [70°, 120°]. In the invalid reference quadrilateral region 900c, the angle 920 between the starting line segment 906 and the ending line segment 908 is, for example, 45°, which is outside the predetermined range, and thus the invalid reference quadrilateral region 900c may be considered as an invalid reference quadrilateral region and may be removed.

[0112] It should be understood that the above reference Fig. 9 The invalid reference quadrilateral regions 900a to 900c described are only exemplary. According to actual application requirements, the invalid reference quadrilateral region may also have other forms and may be determined in other ways accordingly.

[0113] After the reference quadrilateral region set is updated, at 716, a reference quadrilateral region that is closest to the initial quadrilateral region and whose distance from the initial quadrilateral region is lower than a threshold value may be selected from the updated reference quadrilateral region set. For example, the distance between each reference quadrilateral region in the updated reference quadrilateral region set and the initial quadrilateral region may be calculated. In one embodiment, the distance between each edge of the reference quadrilateral region and the corresponding edge of the initial quadrilateral region may be first calculated to obtain four distances corresponding to the four edges of the reference quadrilateral region. The distance between the reference quadrilateral region and the initial quadrilateral region may then be obtained by summing and averaging the four distances. The distance between the edge of the reference quadrilateral region and the edge of the initial quadrilateral region may be obtained, for example, by combining the above. Figure 5 The described procedure is used for calculation.

[0114] At 718, the initial quadrilateral region may be replaced with the selected reference quadrilateral region.

[0115] It should be understood that Figure 7 The process 700 in is merely an example of a process for calibrating an initial quadrilateral region using a reference quadrilateral region constructed by a set of line segments. According to actual application requirements, the steps in the process for calibrating an initial quadrilateral region by using a reference quadrilateral region may be replaced or modified in any manner, and the process may include more or fewer steps. For example, the process of updating the reference quadrilateral region set may be omitted from process 700, thereby directly selecting a reference quadrilateral region that is closest to the initial quadrilateral region and whose distance from the initial quadrilateral region is lower than a threshold from a reference quadrilateral region set composed of a plurality of reference quadrilateral regions corresponding to the set of line segments. In addition, the specific order or hierarchy of the steps in process 700 is merely exemplary, and the process for calibrating an initial quadrilateral region by using a reference quadrilateral region may be performed in an order different from the described order.

[0116] Fig.10 1000a to 1000c are schematic diagrams showing an example of calibrating an initial quadrilateral region using a reference quadrilateral region constructed by a set of line segments according to an embodiment of the present disclosure. In 1000a to 1000c, the image 1002 may be, for example, a frame of an image from a video, which may correspond to Figure 6 602 in FIG. 1000a may correspond to Figure 6600b in FIG. 600b shows a painting 1004, an initial quadrilateral region 1006 detected in the image 1002, and a set of line segments detected in the image 1002. The set of line segments may include, for example, line segments 1008, 1010, and 1012 corresponding to the outer frame of the painting 1004 and line segments 1014, 1016, 1018, and 1020 corresponding to the inner frame of the painting 1004.

[0117] The set of line segments can be used to construct a reference quadrilateral region. FIG. 1000 b shows two constructed reference quadrilateral regions 1022 and 1024, which can be constructed by, for example Figure 7 The reference quadrilateral region 1022 may be constructed by using the line segments 1008, 1010, and 1012. For example, the reference quadrilateral region 1022 may be formed by combining the line segments 1008, 1010, and 1012 and connecting the endpoints of the line segments 1008 and 1012. The reference quadrilateral region 1024 may be constructed by using the line segments 1014, 1016, 1018, and 1020.

[0118] The reference quadrilateral region that is closest to the initial quadrilateral region 1006 and whose distance to the initial quadrilateral region is lower than a threshold value can be selected from these two reference quadrilateral regions 1022 and 1024. The reference quadrilateral region that is closest to the initial quadrilateral region 1006 and whose distance to the initial quadrilateral region 1006 is lower than a threshold value can be, for example, the reference quadrilateral region 1022, as shown in the thick solid line frame in Figure 1000c. The initial quadrilateral region 1006 can be replaced by the reference quadrilateral region 1022. The reference quadrilateral region 1022 can be used as a target region for adding digital content. As can be seen from the figure, the reference quadrilateral region 1022 fits the painting 1004 and is constructed only by the edges corresponding to the outer frame of the painting 1004.

[0119] Fig.11 11 is a flowchart of an exemplary method 1100 for extracting a target area for adding digital content according to an embodiment of the present disclosure.

[0120] At 1110, an image can be obtained.

[0121] At 1120, at least one initial quadrilateral region may be detected in the image.

[0122] At 1130, a set of line segments can be detected in the image.

[0123] At 1140 , the initial quadrilateral area may be calibrated using the set of line segments to obtain a target area for adding digital content.

[0124] In one embodiment, detecting at least one initial quadrilateral area may include: obtaining a set of heat maps corresponding to center points and corner points in the image; obtaining a set of offsets corresponding to the center points based on the offset between each center point in the image and the corresponding corner point; and detecting the initial quadrilateral area based on the set of heat maps and multiple offset sets corresponding to multiple center points in the image.

[0125] In one embodiment, the initial quadrilateral region may be detected by a quadrilateral detection model. The training data of the quadrilateral detection model may include a plurality of training images. Each training image may be marked with one or more quadrilateral regions suitable for adding digital content.

[0126] In one embodiment, calibrating the initial quadrilateral region may include: calibrating at least one side of the initial quadrilateral region using the line segment set to obtain a calibrated quadrilateral region.

[0127] The calibrating of at least one side of the initial quadrilateral area may include: determining whether there is a line segment subset in the line segment set whose angle and / or distance with the side meet predetermined requirements; and in response to determining that the line segment subset exists in the line segment set, selecting a line segment closest to the side from the line segment subset to replace the side.

[0128] In one embodiment, calibrating the initial quadrilateral region may include: using the set of line segments to construct a reference quadrilateral region set; selecting a reference quadrilateral region from the reference quadrilateral region set that is closest to the initial quadrilateral region and whose distance to the initial quadrilateral region is lower than a threshold; and replacing the initial quadrilateral region with the selected reference quadrilateral region.

[0129] The construction of the reference quadrilateral region set may include: taking each line segment in the line segment set as a starting line segment, iteratively searching the line segment set for the next line segment adjacent to the current line segment, so as to form at least one reference quadrilateral region corresponding to the line segment; and combining multiple reference quadrilateral regions corresponding to the line segment set into the reference quadrilateral region set.

[0130] Method 1100 may also include: establishing a set of adjacency matrices, wherein the set of adjacency matrices indicates the adjacency relationship between every two line segments in the line segment set. The searching for the next line segment adjacent to the current line segment may include: searching for the next line segment adjacent to the current line segment from the line segment set based on the set of adjacency matrices.

[0131] The establishing of a set of adjacency matrices may include: defining a set of adjacency types based on a combination of endpoint types of adjacent endpoints of two line segments; and establishing an adjacency matrix for each adjacency type in the set of adjacency types to obtain the set of adjacency matrices.

[0132] The establishing of the adjacency matrix for the adjacency type may include: for each two line segments in the line segment set, determining the adjacency value between the two line segments based on the angle between the two line segments and / or the distance between two endpoints of the two line segments corresponding to the adjacency type to obtain a set of adjacency values; and combining the set of adjacency values ​​into the adjacency matrix.

[0133] Method 1100 may also include: updating the reference quadrilateral region set by removing at least one of the following from the reference quadrilateral region set: a reference quadrilateral region of a non-convex quadrilateral, a reference quadrilateral region whose intersection point of a starting line segment and an ending line segment falls on the starting line segment, and a reference quadrilateral region whose angle between the starting line segment and the ending line segment is outside a predetermined range. The selecting of the reference quadrilateral region may include: selecting a reference quadrilateral region from the updated reference quadrilateral region set that is closest to the initial quadrilateral region and whose distance from the initial quadrilateral region is lower than a threshold.

[0134] In one embodiment, the method 1100 may further include: determining whether the target area is suitable for adding digital content based on the content of the target area; and removing the target area in response to determining that the target area is not suitable for adding digital content.

[0135] In one embodiment, the image may be from a group of images in a video. The group of images may include a group of target areas. The group of images may be taken using different camera parameters.

[0136] The method 1100 may further include: for each target area in the set of target areas, scoring the target area based on the quality of the target area; and selecting one or more target areas with the highest scores from the set of target areas for adding digital content.

[0137] Scoring the target region may include scoring the target region based on one or more of a confidence level, an aspect ratio, an area, a rectangular degree, and a clarity of the target region.

[0138] The method 1100 may also include: updating the set of target areas by removing at least one of the following from the set of target areas: a repeated target area, an obscured target area, a target area that failed to be tracked, a target area with a duration below a threshold, and a target area with a score below a threshold. The selecting one or more target areas with the highest scores may include: selecting one or more target areas with the highest scores from the updated set of target areas for adding digital content.

[0139] It should be understood that the method 1100 may also include any steps / processes for extracting a target area for adding digital content according to the above-mentioned embodiments of the present disclosure.

[0140] Fig.12 An exemplary apparatus 1200 for extracting a target area for adding digital content according to an embodiment of the present disclosure is shown.

[0141] The apparatus 1200 may include: an image acquisition module 1210 for acquiring an image; a quadrilateral detection module 1220 for detecting at least one initial quadrilateral region in the image; a line segment detection module 1230 for detecting a line segment set in the image; and a quadrilateral calibration module 1240 for calibrating the initial quadrilateral region using the line segment set to obtain a target region for adding digital content. In addition, the apparatus 1200 may also include any other module configured for extracting a target region for adding digital content according to the above-mentioned embodiments of the present disclosure.

[0142] Fig.13 An exemplary apparatus 1300 for extracting a target area for adding digital content according to an embodiment of the present disclosure is shown.

[0143] The apparatus 1300 may include: at least one processor 1310; and a memory 1320 storing computer executable instructions. When the computer executable instructions are executed, the at least one processor 1310 may: obtain an image, detect at least one initial quadrilateral region in the image, detect a line segment set in the image, and calibrate the initial quadrilateral region using the line segment set to obtain a target region for adding digital content.

[0144] In one embodiment, calibrating the initial quadrilateral region may include: calibrating at least one side of the initial quadrilateral region using the line segment set to obtain a calibrated quadrilateral region.

[0145] In one embodiment, calibrating the initial quadrilateral region may include: using the set of line segments to construct a reference quadrilateral region set; selecting a reference quadrilateral region from the reference quadrilateral region set that is closest to the initial quadrilateral region and whose distance to the initial quadrilateral region is lower than a threshold; and replacing the initial quadrilateral region with the selected reference quadrilateral region.

[0146] It should be understood that the processor 1310 may also execute any other steps / processes of the method for extracting a target area for adding digital content according to the above-mentioned embodiment of the present disclosure.

[0147] The embodiment of the present disclosure proposes a computer program product for extracting a target area for adding digital content, including a computer program, which is executed by at least one processor to: obtain an image; detect at least one initial quadrilateral area in the image; detect a line segment set in the image; and calibrate the initial quadrilateral area using the line segment set to obtain a target area for adding digital content. In addition, the computer program can also be executed to implement any other steps / processes of the method for extracting a target area for adding digital content according to the above-mentioned embodiment of the present disclosure.

[0148] The embodiments of the present disclosure may be embodied in a non-transitory computer-readable medium. The non-transitory computer-readable medium may include instructions, which, when executed, cause one or more processors to perform any operation of the method for extracting a target area for adding digital content according to the embodiments of the present disclosure as described above.

[0149] It should be understood that all operations in the above-described methods are merely exemplary, and the present disclosure is not limited to any operations in the methods or the order of these operations, but should cover all other equivalent transformations under the same or similar concept. In addition, unless otherwise specified or it is clear from the context that it is directed to a singular form, the articles "a" and "an" as used in this specification and the appended claims should generally be interpreted as meaning "one" or "one or more".

[0150] It should also be appreciated that all modules in the above described device can be implemented in various ways. These modules can be implemented as hardware, software, or a combination thereof. In addition, any module in these modules can be further divided into submodules or combined together in function.

[0151] Processors have been described in conjunction with various devices and methods. These processors can be implemented using electronic hardware, computer software or any combination thereof. Whether these processors are implemented as hardware or software will depend on specific application and the overall design constraints imposed on the system. As an example, the processor provided in the present disclosure, any part of the processor or any combination of processors can be realized using a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a state machine, a gated logic unit, a discrete hardware circuit, and other suitable processing components configured to perform the various functions described in the present disclosure. The function of the processor provided in the present disclosure, any part of the processor or any combination of processors can be realized using software executed by a microprocessor, a microcontroller, a DSP or other suitable platforms.

[0152] Software should be broadly considered to mean instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, running threads, processes, functions, etc. Software can reside in a computer-readable medium. A computer-readable medium can include, for example, a memory, which can be, for example, a magnetic storage device (e.g., a hard disk, a floppy disk, a magnetic stripe), an optical disk, a smart card, a flash memory device, a random access memory (RAM), a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a register, or a removable disk. Although the memory is shown as being separated from the processor in the multiple aspects provided in the present disclosure, the memory can also be located inside the processor, such as a cache or a register.

[0153] The above description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein. All structural and functional equivalents to the elements of the various aspects described in this disclosure that are known or to be known to those of ordinary skill in the art are expressly incorporated herein and covered by the claims.

Claims

1. A method for extracting a target area for adding digital content, comprising: Obtaining images; detecting at least one initial quadrilateral region in the image; detecting a set of line segments in the image; as well as Calibrate the initial quadrilateral area using the line segment set to obtain a target area for adding digital content, wherein calibrating the initial quadrilateral area includes: Determining whether there is a line segment subset in the line segment set whose angle and / or distance with at least one side of the initial quadrilateral region meet predetermined requirements, and in response to determining that the line segment subset exists in the line segment set, selecting a line segment closest to the side from the line segment subset to replace the side; and / or A reference quadrilateral region set is constructed using the line segment set, a reference quadrilateral region that is closest to the initial quadrilateral region and whose distance to the initial quadrilateral region is lower than a threshold is selected from the reference quadrilateral region set, and the initial quadrilateral region is replaced by the selected reference quadrilateral region.

2. The method according to claim 1, wherein: The detecting at least one initial quadrilateral region comprises: Obtaining a set of heat maps corresponding to the center point and the corner points in the image; Obtaining an offset set corresponding to each center point based on an offset between each center point and a corresponding corner point in the image; and The initial quadrilateral region is detected based on the set of heat maps and a plurality of sets of offsets corresponding to a plurality of center points in the image.

3. The method according to claim 1, wherein: The initial quadrilateral region is detected by a quadrilateral detection model, the training data of the quadrilateral detection model includes a plurality of training images, and each training image is marked with one or more quadrilateral regions suitable for adding digital content.

4. The method according to claim 1, wherein: The construction of the reference quadrilateral region set includes: By taking each line segment in the line segment set as a starting line segment, iteratively searching for a next line segment adjacent to a current line segment from the line segment set, so as to form at least one reference quadrilateral region corresponding to the line segment; and A plurality of reference quadrilateral regions corresponding to the line segment set are combined into the reference quadrilateral region set.

5. The method according to claim 4, further comprising: A set of adjacency matrices is established, wherein the set of adjacency matrices indicates the adjacency relationship between every two line segments in the line segment set, and The searching for the next line segment adjacent to the current line segment includes: searching for the next line segment adjacent to the current line segment from the line segment set based on the set of adjacency matrices.

6. The method according to claim 5, wherein: The step of establishing a set of adjacency matrices comprises: defining a set of adjacency types based on combinations of endpoint types of adjacent endpoints of two line segments; and For each adjacency type in the set of adjacency types, an adjacency matrix for the adjacency type is established to obtain the set of adjacency matrices.

7. The method according to claim 6, wherein: The establishing of the adjacency matrix for the adjacency type comprises: For each two line segments in the set of line segments, determining an adjacency value between the two line segments based on an angle between the two line segments and / or a distance between two endpoints of the two line segments corresponding to the adjacency type to obtain a set of adjacency values; and The set of adjacency values ​​are combined into the adjacency matrix.

8. The method according to claim 1, further comprising: The reference quadrilateral region set is updated by removing at least one of the following from the reference quadrilateral region set: a reference quadrilateral region of a non-convex quadrilateral, a reference quadrilateral region whose intersection point of a starting line segment and an ending line segment falls on the starting line segment, and a reference quadrilateral region whose angle between the starting line segment and the ending line segment is outside a predetermined range, and The selecting the reference quadrilateral region comprises: selecting, from the updated reference quadrilateral region set, a reference quadrilateral region that is closest to the initial quadrilateral region and has a distance from the initial quadrilateral region lower than a threshold.

9. The method according to claim 1, further comprising: determining whether the target area is suitable for adding digital content based on the content of the target area; as well as In response to determining that the target area is not suitable for adding digital content, the target area is removed.

10. The method according to claim 1, wherein: The image comes from a group of images in a video, the group of images includes a group of target areas, and the group of images is taken using different camera parameters.

11. The method according to claim 10, further comprising: For each target region in the set of target regions, scoring the target region based on a quality of the target region; as well as One or more highest-scoring target areas are selected from the set of target areas for adding digital content.

12. The method according to claim 11, wherein: The scoring of the target area comprises: The target region is scored based on one or more of the confidence, aspect ratio, area, rectangularity, and clarity of the target region.

13. The method according to claim 11, further comprising: The set of target regions is updated by removing at least one of the following from the set of target regions: a repeated target region, an occluded target region, a target region whose tracking fails, a target region whose duration is below a threshold, and a target region whose score is below a threshold, and The selecting one or more target areas with the highest score includes: selecting one or more target areas with the highest score from the updated set of target areas for adding digital content.

14. A device for extracting a target area for adding digital content, comprising: at least one processor; as well as a memory storing computer executable instructions that, when executed, cause the at least one processor to: Get the image, detecting at least one initial quadrilateral region in the image, detecting a set of line segments in the image, and Calibrate the initial quadrilateral area using the line segment set to obtain a target area for adding digital content, wherein calibrating the initial quadrilateral area includes: Determining whether there is a line segment subset in the line segment set whose angle and / or distance with at least one side of the initial quadrilateral region meet predetermined requirements, and in response to determining that the line segment subset exists in the line segment set, selecting a line segment closest to the side from the line segment subset to replace the side; and / or A reference quadrilateral region set is constructed using the line segment set, a reference quadrilateral region that is closest to the initial quadrilateral region and whose distance to the initial quadrilateral region is lower than a threshold is selected from the reference quadrilateral region set, and the initial quadrilateral region is replaced by the selected reference quadrilateral region.

15. A computer program product for extracting a target area for adding digital content, comprising a computer program, wherein the computer program is executed by at least one processor to: Obtaining images; detecting at least one initial quadrilateral region in the image; detecting a set of line segments in the image; and The initial quadrilateral area is calibrated using the line segment set to obtain a target area for adding digital content, wherein: The calibrating the initial quadrilateral area comprises: Determining whether there is a line segment subset in the line segment set whose angle and / or distance with at least one side of the initial quadrilateral region meet predetermined requirements, and in response to determining that the line segment subset exists in the line segment set, selecting a line segment closest to the side from the line segment subset to replace the side; and / or A reference quadrilateral region set is constructed using the line segment set, a reference quadrilateral region that is closest to the initial quadrilateral region and whose distance to the initial quadrilateral region is lower than a threshold is selected from the reference quadrilateral region set, and the initial quadrilateral region is replaced by the selected reference quadrilateral region.

Images