Live image contour correction method, device, equipment and storage medium

Abstract

The present application relates to a method, device, equipment and storage medium for contour correction of a live image. The method extracts straight line segments from a contour map of a live image, converts the straight line segments into straight lines and obtains the intersection points between every two straight lines, and merges the intersection points that meet the merging conditions according to a preset merging condition, thereby reducing the amount of data calculation for contour correction of the live image. The method obtains an affine transformation matrix based on the position information of the four intersection points of the quadrilateral with the largest area, and the affine transformation matrix is ​​used to correct the contour map of the live image. The method for contour correction of the live image can quickly correct the contour map of the extracted live image, so that the extracted contour map of the live image is easier to be recognized, thereby improving the efficiency of contour detection of the live image.

Description

Technical Field

[0001] The present application relates to the technical field of network live broadcasting, and in particular to a method, device, equipment and storage medium for correcting the contour of a live broadcast image. Background Art

[0002] With the advancement of network communication technology, live streaming has become a new form of online interaction. It is also loved by more and more audiences because of its real-time and interactive features.

[0003] During live broadcasts, online hosts often need to interact with the audience. In some live broadcast scenarios, when the host shows an object to the audience, it is necessary to perform contour detection on the object. After the object is detected through contour detection, the object can be enlarged, special effects can be added, and it can be displayed separately.

[0004] However, objects in the captured live images are prone to offset or distortion, and the object contour map extracted by contour detection may have distortion problems, thereby affecting the recognition of the object. Summary of the invention

[0005] Based on this, the purpose of this application is to provide a method, device, equipment and storage medium for contour correction of a live image, which can quickly realize the correction of the contour map of the live image, make the extracted contour map of the live image easier to identify, and improve the detection efficiency.

[0006] According to a first aspect of an embodiment of the present application, a contour correction method for a live image is provided, and the contour correction method for a live image comprises:

[0007] Acquire the live image to be detected;

[0008] Performing contour extraction on the live broadcast image to obtain a contour map of the live broadcast image;

[0009] Extracting straight line segments in the contour map of the live image;

[0010] Convert the straight line segments into straight lines, obtain the position information of the intersection points between every two straight lines, and merge the intersection points that meet the merging conditions according to a preset merging condition;

[0011] Obtain the area of ​​a quadrilateral formed by every four intersections, and obtain the position information of the four intersections with the largest quadrilateral areas;

[0012] An affine transformation matrix is ​​obtained based on preset target image position information and position information of the four intersection points, and the contour map of the live image is corrected using the affine transformation matrix to obtain a corrected contour map of the live image.

[0013] According to a second aspect of an embodiment of the present application, a contour correction device for a live image is provided, and the contour correction device for a live image comprises:

[0014] An acquisition module, used for acquiring a live image to be detected;

[0015] A contour map acquisition module, used to extract the contour of the live image and acquire the contour map of the live image;

[0016] A straight line segment extraction module, used to extract straight line segments in the contour map of the live image;

[0017] An intersection acquisition module, used for converting the straight line segments into straight lines, acquiring the position information of the intersections between every two straight lines, and merging the intersections satisfying the merging conditions according to a preset merging condition;

[0018] A position information acquisition module, used to acquire the area of ​​a quadrilateral formed by every four intersections, and acquire the position information of the four intersections with the largest quadrilateral area;

[0019] The correction module is used to obtain an affine transformation matrix based on preset target image position information and the position information of the four intersection points, and use the affine transformation matrix to correct the contour map of the live image to obtain the corrected contour map of the live image.

[0020] According to a third aspect of an embodiment of the present application, there is provided an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing any one of the methods for contour correction of live images.

[0021] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the contour correction method of any one of the live image correction methods is implemented.

[0022] The present application obtains a live image to be detected, extracts straight line segments in a contour map of the live image, converts the straight line segments into straight lines and obtains the intersection points between every two straight lines, and merges the intersection points that meet the merging conditions according to a preset merging condition, thereby reducing the amount of data calculation for contour correction of the live image; obtains an affine transformation matrix based on the preset target image position information and the position information of the four intersection points by forming the position information of the four intersection points of the quadrilateral with the largest area, and uses the affine transformation matrix to correct the contour map of the live image. The contour correction method of the live image can quickly correct the contour map of the extracted live image, making the extracted contour map of the live image easier to be recognized, thereby improving the contour detection efficiency of the live image.

[0023] For better understanding and implementation, the present application is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of an application scenario of a contour correction method for a live image provided by an embodiment of the present application;

[0025] Figure 2 A flowchart of a method for correcting the contour of a live broadcast image provided by one embodiment of the present application;

[0026] Figure 3 An example diagram of extracting a contour map of a live broadcast image using the method described in the present application provided in one embodiment of the present application;

[0027] Figure 4 An example diagram of extracting a contour map of a live broadcast image using the method described in the present application, provided in another embodiment of the present application;

[0028] Figure 5 A schematic structural diagram of a live image contour correction device provided by an embodiment of the present application;

[0029] Figure 6 A schematic block diagram of the structure of an electronic device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the objectives, technical solutions and advantages of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0031] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0032] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0033] In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. The singular forms of "a", "said", and "the" used in the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. The words "if" / "if" used herein can be interpreted as "at the time of" or "when" or "in response to determination". In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0034] See also Figure 1 , which is a schematic diagram of an application scenario of the contour correction method of a live broadcast image provided in the present application. The application scenario includes a live broadcast client 10 and a server 20, and the live broadcast client 10 interacts with the server 20.

[0035] The hardware pointed to by the live broadcast client 10 is essentially a computer device, specifically, it can be a computer device of the type of a smart phone, a smart interactive tablet, a personal computer, etc. The live broadcast client 10 can access the Internet through a well-known network access method and establish a data communication link with the server 20.

[0036] The server 20 is a service server, which can be responsible for further connecting to related audio data servers, video streaming servers and other servers providing related support services, so as to form a logically related service cluster to provide services for related terminal devices, such as Figure 1 The live broadcast client 10 shown in provides services.

[0037] The contour correction method of the live broadcast image can be run on the live broadcast client 10 and / or the server 20. When the contour correction method of the live broadcast image is run on the live broadcast client 10, the live broadcast client 10 executes the contour correction method of the live broadcast image on the live broadcast image obtained locally, and obtains the contour map of the corrected live broadcast image. When the contour correction method of the live broadcast image is run on the server, the server 20 obtains the live broadcast image from the live broadcast client, executes the contour correction method of the live broadcast image, obtains the contour map of the corrected live broadcast image, and can return the contour map of the live broadcast image to the live broadcast client 10.

[0038] Embodiment 1:

[0039] The embodiment of the present application discloses a method for correcting the contour of a live image.

[0040] The following will be combined with the attached Figure 2 , a contour correction method for a live image provided in an embodiment of the present application is introduced in detail.

[0041] The contour correction method of the live image provided in the embodiment of the present application includes:

[0042] S101: Acquire a live image to be detected.

[0043] The live broadcast image may be a live broadcast image acquired by a live broadcast client, or a part of the live broadcast image, such as a partial screenshot of the live broadcast image.

[0044] S102: extracting the contour of the live image to obtain a contour map of the live image;

[0045] In one embodiment, an existing contour extraction model can be used to extract contours from live images. In another embodiment, a pre-trained weak semantic contour detection model can be used to obtain contour maps of live images. The weak semantic contour detection model only focuses on complete and significant target objects, which can effectively reduce the amount of calculation in contour detection and improve detection efficiency; for live images without complete objects, contour detection is terminated in a timely manner, thereby reducing the false detection rate.

[0046] Wherein, the weak semantic contour detection model is used to detect the contours of significant complete objects in the live image. The weak semantic contour detection model does not care about the type of objects in the live image. As long as there are significant complete objects in the live image, the contours of the complete objects can be detected. The design of the weak semantic contour detection model is based on the encoder-decoder framework. The decoder-encoder framework is a model architecture that uses different algorithms to solve different tasks. Encoding refers to an encoder converting an input sequence into a dense vector of a fixed dimension, and decoding refers to converting the dense vector obtained by encoding into target data. The weak semantic contour detection model includes an encoder, a classification module and a decoder.

[0047] Specifically, the step of extracting the contour of the live image includes:

[0048] Acquire contour features in the live image through an encoder in a pre-trained weak semantic contour detection model; wherein the weak semantic contour detection model includes an encoder, a classification module and a decoder;

[0049] Determining, by the classification module, whether the live broadcast image contains at least one complete object according to the contour feature;

[0050] If it is detected that the live image contains at least one complete object, the contour of the object in the live image is extracted by the decoder.

[0051] The encoder comprises an input layer and a plurality of encoding layers connected in sequence;

[0052] The step of obtaining the contour features in the live image through the encoder in the pre-trained weak semantic contour detection model comprises:

[0053] Convolving the live image through the input layer, downsampling to a first preset resolution and then outputting it to the plurality of encoding layers;

[0054] Separately convolve the live image through the plurality of coding layers to obtain a contour feature map in the live image;

[0055] The input layer downsamples the live image to a first preset resolution to reduce the feature computation amount output to the encoding layer and improve the efficiency of extracting the object contour by the weak semantic contour detection model. The first preset resolution can be set according to the image size of the input live image.

[0056] The coding layer is used to perform convolution operation on the live image to obtain a contour feature map in the live image. Preferably, the convolution method of the coding layer is depthwise separable convolution. Compared with conventional convolution, depthwise separable convolution can reduce parameters and can improve the network's computing speed to a certain extent.

[0057] The classification module is a binary classification module, which divides each pixel of the input image into contour pixels and non-contour pixels, and determines the connectivity of each contour pixel to determine whether the live image contains at least one complete object. In the embodiment of the present application, whether the live image has significance is determined by judging whether the live image contains at least one or more complete objects: if the live image does not contain at least one or more complete objects, the live image has significance; if the live image does not contain a complete object, the live image has no significance. If the live image is judged to be significant, contour detection continues; if the live image is not significant, contour detection is terminated. The conclusion that the live image does not contain a complete object is drawn in advance, and contour detection is not required, which reduces contour detection of invalid images and reduces the false detection rate.

[0058] The classification module includes an average pooling layer, a vector conversion layer and several fully connected layers connected in sequence;

[0059] Downsampling the contour feature map output by the encoder to a second preset resolution through the average pooling layer;

[0060] The second preset resolution contour feature map is converted into a one-dimensional vector of a preset length through the vector conversion layer, and a binary classification value indicating whether the live broadcast image contains at least one complete object is obtained through the multiple fully connected layers.

[0061] The vector conversion layer converts the contour feature map of the second preset resolution into a one-dimensional vector with a length of 512. The classification module includes three fully connected layers with node numbers of 64, 16, and 1. The three fully connected layers connect the one-dimensional vectors to obtain a binary classification value indicating whether the live image contains at least one complete object.

[0062] The decoder is used to decode the contour feature map obtained by encoding by the encoder to obtain the contour of the object in the live image. In one embodiment, the decoder includes a plurality of decoding layers and output layers connected in sequence; wherein each decoding layer corresponds to an encoding layer; the step of extracting the contour of the object in the live image by the decoder includes:

[0063] Performing bilinear interpolation on the outputs of the corresponding encoding layers through the decoding layers respectively, and upsampling to the first preset resolution;

[0064] The object contours in the live image are extracted through the output layer.

[0065] Bilinear interpolation is an upsampling method that uses the existing four pixels in the original image to interpolate new pixels in two directions to improve the resolution of the image. Compared with other upsampling methods, bilinear interpolation is calculated based on the pixels in the original image to avoid jagged edges and obtain smoother, high-resolution images.

[0066] S103: extracting straight line segments in the contour map of the live broadcast image;

[0067] In one embodiment, a line segment detection algorithm can be used to extract the line segments in the contour map of the live image. The line segment detection algorithm (a Line Segment Detector, LSD) refers to detecting the gradient value and gradient direction of each pixel point of the image, thereby performing line segmentation on the input grayscale image based on the gradient value and gradient direction to obtain a plurality of line segments. Specifically, the step of extracting the line segments in the contour map of the live image includes:

[0068] Based on the Gaussian downsampling method, downsampling the contour map of the live broadcast image to a preset image scale;

[0069] Obtaining the gradient value and gradient direction value of each pixel point of the contour map of the live image;

[0070] Eliminate pixels whose gradient values ​​are less than the preset gradient threshold, and select the pixel with the maximum gradient value as the seed point;

[0071] Determine a direction value range based on the gradient direction value of the seed point and a preset range threshold, obtain pixel points whose gradient direction values ​​are within the direction value range, and obtain a plurality of homogeneous points;

[0072] Based on the position information of the plurality of isotropic points, generating a rectangle including the plurality of isotropic points;

[0073] Obtain the length and width of the rectangle, and calculate the density of homosexual points of the rectangle according to the number of homosexual points in the rectangle;

[0074] If the density of the isotropic points is greater than or equal to a set density threshold, based on a fitted rectangle accuracy calculation function, an error value of the rectangle in the contour map is obtained;

[0075] If the error value is less than or equal to a preset threshold, the straight line segment of the rectangle is used as the straight line segment in the contour map of the live image; if the error value is greater than the preset threshold, the side length of the rectangle is adjusted until the error value of the rectangle obtained based on the fitting rectangle accuracy calculation function is less than or equal to the preset threshold.

[0076] The contour map of the live image is downsampled based on the Gaussian downsampling method, and the jagged phenomenon of the image can be effectively solved by reducing the image, thereby improving the accuracy of straight line segment detection. In the embodiment of the present application, the preset image scale can be 0.8.

[0077] Specifically, the gradient value can be calculated according to the grayscale values ​​i(x+1, y), i(x, y+1) and i(x+1, y) of the pixel point (x, y) and its neighboring pixel points (x+1, y), (x, y+1) and (x+1, y) according to the gradient value calculation formula.

[0078]

[0079]

[0080] Among them, G(x,y) is the gradient value, g x (x, y) is the first gray value, g y (x, y) is the second grayscale value.

[0081] The gradient direction value can be calculated according to the grayscale values ​​i(x+1, y), i(x, y+1) and i(x+1, y) of the pixel point (x, y) and its neighboring pixel points (x+1, y), (x, y+1) and (x+1, y) according to the gradient direction value calculation formula.

[0082] The calculation formula of the gradient direction value is:

[0083]

[0084] Among them, θ is the gradient direction value.

[0085] The gradient threshold can be set according to actual needs.

[0086] The direction value range is determined based on the gradient direction value of the seed point and a preset range threshold. Specifically, the direction value range can be [at, a+t], where a is the gradient direction value of the seed point, and t is the preset range threshold. The range threshold can be set according to the actual needs of the user.

[0087] After determining the direction value range, the seed point is used as the starting point to search for each pixel point of the contour map of the live image to obtain pixel points (i.e., homogeneous points) whose gradient direction values ​​are within the direction value range, and generate a rectangle containing all homogeneous points.

[0088] The density of homogeneous points is used to determine the number of homogeneous points in the rectangle, which can be obtained by dividing the number of homogeneous points by the area of ​​the rectangle. The density threshold can be set according to the size of the input image and the actual needs of the user. When the homogeneous point density of a rectangle is less than the set density threshold, the rectangle can be truncated to convert it into multiple rectangles, and the homogeneous point density of the truncated rectangle is recalculated until the homogeneous point density of the obtained rectangle is greater than or equal to the set density threshold.

[0089] The fitted rectangle accuracy calculation function (Number of False Alarms, NFA) is used to evaluate the accuracy of the fitted rectangle. In the embodiment of the present application, when the error value is less than or equal to the preset threshold, it is judged that the current fitted rectangle meets the set requirements, and the straight line segments of the rectangle are used as the straight line segments in the contour map of the live image. If the error value is greater than the preset threshold, the side length of the rectangle is adjusted to cut it into multiple rectangular frames, and the error value is obtained based on the fitted rectangle accuracy calculation function until the error value is less than or equal to the preset threshold.

[0090] S104: converting the straight line segments into straight lines, obtaining the position information of the intersections between every two straight lines, and merging the intersections satisfying the merging conditions according to a preset merging condition;

[0091] In one embodiment, the step of converting the straight line segment into a straight line specifically includes:

[0092] According to the position information of the endpoints of the straight line segment, the slope and intercept of the straight line corresponding to the straight line segment are obtained;

[0093] According to the slope and the intercept, a straight line corresponding to the straight line segment is obtained.

[0094] The position information of the endpoints of the straight line segment includes the position information of the two endpoints of the straight line segment, and the slope and intercept of the straight line corresponding to the two straight line segments are obtained according to the position information of the endpoints of the two straight line segments, and the straight line corresponding to the straight line segment is determined according to the slope and intercept. In a preferred embodiment, the amount of data calculation can be reduced and the correction efficiency can be improved by merging close straight lines. Therefore, after the step of obtaining the straight line corresponding to the straight line segment, the following steps are also included:

[0095] According to the slope and the intercept, merging straight lines that meet a preset merging condition;

[0096] The preset merging condition includes: the slope difference between at least two straight lines is within a preset slope difference range, and the intercept difference between the at least two straight lines is within an intercept difference range.

[0097] The intersection point is the intersection point between two straight lines. For adjacent intersection points, the amount of data calculation can be reduced by merging the intersection points to improve the correction efficiency. Specifically, the steps of merging the intersection points that meet the merging conditions according to the preset merging conditions include:

[0098] When there are at least two intersections whose distance is less than a set threshold, obtaining the average of the position information of the at least two intersections;

[0099] The at least two intersection points are merged, and a merged intersection point is generated according to an average of the position information of the at least two intersection points.

[0100] By merging adjacent intersections and generating a new intersection at the midpoint of the adjacent intersections according to the average of the position information of the adjacent intersections, the number of intersections is reduced and the correction efficiency is improved.

[0101] S105: Obtaining the area of ​​a quadrilateral formed by every four intersections, and obtaining the position information of the four intersections with the largest quadrilateral areas;

[0102] Every four intersection points can determine a quadrilateral, and the quadrilateral area to be corrected in the contour map of the live image can be determined by using the position information of the four intersection points with the largest quadrilateral area.

[0103] In one embodiment, after the step of obtaining the area of ​​the quadrilateral formed by every four intersection points, the method further includes:

[0104] Obtain a quadrilateral formed by every four intersection points, and obtain a quadrilateral that satisfies a preset quadrilateral screening condition;

[0105] Among them, the preset quadrilateral screening conditions include: the angle between adjacent sides of the quadrilateral is greater than the set angle threshold, the length ratio of the opposite sides of the quadrilateral is greater than the set ratio threshold, the quadrilateral has at least one set of parallel opposite sides, and the aspect ratio of the quadrilateral is greater than the set aspect ratio threshold.

[0106] The quadrilateral has at least one set of parallel opposite sides, so that the final corrected quadrilateral area is a trapezoid or a parallelogram, which is more convenient for affine transformation. The angle threshold, the ratio threshold, and the aspect ratio threshold can be set according to the size of the input image and the size of the contour contained in the image. For example, the quadrilateral screening conditions can be set as follows: the angle between adjacent sides of the quadrilateral is greater than 4 degrees, the length ratio of the opposite sides of the quadrilateral is greater than 0.5, the quadrilateral has at least one set of parallel opposite sides, and the ratio of the shortest side to the longest side of the quadrilateral is greater than 0.15.

[0107] S106: acquiring an affine transformation matrix based on preset target image position information and position information of the four intersection points, and correcting the contour map of the live image using the affine transformation matrix to acquire a corrected contour map of the live image.

[0108] Affine transformation refers to a linear transformation between two-dimensional coordinates. After affine transformation, the lines in the image can maintain their original straightness and relative position relationship. Affine transformation includes transformations such as translation, scaling, flipping, rotation and shearing. In the embodiment of the present application, an affine transformation matrix can be constructed based on the target image position information and the position information of the four intersections. The affine transformation matrix is ​​used to correct the contour map of the live image, so that the live contour map extracted by contour detection is clearer and easier to be recognized, thereby improving the recognition efficiency of the contour image.

[0109] like Figure 3 As shown in the figure, it is an example of extracting the contour map of a live broadcast image using a pre-trained weak semantic contour detection model. The input image includes 256×192×3 feature points, and the weak semantic contour detection model includes an encoder, a classification module (cls_out), a decoder, and 5 connection layers (skip-layer5, skip-layer4, skip-layer3, skip-layer2, skip-layer1);

[0110] Among them, the encoder includes an input layer (InConv) and 5 encoding layers (Encoder1, Encoder2, Encoder3, Encoder4, Encoder5) connected in sequence, and the input layer and the 5 encoding layers are used to convolve and downsample the live image to obtain contour features in the live image.

[0111] The decoder includes 5 decoding layers (Decoder1, Decoder2, Decoder 3, Decoder4, Decoder 5) corresponding to the 5 coding layers and an output layer (OutConv). Each coding layer is connected to the corresponding decoding layer through a fully connected layer. Each decoding layer is used to upsample the output of its coding layer and the output of the previous coding layer to extract the contour of the object in the live image. Using a fully connected layer to connect the coding layer and the decoding layer can avoid information loss during the encoding process, so that the decoding layer can combine the information that has not been lost in the corresponding coding layer before encoding to encode when decoding, so that the extracted object contour is more accurate.

[0112] like Figure 4 As shown, it is a contour map extracted by using the contour correction method of the live image described in this application. It can be seen that the contour map of the image obtained after correction is clearer and easier to recognize, which can effectively improve the detection efficiency of the target object.

[0113] According to experiments, when the contour correction method of live images described in this application is applied to most mid-to-high-end mobile phones (2000+ models) for contour detection, the number of floating-point operations per second (flops) is 174M, the model parameters are 386.56k, and the calculation speed is about 30ms. It can be seen that the contour correction method of live images described in this application can achieve ultra-real-time detection of contours.

[0114] In an embodiment of the present application, by extracting straight line segments in the contour map of the live image, converting the straight line segments into straight lines and obtaining the intersection points between every two straight lines, the intersection points that meet the merging conditions are merged according to preset merging conditions, thereby reducing the amount of data calculation for contour correction of the live image; by forming the position information of the four intersection points with the largest area of ​​the quadrilateral, and then obtaining the affine transformation matrix based on the preset target image position information and the position information of the four intersection points, the contour map of the live image is corrected using the affine transformation matrix. The contour correction method of the live image can quickly correct the contour map of the extracted live image, making the extracted contour map of the live image easier to be recognized, thereby improving the contour detection efficiency of the live image.

[0115] Optionally, before the step of acquiring the live image to be detected, the contour correction method of the live image further includes:

[0116] Acquire a preset weak semantic training sample set; the weak semantic training sample set includes a plurality of images with complete objects and their corresponding contour images and a plurality of images with incomplete objects and their corresponding contour images;

[0117] Based on the decoding-encoding framework, a weak semantic contour detection model with contour extraction function is constructed;

[0118] The weak semantic contour detection model is pre-trained using the weak semantic training sample set until the loss value of the weak semantic contour detection model meets the target loss, thereby obtaining the pre-trained weak semantic contour detection model.

[0119] The plurality of images with complete objects and the plurality of images with incomplete objects may be a collection of manually photographed images of indoor scenes, street scenes or landscapes, and the corresponding contour images may be images extracted by a contour detection algorithm. In an embodiment of the present application, the contour images corresponding to the images with incomplete objects are black images without contours.

[0120] In one embodiment, the weak semantic training sample set is an image collected from a live broadcast scene; or, the weak semantic training sample set is an image collected from a live broadcast room, and the plurality of images with complete objects include a combination of multiple live broadcast scenes and multiple complete objects. The live broadcast scene may include but is not limited to desktop scenes (such as solid color desktops, wooden desktops, floral desktops, desktops with sundries), handheld scenes (such as hand-held, hand-held), wall backgrounds (such as solid color walls, floral walls) and lighting change scenes (such as dark light, strong light, backlight, reflection), etc. The multiple objects may include but are not limited to cards (such as work cards, membership cards, ID cards), bills (such as receipts, tax bills, registration forms), electronic (mobile phones, computers, tablets) and other objects (such as notices, books, signs, paper boxes), etc.

[0121] The plurality of images with complete objects and their corresponding contour maps are used as positive samples of the weak semantic contour detection model, and the plurality of images with incomplete objects and their corresponding contour maps are used as negative samples of the weak semantic contour detection model; the weak semantic contour detection model is pre-trained by using the positive samples and negative samples until the loss value of the weak semantic contour detection model meets the target loss, thereby obtaining the pre-trained weak semantic contour detection model.

[0122] Specifically, the step of pre-training the weak semantic contour detection model using the weak semantic training sample set includes: adjusting model parameters of the weak semantic contour detection model based on the Adam optimization algorithm.

[0123] Model parameters may include parameters such as learning rate and training period.

[0124] The Adam optimization algorithm is a method for calculating an adaptive learning rate for each parameter, and can iteratively update the neural network weights based on training data; in an embodiment of the present application, the initial learning rate of the weak semantic contour detection model is set to 0.001, and the training is performed for 200 epochs; wherein, all weak semantic training sample sets are trained once in one training epoch, and then the learning rate is decayed to 0.0005, and the learning rate is decayed to 0.0001 after the 300th epoch, and fine-tuned after the 400th epoch, freezing all batch normalization layers (Batch Normalization, BN) in the network, and the learning rate is decayed to 0.00005 again, wherein freezing all batch normalization layers in the network means that the batch normalization layers are not allowed to participate in network training, that is, the parameters of the batch normalization layers are not updated.

[0125] The weak semantic contour detection model described in the embodiment of the present application performs contour detection by dividing each pixel of the input image into contour pixels and non-contour pixels. However, since the number of contour pixels is usually small, it is easy to cause imbalance in category prediction. Therefore, in a preferred embodiment, the loss function of the weak semantic contour detection model is:

[0126]

[0127] Where L represents the loss value, β represents the first coefficient, β=|Y - | / |Y + |,|Y _ | represents the number of non-contour pixels in the live image, |Y + | represents the number of contour pixels in the live image, y j represents the predicted value of the weak semantic contour detection model at pixel j. j =1|X)=σ(a j )∈[0,1],P(y j =0|X)=σ(a j )∈[0,1], σ(*) represents the sigmoid function, wherein the sigmoid function is an activation function of a neural network, which maps variables to between 0 and 1. By introducing a category-balanced cross entropy loss function as the loss function of the weak semantic contour detection model, the imbalance of category prediction can be avoided and the accuracy of contour pixel detection can be improved.

[0128] Embodiment 2:

[0129] This embodiment provides a contour correction device for a live image, which can be used to execute the contour correction method for a live image of Embodiment 1 of the present application. For details not disclosed in this embodiment, please refer to Embodiment 1 of the present application.

[0130] See also Figure 5 , Figure 5 1 is a schematic diagram of the structure of a contour correction device for a live image disclosed in an embodiment of the present application. The contour correction device for a live image can be run in a server or a live client. The contour correction device for a live image includes:

[0131] An acquisition module 301 is used to acquire a live image to be detected;

[0132] The contour map acquisition module 302 is used to extract the contour of the live image and obtain the contour map of the live image;

[0133] A straight line segment extraction module 303 is used to extract straight line segments in the contour map of the live image;

[0134] The intersection acquisition module 304 is used to convert the straight line segments into straight lines, obtain the position information of the intersections between every two straight lines, and merge the intersections that meet the merging conditions according to a preset merging condition;

[0135] The position information acquisition module 305 is used to acquire the area of ​​a quadrilateral formed by every four intersections, and acquire the position information of the four intersections with the largest quadrilateral area;

[0136] The correction module 306 is used to obtain an affine transformation matrix based on the preset target image position information and the position information of the four intersection points, and use the affine transformation matrix to correct the contour map of the live image to obtain the corrected contour map of the live image.

[0137] The embodiment of the present application obtains a live image to be detected, uses a classification module in a trained weak semantic contour detection model to determine whether the live image contains at least one complete object, and extracts the object contour in the live image through a decoder when detecting that the live image contains at least one complete object. The weak semantic contour detection model only focuses on complete and significant target objects, which can effectively reduce the amount of calculation in contour detection and improve detection efficiency. For live images without complete objects, contour detection is terminated in time, thereby reducing the false detection rate.

[0138] Embodiment three:

[0139] This embodiment provides an electronic device that can be used to perform all or part of the steps of the contour correction method of the live image in the first embodiment of the present application. For details not disclosed in this embodiment, please refer to the first embodiment of the present application.

[0140] See also Figure 6 , Figure 6 The electronic device 400 may be, but is not limited to, a combination of one or more of various servers, personal computers, laptops, smart phones, tablet computers, and the like.

[0141] In a preferred embodiment of the present application, the electronic device 400 includes a memory 401 , at least one processor 402 , at least one communication bus 403 and a transceiver 404 .

[0142] Those skilled in the art should understand that Figure 6 The structure of the electronic device shown does not constitute a limitation of the embodiments of the present application, and may be either a bus structure or a star structure. The electronic device 400 may also include more or less other hardware or software than shown in the figure, or a different component arrangement.

[0143] In some embodiments, the electronic device 400 is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits, programmable gate arrays, digital processors, and embedded devices. The electronic device 400 may also include client devices, which include but are not limited to any electronic product that can interact with a client through a keyboard, mouse, remote control, touchpad, or voice-controlled device, such as a personal computer, tablet computer, smart phone, digital camera, etc.

[0144] It should be noted that the electronic device 400 is only an example, and other existing or future electronic products that are suitable for the present application should also be included in the protection scope of the present application and included here by reference.

[0145] In some embodiments, the memory 401 stores a computer program, and when the computer program is executed by the at least one processor 402, all or part of the steps in the contour correction method of the live image in the first and second embodiments are implemented. The memory 401 includes a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0146] In some embodiments, the at least one processor 402 is the control core (Control Unit) of the electronic device 400, and uses various interfaces and lines to connect the various components of the entire electronic device 400, and executes or executes the programs or modules stored in the memory 401, and calls the data stored in the memory 401 to perform various functions of the electronic device 400 and process data. For example, when the at least one processor 402 executes the computer program stored in the memory, it implements all or part of the steps of the contour correction method of the live image described in the embodiment of the present application; or implements all or part of the functions of the contour correction device of the live image. The at least one processor 402 can be composed of an integrated circuit, for example, it can be composed of a single packaged integrated circuit, or it can be composed of multiple integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips.

[0147] In some embodiments, the at least one communication bus 403 is configured to implement connection and communication between the memory 401 and the at least one processor 402, etc.

[0148] The electronic device 400 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0149] Embodiment four:

[0150] This embodiment provides a computer-readable storage medium on which a computer program is stored. The instructions are suitable for being loaded by a processor and executing the contour correction method of a live image of the first embodiment of the present application. The specific execution process can be found in the specific description of the first embodiment and will not be repeated here.

[0151] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the components described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present application scheme. Ordinary technicians in this field can understand and implement it without paying creative work.

[0152] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0153] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0154] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for correcting the contour of a live image, characterized in that: The following steps are involved: Acquire the live image to be detected; Performing contour extraction on the live broadcast image to obtain a contour map of the live broadcast image; Extracting straight line segments in the contour map of the live image; Convert the straight line segments into straight lines, obtain the position information of the intersection points between every two straight lines, and merge the intersection points that meet the merging conditions according to a preset merging condition; Obtain the area of ​​a quadrilateral formed by every four intersections, and obtain the position information of the four intersections with the largest quadrilateral areas; Acquire an affine transformation matrix based on preset target image position information and position information of the four intersection points, and use the affine transformation matrix to correct the contour map of the live image to obtain a corrected contour map of the live image; The step of extracting straight line segments in the contour map of the live image comprises: Based on the Gaussian downsampling method, downsampling the contour map of the live broadcast image to a preset image scale; Obtaining the gradient value and gradient direction value of each pixel point of the contour map of the live image; Eliminate pixels whose gradient values ​​are less than the preset gradient threshold, and select the pixel with the maximum gradient value as the seed point; Determine a direction value range based on the gradient direction value of the seed point and a preset range threshold, obtain pixel points whose gradient direction values ​​are within the direction value range, and obtain a plurality of homogeneous points; Based on the position information of the plurality of isotropic points, generating a rectangle including the plurality of isotropic points; Obtain the length and width of the rectangle, and calculate the density of homosexual points of the rectangle according to the number of homosexual points in the rectangle; If the density of the isotropic points is greater than or equal to a set density threshold, based on a fitted rectangle accuracy calculation function, an error value of the rectangle in the contour map is obtained; If the error value is less than or equal to a preset threshold, the straight line segment of the rectangle is used as the straight line segment in the contour map of the live image; if the error value is greater than the preset threshold, the side length of the rectangle is adjusted until the error value of the rectangle obtained based on the fitting rectangle accuracy calculation function is less than or equal to the preset threshold; After the step of obtaining the area of ​​the quadrilateral formed by every four intersection points, the method further includes: Obtain a quadrilateral formed by every four intersection points, and obtain a quadrilateral that satisfies a preset quadrilateral screening condition; Among them, the preset quadrilateral screening conditions include: the angle between adjacent sides of the quadrilateral is greater than the set angle threshold, the length ratio of the opposite sides of the quadrilateral is greater than the set ratio threshold, the quadrilateral has at least one set of parallel opposite sides, and the aspect ratio of the quadrilateral is greater than the set aspect ratio threshold.

2. The method for correcting the contour of a live image according to claim 1, characterized in that: The step of converting the straight line segment into a straight line specifically comprises: According to the position information of the endpoints of the straight line segment, the slope and intercept of the straight line corresponding to the straight line segment are obtained; According to the slope and the intercept, a straight line corresponding to the straight line segment is obtained.

3. The method for correcting the contour of a live image according to claim 2, characterized in that: After the step of obtaining the straight line corresponding to the straight line segment, the method further includes: According to the slope and the intercept, merging straight lines that meet a preset merging condition; The preset merging condition includes: the slope difference between at least two straight lines is within a preset slope difference range, and the intercept difference between the at least two straight lines is within an intercept difference range.

4. The method for correcting the contour of a live image according to claim 1, characterized in that: The step of merging the intersections satisfying the merging condition according to the preset merging condition comprises: When there are at least two intersections whose distance is less than a set threshold, obtaining the average of the position information of the at least two intersections; The at least two intersection points are merged, and a merged intersection point is generated according to an average of the position information of the at least two intersection points.

5. The method for correcting the contour of a live image according to any one of claims 1 to 4, characterized in that: The step of extracting the contour of the live image comprises: Acquire contour features in the live image through an encoder in a pre-trained weak semantic contour detection model; wherein the weak semantic contour detection model includes an encoder, a classification module and a decoder; Determining, by the classification module, whether the live broadcast image contains at least one complete object according to the contour feature; If it is detected that the live image contains at least one complete object, the contour of the object in the live image is extracted by the decoder.

6. A device for correcting the contour of a live image, characterized in that: The device comprises: An acquisition module, used for acquiring a live image to be detected; A contour map acquisition module, used to extract the contour of the live image and acquire the contour map of the live image; The straight line segment extraction module is used to extract the straight line segments in the contour map of the live image: based on the Gaussian downsampling method, the contour map of the live image is downsampled to a preset image scale; the gradient value and gradient direction value of each pixel point of the contour map of the live image are obtained; the pixel points with gradient values ​​less than a preset gradient threshold are eliminated, and the pixel points with the maximum gradient value are selected as seed points; the direction value range is determined based on the gradient direction value of the seed point and a preset range threshold, and the pixel points with gradient direction values ​​within the direction value range are obtained to obtain a number of homogeneous points; based on the position information of the several homogeneous points, a plurality of homogeneous points are generated. A rectangle including the plurality of isotropic points is formed; the length and width of the rectangle are obtained, and the density of isotropic points of the rectangle is calculated according to the number of isotropic points in the rectangle; if the density of isotropic points is greater than or equal to a set density threshold, the error value of the rectangle in the contour map is obtained based on a fitting rectangle precision calculation function; if the error value is less than or equal to a preset threshold, the straight line segment of the rectangle is used as the straight line segment in the contour map of the live image; if the error value is greater than a preset threshold, the side length of the rectangle is adjusted until the error value of the rectangle obtained based on the fitting rectangle precision calculation function is less than or equal to the preset threshold; An intersection acquisition module, used for converting the straight line segments into straight lines, acquiring the position information of the intersections between every two straight lines, and merging the intersections satisfying the merging conditions according to a preset merging condition; A position information acquisition module is used to acquire the area of ​​a quadrilateral formed by every four intersections, acquire a quadrilateral formed by every four intersections, and acquire a quadrilateral that meets a preset quadrilateral screening condition; wherein the preset quadrilateral screening condition includes: the angle between adjacent sides of the quadrilateral is greater than a set angle threshold, the length ratio of opposite sides of the quadrilateral is greater than a set ratio threshold, the quadrilateral has at least one set of opposite sides that are parallel, and the aspect ratio of the quadrilateral is greater than a set aspect ratio threshold; acquire the position information of the four intersections with the largest area of ​​the formed quadrilateral; The correction module is used to obtain an affine transformation matrix based on preset target image position information and the position information of the four intersection points, and use the affine transformation matrix to correct the contour map of the live image to obtain the corrected contour map of the live image.

7. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the contour correction method of a live image as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the contour correction method of a live image as described in any one of claims 1 to 5 is implemented.

Images