Screen positioning method, device, electronic device and computer readable storage medium

Abstract

The present application is applied to the field of image processing technology, and discloses a screen positioning method, device, electronic device and computer-readable storage medium, including: obtaining image data captured by a camera, calling a straight line detection model to detect each straight line segment in the image data, and calling a target area detection model to detect each rectangular area in the image data; determining at least one candidate closed area from the image data according to each rectangular area, each straight line segment and the potential intersection between each straight line segment, and respectively determining the closed area of ​​each candidate closed area and the parallelism between each candidate straight line segment on the candidate closed area; respectively determining the candidate value of each candidate closed area based on the closed area and parallelism corresponding to each candidate closed area; and selecting the candidate closed area with the largest candidate value from each candidate closed area as the target screen area. The present application aims to solve the technical problem that it is difficult to accurately locate the area of ​​an electronic screen when the shooting background is complex.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology and relates to a screen positioning method, device, electronic device and computer-readable storage medium. Background Art

[0002] In scenarios such as multimedia teaching automatic recording and broadcasting, video conference recording, etc., electronic screens are often used as the main carrier of information interaction. Therefore, how to record the electronic screen in real time through video is of great value. However, in natural scenes, after the electronic screen is photographed, the electronic screen's performance in the image is relatively complex. For example, the scene where the electronic screen is located may have a relatively complex background, which will interfere with the positioning of the screen area during recording. In addition, the camera lens is tilted relative to the surface of the electronic screen during shooting, and the shooting distance is uncertain. Factors such as these make it impossible to accurately locate the position of the electronic screen, thereby meeting the user's needs for shooting and recording the electronic screen.

[0003] At present, screen areas are often extracted through image segmentation technology, such as edge segmentation, image binarization segmentation and other classic image processing technologies. However, in scenes with complex backgrounds, it is still difficult to accurately identify the position of the electronic screen, and thus it is difficult to meet users' needs for shooting and recording electronic screens.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of the present application is to provide a screen positioning method, device, electronic device and computer-readable storage medium, aiming to solve the technical problem that it is difficult to accurately locate the area of ​​the electronic screen when the shooting background is complex.

[0006] To achieve the above object, the present application provides a screen positioning method, the screen positioning method comprising:

[0007] Acquire image data captured by a camera, call a straight line detection model to detect straight line segments in the image data, and call a target area detection model to detect rectangular areas in the image data;

[0008] Determine at least one candidate closed region from the image data according to each of the rectangular regions, each of the straight line segments, and the potential intersection points between the straight line segments, and respectively determine a closed area ratio of each of the candidate closed regions and a straight line parallelism of the candidate closed regions;

[0009] Determine the candidate value of each candidate closed region based on the closed area and the parallelism corresponding to each candidate closed region;

[0010] A candidate closed area with the largest candidate value is selected from among the candidate closed areas as the target screen area.

[0011] To achieve the above object, the present application provides a screen positioning device, the screen positioning device comprising:

[0012] An acquisition module is used to acquire image data captured by a camera, call a straight line detection model to detect each straight line segment in the image data, and call a target area detection model to detect each rectangular area in the image data;

[0013] a candidate closed region determination module, configured to determine at least one candidate closed region from the image data according to each of the rectangular regions, each of the straight line segments and potential intersections between the straight line segments, and to respectively determine a closed area ratio of each of the candidate closed regions and a straight line parallelism of the candidate closed regions;

[0014] A candidate value determination module, configured to determine the candidate value of each candidate closed region based on the closed area and the parallelism corresponding to each candidate closed region;

[0015] The screen determination module is used to select a candidate closed area with the largest candidate value from among the candidate closed areas as a target screen area.

[0016] The present application also provides an electronic device, which includes: a memory, a processor, and a program of the screen positioning method stored in the memory and executable on the processor. When the program of the screen positioning method is executed by the processor, the steps of the screen positioning method as described above can be implemented.

[0017] The present application also provides a computer-readable storage medium, on which is stored a program for implementing the screen positioning method. When the program of the screen positioning method is executed by a processor, the steps of the screen positioning method as described above are implemented.

[0018] The present application also provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the steps of the screen positioning method as described above are implemented.

[0019] The present application provides a screen positioning method, device, electronic device and computer-readable storage medium, which obtain image data captured by a camera, call a straight line detection model to detect each straight line segment in the image data, and call a target area detection model to detect each rectangular area in the image data; determine at least one candidate closed area from the image data according to each rectangular area, each straight line segment and the potential intersection between each straight line segment, and respectively determine the closed area ratio of each candidate closed area and the straight line parallelism of the candidate closed area; determine the candidate value of each candidate closed area based on the closed area and the parallelism corresponding to each candidate closed area; select the candidate closed area with the largest candidate value among the candidate closed areas as the target screen area.

[0020] The present application obtains image data captured by a camera, and calls a straight line detection model and a target area detection model respectively, so as to detect a rectangular area and each straight line segment in the image data; then, according to the rectangular area, the straight line segment and the potential intersection points between the straight line segments, at least one candidate closed area is determined from the image data, and then each potential closed area is analyzed to obtain a target screen area, so as to determine a candidate closed area that can be formed by straight line segments through the rectangular area, the straight line segment and the potential intersection points between the straight line segments, so as to exclude straight line segments that will not form a closed area, and reduce the interference of straight line segments corresponding to non-closed areas under complex backgrounds; further, by determining the closed area ratio and straight line parallelism of the candidate closed area, and determining the candidate value of the candidate closed area through the closed area and parallelism, and selecting the candidate closed area with the largest candidate value as the target screen area, it is achieved that a candidate closed area with relatively high levels of closed area and parallelism is selected from each candidate closed area as the target screen area, so as to exclude the interference of candidate areas with small areas and / or low parallelism of the candidate closed area, so as to achieve accurate positioning of the target screen area even when the shooting background is complex, and improve the accuracy of positioning the target screen area. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the description, are used to explain the principles of the present application.

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0023] Figure 1 This is a flow chart of the first embodiment of the screen positioning method of the present application;

[0024] Figure 2 A schematic diagram of data annotation of a training image in the screen positioning method of this application;

[0025] Figure 3 This is a flow chart of the second embodiment of the screen positioning method of the present application;

[0026] Figure 4 A schematic diagram of generating a candidate closed area for the screen positioning method of this application;

[0027] Figure 5 A schematic diagram of a device according to an embodiment of the screen positioning method of the present application;

[0028] Figure 6 Schematic diagram of the device structure of the hardware operating environment involved in the screen positioning method in the embodiment of the present application.

[0029] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0030] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of this application.

[0031] Embodiment 1

[0032] In scenarios such as multimedia teaching automatic recording and broadcasting, video conference recording, etc., electronic screens are often used as the main carrier of information interaction. Therefore, it is of great value to use algorithms to automatically locate the area where they are located and detect their status. However, in natural scenes, after the electronic screen is photographed by a camera, the "way and content" of its expression in the image is relatively complex, which is specifically reflected in the following aspects: 1) The imaging conditions are complex, the indoor light intensity is uncertain, and the tilt, pitch angle, and shooting distance of the camera lens relative to the screen surface are uncertain, which makes the position and size of the screen area in the image uncertain, the brightness is uncertain, and the shape is an irregular convex quadrilateral; 2) The imaging background is relatively complex, and there is a relatively complex background indoors, which interferes with the positioning of the screen area; 3) The screen content is uncertain, and various situations such as blue screen, black screen, and flower screen will appear, which brings challenges to the robustness of the algorithm; the existing methods often extract the screen area through image segmentation technology, and use some classic image processing technologies such as edge segmentation and image binary segmentation. The generalization of its effect is weak in complex scenes, and it is difficult to meet large-scale commercialization.

[0033] Reference Figure 1 The present application provides a screen positioning method. In a first embodiment of the screen positioning method of the present application, the screen positioning method includes:

[0034] Step S10, acquiring image data captured by a camera, calling a straight line detection model to detect straight line segments in the image data, and calling a target area detection model to detect rectangular areas in the image data;

[0035] It should be noted that the embodiments of the present application can be applied to a screen recording system, and the screen recording system can obtain image data captured by a camera. Specifically, the screen recording system can obtain image data captured by a camera through a SOC (System of Chip) interface, and the screen recording system can store image data. The image data is an image frame captured by a camera, and the image data captured by the camera can be obtained in real time, and each image data can be analyzed to accurately locate the screen area in the image data.

[0036] The line detection model is used to identify straight line segments in image data, and the target region detection model is used to identify multiple rectangular regions in image data. Rectangular regions are regular quadrilaterals. Both the line detection model and the target region detection model are pre-trained.

[0037] Exemplarily, before step S10, the training process of the straight line detection model and the target area detection model may be: obtaining training samples, determining the straight line model to be trained and the target detection model to be trained, training the straight line model to be trained based on the training samples and the loss function, obtaining the straight line model to be quantized after the straight line model to be trained is successfully trained, and training the target detection model to be trained according to the training samples and the loss function, obtaining the target detection model to be quantized after the target detection model to be trained is successfully trained; quantizing the straight line model to be quantized to obtain the straight line detection model, and obtaining the target area detection model from the quantized target detection model. The straight line detection model and the target area detection model may be deployed on the screen recording system. The straight line detection model and the target area detection model deployed on the screen recording system may be updated, for example, the weight of the straight line detection model may be updated, and the weight corresponding to the target area detection model may also be updated. Among them, the model of the YOLO series may be used as the target detection model to be trained, for example, the target detection model to be trained may be a YOLO v3 model, and the straight line model to be trained may be an MLSD (straight line detection) model.

[0038] Among them, the step of obtaining training samples can be to obtain an image captured by a camera in an actual scene as a training image, and perform data annotation on the training image, including: annotating the position of the electronic screen in the training image, the annotation format of the electronic screen is bbx (bounding box), and also including annotating the straight line segments in the training image, and the annotation method of the straight line segments can be to mark the starting point and the end point of the straight selection segment.

[0039] To better understand the process of data annotation in training images, refer to Figure 2 , Figure 2 Schematic diagram of the annotation of the straight line segment of the electronic screen in the training image: P can be used to represent the coordinates of a point in the training image, and the upper left corner point of the electronic screen (P lt ), and the lower right corner (P rd ) to mark the position of the electronic screen in the training image bbx(P lt ,P rd ), you can mark the starting point of the straight line segment (P start ) and the end point (P end ) to mark the position of the straight line segment in the training image Line(P start ,P end ), 3 is a training image, the upper left corner of the training image is the coordinate origin, X is the horizontal coordinate, Y is the vertical coordinate, 1 is the real electronic screen, and 2 is the position of the marked electronic screen. The training sample is obtained by marking the straight line segments in the training image and the corresponding positions of the electronic screen. The training sample can include multiple training images.

[0040] The embodiment of the present application obtains training samples to train a straight line model to be trained and a target detection model to be trained, and quantizes the trained models to obtain target detection models and target area detection models, thereby improving the accuracy of identifying straight line segments and rectangular areas in images. Moreover, since the trained models are quantized, the running amount of the straight line detection model and the target area detection model is reduced, and the speed of identifying straight line segments and rectangular areas in image data is improved, thereby improving the efficiency of positioning the electronic screen in the image.

[0041] Exemplarily, step S10 includes: the embodiment of the present application is applied to a screen recording system, and a straight line detection model and a target area detection model are deployed on the screen recording system; and preset closing conditions and preset screen state abnormal conditions can be configured in advance on the screen recording system; image data captured by a camera is acquired, and the straight line detection model is called to identify each straight line segment in the image data, and the target area detection model is called to identify each rectangular area in the image data; wherein each straight line segment in the image data can be represented by the starting point coordinates and the ending point coordinates of the straight line segment, and each rectangular area in the image data can be represented by the coordinates of the two diagonal points of the rectangular area, and the two diagonal point coordinates can be the upper left corner coordinates and the lower right corner coordinates, or the lower left corner coordinates and the upper right corner coordinates. The embodiment of the present application detects the straight line segments and rectangular areas in the image data through the model, thereby improving the detection accuracy of the image data.

[0042] Step S20, determining at least one candidate closed region from the image data according to each of the rectangular regions, each of the straight line segments and the potential intersections between the straight line segments, and determining a closed area ratio of each of the candidate closed regions and a straight line parallelism of the candidate closed region respectively;

[0043] It should be noted that potential intersections are characterized as possible intersections between two straight line segments, which can be intersections where two straight line segments have already intersected, or intersections where two straight line segments may intersect. The candidate closed area can be expressed as an irregular quadrilateral. The closed area ratio is characterized as the ratio of the area size of the candidate closed area to the image area of ​​the image data, and the candidate straight line segment is any straight line segment on the edge of the candidate closed area. Line parallelism is the parallelism between two non-adjacent candidate straight line segments in the candidate closed area. The closer the line parallelism is to 1, the higher the line parallelism of the candidate closed area, and the higher the possibility that the candidate closed area is the area where the electronic screen is located. The candidate closed area is a closed area that may be formed by combining the straight line segments, so that the interference of other straight line segments that will not constitute a closed area can be eliminated.

[0044] Exemplarily, based on each of the straight line segments, potential intersections between the straight line segments are determined, and based on each of the potential intersections, at least one potential closed region is determined from the candidate image, and for any potential closed region, each of the rectangular regions is traversed to determine the area overlap ratio between the potential closed region and the corresponding rectangular region, and when the area overlap ratio satisfies a preset candidate threshold, the potential closed region is used as the candidate closed region. After multiple candidate closed regions are determined, the closed area corresponding to the candidate closed region and the degree of parallelism between each of the candidate straight line segments on the candidate closed region can be calculated.

[0045] Step S30, determining candidate values ​​of each candidate closed region based on the closed area and the parallelism corresponding to each candidate closed region;

[0046] Step S40 , selecting a candidate closed area with the largest candidate value from among the candidate closed areas as the target screen area.

[0047] It should be noted that the candidate value is the sum of the closed area and the parallelism, and each candidate closed area has a corresponding candidate value. The target screen area is the area where the electronic screen is located. Exemplarily, the closed area and the parallelism of the candidate closed areas are aggregated respectively to obtain the candidate values ​​corresponding to each of the candidate closed areas, and the candidate closed area with the largest candidate value is selected from among the candidate closed areas as the target screen area.

[0048] The present application obtains image data captured by a camera, and calls a straight line detection model and a target area detection model respectively, so as to detect a rectangular area and each straight line segment in the image data; then, according to the rectangular area, the straight line segment and the potential intersection points between the straight line segments, at least one candidate closed area is determined from the image data, and then each potential closed area is analyzed to obtain a target screen area, so as to determine a candidate closed area that can be formed by straight line segments through the rectangular area, the straight line segment and the potential intersection points between the straight line segments, so as to exclude straight line segments that will not form a closed area, and reduce the interference of straight line segments corresponding to non-closed areas under complex backgrounds; further, by determining the closed area ratio and straight line parallelism of the candidate closed area, and determining the candidate value of the candidate closed area through the closed area and parallelism, and selecting the candidate closed area with the largest candidate value as the target screen area, it is achieved that a candidate closed area with relatively high levels of closed area and parallelism is selected from each candidate closed area as the target screen area, so as to exclude the interference of candidate areas with small areas and / or low parallelism of the candidate closed area, so as to achieve accurate positioning of the target screen area even when the shooting background is complex, and improve the accuracy of positioning the target screen area.

[0049] Embodiment 2

[0050] Further, refer to Figure 3 Based on the above embodiment of the present application, in another embodiment of the present application, the same or similar contents as the above embodiment can be referred to the above introduction, and will not be described in detail later. On this basis, the step of determining at least one candidate closed area from the image data according to each of the rectangular areas, each of the straight line segments, and the potential intersections between the straight line segments includes:

[0051] Step A10, determining at least one potential closed area from the image data according to each of the straight line segments and the potential intersection points between the straight line segments;

[0052] It should be noted that the potential closed area is an area that may be closed, and there may be multiple potential closed areas in the image data. Potential intersections can be expressed as coordinates. Since the image data can be an image frame, the two adjacent edges of the image frame and the intersection of the two edges can be used as a two-dimensional coordinate system. The coordinates of each point in the image data can be determined based on the two-dimensional coordinate system, wherein the intersection of the two edges can be the origin of the two-dimensional coordinate system, and the two edges can be the x-axis and y-axis, respectively. The potential closed area can also be expressed in the form of coordinates, which can be the coordinate representation of each potential intersection on the potential closed area. Exemplarily, based on each of the straight line segments, the potential intersections between the straight line segments are determined, and based on each of the potential intersections, at least one potential closed area in the image data is determined.

[0053] In one possible embodiment, step A10 includes:

[0054] Step A11, determining potential intersection points between the straight line segments according to the extended trend lines of the straight line segments;

[0055] Step A12, selecting a plurality of potential vertices satisfying a preset closing condition from each of the potential intersection points;

[0056] Step A13: determining at least one potential closed area according to the vertex identifier of each potential vertex, wherein the vertex identifier is determined based on the straight line identifiers corresponding to two straight line segments that potentially intersect the potential vertex.

[0057] It should be noted that the extended trend line is characterized as a straight line extending to both ends of the straight line segment. The potential intersection point can be on the extended trend line of the straight line segment. The preset closing condition includes a preset closing distance condition and a preset closing angle threshold. The preset closing condition is used to determine whether each potential intersection point can be used as a vertex of a potential closed area. The potential vertex is characterized as a potential intersection point that can be constructed into a potential closed area. The vertex identifier is determined based on the straight line identifiers corresponding to the two straight line segments that potentially intersect at the potential vertex. The straight line identifier is the identifier of the straight line segment. The straight line identifier can be identified by letters or numbers. The vertex identifier includes the straight line identifiers of the two straight lines that potentially intersect with the potential vertex.

[0058] Exemplarily, steps A11 to A13 include: judging whether each of the straight line segments intersects or potentially intersects based on the extended trend lines of each of the straight line segments; for any straight line segment, if there is a straight line segment intersecting with the straight line segment, the point where the two straight line segments intersect is taken as a potential intersection point; if there is a straight line segment intersecting with the extended trend line of the straight line segment, the point where the two straight line segments intersect is taken as a potential intersection point; screening out potential intersection points that meet preset closing conditions, and determining potential intersection points between each of the straight line segments; determining at least one potential closed area based on the vertex identifier of each of the potential vertices, wherein the potential closed area includes four potential vertices.

[0059] The embodiment of the present application determines potential intersections and screens out potential vertices that may form a closed area through preset closing conditions, so as to form a potential closed area according to each potential vertex, thereby excluding potential vertices that cannot form a potential closed area, and improving the accuracy of determining the potential closed area. The embodiment of the present application determines multiple potential vertices through each potential intersection and preset closing conditions, and thus determines multiple potential closed areas in the image data according to each potential vertex, so as to more comprehensively detect the area that may correspond to the electronic screen and avoid missing the area corresponding to the electronic screen.

[0060] In a feasible embodiment, step A12 further includes:

[0061] Step A121, for any of the potential intersection points, determining a first potential straight line segment and a second potential straight line segment that potentially intersect the potential intersection point;

[0062] Step A122, calculating, according to a first distance from a first straight line midpoint of the first potential straight line segment to the potential intersection point, a ratio of the first distance to a first line segment length of the first potential straight line segment to obtain a first distance overlap ratio;

[0063] Step A123, calculating, according to a second distance from a second straight line midpoint of the second potential straight line segment to the potential intersection point, a ratio of the second distance to a second line segment length of the second potential straight line segment to obtain a second distance overlap ratio;

[0064] It should be noted that the first potential straight line segment and the second potential straight line segment are both straight line segments that potentially intersect at the potential intersection point; the first potential straight line segment includes a first potential intersection far end point and a first potential intersection near end point, the midpoint of the first straight line is the midpoint of the first potential straight line segment, the midpoint of the second straight line is the midpoint of the second potential straight line segment, the first potential intersection near end point is the straight line end point closest to the potential intersection point in the first potential straight line segment, and the first potential intersection far end point is the straight line end point farthest from the potential intersection point in the first potential straight line segment. The first distance overlap ratio is characterized by the ratio of the distance from the potential intersection point to the midpoint of the first straight line to the first potential straight line segment. When the first distance overlap ratio is closer to 0.5, it means that the potential intersection point coincides with the first potential intersection near end point of the first potential straight line segment.

[0065] The preset closing condition includes a preset closing distance condition and a preset closing angle condition. The preset closing distance condition includes a preset closing distance ratio range, and the preset closing distance ratio range can float above and below the preset closing distance threshold. The preset closing distance condition can be based on the actual snow top, and the preset closing distance threshold can be 0.5. The preset closing angle condition includes a preset closing angle range, and the angle range can be determined based on the actual situation.

[0066] The second potential intersection straight line segment includes a second potential intersection proximal endpoint and a second potential intersection distal endpoint. The second potential intersection proximal endpoint is the straight line endpoint closest to the potential intersection point in the second potential straight line segment. The second potential intersection distal endpoint is the straight line endpoint farthest from the potential intersection point in the second potential straight line segment. The first distance is the straight line distance between the midpoint of the first straight line and the potential intersection point, and the second distance is the straight line distance between the midpoint of the second straight line and the potential intersection point. The second distance overlap ratio is characterized by the ratio of the distance from the potential intersection point to the midpoint of the second straight line to the second potential straight line segment. When the second distance overlap ratio is closer to 0.5, it means that the potential intersection point and the second potential intersection proximal endpoint of the second potential straight line segment overlap.

[0067] Exemplary: for any of the potential intersections, determine the first potential straight line segment and the second potential straight line segment that potentially intersect the potential intersection; select the endpoint closest to the potential intersection from the two endpoints on the first potential straight line segment as the first potential intersection near endpoint, and select the endpoint closest to the potential intersection from the second potential straight line segment as the second potential intersection near endpoint; calculate the distance between the first potential intersection near endpoint and the potential intersection based on the first near endpoint coordinates of the first potential intersection near endpoint and the potential intersection coordinates to obtain the first near endpoint distance; calculate the distance between the first potential intersection far endpoint and the potential intersection based on the first far endpoint coordinates of the first potential intersection far endpoint and the potential intersection coordinates to obtain the first far endpoint distance, and use the average of the first near endpoint distance and the first far endpoint distance as the first distance; calculate the length of the first potential straight line segment based on the first near endpoint coordinates and the first far endpoint coordinates to obtain the first line segment length, and use the ratio of the first distance to the first line segment length as the first distance overlap ratio.

[0068] According to the second distal end point coordinates of the second potential intersecting distal end point and the potential intersection point coordinates, the distance between the second potential intersecting distal end point and the potential intersection point is calculated to obtain the second distal end point distance. According to the second distal end point coordinates of the second potential intersecting distal end point and the potential intersection point coordinates, the distance between the second potential intersecting distal end point and the potential intersection point is calculated to obtain the second distal end point distance, and the average of the second distal end point distance and the second distal end point distance is used as the second distance. According to the second proximal end point coordinates and the second distal end point coordinates, the length of the second potential straight line segment is calculated to obtain the second line segment length, and the ratio of the second distance to the second line segment length is used as the second distance coincidence ratio.

[0069] The embodiment of the present application determines the first distance overlap ratio and the second distance overlap ratio, thereby excluding potential intersection points corresponding to two straight line segments whose first distance overlap ratio and second distance overlap ratio do not satisfy the preset closing distance condition. If the first distance is too long or the second distance is too long, it means that there is no possibility of combining the first potential straight line segment and the second potential straight line segment into a closed area. Therefore, the corresponding potential intersection points in this case can be excluded to improve the accuracy of determining the candidate closed area.

[0070] Exemplarily, the formula for calculating the first distance overlap ratio is:

[0071]

[0072] Among them, dis i is the first distance overlap ratio, is the first potential intersection point, is the first potential intersection far point, v ij is a potential intersection point, thr i is the first distance overlap ratio, is the first proximal endpoint distance, is the first far end point distance, is the length of the first line segment.

[0073] Exemplarily, the formula for calculating the second distance overlap ratio is:

[0074]

[0075] Among them, dis j is the second distance overlap ratio, is the second potential intersection point, is the second potential intersection far point, v ij is a potential intersection point, thr j is the second distance overlap ratio, is the second proximal endpoint distance, is the distance to the second far end point, is the length of the second line segment.

[0076] Step A124, calculating a potential angle between the first potential straight line segment and the second potential straight line segment;

[0077] Step A125: If the first distance overlap ratio and the second distance overlap ratio both satisfy the preset closing distance condition, and the potential angle satisfies the preset closing angle condition, then the potential intersection point is determined to be the potential vertex.

[0078] It should be noted that the potential angle is characterized as the angle between the first potential straight line segment and the second potential straight line segment. The preset closing distance condition includes that the first distance overlap ratio is within the preset closing angle range, and the second distance overlap ratio is within the preset closing angle range; the preset closing angle condition includes that the potential angle is within the preset closing distance ratio range. When both the first distance and the second distance meet the preset closing condition, and the potential angle meets the preset closing angle threshold, it indicates that there is a potential vertex between the first straight line segment and the second straight line segment.

[0079] Exemplarily, steps A124 to A125 include: calculating the potential angle between the first potential straight line segment and the second potential straight line segment; if the first distance overlap ratio and the second distance overlap ratio both satisfy the preset closed distance condition, and the potential angle satisfies the preset closed angle condition, then determining the potential intersection as the potential vertex; if the first distance overlap ratio and / or the second distance overlap ratio do not satisfy the preset closed distance condition, or the potential angle does not satisfy the preset closed angle condition, then obtaining a new potential intersection, and returning to step A121. The embodiment of the present application determines whether an existing potential intersection can be used as a potential vertex by using distance ratios and potential angles, and then screens out vertices that can constitute a closed area, thereby improving the accuracy of determining a potential closed area and also improving the efficiency of screening out potential vertices from each potential intersection.

[0080] Exemplarily, the formula for calculating the potential angle between the first potential straight line segment and the second potential straight line segment may be:

[0081] thr ij =abs(degree(L i )-degree(L j ))

[0082] Among them, thr ij is the potential angle between the first potential straight line segment and the second potential straight line segment, L i is the first potential straight line segment, L j is the second potential straight line segment, abs represents the absolute value, degree(L i ) is the straight line L i The angle between the horizontal axes corresponding to the image data, degree(L j ) is the straight line L j The angle between the horizontal axes corresponding to the image data.

[0083] In a feasible embodiment, step A13 further includes:

[0084] Step A131, for any target potential vertex among the potential vertices, according to the target vertex identifier of the target potential vertex, determining a first target closed vertex and a second target closed vertex that match the target vertex identifier among the potential vertices;

[0085] Step A132, determining a third target closed vertex that matches both the second target closed vertex and the second target closed vertex among the potential vertices;

[0086] Step A133: determining the potential closed area according to the target potential vertex, the first target closed vertex, the second target closed vertex, and the third target closed vertex.

[0087] It should be noted that the target potential vertex is any potential vertex, the target vertex identifier is the vertex identifier of the target potential vertex, and the target vertex identifier is composed of the first straight line identifier of the target first potential straight line segment that potentially intersects the target potential vertex and the second straight line identifier of the target second potential straight line segment. It can be understood that the target vertex identifier includes the first straight line identifier and the second straight line identifier. A straight line segment has two endpoints, so a straight line segment can also correspond to two potential vertices. The first target closed vertex is the potential vertex corresponding to the other end of the target first potential straight line segment that intersects with the target potential vertex, and the second target closed vertex is the potential vertex corresponding to the other end of the second potential straight line segment that intersects with the target potential vertex.

[0088] The first target closed vertex carries a first target closed vertex identifier, which is a vertex identifier corresponding to the first target closed vertex, and is a combination of the first straight line identifier and the third straight line identifier; the second target closed vertex carries a second target closed vertex identifier, which is a vertex identifier corresponding to the second target closed vertex, and is a combination of the second straight line identifier and the fourth straight line identifier. The third straight line identifier is an identifier of the third straight line segment, and the fourth straight line identifier is an identifier of the fourth straight line segment. The third target closed vertex carries a third closed vertex identifier, and the third closed vertex identifier is a combination of the third straight line identifier and the fourth straight line identifier.

[0089] Exemplarily, steps A131 to A133 include: for any target potential vertex among the potential vertices, according to the target vertex identifier of the target potential vertex, determining in each of the potential vertices a first target closed vertex and a second target closed vertex that match the target vertex identifier; specifically, the target vertex identifier includes a first straight line identifier and a second straight line identifier, searching in each of the potential vertices for a first target closed vertex whose vertex identifier includes the first straight line identifier and does not include the second straight line identifier, and searching in each of the potential vertices for a second target closed vertex whose vertex identifier includes the second straight line identifier and does not include the first straight line identifier.

[0090] Determine the third straight line identifier from the second target closed vertex identifier of the second target closed vertex, determine the fourth straight line identifier from the third closed vertex identifier of the third target closed vertex, and search for the third target closed vertex whose vertex identifier includes the third straight line identifier and the fourth straight line identifier in each of the potential vertices; according to the target vertex identifier, the first target closed vertex identifier, the second target closed vertex identifier and the third closed vertex identifier, splice the target potential vertex, the first target closed vertex, the second target closed vertex and the third target closed vertex to obtain a potential closed area. The embodiment of the present application constructs a candidate closed area through the vertex identifier of each potential vertex, determines multiple candidate closed areas from multiple potential vertices, improves the efficiency and accuracy of determining the candidate closed area, and avoids the inaccurate determination of the candidate closed area due to random combinations between potential vertices.

[0091] Step A20, determining the area overlap ratio of each of the potential closed areas according to each of the rectangular areas;

[0092] Step A30: Screen out at least one candidate closed region whose area overlap ratio is greater than or equal to a preset candidate threshold value from among the potential closed regions.

[0093] It should be noted that the area overlap ratio is characterized by the overlap between the area of ​​the potential closed area and the area of ​​the rectangular area corresponding to the potential closed area, and the ratio of the area of ​​the potential closed area to the sum of the areas of the rectangular area corresponding to the potential closed area. The preset candidate threshold is set in advance, and the preset candidate threshold can be a numerical value, characterized by the minimum area overlap threshold that can be used as a candidate closed area. In this way, areas that can constitute a potential closed area but have a low area overlap between the corresponding rectangular area can be excluded, and areas that are obviously not corresponding to the area where the electronic screen is located can be screened out. In this way, the accuracy of locating the target screen area can be improved.

[0094] Exemplarily, steps A20 to A30 include: respectively determining the area overlap ratios between each of the potential closed regions and their corresponding rectangular regions; and selecting, from among the potential closed regions, potential closed regions whose area overlap ratios are greater than or equal to a preset candidate threshold as candidate closed regions.

[0095] For a better understanding of the embodiments of the present application, refer to Figure 4 , Figure 4 is a schematic diagram of the generation of candidate closed regions. i , L k , L l and L j are the candidate straight line segments that constitute the candidate closed region, V ij 、V jk 、Vkl and V lj They are the four potential vertices corresponding to the candidate closed area. and They are respectively the candidate straight line segments L i The two endpoints of .

[0096] In a feasible embodiment, step A20 further includes:

[0097] Step A21, for any of the potential closed areas, calculating the area overlap between the potential closed area and each of the rectangular areas in the image data, and obtaining the area overlaps to be selected;

[0098] Step A22, respectively determining the sum of the areas between each of the rectangular regions and the potential closed region, and obtaining the sum of the areas to be selected corresponding to each of the rectangular regions.

[0099] Step A23, respectively calculating the ratio of the overlap degree of the to-be-selected area corresponding to each of the rectangular areas to the sum of the to-be-selected areas, to obtain the to-be-selected area ratio corresponding to each of the rectangular areas;

[0100] Step A24, selecting the largest area ratio among the area ratios to be selected as the area overlap ratio of the potential closed area.

[0101] It should be noted that for each potential closed area, steps A21 to A22 need to be executed. The embodiment of the present application can identify multiple rectangular areas in the image data, and the overlap of the selected areas is characterized by the overlap of the areas between the potential closed area and the rectangular area. The sum of the areas to be selected is the sum of the areas of the potential closed area and the rectangular area, and the ratio of the areas to be selected is the ratio of the overlap of the areas to be selected to the sum of the areas to be selected. For each potential closed area, it is necessary to calculate the overlap of the areas to be selected and the sum of the areas to be selected with each rectangular area, and then select the largest ratio of the areas to be selected among the ratios of the areas to be selected as the overlap ratio of the area corresponding to the potential closed area.

[0102] Exemplarily, steps A21 to A22 include: for any of the potential closed areas, calculating the area overlap between the potential closed area and each of the rectangular areas in the image data, and obtaining the area overlap of each of the rectangular areas; calculating the sum of the areas between each of the rectangular areas and the potential closed area, and obtaining the sum of the areas to be selected corresponding to each of the rectangular areas; calculating the ratio of the overlap of the areas to be selected corresponding to each of the rectangular areas to the sum of the areas to be selected, and obtaining the ratio of the areas to be selected corresponding to each of the rectangular areas; and selecting the largest area ratio among the area ratios to be selected as the area overlap ratio of the potential closed area.

[0103] Exemplarily, the formula for calculating the area overlap ratio is:

[0104] S 1 =calc_max_IOU(region1,bbxs)

[0105] Among them, S 1 is the area overlap ratio of the regions, IOU is Intersection of union, which refers to the area of ​​the overlapping parts of two regions divided by the area of ​​the regions, region1 is a potential closed area, bbxs is each rectangular area, which can be the area ratio to be selected in this embodiment; calc_max_IOU(region,bbxs) is to select the largest area ratio to be selected from each area ratio to be selected as the area overlap ratio of the regions.

[0106] In a feasible embodiment, the step of respectively determining the closed area of ​​each candidate closed region and the parallelism between each candidate straight line segment on the candidate closed region comprises:

[0107] Step X10, calculating the closed area of ​​the candidate closed area according to the potential coordinates of each potential vertex on the candidate closed area;

[0108] Step X20, taking the ratio of the closed region area to the image area of ​​the image data as the closed area ratio;

[0109] Step X30, determining a first group of adjacent straight line segments and a second group of adjacent straight line segments from each candidate straight line segment of the candidate closed region, determining a first candidate corner of the candidate closed region according to the first group of adjacent straight line segments, and determining a second candidate corner of the candidate closed region according to the second group of adjacent straight line segments;

[0110] Step X40, determining the straight line parallelism of the candidate closed area according to the first candidate angle and the second candidate angle.

[0111] It should be noted that the closed area area is the area of ​​the candidate closed area, and the image area is the image area corresponding to the image data. The image area can be calculated based on the length and width of the image data. The four candidate straight line segments corresponding to the candidate closed area can be the first candidate straight line segment, the second candidate straight line segment, the third candidate straight line segment and the fourth candidate straight line segment. The first group of adjacent straight line segments includes two adjacent candidate straight line segments in the candidate closed area, and the second group of adjacent straight line segments includes two adjacent candidate straight line segments in the candidate closed area. The first group of adjacent straight line segments does not include the candidate straight line segment corresponding to the second group of adjacent straight line segments. The first candidate angle and the second candidate angle are a group of diagonals.

[0112] Exemplarily, step X10 to step X40 include: calculating the closed area area of ​​the candidate closed area according to the potential coordinates of each potential vertex on the candidate closed area; calculating the image area corresponding to the image data according to the image length and image width corresponding to the image data, and taking the ratio of the closed area area to the image area of ​​the image data as the closed area ratio; determining a first group of adjacent straight line segments and a second group of adjacent straight line segments from each candidate straight line segment of the candidate closed area, determining the first candidate angle of the candidate closed area according to the first group of adjacent straight line segments, and determining the second candidate angle of the candidate closed area according to the second group of adjacent straight line segments; comparing the size of the first candidate angle and the second candidate angle to determine the ratio between the first candidate angle and the second candidate angle, and obtaining the straight line parallelism, which is less than or equal to 1. The embodiment of the present application can screen out the area in the image data that is most like an electronic screen by determining the closed area area ratio and the straight line parallelism of the candidate closed area. And determining the straight line parallelism by the diagonal relationship can simplify the calculation amount, and there is no need to calculate the angle between each pair of candidate straight line segments, thereby improving the calculation efficiency of the straight line parallelism.

[0113] Exemplarily, the formula for calculating the candidate area ratio is:

[0114] Calc_region = region / (h*w)

[0115] Among them, Calc_region is the candidate area ratio, region is the closed area of ​​the candidate closed area, (h*w) is the image area, h is the length of the image data, and w is the width of the image data.

[0116] Exemplarily, the formula for calculating the parallelism of a line is:

[0117] Calc_degree=min(degree(i,j),degree(k,l)) / max(degree(i,j),degree(k,l))

[0118] Among them, Calc_degree is the parallelism of the line, degree(i,j) is the first candidate angle, degree(k,l) is the second candidate angle, i and j represent two adjacent straight line segments in the first set of adjacent straight line segments, and k and l represent two adjacent straight line segments in the second set of adjacent straight line segments.

[0119] In a feasible embodiment, after the step of selecting the candidate closed area with the largest candidate value as the target screen area from among the candidate closed areas, the method further includes:

[0120] Step Y10, performing geometric correction on the target screen area to obtain a standard screen area, and calculating a current pixel value corresponding to the standard screen area;

[0121] Step Y20, obtaining the historical adjacent pixel values ​​of the historical adjacent image data of the previous time step of the image data corresponding to the standard screen area;

[0122] Step Y30, calculating the difference between the current pixel value and the historical adjacent pixel value to obtain a current pixel difference value, and obtaining a preset number of historical pixel difference values ​​that are sequentially adjacent to the current pixel difference value;

[0123] Step Y40, if the current pixel difference value and each of the historical pixel difference values ​​satisfy a preset period value change condition, then determining the flicker frequency abnormality information of the standard screen area;

[0124] It should be noted that the target screen area may be an irregular quadrilateral, so it is necessary to perform geometric correction on the target screen area. For example, the target screen area is restored to a regular quadrilateral. The current pixel value is the pixel value of the standard screen area. The standard screen area can be determined from the image data, and then the pixel value of the standard screen area is calculated. The historical adjacent pixel value is the historical adjacent image data of the previous time step of the image data corresponding to the current pixel value. The current pixel difference value is the difference between the current pixel value and the historical ringing pixel value.

[0125] The preset number is preset in advance, and the preset number is the number of historical pixel differences. The historical pixel difference is the pixel difference between the standard screen areas in each adjacent historical image data, and each historical pixel difference is sequentially adjacent to the current pixel difference. For example, the current image data corresponding to the current pixel value is i, the current pixel value corresponding to the current image data is represented as feature(i), the historical adjacent image data is i-1, the historical adjacent pixel value is feature(i-1), and the current pixel difference is feature(i)-feature(i-1). When the preset number is 2, the 2 historical pixel differences sequentially adjacent to the current pixel difference are: feature(i-1)-feature(i-2), feature(i-2)-feature(i-3), feature(i-2) is the pixel value of the standard screen area of ​​the image data of the previous time step adjacent to the historical adjacent pixel value, and feature(i-3) is the pixel value of the standard screen area of ​​the image data of the previous time step adjacent to the image data corresponding to feature(i-2).

[0126] The preset periodic change condition includes that the current pixel difference value and each historical pixel difference value conform to the periodic change relationship. For example, when the current pixel difference value is A and each historical pixel difference value adjacent to the current pixel difference value is B, A, B and A, it means that the current pixel difference value and each historical pixel difference value conform to the preset periodic change condition. The specific number of cycles is not limited in the embodiment of the present application. The flicker abnormality information is used to prompt the user that the electronic screen has a flicker abnormality. The flicker abnormality information can be expressed as text of the flicker abnormality.

[0127] Exemplarily, step Y10 to step Y40 include: performing geometric correction on the target screen area to obtain a standard screen area, and calculating a current pixel value corresponding to the standard screen area; obtaining historical adjacent pixel values ​​of the historical standard screen area in the historical adjacent image data of the image data corresponding to the standard screen area in the previous time step; calculating the difference between the current pixel value and the historical adjacent pixel value to obtain a current pixel difference, and obtaining a preset number of historical pixel differences that are sequentially adjacent to the current pixel difference; judging whether the current pixel difference and each of the historical pixel differences satisfy a preset periodic value change condition; if the current pixel difference and each of the historical pixel differences satisfy a preset periodic value change condition, determining the flicker abnormality information of the standard screen area; if there is no preset periodic value change condition between the current pixel difference and each of the historical pixel differences, there is no flicker in the standard screen area.

[0128] Step Y50, if the current pixel value matches the preset black screen state value, then determining the black screen abnormality information of the standard screen area;

[0129] Step Y60: If the current pixel value matches the preset blue screen status value, the blue screen abnormality information of the standard screen area is determined.

[0130] It should be noted that the preset black screen status value is the pixel value corresponding to the electronic screen when it is black, the preset blue screen status value is the pixel value corresponding to the electronic screen when it is blue, the black screen abnormality information is used to prompt the user that the electronic screen has a black screen abnormality, and the blue screen abnormality information is used to prompt the user that the electronic screen has a blue screen abnormality. Both the black screen abnormality information and the blue screen abnormality information can be expressed in text form, and the blue screen abnormality information, the black screen abnormality information and the flashing screen abnormality information can all be sent to the SOC interface on the screen recording system.

[0131] Exemplarily, it is determined whether the current pixel value matches the preset black screen status value. If the current pixel value matches the preset black screen status value, the black screen abnormality information of the standard screen area is determined; if the current pixel value does not match the preset black screen status value, it is determined that the black screen status does not exist in the standard screen area; it is determined whether the current pixel value matches the preset blue screen status value. If the current pixel value matches the preset blue screen status value, the blue screen abnormality information of the standard screen area is determined; if the current pixel value does not match the preset blue screen status value, it is determined that the blue screen status does not exist in the standard screen area.

[0132] The embodiment of the present application not only identifies the electronic screen in the image data but also monitors the status of the electronic screen, so that the abnormal status of the electronic screen can be monitored in real time to ensure the accuracy of screen recording.

[0133] Embodiment 3

[0134] Reference Figure 5 , the embodiment of the present application further provides a screen positioning device, the screen positioning device comprising:

[0135] An acquisition module 10 is used to acquire image data captured by a camera, call a straight line detection model to detect straight line segments in the image data, and call a target area detection model to detect rectangular areas in the image data;

[0136] A candidate closed region determining module 20, configured to determine at least one candidate closed region from the image data according to each of the rectangular regions, each of the straight line segments and potential intersections between the straight line segments, and to determine a closed area ratio of each of the candidate closed regions and a straight line parallelism of the candidate closed regions respectively;

[0137] A candidate value determination module 30, configured to determine the candidate value of each candidate closed region based on the closed area and the parallelism corresponding to each candidate closed region;

[0138] The screen determination module 40 is used to select a candidate closed area with the largest candidate value from among the candidate closed areas as a target screen area.

[0139] Optionally, the candidate closed area determining module 20 further includes:

[0140] Determining at least one potential closed region from the image data based on the straight line segments and the potential intersection points between the straight line segments;

[0141] According to each of the rectangular areas, respectively determine the area overlap ratio of each of the potential closed areas;

[0142] At least one candidate closed region whose area overlap ratio is greater than or equal to a preset candidate threshold is screened out from each of the potential closed regions.

[0143] Optionally, the candidate closed area determining module 20 further includes:

[0144] Determining potential intersection points between the straight line segments according to the extended trend lines of the straight line segments;

[0145] Screening out a plurality of potential vertices satisfying a preset closing condition from each of the potential intersection points;

[0146] At least one potential closed area is determined according to the vertex identifier of each potential vertex, wherein the vertex identifier is determined based on the straight line identifiers respectively corresponding to two straight line segments that potentially intersect the potential vertex.

[0147] Optionally, the candidate closed area determining module 20 further includes:

[0148] For any of the potential intersection points, determining a first potential straight line segment and a second potential straight line segment that potentially intersect the potential intersection point;

[0149] According to a first distance from a first straight line midpoint of the first potential straight line segment to the potential intersection point, a ratio of the first distance to a first line segment length of the first potential straight line segment is calculated to obtain a first distance overlap ratio;

[0150] According to a second distance from a second straight line midpoint of the second potential straight line segment to the potential intersection point, a ratio of the second distance to a second line segment length of the second potential straight line segment is calculated to obtain a second distance coincidence ratio;

[0151] Calculating a potential angle between the first potential straight line segment and the second potential straight line segment;

[0152] If the first distance overlap ratio and the second distance overlap ratio both satisfy the preset closing distance condition, and the potential angle satisfies the preset closing angle condition, the potential intersection point is determined to be the potential vertex.

[0153] Optionally, the candidate closed area determining module 20 further includes:

[0154] For any target potential vertex among the potential vertices, according to the target vertex identifier of the target potential vertex, determining a first target closed vertex and a second target closed vertex matching the target vertex identifier among the potential vertices;

[0155] Determine, among the potential vertices, a third target closed vertex that matches both the second target closed vertex and the second target closed vertex;

[0156] The potential closed area is determined according to the target potential vertex, the first target closed vertex, the second target closed vertex, and the third target closed vertex.

[0157] Optionally, the candidate closed area determining module 20 further includes:

[0158] For any of the potential closed areas, calculating the area overlap between the potential closed area and each of the rectangular areas in the image data, and obtaining the area overlap of each of the rectangular areas to be selected;

[0159] Determine the sum of the areas between each of the rectangular areas and the potential closed area respectively, and obtain the sum of the areas to be selected corresponding to each of the rectangular areas;

[0160] Respectively calculating the ratio of the overlap degree of the to-be-selected area corresponding to each of the rectangular areas to the sum of the to-be-selected areas, to obtain the to-be-selected area ratio corresponding to each of the rectangular areas;

[0161] The largest area ratio among the area ratios to be selected is selected as the area overlap ratio of the potential closed area.

[0162] Optionally, the candidate closed area determining module 20 further includes:

[0163] Calculating the closed area of ​​the candidate closed area according to the potential coordinates of each potential vertex on the candidate closed area;

[0164] Taking the ratio of the closed region area to the image area of ​​the image data as the closed area ratio;

[0165] Determine a first group of adjacent straight line segments and a second group of adjacent straight line segments from each candidate straight line segment of the candidate closed region, determine a first candidate corner of the candidate closed region according to the first group of adjacent straight line segments, and determine a second candidate corner of the candidate closed region according to the second group of adjacent straight line segments;

[0166] The straight line parallelism of the candidate closed area is determined according to the first candidate angle and the second candidate angle.

[0167] Optionally, the screen determination module 40 further includes:

[0168] Performing geometric correction on the target screen area to obtain a standard screen area, and calculating a current pixel value corresponding to the standard screen area;

[0169] Acquire historical adjacent pixel values ​​of historical adjacent image data of the previous time step of the image data corresponding to the standard screen area;

[0170] Calculating the difference between the current pixel value and the historical adjacent pixel value to obtain a current pixel difference value, and obtaining a preset number of historical pixel difference values ​​that are sequentially adjacent to the current pixel difference value;

[0171] If the current pixel difference value and each of the historical pixel difference values ​​satisfy a preset period value change condition, determining the flicker frequency abnormality information of the standard screen area;

[0172] If the current pixel value matches the preset black screen state value, determining the black screen abnormality information of the standard screen area;

[0173] If the current pixel value matches the preset blue screen status value, the blue screen abnormality information of the standard screen area is determined.

[0174] The screen positioning device provided by the present application adopts the screen positioning method in the above embodiment, aiming to solve the technical problem that it is difficult to accurately locate the area of ​​the electronic screen when the shooting background is complex. Compared with the prior art, the beneficial effects of the screen positioning method provided by the embodiment of the present application are the same as the beneficial effects of the screen positioning method provided by the above embodiment, and the other technical features in the screen positioning device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0175] Embodiment 4

[0176] An embodiment of the present application provides an electronic device, which may be a playback device, and the electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the screen positioning method in the above embodiment.

[0177] Reference below Figure 6 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (portable Android devices), PMPs (portable media players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0178] like Figure 6As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a ROM (Read-Only Memory) or a program loaded from a storage device into a RAM (Random Access Memory). In the RAM, various programs and data required for the operation of the electronic device are also stored. The processing device, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0179] Typically, the following systems can be connected to the I / O interface: input devices such as touch screens, touch pads, keyboards, mice, image sensors, microphones, tachometers, gyroscopes, etc.; output devices such as LCDs (Liquid Crystal Displays), speakers, vibrators, etc.; storage devices such as magnetic tapes, hard disks, etc.; and communication devices. The communication device can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an electronic device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0180] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication system, or installed from a storage system, or installed from a ROM. When the computer program is executed by a processing system, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0181] The electronic device provided by the present application adopts the screen positioning method in the above-mentioned embodiment 1 to solve the technical problem that it is difficult to accurately locate the area of ​​the electronic screen when the shooting background is complex. Compared with the prior art, the beneficial effect of the product flow data distribution provided by the embodiment of the present application is the same as the beneficial effect of the screen positioning method provided by the above-mentioned embodiment, and the other technical features in the screen positioning device are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0182] It should be understood that the various parts of the present disclosure can be implemented with hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0183] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0184] Embodiment 5

[0185] This embodiment provides a computer-readable storage medium having computer-readable program instructions stored thereon, and the computer-readable program instructions are used to execute the screen positioning method in the above-mentioned embodiment 1.

[0186] The computer-readable storage medium provided in the embodiment of the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor devices, equipment or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable EPROM (Electrical Programmable Read Only Memory, read-only memory) or flash memory, an optical fiber, a portable compact disk CD-ROM (compact disc read-only memory, read-only memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution device, device or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency, radio frequency), etc., or any suitable combination of the above.

[0187] The computer-readable storage medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0188] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device: obtains image data captured by a camera, calls a straight line detection model to detect each straight line segment in the image data, and calls a target area detection model to detect each rectangular area in the image data; determines at least one candidate closed area from the image data according to each rectangular area, each straight line segment, and potential intersections between each straight line segment, and determines the closed area of ​​each candidate closed area and the parallelism between each candidate straight line segment on the candidate closed area respectively; determines the candidate value of each candidate closed area based on the closed area and parallelism corresponding to each candidate closed area; and selects the candidate closed area with the largest candidate value as the target screen area among the candidate closed areas.

[0189] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a LAN (local area network) or WAN (Wide Area Network), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0190] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based device that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0191] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of the module does not limit the unit itself in some cases.

[0192] The computer-readable storage medium provided by the present application stores computer-readable program instructions for executing the above-mentioned screen positioning method, aiming to solve the technical problem that it is difficult to accurately locate the area of ​​the electronic screen when the shooting background is complex. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the embodiment of the present application are the same as the beneficial effects of the screen positioning method provided in the above-mentioned embodiment, and will not be repeated here.

[0193] Embodiment 6

[0194] The present application also provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the steps of the screen positioning method as described above are implemented.

[0195] The computer program product provided by the present application is intended to solve the technical problem that it is difficult to accurately locate the area of ​​the electronic screen when the shooting background is complex. Compared with the prior art, the beneficial effects of the computer program product provided by the embodiment of the present application are the same as the beneficial effects of the screen positioning method provided by the above embodiment, and will not be repeated here.

[0196] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. A screen positioning method, characterized in that: The screen positioning method comprises: Acquire image data captured by a camera, call a straight line detection model to detect straight line segments in the image data, and call a target area detection model to detect rectangular areas in the image data; Determine at least one candidate closed region from the image data according to each of the rectangular regions, each of the straight line segments, and the potential intersection points between the straight line segments, and respectively determine a closed area ratio of each of the candidate closed regions and a straight line parallelism of the candidate closed regions; Determine the candidate value of each candidate closed region based on the closed area ratio and the straight line parallelism corresponding to each candidate closed region; Selecting a candidate closed area with the largest candidate value from among the candidate closed areas as the target screen area; The step of determining at least one candidate closed region from the image data according to each of the rectangular regions, each of the straight line segments and the potential intersections between the straight line segments comprises: Determining at least one potential closed region from the image data based on the straight line segments and the potential intersection points between the straight line segments; According to each of the rectangular areas, respectively determine the area overlap ratio of each of the potential closed areas; Screen out at least one candidate closed area whose area overlap ratio is greater than or equal to a preset candidate threshold value from among the potential closed areas; the area overlap ratio is characterized by the ratio of the overlap between the area of ​​the potential closed area and the area of ​​the rectangular area corresponding to the potential closed area to the sum of the area of ​​the potential closed area and the area of ​​the rectangular area corresponding to the potential closed area; According to the extended trend lines of each of the straight line segments, the potential intersection points between the straight line segments are determined; for any of the potential intersection points, a first potential straight line segment and a second potential straight line segment that potentially intersect the potential intersection point are determined; according to a first distance from a midpoint of a first straight line of the first potential straight line segment to the potential intersection point, a ratio of the first distance to a first line segment length of the first potential straight line segment is calculated to obtain a first distance overlap ratio; according to a second distance from a midpoint of a second straight line of the second potential straight line segment to the potential intersection point, a ratio of the second distance to a second line segment length of the second potential straight line segment is calculated to obtain a second distance overlap ratio; a potential angle between the first potential straight line segment and the second potential straight line segment is calculated; if both the first distance overlap ratio and the second distance overlap ratio satisfy a preset closed distance condition, and the potential angle satisfies a preset closed angle condition, the potential intersection point is determined to be a potential vertex; Calculate the closed area of ​​the candidate closed area according to the potential coordinates of each potential vertex on the candidate closed area; calculate the image area corresponding to the image data according to the image length and image width corresponding to the image data, and use the ratio of the closed area area to the image area of ​​the image data as the closed area ratio; Determine a first group of adjacent straight line segments and a second group of adjacent straight line segments from each candidate straight line segment of the candidate closed region, determine a first candidate corner of the candidate closed region according to the first group of adjacent straight line segments, and determine a second candidate corner of the candidate closed region according to the second group of adjacent straight line segments; The first candidate angle and the second candidate angle are compared to determine a ratio between the first candidate angle and the second candidate angle, and the straight line parallelism is obtained, and the straight line parallelism is less than or equal to 1.

2. The screen positioning method according to claim 1, characterized in that: The step of determining at least one potential closed area from the image data according to each of the straight line segments and the potential intersection points between the straight line segments comprises: Determining potential intersection points between the straight line segments according to the extended trend lines of the straight line segments; Screening out a plurality of potential vertices satisfying a preset closing condition from each of the potential intersection points; At least one potential closed area is determined based on the vertex identifier of each potential vertex, wherein the vertex identifier is determined based on the straight line identifiers corresponding to two straight line segments that potentially intersect at the potential vertex; the preset closing condition includes a preset closing distance condition and a preset closing angle condition.

3. The screen positioning method according to claim 1, characterized in that: The step of determining at least one potential closed area according to the vertex identifiers of the potential vertices comprises: For any target potential vertex among the potential vertices, according to the target vertex identifier of the target potential vertex, determining a first target closed vertex and a second target closed vertex matching the target vertex identifier among the potential vertices; Determine, among the potential vertices, a third target closed vertex that matches both the second target closed vertex and the second target closed vertex; The potential closed area is determined according to the target potential vertex, the first target closed vertex, the second target closed vertex, and the third target closed vertex.

4. The screen positioning method according to claim 1, characterized in that: The step of determining the area overlap ratio of each of the potential closed areas according to each of the rectangular areas comprises: For any of the potential closed areas, calculating the area overlap between the potential closed area and each of the rectangular areas in the image data, and obtaining the area overlap of each of the rectangular areas to be selected; Determine the sum of the areas between each of the rectangular areas and the potential closed area respectively, and obtain the sum of the areas to be selected corresponding to each of the rectangular areas; Respectively calculating the ratio of the overlap degree of the to-be-selected area corresponding to each of the rectangular areas to the sum of the to-be-selected areas, to obtain the to-be-selected area ratio corresponding to each of the rectangular areas; The largest area ratio among the area ratios to be selected is selected as the area overlap ratio of the potential closed area.

5. The screen positioning method according to claim 1, characterized in that: After the step of selecting the candidate closed area with the largest candidate value from among the candidate closed areas as the target screen area, the method further includes: Performing geometric correction on the target screen area to obtain a standard screen area, and calculating a current pixel value corresponding to the standard screen area; Acquire historical adjacent pixel values ​​of historical adjacent image data of the previous time step of the image data corresponding to the standard screen area; Calculating the difference between the current pixel value and the historical adjacent pixel value to obtain a current pixel difference value, and obtaining a preset number of historical pixel difference values ​​that are sequentially adjacent to the current pixel difference value; If the current pixel difference value and each of the historical pixel difference values ​​satisfy a preset period value change condition, determining the flicker frequency abnormality information of the standard screen area; If the current pixel value matches the preset black screen state value, determining the black screen abnormality information of the standard screen area; If the current pixel value matches the preset blue screen status value, the blue screen abnormality information of the standard screen area is determined.

6. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the screen positioning method described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program for implementing the screen positioning method, and the program for implementing the screen positioning method is executed by a processor to implement the steps of the screen positioning method according to any one of claims 1 to 5.

Images