A window opening detection method based on image processing

Abstract

This application relates to the field of heating, ventilation, and air conditioning (HVAC) and discloses a method for detecting window openings based on image processing, including: fitting a motion trajectory curve of a window sash marker at different window openings; performing corner point detection on the window image and retaining corner points within a set range of the motion trajectory curve; performing line detection on the window image and searching for corner points within the retained corner points that are simultaneously within a set distance from both ends of the line as target marker points; and determining the window opening based on the position of the target marker point. The above-mentioned window opening detection method combines corner point detection with line detection in the image processing method and the motion trajectory curve to filter out corner points and lines detected on the background, thereby accurately detecting the specific window opening and is suitable for more complex scenarios.

Description

Technical Field

[0001] The present invention relates to the field of heating, ventilation and air conditioning, and in particular to a window opening detection method based on image processing. Background Art

[0002] Relevant research has shown that the number of air changes varies significantly depending on the window opening area. For example, the number of air changes is almost nine times higher when the window area is 25% open than when it is 100% open. Air changes significantly impact indoor air quality, occupant thermal comfort, and building energy consumption. Accurately capturing window opening behavior data can help further improve the accuracy of building energy consumption simulations and optimize the operational control performance of building ventilation and air conditioning systems, thereby achieving building energy conservation and emission reduction.

[0003] Currently, in the HVAC field, questionnaires and door and window sensors are primarily used to obtain data on both window opening and closing states, enabling a quantitative description of behavioral characteristics such as window opening frequency, duration, and probability of opening. Questionnaire surveys typically involve researchers verbally or handing out questionnaires to each participant, recording information on their window opening behavior. This method is labor-intensive and time-consuming, and the accuracy of window states captured by the questionnaires cannot be guaranteed. Door and window sensors, on the other hand, are installed on windows, and the recorded window opening and closing data is uploaded to a terminal. This method captures both window opening and closing states. However, considering that simply capturing window opening and closing states alone is insufficient for in-depth analysis of window opening behavior and its impact on building performance, some researchers have recently explored using deep learning methods to detect different window opening ranges (i.e., opening angles), such as closed, partially open, and open. Specifically, they create a relevant window opening and closing dataset to train a deep learning-based object detection algorithm to detect window openings across a range of different degrees. However, deep learning-based recognition of window openings still has certain limitations. Specifically, deep learning not only requires time and manpower to produce a large amount of data, but also the image features of windows with similar opening angles are similar, while the feature differences learned by deep networks are small, resulting in poor detection results. Therefore, what is obtained is often a range of window openings rather than specific window openings, making it difficult to fully reflect the impact of different window openings on indoor cooling and heating loads and air quality in subsequent applications and research. Considering that the window opening behaviors of different users in real life often vary significantly, compared to simplifying the window openings into a range of openings with a large variation interval, detecting the specific window openings can better help analyze the differences in window opening behaviors of different groups of people, and the window opening behavior data obtained is more accurate and more applicable.

[0004] Therefore, how to design a method for detecting the specific opening of a window is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a window opening detection method based on image processing, which can accurately detect the specific window opening and is applicable to more complex scenarios. The specific scheme is as follows:

[0006] A window opening detection method based on image processing, comprising:

[0007] Fit the motion trajectory curve of the window sash marker point at different window openings;

[0008] Performing corner detection on the window image and retaining the corner points within the set range of the motion trajectory curve;

[0009] Perform straight line detection on the window image and find corner points that are within a set distance at both ends of the straight line among the retained corner points as target markers;

[0010] The opening degree of the window is determined according to the position of the target marking point.

[0011] Preferably, in the above-mentioned window opening detection method based on image processing provided by an embodiment of the present invention, fitting the motion trajectory curve of the window sash mark point at different window openings includes:

[0012] Get the coordinate values ​​of the window sash mark points at different window openings;

[0013] According to the coordinate values ​​of the window sash marking points, a polynomial is used to fit the motion trajectory curve of the window sash marking points at different window openings.

[0014] Preferably, in the above-mentioned window opening detection method based on image processing provided by an embodiment of the present invention, performing corner point detection on the window image includes:

[0015] Use the corner detection algorithm to detect corners of the window image and obtain all corner points on the window image.

[0016] Preferably, in the above-mentioned window opening detection method based on image processing provided by an embodiment of the present invention, retaining corner points within a set range of the motion trajectory curve includes:

[0017] All corner points on the window image are roughly screened by the set interval obtained by moving the motion trajectory curve upward and downward, and the corner points within the set interval of the motion trajectory curve are retained, while the corner points outside the set interval of the motion trajectory curve are eliminated.

[0018] Preferably, in the above-mentioned window opening detection method based on image processing provided by an embodiment of the present invention, performing straight line detection on the window image includes:

[0019] The line detection algorithm is used to detect lines in the window image and obtain all the lines on the window image.

[0020] Preferably, in the above-mentioned window opening detection method based on image processing provided by an embodiment of the present invention, searching for corner points located within a set distance from both ends of a straight line as target marking points among the retained corner points includes:

[0021] Pair all the straight lines on the window image with the retained corner points, and sequentially search for two corner points that are simultaneously within a set distance from each straight line endpoint, and use the found corner points as target marking points.

[0022] Preferably, in the above-mentioned window opening detection method based on image processing provided by an embodiment of the present invention, the relationship between the coordinate value of the target marking point and the coordinate value of the corresponding straight line endpoint satisfies the following formula:

[0023]

[0024] Wherein, (X1, Y1) is the coordinate value of the endpoint of the straight line, (X2, Y2) is the coordinate value of the target mark point, and ρ is the set distance.

[0025] Preferably, in the above-mentioned window opening detection method based on image processing provided by an embodiment of the present invention, determining the window opening according to the position of the target marking point includes:

[0026] Establish the relationship between the window opening and the coordinate value of the window sash mark point;

[0027] The position coordinates of the target marking point are input into the established relationship to calculate the window opening.

[0028] Preferably, in the above-mentioned window opening detection method based on image processing provided by an embodiment of the present invention, establishing a relationship between the window opening and the coordinate values ​​of the window sash marking points includes:

[0029] The horizontal and vertical coordinates of the window sash mark points were used as independent variables and the window opening was used as the dependent variable to fit a polynomial regression equation.

[0030] The relationship between the window opening and the coordinate values ​​of the window sash mark points is established by fitting the polynomial regression equation.

[0031] Preferably, in the above-mentioned window opening detection method based on image processing provided by an embodiment of the present invention, the formula of the polynomial regression equation is:

[0032]

[0033] Among them, x iis the horizontal coordinate of the window sash mark point, y i is the ordinate of the window sash mark point, z is the window opening, μ i and ν j are polynomial coefficients, i = 0,…,n, j = 0,…,n, n is the degree of the polynomial, and c is a constant term.

[0034] It can be seen from the above technical solution that the present invention provides a window opening detection method based on image processing, including: fitting the motion trajectory curve of the window sash mark point under different window openings; performing corner point detection on the window image, and retaining the corner points within the set range of the motion trajectory curve; performing straight line detection on the window image, and searching for corner points that are simultaneously located within a set distance at both ends of the straight line among the retained corner points as target marking points; and determining the window opening according to the position of the target marking point.

[0035] The above-mentioned window opening detection method provided by the present invention, through the combination of corner point detection and straight line detection in the image processing method and the motion trajectory curve, filters out the corner points and straight lines detected on the background, and can realize accurate detection of the specific window opening, which is suitable for more complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0037] Figure 1 A flowchart of a window opening detection method based on image processing provided by an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of the casement window structure and background provided by an embodiment of the present invention;

[0039] Figure 3 A schematic diagram of a coarse screening process provided by an embodiment of the present invention;

[0040] Figure 4 Schematic diagram of the fine screening process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In the HVAC field, existing methods for detecting window opening and closing behavior primarily include questionnaires, door and window sensors, and deep learning-based target detection. However, these methods have the following drawbacks: questionnaires are labor-intensive and time-consuming, and the accuracy of the window status captured by the questionnaire cannot be guaranteed; door and window sensors can interfere with the subjects' window opening and closing behavior; and deep learning-based target detection methods require significant labor and time to generate datasets. A common drawback of these three methods is their inability to accurately detect the specific window opening degree. The present invention, however, provides a window opening detection method based on image processing that can accurately detect the specific window opening degree.

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0043] The present invention provides a window opening detection method based on image processing, such as Figure 1 As shown, the following steps are included:

[0044] S101, fitting the motion trajectory curve of the window sash mark point at different window openings;

[0045] Specifically, in step S101 , the motion trajectory curve of the marker point during the windowing process may be fitted to prepare for the next rough screening step.

[0046] S102, performing corner point detection on the window image and retaining the corner points within the set range of the motion trajectory curve;

[0047] Specifically, step S102 is a step of performing corner point detection and coarse screening after obtaining the motion trajectory of the marker point, wherein the corner points within the set range of the motion trajectory curve are retained, while the corner points outside the set range of the motion trajectory curve are eliminated.

[0048] S103, performing straight line detection on the window image, and searching for corner points among the retained corner points that are simultaneously within a set distance from both ends of the straight line as target marking points;

[0049] Specifically, step S103 is to perform line detection and fine screening to further determine the marker points based on the corner points obtained in the motion trajectory area through coarse screening. The main purpose is to locate the target marker point using the obtained straight line (i.e., marker line).

[0050] The above screening steps can accurately detect the specific opening degree of the window even when the window is in a more complex scene.

[0051] S104: Determine the window opening according to the position of the target marking point.

[0052] It's understandable that windows have distinct features. They're generally rectangular in shape, with straight lines (such as the four sides of a sash) and corners (such as its apex). Corners are the intersections of contours in an image, and their grayscale values ​​exhibit significant gradient variations, reflecting local image characteristics. Straight lines are another important geometric feature in an image, represented by a continuous set of pixels with consistent orientation.

[0053] Therefore, based on the characteristics of the window, the present invention establishes a window opening detection method based on image processing. The specific ideas are as follows: Taking the casement window as an example, it generally consists of two parts: a fixed window frame and a movable window sash. Its structure is shown in the attached figure. Figure 2 As shown. Since the vertices of the movable window sash in the image (i.e., points a, b, c, and d) are the intersection points between the contours, and their right-angled sides (i.e., ab, bc, cd, and da) are a group of pixels with the same direction and continuity, they respectively conform to the concepts of corner points and straight lines in the above image features. Considering that the distances between points a and b and the window frame show obvious regular changes under different window openings, while points c and d are relatively fixed, points a and b are used as marking points for judging the window opening, and the corresponding line ab is the marking line (i.e., a straight line). On this basis, the present invention combines corner point detection and straight line detection in image processing technology to obtain the position of the target marking point, and further determines the window opening based on the positions of the two points. Taking into account that when performing corner point detection on an image, not only the marking points will be detected, but also many other corner points (hereinafter referred to as pseudo-marking points), it is necessary to formulate a screening step to filter out other pseudo points, so that the marking points a and b can be accurately obtained.

[0054] In the above-mentioned window opening detection method based on image processing provided in an embodiment of the present invention, by combining corner point detection with straight line detection and motion trajectory curve in the image processing method, the corner points and straight lines detected on the background are screened out, thereby realizing accurate detection of the specific window opening, which is suitable for more complex scenes.

[0055] In a specific implementation, in the above-mentioned image processing-based window opening detection method provided by an embodiment of the present invention, step S101 of fitting the motion trajectory curve of the window sash marker point at different window openings may specifically include: obtaining the coordinate values ​​of the window sash marker point at different window openings; and fitting the motion trajectory curve of the window sash marker point at different window openings using a polynomial based on the coordinate values ​​of the window sash marker point. In other words, the motion trajectory curve of the window sash marker point during the window opening process is fitted using the coordinate values ​​of the actual marker point.

[0056] Specifically, obtain the coordinate values ​​of the marked points under different window openings, such as Figure 2As shown in the figure, by taking the point e in the upper left corner of the image as the coordinate origin, taking the horizontal right direction of point e as the positive direction of the X axis and the vertical downward direction as the positive direction of the Y axis, the pixel coordinate values ​​of the marked points a and b under several different window openings {(x1, y1)(x2, y2)...(x i ,y i )}, and each coordinate value is brought into the polynomial formula to fit the motion trajectory curves y(a) and y(b) of the marker points a and b under different window openings. The corresponding formulas are as follows:

[0057]

[0058] Among them, a0a1a2...a n-1 are the polynomial coefficients, and n is the degree of the polynomial.

[0059] In a specific implementation, in the above-mentioned window opening detection method based on image processing provided in an embodiment of the present invention, step S102 performs corner point detection on the window image and retains the corner points within the set range of the motion trajectory curve. Specifically, it may include: first, using a corner point detection algorithm to perform corner point detection on the window image to obtain all corner points on the window image; then, all corner points on the window image are coarsely screened by using the set interval obtained by moving the motion trajectory curve up and down, retaining the corner points within the set interval of the motion trajectory curve, and eliminating the corner points outside the set interval of the motion trajectory curve.

[0060] Specifically, corner detection is performed on the window image to obtain all corner points on the image, including marked points and pseudo-marked points. Most of the pseudo-marked points on the image are far away from the marked point motion trajectory curve. Therefore, the marked point motion trajectory curve can be used to eliminate these pseudo-marked points. The specific method is to roughly screen the corner points on the image through the interval obtained by moving the marked point motion trajectory curve up and down, retain the points within the marked point motion trajectory curve interval, and eliminate the points outside the marked point motion trajectory curve interval, that is, Figure 3 As shown, two groups of points C can be obtained, which are located in the motion trajectory curve interval of the marked points a and b respectively. a (p) and C b (q), p and q are the number of corner points after screening. Preferably, in order to ensure that the actual marking point is within the interval, the present invention can set the translation distance to 10 pixels.

[0061] In specific implementation, in the above-mentioned window opening detection method based on image processing provided in an embodiment of the present invention, step S103 performs straight line detection on the window image, and searches for corner points that are simultaneously located within a set distance from both ends of the straight line among the retained corner points as target marking points. Specifically, it may include: first, using a straight line detection algorithm to perform straight line detection on the window image to obtain all straight lines on the window image; then, pairing all straight lines on the window image with the retained corner points, and searching for two corner points that are simultaneously located within a set distance from the end points of each straight line among the retained corner points, and using the found corner points as target marking points.

[0062] Specifically, since the above-mentioned known marking points a and b are the endpoints of the marking line ab, all the straight lines obtained by straight line detection on the image can be paired with the two groups of corner points after rough screening according to this correspondence, as shown in the following example: Figure 4 As shown, in the two sets of corner points C a (p) and C b Find two points C in (q) that are both endpoints of a line a (m) and C b (n), thereby locating the marker points a and b. This step is the fine screening step. It should be noted that due to the different algorithm principles and detection accuracy of corner point detection and line detection, the coordinate values ​​of the endpoints of the marker line ab obtained by line detection and the positions of the marker points a and b obtained by corner point detection are only approximately coincident, and there is often a certain difference between the coordinate values ​​of the two. Therefore, the pairing of the line on the image and the two groups of corner points after coarse screening needs to be calculated by calculating the Euclidean distance between the endpoints of the line and the two groups of corner points after coarse screening, so as to find the corner points that are simultaneously near the two ends of the line to determine the marker points a and b.

[0063] In a specific implementation, the relationship between the coordinate value of the target mark point and the coordinate value of the corresponding straight line endpoint satisfies the following formula (2):

[0064]

[0065] Among them, (X1, Y1) is the coordinate value of the endpoint of the straight line, (X2, Y2) is the coordinate value of the target marking point, and ρ is the set distance. Preferably, the value of ρ can be 5.

[0066] In specific implementation, in the above-mentioned window opening detection method based on image processing provided in an embodiment of the present invention, step S104 determines the window opening according to the position of the target marking point, which may specifically include: establishing a relationship between the window opening and the coordinate value of the window sash marking point; inputting the position coordinates of the target marking point into the established relationship to calculate the window opening. It should be noted that after the coordinate values ​​of the marking points a and b obtained based on the above steps, one of them can be selected to further determine the window opening. It is known that each window opening marking point has only a unique corresponding coordinate value in the coordinate system, so the relationship between the window opening and the coordinate value can be obtained by establishing a polynomial equation.

[0067] Specifically, during implementation, the relationship between the window opening and the coordinate values ​​of the window sash marking points is established in the above steps, which may include:

[0068] First, mark the horizontal coordinates of the window sash point a or b {x1, x2, ..., x i} and the vertical coordinates {y1,y2,...,y i} as independent variables, window opening {z1,z2,...,z i} as the dependent variable, and fit the polynomial regression equation; the formula of the polynomial regression equation can be:

[0069]

[0070] Among them, x i is the horizontal coordinate of the window sash mark point, y i is the ordinate of the window sash mark point, z is the window opening, μ i and ν j are polynomial coefficients, i = 0,…,n, j = 0,…,n, n is the degree of the polynomial, and c is a constant term.

[0071] Then, the relationship between the window opening and the coordinate values ​​of the window sash mark points is established through the fitted polynomial regression equation.

[0072] Finally, the window opening can be calculated by substituting the coordinate values ​​of the marking points a and b obtained in the previous steps into the above polynomial equation.

[0073] It should be pointed out that the present invention combines motion trajectory curves and corner detection and line detection in image processing methods to propose a dual-track corner point and line detection method that can obtain the specific window opening, and sets some screening steps so that the specific opening of the window can be accurately detected even in more complex scenes. The specific advantages are as follows: First, for the study of other window opening behaviors in the HVAC field, the present invention realizes the detection of the specific opening angle of the window; second, the present invention is targeted at the HVAC field, and for the HVAC field, the specific opening angle of the window is of practical significance, but it is currently difficult to achieve accurate detection of the window opening; third, the present invention can be applied to more complex scenes. Through the combination of line detection, corner detection and motion trajectory curves, multiple steps filter out the lines and corners detected on the background to achieve accurate detection of the specific window opening.

[0074] In practical applications, this invention primarily targets casement window opening detection, but can also be applied to other window types, such as sliding and top-hung windows. Furthermore, this invention primarily targets placing a camera inside a room to detect the opening of a single window, but can also be applied to an outdoor location to detect the opening of an entire wall.

[0075] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0076] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0077] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0078] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0079] The above is a detailed introduction to the window opening detection method based on image processing provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A window opening detection method based on image processing, characterized in that: include: Fit the motion trajectory curve of the window sash marker point at different window openings; Performing corner detection on the window image using a corner detection algorithm to obtain all corner points on the window image, and performing a coarse screening of all corner points on the window image using a set interval obtained by moving the motion trajectory curve upward and downward, retaining corner points within the set interval of the motion trajectory curve and removing corner points outside the set interval of the motion trajectory curve; Use the line detection algorithm to detect lines on the window image, obtain all the lines on the window image, and pair all the lines on the window image with the retained corner points. Then, find two corner points that are both within a set distance from each line endpoint among the retained corner points, and use the found corner points as target marking points. The relationship between the coordinate values ​​of the target marking points and the coordinate values ​​of the corresponding line endpoints satisfies the following formula: ； in, are the coordinates of the endpoints of the line, is the coordinate value of the target marking point, Setting a distance for said Establish the relationship between the window opening and the coordinate value of the window sash mark point; The position coordinates of the target marking point are input into the established relationship to calculate the window opening.

2. The window opening detection method based on image processing according to claim 1, characterized in that: Fitting the motion trajectory curve of the window sash marker at different window openings, including: Get the coordinate values ​​of the window sash mark points at different window openings; According to the coordinate values ​​of the window sash marking points, a polynomial is used to fit the motion trajectory curve of the window sash marking points at different window openings.

3. The window opening detection method based on image processing according to claim 2, characterized in that: Establish the relationship between the window opening and the coordinate value of the window sash mark point, including: The horizontal and vertical coordinates of the window sash mark points were used as independent variables and the window opening was used as the dependent variable to fit a polynomial regression equation. The relationship between the window opening and the coordinate values ​​of the window sash mark points is established by fitting the polynomial regression equation.

4. The window opening detection method based on image processing according to claim 3, characterized in that: The formula of the polynomial regression equation is: ； in, is the horizontal coordinate of the window sash mark point, is the vertical coordinate of the window sash mark point, For window opening, and are polynomial coefficients, i=0,…,n, j=0,…,n, n is the degree of the polynomial, and c is a constant term.

Images