An open and close state detection method, electronic equipment and computer readable storage medium
By acquiring and comparing the salient regions of the target image and the reference image, the opening and closing state of the target object is automatically identified, solving the problem of low efficiency in manual monitoring and achieving efficient opening and closing state detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG DAHUA TECH CO LTD
- Filing Date
- 2022-11-21
- Publication Date
- 2026-06-26
AI Technical Summary
In existing technologies, the control and monitoring of the opening and closing devices of target objects mainly rely on manual labor, which cannot be carried out in a timely and effective manner, resulting in untimely monitoring and low efficiency.
By acquiring salient regions in the target image and reference image, and using a salient recognition model and a target entity detection model, the opening and closing state of the target object is automatically identified. By comparing the differences in salient regions in the target image and reference image, the state of the opening and closing device is determined.
It enables automatic identification of the opening and closing state of the target object, reduces manual intervention, improves the timeliness and efficiency of monitoring, has a wide range of applications, and does not require the addition of obvious marking materials on the target object.
Smart Images
Figure CN115797757B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to an opening / closing state detection method, electronic device, and computer-readable storage medium. Background Technology
[0002] Currently, in some sectors (such as power energy, petrochemicals, rail transportation, and production workshops), it is necessary to control and monitor the opening and closing devices of certain objects (such as distribution boxes and containers). However, this control and monitoring is currently mainly carried out manually, which cannot be done in a timely and effective manner. Summary of the Invention
[0003] The main technical problem addressed by this application is to provide an opening / closing state detection method, electronic device, and computer-readable storage medium that can automatically identify the opening / closing state of an object's opening / closing device.
[0004] To solve the above-mentioned technical problems, one technical solution adopted in this application is: providing an opening / closing state detection method, the method comprising: acquiring at least one reference salience region on a target object in a reference image; wherein the target object is provided with an opening / closing device, and the reference image is acquired from the target object with the opening / closing device in a first opening / closing state; performing salience identification on the target object in the target image to obtain at least one target salience region; wherein the target image and the reference image are acquired from the target object at the same angle and the same position, and the entities corresponding to the reference salience region and the target salience region belong to the target object; determining the opening / closing state of the opening / closing device of the target object based on the difference between at least one target salience region and at least one reference salience region.
[0005] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide an electronic device, which includes a processor and a memory. The memory stores program instructions, and the processor executes the program instructions to implement the above-mentioned method for detecting the opening and closing state.
[0006] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a computer-readable storage medium for storing program instructions that can be executed to implement the above-mentioned method for detecting the opening and closing state.
[0007] The above technical solution, by comparing the differences between at least one target salient region of the target object in the target image and at least one reference salient region of the target object in the reference image, can determine the difference between the opening / closing state of the opening / closing device of the target object in the target image and the first opening / closing state of the opening / closing device of the target object in the reference image. Since the first opening / closing state of the opening / closing device of the target object in the reference image is fixed and known, the opening / closing state of the opening / closing device of the target object in the target image can be determined by comparing the differences between at least one target salient region of the target object in the target image and at least one reference salient region of the target object in the reference image. Furthermore, since the entities corresponding to the reference salient region and the target salient region belong to the target object, that is, since both the reference salient region and the target salient region are regions related to the target object's own attributes, the opening / closing state of the target object's opening / closing device can be detected based on the target object itself, without needing to add additional obvious labeling material to the target object, thus broadening the applicability of the opening / closing state detection method provided in this application. Attached Figure Description
[0008] Figure 1 This is a flowchart illustrating an embodiment of the opening / closing state detection method provided in this application;
[0009] Figure 2 This is a schematic diagram of an embodiment of the target image provided in this application;
[0010] Figure 3 This is a schematic diagram of another embodiment of the target image provided in this application;
[0011] Figure 4 This is a schematic diagram of another embodiment of the target image provided in this application;
[0012] Figure 5 This is a schematic diagram of another embodiment of the target image provided in this application;
[0013] Figure 6 This is a flowchart illustrating an embodiment of determining a target entity in a target image provided in this application;
[0014] Figure 7 This is a schematic diagram of an embodiment of the detected target image provided in this application;
[0015] Figure 8 This is a schematic diagram of another embodiment of the detected target image provided in this application;
[0016] Figure 9 yes Figure 1 The flowchart of step S12 shown is a schematic diagram of one embodiment;
[0017] Figure 10This is a schematic diagram of an embodiment of the image to be detected provided in this application;
[0018] Figure 11 This is a schematic diagram of another embodiment of the image to be detected provided in this application;
[0019] Figure 12 This is a schematic diagram of an embodiment of the location information of the target object provided in this application;
[0020] Figure 13 This is a schematic diagram of another embodiment of the location information of the target object provided in this application;
[0021] Figure 14 This is a schematic diagram of an embodiment of the target salience region provided in this application;
[0022] Figure 15 This is a schematic flowchart of an embodiment of determining the opening and closing state of an opening and closing device based on the difference between parameter information of at least one target salience region and parameter information of at least one reference salience region, provided in this application.
[0023] Figure 16 yes Figure 15 The flowchart of an embodiment of step S1503 is shown.
[0024] Figure 17 yes Figure 16 The flowchart shown is a representation of step S1605 in one embodiment.
[0025] Figure 18 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application;
[0026] Figure 19 This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. Detailed Implementation
[0027] The embodiments of this application will now be described in detail with reference to the accompanying drawings.
[0028] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.
[0029] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.
[0030] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the opening / closing state detection method provided in this application. It should be noted that if substantially the same result is obtained, this embodiment does not necessarily replace it. Figure 1 The illustrated process sequence is limited. For example... Figure 1 As shown, this embodiment includes:
[0031] Step S11: Obtain the target image and at least one reference salient region on the target object in the reference image.
[0032] The method of this embodiment is used to determine the opening and closing state of the opening and closing device of a target object. The target object described herein includes, but is not limited to, electrical distribution boxes, electrical distribution cabinets, containers, etc., and the opening and closing device includes, but is not limited to, doors, etc.; for example, taking a door as the opening and closing device and an electrical distribution box as the target object, the method of this embodiment is used to determine the opening and closing state of the door of the electrical distribution box.
[0033] In this embodiment, a target image is acquired, which is a captured image of the target object. In one embodiment, the target image can be obtained from local storage or cloud storage. Of course, in other embodiments, the target object can also be captured in real time using an image acquisition device; this is not specifically limited here. For example, such as... Figures 2-5 As shown, Figure 2 This is a schematic diagram of an embodiment of the target image provided in this application. Figure 3 This is a schematic diagram of another embodiment of the target image provided in this application. Figure 4 This is a schematic diagram of another embodiment of the target image provided in this application. Figure 5 This is a schematic diagram of another embodiment of the target image provided in this application, taking a power distribution box as an example. Figures 2-5 The target image shown was acquired from the distribution box at the same angle, the same position, and at different acquisition times.
[0034] In addition, in this embodiment, at least one reference salient region on the target object in the reference image is also acquired. The target image and the reference image are acquired from the same angle and position of the target object. The target object is equipped with an opening and closing device, and the reference image is acquired from the target object with the opening and closing device in a first open / closed state. The first open / closed state of the opening and closing device is not limited and can be specifically set according to actual usage needs; for example, the first open / closed state of the opening and closing device can be the closed state of the opening and closing device, or it can be the open state of the opening and closing device at a 90° angle to the target object, etc.
[0035] In one embodiment, at least one reference salient region on the target object in the reference image can be obtained from local storage or cloud storage. Of course, in other embodiments, at least one reference salient region on the target object in the reference image can also be obtained by performing salientity identification on the reference image in real time, and no specific limitation is made here.
[0036] Step S12: Perform saliency identification on the target object in the target image to obtain at least one target saliency region.
[0037] In this embodiment, the saliency of target objects in the target image is identified to obtain at least one target saliency region. In one embodiment, a saliency identification model can be used to identify the saliency of target objects in the target image. Of course, in other embodiments, a saliency identification algorithm can also be used to identify the saliency of target objects in the target image, and no specific limitation is made here.
[0038] In one specific implementation, a saliency recognition model is used to perform saliency recognition on target objects in a target image; wherein the training process of the saliency recognition model specifically includes the following steps:
[0039] Step 1: Obtain an initial saliency identification model, such as a detection network like U2-Net; Step 2: Train the initial saliency identification model using the sample dataset to obtain the target saliency identification model.
[0040] To reduce the impact of elements other than the target object in the target image on the saliency identification of the target object, that is, to improve the accuracy of the obtained at least one target saliency region, in one embodiment, the target image can be cropped to obtain a detection image containing only the target object, and saliency identification can be performed on the detection image. In other embodiments, no processing can be performed on the target image, and the saliency identification of the target object in the target image can be performed directly.
[0041] Because the presence of a target entity in the target image, or the presence of a target entity in the target image where the position of the target entity overlaps with the position of the target object, affects the accuracy of determining the opening and closing state of the opening and closing device for the target object. Therefore, in one embodiment, such as Figure 6 As shown, Figure 6 This is a flowchart illustrating an embodiment of determining a target entity in a target image provided in this application. Before performing salience identification on the target object in the target image to obtain at least one salient target region, the method further includes the following sub-steps:
[0042] Step S61: Determine whether a target entity exists in the target image.
[0043] In this embodiment, it is determined whether a target entity exists in the target image. Specifically, target entity detection is performed on the target image to determine whether a target entity exists in the target image. In one embodiment, a target entity recognition model can be used to recognize target entities in the target image. Of course, in other embodiments, a target entity recognition algorithm can also be used to recognize target entities in the target image, and no specific limitation is made here.
[0044] In one specific implementation, a target entity recognition model is used to perform target entity recognition on a target image; wherein, the training process of the target entity recognition model specifically includes the following steps: Step 1: Obtain an initial target entity recognition model, such as a detection network like YOLOv4; Step 2: Train the initial target entity recognition model using a sample dataset to obtain a converged target entity recognition model.
[0045] In one embodiment, the target entity is a person. Due to the movement of people, the captured image of the target object may contain people, or even obstruct the target object. Of course, in other embodiments, the target entity can be other entities, and this is not specifically limited here. For example, such as Figure 4 and Figure 5 The target image shown contains a target entity – a person.
[0046] Step S62: In response to the absence of a target entity in the target image, or the presence of a target entity in the target image and the positional relationship between the target object and the target entity in the target image not satisfying the overlap condition, perform saliency identification of the target object in the target image to obtain at least one target saliency region and subsequent steps.
[0047] In this embodiment, in response to the absence of a target entity in the target image, or the presence of a target entity in the target image but the positional relationship between the target object and the target entity does not satisfy the overlap condition, saliency identification of the target object in the target image is performed to obtain at least one target saliency region and subsequent steps. That is, when there is no target entity in the target image, or when a target entity exists in the target image but the positional relationship between the target entity and the target object does not satisfy the overlap condition, saliency identification of the target object in the target image is performed. This reduces the impact of the appearance of the target entity on the accuracy of the subsequently determined at least one target saliency region, thereby avoiding interference with the determination of the opening and closing state of the subsequent target object opening and closing device.
[0048] In one embodiment, the overlap condition is that the positions of the target object and the target entity in the target image overlap. In a specific embodiment, the overlap between the positions of the target object and the target entity in the target image is determined by determining whether the bounding rectangle corresponding to the target object and the bounding rectangle corresponding to the target entity overlap.
[0049] For example, consider the saliency identification of a target object in a target image when the target entity does not exist in the target image, where the target entity is a person and the target object is an electrical distribution box: For example... Figure 2 The target image shown is used for personnel detection. If no people are found in the target image, then the electrical distribution box in the target image is salientically identified. Figure 7 As shown, Figure 7 This is a schematic diagram of an embodiment of the detected target image provided in this application, for example... Figure 4 The target image shown is used for person detection, and the result is as follows: Figure 7 The detected image shows that a person is identified in the target image. At this point, no saliency identification is performed on the target object in the target image.
[0050] For example, consider the saliency identification of a target object in a target image when the positional relationship between the target object and the target entity does not satisfy the overlap condition. The overlap condition is that the bounding rectangle of the target object overlaps with the bounding rectangle of the target entity, and the target entity is a person while the target object is a distribution box. Figure 4 The target image shown is used to detect people and electrical boxes, resulting in the following: Figure 7 The bounding rectangles of the person and the distribution box are shown. Since the bounding rectangles of the person and the distribution box do not overlap, it indicates that the distribution box and the person in the target image do not overlap. That is, it indicates that a person exists in the target image, and the positional relationship between the distribution box and the person in the target image does not meet the overlap requirement. At this time, saliency identification is performed on the target image. Figure 8 As shown, Figure 8This is a schematic diagram of another embodiment of the detected target image provided in this application, for example... Figure 5 The target image shown is used to detect people and electrical boxes, resulting in the following: Figure 8 The bounding rectangles of the person and the distribution box are shown. Since the bounding rectangles of the person and the distribution box overlap, it indicates that the distribution box and the person in the target image overlap. That is, it indicates that there is a person in the target image and the positional relationship between the distribution box and the person in the target image meets the overlap requirement. At this time, no saliency recognition is performed on the target image.
[0051] Step S13: Determine the opening / closing state of the target object's opening / closing device based on the difference between at least one target salient region and at least one reference salient region.
[0052] In this embodiment, the opening / closing state of the opening / closing device of the target object is determined based on the difference between at least one target salient region and at least one reference salient region. By comparing the difference between at least one target salient region of the target object in the target image and at least one reference salient region of the target object in the reference image, the difference between the opening / closing state of the opening / closing device of the target object in the target image and the first opening / closing state of the opening / closing device of the target object in the reference image can be determined. Since the first opening / closing state of the opening / closing device of the target object in the reference image is fixed and known, the opening / closing state of the opening / closing device of the target object in the target image can be determined by comparing the difference between at least one target salient region of the target object in the target image and at least one reference salient region of the target object in the reference image. That is, by comparing the difference between at least one target salient region of the target object in the target image and at least one reference salient region of the target object in the reference image, the opening / closing state of the opening / closing device of the target object in the target image can be automatically identified. Furthermore, since the entities corresponding to the reference salient region and the target salient region belong to the target object, that is, since both the reference salient region and the target salient region are regions related to the target object's own attributes, the opening and closing state of the target object's opening and closing device can be detected based on the target object itself, without the need to add obvious marking materials to the target object, thus making the opening and closing state detection method provided in this application more widely applicable.
[0053] For example, taking the first opening / closing state as the closed state, the target object as a distribution box, and the opening / closing device as the box door as an example; by comparing the differences between at least one target salient region of the distribution box in the target image and at least one reference salient region of the distribution box in the reference image, the difference between the opening / closing state of the distribution box door in the target image and the closed state of the distribution box door in the reference image can be determined; if there is no difference between at least one target salient region of the distribution box in the target image and at least one reference salient region of the distribution box in the reference image, then This indicates that the opening / closing state of the distribution box door in the target image is the same as the first opening / closing state (closed state) of the distribution box in the reference image, that is, the door of the distribution box in the target image is in a closed state; if there is a difference between at least one target salient area of the distribution box in the target image and at least one reference salient area of the distribution box in the reference image, it indicates that the opening / closing state of the distribution box door in the target image is different from the first opening / closing state (closed state) of the distribution box in the reference image, that is, the door of the distribution box in the target image is in an open state.
[0054] For example, taking the first opening / closing state as the open state and the opening / closing device of the target object in the reference image at a 90° angle to the target object, the target object as an electrical distribution box, and the opening / closing device as a box door as an example; by comparing the difference between at least one target salient area of the electrical distribution box in the target image and at least one reference salient area of the electrical distribution box in the reference image, the difference between the opening / closing state of the box door in the target image and the opening / closing state of the box door in the reference image can be determined; if at least one target salient area of the electrical distribution box in the target image and at least one reference salient area of the electrical distribution box in the reference image... If there is no difference between the reference salient regions, it indicates that the door of the distribution box in the target image is open; if there is a difference between at least one target salient region of the distribution box in the target image and at least one reference salient region of the distribution box in the reference image, and the difference is within a preset range, it indicates that the door of the distribution box in the target image is open; if there is a difference between at least one target salient region of the distribution box in the target image and at least one reference salient region of the distribution box in the reference image, and the difference is not within a preset range, it indicates that the door of the distribution box in the target image is closed.
[0055] Since the opening / closing state of the opening / closing device of the target object in the target image differs from the first opening / closing state of the opening / closing device of the target object in the reference image, the parameter information of at least one target salient region corresponding to the target object in the target image and the parameter information of at least one reference salient region corresponding to the target object in the reference image will inevitably differ. Therefore, in one embodiment, the opening / closing state of the opening / closing device of the target object in the target image can be determined based on the parameter difference between the parameter information of at least one target salient region and the parameter information of at least one reference salient region; wherein the parameter information includes at least one of the parameters representing the number of regions, position, and size.
[0056] In one specific embodiment, the dimensional characterization parameter includes at least one of area and contour length, and the contour length includes at least one of contour perimeter and contour single-side length. It is understood that in other specific embodiments, the dimensional characterization parameter may also include radius, diameter, etc., and is not specifically limited herein.
[0057] In the above embodiments, by comparing the differences between at least one target salient region of the target object in the target image and at least one reference salient region of the target object in the reference image, the difference between the opening / closing state of the opening / closing device of the target object in the target image and the first opening / closing state of the opening / closing device of the target object in the reference image can be determined. Since the first opening / closing state of the opening / closing device of the target object in the reference image is fixed and known, the opening / closing state of the opening / closing device of the target object in the target image can be determined by comparing the differences between at least one target salient region of the target object in the target image and at least one reference salient region of the target object in the reference image. Furthermore, since the entities corresponding to the reference salient region and the target salient region belong to the target object—that is, since both the reference salient region and the target salient region are regions related to the target object's own attributes—the opening / closing state of the target object can be detected based on the target object itself, without needing to add additional obvious labeling material to the target object. This makes the opening / closing state detection method provided in this application more widely applicable.
[0058] Please see Figure 9 , Figure 9 yes Figure 1 The flowchart shown is a schematic diagram of one embodiment of step S12. It should be noted that if substantially the same result is achieved, this embodiment does not necessarily follow the same pattern. Figure 9 The illustrated process sequence is limited. For example... Figure 9 As shown, in this embodiment, the target image is cropped to obtain a detection image containing only the target object, and saliency identification is performed on the detection image, specifically including:
[0059] Step S91: Based on the location information of the target object in the target image, crop the target image to obtain the image to be detected corresponding to the target object.
[0060] In this embodiment, the target image is cropped using the location information of the target object in the target image to obtain a detection image corresponding to the target object. Since the cropped detection image contains only the target object, or rather, since the cropped detection image contains very few other elements besides the target object, the accuracy of obtaining at least one salient region of the target object through subsequent saliency identification of the detection image is high, thereby improving the accuracy of subsequent determination of the opening / closing state of the target object. For example, such as... Figure 2 and Figure 10 As shown, Figure 10 This is a schematic diagram of an embodiment of the image to be detected provided in this application, taking a power distribution box as an example; based on... Figure 2 The location information of the distribution box in the target image is shown. The target image is then cropped to obtain the following result: Figure 10 The image to be detected corresponds to the distribution box shown. For example, such as... Figure 3 and Figure 11 As shown, Figure 11 This is a schematic diagram of another embodiment of the image to be detected provided in this application, taking a power distribution box as an example; based on... Figure 3 The location information of the distribution box in the target image is shown. The target image is then cropped to obtain the following result: Figure 11 The image to be tested corresponds to the distribution box shown.
[0061] Since the target object is imaged from the same angle and position, and the target object is generally stationary, in one embodiment, the target object's position information can be obtained based on prior position information of the target object in several acquired images. That is, since the target object's position is stationary, after determining its position information based on several acquired images, the position information of the target object in the target image does not need to be detected again; the position information determined based on several acquired images can be directly reused. This reduces the overall time consumption of the method, improves detection efficiency, and reduces the equipment resources used. Specifically, the acquired images used to determine the target object's position information either do not contain a target entity or contain a target entity but the positional relationship between the target entity and the target object does not meet the overlap requirement. Then, the position information of the target object in each acquired image is determined. Finally, the position information of the target object in each acquired image is averaged to obtain the final position information of the target object.
[0062] In other embodiments, the location information of the target object can also be obtained by real-time target detection of the target image. Specifically, the location information of the target object can be obtained by performing target detection on the target image using a target detection model or algorithm. In one specific embodiment, a target detection model is used to detect the target object in the target image; the training process of the target detection model specifically includes the following steps: Step 1: Obtain an initial target detection model, such as a detection network like YOLOv4; Step 2: Train the initial target detection model using a sample dataset to obtain a converged target detection model.
[0063] In one specific implementation, the location information of the target object is specifically the bounding rectangle of the target object. For example, assuming the location information of the target object is obtained by target detection from a target image, the location information of the target object is its bounding rectangle, and the target object is a power distribution box: First, as... Figure 2 , Figure 10 and Figure 12 As shown, Figure 12 This is a schematic diagram of an embodiment of the location information of the target object provided in this application. Figure 2 Target detection is performed on the target image shown, and the result is as follows: Figure 12 The image shows the outer rectangular frame of the distribution box; then, the target image is cropped based on the outer rectangular frame of the distribution box to obtain the result shown below. Figure 10 The image to be detected corresponds to the distribution box shown. For example, taking a distribution box as an example where the target object's location information is obtained by target detection on the target image, the target object's location information is its bounding rectangle, and the target object is the distribution box: First, for example... Figure 3 , Figure 11 and Figure 13 As shown, Figure 13 This is a schematic diagram of another embodiment of the location information of the target object provided in this application, for example... Figure 3 Target detection is performed on the target image shown, and the result is as follows: Figure 13 The image shows the outer rectangular frame of the distribution box; then, the target image is cropped based on the outer rectangular frame of the distribution box to obtain the result shown below. Figure 13 The image to be tested corresponds to the distribution box shown.
[0064] In other specific embodiments, the location information of the target object may also be the coordinates of the center of the target object, etc., which are not specifically limited here.
[0065] Step S92: Perform saliency identification on the image to be detected to obtain at least one target saliency region.
[0066] In this embodiment, saliency identification is performed on the image to be detected to obtain at least one target saliency region. Since the cropped image to be detected contains only the target object, or rather, since the cropped image to be detected contains very few other elements besides the target object, the accuracy of the at least one target saliency region obtained by saliency identification of the image to be detected is high, thereby improving the accuracy of subsequent determination of the opening and closing state of the target object.
[0067] For example, such as Figure 14 As shown, Figure 14 This is a schematic diagram of an embodiment of the target salience region provided in this application. Figure 14 The images to be detected (a), (b), (c), and (d) are used to perform saliency identification, and at least one target saliency region is obtained for each image to be detected.
[0068] Please see Figure 15 , Figure 15 This is a flowchart illustrating an embodiment of this application for determining the opening / closing state of an opening / closing device based on the difference in target parameters between parameter information of at least one target salience region and parameter information of at least one reference salience region. It should be noted that if substantially the same result is obtained, this embodiment does not necessarily use it. Figure 15 The illustrated process sequence is limited. For example... Figure 15 As shown, this embodiment includes:
[0069] Step S1501: Determine whether the parameter difference belongs to the state change difference.
[0070] In this embodiment, it is determined whether the parameter difference belongs to the state change difference. The state change difference characterizes the parameter difference between the salient regions of the target object when the opening and closing device is in different opening and closing states. Therefore, by determining whether the parameter difference between at least one salient region of the target object in the target image and at least one reference salient region of the target object in the reference image belongs to the state change difference, it is possible to determine whether the opening and closing state of the opening and closing device of the target object in the target image is the same as the first opening and closing state of the opening and closing device of the target object in the reference image, thereby determining the opening and closing state of the opening and closing device of the target object in the target image.
[0071] Specifically, if the parameter difference between at least one target salient region of the target object in the target image and at least one reference salient region of the target object in the reference image is a state change difference, step S1503 is executed; and if the parameter difference between at least one target salient region of the target object in the target image and at least one reference salient region of the target object in the reference image is not a state change difference, step S1502 is executed.
[0072] In one embodiment, the state change difference includes at least one of the following: the number of regions differs between at least one target salient region and at least one reference salient region; there exists a region pair whose difference in position exceeds a preset position change range; there exists a region pair whose difference in size characterization parameters exceeds a preset size change range; wherein, the region pair includes a reference salient region and a corresponding target salient region.
[0073] There are no limitations on the preset position variation range and preset size variation range; they can be set according to actual usage needs.
[0074] Step S1502: In response to the parameter difference not being a state change difference, determine that the opening and closing device is in the first opening and closing state.
[0075] In this embodiment, in response to the parameter difference not being a state change difference, it is determined that the opening and closing device of the target object in the target image is in a first opening and closing state. That is, when the parameter difference is not a state change difference, it is indicated that the opening and closing device of the target object in the target image and the opening and closing device of the target object in the reference image are in the same opening and closing state, i.e., the opening and closing device of the target object in the target image is in the first opening and closing state.
[0076] Step S1503: In response to the parameter difference being a state change difference, determine that the opening and closing device is in the second opening and closing state.
[0077] In this embodiment, in response to the target parameter difference being a state change difference, the opening and closing device of the target object in the target image is determined to be in a second opening and closing state. Here, one of the first and second opening and closing states is a closed state, and the other is an open state. That is, when the first opening and closing state is closed, if the parameter difference is a state change difference, it indicates that the opening and closing device of the target object in the target image and the opening and closing device of the target object in the reference image are in different opening and closing states, i.e., the opening and closing device of the target object in the target image is in the open state; when the first opening and closing state is open, if the parameter difference is a state change difference, it indicates that the opening and closing device of the target object in the target image and the opening and closing device of the target object in the reference image are in different opening and closing states, i.e., the opening and closing device of the target object in the target image is in the closed state.
[0078] Since the opening and closing devices of the target object in the target image and the target object in the reference image are both in an open state but to different degrees, there may be parameter differences between them. In this case, if the first opening and closing state is set to an open state, it may lead to misjudgment of the opening and closing state of the target object's opening and closing device in the target image. Therefore, when the first opening and closing state is set to an open state, it is necessary to further determine the opening and closing state of the target object's opening and closing device in the target image, which is relatively complex and time-consuming. Therefore, in one embodiment, the first opening and closing state is set to a closed state to improve the accuracy and efficiency of detecting the opening and closing state of the target object's opening and closing device in the target image.
[0079] The first opening / closing state is a closed state, and the second opening / closing state is an open state; the state change differences include a difference in the number of regions between at least one target salient region and at least one reference salient region, the existence of region pairs whose positional differences exceed a preset range of positional changes, and the existence of region pairs whose size representation parameters exceed a preset range of size changes. In one embodiment, regions with a difference in the number of regions between at least one target salient region and at least one reference salient region, regions whose positional differences exceed a preset range of positional changes, and regions whose size representation parameters exceed a preset range of size changes are determined respectively. To improve the efficiency of determining the opening / closing state of the opening / closing device for a target object in a target image and reduce resource consumption, in one embodiment, such as Figure 16 As shown, Figure 16 yes Figure 15 The flowchart of step S1503 shown is a schematic diagram of an embodiment. It can sequentially execute pairs of regions where the number of regions differs between at least one target salient region and at least one reference salient region, pairs of regions where the difference in position exceeds a preset range of positional variation, and pairs of regions where the difference in size characterization parameters exceeds a preset range of size variation. After determining the opening / closing state of the opening / closing device through one of these methods, subsequent steps are stopped. Specifically, the steps include the following:
[0080] Step S1601: Obtain the number of first regions of at least one target saliency region and the number of second regions of at least one reference saliency region.
[0081] Compared to differences in location and size representation parameters, the difference in the number of regions between at least one target salient region and at least one reference salient region is more intuitive, making the determination of this difference in region number faster and simpler. Therefore, in this embodiment, the first number of regions of at least one target salient region and the second number of regions of at least one reference salient region are first obtained.
[0082] Step S1602: In response to the difference between the number of the first region and the number of the second region, determine that the opening and closing device is in the second opening and closing state.
[0083] In this embodiment, in response to the difference between the number of first regions of at least one target salient region and the number of second regions of at least one reference salient region, it is determined that the opening and closing device of the target object in the target image is in a second opening and closing state, that is, it is determined that the opening and closing device of the target object in the target image is in an open state. Since the number of first regions of at least one target salient region is different from the number of second regions of at least one reference salient region, the opening and closing state of the opening and closing device of the target object in the target image is different from the opening and closing state of the opening and closing device of the target object in the reference image. Since the opening and closing device of the target object in the reference image is in a closed state, it is determined that the opening and closing device of the target object in the target image is in an open state.
[0084] For example, such as Figure 14 As shown, assuming Figure 14 (a) is a reference image. Figure 14 (b) Figure 14 (c) and Figure 14 (d) shows the target images acquired at different acquisition times; due to the target image - Figure 14 In (d), the number of the first region of the target salience region corresponding to the target object is 3, and the reference image - Figure 14 The number of second regions corresponding to the reference saliency region of the target object in (a) is 3. Therefore, it is determined that the opening and closing device of the target object in target image-14(d) is in the same opening and closing state as the opening and closing device of the target object in reference image-14(a), that is, it is determined that the opening and closing device of the target object in target image-14(d) is in the closed state; since the target image- Figure 14 (b) and the target image - Figure 14 (c) The number of the first region of the target salience region corresponding to the target object is 1, and the reference image - Figure 14 The number of second regions corresponding to the reference saliency region of the target object in (a) is 3. Therefore, it is determined that the opening and closing device of the target object in target image-14(b) and target image-14(c) is in a different opening and closing state than the opening and closing device of the target object in reference image-14(a). That is, it is determined that the opening and closing device of the target object in target image-14(b) and target image-14(c) is in the open state.
[0085] Step S1603: In response to the fact that the number of the first region is the same as the number of the second region, obtain the first difference of each region with respect to location.
[0086] In this embodiment, in response to the fact that the number of first regions of at least one target salient region is the same as the number of second regions of at least one reference salient region, a first difference in position between each region pair is obtained. Since the number of at least one target salient region corresponding to the target object may not change when the opening and closing device of the target object in the target image is open to a certain extent, the number of first regions of at least one target salient region may be the same as the number of second regions of at least one reference salient region. However, the opening and closing device of the target object in the target image is actually in an open state. Therefore, when the number of first regions is the same as the number of second regions, the opening and closing state of the opening and closing device of the target object in the target image cannot be directly determined. At this time, the first difference in position between each region pair is obtained so as to determine the opening and closing state of the opening and closing device of the target object in the target image through the positional difference between each region pair.
[0087] Step S1604: In response to the existence of a region pair with a first difference exceeding the preset position change range, determine that the opening and closing device is in the second opening and closing state.
[0088] In this embodiment, in response to the existence of a pair of regions where the first difference exceeds a preset positional change range, it is determined that the opening and closing device of the target object in the target image is in a second opening and closing state, that is, it is determined that the opening and closing device of the target object in the target image is in an open state. Since there are pairs of regions where the first difference exceeds the preset positional change range, the opening and closing state of the opening and closing device of the target object in the target image is different from the opening and closing state of the opening and closing device of the target object in the reference image. Since the opening and closing device of the target object in the reference image is in a closed state, it is determined that the opening and closing device of the target object in the target image is in an open state.
[0089] Step S1605: In response to the fact that the first difference of each region pair is within the preset position change range, the second difference of each region pair with respect to the size characterization parameter is obtained, and in response to the existence of a region pair with the second difference exceeding the preset size change range, the opening and closing device is determined to be in the second opening and closing state.
[0090] In this embodiment, in response to the fact that the first difference of each region pair is within a preset position variation range, the second difference of each region pair with respect to the size characterization parameter is obtained. Since the opening and closing range of the target object's opening and closing device in the target image is within a certain range, the first difference of each region pair with respect to the position may all be within the preset position variation range. However, the opening and closing device of the target object in the target image is actually in an open state. Therefore, even if the first difference of each region pair is within the preset position variation range, the opening and closing state of the target object's opening and closing device in the target image cannot be directly determined. At this time, the second difference of each region pair with respect to the size characterization parameter is obtained so as to determine the opening and closing state of the target object's opening and closing device in the target image through the size difference of each region pair.
[0091] Specifically, in response to the existence of a pair of regions where the second difference exceeds a preset size variation range, the opening and closing device is determined to be in a second opening and closing state; that is, the opening and closing device of the target object in the target image is determined to be in an open state. Because there are pairs of regions where the second difference exceeds the preset size variation range, the opening and closing state of the target object in the target image differs from that in the reference image, where the target object's opening and closing device is in a closed state. Therefore, the opening and closing device of the target object in the target image is determined to be in an open state.
[0092] In one implementation, such as Figure 17 As shown, Figure 17 yes Figure 16 The flowchart shown in step S1605 is a schematic diagram of an embodiment. The size characterization parameters include area and contour length parameters. Determining the opening and closing state of the opening and closing device based on the second difference of each region with respect to the size characterization parameters specifically includes the following sub-steps:
[0093] Step S1701: Obtain the second difference in area for each region.
[0094] In this embodiment, a second difference in area is obtained for each region. In one embodiment, the area of each region is specifically the number of pixels contained in each region.
[0095] Step S1702: In response to the existence of a pair of regions with a second difference in area that exceeds a preset area change range, determine that the opening and closing device is in a second opening and closing state.
[0096] In this embodiment, in response to the existence of a pair of regions where the second difference in area exceeds a preset area variation range, it is determined that the opening and closing device of the target object in the target image is in a second opening and closing state, that is, it is determined that the opening and closing device of the target object in the target image is in an open state. Since there are pairs of regions where the second difference in area exceeds the preset area variation range, the opening and closing state of the opening and closing device of the target object in the target image is different from the opening and closing state of the opening and closing device of the target object in the reference image. Since the opening and closing device of the target object in the reference image is in a closed state, it is determined that the opening and closing device of the target object in the target image is in an open state.
[0097] Step S1703: In response to the fact that the second difference of each region with respect to area is within the preset area variation range, obtain the second difference of each region with respect to contour length.
[0098] In this embodiment, in response to the fact that the second difference in area of each region is within a preset area variation range, the second difference in contour length of each region is obtained. Since the opening and closing mechanism of the target object in the target image may be within a preset area variation range when the opening range is a certain size, and the opening and closing mechanism of the target object in the target image is actually in an open state, the opening and closing state of the target object in the target image cannot be directly determined when the second difference in area of each region is within the preset area variation range. At this time, the second difference in contour length of each region is obtained so as to determine the opening and closing state of the target object in the target image by the difference in contour length of each region.
[0099] Step S1704: In response to the existence of a pair of regions where the second difference in the profile length exceeds the preset length variation range, determine that the opening and closing device is in the second opening and closing state.
[0100] In this embodiment, in response to the existence of a pair of regions where the second difference in contour length exceeds a preset length variation range, the opening and closing device is determined to be in a second opening and closing state; that is, the opening and closing device of the target object in the target image is determined to be in an open state. Since there are pairs of regions where the second difference in contour length exceeds the preset length variation range, the opening and closing state of the target object in the target image differs from that in the reference image, where the target object's opening and closing device is in a closed state. Therefore, the opening and closing device of the target object in the target image is determined to be in an open state.
[0101] Furthermore, when the second difference in the contour length in each region is within the preset length variation range, the opening and closing state of the target object's opening and closing device in the target image is the same as the opening and closing state of the target object's opening and closing device in the reference image, while the target object's opening and closing device in the reference image is in a closed state. Therefore, it is determined that the target object's opening and closing device in the target image is in a closed state.
[0102] Please see Figure 18 , Figure 18 This is a schematic diagram of an embodiment of the electronic device provided in this application. The electronic device 180 includes a memory 1801 and a processor 1802 coupled to each other. The processor 1802 is used to execute program instructions stored in the memory 1801 to implement the steps of any of the above-described embodiments of the opening / closing state detection method. In a specific implementation scenario, the electronic device 180 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 180 may also include mobile devices such as laptops and tablets, which are not limited here.
[0103] Specifically, processor 1802 controls itself and memory 1801 to implement the steps of any of the above-described open / closed state detection method embodiments. Processor 1802 can also be referred to as a CPU (Central Processing Unit). Processor 1802 may be an integrated circuit chip with signal processing capabilities. Processor 1802 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 1802 can be implemented using integrated circuit chips.
[0104] Please see Figure 19 , Figure 19 This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. The computer-readable storage medium 190 of this application embodiment stores program instructions 1901. When executed, these program instructions 1901 implement the method provided by any embodiment of the opening / closing state detection method and any non-conflicting combination thereof. The program instructions 1901 can form a program file and be stored in the aforementioned computer-readable storage medium 190 in the form of a software product, so that a computer device (which may be a personal computer, server, or network device, etc.) can execute all or part of the steps of the methods of various embodiments of this application. The aforementioned computer-readable storage medium 190 includes various media capable of storing program code, such as a USB flash drive, mobile hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or terminal devices such as computers, servers, mobile phones, and tablets.
[0105] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. A method for detecting an open / closed state, characterized in that, The method includes: At least one reference salient region on a target object in a reference image is obtained; wherein the target object is provided with an opening and closing device, and the reference image is acquired from the target object with the opening and closing device in a first opening and closing state; The saliency of a target object in a target image is identified to obtain at least one target saliency region; wherein the target image and the reference image are acquired from the same angle and the same position of the target object, and the entities corresponding to the reference saliency region and the target saliency region belong to the target object; Based on the difference between the at least one target salience region and the at least one reference salience region, the opening and closing state of the opening and closing device of the target object is determined; The step of determining the opening / closing state of the target object's opening / closing device based on the difference between the at least one target salience region and the at least one reference salience region includes: The opening and closing state of the opening and closing device is determined based on the parameter difference between the parameter information of the at least one target salience region and the parameter information of the at least one reference salience region; wherein the parameter information includes at least one of the region number, position and size characterization parameters.
2. The method according to claim 1, characterized in that, The dimensional characterization parameters include at least one of area and contour length, and the contour length includes at least one of contour perimeter and the length of one side of the contour.
3. The method according to claim 1, characterized in that, Determining the opening / closing state of the opening / closing device based on the parameter difference between the parameter information of the at least one target saliency region and the parameter information of the at least one reference saliency region includes: In response to the parameter difference being a state change difference, the opening and closing device is determined to be in a second opening and closing state, where one of the first opening and closing state and the second opening and closing state is a closed state and the other is an open state. The state change difference characterizes the parameter difference between significant regions of the target object when the opening and closing device is in different opening and closing states. In response to the parameter difference not belonging to the state change difference, it is determined that the opening and closing device is in the first opening and closing state.
4. The method according to claim 3, characterized in that, The first opening / closing state is a closed state, and the second opening / closing state is an open state; The state change differences include at least one of the following: the number of regions differs between the at least one target salience region and the at least one reference salience region; there are regions whose differences regarding position exceed a preset range of positional changes; there are regions whose differences regarding size characterization parameters exceed a preset range of size changes. The region pair includes a reference salience region and a corresponding target salience region.
5. The method according to claim 4, characterized in that, The step of determining that the opening / closing device is in a second opening / closing state in response to the parameter difference being a state change difference includes: The number of first regions of the at least one target saliency region and the number of second regions of the at least one reference saliency region are obtained respectively; In response to the difference between the number of the first region and the number of the second region, it is determined that the opening and closing device is in a second opening and closing state; In response to the fact that the number of the first region is the same as the number of the second region, a first difference between each of the regions with respect to the location is obtained; In response to the existence of a region pair where the first difference exceeds the preset position change range, the opening and closing device is determined to be in a second opening and closing state; In response to the fact that the first difference of each of the region pairs is within the preset position change range, the second difference of each of the region pairs with respect to the size characterization parameter is obtained, and in response to the existence of a region pair where the second difference exceeds the preset size change range, the opening and closing device is determined to be in a second opening and closing state.
6. The method according to claim 5, characterized in that, The size characterization parameters include area and contour length parameters; obtaining the second difference of each region pair with respect to the size characterization parameters, and determining that the opening and closing device is in a second opening and closing state in response to the existence of a region pair where the second difference exceeds a preset size change range, includes: Obtain the second difference between each of the regions with respect to the area; In response to the existence of a pair of regions where the second difference in area exceeds a preset area variation range, the opening and closing device is determined to be in a second opening and closing state; In response to the fact that the second difference of each region with respect to the area is within a preset area variation range, the second difference of each region with respect to the contour length is obtained; In response to the existence of a pair of regions where the second difference in the length of the outline exceeds a preset length variation range, the opening and closing device is determined to be in a second opening and closing state.
7. The method according to claim 1, characterized in that, Before performing saliency identification on the target object in the target image to obtain at least one salient region of the target, the method further includes: Determine whether a target entity exists in the target image; In response to the absence of the target entity in the target image, or the presence of the target entity in the target image and the positional relationship between the target object in the target image and the target entity not satisfying the overlap condition, the saliency identification of the target object in the target image is performed to obtain at least one target saliency region and subsequent steps.
8. The method according to claim 1, characterized in that, The step of identifying the saliency of targets in a target image to obtain at least one salient region of the target includes: Based on the location information of the target object in the target image, the target image is cropped to obtain the image to be detected corresponding to the target object; wherein, the location information of the target object is obtained by target detection in the target image, or is obtained based on the prior location information of the target object in several acquired images; The image to be detected is subjected to saliency identification to obtain at least one target saliency region.
9. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing program instructions, and the processor executing the program instructions to implement the method for detecting the opening / closing state as described in any one of claims 1-8.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program instructions that can be executed to implement the method for detecting open / closed states as described in any one of claims 1-8.