Decoration change action recognition method, device and equipment based on decoration semantic constraint and medium

Abstract

Embodiments of the present disclosure disclose a decoration change action recognition method and device based on decoration semantic constraint, equipment and medium. A specific embodiment of the method comprises: acquiring a first time image and a second time image; performing feature-guided spatial alignment processing on the second time image; performing semantic segmentation processing on the aligned second time image; performing double-threshold change probability filtering processing on the semantic segmentation image and the pixel-level change probability image; performing connected domain labeling processing on the decoration change mask image; for each connected domain position information, performing decoration change action recognition processing on the image at the position represented by the connected domain position information in the first time image and the image at the position represented by the connected domain position information in the aligned second time image, to obtain decoration action recognition information. The embodiment relates to the technical field of computers, and improves the accuracy of decoration change action recognition.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of computer technology, and specifically to a method, apparatus, device, and medium for recognizing decoration change actions based on decoration semantic constraints. Background Technology

[0002] The decoration change action recognition method based on decoration semantic constraints is a technology used to automatically identify decoration changes during the decoration construction process.

[0003] According to statistics, the annual output value of my country's building decoration industry has exceeded 4 trillion RMB. Decoration projects are characterized by long construction cycles, numerous participants, and frequent changes. Therefore, the accuracy and efficiency of detecting changes in the quantity of work, i.e., decoration changes, directly affect the accurate payment of progress payments, the rational allocation of construction resources, and the effective control of project costs. Currently, the detection of changes in the quantity of decoration work mainly relies on the following method: using frame difference method to detect pixel changes in frames before and after construction to identify decoration changes during the construction process. However, when using this solution, the following technical problems often exist: Image acquisition at construction sites typically utilizes portable devices like smartphones, taken by supervisors or site representatives. However, the camera position, angle, and height are rarely perfectly consistent across shots. When the shooting position shifts by even a centimeter or the angle by several degrees, the coordinates of the same physical point in the pixel coordinate system shift accordingly between frames. Since the frame difference method performs pixel-by-pixel difference between two frames, it assumes that the two frames are precisely aligned in space, with the same physical point located at the same pixel coordinates in both frames. However, in actual construction scenarios, the handheld posture during two shots is unlikely to be exactly the same, failing to meet this alignment prerequisite. Even if there are no changes in the amount of work done at the construction site (e.g., no additions or removals of components within two weeks), the pixel position of the same wall area will still shift significantly between the two frames. The frame difference method will misinterpret this shift as a change in pixel value, thus incorrectly identifying it as a change in construction, resulting in poor accuracy in recognizing such changes.

[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not form prior art known to those skilled in the art. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure propose a method, apparatus, electronic device, and computer-readable medium for recognizing decoration change actions based on decoration semantic constraints, in order to solve one or more of the technical problems mentioned in the background section above.

[0007] In a first aspect, some embodiments of this disclosure provide a method for recognizing decoration change actions based on decoration semantic constraints. The method includes: acquiring a first temporal image of the same construction space collected during a first construction phase and a second temporal image collected during a second construction phase; performing feature-guided spatial alignment processing on the second temporal image based on the first temporal image to obtain an aligned second temporal image; inputting the first temporal image and the aligned second temporal image into a pre-trained dual-temporal pixel-level change detection Siamese network to obtain a pixel-level change probability map; performing semantic segmentation processing on the aligned second temporal image to obtain a semantic segmentation map; performing dual-threshold change probability filtering processing on the semantic segmentation map and the pixel-level change probability map to generate a decoration change mask map; performing connected component labeling processing on the decoration change mask map to obtain location information of each connected component; and performing decoration change action recognition processing on the image representing the location of the connected component location information in the first temporal image and the image representing the location of the connected component location information in the aligned second temporal image to obtain decoration action recognition information.

[0008] Secondly, some embodiments of this disclosure provide a decoration change action recognition device based on decoration semantic constraints. The device includes: an acquisition unit configured to acquire a first temporal image of the same construction space collected in a first construction phase and a second temporal image collected in a second construction phase; an alignment unit configured to perform feature-guided spatial alignment processing on the second temporal image based on the first temporal image to obtain an aligned second temporal image; an input unit configured to input the first temporal image and the aligned second temporal image into a pre-trained dual-temporal pixel-level change detection Siamese network to obtain a pixel-level change probability map; and a semantic segmentation unit configured to segment the aligned second temporal image... The image is subjected to semantic segmentation to obtain a semantic segmentation map; a dual-threshold change probability filtering unit is configured to perform dual-threshold change probability filtering on the semantic segmentation map and the pixel-level change probability map to generate a decoration change mask map; a connected component labeling unit is configured to perform connected component labeling on the decoration change mask map to obtain the location information of each connected component; a decoration change action recognition unit is configured to perform decoration change action recognition on the image of the location represented by the connected component location information in the first temporal image and the image of the location represented by the connected component location information in the second temporal image for each connected component location information to obtain decoration action recognition information.

[0009] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0010] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0011] The above embodiments of this disclosure have the following beneficial effects: the method for recognizing decoration change actions based on decoration semantic constraints according to some embodiments of this disclosure improves the accuracy of decoration change action recognition. Specifically, the reason for the poor accuracy of decoration change action recognition is that: image acquisition at decoration construction sites is usually done using portable devices such as ordinary mobile phones, and is taken by supervisors or site representatives. The camera position, angle, and height are difficult to be completely consistent each time. When the shooting position shifts by a centimeter or the angle shifts by several degrees, the coordinates of the same physical point in the pixel coordinate system in the preceding and following frames will shift accordingly. Since the frame difference method performs pixel-by-pixel difference between two frames, it assumes that the two frames are precisely aligned in space, and that the same physical point is located at the same pixel coordinates in both frames. However, in actual construction scenarios, the handheld postures in the two shots are unlikely to be completely identical, and this alignment premise cannot be met. Once there is a difference in the shooting perspective, even if there is no change in the amount of work at the construction site, i.e., no decoration changes (such as no addition or removal of any components in two weeks), the pixel position of the same wall area in the two frames will also be significantly offset. The frame difference method will misjudge this offset as a change in pixel value, and thus incorrectly identify it as a decoration change, resulting in poor accuracy in recognizing decoration change actions. Based on this, some embodiments of the decoration change action recognition method based on decoration semantic constraints of this disclosure first acquire a first temporal image of the same construction space acquired in the first construction stage and a second temporal image acquired in the second construction stage. Thus, two temporal images of the same construction space at different construction stages can be obtained. Then, based on the first temporal image, feature-guided spatial alignment processing is performed on the second temporal image to obtain an aligned second temporal image. This eliminates geometric distortion caused by inconsistencies in camera position, angle, and height between two shots, ensuring precise alignment of the preceding and following temporal images in the spatial coordinate system. Since the two images are spatially aligned, the same physical point resides at the same pixel coordinates in both frames, preventing pixel shifts caused by differences in shooting perspective from being misinterpreted as pixel value changes. Next, the first temporal image and the aligned second temporal image are input into a pre-trained dual-temporal pixel-level change detection Siamese network to obtain a pixel-level change probability map. Because the two input images are spatially aligned, the Siamese network can focus on the true structural changes at the semantic level, without needing to handle pixel coordinate shifts caused by perspective differences. Thus, it can accurately perceive the probability of a true decoration change at each pixel location in the precisely aligned image pair, obtaining a pixel-level change probability map. Then, semantic segmentation is performed on the aligned second temporal image to obtain a semantic segmentation map. This allows us to determine the decoration semantic category (e.g., wall, floor, door, window, background, etc.) for each pixel in the aligned second temporal image.Next, the semantic segmentation map and the pixel-level change probability map are subjected to dual-threshold change probability filtering to generate a decoration change mask map. This maintains high sensitivity within the decoration area to ensure that genuine structural changes are not missed, while effectively suppressing false changes such as worker movement and temporary facilities in non-decoration areas, thereby improving the accuracy of change detection. Then, the decoration change mask map is subjected to connected component labeling to obtain the location information of each connected component. This transforms discrete change points at the pixel level into independent change regions with engineering semantics. Then, for each connected component location information, decoration change action recognition processing is performed on the image representing the location of the connected component location information in the first temporal image and the image representing the location of the connected component location information in the aligned second temporal image to obtain decoration action recognition information. This allows for the automatic identification of decoration action recognition information for the three types of decoration actions: "addition," "reduction," and "replacement," corresponding to each valid connected component. It is precisely because the feature-guided spatial alignment process eliminates the geometric distortion introduced by the difference in shooting perspective that the pixel shift caused by the difference in shooting perspective is avoided from being misjudged as a change in pixel value. The Siamese network can accurately perceive the structural changes at the semantic level on precisely aligned image pairs. The synergistic effect of semantic segmentation and dual-threshold differential filtering further filters out the interference of pseudo-changes, thereby improving the accuracy of decoration change action recognition and obtaining more accurate decoration action recognition information. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0013] Figure 1 This is a flowchart of some embodiments of the decoration change action recognition method based on decoration semantic constraints according to the present disclosure; Figure 2 This is a structural schematic diagram of some embodiments of the decoration change action recognition device based on decoration semantic constraints according to the present disclosure; Figure 3 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0016] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0017] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0018] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0019] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] Figure 1 A flowchart 100 is shown, illustrating some embodiments of a decoration change action recognition method based on decoration semantic constraints according to this disclosure. This decoration change action recognition method based on decoration semantic constraints includes the following steps: Step 101: Obtain a first temporal image of the same construction space acquired during the first construction phase and a second temporal image acquired during the second construction phase.

[0021] In some embodiments, the execution subject (e.g., a computing device) of the decoration change action recognition method based on decoration semantic constraints can acquire, via wired or wireless connection, a first temporal image of the same construction space acquired during the first construction phase and a second temporal image acquired during the second construction phase from a preset image folder. The construction space refers to the physical space involved in the decoration construction. This physical space can be a room (e.g., bedroom, bathroom) or a corridor in a residential building, or an office area or public area in a commercial building. The same construction space means that the images are acquired at the same physical space during different construction phases. The construction phases refer to different time points in the decoration construction process. These time points can be the start and end times of various sub-projects stipulated in the decoration construction contract, such as the water and electricity renovation phase. Alternatively, the construction phases can be time points determined by the supervisor or site representative based on the actual construction progress, such as the first week of construction, the second week of construction, the fourth week of construction, etc. The first construction phase refers to a time point in the decoration construction process that precedes the second construction phase. The aforementioned first temporal image may be an image taken by the supervisor or site representative using a portable image acquisition device (e.g., mobile phone, digital camera, tablet computer, or dedicated engineering recorder) of the construction space after the decoration and construction work commences, at or before the start of the first construction phase. The aforementioned second temporal image may be an image taken by the supervisor or site representative using a portable image acquisition device (e.g., mobile phone, tablet computer, or dedicated engineering recorder) of the construction space after the decoration and construction work commences, at or before the start of the second construction phase.

[0022] It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other currently known or future wireless connection methods.

[0023] Step 102: Based on the first temporal image, perform feature-guided spatial alignment processing on the second temporal image to obtain an aligned second temporal image.

[0024] In some embodiments, the execution entity may perform feature-guided spatial alignment processing on the second temporal image based on the first temporal image to obtain an aligned second temporal image.

[0025] In addressing the technical problems mentioned above using the aforementioned technical solution (i.e., an image alignment scheme based on semantically perceptive feature point extraction), the following technical issues arise when aligning two-temporal images captured by portable devices such as ordinary mobile phones during the decoration and construction process: Image acquisition at decoration and construction sites typically involves portable devices like ordinary mobile phones, taken by supervisors or site representatives. The camera position, angle, and height are rarely perfectly consistent each time, leading to geometric distortion between consecutive frames. To eliminate this geometric distortion, existing solutions usually employ feature point matching-based image alignment methods. This involves extracting and matching feature points from two images, calculating the homography matrix based on the matched points, and then performing a projection transformation on the images to be aligned using the homography matrix. However, when using existing solutions to align two-temporal images in decoration and construction scenarios, the following technical problems often arise: Existing feature point detection algorithms (such as SIFT or SuperPoint) typically employ globally uniform feature point detection parameters (i.e., using the same feature point detection density and threshold across the entire image) to extract feature points. However, due to the significant differences in texture characteristics across different areas in a construction scene, a globally uniform threshold cannot simultaneously accommodate the needs of all textured regions. Solid-color wall areas have poor texture and generally low pixel response scores, resulting in very few pixels that can be identified as feature points under a globally uniform threshold. This leads to insufficient feature point extraction from wall areas, and the homography matrix calculation suffers from insufficient precision due to a lack of adequate constraints, resulting in a low accuracy of the determined homography matrix. Consequently, a large alignment error exists between the aligned second-temporal image and the first-temporal image. Simultaneously, floor tile areas have rich but highly repetitive textures and generally high pixel response scores, resulting in a large number of pixels that can be identified as feature points under a globally uniform threshold. This leads to the extraction of a large number of redundant feature points in floor tile areas, increasing computational load and wasting computational resources. The following characteristics are required for this application scenario: Construction site supervisors or representatives often lack professional image acquisition skills and typically use portable devices like smartphones to take photos during construction, making it difficult to guarantee consistent perspectives between shots. Therefore, the image alignment solution needs to be adaptable to different shooting perspectives and capable of calculating spatial transformation relationships from the image content itself. Furthermore, the texture characteristics of different areas in a construction site are complex and random. The image alignment solution needs to be adaptable to different texture regions, adaptively adjusting feature point detection parameters based on the texture characteristics of each region. In low-texture areas such as walls, the detection threshold should be lowered to extract sufficient feature points, ensuring sufficient constraints for homography matrix solving. Simultaneously, in high-texture, repetitive areas such as floor tiles, the detection threshold should be increased to reasonably control the number of feature points and avoid wasting computational resources due to redundant feature points. In some optional implementations of certain embodiments, the execution entity may perform feature-guided spatial alignment processing on the second temporal image based on the first temporal image to obtain an aligned second temporal image by performing the following steps: The first step involves semantic segmentation of the first temporal image to obtain a first semantic segmentation map. In practice, the execution entity can input the first temporal image into a pre-trained semantic segmentation network (e.g., DeepLabV3+ or SegFormer network) to obtain the first semantic segmentation map. The semantic segmentation network performs encoder-decoder processing on the input image: the encoder extracts multi-scale semantic features of the input image through convolutional and downsampling layers; the decoder restores the multi-scale semantic features to the spatial resolution of the input image through upsampling and feature fusion layers, and outputs a probability vector at each pixel location. The dimension of this probability vector is equal to the total number of preset semantic categories, and each element represents the probability that the pixel belongs to the corresponding semantic category. The execution entity takes the semantic category identifier corresponding to the maximum probability value in the probability vector at each pixel location as the semantic category identifier for that pixel location, thus obtaining the first semantic segmentation map. The value of each semantically segmented pixel in the semantic segmentation map is the semantic category identifier of the corresponding pixel. This semantic category identifier can be an identifier representing the semantic category. For example, semantic category identifier "0" corresponds to the semantic category "background", semantic category identifier "1" corresponds to the semantic category "wall", and semantic category identifier "2" corresponds to the semantic category "floor".

[0026] The second step involves dividing the first temporal image into multiple first detection region sub-images based on the first semantic segmentation map. Each first detection region sub-image has a corresponding semantic category identifier. In practice, the execution entity can traverse each pixel position in the first semantic segmentation map, obtain the semantic category identifier of that pixel position, and then group pixel positions with the same semantic category identifier in the first temporal image into a set, forming a set of pixel positions corresponding to that semantic category identifier. The multiple sets of pixel positions are then deduplicated. For each set of pixel positions in the deduplicated set, the execution entity can crop out the local image region corresponding to that set of pixel positions in the first temporal image, using it as the first detection region sub-image corresponding to that semantic category identifier. Each first detection region sub-image is associated with a corresponding semantic category identifier. For example, based on the first semantic segmentation map, the execution entity extracts all pixel regions in the first temporal image whose semantic category identifier represents "wall," forming a wall detection region sub-image; and extracts all pixel regions in the first temporal image whose semantic category identifier represents "doorway," forming a first detection region sub-image.

[0027] Third, for each of the multiple first detection region sub-images mentioned above, perform the following semantic-aware feature point extraction process: The first sub-step involves determining the semantic category identifier corresponding to the first detection region sub-graph as the semantic category identifier to be queried.

[0028] The second sub-step involves querying the feature point detection density and feature point detection threshold corresponding to the semantic category identifier to be queried. Since image regions of different semantic categories have different texture characteristics, pixels in texture-poor regions (such as solid-color walls) generally have low response scores, requiring a lower detection threshold to extract sufficient feature points. Conversely, pixels in texture-rich but highly repetitive regions (such as tiled floors) generally have high response scores, but if the detection threshold is too low, a large number of redundant feature points that easily lead to ambiguous matching will be extracted. Therefore, a higher detection threshold is needed to extract only the feature points with the strongest response. In practice, the execution entity can query a preset mapping table between semantic category identifiers and feature point detection parameters to obtain the feature point detection density and feature point detection threshold corresponding to the semantic category identifier to be queried. Each mapping relationship in the mapping table records the feature point detection density and feature point detection threshold corresponding to each semantic category identifier. The feature point detection density refers to the number of feature points expected to be extracted per unit area. The feature point detection threshold is a response score threshold used to filter feature points; only pixel positions with a response score greater than or equal to this threshold are determined to be feature points. The third sub-step involves performing feature point detection processing on the first detection region sub-image according to the aforementioned feature point detection density and feature point detection threshold, to obtain at least one first feature point information. Here, the at least one first feature point information refers to the feature point information corresponding to each feature point extracted from the first detection region sub-image. Feature point information refers to structured data describing a feature point, including at least the feature point's position coordinates in the image, and may also include the feature point's scale, orientation, and feature descriptor. In practice, the execution entity can perform feature point detection processing on the first detection region sub-image using a feature point detection algorithm (e.g., Scale Invariant Feature Transform (SIFT) or SuperPoint deep feature algorithm) according to the aforementioned feature point detection density and feature point detection threshold. Specifically, a response score (i.e., response value) can be calculated for each pixel in the first detection region sub-image. Pixels with response scores greater than or equal to the aforementioned feature point detection threshold are identified as candidate feature points. From these candidate feature points, feature points are selected in descending order of response score, with a number not exceeding the aforementioned feature point detection density. The position coordinates, scale, orientation, and feature descriptor of each feature point are then extracted as the first feature point information.

[0029] The fourth step is to determine the obtained information of each first feature point as the first feature point information set.

[0030] The fifth step involves semantic segmentation of the second temporal image to obtain a second semantic segmentation map. In practice, the executing entity can input the second temporal image into a pre-trained semantic segmentation network (e.g., a DeepLabV3+ network or a SegFormer network) to obtain the second semantic segmentation map.

[0031] Step 6: Based on the second semantic segmentation map, the second temporal image is divided into multiple second detection region sub-images, where each second detection region sub-image has a corresponding semantic category label. In practice, the executing entity can use the same method as in step 2 to divide the second temporal image into multiple second detection region sub-images based on the second semantic segmentation map. Here, the second detection region sub-image refers to a local image cropped from the second temporal image that corresponds to a semantic category label.

[0032] Step 7: Perform semantic-aware feature point extraction processing on each of the multiple second detection region sub-images to obtain at least one second feature point. In practice, the execution entity can use the same method as in Step 3 to perform semantic-aware feature point extraction processing on each of the multiple second detection region sub-images to obtain at least one second feature point.

[0033] The eighth step is to determine the obtained information of each second feature point as the second feature point information set.

[0034] Step nine involves performing feature matching processing on the aforementioned first feature point information set and the aforementioned second feature point information set to obtain a set of feature point information matching pairs. In practice, for each first feature point information in the aforementioned first feature point information set, the feature descriptors included in the first feature point information can be extracted as first feature descriptors. Then, the matching similarity between the first feature descriptors and the feature descriptors included in each second feature point information in the aforementioned second feature point information set is determined. The matching similarity can be represented by Euclidean distance or cosine similarity. Then, the executing entity can determine the matching similarity with the highest similarity among the determined matching similarities as the target matching similarity. Afterwards, the executing entity can determine the second feature point information corresponding to the target matching similarity as the target second feature point information. Then, the executing entity can determine the aforementioned first feature point information and the aforementioned target second feature point information as feature point information matching pairs. Finally, the executing entity can determine the determined feature point information matching pairs as a set of feature point information matching pairs.

[0035] Step 10: Based on the aforementioned set of feature point information matching pairs, generate a homography matrix between the first temporal image and the second temporal image. This homography matrix is ​​a 3×3 projection transformation matrix used to describe the coordinate mapping relationship of a point on a plane between images from two different viewpoints. In practice, the executing entity uses the Random Sample Consensus Algorithm (RANSAC) to generate the homography matrix based on the aforementioned set of feature point information matching pairs. Specifically, the executing entity randomly selects four non-collinear feature point information matching pairs from the aforementioned set and uses these four pairs to solve for an initial homography matrix. Then, it uses this initial homography matrix to verify all matching pairs in the aforementioned set of feature point information matching pairs, counting the number of matching pairs that satisfy the mapping relationship of the homography matrix (i.e., the number of inliers). The criterion for satisfying the mapping relationship is: after mapping the position coordinates of the first feature point information through the homography matrix, the distance between the position coordinates of the corresponding second feature point information and the first feature point information is less than a preset distance threshold (e.g., 3 pixels). The aforementioned executing entity repeats the above random sampling, solution, and verification process multiple times (e.g., 1000 times) and selects the initial homography matrix with the largest number of interior points as the homography matrix.

[0036] Step 11: Based on the homography matrix described above, the second temporal image is projected onto the coordinate system of the first temporal image to obtain an aligned second temporal image. The coordinate system refers to the spatial reference system for pixel positions in the image, with the top-left corner of the image as the origin, the horizontal direction to the right as the positive x-axis, and the vertical direction downwards as the positive y-axis. The position of each pixel is determined by its coordinate values ​​on the x-axis and y-axis. The aligned second temporal image refers to the image obtained by projecting the second temporal image according to the homography matrix, which is aligned with the first temporal image in the spatial coordinate system. In practice, for the current pixel in the second temporal image, the executing entity substitutes the pixel's x-coordinate and y-coordinate into the mapping relationship described by the homography matrix, and through matrix multiplication, obtains the target x-coordinate and target y-coordinate of the pixel in the coordinate system of the first temporal image. The executing entity then assigns the pixel value (including red channel value, green channel value, and blue channel value) of the current pixel in the second temporal image to the target position. The aforementioned execution entity performs the above operation on each pixel in the second temporal image, rearranging the pixels in the second temporal image according to the spatial transformation relationship described by the homography matrix, to form a preliminary projected image. Optionally, for unfilled pixel positions (i.e., hole positions) in the projected image, the aforementioned execution entity uses bilinear interpolation to fill them.

[0037] The above-described technical solution and its related content, as an inventive point of this disclosure, solve the technical problem of "a large alignment error and wasted computing resources between the aligned second temporal image and the first temporal image". The factors leading to a large alignment error and wasted computing resources between the aligned second temporal image and the first temporal image are often as follows: Existing feature point detection algorithms (such as the SIFT algorithm or the SuperPoint algorithm) typically use globally uniform feature point detection parameters (i.e., using the same feature point detection density and the same feature point detection threshold for the entire image) to extract feature points. Due to the significant differences in texture characteristics between different areas in a construction scene, a globally uniform feature point detection threshold cannot simultaneously meet the needs of all textured areas. Solid-color wall areas have poor texture and generally low pixel response scores. Under a globally uniform threshold, the number of pixels that can be identified as feature points is extremely small, resulting in insufficient feature points being extracted from the wall area. The calculation of the homography matrix lacks sufficient constraints, leading to insufficient accuracy and thus a low precision in the determined homography matrix. This, in turn, results in a large alignment error between the aligned second temporal image and the first temporal image. Meanwhile, the texture of the tiled surface area is rich but highly repetitive, and the pixel response scores are generally high. Under a globally uniform threshold, the number of pixels that can be identified as feature points is extremely large, leading to the extraction of a large number of redundant feature points in the tiled area, increasing computational load and wasting computational resources. Solving these factors can reduce the alignment error between the aligned second-temporal image and the first-temporal image, and also reduce the waste of computational resources. To achieve this, firstly, semantic segmentation is performed on the first-temporal image to obtain a first semantic segmentation map. Each pixel in the first-temporal image can then be labeled with a corresponding semantic category identifier, resulting in the first semantic segmentation map. Then, based on the first semantic segmentation map, the first-temporal image is divided into multiple first detection region sub-images, each with a corresponding semantic category identifier. This allows regions with different texture characteristics (such as solid-color wall areas and tiled surface areas) to be separated, resulting in multiple first detection region sub-images, making the texture characteristics within each first detection region sub-image relatively consistent. Subsequently, for each of the aforementioned first detection region sub-images, the following semantic-aware feature point extraction processing is performed: First, the semantic category identifier corresponding to the aforementioned first detection region sub-image is determined as the semantic category identifier to be queried. Second, the feature point detection density and feature point detection threshold corresponding to the aforementioned semantic category identifier to be queried are queried. Third, according to the aforementioned feature point detection density and the aforementioned feature point detection threshold, feature point detection processing is performed on the aforementioned first detection region sub-image to obtain at least one first feature point.Therefore, the feature point detection parameters can be adaptively adjusted according to the texture characteristics of each first detection region sub-image (represented by its corresponding semantic category identifier), allowing different feature point detection parameters to be used for feature point extraction in regions with different texture characteristics. For texture-poor wall areas, a lower detection threshold is used, allowing pixels with lower response scores to be identified as feature points, thereby increasing the number of feature points extracted from the wall area and alleviating the decrease in homography matrix accuracy due to insufficient feature points. For texture-rich but highly repetitive floor tile areas, a higher detection threshold is used, ensuring that only pixels with higher response scores are identified as feature points, thereby reducing the number of redundant feature points extracted from the floor tile area, reducing the computational load of feature point extraction and feature matching, and alleviating the waste of computational resources caused by redundant feature points. Then, the obtained first feature point information is determined as the first feature point information set. Next, semantic segmentation processing is performed on the aforementioned second temporal image to obtain the second semantic segmentation map. Thus, each pixel in the aforementioned second temporal image can be labeled with its corresponding semantic category identifier. Furthermore, based on the aforementioned second semantic segmentation map, the aforementioned second temporal image is divided into multiple second detection region sub-images, each of which has a corresponding semantic category identifier. Thus, the aforementioned second temporal image can be divided into different detection region sub-images according to texture characteristics. Next, semantic-aware feature point extraction processing is performed on each of the multiple second detection region sub-images to obtain at least one second feature point. Therefore, regions with different texture characteristics in the aforementioned second temporal image are also subjected to feature point extraction using their respective corresponding feature point detection parameters, ensuring that sufficient feature points are extracted from the wall region of the aforementioned second temporal image, while avoiding the extraction of a large number of redundant feature points from the floor tile region. Then, the obtained second feature point information is determined as a second feature point information set. Subsequently, feature matching processing is performed on the aforementioned first feature point information set and the aforementioned second feature point information set to obtain a feature point information matching pair set. Thus, a feature point correspondence can be established between the first temporal image and the second temporal image. Since the number of feature points in both the first and second feature point information sets is within a reasonable range across different regions, the feature matching process can obtain a moderate number of reliable feature point information matching pairs. Then, based on the aforementioned set of feature point information matching pairs, a homography matrix is ​​generated between the first and second temporal images. Finally, according to the homography matrix, the second temporal image is projected onto the coordinate system of the first temporal image to obtain an aligned second temporal image. Thus, an aligned second temporal image that is precisely aligned with the first temporal image can be obtained.Since the homography matrix is ​​generated based on a moderate number of reliable feature point matching pairs, the wall region provides sufficient matching pairs as constraints for solving the homography matrix, while the floor tile region reduces the interference of ambiguous matching on the accuracy of the homography matrix. Therefore, the accuracy of the generated homography matrix is ​​improved, thereby reducing the alignment error between the aligned second-temporal image and the first-temporal image. Simultaneously, the reduced number of feature points extracted from the floor tile region lowers the computational cost of feature point extraction and feature matching, reducing the waste of computing resources. Furthermore, the use of semantic segmentation-based differential feature point detection (i.e., using a low detection threshold for the wall region to increase the number of feature points and a high detection threshold for the floor tile region to reduce redundant feature points) achieves balanced feature point extraction from different texture regions. This ensures that sufficient feature points are extracted from the wall region, guaranteeing sufficient constraints for solving the homography matrix. At the same time, the reduced number of feature points extracted from the floor tile region lowers the computational cost, thus reducing the alignment error between the aligned second-temporal image and the first-temporal image, and minimizing the waste of computing resources.

[0038] Step 103: Input the first temporal image and the aligned second temporal image into the pre-trained dual-temporal pixel-level change detection Siamese network to obtain a pixel-level change probability map.

[0039] In some embodiments, the execution entity can input the first temporal image and the aligned second temporal image into a pre-trained dual-temporal pixel-level change detection Siamese network to obtain a pixel-level change probability map. The dual-temporal pixel-level change detection Siamese network refers to a deep learning network model based on a Siamese network architecture, used to perform pixel-by-pixel change detection on two temporal images to obtain a pixel-level change probability map. The pixel-level change probability map refers to an image with the same spatial resolution as the input image, where the value at each pixel position represents the probability of a real structural change occurring at that pixel position.

[0040] In some optional implementations of certain embodiments, the execution entity may input the first temporal image and the aligned second temporal image into a pre-trained dual-temporal pixel-level change detection Siamese network through the following steps to obtain a pixel-level change probability map: The first step involves inputting the aforementioned first temporal image into the first feature extraction branch of a pre-trained dual-temporal pixel-level change detection Siamese network. After multi-scale convolutional feature extraction processing by this first feature extraction branch, a first temporal semantic feature map set is obtained. The dual-temporal pixel-level change detection Siamese network includes the first feature extraction branch, a second feature extraction branch, and a pixel-level change prediction module. The first feature extraction branch can employ a multi-scale feature extraction module to perform multi-scale feature extraction processing with the first temporal image as input and the first temporal semantic feature map set as output. For example, the first feature extraction branch can use a ResNet-50 network or a VGG-16 network with fully connected layers removed as the backbone network. The first temporal semantic feature map set refers to a collection of semantic feature maps extracted from the first temporal image, containing multiple semantic feature maps at different spatial scales. Each semantic feature map covers the complete spatial range of the first temporal image. Each semantic feature map is a three-dimensional tensor (i.e., a data array with three dimensions: height, width, and number of channels), where height and width constitute the spatial dimension, and the number of channels constitutes the feature dimension. The semantic feature map mentioned above preserves the spatial structure information of each pixel position in the input image. That is, the position with coordinates (h, w) on the feature map corresponds to the feature vector of a local region in the input image. For example, the position with coordinates (h, w) on the feature map (i.e., the position in the h-th row and w-th column) corresponds to the feature vector of a local region in the input image.

[0041] The second step involves inputting the aligned second temporal image into the second feature extraction branch of the Siamese network. After multi-scale convolutional feature extraction processing by the second feature extraction branch, a second temporal semantic feature map set is obtained. The first and second feature extraction branches share network weights, and each second temporal semantic feature map in the second temporal semantic feature map set is associated with each first temporal semantic feature map in the first temporal semantic feature map set. Figure 1One-to-one correspondence. The second feature extraction branch has the exact same network structure as the first feature extraction branch. This second feature extraction branch can be implemented using a multi-scale feature extraction module, taking the aligned second temporal image as input and the second temporal semantic feature map set as output. The shared network weights refer to the fact that the network parameters of the first and second feature extraction branches are exactly the same, i.e., the convolutional kernel weights and bias parameters of the two branches are completely identical. The second temporal semantic feature map set refers to the set of semantic feature maps extracted from the aligned second temporal image, containing multiple semantic feature maps of different spatial scales, where each semantic feature map covers the complete spatial range of the aligned second temporal image. The one-to-one correspondence means that each semantic feature map in the second temporal semantic feature map set corresponds one-to-one with each semantic feature map in the first temporal semantic feature map set in terms of spatial scale. For example, when the first feature extraction branch outputs first temporal semantic feature maps at scales of 1 / 4, 1 / 8, and 1 / 16, since the second feature extraction branch shares network weights with the first feature extraction branch, it also outputs second temporal semantic feature maps at scales of 1 / 4, 1 / 8, and 1 / 16. The 1 / 4-scale second temporal semantic feature map corresponds to the 1 / 4-scale first temporal semantic feature map, the 1 / 8-scale second temporal semantic feature map corresponds to the 1 / 8-scale first temporal semantic feature map, and the 1 / 16-scale second temporal semantic feature map corresponds to the 1 / 16-scale first temporal semantic feature map. The executing entity then combines these second temporal semantic feature maps at each scale to form the aforementioned set of second temporal semantic feature maps.

[0042] The third step involves performing pixel-by-pixel difference detection on the corresponding first-temporal semantic feature maps and second-temporal semantic feature maps of the same scale within the first and second temporal semantic feature map sets, resulting in an initial change heatmap. Pixel-by-pixel difference detection refers to the process of calculating the difference between the feature vectors of two semantic feature maps with the same spatial scale at each spatial location (i.e., each height and width coordinate position). The initial change heatmap is a response map reflecting the degree of change at each spatial location, obtained after pixel-by-pixel difference detection of the semantic feature maps; its spatial resolution is the same as that of the detected semantic feature maps. In practice, the execution entity performs pixel-by-pixel difference detection on each pair of corresponding semantic feature maps. For a 1 / 4 scale feature map pair, the execution entity takes a 1 / 4 scale first-temporal semantic feature map from the first temporal semantic feature map set and a 1 / 4 scale second-temporal semantic feature map from the second temporal semantic feature map set. At each spatial location (h, w) in the feature maps, it calculates the difference between the feature vector at that location in the first temporal semantic feature map and the feature vector at that location in the second temporal semantic feature map. The difference can be calculated using either the absolute value of the difference (i.e., the sum or average of the absolute values ​​of the element-wise differences between the two feature vectors) or cosine similarity (i.e., the cosine similarity between the two feature vectors is calculated, and then 1 is subtracted from the similarity to obtain the difference value). After the above calculation, the execution entity obtains an initial change heatmap at the 1 / 4 scale. Similarly, for 1 / 8 scale and 1 / 16 scale feature map pairs, the execution entity performs pixel-wise difference detection in the same way to obtain initial change heatmaps at the 1 / 8 scale and 1 / 16 scale, respectively.

[0043] The fourth step involves feature fusion processing of the obtained initial change heatmaps to obtain a fused change heatmap. In practice, the execution entity can upsample the initial change heatmaps at each scale to the same spatial resolution (e.g., upsample the 1 / 8 and 1 / 16 scale initial change heatmaps to a 1 / 4 scale spatial resolution), and then add the upsampled initial change heatmaps element-wise to obtain the fused change heatmap. The upsampling operation can be implemented using bilinear interpolation. Through multi-scale fusion, the fused change heatmap balances the sensitivity of high-scale (e.g., 1 / 4 scale) feature maps to detailed changes with the robustness of low-scale (e.g., 1 / 16 scale) feature maps to global changes. The fused change heatmap refers to a single change heatmap that integrates multi-scale change information, obtained after feature fusion processing of the initial change heatmaps at each scale.

[0044] Fifth, the fused change heatmap is input into the pixel-level change prediction module to obtain a pixel-level change probability map. This module maps the fused change heatmap to pixel-level change probabilities. It includes at least one convolutional layer and a sigmoid activation layer. The convolutional layer performs further feature extraction and channel transformation on the fused change heatmap, converting it from a multi-channel feature map to a single-channel response map. The sigmoid activation layer maps the value at each spatial location in the single-channel response map output by the convolutional layer to a value between 0 and 1, resulting in the pixel-level change probability map. The spatial resolution of the pixel-level change probability map is the same as that of the fused change heatmap, and the value at each pixel location represents the probability of a real structural change occurring at that location.

[0045] Step 104: Perform semantic segmentation on the aligned second temporal image to obtain a semantic segmentation map.

[0046] In some embodiments, the execution entity can perform semantic segmentation processing on the aligned second temporal image to obtain a semantic segmentation map. In practice, the execution entity can input the aligned second temporal image into a pre-trained semantic segmentation network (e.g., a DeepLabV3+ network or a SegFormer network). This semantic segmentation network performs encoder-decoder processing on the input image, outputting a probability vector for each pixel location. Then, it takes the semantic category identifier corresponding to the highest probability value in the probability vector as the semantic category identifier for that pixel location, thereby obtaining the semantic segmentation map. The pre-trained semantic segmentation network refers to a semantic segmentation network whose parameters have been optimized using a large number of training images labeled with semantic category identifiers.

[0047] In some optional implementations of certain embodiments, the aforementioned execution entity may perform semantic segmentation processing on the aforementioned aligned second temporal image through the following steps to obtain a semantic segmentation map: The first step involves inputting the aligned second temporal image into the encoder of a pre-trained semantic segmentation network. The encoder then performs multi-scale feature extraction on the aligned second temporal image to obtain a multi-scale semantic feature map. This multi-scale semantic feature map refers to the set of semantic feature maps extracted from the input aligned second temporal image, containing multiple semantic feature maps at different spatial scales.

[0048] The second step involves inputting the aforementioned multi-scale semantic feature map into the decoder to perform feature fusion and upsampling processing, resulting in a pixel-level category probability map. The spatial resolution of this pixel-level category probability map is the same as that of the aligned second temporal image. Each pixel in the pixel-level category probability map has a probability vector at its pixel location. Each probability element in the probability vector represents the probability that the pixel location belongs to a semantic category identifier. The pixel-level category probability map refers to a three-dimensional tensor with the same spatial resolution as the input image, its height and width are the same as the input image, and the number of channels equals the preset total number of semantic categories. The probability vector refers to the channel dimension vector at each pixel location in the pixel-level category probability map, i.e., the vector formed by the values ​​of that location across all channels. The length of the probability vector equals the preset total number of semantic categories, and the value of each element is between 0 and 1, with the sum of all elements being 1. Each element represents the probability that the pixel location belongs to the semantic category represented by the corresponding semantic category identifier. In practice, the decoder uses the aforementioned multi-scale semantic feature map as input. The decoder first upsamples the low-scale semantic feature map to a high scale, then concatenates or adds the upsampled feature map to the high-scale semantic feature map element-wise to achieve feature fusion. The decoder repeats the upsampling and fusion operations to gradually restore the spatial resolution of the feature map. Finally, the decoder maps the fused feature map through a convolutional layer to a pixel-level category probability map with the same spatial resolution as the aligned second temporal image. For example, when the preset semantic categories include three categories: semantic category identifier "0" corresponding to the "background" semantic category, semantic category identifier "1" corresponding to the "wall" semantic category, and semantic category identifier "2" corresponding to the "floor" semantic category, for the pixel position at coordinates (100, 200) in the pixel-level category probability map, the corresponding probability vector is [0.05, 0.8, 0.15], indicating that the probability of this pixel position belonging to the "background" semantic category is 5%, the probability of belonging to the "wall" semantic category is 80%, and the probability of belonging to the "floor" semantic category is 15%.

[0049] Third, for each pixel in the pixel-level category probability map above, perform the following steps: The first sub-step involves determining the probability vector at the pixel location of the aforementioned pixel as the target probability vector. In practice, the execution entity obtains the channel dimension vector (i.e., the values ​​of all channels at that location) of the current pixel location in the aforementioned pixel-level category probability map, and determines this vector as the aforementioned target probability vector.

[0050] The second sub-step involves determining the semantic category identifier corresponding to the probability element with the highest probability value in the target probability vector as the semantic category identifier corresponding to the pixel.

[0051] The fourth step involves combining the semantic category identifiers corresponding to each pixel according to their spatial positions in the pixel-level category probability map to generate a semantic segmentation map. The spatial resolution of this semantic segmentation map is the same as that of the aligned second temporal image, and the value of each semantic segmentation pixel in the semantic segmentation map is its corresponding semantic category identifier. Specifically, the semantic segmentation map is a two-dimensional matrix composed of the semantic category identifiers at each pixel position, with the same spatial resolution as the aligned second temporal image. Each element in the matrix is ​​the semantic category identifier of its corresponding pixel position. In practice, the executing entity fills the corresponding two-dimensional matrix position with the semantic category identifiers obtained from each pixel position in the pixel-level category probability map, according to the spatial coordinates of that pixel position, to form the semantic segmentation map. The semantic segmentation map is a two-dimensional matrix, where each position has an integer value (i.e., a semantic category identifier) ​​representing the category of the corresponding decorative component. For example, a region with a value of 1 in the semantic segmentation map is classified as "wall," a region with a value of 2 is classified as "floor," and a region with a value of 3 is classified as "door."

[0052] Step 105: Perform dual-threshold change probability filtering on the semantic segmentation map and the pixel-level change probability map to generate a decoration change mask map.

[0053] In some embodiments, the execution entity may perform dual-threshold change probability filtering on the semantic segmentation map and the pixel-level change probability map to generate a decoration change mask map. Each pixel-level change probability pixel in the pixel-level change probability map corresponds one-to-one with each semantic segmentation pixel in the semantic segmentation map, and the pixel value of each pixel-level change probability pixel is a change probability value.

[0054] In addressing the technical problems described above using the aforementioned technical solutions, the following technical issues arise when considering the application scenario: generating a decoration change mask by detecting pixel-by-pixel changes in temporal images before and after decoration construction. While the pixel-level change probability map output by the aforementioned dual-temporal pixel-level change detection Siamese network provides the probability value of each pixel's position changing, this probability value itself cannot be directly used as the final judgment result. This is because non-decoration factors such as construction worker movement, temporary facility placement, and lighting changes can also generate high probability responses in the pixel-level change probability map. If a single global threshold is used to binarize the aforementioned pixel-level change probability map (i.e., all pixels are judged using the same threshold), the following dilemma arises: if the threshold is lowered, while the actual changes in the decoration area (such as newly added doors and windows) can be detected, interference such as construction worker movement and lighting changes will also be falsely reported as decoration changes, resulting in poor accuracy of the decoration change mask. Increasing the threshold can filter out some interference, but it will also miss real changes with relatively low probability responses in the decoration area (such as changes in small components like door frame edges and baseboards), resulting in poor accuracy of the decoration change mask. The following characteristics are required for this application scenario: Decoration quantity change detection needs to ensure complete detection of real structural changes while effectively suppressing interference from non-decoration factors such as construction workers, temporary facilities, and lighting changes. In other words, it needs to simultaneously meet the requirements of "high sensitivity" and "high robustness" so that subsequent connected component analysis and decoration change action recognition can be performed based on an accurate decoration change mask.

[0055] In some optional implementations of certain embodiments, the execution entity may perform dual-threshold change probability filtering on the semantic segmentation map and the pixel-level change probability map through the following steps to generate a decoration change mask map: The first step, for each semantic segmentation pixel in the semantic segmentation map above, is to perform the following steps: The first sub-step involves determining the pixel values ​​of the aforementioned semantic segmentation pixels as the semantic category identifiers to be classified.

[0056] The second sub-step involves, in response to determining that the aforementioned semantic category identifier to be classified is a decoration semantic category identifier from a preset set of decoration semantic category identifiers, setting the pixel value of the aforementioned semantic segmentation pixel to a first preset value to update the semantic segmentation pixel. Here, the aforementioned set of decoration semantic category identifiers refers to a collection of all decoration-related semantic category identifiers, which in this application includes semantic category identifiers for decoration components such as "walls," "floors," "doors," "windows," "ceilings," and "baseboards." The aforementioned first preset value is a value used to identify "belonging to the decoration semantic region," and in this application, it is 1.

[0057] The third sub-step involves determining that the semantic category identifier to be classified is not any of the preset decoration semantic category identifiers in the set of decoration semantic category identifiers, and then setting the pixel value of the semantic segmentation pixel to a second preset value to update the semantic segmentation pixel. In practice, if the semantic category identifier to be classified is not any of the decoration semantic category identifiers in the set of decoration semantic category identifiers (i.e., the pixel position belongs to non-decoration semantic categories such as "background" or "construction workers"), then the pixel value of the current semantic segmentation pixel is updated to the second preset value (i.e., 0).

[0058] The second step involves updating the semantic segmentation map for each semantic segmentation pixel and defining it as a decoration region mask. Each pixel in the decoration region mask corresponds one-to-one with each pixel-level change probability pixel in the pixel-level change probability map. The decoration region mask is a binary mask image generated from the semantic segmentation map to identify which pixel locations in the image belong to the decoration semantic region. In the decoration region mask, a pixel location with a value of the first preset value (i.e., 1) indicates that the location belongs to the decoration semantic region, while a pixel location with a value of the second preset value (i.e., 0) indicates that the location does not belong to the decoration semantic region.

[0059] Third, for each pixel with a change probability in the pixel-level change probability map above, perform the following steps: The first sub-step is to set the pixel-level change probability values ​​of the above-mentioned pixels to the target change probability values.

[0060] The second sub-step is to determine the pixel value corresponding to the pixel-level change probability pixel in the above-mentioned decoration area mask image as the target pixel value.

[0061] The third sub-step, in response to determining that the target pixel value is a first preset value, compares the target change probability value with a preset lower probability threshold, and performs the following steps: First, in response to determining that the target change probability value is less than or equal to a preset lower probability threshold, the pixel value corresponding to the pixel-level change probability pixel is updated to a second preset value to update the pixel-level change probability pixel. Here, the first detection threshold is a change probability determination threshold applied to the decoration semantic region, which in this application can be 0.4, a relatively low value to ensure sensitivity in detecting real structural changes within the decoration region. In practice, if the target pixel value is the first preset value (i.e., 1), it indicates that the current pixel position belongs to the decoration semantic region, and the executing entity compares the target change probability value with the first detection threshold. In practice, if the target change probability value is less than the first detection threshold, it indicates that the change probability of the current pixel position does not meet the detection standard of the decoration semantic region, and the pixel position is determined to be unchanged. The executing entity updates the pixel value of the pixel position in the pixel-level change probability map to the second preset value (i.e., 0).

[0062] Second, in response to determining that the target change probability value is greater than a preset lower probability threshold, the pixel value corresponding to the pixel-level change probability pixel is updated to a first preset value to update the pixel-level change probability pixel. In practice, if the target change probability value is greater than or equal to the first detection threshold, it indicates that the change probability of the current pixel position reaches the detection standard of the decoration semantic region, and the pixel position is determined to have undergone a real decoration structure change. The execution entity updates the pixel value of the pixel position in the pixel-level change probability map to the first preset value (i.e., 1).

[0063] The fourth sub-step, in response to determining that the above target pixel value is a second preset value, performs the following steps: The second detection threshold mentioned above is a threshold for determining the change probability in non-decoration semantic regions. In this application, it can be 0.8, which is relatively high to suppress false changes in non-decoration regions (such as construction workers, temporary facilities, etc.). In practice, if the target pixel value is the second preset value (i.e., 0), it means that the current pixel position does not belong to the decoration semantic region. Then, the execution entity compares the target change probability value with the second detection threshold.

[0064] Step one: In response to determining that the target change probability value is less than or equal to a preset upper probability threshold, the pixel value corresponding to the pixel-level change probability pixel is updated to a second preset value to update the pixel-level change probability pixel. In practice, if the target change probability value is less than the second detection threshold, it means that the change probability of the current pixel position does not meet the judgment standard for non-decoration semantic regions, and the pixel position is judged as unchanged or masked. The executing entity updates the pixel value of the pixel position in the pixel-level change probability map to the second preset value (i.e., 0).

[0065] Step two: In response to determining that the target change probability value is greater than a preset upper probability threshold, the pixel value corresponding to the pixel-level change probability pixel is updated to a first preset value to update the pixel-level change probability pixel. In practice, if the target change probability value is greater than or equal to the second detection threshold, it indicates that the change probability of the current pixel position reaches the judgment standard for non-decoration semantic regions, and the pixel position is judged to have changed. The execution entity updates the pixel value of the pixel position in the pixel-level change probability map to the first preset value (i.e., 1).

[0066] The fourth step is to determine the updated pixel-level change probability map as the decoration change mask map. This decoration change mask map refers to the binary mask image obtained by applying the aforementioned dual-threshold change probability filtering process to the pixel-level change probability map, used to identify which pixel locations in the image are determined to have undergone actual decoration structural changes. In the decoration change mask map, a pixel location with a value of the first preset value (i.e., 1) indicates that the location has been determined to have undergone actual decoration structural changes, while a pixel location with a value of the second preset value (i.e., 0) indicates that the location has been determined to have no change or is masked.

[0067] Step 106: Perform connected component labeling on the decoration change mask to obtain the location information of each connected component.

[0068] In some embodiments, the execution entity can perform connected component labeling on the aforementioned decoration change mask to obtain the location information of each connected component. In practice, the execution entity can use a seed filling algorithm (e.g., seed filling method) to perform connected component labeling on the aforementioned decoration change mask to obtain the location information of each connected component. Each connected component location information can describe the position of a valid connected component in the aforementioned decoration change mask, and can be a set of coordinates including all pixels contained in that connected component.

[0069] The above-described technical solution and related content, as an inventive point of this disclosure, solve the technical problem of "poor accuracy of the decoration change mask". Factors leading to poor accuracy of the decoration change mask often include: Although the pixel-level change probability map output by the aforementioned dual-temporal pixel-level change detection twin network provides the probability value of a change in each pixel position, this probability value itself cannot be directly used as the final judgment result, because non-decoration factors such as construction worker movement, temporary facility placement, and lighting changes will also produce high probability responses in the pixel-level change probability map. If a single global threshold is used to binarize the aforementioned pixel-level change probability map (i.e., all pixels are judged using the same threshold), the following dilemma arises: If the threshold is lowered, although the actual changes in the decoration area (such as newly added doors and windows) can be detected, interference such as construction worker movement and lighting changes will also be falsely reported as decoration changes, resulting in poor accuracy of the decoration change mask. Increasing the threshold can filter out some interference, but it also misses real changes with relatively low probability of change in the decoration area (such as changes in small components like door frame edges and baseboards), resulting in poor accuracy of the decoration change mask. Addressing these factors can improve the accuracy of the decoration change mask. To achieve this, firstly, for each semantic segmentation pixel in the semantic segmentation image, the following steps are performed: The pixel value of the semantic segmentation pixel is determined as the semantic category identifier to be classified. Then, in response to determining that the semantic category identifier to be classified is a decoration semantic category identifier from a preset set of decoration semantic category identifiers, the pixel value of the semantic segmentation pixel is determined as a first preset value to update the semantic segmentation pixel. Thus, the pixel values ​​of pixels belonging to decoration semantics are determined as the first preset value. Next, in response to determining that the semantic category identifier to be classified is not any decoration semantic category identifier from the preset set of decoration semantic category identifiers, the pixel value of the semantic segmentation pixel is determined as a second preset value to update the semantic segmentation pixel. Thus, the pixel values ​​of pixels not belonging to decoration semantics are determined as the second preset value. Next, the semantic segmentation map updated by each semantic segmentation pixel is determined as a decoration region mask map, wherein each pixel in the decoration region mask map corresponds one-to-one with each pixel-level change probability pixel in the pixel-level change probability map. This yields a binary mask image, where pixel positions with values ​​of a first preset value represent pixels belonging to the decoration semantics, and pixel positions with values ​​of a second preset value represent pixels not belonging to the decoration semantics. Then, for each pixel-level change probability pixel in the pixel-level change probability map, the following steps are performed: First sub-step: Set the pixel value of the pixel-level change probability pixel to the target change probability value. Second sub-step: Determine the pixel value in the decoration region mask map corresponding to the pixel-level change probability pixel as the target pixel value.The third sub-step, in response to determining that the target pixel value is a first preset value, compares the target change probability value with a preset lower probability threshold, and performs the following steps: In response to determining that the target change probability value is less than or equal to the preset lower probability threshold, updates the pixel value corresponding to the pixel-level change probability pixel to a second preset value, thereby updating the pixel-level change probability pixel. In response to determining that the target change probability value is greater than the preset lower probability threshold, updates the pixel value corresponding to the pixel-level change probability pixel to the first preset value, thereby updating the pixel-level change probability pixel. Therefore, within the decoration area, i.e., when the target pixel value is the first preset value, a low threshold (lower probability threshold) is used, ensuring that pixels with relatively low change probability responses but belonging to real decoration changes (such as changes in small-sized components like door frame edges and baseboards) can also be effectively detected, thus guaranteeing the detection sensitivity of real structural changes within the decoration semantic area. Subsequently, in response to determining that the target pixel value is a second preset value, the following steps are performed: In response to determining that the target change probability value is less than or equal to a preset upper probability threshold, the pixel value corresponding to the pixel-level change probability pixel is updated to the second preset value to update the pixel-level change probability pixel. In response to determining that the target change probability value is greater than the preset upper probability threshold, the pixel value corresponding to the pixel-level change probability pixel is updated to a first preset value to update the pixel-level change probability pixel. Thus, in non-decoration areas, i.e., when the target pixel value is the second preset value, a high threshold (upper probability threshold) is used, effectively filtering out moderate probability responses caused by non-decoration factors such as construction worker movement, temporary facility placement, and lighting changes, thereby suppressing false change false alarms in non-decoration semantic areas. Finally, the pixel-level change probability map updated for each pixel-level change probability pixel is determined as the decoration change mask map. It is precisely because of the adoption of a differentiated threshold processing strategy based on the decoration semantic region mask that the detection threshold is lower for pixels with change probability falling within the decoration semantic region to ensure that real decoration structural changes are effectively detected, and higher for pixels outside the decoration semantic region to suppress false changes caused by non-decoration factors such as construction workers walking, temporary facilities being placed, and lighting changes. This ensures that real changes with relatively low change probability responses in the decoration semantic region (such as changes in small-sized components such as door frame edges and baseboards) are no longer missed, and that interfering factors such as construction workers walking and lighting changes outside the decoration semantic region are no longer falsely reported as decoration changes, thereby improving the accuracy of the decoration change mask map.

[0070] Step 107: For each connected component location information, perform decoration change action recognition processing on the image representing the location of the connected component location information in the first temporal image and the image representing the location of the connected component location information in the second temporal image to obtain decoration action recognition information.

[0071] In some embodiments, the execution entity may perform decoration change action recognition processing on the image representing the position of the connected component position information in the first temporal image and the image representing the position of the connected component position information in the aligned second temporal image for each connected component position information, to obtain decoration action recognition information.

[0072] In some optional implementations of certain embodiments, the execution entity may perform decoration change action recognition processing on the image representing the position of the connected component position information in the first temporal image and the image representing the position of the connected component position information in the aligned second temporal image through the following steps to obtain decoration action recognition information: The first step is to determine the image at the location represented by the connected component location information in the first temporal image as the first image to be identified.

[0073] The second step involves performing semantic category recognition processing on the first image to be recognized to obtain a first semantic category identifier. In practice, the executing entity crops a local region corresponding to the connected component location information from the first semantic segmentation map, and statistically analyzes the semantic category identifiers of all pixels within this local region (for example, taking the semantic category identifier with the highest frequency). The semantic category identifier with the highest frequency is then used as the first semantic category identifier of the first image to be recognized.

[0074] The third step is to determine the image whose position is represented by the connected component position information in the above-mentioned aligned second temporal image as the second image to be identified.

[0075] The fourth step is to perform semantic category recognition processing on the second image to be recognized to obtain a second semantic category identifier. In practice, the executing entity crops out a local region corresponding to the connected component location information from the semantic segmentation map, counts the semantic category identifiers of all pixels in the local region (for example, takes the semantic category identifier with the highest frequency), and uses the semantic category identifier with the highest frequency as the second semantic category identifier of the second image to be recognized.

[0076] The fifth step involves generating new action information as decoration action identification information in response to the first semantic category identifier not being any of the preset decoration semantic category identifiers and the second semantic category identifier being one of the preset decoration semantic category identifiers. Here, the preset decoration semantic category identifier set refers to the set of all decoration-related semantic category identifiers. In practice, when the first semantic category identifier is not any of the preset decoration semantic category identifiers (i.e., in the first temporal state, the location does not belong to any decoration component category but to the background category), and the second semantic category identifier is one of the preset decoration semantic category identifiers (i.e., in the second temporal state, the location belongs to a certain decoration component category), the executing entity can determine the second semantic category identifier (decoration component identifier) ​​and the preset decoration action identifier representing the new action as the new action information. For example, if the first semantic category identifier is an empty doorway (background) and the second semantic category identifier is a door, then the new action information would be "Preset decoration action identifier: New, Second semantic category identifier: Door".

[0077] Step 6: In response to the first semantic category identifier being a decoration semantic category identifier in the aforementioned decoration semantic category identifier set and the second semantic category identifier not being any decoration semantic category identifier in the preset decoration semantic category identifier set, reduction action information is generated as decoration action identification information. In practice, when the first semantic category identifier is a decoration semantic category identifier in the aforementioned decoration semantic category identifier set (i.e., in the first temporal state, the location belongs to a certain decoration component category), and the second semantic category identifier is not any decoration semantic category identifier in the aforementioned decoration semantic category identifier set (i.e., in the second temporal state, the location does not belong to any decoration component category, but belongs to the background category), the executing entity can determine the first semantic category identifier (i.e., as a decoration component identifier) ​​and the decoration action identifier representing demolition as reduction action information.

[0078] Step 7: In response to the fact that both the first semantic category identifier and the second semantic category identifier are decoration semantic category identifiers in the decoration semantic category identifier set, and that the first semantic category identifier and the second semantic category identifier are different, replacement action information is generated as decoration action identification information. In practice, the executing entity can determine the first semantic category identifier and the second semantic category identifier as identifiers of various decoration components. Then, the identifiers of various decoration components and the decoration action identifier representing the replacement are determined as decoration action identification information. Optionally, the executing entity can also perform the following steps: The first step is to perform the following unexpected change detection processing on each of the generated renovation action recognition information: The first sub-step involves determining that the decoration component identifier included in the aforementioned decoration action identification information is a non-removable component identifier from a preset set of non-removable component identifiers, and then identifying the decoration action identifier included in the aforementioned decoration action identification information as the target decoration action identifier. Here, the aforementioned decoration component identifier refers to a number used to uniquely identify a type of decoration component. The aforementioned set of non-removable component identifiers refers to a collection consisting of the identifiers of all non-removable decoration components.

[0079] The second sub-step, in response to determining that the target renovation action identifier represents demolition, adds the renovation component identifier to a preset initial unexpected change event information, thus obtaining the added unexpected change event information. Here, the initial unexpected change event information refers to an initially empty data structure used to record unexpected change events. In practice, when the target renovation action identifier represents "reduction" (i.e., the non-removable component is demolished), the executing entity adds the renovation component identifier to the initial unexpected change event information, generating the unexpected change event information.

[0080] The second step involves, based on at least one unexpected change event information obtained, triggering an audible and visual alarm, an SMS gateway, or a smart bracelet to issue an alert to the supervisor. The audible and visual alarm refers to an electronic device capable of emitting sound and flashing alarms. The SMS gateway refers to a communication device capable of sending SMS messages to a designated mobile phone number. The smart bracelet refers to a wearable device worn by the supervisor, capable of receiving information and generating vibration alerts. In practice, when the aforementioned executing entity generates at least one unexpected change event information, it sends an alarm signal to the audible and visual alarm to trigger an audible and visual alarm, or sends an alarm SMS instruction to the SMS gateway to trigger an alarm SMS message to the supervisor's mobile phone, or sends an alarm notification to the smart bracelet to trigger vibration and / or display alarm information, thereby reminding the supervisor to promptly address the unexpected change event.

[0081] Optionally, the aforementioned implementing entity may also perform the following steps: The first step is to define the obtained renovation action recognition information as a renovation action recognition information set, where each renovation action recognition information corresponds to one of the connected component location information sets mentioned above. The renovation action recognition information set refers to the collection composed of each renovation action recognition information. Each renovation action recognition information corresponds one-to-one with a connected component location information set; that is, each connected component corresponds to one renovation action recognition information set.

[0082] The second step involves performing the following steps for each renovation action recognition piece in the aforementioned renovation action recognition information set: The first sub-step involves determining the target connected component location information as the connected component location information that corresponds to the aforementioned decoration action recognition information. Here, the target connected component location information refers to the connected component location information corresponding to the current decoration action recognition information, i.e., the position of the connected component described by the decoration action recognition information in the aforementioned decoration change mask.

[0083] The second sub-step generates engineering quantity information based on the aforementioned decoration action recognition information and the target connected component location information. This engineering quantity information describes the changes in decoration components and may include the type of changed component, the type of change (addition / reduction / replacement), and the changed physical quantity (e.g., area, volume, or length). In practice, the executing entity first calculates the pixel area of ​​the connected component in the decoration change mask based on the target connected component location information. Then, the executing entity combines the camera calibration parameters (including intrinsic and extrinsic parameter matrices) to convert the pixel area into the actual physical area (or volume / length). The intrinsic parameter matrix describes the optical characteristics of the image acquisition device, including parameters such as focal length and principal point coordinates, and is used to convert image pixel coordinates into normalized coordinates in the camera coordinate system. The extrinsic parameter matrix describes the position and orientation of the image acquisition device in the world coordinate system and is used to convert coordinates in the camera coordinate system into coordinates in the world coordinate system. Finally, the executing entity calculates the quantity difference value corresponding to the connected domain based on the decoration action type and decoration component category in the decoration action identification information, combined with predefined decoration component quantity calculation rules (e.g., the quantity of wooden doors is calculated by area, the quantity of baseboards is calculated by length, and the quantity of cabinets is calculated by volume, etc.), and generates the above-mentioned quantity information. The above-mentioned quantity information can be directly used to update the quantity ledger database.

[0084] The above embodiments of this disclosure have the following beneficial effects: the method for recognizing decoration change actions based on decoration semantic constraints according to some embodiments of this disclosure improves the accuracy of decoration change action recognition. Specifically, the reason for the poor accuracy of decoration change action recognition is that: image acquisition at decoration construction sites is usually done using portable devices such as ordinary mobile phones, and is taken by supervisors or site representatives. The camera position, angle, and height are difficult to be completely consistent each time. When the shooting position shifts by a centimeter or the angle shifts by several degrees, the coordinates of the same physical point in the pixel coordinate system in the preceding and following frames will shift accordingly. Since the frame difference method performs pixel-by-pixel difference between two frames, it assumes that the two frames are precisely aligned in space, and that the same physical point is located at the same pixel coordinates in both frames. However, in actual construction scenarios, the handheld postures in the two shots are unlikely to be completely identical, and this alignment premise cannot be met. Once there is a difference in the shooting perspective, even if there is no change in the amount of work at the construction site, i.e., no decoration changes (such as no addition or removal of any components in two weeks), the pixel position of the same wall area in the two frames will also be significantly offset. The frame difference method will misjudge this offset as a change in pixel value, and thus incorrectly identify it as a decoration change, resulting in poor accuracy in recognizing decoration change actions. Based on this, some embodiments of the decoration change action recognition method based on decoration semantic constraints of this disclosure first acquire a first temporal image of the same construction space acquired in the first construction stage and a second temporal image acquired in the second construction stage. Thus, two temporal images of the same construction space at different construction stages can be obtained. Then, based on the first temporal image, feature-guided spatial alignment processing is performed on the second temporal image to obtain an aligned second temporal image. This eliminates geometric distortion caused by inconsistencies in camera position, angle, and height between two shots, ensuring precise alignment of the preceding and following temporal images in the spatial coordinate system. Since the two images are spatially aligned, the same physical point resides at the same pixel coordinates in both frames, preventing pixel shifts caused by differences in shooting perspective from being misinterpreted as pixel value changes. Next, the first temporal image and the aligned second temporal image are input into a pre-trained dual-temporal pixel-level change detection Siamese network to obtain a pixel-level change probability map. Because the two input images are spatially aligned, the Siamese network can focus on the true structural changes at the semantic level, without needing to handle pixel coordinate shifts caused by perspective differences. Thus, it can accurately perceive the probability of a true decoration change at each pixel location in the precisely aligned image pair, obtaining a pixel-level change probability map. Then, semantic segmentation is performed on the aligned second temporal image to obtain a semantic segmentation map. This allows us to determine the decoration semantic category (e.g., wall, floor, door, window, background, etc.) for each pixel in the aligned second temporal image.Next, the semantic segmentation map and the pixel-level change probability map are subjected to dual-threshold change probability filtering to generate a decoration change mask map. This maintains high sensitivity within the decoration area to ensure that genuine structural changes are not missed, while effectively suppressing false changes such as worker movement and temporary facilities in non-decoration areas, thereby improving the accuracy of change detection. Then, the decoration change mask map is subjected to connected component labeling to obtain the location information of each connected component. This transforms discrete change points at the pixel level into independent change regions with engineering semantics. Then, for each connected component location information, decoration change action recognition processing is performed on the image representing the location of the connected component location information in the first temporal image and the image representing the location of the connected component location information in the aligned second temporal image to obtain decoration action recognition information. This allows for the automatic identification of decoration action recognition information for the three types of decoration actions: "addition," "reduction," and "replacement," corresponding to each valid connected component. It is precisely because the feature-guided spatial alignment process eliminates the geometric distortion introduced by the difference in shooting perspective that the pixel shift caused by the difference in shooting perspective is avoided from being misjudged as a change in pixel value. The Siamese network can accurately perceive the structural changes at the semantic level on precisely aligned image pairs. The synergistic effect of semantic segmentation and dual-threshold differential filtering further filters out the interference of pseudo-changes, thereby improving the accuracy of decoration change action recognition and obtaining more accurate decoration action recognition information.

[0085] Further reference Figure 2 As an implementation of the methods shown in the figures, this disclosure provides some embodiments of a decoration change action recognition device based on decoration semantic constraints. These device embodiments are similar to... Figure 1 Corresponding to the method embodiments shown, this decoration change action recognition device based on decoration semantic constraints can be specifically applied to various electronic devices.

[0086] like Figure 2As shown, a decoration change action recognition device 200 based on decoration semantic constraints in some embodiments includes: an acquisition unit 201, an alignment unit 202, an input unit 203, a semantic segmentation unit 204, a dual-threshold change probability filtering unit 205, a connected component labeling unit 206, and a decoration change action recognition unit 207. The acquisition unit 201 is configured to acquire a first temporal image of the same construction space collected in a first construction phase and a second temporal image collected in a second construction phase; the alignment unit 202 is configured to perform feature-guided spatial alignment processing on the second temporal image based on the first temporal image to obtain an aligned second temporal image; the input unit 203 is configured to input the first temporal image and the aligned second temporal image into a pre-trained dual-temporal pixel-level change detection Siamese network to obtain a pixel-level change probability map; and the semantic segmentation unit 204 is configured to perform semantic segmentation processing on the aligned second temporal image to obtain a semantic segmentation map. The dual-threshold change probability filtering unit 205 is configured to perform dual-threshold change probability filtering on the semantic segmentation map and the pixel-level change probability map to generate a decoration change mask map; the connected component labeling unit 206 is configured to perform connected component labeling on the decoration change mask map to obtain the location information of each connected component; the decoration change action recognition unit 207 is configured to perform decoration change action recognition processing on the image of the location represented by the connected component location information in the first temporal image and the image of the location represented by the connected component location information in the aligned second temporal image for each connected component location information to obtain decoration action recognition information.

[0087] It is understandable that the units recorded in the decoration change action recognition device 200 based on decoration semantic constraints are related to the reference. Figure 1 The steps described in the method correspond to each other. Therefore, the operations, features, and beneficial effects described above are also applicable to the decoration change action recognition device 200 based on decoration semantic constraints and the units contained therein, and will not be repeated here.

[0088] The following is for reference. Figure 3 It shows a schematic diagram of the structure of an electronic device 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0089] like Figure 3As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory 302 or a program loaded from a storage device 308 into a random access memory 303. The random access memory 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, the read-only memory 302, and the random access memory 303 are interconnected via a bus 304. An input / output interface 305 is also connected to the bus 304.

[0090] Typically, the following devices can be connected to the input / output interface 305: input devices 306 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 308 including, for example, magnetic tape, hard disk, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0091] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 309, or installed from a storage device 308, or installed from a read-only memory 302. When the computer program is executed by the processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0092] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical fiber, a portable compact disk read-only memory, an optical storage device, a magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0093] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0094] Computer-readable media may be contained within an electronic device or may exist independently of the electronic device. A computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire a first temporal image of the same construction space acquired during a first construction phase and a second temporal image acquired during a second construction phase; perform feature-guided spatial alignment processing on the second temporal image based on the first temporal image to obtain an aligned second temporal image; input the first temporal image and the aligned second temporal image into a pre-trained dual-temporal pixel-level change detection Siamese network to obtain a pixel-level change probability map; perform semantic segmentation processing on the aligned second temporal image to obtain a semantic segmentation map; perform dual-threshold change probability filtering processing on the semantic segmentation map and the pixel-level change probability map to generate a decoration change mask map; perform connected component labeling processing on the decoration change mask map to obtain the location information of each connected component; and for each connected component location information, perform decoration change action recognition processing on the image representing the location of the connected component location information in the first temporal image and the image representing the location of the connected component location information in the aligned second temporal image to obtain decoration action recognition information.

[0095] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0096] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0097] The units described in some embodiments of this disclosure can be implemented in software or in hardware. The described units can also be located in a processor.

[0098] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0099] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of technical features, but should also cover other technical solutions formed by arbitrary combinations of technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for recognizing decoration change actions based on decoration semantic constraints, characterized in that, include: Acquire a first temporal image of the same construction space acquired during the first construction phase and a second temporal image acquired during the second construction phase; Based on the first temporal image, feature-guided spatial alignment processing is performed on the second temporal image to obtain an aligned second temporal image; The first temporal image and the aligned second temporal image are input into a pre-trained dual-temporal pixel-level change detection Siamese network to obtain a pixel-level change probability map; The aligned second temporal image is subjected to semantic segmentation processing to obtain a semantic segmentation map; The semantic segmentation map and the pixel-level change probability map are subjected to dual-threshold change probability filtering to generate a decoration change mask map; The decoration change mask is processed by connected component labeling to obtain the location information of each connected component; For each connected component location information, the image representing the location of the connected component location information in the first temporal image and the image representing the location of the connected component location information in the aligned second temporal image are subjected to decoration change action recognition processing to obtain decoration action recognition information.

2. The method according to claim 1, characterized in that, Each decoration action identification information includes decoration component identifiers and decoration action identifiers, and the method further includes: For each generated renovation action identification information, the following unexpected change detection processing is performed: In response to determining that the decoration component identifier included in the decoration action identification information is a non-removable component identifier in a preset set of non-removable component identifiers, the decoration action identifier included in the decoration action identification information is determined as the target decoration action identifier; In response to determining that the target decoration action identifier represents demolition, the decoration component identifier is added to the preset initial unexpected change event information to obtain the unexpected change event information after addition; Based on information about at least one unexpected change event, the system triggers an alarm via sound and light, SMS gateway, or smart bracelet to send a warning to the supervisor.

3. The method according to claim 1, characterized in that, The method further includes: The obtained renovation action recognition information is determined as a renovation action recognition information set, wherein each renovation action recognition information corresponds to one of the connected component location information in the various connected component location information; For each renovation action recognition information in the aforementioned renovation action recognition information set, the following steps are performed: The connected component location information that corresponds to the decoration action recognition information in each connected component location information is determined as the target connected component location information; Based on the renovation action recognition information and the target connected domain location information, the engineering quantity information is generated.

4. The method according to claim 1, characterized in that, The step of inputting the first temporal image and the aligned second temporal image into a pre-trained dual-temporal pixel-level change detection Siamese network to obtain a pixel-level change probability map includes: The first temporal image is input into the first feature extraction branch of a pre-trained dual-temporal pixel-level change detection Siamese network. After multi-scale convolutional feature extraction processing by the first feature extraction branch, a first temporal semantic feature map set is obtained. The dual-temporal pixel-level change detection Siamese network includes the first feature extraction branch, the second feature extraction branch, and a pixel-level change prediction module. The aligned second temporal image is input into the second feature extraction branch of the Siamese network. After multi-scale convolutional feature extraction processing by the second feature extraction branch, a second temporal semantic feature map set is obtained. The first feature extraction branch and the second feature extraction branch share network weights. Each second temporal semantic feature map in the second temporal semantic feature map set corresponds one-to-one with each first temporal semantic feature map in the first temporal semantic feature map set. Pixel-by-pixel difference detection is performed on the first temporal semantic feature map and the second temporal semantic feature map of the same scale that correspond one-to-one in the first temporal semantic feature map set and the second temporal semantic feature map set to obtain an initial change heatmap; The initial change heatmaps are subjected to feature fusion processing to obtain a fused change heatmap. The fused change heatmap is input into the pixel-level change prediction module to obtain a pixel-level change probability map.

5. The method according to claim 1, characterized in that, The semantic segmentation process performed on the aligned second temporal image to obtain a semantic segmentation map includes: The aligned second temporal image is input into the encoder of a pre-trained semantic segmentation network. The encoder performs multi-scale feature extraction processing on the aligned second temporal image to obtain a multi-scale semantic feature map. The multi-scale semantic feature map is input to the decoder to perform feature fusion and upsampling processing on the multi-scale semantic feature map to obtain a pixel-level category probability map. The spatial resolution of the pixel-level category probability map is the same as the spatial resolution of the aligned second temporal image. Each pixel in the pixel-level category probability map has a probability vector at its pixel position. Each probability element in the probability vector represents the probability that the pixel position belongs to a semantic category identifier. For each pixel in the pixel-level category probability map, perform the following steps: The probability vector of the pixel position of the pixel is determined as the target probability vector; The semantic category identifier corresponding to the probability element with the highest probability value in the target probability vector is determined as the semantic category identifier corresponding to the pixel; The semantic category identifiers corresponding to each pixel are combined according to the spatial position of each pixel in the pixel-level category probability map to generate a semantic segmentation map. The spatial resolution of the semantic segmentation map is the same as the spatial resolution of the aligned second temporal image, and the value of each semantic segmentation pixel in the semantic segmentation map is the semantic category identifier of the corresponding pixel.

6. The method according to claim 1, characterized in that, The process of performing decoration change action recognition processing on the image representing the position of the connected component location information in the first temporal image and the image representing the position of the connected component location information in the aligned second temporal image to obtain decoration action recognition information includes: The image representing the location of the connected component location information in the first temporal image is determined as the first image to be identified; The first image to be identified is subjected to semantic category recognition processing to obtain a first semantic category identifier; The image representing the location of the connected component position information in the aligned second temporal image is determined as the second image to be identified; The second image to be identified is subjected to semantic category recognition processing to obtain a second semantic category identifier; In response to the first semantic category identifier not being any one of the preset decoration semantic category identifiers and the second semantic category identifier being one of the preset decoration semantic category identifiers, new action information is generated as decoration action recognition information; In response to the first semantic category identifier being a decoration semantic category identifier in the decoration semantic category identifier set and the second semantic category identifier not being any decoration semantic category identifier in the preset decoration semantic category identifier set, a reduction action information is generated as decoration action recognition information; In response to the fact that both the first semantic category identifier and the second semantic category identifier are decoration semantic category identifiers in the decoration semantic category identifier set and the first semantic category identifier and the second semantic category identifier are different, replacement action information is generated as decoration action identification information.

7. A device for recognizing decoration change actions based on decoration semantic constraints, characterized in that, include: The acquisition unit is configured to acquire a first temporal image of the same construction space acquired during the first construction phase and a second temporal image acquired during the second construction phase. The alignment unit is configured to perform feature-guided spatial alignment processing on the second temporal image based on the first temporal image to obtain an aligned second temporal image; The input unit is configured to input the first temporal image and the aligned second temporal image into a pre-trained dual-temporal pixel-level change detection Siamese network to obtain a pixel-level change probability map; A semantic segmentation unit is configured to perform semantic segmentation processing on the aligned second temporal image to obtain a semantic segmentation map; A dual-threshold change probability filtering unit is configured to perform dual-threshold change probability filtering on the semantic segmentation map and the pixel-level change probability map to generate a decoration change mask map. A connected component marking unit is configured to perform connected component marking processing on the decoration change mask to obtain the location information of each connected component. The decoration change action recognition unit is configured to perform decoration change action recognition processing on the image of the position represented by the connected component position information in the first temporal image and the image of the position represented by the connected component position information in the aligned second temporal image for each connected component position information, so as to obtain decoration action recognition information.

8. An electronic device, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 6.

9. A computer-readable medium, characterized in that, It stores a computer program thereon, wherein the program, when executed by a processor, implements the method as described in any one of claims 1 to 6.

Images