Patrol method and device, nonvolatile storage medium and electronic equipment

Abstract

The application discloses a kind of inspection methods, device, nonvolatile storage medium and electronic equipment.Therein, the method comprises: determining attention window and mask weight map in target inspection image, global feature and local texture feature of target inspection image, axis geometric alignment of attention window and image region;According to attention window and axis direction, determine the basic sampling point set, the receptive field of basic sampling point set in axis direction is greater than the receptive field perpendicular to axis direction;According to local texture feature, the target sampling point set is obtained by correcting basic sampling point set;According to target sampling point set and mask weight map, target detail feature is extracted;According to target detail feature, determine target fusion feature, and determine the risk information of object to be detected according to target fusion feature.The application solves the technical problem that the risk information that small target object can exist cannot be discovered in time due to the fact that small target object in image cannot be accurately distinguished in the related art.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically, to an inspection method, apparatus, non-volatile storage medium, and electronic device. Background Technology

[0002] In related technologies, when identifying inspection images to determine risk information, the inability to accurately distinguish small target objects in the inspection images leads to the failure to promptly detect potential risk information associated with these small target objects.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides an inspection method, apparatus, non-volatile storage medium, and electronic device to at least solve the technical problem that the inability to accurately distinguish small target objects in images in related technologies leads to the inability to timely detect potential risk information of small target objects.

[0005] According to one aspect of the embodiments of this application, a patrol inspection method is provided, comprising: determining a window of interest and a mask weight map, global features and local texture features of the target patrol image in a target patrol image, wherein the window of interest is used to identify the image region corresponding to the object to be detected in the target patrol image, and the axis of the window of interest and the image region are geometrically aligned, and the mask weight map is used to represent the importance of each region in the target patrol image; determining a set of basic sampling points in the global features based on the axis direction of the window of interest and the axis, wherein the receptive field of the set of basic sampling points in the axis direction is larger than the receptive field perpendicular to the axis direction; correcting the position of the sampling points in the set of basic sampling points based on the local texture features, and determining the importance index of the sampling points to obtain a target sampling point set; extracting target detail features from the global features based on the target sampling point set and the mask weight map, wherein the target detail features include detail features of the object to be detected along the axis direction; fusing the global features, local texture features and target detail features to obtain a target fused feature, and determining the risk information of the object to be detected based on the target fused feature.

[0006] Optionally, extracting target detail features from global features based on the target sampling point set and mask weight map includes: performing convolution calculation on global features using an adaptive rectangular convolution method based on the position and importance index of sampling points in the target sampling point set to obtain a first sampling result, wherein the importance index of the sampling point is used to reflect the importance of the feature at the sampling point; and performing residual weight calculation on the first sampling result based on the mask weight map to obtain target detail features.

[0007] Optionally, the target fusion features are obtained by fusing global features, local texture features, and target detail features, including: determining the reference axis direction of the axis based on global features, and determining the deviation vector between the reference axis direction and the axis direction; correcting the interest window based on the deviation vector, and generating a soft mask weight map corresponding to the reference axis direction based on the corrected interest window; extracting target local texture features from the target inspection image based on the soft mask weight map; and fusing global features, target local texture features, target detail features, and downsampled feature maps of the target inspection image to obtain the target fusion features, wherein the downsampled feature map is the feature map obtained by directly downsampling the target inspection image.

[0008] Optionally, determining the risk information of the object to be detected based on the target fusion features includes: determining the first classification confidence, the second classification confidence, and the location information of abnormal regions in the target inspection image based on the target fusion features, wherein the abnormal regions are areas with safety risks, the first classification confidence includes the attitude displacement confidence of the object to be detected, and the second classification confidence includes the fall risk confidence of the object to be detected; if the first classification confidence is greater than the first confidence threshold and the second classification confidence is greater than the second confidence threshold, determining the risk level of the object to be detected based on the first classification confidence and the second classification confidence, and using the risk level as risk information, wherein the risk level is positively correlated with the probability of an accident occurring in the object to be detected, and the values ​​of the first classification confidence and the second classification confidence are positively correlated with the risk level; if the first classification confidence is less than the first confidence threshold or the second classification confidence is less than the second confidence threshold, updating the axis direction based on the reference axis direction, and redetermining the basic sampling point set based on the updated axis direction, and then redetermining the target fusion features based on the redetermined basic sampling point set.

[0009] Optionally, determining the first classification confidence, the second classification confidence, and the location information of the abnormal region in the target inspection image based on the target fusion features includes: performing linear projection processing and global average pooling processing on the target fusion features to obtain the target semantic vector; and using an anchorless detection head to process the target semantic vector to obtain the first classification confidence, the second classification confidence, and the location information of the abnormal region in the target inspection image.

[0010] Optionally, after determining the risk information of the object to be detected based on the target fusion features, the method further includes: determining the operation and maintenance personnel information associated with the object to be detected; generating operation and maintenance work orders corresponding to the risk level, and dispatching operation and maintenance work orders based on the operation and maintenance personnel information.

[0011] Optionally, determining the window of interest in the target inspection image includes: using a geometric polygon attention mechanism to determine the window of interest in the target inspection image; and / or determining the mask weight map by: determining a multi-scale image feature set of the target inspection image, wherein the multi-scale image feature set includes a first image feature, a second image feature, and a third image feature, wherein the first image feature is an image feature obtained after downsampling the target inspection image, the second image feature is an image feature obtained after downsampling the first image feature, and the third image feature is an image feature obtained after downsampling the second image feature; generating a prediction weight map based on the second image feature, and determining elliptical Gaussian prior information elongated along the axial direction based on the second image feature, wherein the elliptical Gaussian prior information is used to represent the axial direction, and the prediction weight map is used to indicate the importance of each region in the target inspection image; and fusing the prediction weight map and the elliptical Gaussian prior information to obtain the mask weight map.

[0012] Optionally, determining the global and local texture features of the target inspection image in the target inspection image includes: based on the attention window, performing aligned sampling on the calibrated second image features within the attention window to obtain a second sampling result; performing residual calculation on the second sampling result based on the mask weight map to obtain global features; based on the attention window, performing aligned sampling on the calibrated third image features within the attention window to obtain a third sampling result; and performing residual calculation on the third sampling result based on the mask weight map to obtain local texture features.

[0013] According to another aspect of the embodiments of this application, a method for training an inspection model is also provided. The inspection model is used to perform an inspection method, including: determining a shape consistency loss function and a region overlap loss function for the inspection model, wherein the shape consistency loss function is used to represent the degree of geometric deviation between the interest window and the ground truth bounding box corresponding to the object to be detected, and the region overlap loss function is used to represent the degree of deviation in the spatial dimension between the interest window and the sampling domain corresponding to the target sampling point set; performing a weighted summation of the shape consistency loss function and the region overlap loss function to obtain a comprehensive loss function; and training the inspection model based on the comprehensive loss function.

[0014] Optionally, the shape consistency loss function is determined by: determining the degree of overlap between the coverage areas of the window of interest and the true bounding box; determining the distance between the center of the window of interest and the center of the true bounding box; determining the diagonal length of the circumscribed union rectangle of the window of interest and the true bounding box; determining the deviation between the first aspect ratio of the window of interest and the second aspect ratio of the true bounding box, wherein the first aspect ratio is the ratio of the long side to the short side of the window of interest, and the second aspect ratio is the ratio of the long side to the short side of the true bounding box; and determining the shape consistency loss function based on the degree of overlap, distance, diagonal length, and deviation.

[0015] Optionally, the regional overlap loss function is determined by: determining the overlap evaluation index of the sampling domain corresponding to the focus window and the target sampling point set at each feature level, wherein the sum of the overlap evaluation index and the intersection-union ratio of the sampling domain corresponding to the focus window and the target sampling point set at each feature level is 1; and determining the average value of the overlap evaluation index corresponding to each feature level as the regional overlap loss function.

[0016] According to another aspect of the embodiments of this application, an inspection device is provided, comprising: a first processing module, configured to determine a window of interest and a mask weight map, global features and local texture features of the target inspection image in a target inspection image, wherein the window of interest is used to identify the image region corresponding to the object to be detected in the target inspection image, and the axis of the window of interest and the image region are geometrically aligned, and the mask weight map is used to represent the importance of each region in the target inspection image; a second processing module, configured to determine a set of basic sampling points in the global features based on the axial direction of the window of interest and the axis, wherein the receptive field of the set of basic sampling points in the axial direction is larger than the receptive field perpendicular to the axis direction; a third processing module, configured to correct the position of the sampling points in the set of basic sampling points based on the local texture features, and determine the importance index of the sampling points to obtain a target sampling point set; a fourth processing module, configured to extract target detail features from the global features based on the target sampling point set and the mask weight map, wherein the target detail features include detail features of the object to be detected along the axis direction; and a fifth processing module, configured to fuse the global features, local texture features and target detail features to obtain target fused features, and determine the risk information of the object to be detected based on the target fused features.

[0017] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, wherein a program is stored in the non-volatile storage medium, wherein the program controls the device where the non-volatile storage medium is located to execute an inspection method or an inspection model training method when it runs.

[0018] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes an inspection method or an inspection model training method during runtime.

[0019] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of an inspection method or an inspection model training method.

[0020] In this embodiment, a focus window and a mask weight map are determined in the target inspection image, along with global and local texture features of the target inspection image. The focus window identifies the image region corresponding to the object to be detected in the target inspection image, and the axis of the focus window and the image region are geometrically aligned. The mask weight map represents the importance of each region in the target inspection image. Based on the axis direction of the focus window and the axis, a basic sampling point set is determined in the global features, wherein the receptive field of the basic sampling point set along the axis direction is larger than the receptive field perpendicular to the axis direction. The positions of the sampling points in the basic sampling point set are corrected based on the local texture features, and the importance index of the sampling points is determined to obtain the target sampling point set. Based on the target sampling point set and mask weight map, target detail features are extracted from global features. These target detail features include the detail features of the object to be detected along the axis direction. Global features, local texture features, and target detail features are fused to obtain target fusion features. The risk information of the object to be detected is determined based on the target fusion features. By adjusting the receptive field of the sampling point set in each direction, the method achieves the goal of focusing on extracting features along the axis direction of the object to be detected. This achieves the technical effect of accurately identifying small target objects and extracting their key features, thereby solving the technical problem that the inability to accurately distinguish small target objects in images in related technologies leads to the inability to detect potential risk information of small target objects in a timely manner. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0022] Figure 1 This is a schematic diagram of the structure of a computer terminal (or mobile device) according to an embodiment of this application;

[0023] Figure 2 This is a flowchart illustrating an inspection method provided according to an embodiment of this application;

[0024] Figure 3 This is a schematic diagram of the internal feature processing flow of an inspection model according to an embodiment of this application;

[0025] Figure 4 This is a flowchart illustrating an inspection model training method according to an embodiment of this application;

[0026] Figure 5 This is a schematic diagram illustrating a model performance comparison result provided in an embodiment of this application;

[0027] Figure 6This is a schematic diagram of the structure of an inspection system according to an embodiment of this application;

[0028] Figure 7 This is a schematic diagram of a detection result provided according to an embodiment of this application;

[0029] Figure 8 This is a schematic diagram of another detection result provided according to an embodiment of this application;

[0030] Figure 9 This is a schematic diagram of another detection result provided according to an embodiment of this application;

[0031] Figure 10 This is a schematic diagram of an inspection interaction interface provided according to an embodiment of this application;

[0032] Figure 11 This is a schematic diagram of the structure of an inspection device provided according to an embodiment of this application. Detailed Implementation

[0033] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0034] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0035] To better understand the embodiments of this application, the technical terms involved in the embodiments of this application are explained below:

[0036] Adaptive Rectangular Convolution (AR Conv): A convolution mechanism that dynamically modulates the aspect ratio and sampling position of the convolution kernel along the antenna axis to amplify detailed features related to antenna attitude shift and corrosion deformation, while suppressing irrelevant textures in other directions.

[0037] Quadrangle Attention (QA): This technique generates a quadrilateral region of interest aligned with the antenna axis. By regressing affine transformation parameters in the image, it enables the model to focus on specific components of the antenna, reducing the impact of background complexity and lighting interference.

[0038] Vision Transformer (ViT): The basic architecture of deep learning models used to process image data. It captures global features through a self-attention mechanism, providing feature extraction capabilities for the detection of abnormal poses of spotlight antennas and fall risk warnings.

[0039] QA mechanism: refers to the geometric polygon attention mechanism, which guides the model to focus on key parts of the antenna by generating a quadrilateral region of interest that is consistent with the antenna axis, thereby improving the extraction accuracy of attitude anomalies and potential risk features.

[0040] In residential network coverage, spotlight antennas achieve directional signal coverage over densely built-up areas through preset elevation and azimuth angles. However, environmental factors such as prolonged exposure to wind and sun can easily lead to corrosion, loosening, and deformation of antenna supports, resulting in abnormal antenna postures such as "tilting up" or "lowering down." This not only causes the main lobe of electromagnetic waves to deviate from its designed direction, significantly degrading user perception, but more seriously, if not detected and addressed in time, it also poses safety hazards such as support breakage and antenna falls, directly endangering residents' safety. Therefore, timely inspection of the health status of spotlight antennas in residential areas to determine whether there are risks of abnormal postures or falls is a key aspect of operators' infrastructure operation and maintenance work.

[0041] Traditional methods of inspection, such as manual inspection or drone aerial photography combined with human visual verification, suffer from the following problems: ① Small targets are difficult to identify (spotlight antennas occupy a low proportion in aerial footage), and the visual characteristics of attitude deviation and falling due to corrosion and deformation are not sufficiently significant; ② Environmental noise, such as complex lighting interference (strong reflections / multi-period shadow changes), significantly reduces the identifiability of abnormal spotlight attitude features; ③ Experience dependence and subjective bias: Human visual verification relies heavily on the experience and knowledge of the personnel involved, is inefficient, and its accuracy is easily affected by subjective factors such as visual fatigue; ④ Efficiency and safety bottlenecks: Maintenance personnel working at heights pose safety hazards, and the overall response time is insufficient to meet emergency inspection needs. Especially after extreme weather events such as typhoons, traditional inspection methods struggle to quickly and accurately locate and address abnormal spotlight attitude risks, further amplifying network outages and public safety risks. Therefore, a highly efficient and automated intelligent inspection method is urgently needed to accurately detect and warn of spotlight antenna attitude deviation and falling risks caused by bracket corrosion / deformation, in order to address the maintenance challenges in complex environments.

[0042] To address this issue, relevant solutions are provided in the embodiments of this application, which are described in detail below.

[0043] According to an embodiment of this application, a method embodiment of an inspection method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0044] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing an inspection method or inspection model training method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0045] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0046] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the inspection method or inspection model training method in the embodiments of this application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned inspection method or inspection model training method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0047] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0048] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0049] Under the above operating environment, this application embodiment provides an inspection method, such as... Figure 2 As shown, the method includes the following steps:

[0050] Step S202: Determine the attention window and mask weight map, global features and local texture features of the target inspection image in the target inspection image. The attention window is used to identify the image region corresponding to the object to be detected in the target inspection image, and the axis of the attention window and the image region are geometrically aligned. The mask weight map is used to represent the importance of each region in the target inspection image.

[0051] In the technical solution provided in step S202, the step of determining the window of interest in the target inspection image includes: using a geometric polygon attention mechanism to determine the window of interest in the target inspection image; and / or determining the mask weight map in the following ways: determining a multi-scale image feature set of the target inspection image, wherein the multi-scale image feature set includes a first image feature, a second image feature, and a third image feature, wherein the first image feature is an image feature obtained after downsampling the target inspection image, the second image feature is an image feature obtained after downsampling the first image feature, and the third image feature is an image feature obtained after downsampling the second image feature; generating a prediction weight map based on the second image feature, and determining elliptical Gaussian prior information stretched along the axial direction based on the second image feature, wherein the elliptical Gaussian prior information is used to reflect the axial direction, and the prediction weight map is used to indicate the importance of each region in the target inspection image; fusing the prediction weight map and the elliptical Gaussian prior information to obtain the mask weight map.

[0052] As an optional implementation, the steps of determining the global features and local texture features of the target inspection image include: based on the window of interest, performing aligned sampling on the calibrated second image features within the window of interest to obtain a second sampling result; performing residual calculation on the second sampling result based on the mask weight map to obtain global features; based on the window of interest, performing aligned sampling on the calibrated third image features within the window of interest to obtain a third sampling result; and performing residual calculation on the third sampling result based on the mask weight map to obtain local texture features.

[0053] Optionally, the aforementioned target inspection images include aerial images taken by the drone during the inspection process, containing the objects to be inspected. The objects to be inspected include spotlight antennas and their supports, etc.

[0054] Optionally, in the inspection method provided in this application embodiment, the processing of the target inspection image (aerial image) can be divided into three stages, including the first stage: regional feature focusing guided by the QA mechanism, the second stage: fine-grained feature extraction based on ARConv, and the third stage: high-order semantic verification and feature aggregation.

[0055] In some embodiments of this application, taking a spotlight antenna as an example, in spotlight attitude health inspection tasks, the antenna in aerial images is often affected by factors such as small target weakening and complex lighting environments. Its attitude anomalies and support deformation, among other micro-features, are easily diluted by background noise. Meanwhile, the rectangular attention of traditional ViT is difficult to fit the real axis and contour, further weakening the recognition accuracy. To address this, in the first stage, to achieve a unified alignment coordinate system, this application introduces a QA mechanism. Based on affine transformation, the image is slightly rotated / scaled / translated to align the region of interest with the antenna attitude, and the rectangular window is dynamically fitted into a quadrilateral region of interest that fits the antenna component. Simultaneously, a weight mask in the same direction as the axis is generated, with emphasis on weighting areas strongly correlated with attitude anomalies, such as bolt textures and connection points, thereby obtaining aligned global orientation features and aligned local detail features. This eliminates reference system differences caused by different specifications through geometric alignment, and improves the signal-to-noise ratio of key parts through semantic focusing, significantly enhancing the overall ability to capture attitude shifts and providing reliable input for subsequent fall risk assessment.

[0056] Optionally, the input data for the first stage includes preprocessed images (target inspection images). (Size uniformly 512×512×3) and dimensionality features of the preprocessed image. The feature extraction backbone can use ViT, followed by a lightweight pyramid to construct multidimensional scale features. , Then, QA can be used to first regress the affine parameters and form the region of interest V (the aforementioned window of interest), aligning the region of interest with the spotlight antenna axis. Then... The generated prediction weight map is fused with the axially elongated elliptical Gaussian prior (the probability distribution along the rod direction, used to emphasize the rod direction) to obtain the weight map M. (The above mask weight graph). Then, [the following is done]... , The layer feature map is aligned and sampled within V, and residual amplification is performed using M to highlight details related to attitude anomalies / fall risks, thus obtaining the aligned / focused features after QA. (The above global features) (The aforementioned local texture features), these two parts and M are used as inputs to the second-stage AR Conv, to further enhance linear details related to pose anomalies and suppress background interference such as lighting. The specific calculation formula for this stage is as follows:

[0057]

[0058]

[0059]

[0060]

[0061]

[0062] in It is an affine transformation matrix; () represents the center pixel coordinates; () represents the dimensional parameters in the axial / vertical direction; It is a 1×1 convolution operation; yes and An elliptic Gaussian prior constructed for the parameters, The direction parameter (which can be a direction angle or a unit direction vector determined by the direction angle) represents the direction of the axis and is used to determine the orientation of the rotation matrix R(θ). The output is a 64×64 weighted graph. The fusion coefficient; It is a pixel-by-pixel multiplication operation; This is the residual amplification factor.

[0063] In some embodiments of this application, the task objectives, input data, core methods, output data, and the use of the output data in the next stage in the first stage are shown in the table below:

[0064]

[0065] Step S204: Based on the axial direction of the focus window and the axis, determine the basic sampling point set in the global features, wherein the receptive field of the basic sampling point set in the axial direction is larger than the receptive field in the direction perpendicular to the axis.

[0066] Step S206: Correct the position of the sampling points in the basic sampling point set based on the local texture features, and determine the importance index of the sampling points to obtain the target sampling point set;

[0067] Step S208: Extract target detail features from global features based on the target sampling point set and mask weight map, wherein the target detail features include detail features of the object to be detected along the axis direction;

[0068] In the technical solution provided in step S208, the step of extracting target detail features from global features based on the target sampling point set and the mask weight map includes: performing convolution calculation on global features using an adaptive rectangular convolution method based on the position and importance index of the sampling points in the target sampling point set to obtain a first sampling result, wherein the importance index of the sampling point is used to reflect the importance of the features at the sampling point; and performing residual weight calculation on the first sampling result based on the mask weight map to obtain the target detail features.

[0069] It is important to note that traditional isotropic convolution kernels, due to their fixed size, symmetrical structure, and limited receptive field, struggle to adapt to the significant scale differences between surrounding buildings and small antenna components. They also lack spatial adaptability to spotlight attitude shifts and irregular deformations, resulting in a weak response to critical geometric areas such as support corrosion and micro-deformation. Therefore, in the second stage, based on the alignment results from the first stage, it is necessary to amplify continuous geometric changes axially and suppress irrelevant textures in the vertical direction within a reference frame aligned with the antenna axis, separating features strongly correlated with attitude anomalies from background noise.

[0070] To achieve the above objectives, the second stage employs AR Conv. Under the constraints of the axis orientation prior provided in the first stage (i.e., the axis direction parameter (direction angle θ / unit direction vector v)) and the weight map M, the sampling position and longitudinal and transverse scale of the convolution kernel are modulated in a learnable orientation manner. This involves elongating the basic sampling grid in the antenna rod direction and tightening it vertically. A bias network then predicts the sampling offset and modulation coefficients from the alignment detail features, transforming the fixed kernel into an asymmetric deformable kernel aligned with the axis. This allows the convolution to both "look in the right direction" and "adaptively look in the right place," significantly amplifying signs of continuous axial offset such as tilting / lowering, highlighting micro-damage and early corrosion at joints. The output of this stage is an axially enhanced feature pixel-wise aligned with the affine region of interest in the first stage. Furthermore, the output of this stage serves as the input to the third stage, where re-QA and anchorless detection are performed to complete consistency verification and risk quantification.

[0071] Optionally, the first stage output {V, M, ... , } is the input, where The main processing layer is defined by M, which represents the spatial weights. Firstly, based on... A directional fundamental sampling grid is generated in the domain of interest reference frame, which elongates the receptive field of the core along the antenna rod direction and tightens it in the vertical direction. The fundamental sampling grid refers to the sampling coordinate grid / relative coordinate layout (i.e., the set of relative positions of each sampling point) composed of a set of fundamental sampling points. Subsequently, the bias network... The sampling offset field and modulation coefficients are learned to fine-tune the base sampling positions, and the modulation coefficients are used to assign different importance to each sampling point. Finally, convolution is performed on this orientation-guided + offset-corrected sampling grid, and residual weighting is performed with M to enhance the feature response only in regions related to attitude anomalies / fall risk. After the stage, the enhanced feature Z, aligned with the coordinates of the region of interest, is obtained. Together with and As input to the third stage "re-QA", it is used for global orientation verification, secondary weighting and cross-scale gating fusion, and finally outputs two confidence scores: "attitude anomaly" and "fall risk". The specific calculation formula for this stage is as follows:

[0072] A(1) Orientation-based sampling grid (anisotropic scaling):

[0073]

[0074]

[0075] A(2) Bias network predicts sampling offset and modulation (deformable kernel):

[0076]

[0077] A(3) Direction-guided + weighted residual coupling output features (pixel-by-pixel):

[0078]

[0079] in for Any coordinate on the layer grid; Based on the axis orientation given in the first stage Constructed two-dimensional rotation matrix; This represents the effective receptive field (i.e., the height and width of the dynamic matrix convolution kernel) in the "axial / vertical" direction. The orientation basis sampling points obtained from equation A(1) (already according to) Rotate and press (Scaled but not yet deformable offset); Basic sampling points; , By bias network exist Position on the layer Learned; That is, the final deformable sampling point that includes both orientation anisotropy and local offset; To upsample Weighted graph of layer resolution; These are the convolution weights. These are the detailed features of the aforementioned target.

[0080] In some embodiments of this application, the task objectives, input data, core methods, output data, and the use of the output data in the next stage of the second stage are shown in the following table:

[0081]

[0082] Step S210: Fuse global features, local texture features and target detail features to obtain target fusion features, and determine the risk information of the object to be detected based on the target fusion features.

[0083] In the technical solution provided in step S210, the step of fusing global features, local texture features, and target detail features to obtain target fusion features includes: determining the reference axis direction of the axis based on global features, and determining the deviation vector between the reference axis direction and the axis direction; correcting the interest window based on the deviation vector, and generating a soft mask weight map corresponding to the reference axis direction based on the corrected interest window; extracting target local texture features from the target inspection image based on the soft mask weight map; and fusing global features, target local texture features, target detail features, and downsampled feature map of the target inspection image to obtain target fusion features, wherein the downsampled feature map is the feature map obtained by directly downsampling the target inspection image.

[0084] In some embodiments of this application, the step of determining the risk information of the object to be detected based on the target fusion features includes: determining a first classification confidence level, a second classification confidence level, and the location information of abnormal regions in the target inspection image based on the target fusion features, wherein the abnormal regions are regions with safety risks, the first classification confidence level includes the attitude offset confidence level of the object to be detected, and the second classification confidence level includes the fall risk confidence level of the object to be detected; if the first classification confidence level is greater than a first confidence level threshold and the second classification confidence level is greater than a second confidence level threshold, determining the risk level of the object to be detected based on the first classification confidence level and the second classification confidence level, and using the risk level as risk information, wherein the risk level is positively correlated with the probability of an accident occurring in the object to be detected, and the values ​​of the first classification confidence level and the second classification confidence level are positively correlated with the risk level; if the first classification confidence level is less than the first confidence level threshold or the second classification confidence level threshold is less than the second confidence level threshold, updating the axis direction based on the reference axis direction, and redetermining the basic sampling point set based on the updated axis direction, and then redetermining the target fusion features based on the redetermined basic sampling point set.

[0085] As an optional implementation, the steps of determining the first classification confidence, the second classification confidence, and the location information of the abnormal region in the target inspection image based on the target fusion features include: performing linear projection processing and global average pooling processing on the target fusion features to obtain the target semantic vector; and processing the target semantic vector using an anchorless detection head to obtain the first classification confidence, the second classification confidence, and the location information of the abnormal region in the target inspection image.

[0086] In some embodiments of this application, the aforementioned first classification confidence level can be expressed as: The confidence level for the second category can be expressed as: .in, The confidence level indicates that the antenna exhibits abnormal attitude deviations such as tilting, low-angle movement, or tilting. This indicates the confidence level of the potential risk of the antenna support structure falling due to corrosion / deformation.

[0087] Optionally, set 0.3, 0.7,

[0088] Level

[0089] In some embodiments of this application, after determining the risk information of the object to be detected based on the target fusion features, the method further includes: determining the operation and maintenance personnel information associated with the object to be detected; generating an operation and maintenance work order corresponding to the risk level, and dispatching the operation and maintenance work order based on the operation and maintenance personnel information.

[0090] Different rectification methods and time limits can be set for different risk levels. For example, low risk can be subject to manual verification, medium risk can be subject to verification and feedback within seven days, and high risk can be subject to verification and feedback within one day.

[0091] It is important to note that relying solely on local details to determine the risk of an object under inspection can easily lead to misjudging reflections and shadows as attitude deviations; conversely, relying solely on global orientation can easily overlook micro-features such as scaffold corrosion and loosening. To reduce these biases, in the third stage, a mechanism for global-local consistency verification and cross-scale evidence convergence needs to be established, building upon the alignment in the first stage and the axial enhancement in the second stage, before completing the risk assessment within a unified coordinate system. Therefore, the third stage introduces an integrated process of "re-attention - cross-scale convergence - anchorless detection" at a higher semantic level: using the affine quadrilateral issued in the first stage as a priori, the axis orientation is first verified at the global level and the deviation vector is estimated (to check for consistency with the previous stage); based on this, a small affine fine-tuning and secondary soft mask gating are performed to suppress reflections / shadows unrelated to the axis from top to bottom, and to strengthen areas strongly correlated with pose anomalies / corrosion deformation; then, in the aligned coordinate system, the global orientation cues and local axial enhancement textures are fused through cross-scale attention and fed into the anchorless detection head (which does not depend on the preset box size and aspect ratio, and is adapted to small targets), while simultaneously outputting two confidence scores and localization boxes for "pose anomalies" and "fall risk". The multi-layer results are weighted and mapped to low / medium / high levels according to a threshold, and linked for order dispatch, thus forming a dynamic closed loop of "QA → AR Conv → re-QA".

[0092] Optional, axial enhancement detail Z output from the second stage. and the global features obtained from the first stage alignment With orientation Prior V is the input. First, in Upward verification orientation ,get And calculate the deviation vector; based on this, make small affine adjustments to V to generate a vector that is consistent with the original vector. 'Consistent secondary soft mask' (The above soft mask weighting diagram), using For alignment Channel and spatial gating were implemented, gradually shifting attention from the top layer to finer-grained layers; subsequently, Z was upsampled to 64×64. With gated (and after downsampling and alignment) Gated attention fusion is performed in the aligned coordinate system to obtain the fused feature map. Then, a 256-dimensional semantic vector is obtained through 1×1 linear projection and global average pooling; the anchorless detection head in Simultaneously output two classification confidence scores and With the regression parameter (coordinates of the outlier region), when the confidence orientation deviation exceeds a predetermined threshold, 'right Perform a soft update and revert to Phase 2 to re-run ARConv to obtain the final result. The regression parameters are then used. The multi-level output results are then weighted and fused, mapped to a risk level, and fed back to the operations and maintenance side, forming a dynamic modeling closed loop of "QA→AR Conv→re-QA". The calculation formulas involved in this stage are as follows:

[0093] R(1) orientation kernel and secondary soft mask:

[0094]

[0095]

[0096]

[0097] R(2) Integration of top-down gating and cross-scale attention:

[0098]

[0099]

[0100]

[0101]

[0102] in For global orientation estimation of the convolutional structure, output the orientation angle of the main axis; This is the angle difference normalization function, used to map the angle difference to the range (-π, π]. It is the fine-tuning amplitude coefficient; It is a channel-based stitching operation that maintains the same spatial resolution; For global average pooling, H (the aforementioned target fusion features) is compressed into a vector; This is a downsampling operation; It is a 256-dimensional semantic vector (the target semantic vector mentioned above), which serves as the global baseline vector for subsequent classification branches.

[0103] In some embodiments of this application, in order to ensure spatial alignment and geometric consistency, shape-aware composite loss can be used to optimize the inspection model through end-to-end backpropagation: on the one hand, the shape consistency term L_(shape-IoU) improves the overlap between the predicted box and the ground truth box, while explicitly penalizing the mismatch between the center distance and the aspect ratio, so as to enhance the discrimination of appearance changes such as "head up / lowering" and "bracket corrosion deformation".

[0104] In some embodiments of this application, each of the above stages can be achieved through, for example... Figure 3 The inspection model shown is implemented as follows. The internal feature processing flow of the inspection model during operation is as follows: Figure 3 As shown.

[0105] In summary, this application provides an intelligent inspection method for alarming abnormal attitude deviation of residential spotlight antennas and warning of fall risks for telecommunications operators. It adopts a "terminal-network-cloud" collaborative architecture and UAV fixed-point multi-view acquisition and 5G backhaul. It has a built-in three-stage AI vision engine of "QA geometric alignment - AR Conv axial modulation - re-attention / anchorless detection". It integrates global orientation semantic features and local detail textures in the alignment coordinate system. After weighting the multi-view results, it outputs two confidence scores for spotlight antenna attitude deviation and fall risk. After threshold classification (low / medium / high) mapping, it automatically dispatches orders and realizes hourly closed-loop processing. It provides technical support for optimizing network coverage quality in residential areas and preventing high-altitude fall risks, and promotes the upgrade of operators' operation and maintenance mode from traditional delayed passive maintenance to preventive proactive operation and maintenance.

[0106] Furthermore, by determining the interest window and mask weight map in the target inspection image, the global features and local texture features of the target inspection image are analyzed. The interest window is used to identify the image region corresponding to the object to be detected in the target inspection image, and the axis of the interest window and the image region are geometrically aligned. The mask weight map is used to represent the importance of each region in the target inspection image. Based on the axis direction of the interest window and the axis, a basic sampling point set is determined in the global features, where the receptive field of the basic sampling point set along the axis direction is larger than the receptive field perpendicular to the axis direction. The positions of the sampling points in the basic sampling point set are corrected based on the local texture features, and the importance index of the sampling points is determined to obtain the target sampling point set. Based on the target... The method extracts target detail features from global features using a set of sampling points and a mask weight map. These target detail features include the detail features of the object to be detected along the axial direction. By fusing global features, local texture features, and target detail features, a target fusion feature is obtained. The risk information of the object to be detected is determined based on the target fusion feature. By adjusting the receptive field of the sampling point set in each direction, the method achieves the goal of focusing on extracting features along the axial direction of the object to be detected. This achieves the technical effect of accurately identifying small target objects and extracting their key features, thereby solving the technical problem that the inability to accurately distinguish small target objects in images in related technologies leads to the inability to detect potential risk information of small target objects in a timely manner.

[0107] Compared to the method provided in this application, the spotlight antenna inspection method in related technologies relies on manual climbing or drone aerial photography followed by human visual verification. This method is inefficient, poses significant safety risks, and has poor real-time performance. Furthermore, in densely populated residential areas, it is easily affected by environmental factors such as obstruction and reflection, resulting in poor inspection results. This application, however, constructs a three-dimensional inspection system integrating "end-network-cloud." Drones utilize autonomous obstacle avoidance and multi-view coverage for data collection. Wireless communication enables low-latency transmission of video and spatiotemporal metadata. An AI decision engine is deployed in the cloud for online detection and identification, forming a closed loop of "collection-identification-alarm-handling." This significantly reduces the risks of manual intervention and high-altitude operations, achieving hourly closed-loop operation and improving inspection efficiency by tens of times.

[0108] AI models in related technologies lack robustness in complex backgrounds and small target scenarios, are susceptible to interference from strong reflections, shadows, and occlusions, and struggle to stably capture minute attitude shifts of spotlights. The method provided in this application employs a joint dynamic modeling approach combining "geometric polygon attention + dynamic matrix convolution + visual sensor": affine adjustment aligns the attention window with the antenna geometry, dynamic matrix convolution adapts to the sampling shift of the antenna attitude, and after multi-scale semantic fusion, directly outputs the attitude anomaly confidence and warning level. This model effectively suppresses background noise and information loss due to scale reduction, significantly improving the detection accuracy and stability of spotlight attitude anomalies in complex scenarios.

[0109] Traditional maintenance methods often fail to promptly identify the risk of antenna brackets falling due to corrosion or deformation, leading to reactive, reactive repairs rather than proactive management. The method provided in this application, however, utilizes an AI decision-making model centered on QA-AR Conv-ViT. It learns end-to-end visual evidence strongly correlated with the likelihood of a fall from aerial imagery, such as corrosion textures, oxidation edges, abnormal axis deflection, and loose connections. By aligning the attention and sampling domains through shape-perception loss and aggregating cross-scale linear component and subtle defect information using a dynamic feature pyramid, it stably outputs the fall risk confidence level. Furthermore, it automatically triggers work order dispatch by combining risk scoring and threshold grading, shifting maintenance from "post-fault repair" to "pre-hazard intervention," significantly reducing public safety and compliance risks and improving the certainty and controllability of large-scale maintenance.

[0110] This application provides a method for training an inspection model, which can be used to perform tasks such as... Figure 4 The inspection method shown. Figure 4 As shown, the inspection model training method includes the following steps:

[0111] Step S402: Determine the shape consistency loss function and the region overlap loss function of the inspection model. The shape consistency loss function is used to represent the degree of geometric deviation between the window of interest and the ground truth bounding box corresponding to the object to be detected, and the region overlap loss function is used to represent the degree of deviation between the window of interest and the sampling domain corresponding to the target sampling point set in the spatial dimension.

[0112] In some embodiments of this application, the shape consistency loss function is determined by: determining the degree of overlap between the coverage areas of the window of interest and the true annotation box; determining the distance between the center of the window of interest and the center of the true annotation box; determining the diagonal length of the circumscribed union rectangle of the window of interest and the true annotation box; determining the deviation value between the first aspect ratio of the window of interest and the second aspect ratio of the true annotation box, wherein the first aspect ratio is the ratio of the long side to the short side of the window of interest, and the second aspect ratio is the ratio of the long side to the short side of the true annotation box; and determining the shape consistency loss function based on the degree of overlap, distance, diagonal length, and deviation value.

[0113] As an optional implementation, the regional overlap loss function is determined as follows: the overlap evaluation index of the sampling domain corresponding to the focus window and the target sampling point set at each feature level is determined, wherein the sum of the overlap evaluation index and the intersection-union ratio of the sampling domain corresponding to the focus window and the target sampling point set at each feature level is 1; the average value of the overlap evaluation index corresponding to each feature level is determined as the regional overlap loss function.

[0114] Step S404: The shape consistency loss function and the region overlap loss function are weighted and summed to obtain the comprehensive loss function;

[0115] Step S406: Train the inspection model based on the comprehensive loss function.

[0116] In some embodiments of this application, in order to ensure spatial alignment and geometric consistency, shape-aware composite loss can be used to optimize the inspection model through end-to-end backpropagation: on the one hand, through the shape consistency term While increasing the overlap between the predicted bounding box and the ground truth bounding box, an explicit penalty is applied to the center distance and aspect ratio mismatch to enhance the ability to distinguish between "head-up / head-down" and "bracket corrosion deformation" appearance changes. As shown in the following formula:

[0117]

[0118] in This indicates the degree of overlap between the predicted bounding box and the truth bounding box. The Euclidean distance between the centers of the two frames is given. The diagonal length of the rectangle containing the bounding union of predicted and true values ​​is used to normalize the center distance to eliminate scale effects. Used to measure aspect ratio differences To control the nonlinear sensitivity of the aspect ratio term (usually set to 1). The weights are selected based on optimization using the validation set. On the other hand, the overlapping region term is considered. constraint The affine quadrilateral attention domain of layer re-attention is spatially consistent with the corresponding AR Conv effective sampling domain to suppress attentional bias and missampling caused by strong reflections or shadows. This is illustrated in the following equation:

[0119]

[0120] in The pyramid feature layers (P2-P4 layers); (A,B) represents the intersection-union ratio of regions A and B; Is the affine quadrilateral in the th... Polygonal regions in layer coordinate system; Is this AR Conv in the 19th Effective sampling domain in layer coordinate system.

[0121] The two losses are adaptively weighted according to the course learning strategy to obtain the shape-aware loss function. This dynamic, alternating perception characteristic of "attention-convolution-re-attention" first focuses, then enhances details and performs a secondary geometric verification, taking into account both the overall posture judgment of "head up / lowering down" and the aggregation of local evidence of "potential fall risk due to corrosion deformation." This collectively drives the AI ​​decision-making model to stably focus on areas related to abnormal spotlight antenna posture under complex lighting and small target conditions. Furthermore, as... Figure 5 As shown, this can significantly improve the performance of the inspection model.

[0122] This application provides an inspection system. Figure 6 This is a schematic diagram of the system structure, which can be used to execute... Figure 2 The inspection method shown and Figure 4 The inspection model training method is shown in the diagram. From... Figure 6 As can be seen, the system comprises a physical layer, a network layer, and an application layer, and through this three-tiered architecture, it collaboratively completes a closed-loop process of "data acquisition—reliable transmission—real-time analysis and feedback." In complex production environments characterized by high building density in residential areas, complex optical environments (strong reflections and shadows), and a high proportion of small targets, the system, with end-to-end visual feature learning at its core, directly outputs the attitude health and fall risk warning confidence levels of spotlight antennas transmitted via UAV aerial photography through a cloud-based analysis and decision-making model. The system also provides real-time image feedback to maintenance personnel, who then manually verify the results on-site and issue rectification reports, forming an hourly maintenance closed loop. This system effectively overcomes the shortcomings of traditional inspection methods or "UAV photography + human verification" in areas such as missed small targets, reliance on experience, low efficiency, and risks associated with high-altitude operations.

[0123] In some embodiments of this application, the overall inspection process of the above-mentioned inspection system is as follows: UAV multi-view acquisition → data transmission → data preprocessing (512×512, normalization, histogram equalization) → Stage-1 positioning and alignment: a focus window V geometrically aligned with the antenna axis is generated through the geometric polygon attention (QA) module, and an elliptic Gaussian prior is used to enhance the axial features and output a soft mask weight map M to suppress reflection / shadow interference. Simultaneously, a system is established... Multi-dimensional scale features; → Stage-2 detail enhancement: in Upward embedding of dynamic matrix convolution (AR Conv) amplifies the corrosion / deformation and antenna attitude shift features of the support along the antenna mast direction, suppresses irrelevant textures in the vertical direction, and outputs enhanced features aligned with V; → Stage-3 judgment and classification: fusion global orientation and After local texturing, orientation verification and cross-scale aggregation are performed, followed by an anchorless detection mechanism to output two confidence levels: "posture anomaly" and "fall risk". , The system maps risks to low, medium, and high risks based on thresholds and automatically dispatches tasks, forming an hourly operation and maintenance closed loop.

[0124] In some embodiments of this application, a hexacopter UAV with autonomous navigation capabilities can be deployed at the physical layer of the system. Its high-definition image acquisition module, based on a fusion technology of navigation satellite system and real-time visual positioning (RTK-GNSS), can achieve meter-level precise positioning and stable hovering in densely populated residential environments. Through dynamic path planning algorithms, the UAV can autonomously avoid obstacles within its coverage radius and acquire raw image data of the rooftop spotlight antenna and its support structure from multiple perspectives, including overhead, side, and 45° oblique views.

[0125] At the physical layer, relying on 5G networks and airborne 5G communication modules, communication resources can be dynamically allocated according to the real-time requirements of inspection tasks. This ensures that high-definition image data and metadata (including time, latitude and longitude information for subsequent feedback) collected during a single inspection task are quickly and stably transmitted back to the cloud processing platform. Other wireless communication networks can be used here; 5G is not mandatory.

[0126] Optionally, a cloud processing platform can be deployed at the application layer. This platform can perform preprocessing operations such as data cleaning and size normalization on the returned spotlight antenna images before inputting the structured data source into the AI ​​visual analysis decision engine. This engine can then run the aforementioned inspection model. Furthermore, based on a visual transducer (ViT), the engine can alternately stack and embed QA and AR Conv to form a joint modeling mechanism of "attention-convolution-re-attention." Through end-to-end feature learning, the decision engine no longer needs to rely on traditional human experience. It can directly extract shallow texture features and deep semantic features from aerial images, identifying potential fall risks due to spotlight antenna attitude shift and component corrosion deformation. Then, it uses a feature pyramid to stack and fuse multi-stage deep and shallow features to address the feature weakening problem in small target detection in aerial images. Finally, the detection head mechanism directly outputs the confidence levels for spotlight attitude shift anomaly alarms and fall risk warnings. Simultaneously, a shape-aware composite loss function is used for backpropagation to jointly optimize the decision engine, further improving its detection and recognition accuracy.

[0127] Optionally, for the input spotlight antenna image I, the aforementioned AI visual analysis and decision engine can call the inspection model to calculate the model based on a large number of pre-labeled spotlight attitude anomaly images. and The two discriminant values ​​represent the probability of the spotlight antenna exhibiting abnormal postures such as tilting up or drooping down, and the probability of the bracket posing a potential risk of falling due to corrosion and deformation, respectively. These probabilities are then evaluated using the Sigmoid function. Mapped to confidence probability:

[0128]

[0129] in, This indicates the confidence level that the antenna exhibits abnormal attitude deviations such as tilting, low-angle movement, or tilting. This indicates the confidence level of the potential fall risk caused by corrosion / deformation of the antenna support structure. These confidence levels are fused and calibrated based on the results collected from multiple perspectives, and then converted into operation and maintenance alarm information through pre-set classification rules. This information is then fed back to operation and maintenance personnel in real time for on-site confirmation and rectification.

[0130] Optionally, set 0.3, 0.7,

[0131] Level

[0132] Different rectification methods and time limits can be set for different risk levels. For example, low risk can be subject to manual verification, medium risk can be subject to verification and feedback within seven days, and high risk can be subject to verification and feedback within one day.

[0133] In some embodiments of this application, taking the period after a strong typhoon or during routine inspections as examples, the cloud control platform can issue inspection tasks for spotlights in residential areas. The hexacopter UAV autonomously plans and avoids obstacles based on a 3D vector map, collecting rooftop spotlight antenna images and spatiotemporal metadata from multiple perspectives (top-down, side-down, oblique) within its coverage radius. The onboard 5G module streams the data back to the cloud, and the visual analysis decision engine performs dynamic modeling and multi-scale fusion of QA—AR Conv—re-QA, directly outputting the confidence level of the judgment result and mapping low / medium / high-risk alarm levels and evidence layers according to rules, automatically dispatching tasks to the maintenance end. A schematic diagram of the final result is shown below. Figures 7 to 9 As shown. Among them, Figure 7 and Figure 8 The test results indicate a medium-risk warning. Figure 9 The option result is a high-risk warning.

[0134] In some embodiments of this application, the inspection system also provides, for example... Figure 10 The inspection interface shown is aggregating multi-view identification results at the station level and linking work orders and handling time limits according to risk level, driving the implementation of downstream closed-loop indicators. This can form an hourly closed loop of "collection—return transmission—identification—alarm—dispatch—return order". Additionally, this interface allows setting a unified closed-loop timeframe: in routine inspections, the overall process of collection → return transmission → identification → dispatch should be ≤60 minutes (with single-frame inference latency ≤10ms and single-site task return link bandwidth requirement not exceeding 500Mbps).

[0135] As can be seen, the above-mentioned inspection system can achieve the entire process without relying on manual inspection, and abandon the traditional physical threshold judgment such as spotlight offset angle. It can directly complete the determination of spotlight attitude offset and fall risk by end-to-end visual feature learning, which significantly improves operation and maintenance efficiency.

[0136] This application provides an inspection device. Figure 11 This is a schematic diagram of the device. From Figure 11 As can be seen from the diagram, the device includes: a first processing module 110, used to determine a window of interest and a mask weight map, global features and local texture features of the target inspection image in the target inspection image, wherein the window of interest is used to identify the image region corresponding to the object to be detected in the target inspection image, and the axis of the window of interest and the image region are geometrically aligned; the mask weight map is used to represent the importance of each region in the target inspection image; and a second processing module 112, used to determine a set of basic sampling points in the global features based on the axis direction of the window of interest and the axis, wherein the receptive field of the set of basic sampling points in the axis direction is greater than that in the vertical direction. The receptive field is perpendicular to the axis direction; the third processing module 114 is used to correct the position of the sampling points in the basic sampling point set based on local texture features, and to determine the importance index of the sampling points to obtain the target sampling point set; the fourth processing module 116 is used to extract target detail features from global features based on the target sampling point set and the mask weight map, wherein the target detail features include the detail features of the object to be detected along the axis direction; the fifth processing module 118 is used to fuse global features, local texture features and target detail features to obtain target fused features, and to determine the risk information of the object to be detected based on the target fused features.

[0137] In some embodiments of this application, the step of the first processing module 110 determining the window of interest in the target inspection image includes: using a geometric polygon attention mechanism to determine the window of interest in the target inspection image. The step of the first processing module 110 determining the mask weight map includes: determining a multi-scale image feature set of the target inspection image, wherein the multi-scale image feature set includes a first image feature, a second image feature, and a third image feature, wherein the first image feature is an image feature obtained after downsampling the target inspection image, the second image feature is an image feature obtained after downsampling the first image feature, and the third image feature is an image feature obtained after downsampling the second image feature; generating a prediction weight map based on the second image feature, and determining elliptical Gaussian prior information stretched along the axial direction based on the second image feature, wherein the elliptical Gaussian prior information is used to represent the axial direction, and the prediction weight map is used to indicate the importance of each region in the target inspection image; fusing the prediction weight map and the elliptical Gaussian prior information to obtain the mask weight map.

[0138] In some embodiments of this application, the steps of the first processing module 110 in determining the global features and local texture features of the target inspection image include: based on the window of interest, performing aligned sampling on the calibrated second image features within the window of interest to obtain a second sampling result; performing residual calculation on the second sampling result based on the mask weight map to obtain global features; based on the window of interest, performing aligned sampling on the calibrated third image features within the window of interest to obtain a third sampling result; and performing residual calculation on the third sampling result based on the mask weight map to obtain local texture features.

[0139] In some embodiments of this application, the step of the fourth processing module 116 extracting target detail features from global features based on the target sampling point set and the mask weight map includes: performing convolution calculation on global features using an adaptive rectangular convolution method based on the position and importance index of the sampling points in the target sampling point set to obtain a first sampling result, wherein the importance index of the sampling point is used to reflect the importance of the feature at the sampling point; and performing residual weight calculation on the first sampling result based on the mask weight map to obtain the target detail features.

[0140] In some embodiments of this application, the fifth processing module 118 fuses global features, local texture features, and target detail features to obtain target fused features. The steps include: determining the reference axis direction of the axis based on the global features, and determining the deviation vector between the reference axis direction and the axis direction; correcting the interest window based on the deviation vector, and generating a soft mask weight map corresponding to the reference axis direction based on the corrected interest window; extracting target local texture features from the target inspection image based on the soft mask weight map; and fusing global features, target local texture features, target detail features, and downsampled feature map of the target inspection image to obtain target fused features, wherein the downsampled feature map is the feature map obtained by directly downsampling the target inspection image.

[0141] In some embodiments of this application, the step of the fifth processing module 118 in determining the risk information of the object to be detected based on the target fusion features includes: determining the location information of the first classification confidence, the second classification confidence, and the abnormal region in the target inspection image based on the target fusion features, wherein the abnormal region is a region with safety risks, the first classification confidence includes the attitude offset confidence of the object to be detected, and the second classification confidence includes the fall risk confidence of the object to be detected; if the first classification confidence is greater than the first confidence threshold and the second classification confidence is greater than the second confidence threshold, determining the risk level of the object to be detected based on the first classification confidence and the second classification confidence, and using the risk level as risk information, wherein the risk level is positively correlated with the probability of an accident occurring in the object to be detected, and the values ​​of the first classification confidence and the second classification confidence are positively correlated with the risk level; if the first classification confidence is less than the first confidence threshold or the second classification confidence is less than the second confidence threshold, updating the axis direction based on the reference axis direction, and redetermining the basic sampling point set based on the updated axis direction, and then redetermining the target fusion features based on the redetermined basic sampling point set.

[0142] In some embodiments of this application, the fifth processing module 118 determines the first classification confidence, the second classification confidence, and the location information of the abnormal region in the target inspection image based on the target fusion features. This includes: performing linear projection processing and global average pooling processing on the target fusion features to obtain the target semantic vector; and processing the target semantic vector using an anchorless detection head to obtain the first classification confidence, the second classification confidence, and the location information of the abnormal region in the target inspection image.

[0143] In some embodiments of this application, after determining the risk information of the object to be detected based on the target fusion features, the fifth processing module 118 is further configured to: determine the operation and maintenance personnel information associated with the object to be detected; generate an operation and maintenance work order corresponding to the risk level, and dispatch the operation and maintenance work order based on the operation and maintenance personnel information.

[0144] It should be noted that each module in the above-mentioned inspection device can be a program module (for example, a set of program instructions to implement a certain function) or a hardware module. For the latter, it can be manifested in the following forms, but is not limited to them: each of the above modules is manifested as a processor, or the functions of each of the above modules are implemented by a processor.

[0145] According to an embodiment of this application, a non-volatile storage medium is provided, which stores a program. During program execution, the device containing the non-volatile storage medium performs the following inspection method: In a target inspection image, a focus window and a mask weight map, global features, and local texture features of the target inspection image are determined. The focus window is used to identify the image region corresponding to the object to be detected in the target inspection image, and the axis of the focus window and the image region are geometrically aligned. The mask weight map is used to represent the importance of each region in the target inspection image. Based on the axis direction of the focus window and the axis, a set of basic sampling points is determined in the global features. The receptive field of the basic sampling point set in the axis direction is larger than the receptive field perpendicular to the axis direction. Based on the local texture features, the positions of the sampling points in the basic sampling point set are corrected, and the importance index of the sampling points is determined to obtain a target sampling point set. Based on the target sampling point set and the mask weight map, target detail features are extracted from the global features. The target detail features include detail features of the object to be detected along the axis direction. The global features, local texture features, and target detail features are fused to obtain target fused features, and the risk information of the object to be detected is determined based on the target fused features.

[0146] Alternatively, the following inspection model training method can be implemented: Determine the shape consistency loss function and region overlap loss function of the inspection model, where the shape consistency loss function is used to represent the degree of geometric deviation between the interest window and the ground truth bounding box corresponding to the object to be detected, and the region overlap loss function is used to represent the degree of spatial deviation between the interest window and the sampling domain corresponding to the target sampling point set; perform a weighted summation of the shape consistency loss function and the region overlap loss function to obtain the comprehensive loss function; train the inspection model based on the comprehensive loss function.

[0147] According to an embodiment of this application, an electronic device is provided, including a memory and a processor. The processor is used to run a program stored in the memory, wherein the program executes the following inspection method during runtime: determining a window of interest and a mask weight map, global features and local texture features of the target inspection image in a target inspection image, wherein the window of interest is used to identify the image region corresponding to the object to be detected in the target inspection image, and the axis of the window of interest and the image region are geometrically aligned; the mask weight map is used to represent the importance of each region in the target inspection image; determining a set of basic sampling points in the global features based on the axis direction of the window of interest and the axis, wherein the receptive field of the set of basic sampling points in the axis direction is larger than the receptive field perpendicular to the axis direction; correcting the position of the sampling points in the set of basic sampling points based on the local texture features, and determining the importance index of the sampling points to obtain a target sampling point set; extracting target detail features from the global features based on the target sampling point set and the mask weight map, wherein the target detail features include detail features of the object to be detected along the axis direction; fusing the global features, local texture features and target detail features to obtain target fused features, and determining the risk information of the object to be detected based on the target fused features.

[0148] Alternatively, the following inspection model training method can be implemented: Determine the shape consistency loss function and region overlap loss function of the inspection model, where the shape consistency loss function is used to represent the degree of geometric deviation between the interest window and the ground truth bounding box corresponding to the object to be detected, and the region overlap loss function is used to represent the degree of spatial deviation between the interest window and the sampling domain corresponding to the target sampling point set; perform a weighted summation of the shape consistency loss function and the region overlap loss function to obtain the comprehensive loss function; train the inspection model based on the comprehensive loss function.

[0149] According to an embodiment of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the following steps of an inspection method: determining a focus window and a mask weight map, global features and local texture features of the target inspection image in the target inspection image, wherein the focus window is used to identify the image region corresponding to the object to be detected in the target inspection image, and the axis of the focus window and the image region are geometrically aligned; the mask weight map is used to represent the importance of each region in the target inspection image; determining a set of basic sampling points in the global features based on the axis direction of the focus window and the axis, wherein the receptive field of the set of basic sampling points in the axis direction is larger than the receptive field perpendicular to the axis direction; correcting the position of the sampling points in the set of basic sampling points based on the local texture features, and determining the importance index of the sampling points to obtain a target sampling point set; extracting target detail features from the global features based on the target sampling point set and the mask weight map, wherein the target detail features include detail features of the object to be detected along the axis direction; fusing the global features, local texture features and target detail features to obtain target fused features, and determining the risk information of the object to be detected based on the target fused features.

[0150] Alternatively, the following steps can be taken to train the inspection model: Determine the shape consistency loss function and the region overlap loss function of the inspection model, where the shape consistency loss function represents the degree of geometric deviation between the interest window and the ground truth bounding box corresponding to the object to be detected, and the region overlap loss function represents the degree of spatial deviation between the interest window and the sampling domain corresponding to the target sampling point set; perform a weighted summation of the shape consistency loss function and the region overlap loss function to obtain the comprehensive loss function; train the inspection model based on the comprehensive loss function.

[0151] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0152] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0153] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0154] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0155] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0156] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An inspection method, characterized in that, include: In the target inspection image, a window of interest and a mask weight map are determined, which are the global features and local texture features of the target inspection image. The window of interest is used to identify the image region corresponding to the object to be detected in the target inspection image, and the axis of the window of interest and the image region are geometrically aligned. The mask weight map is used to represent the importance of each region in the target inspection image. Based on the focus window and the axial direction of the axis, a set of basic sampling points is determined in the global features, wherein the receptive field of the set of basic sampling points in the axial direction is larger than the receptive field perpendicular to the axial direction. The positions of the sampling points in the basic sampling point set are corrected based on the local texture features, and the importance index of the sampling points is determined to obtain the target sampling point set; Based on the target sampling point set and the mask weight map, target detail features are extracted from the global features, wherein the target detail features include the detail features of the object to be detected along the axis direction; The global features, the local texture features, and the target detail features are fused to obtain the target fusion features, and the risk information of the object to be detected is determined based on the target fusion features.

2. The inspection method according to claim 1, characterized in that, Extracting target detail features from the global features based on the target sampling point set and the mask weight map includes: Based on the position and importance index of the sampling points in the target sampling point set, the global features are convolved using an adaptive rectangular convolution method to obtain the first sampling result. The importance index of the sampling points is used to reflect the importance of the features at the sampling points. The target detail features are obtained by performing residual weighting calculation on the first sampling result based on the mask weight map.

3. The inspection method according to claim 1, characterized in that, By fusing the global features, the local texture features, and the target detail features, the target fusion features are obtained, including: Based on the global features, the reference axis direction of the axis is determined, and the deviation vector between the reference axis direction and the axis direction is determined; The focus window is corrected based on the deviation vector, and a soft mask weight map corresponding to the direction of the reference axis is generated based on the corrected focus window. Based on the soft mask weight map, local texture features of the target are extracted from the target inspection image; The target fusion feature is obtained by fusing the global feature, the target local texture feature, the target detail feature, and the downsampled feature map of the target inspection image. The downsampled feature map is the feature map obtained by directly downsampling the target inspection image.

4. The inspection method according to claim 3, characterized in that, Determining the risk information of the object to be detected based on the target fusion features includes: Based on the target fusion features, a first classification confidence level, a second classification confidence level, and the location information of abnormal regions in the target inspection image are determined. The abnormal regions are regions with safety risks. The first classification confidence level includes the attitude offset confidence level of the object to be detected, and the second classification confidence level includes the fall risk confidence level of the object to be detected. When the confidence level of the first category is greater than the first confidence threshold and the confidence level of the second category is greater than the second confidence threshold, the risk level of the object to be detected is determined based on the confidence level of the first category and the confidence level of the second category, and the risk level is used as the risk information. The risk level is positively correlated with the probability of the accident occurring in the object to be detected, and the values ​​of the confidence level of the first category and the confidence level of the second category are positively correlated with the risk level. If the confidence level of the first classification is less than the first confidence level threshold, or the confidence level of the second classification is less than the second confidence level threshold, the axis direction is updated according to the reference axis direction, and the basic sampling point set is re-determined according to the updated axis direction, and then the target fusion feature is re-determined according to the re-determined basic sampling point set.

5. The inspection method according to claim 4, characterized in that, Determining the first classification confidence, the second classification confidence, and the location information of the abnormal region in the target inspection image based on the target fusion features includes: The target fusion features are subjected to linear projection and global average pooling to obtain the target semantic vector; The target semantic vector is processed using an anchorless detection head to obtain the first classification confidence, the second classification confidence, and the location information of the abnormal region in the target inspection image.

6. The inspection method according to claim 4, characterized in that, After determining the risk information of the object to be detected based on the target fusion features, the method further includes: Determine the maintenance personnel information associated with the object to be detected; Generate an operation and maintenance work order corresponding to the risk level, and dispatch the operation and maintenance work order based on the operation and maintenance personnel information.

7. The inspection method according to claim 1, characterized in that, Determining the window of interest in the target inspection image includes: using a geometric polygon attention mechanism to determine the window of interest in the target inspection image; The mask weight map is determined as follows: A multi-scale image feature set of the target inspection image is determined, wherein the multi-scale image feature set includes a first image feature, a second image feature, and a third image feature. The first image feature is an image feature obtained after downsampling the target inspection image, the second image feature is an image feature obtained after downsampling the first image feature, and the third image feature is an image feature obtained after downsampling the second image feature. A prediction weight map is generated based on the second image feature, and elliptical Gaussian prior information elongated along the axis direction is determined based on the second image feature. The elliptical Gaussian prior information is used to represent the axis direction, and the prediction weight map is used to indicate the importance of each region in the target inspection image. The prediction weight map and the elliptical Gaussian prior information are fused to obtain the mask weight map.

8. The inspection method according to claim 7, characterized in that, Determining the global and local texture features of the target inspection image includes: Based on the attention window, the calibrated second image features are aligned and sampled within the attention window to obtain a second sampling result; The global features are obtained by performing residual calculation on the second sampling result based on the mask weight map; Based on the attention window, the calibrated third image features are aligned and sampled within the attention window to obtain the third sampling result; The local texture features are obtained by performing residual calculation on the third sampling result based on the mask weight map.

9. A method for training an inspection model, wherein the inspection model is used to execute the inspection method according to any one of claims 1 to 8, characterized in that, include: The shape consistency loss function and the region overlap loss function of the inspection model are determined, wherein the shape consistency loss function is used to represent the degree of geometric deviation between the attention window and the ground truth bounding box corresponding to the object to be detected, and the region overlap loss function is used to represent the degree of spatial deviation between the attention window and the sampling domain corresponding to the target sampling point set. The shape consistency loss function and the region overlap loss function are weighted and summed to obtain the comprehensive loss function; The inspection model is trained based on the comprehensive loss function.

10. The inspection model training method according to claim 9, characterized in that, The shape consistency loss function is determined in the following way: Determine the degree of overlap between the coverage areas of the attention window and the actual annotation box; Determine the distance between the center of the focus window and the center of the actual annotation box; Determine the diagonal length of the bounding union rectangle of the attention window and the actual annotation box; Determine the deviation between the first aspect ratio of the focus window and the second aspect ratio of the actual annotation box, wherein the first aspect ratio is the ratio of the long side to the short side of the focus window, and the second aspect ratio is the ratio of the long side to the short side of the actual annotation box. The shape consistency loss function is determined based on the degree of overlap of the coverage area, the distance, the diagonal length, and the deviation value.

11. The inspection model training method according to claim 9, characterized in that, The region overlap loss function is determined in the following way: Determine the overlap evaluation index of the sampling domain corresponding to the attention window and the target sampling point set at each feature level, wherein the sum of the overlap evaluation index and the intersection-union ratio of the sampling domain corresponding to the attention window and the target sampling point set at each feature level is 1; The average value of the overlap evaluation index corresponding to each feature level is determined as the regional overlap loss function.

12. An inspection device, characterized in that, include: The first processing module is used to determine a window of interest and a mask weight map in a target inspection image, the global features and local texture features of the target inspection image, wherein the window of interest is used to identify the image region corresponding to the object to be detected in the target inspection image, and the axis of the window of interest and the image region are geometrically aligned, and the mask weight map is used to represent the importance of each region in the target inspection image; The second processing module is used to determine a set of basic sampling points in the global features based on the focus window and the axial direction of the axis, wherein the receptive field of the set of basic sampling points in the axial direction is larger than the receptive field perpendicular to the axial direction. The third processing module is used to correct the position of the sampling points in the basic sampling point set based on the local texture features, and to determine the importance index of the sampling points to obtain the target sampling point set. The fourth processing module is used to extract target detail features from the global features based on the target sampling point set and the mask weight map, wherein the target detail features include the detail features of the object to be detected along the axis direction; The fifth processing module is used to fuse the global features, the local texture features, and the target detail features to obtain target fusion features, and to determine the risk information of the object to be detected based on the target fusion features.

13. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a program, wherein when the program is executed, it controls the device where the non-volatile storage medium is located to execute the inspection method according to any one of claims 1 to 8, or the inspection model training method according to any one of claims 9 to 11.

14. An electronic device, characterized in that, include: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, executes the inspection method according to any one of claims 1 to 8, or the inspection model training method according to any one of claims 9 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the inspection method according to any one of claims 1 to 8, or the inspection model training method according to any one of claims 9 to 11.

Images